2024-04-25T13:23:35.417 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T13:23:35.417 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/orig.config.yaml 2024-04-25T13:23:35.470 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T13:23:35.550 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} email: pdonnell@redhat.com first_in_suite: false job_id: '7673154' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0Rp4nG6wfJRTH21CjY/jOug2i9Xd3XmxmZyX72e1grA151CMFx3BjgchHS2JFTcXw0jGZolv0a4ZraQoGdH3I= smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDi7JqY/3/aha3FpS+Jt9r+QK0qS8RQTaE25tDThGS+3YwsrdHfbDxJ0wUdrcXi1zJtRsYV7q+aJ8I5QiHh6+j0= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEkzGXlvnf712wuvBE0c95zh17h554qz5zpmvJAp/jXZWbZjbUDd2v6HGDfqPcFaOkUSt6IYEyGFQ6WgSW0YsQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T13:23:35.550 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T13:23:35.551 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T13:23:35.551 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T13:23:35.553 INFO:teuthology.task.internal:Checking packages... 2024-04-25T13:23:35.577 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T13:23:35.577 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T13:23:35.578 INFO:teuthology.packaging:ref: None 2024-04-25T13:23:35.578 INFO:teuthology.packaging:tag: None 2024-04-25T13:23:35.578 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T13:23:35.578 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:23:35.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T13:23:35.718 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T13:23:35.719 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T13:23:35.725 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T13:23:35.725 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T13:23:35.779 INFO:teuthology.task.internal:Saving configuration 2024-04-25T13:23:35.799 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T13:23:35.805 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T13:23:35.830 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 13:17:25.809231', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0Rp4nG6wfJRTH21CjY/jOug2i9Xd3XmxmZyX72e1grA151CMFx3BjgchHS2JFTcXw0jGZolv0a4ZraQoGdH3I='} 2024-04-25T13:23:35.851 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 13:17:25.806957', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDi7JqY/3/aha3FpS+Jt9r+QK0qS8RQTaE25tDThGS+3YwsrdHfbDxJ0wUdrcXi1zJtRsYV7q+aJ8I5QiHh6+j0='} 2024-04-25T13:23:35.872 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 13:17:25.810999', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEkzGXlvnf712wuvBE0c95zh17h554qz5zpmvJAp/jXZWbZjbUDd2v6HGDfqPcFaOkUSt6IYEyGFQ6WgSW0YsQ='} 2024-04-25T13:23:35.872 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T13:23:35.878 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T13:23:35.878 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T13:23:35.878 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T13:23:35.878 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T13:23:36.006 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7c4ebdd0d0>, signals=[15]) 2024-04-25T13:23:36.007 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T13:23:36.013 INFO:teuthology.task.internal:Opening connections... 2024-04-25T13:23:36.013 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T13:23:36.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:23:36.089 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-04-25T13:23:36.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:23:36.166 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-04-25T13:23:36.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:23:36.236 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T13:23:36.245 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-04-25T13:23:36.263 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-04-25T13:23:36.263 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-04-25T13:23:36.321 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-04-25T13:23:36.321 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T13:23:36.322 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T13:23:36.323 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T13:23:36.323 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-04-25T13:23:36.344 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-04-25T13:23:36.362 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-04-25T13:23:36.362 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-04-25T13:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-04-25T13:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-04-25T13:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-04-25T13:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-04-25T13:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-04-25T13:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-04-25T13:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T13:23:36.420 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-04-25T13:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-04-25T13:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T13:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-04-25T13:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T13:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T13:23:36.421 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T13:23:36.421 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-04-25T13:23:36.444 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-04-25T13:23:36.461 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-04-25T13:23:36.461 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-04-25T13:23:36.518 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T13:23:36.519 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-04-25T13:23:36.519 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T13:23:36.519 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T13:23:36.519 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T13:23:36.519 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-04-25T13:23:36.535 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T13:23:36.545 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T13:23:36.564 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T13:23:36.564 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-04-25T13:23:36.567 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-04-25T13:23:36.569 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-04-25T13:23:36.584 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T13:23:36.592 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T13:23:36.592 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-04-25T13:23:36.624 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-04-25T13:23:36.626 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-04-25T13:23:36.703 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T13:23:36.775 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T13:23:36.790 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T13:23:36.790 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-25T13:23:36.791 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-25T13:23:36.791 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T13:23:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:23:36.809 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-04-25T13:23:36.866 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-437.el9.x86_64 2024-04-25T13:23:36.866 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-437.el9.x86_64 2024-04-25T13:23:36.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-04-25T13:23:39.482 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-04-25T13:23:41.359 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-25T13:23:45.751 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-04-25T13:23:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-04-25T13:23:48.617 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-25T13:23:53.767 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-25T13:23:55.757 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T13:23:55.757 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T13:23:55.852 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T13:23:55.853 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-25T13:23:55.853 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-25T13:23:55.964 DEBUG:teuthology.orchestra.run.smithi027:> echo no | sudo yum reinstall kernel || true 2024-04-25T13:23:56.488 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:23:53 PM UTC. 2024-04-25T13:23:56.614 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T13:23:56.620 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T13:23:56.620 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalling: 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 255 k 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 0 2024-04-25T13:23:56.621 INFO:teuthology.orchestra.run.smithi027.stderr:Operation aborted. 2024-04-25T13:23:56.688 INFO:teuthology.orchestra.run.smithi027.stdout:Is this ok [y/N]: 2024-04-25T13:23:56.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum reinstall -y kernel || true 2024-04-25T13:23:57.204 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 01:23:53 PM UTC. 2024-04-25T13:23:57.325 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T13:23:57.325 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T13:23:57.325 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-25T13:23:57.325 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T13:23:57.326 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalling: 2024-04-25T13:23:57.326 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T13:23:57.326 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:23:57.326 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-25T13:23:57.326 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-25T13:23:57.326 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:23:57.326 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 255 k 2024-04-25T13:23:57.327 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 0 2024-04-25T13:23:57.327 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-25T13:23:57.611 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.9 MB/s | 255 kB 00:00 2024-04-25T13:23:57.612 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-25T13:23:57.612 INFO:teuthology.orchestra.run.smithi027.stdout:Total 891 kB/s | 255 kB 00:00 2024-04-25T13:23:57.633 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-25T13:23:57.817 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-25T13:23:57.817 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-25T13:23:57.822 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-25T13:23:57.823 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-25T13:23:57.954 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-25T13:23:58.062 INFO:teuthology.orchestra.run.smithi027.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T13:23:58.063 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:23:58.768 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:23:58.768 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T13:23:59.451 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:23:59.451 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:23:59.452 INFO:teuthology.orchestra.run.smithi027.stdout:Reinstalled: 2024-04-25T13:23:59.452 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T13:23:59.452 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:23:59.452 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-25T13:23:59.608 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T13:23:59.656 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T13:23:59.656 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T13:23:59.656 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T13:23:59.656 DEBUG:teuthology.parallel:result is None 2024-04-25T13:23:59.657 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T13:23:59.665 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T13:23:59.665 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T13:23:59.668 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T13:23:59.670 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T13:23:59.688 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T13:23:59.695 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T13:23:59.702 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T13:23:59.702 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T13:23:59.723 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T13:23:59.729 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T13:23:59.763 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T13:23:59.770 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T13:23:59.770 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T13:23:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:23:59.808 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T13:23:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:23:59.825 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T13:23:59.827 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T13:23:59.851 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T13:23:59.854 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:23:59.873 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:23:59.884 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:23:59.903 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:23:59.906 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:23:59.922 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:23:59.923 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T13:23:59.931 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T13:23:59.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T13:23:59.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T13:23:59.951 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T13:23:59.990 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T13:23:59.999 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T13:23:59.999 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T13:24:00.001 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T13:24:00.020 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T13:24:00.047 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:24:00.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:24:00.149 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:24:00.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:24:00.270 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:24:00.270 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T13:24:00.334 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:24:00.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:24:00.434 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:24:00.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:24:00.563 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:24:00.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T13:24:00.632 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:24:00.675 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:24:00.723 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:24:00.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:24:00.849 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:24:00.849 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T13:24:00.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-04-25T13:24:00.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-04-25T13:24:00.921 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-04-25T13:24:00.961 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T13:24:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T13:24:00.993 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T13:24:01.437 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T13:24:01.445 INFO:teuthology.task.internal:Starting timer... 2024-04-25T13:24:01.445 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T13:24:01.460 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T13:24:01.468 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T13:24:01.468 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-04-25T13:24:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-04-25T13:24:01.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-04-25T13:24:01.575 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-04-25T13:24:01.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-04-25T13:24:01.652 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-04-25T13:24:01.654 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T13:24:01.654 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-04-25T13:24:01.675 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-04-25T13:24:01.676 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-04-25T13:24:01.699 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-04-25T13:24:01.699 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-04-25T13:24:01.720 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-04-25T13:24:01.720 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-04-25T13:24:01.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T13:24:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:24:01.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T13:24:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:24:01.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T13:24:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:24:01.805 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T13:24:01.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-04-25T13:24:01.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-04-25T13:24:01.866 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-04-25T13:24:01.906 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T13:24:01.915 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T13:24:01.920 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T13:24:02.075 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T13:24:02.090 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T13:24:02.091 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi045.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T13:27:53.769 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-04-25T13:27:53.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-25T13:27:53.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:27:53.848 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-04-25T13:27:53.927 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-25T13:27:53.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-25T13:27:53.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:27:54.007 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-25T13:27:54.104 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-25T13:27:54.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-25T13:27:54.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:27:54.200 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-25T13:27:54.237 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-25T13:27:54.238 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T13:27:54.252 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T13:27:54.252 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T13:27:54.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T13:27:54.255 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T13:27:54.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T13:27:54.258 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T13:27:54.258 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T13:27:54.288 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T13:27:54.289 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T13:27:54.302 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T13:27:54.303 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T13:27:54.308 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T13:27:54.322 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T13:27:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-04-25T13:27:54.328 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-04-25T13:27:54.338 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-04-25T13:27:54.340 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-04-25T13:27:54.346 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-04-25T13:27:54.354 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T13:27:54.355 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T13:27:54.356 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-04-25T13:27:54.367 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T13:27:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T13:27:54.370 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T13:27:54.383 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T13:27:54.425 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-04-25T13:27:54.428 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T13:27:54.428 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-04-25T13:27:54.428 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.428 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.428 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.428 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-25T13:27:54.434 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T13:27:54.434 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-25T13:27:54.434 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.434 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.434 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.434 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.435 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-25T13:27:54.437 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T13:27:54.437 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-25T13:27:54.437 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.437 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.437 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.437 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:27:54.438 INFO:teuthology.run_tasks:Running task install... 2024-04-25T13:27:54.448 DEBUG:teuthology.task.install:project ceph 2024-04-25T13:27:54.449 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T13:27:54.449 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T13:27:54.449 INFO:teuthology.task.install:Using flavor: default 2024-04-25T13:27:54.460 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T13:27:54.461 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T13:27:54.461 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T13:27:54.462 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:27:54.465 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T13:27:54.465 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:27:54.467 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T13:27:54.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:27:54.628 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T13:27:54.629 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T13:27:54.642 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T13:27:54.642 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T13:27:54.662 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T13:27:54.662 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T13:27:54.770 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T13:27:54.770 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:27:54.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T13:27:54.781 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T13:27:54.781 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:27:54.781 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T13:27:54.797 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T13:27:54.797 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T13:27:54.806 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T13:27:54.806 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T13:27:54.817 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T13:27:54.817 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:27:54.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T13:27:54.843 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T13:27:54.844 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T13:27:54.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T13:27:54.877 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T13:27:54.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T13:27:54.938 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T13:27:54.950 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T13:27:54.985 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T13:27:55.002 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-04-25T13:27:55.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-04-25T13:27:55.013 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-04-25T13:27:55.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-04-25T13:27:55.049 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-04-25T13:27:55.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-04-25T13:27:55.238 INFO:teuthology.orchestra.run.smithi027.stdout:48 files removed 2024-04-25T13:27:55.239 INFO:teuthology.orchestra.run.smithi122.stdout:48 files removed 2024-04-25T13:27:55.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T13:27:55.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T13:27:55.283 INFO:teuthology.orchestra.run.smithi045.stdout:48 files removed 2024-04-25T13:27:55.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T13:27:55.883 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 236 kB/s | 84 kB 00:00 2024-04-25T13:27:55.948 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 203 kB/s | 84 kB 00:00 2024-04-25T13:27:55.993 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 205 kB/s | 84 kB 00:00 2024-04-25T13:27:56.242 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-25T13:27:56.361 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-04-25T13:27:56.374 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-25T13:27:56.726 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-04-25T13:27:56.786 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-04-25T13:27:56.908 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-04-25T13:27:57.520 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T13:27:57.643 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T13:27:57.889 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-04-25T13:27:59.383 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-25T13:27:59.603 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-25T13:27:59.848 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-25T13:28:03.673 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.2 MB 00:00 2024-04-25T13:28:03.979 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.2 MB 00:00 2024-04-25T13:28:04.299 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-04-25T13:28:05.254 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-25T13:28:05.778 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-25T13:28:05.905 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-25T13:28:06.609 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-25T13:28:07.123 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-25T13:28:07.632 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-25T13:28:11.598 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T13:28:12.412 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-25T13:28:12.899 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-25T13:28:13.741 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:28:13.741 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:28:13.752 INFO:teuthology.orchestra.run.smithi122.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T13:28:13.753 INFO:teuthology.orchestra.run.smithi122.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T13:28:13.754 INFO:teuthology.orchestra.run.smithi122.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T13:28:13.755 INFO:teuthology.orchestra.run.smithi122.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T13:28:13.756 INFO:teuthology.orchestra.run.smithi122.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T13:28:13.759 INFO:teuthology.orchestra.run.smithi122.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T13:28:13.759 INFO:teuthology.orchestra.run.smithi122.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T13:28:13.820 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T13:28:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T13:28:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T13:28:13.844 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T13:28:13.845 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T13:28:13.846 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T13:28:13.847 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T13:28:13.848 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T13:28:13.849 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T13:28:13.850 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T13:28:13.851 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T13:28:13.852 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T13:28:13.853 INFO:teuthology.orchestra.run.smithi122.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout:Install 148 Packages 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 4 Packages 2024-04-25T13:28:13.854 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:28:13.855 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 214 M 2024-04-25T13:28:13.855 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-25T13:28:14.640 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:28:14.640 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:28:14.652 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T13:28:14.652 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T13:28:14.653 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T13:28:14.654 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T13:28:14.657 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T13:28:14.658 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T13:28:14.659 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T13:28:14.725 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-25T13:28:14.735 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T13:28:14.735 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-25T13:28:14.735 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T13:28:14.735 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-25T13:28:14.735 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T13:28:14.736 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T13:28:14.737 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T13:28:14.738 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T13:28:14.739 INFO:teuthology.orchestra.run.smithi027.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T13:28:14.739 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T13:28:14.739 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-25T13:28:14.739 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T13:28:14.739 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T13:28:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T13:28:14.741 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T13:28:14.742 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T13:28:14.743 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T13:28:14.744 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T13:28:14.745 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T13:28:14.746 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T13:28:14.747 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T13:28:14.748 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-04-25T13:28:14.748 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T13:28:14.748 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:28:14.748 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-25T13:28:14.748 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-25T13:28:14.748 INFO:teuthology.orchestra.run.smithi027.stdout:Install 148 Packages 2024-04-25T13:28:14.748 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 4 Packages 2024-04-25T13:28:14.748 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:28:14.749 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 214 M 2024-04-25T13:28:14.749 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-25T13:28:14.808 INFO:teuthology.orchestra.run.smithi122.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-25T13:28:15.116 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:28:15.117 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:28:15.128 INFO:teuthology.orchestra.run.smithi045.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T13:28:15.129 INFO:teuthology.orchestra.run.smithi045.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T13:28:15.130 INFO:teuthology.orchestra.run.smithi045.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T13:28:15.131 INFO:teuthology.orchestra.run.smithi045.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T13:28:15.132 INFO:teuthology.orchestra.run.smithi045.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T13:28:15.134 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T13:28:15.135 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T13:28:15.141 INFO:teuthology.orchestra.run.smithi122.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 2.5 MB/s | 863 kB 00:00 2024-04-25T13:28:15.202 INFO:teuthology.orchestra.run.smithi122.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 2.5 MB/s | 147 kB 00:00 2024-04-25T13:28:15.202 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-04-25T13:28:15.219 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T13:28:15.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T13:28:15.221 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-04-25T13:28:15.222 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T13:28:15.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T13:28:15.224 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T13:28:15.225 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T13:28:15.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T13:28:15.227 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T13:28:15.228 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T13:28:15.229 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T13:28:15.230 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-04-25T13:28:15.231 INFO:teuthology.orchestra.run.smithi045.stdout:Install 148 Packages 2024-04-25T13:28:15.232 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 4 Packages 2024-04-25T13:28:15.232 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:28:15.232 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 214 M 2024-04-25T13:28:15.232 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-04-25T13:28:15.367 INFO:teuthology.orchestra.run.smithi122.stdout:(4/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 7.3 MB/s | 5.4 MB 00:00 2024-04-25T13:28:15.464 INFO:teuthology.orchestra.run.smithi027.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-04-25T13:28:15.559 INFO:teuthology.orchestra.run.smithi122.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-04-25T13:28:15.909 INFO:teuthology.orchestra.run.smithi122.stdout:(6/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 3.2 MB/s | 2.3 MB 00:00 2024-04-25T13:28:16.110 INFO:teuthology.orchestra.run.smithi122.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 8.9 MB/s | 4.9 MB 00:00 2024-04-25T13:28:16.125 INFO:teuthology.orchestra.run.smithi045.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-04-25T13:28:16.297 INFO:teuthology.orchestra.run.smithi027.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 1.0 MB/s | 863 kB 00:00 2024-04-25T13:28:16.439 INFO:teuthology.orchestra.run.smithi027.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 1.0 MB/s | 147 kB 00:00 2024-04-25T13:28:16.785 INFO:teuthology.orchestra.run.smithi122.stdout:(8/152): ceph-common-19.0.0-3291.g994e2f20.el9. 9.1 MB/s | 20 MB 00:02 2024-04-25T13:28:16.911 INFO:teuthology.orchestra.run.smithi122.stdout:(9/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 13 MB/s | 10 MB 00:00 2024-04-25T13:28:16.936 INFO:teuthology.orchestra.run.smithi122.stdout:(10/152): ceph-selinux-19.0.0-3291.g994e2f20.el 168 kB/s | 25 kB 00:00 2024-04-25T13:28:17.028 INFO:teuthology.orchestra.run.smithi122.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.4 MB/s | 224 kB 00:00 2024-04-25T13:28:17.120 INFO:teuthology.orchestra.run.smithi122.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 294 kB/s | 27 kB 00:00 2024-04-25T13:28:17.220 INFO:teuthology.orchestra.run.smithi122.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 317 kB/s | 32 kB 00:00 2024-04-25T13:28:17.289 INFO:teuthology.orchestra.run.smithi027.stdout:(4/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 2.7 MB/s | 2.3 MB 00:00 2024-04-25T13:28:17.695 INFO:teuthology.orchestra.run.smithi122.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 1.6 MB/s | 756 kB 00:00 2024-04-25T13:28:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:(5/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 2.1 MB/s | 5.4 MB 00:02 2024-04-25T13:28:17.856 INFO:teuthology.orchestra.run.smithi122.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 984 kB/s | 158 kB 00:00 2024-04-25T13:28:18.090 INFO:teuthology.orchestra.run.smithi027.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 1.9 MB/s | 1.5 MB 00:00 2024-04-25T13:28:18.123 INFO:teuthology.orchestra.run.smithi122.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 476 kB/s | 127 kB 00:00 2024-04-25T13:28:18.457 INFO:teuthology.orchestra.run.smithi122.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 1.5 MB/s | 497 kB 00:00 2024-04-25T13:28:18.600 INFO:teuthology.orchestra.run.smithi045.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 349 kB/s | 863 kB 00:02 2024-04-25T13:28:18.699 INFO:teuthology.orchestra.run.smithi122.stdout:(18/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 6.4 MB/s | 18 MB 00:02 2024-04-25T13:28:18.791 INFO:teuthology.orchestra.run.smithi122.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 493 kB/s | 45 kB 00:00 2024-04-25T13:28:18.858 INFO:teuthology.orchestra.run.smithi045.stdout:(3/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 1.8 MB/s | 5.4 MB 00:02 2024-04-25T13:28:18.866 INFO:teuthology.orchestra.run.smithi122.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.8 MB/s | 135 kB 00:00 2024-04-25T13:28:18.884 INFO:teuthology.orchestra.run.smithi045.stdout:(4/152): ceph-immutable-object-cache-19.0.0-329 519 kB/s | 147 kB 00:00 2024-04-25T13:28:18.950 INFO:teuthology.orchestra.run.smithi122.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 1.9 MB/s | 161 kB 00:00 2024-04-25T13:28:19.041 INFO:teuthology.orchestra.run.smithi122.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 3.4 MB/s | 321 kB 00:00 2024-04-25T13:28:19.134 INFO:teuthology.orchestra.run.smithi122.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 3.2 MB/s | 298 kB 00:00 2024-04-25T13:28:19.217 INFO:teuthology.orchestra.run.smithi122.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.2 MB/s | 100 kB 00:00 2024-04-25T13:28:19.301 INFO:teuthology.orchestra.run.smithi122.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.0 MB/s | 86 kB 00:00 2024-04-25T13:28:19.557 INFO:teuthology.orchestra.run.smithi027.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 2.8 MB/s | 4.9 MB 00:01 2024-04-25T13:28:19.734 INFO:teuthology.orchestra.run.smithi045.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 2.6 MB/s | 2.3 MB 00:00 2024-04-25T13:28:19.901 INFO:teuthology.orchestra.run.smithi122.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 5.1 MB/s | 3.0 MB 00:00 2024-04-25T13:28:19.985 INFO:teuthology.orchestra.run.smithi122.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.0 MB/s | 169 kB 00:00 2024-04-25T13:28:20.069 INFO:teuthology.orchestra.run.smithi122.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 410 kB/s | 34 kB 00:00 2024-04-25T13:28:20.152 INFO:teuthology.orchestra.run.smithi122.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.7 MB/s | 142 kB 00:00 2024-04-25T13:28:20.828 INFO:teuthology.orchestra.run.smithi122.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 5.2 MB/s | 3.5 MB 00:00 2024-04-25T13:28:21.109 INFO:teuthology.orchestra.run.smithi045.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 692 kB/s | 1.5 MB 00:02 2024-04-25T13:28:21.545 INFO:teuthology.orchestra.run.smithi122.stdout:(31/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 9.3 MB/s | 43 MB 00:04 2024-04-25T13:28:21.576 INFO:teuthology.orchestra.run.smithi045.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 2.6 MB/s | 4.9 MB 00:01 2024-04-25T13:28:21.612 INFO:teuthology.orchestra.run.smithi122.stdout:(32/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 1.6 MB/s | 5.2 MB 00:03 2024-04-25T13:28:21.637 INFO:teuthology.orchestra.run.smithi122.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.7 MB/s | 249 kB 00:00 2024-04-25T13:28:21.679 INFO:teuthology.orchestra.run.smithi122.stdout:(34/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 738 kB/s | 49 kB 00:00 2024-04-25T13:28:21.713 INFO:teuthology.orchestra.run.smithi122.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3291.g9 218 kB/s | 16 kB 00:00 2024-04-25T13:28:21.871 INFO:teuthology.orchestra.run.smithi122.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 4.7 MB/s | 762 kB 00:00 2024-04-25T13:28:21.905 INFO:teuthology.orchestra.run.smithi122.stdout:(37/152): ceph-volume-19.0.0-3291.g994e2f20.el9 1.2 MB/s | 273 kB 00:00 2024-04-25T13:28:21.997 INFO:teuthology.orchestra.run.smithi122.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 6.3 MB/s | 7.4 MB 00:01 2024-04-25T13:28:22.047 INFO:teuthology.orchestra.run.smithi122.stdout:(39/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 104 kB/s | 18 kB 00:00 2024-04-25T13:28:22.156 INFO:teuthology.orchestra.run.smithi122.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 320 kB/s | 80 kB 00:00 2024-04-25T13:28:22.223 INFO:teuthology.orchestra.run.smithi122.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-25T13:28:22.248 INFO:teuthology.orchestra.run.smithi122.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-04-25T13:28:22.282 INFO:teuthology.orchestra.run.smithi122.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-04-25T13:28:22.349 INFO:teuthology.orchestra.run.smithi122.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.2 MB 00:00 2024-04-25T13:28:22.374 INFO:teuthology.orchestra.run.smithi122.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 854 kB/s | 106 kB 00:00 2024-04-25T13:28:22.400 INFO:teuthology.orchestra.run.smithi122.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-04-25T13:28:22.425 INFO:teuthology.orchestra.run.smithi122.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-25T13:28:22.458 INFO:teuthology.orchestra.run.smithi122.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-25T13:28:22.473 INFO:teuthology.orchestra.run.smithi027.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 3.5 MB/s | 10 MB 00:02 2024-04-25T13:28:22.484 INFO:teuthology.orchestra.run.smithi122.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 8.1 MB/s | 689 kB 00:00 2024-04-25T13:28:22.517 INFO:teuthology.orchestra.run.smithi122.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-04-25T13:28:22.543 INFO:teuthology.orchestra.run.smithi122.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 107 kB 00:00 2024-04-25T13:28:22.559 INFO:teuthology.orchestra.run.smithi122.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 288 kB/s | 21 kB 00:00 2024-04-25T13:28:22.585 INFO:teuthology.orchestra.run.smithi122.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-04-25T13:28:22.602 INFO:teuthology.orchestra.run.smithi122.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 791 kB/s | 33 kB 00:00 2024-04-25T13:28:22.648 INFO:teuthology.orchestra.run.smithi027.stdout:(9/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x86 3.9 MB/s | 18 MB 00:04 2024-04-25T13:28:22.674 INFO:teuthology.orchestra.run.smithi027.stdout:(10/152): ceph-selinux-19.0.0-3291.g994e2f20.el 126 kB/s | 25 kB 00:00 2024-04-25T13:28:22.712 INFO:teuthology.orchestra.run.smithi122.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-04-25T13:28:22.736 INFO:teuthology.orchestra.run.smithi122.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 122 kB/s | 18 kB 00:00 2024-04-25T13:28:22.749 INFO:teuthology.orchestra.run.smithi027.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.9 MB/s | 224 kB 00:00 2024-04-25T13:28:22.761 INFO:teuthology.orchestra.run.smithi122.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 71 kB/s | 11 kB 00:00 2024-04-25T13:28:22.788 INFO:teuthology.orchestra.run.smithi122.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-25T13:28:22.799 INFO:teuthology.orchestra.run.smithi027.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 539 kB/s | 27 kB 00:00 2024-04-25T13:28:22.811 INFO:teuthology.orchestra.run.smithi122.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 908 kB/s | 45 kB 00:00 2024-04-25T13:28:22.837 INFO:teuthology.orchestra.run.smithi122.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-25T13:28:22.841 INFO:teuthology.orchestra.run.smithi027.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 766 kB/s | 32 kB 00:00 2024-04-25T13:28:22.871 INFO:teuthology.orchestra.run.smithi122.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 6.9 MB/s | 582 kB 00:00 2024-04-25T13:28:22.896 INFO:teuthology.orchestra.run.smithi122.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 308 kB/s | 25 kB 00:00 2024-04-25T13:28:22.921 INFO:teuthology.orchestra.run.smithi122.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-04-25T13:28:22.955 INFO:teuthology.orchestra.run.smithi122.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-25T13:28:22.980 INFO:teuthology.orchestra.run.smithi122.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-25T13:28:23.000 INFO:teuthology.orchestra.run.smithi027.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 4.7 MB/s | 756 kB 00:00 2024-04-25T13:28:23.005 INFO:teuthology.orchestra.run.smithi122.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 708 kB/s | 35 kB 00:00 2024-04-25T13:28:23.067 INFO:teuthology.orchestra.run.smithi027.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.3 MB/s | 158 kB 00:00 2024-04-25T13:28:23.117 INFO:teuthology.orchestra.run.smithi027.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 2.5 MB/s | 127 kB 00:00 2024-04-25T13:28:23.155 INFO:teuthology.orchestra.run.smithi122.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-04-25T13:28:23.234 INFO:teuthology.orchestra.run.smithi027.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 4.2 MB/s | 497 kB 00:00 2024-04-25T13:28:23.248 INFO:teuthology.orchestra.run.smithi122.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-25T13:28:23.290 INFO:teuthology.orchestra.run.smithi122.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-04-25T13:28:23.315 INFO:teuthology.orchestra.run.smithi122.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 5.7 MB/s | 911 kB 00:00 2024-04-25T13:28:23.340 INFO:teuthology.orchestra.run.smithi122.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.5 MB/s | 133 kB 00:00 2024-04-25T13:28:23.416 INFO:teuthology.orchestra.run.smithi122.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-25T13:28:23.499 INFO:teuthology.orchestra.run.smithi122.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 31 MB/s | 5.8 MB 00:00 2024-04-25T13:28:23.518 INFO:teuthology.orchestra.run.smithi045.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 5.2 MB/s | 10 MB 00:01 2024-04-25T13:28:23.524 INFO:teuthology.orchestra.run.smithi122.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 480 kB/s | 87 kB 00:00 2024-04-25T13:28:23.585 INFO:teuthology.orchestra.run.smithi045.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 378 kB/s | 25 kB 00:00 2024-04-25T13:28:23.617 INFO:teuthology.orchestra.run.smithi122.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 6.0 MB 00:00 2024-04-25T13:28:23.650 INFO:teuthology.orchestra.run.smithi122.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-04-25T13:28:23.684 INFO:teuthology.orchestra.run.smithi122.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-04-25T13:28:23.709 INFO:teuthology.orchestra.run.smithi122.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 281 kB/s | 26 kB 00:00 2024-04-25T13:28:23.734 INFO:teuthology.orchestra.run.smithi122.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-25T13:28:23.759 INFO:teuthology.orchestra.run.smithi122.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T13:28:23.785 INFO:teuthology.orchestra.run.smithi122.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 463 kB/s | 35 kB 00:00 2024-04-25T13:28:23.835 INFO:teuthology.orchestra.run.smithi122.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-25T13:28:23.910 INFO:teuthology.orchestra.run.smithi122.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-04-25T13:28:23.936 INFO:teuthology.orchestra.run.smithi122.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-04-25T13:28:23.961 INFO:teuthology.orchestra.run.smithi122.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 626 kB/s | 77 kB 00:00 2024-04-25T13:28:23.978 INFO:teuthology.orchestra.run.smithi122.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 634 kB/s | 42 kB 00:00 2024-04-25T13:28:24.003 INFO:teuthology.orchestra.run.smithi122.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 6.3 MB/s | 269 kB 00:00 2024-04-25T13:28:24.028 INFO:teuthology.orchestra.run.smithi122.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-25T13:28:24.054 INFO:teuthology.orchestra.run.smithi122.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-25T13:28:24.079 INFO:teuthology.orchestra.run.smithi122.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T13:28:24.104 INFO:teuthology.orchestra.run.smithi122.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 715 kB/s | 54 kB 00:00 2024-04-25T13:28:24.151 INFO:teuthology.orchestra.run.smithi027.stdout:(18/152): ceph-common-19.0.0-3291.g994e2f20.el9 2.2 MB/s | 20 MB 00:08 2024-04-25T13:28:24.226 INFO:teuthology.orchestra.run.smithi027.stdout:(19/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 5.2 MB/s | 5.2 MB 00:00 2024-04-25T13:28:24.252 INFO:teuthology.orchestra.run.smithi027.stdout:(20/152): python3-ceph-argparse-19.0.0-3291.g99 451 kB/s | 45 kB 00:00 2024-04-25T13:28:24.294 INFO:teuthology.orchestra.run.smithi027.stdout:(21/152): python3-ceph-common-19.0.0-3291.g994e 2.0 MB/s | 135 kB 00:00 2024-04-25T13:28:24.327 INFO:teuthology.orchestra.run.smithi027.stdout:(22/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.1 MB/s | 161 kB 00:00 2024-04-25T13:28:24.378 INFO:teuthology.orchestra.run.smithi027.stdout:(23/152): python3-rados-19.0.0-3291.g994e2f20.e 3.8 MB/s | 321 kB 00:00 2024-04-25T13:28:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 2.0 MB/s | 100 kB 00:00 2024-04-25T13:28:24.470 INFO:teuthology.orchestra.run.smithi027.stdout:(25/152): python3-rbd-19.0.0-3291.g994e2f20.el9 2.0 MB/s | 298 kB 00:00 2024-04-25T13:28:24.480 INFO:teuthology.orchestra.run.smithi122.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-04-25T13:28:24.495 INFO:teuthology.orchestra.run.smithi027.stdout:(26/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 86 kB 00:00 2024-04-25T13:28:24.505 INFO:teuthology.orchestra.run.smithi122.stdout:(93/152): python3-toml-0.10.2-6.el9.noarch.rpm 105 kB/s | 42 kB 00:00 2024-04-25T13:28:24.530 INFO:teuthology.orchestra.run.smithi122.stdout:(94/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 104 kB/s | 44 kB 00:00 2024-04-25T13:28:24.545 INFO:teuthology.orchestra.run.smithi027.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 3.3 MB/s | 169 kB 00:00 2024-04-25T13:28:24.555 INFO:teuthology.orchestra.run.smithi122.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-25T13:28:24.588 INFO:teuthology.orchestra.run.smithi027.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 820 kB/s | 34 kB 00:00 2024-04-25T13:28:24.589 INFO:teuthology.orchestra.run.smithi122.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-04-25T13:28:24.615 INFO:teuthology.orchestra.run.smithi122.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-04-25T13:28:24.640 INFO:teuthology.orchestra.run.smithi122.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 305 kB 00:00 2024-04-25T13:28:24.646 INFO:teuthology.orchestra.run.smithi027.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.4 MB/s | 142 kB 00:00 2024-04-25T13:28:24.665 INFO:teuthology.orchestra.run.smithi122.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 769 kB/s | 58 kB 00:00 2024-04-25T13:28:24.690 INFO:teuthology.orchestra.run.smithi122.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-25T13:28:24.716 INFO:teuthology.orchestra.run.smithi122.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 843 kB/s | 64 kB 00:00 2024-04-25T13:28:24.735 INFO:teuthology.orchestra.run.smithi045.stdout:(10/152): ceph-common-19.0.0-3291.g994e2f20.el9 2.2 MB/s | 20 MB 00:08 2024-04-25T13:28:24.741 INFO:teuthology.orchestra.run.smithi122.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 593 kB/s | 45 kB 00:00 2024-04-25T13:28:24.766 INFO:teuthology.orchestra.run.smithi122.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2024-04-25T13:28:24.792 INFO:teuthology.orchestra.run.smithi122.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-25T13:28:24.851 INFO:teuthology.orchestra.run.smithi045.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.9 MB/s | 224 kB 00:00 2024-04-25T13:28:24.927 INFO:teuthology.orchestra.run.smithi045.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 360 kB/s | 27 kB 00:00 2024-04-25T13:28:25.002 INFO:teuthology.orchestra.run.smithi045.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 424 kB/s | 32 kB 00:00 2024-04-25T13:28:25.034 INFO:teuthology.orchestra.run.smithi122.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-04-25T13:28:25.244 INFO:teuthology.orchestra.run.smithi045.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 3.1 MB/s | 756 kB 00:00 2024-04-25T13:28:25.253 INFO:teuthology.orchestra.run.smithi027.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 5.8 MB/s | 3.5 MB 00:00 2024-04-25T13:28:25.311 INFO:teuthology.orchestra.run.smithi045.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.3 MB/s | 158 kB 00:00 2024-04-25T13:28:25.317 INFO:teuthology.orchestra.run.smithi122.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 562 kB/s | 309 kB 00:00 2024-04-25T13:28:25.355 INFO:teuthology.orchestra.run.smithi027.stdout:(31/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 3.4 MB/s | 3.0 MB 00:00 2024-04-25T13:28:25.378 INFO:teuthology.orchestra.run.smithi045.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.9 MB/s | 127 kB 00:00 2024-04-25T13:28:25.392 INFO:teuthology.orchestra.run.smithi122.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 842 kB/s | 548 kB 00:00 2024-04-25T13:28:25.418 INFO:teuthology.orchestra.run.smithi122.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-04-25T13:28:25.472 INFO:teuthology.orchestra.run.smithi027.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.1 MB/s | 249 kB 00:00 2024-04-25T13:28:25.493 INFO:teuthology.orchestra.run.smithi122.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-04-25T13:28:25.527 INFO:teuthology.orchestra.run.smithi122.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-04-25T13:28:25.547 INFO:teuthology.orchestra.run.smithi027.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 660 kB/s | 49 kB 00:00 2024-04-25T13:28:25.561 INFO:teuthology.orchestra.run.smithi045.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 2.7 MB/s | 497 kB 00:00 2024-04-25T13:28:25.602 INFO:teuthology.orchestra.run.smithi122.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-04-25T13:28:25.615 INFO:teuthology.orchestra.run.smithi027.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 246 kB/s | 16 kB 00:00 2024-04-25T13:28:25.748 INFO:teuthology.orchestra.run.smithi027.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 2.0 MB/s | 273 kB 00:00 2024-04-25T13:28:25.778 INFO:teuthology.orchestra.run.smithi122.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-04-25T13:28:25.811 INFO:teuthology.orchestra.run.smithi122.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-04-25T13:28:25.870 INFO:teuthology.orchestra.run.smithi122.stdout:(114/152): libarrow-9.0.0-10.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-04-25T13:28:25.887 INFO:teuthology.orchestra.run.smithi122.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 399 kB/s | 43 kB 00:00 2024-04-25T13:28:25.904 INFO:teuthology.orchestra.run.smithi122.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 349 kB/s | 32 kB 00:00 2024-04-25T13:28:25.954 INFO:teuthology.orchestra.run.smithi122.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 178 kB/s | 15 kB 00:00 2024-04-25T13:28:25.998 INFO:teuthology.orchestra.run.smithi027.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 3.0 MB/s | 762 kB 00:00 2024-04-25T13:28:26.005 INFO:teuthology.orchestra.run.smithi122.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-25T13:28:26.031 INFO:teuthology.orchestra.run.smithi122.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-25T13:28:26.055 INFO:teuthology.orchestra.run.smithi122.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-25T13:28:26.073 INFO:teuthology.orchestra.run.smithi027.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 244 kB/s | 18 kB 00:00 2024-04-25T13:28:26.097 INFO:teuthology.orchestra.run.smithi122.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-04-25T13:28:26.139 INFO:teuthology.orchestra.run.smithi122.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.8 MB/s | 206 kB 00:00 2024-04-25T13:28:26.191 INFO:teuthology.orchestra.run.smithi027.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 689 kB/s | 80 kB 00:00 2024-04-25T13:28:26.223 INFO:teuthology.orchestra.run.smithi122.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-25T13:28:26.248 INFO:teuthology.orchestra.run.smithi122.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-25T13:28:26.274 INFO:teuthology.orchestra.run.smithi027.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 803 kB 00:00 2024-04-25T13:28:26.316 INFO:teuthology.orchestra.run.smithi027.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-25T13:28:26.332 INFO:teuthology.orchestra.run.smithi122.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-04-25T13:28:26.345 INFO:teuthology.orchestra.run.smithi045.stdout:(18/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 3.4 MB/s | 18 MB 00:05 2024-04-25T13:28:26.350 INFO:teuthology.orchestra.run.smithi122.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 140 kB/s | 18 kB 00:00 2024-04-25T13:28:26.358 INFO:teuthology.orchestra.run.smithi027.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.9 MB/s | 253 kB 00:00 2024-04-25T13:28:26.366 INFO:teuthology.orchestra.run.smithi122.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 197 kB/s | 23 kB 00:00 2024-04-25T13:28:26.395 INFO:teuthology.orchestra.run.smithi045.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 907 kB/s | 45 kB 00:00 2024-04-25T13:28:26.416 INFO:teuthology.orchestra.run.smithi122.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 231 kB/s | 19 kB 00:00 2024-04-25T13:28:26.425 INFO:teuthology.orchestra.run.smithi027.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-25T13:28:26.445 INFO:teuthology.orchestra.run.smithi045.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 2.6 MB/s | 135 kB 00:00 2024-04-25T13:28:26.449 INFO:teuthology.orchestra.run.smithi122.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-04-25T13:28:26.467 INFO:teuthology.orchestra.run.smithi027.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-25T13:28:26.504 INFO:teuthology.orchestra.run.smithi045.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.7 MB/s | 161 kB 00:00 2024-04-25T13:28:26.525 INFO:teuthology.orchestra.run.smithi122.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-04-25T13:28:26.550 INFO:teuthology.orchestra.run.smithi122.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 784 kB/s | 79 kB 00:00 2024-04-25T13:28:26.551 INFO:teuthology.orchestra.run.smithi027.stdout:(44/152): ceph-mgr-diskprediction-local-19.0.0- 5.7 MB/s | 7.4 MB 00:01 2024-04-25T13:28:26.576 INFO:teuthology.orchestra.run.smithi027.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-25T13:28:26.579 INFO:teuthology.orchestra.run.smithi045.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 4.2 MB/s | 321 kB 00:00 2024-04-25T13:28:26.609 INFO:teuthology.orchestra.run.smithi122.stdout:(132/152): python3-logutils-0.3.5-21.el9.noarch 240 kB/s | 46 kB 00:00 2024-04-25T13:28:26.609 INFO:teuthology.orchestra.run.smithi027.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-25T13:28:26.626 INFO:teuthology.orchestra.run.smithi122.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 573 kB/s | 58 kB 00:00 2024-04-25T13:28:26.635 INFO:teuthology.orchestra.run.smithi027.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-25T13:28:26.654 INFO:teuthology.orchestra.run.smithi045.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 3.9 MB/s | 298 kB 00:00 2024-04-25T13:28:26.660 INFO:teuthology.orchestra.run.smithi027.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 14 MB/s | 689 kB 00:00 2024-04-25T13:28:26.668 INFO:teuthology.orchestra.run.smithi122.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-25T13:28:26.693 INFO:teuthology.orchestra.run.smithi122.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 195 kB/s | 16 kB 00:00 2024-04-25T13:28:26.694 INFO:teuthology.orchestra.run.smithi027.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T13:28:26.696 INFO:teuthology.orchestra.run.smithi045.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 2.3 MB/s | 100 kB 00:00 2024-04-25T13:28:26.719 INFO:teuthology.orchestra.run.smithi027.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T13:28:26.727 INFO:teuthology.orchestra.run.smithi122.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 895 kB/s | 90 kB 00:00 2024-04-25T13:28:26.744 INFO:teuthology.orchestra.run.smithi027.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-25T13:28:26.752 INFO:teuthology.orchestra.run.smithi122.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 336 kB/s | 28 kB 00:00 2024-04-25T13:28:26.755 INFO:teuthology.orchestra.run.smithi045.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.4 MB/s | 86 kB 00:00 2024-04-25T13:28:26.770 INFO:teuthology.orchestra.run.smithi027.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T13:28:26.777 INFO:teuthology.orchestra.run.smithi122.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-04-25T13:28:26.795 INFO:teuthology.orchestra.run.smithi027.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 662 kB/s | 33 kB 00:00 2024-04-25T13:28:26.836 INFO:teuthology.orchestra.run.smithi122.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-25T13:28:26.862 INFO:teuthology.orchestra.run.smithi122.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 542 kB/s | 59 kB 00:00 2024-04-25T13:28:26.870 INFO:teuthology.orchestra.run.smithi027.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-04-25T13:28:26.887 INFO:teuthology.orchestra.run.smithi122.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 328 kB/s | 36 kB 00:00 2024-04-25T13:28:26.896 INFO:teuthology.orchestra.run.smithi027.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 189 kB/s | 18 kB 00:00 2024-04-25T13:28:26.929 INFO:teuthology.orchestra.run.smithi122.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-25T13:28:26.980 INFO:teuthology.orchestra.run.smithi045.stdout:(26/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 3.7 MB/s | 5.2 MB 00:01 2024-04-25T13:28:26.981 INFO:teuthology.orchestra.run.smithi122.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-25T13:28:26.981 INFO:teuthology.orchestra.run.smithi027.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 338 kB/s | 11 kB 00:00 2024-04-25T13:28:26.981 INFO:teuthology.orchestra.run.smithi027.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-25T13:28:26.981 INFO:teuthology.orchestra.run.smithi027.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-04-25T13:28:26.996 INFO:teuthology.orchestra.run.smithi122.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 823 kB/s | 90 kB 00:00 2024-04-25T13:28:27.005 INFO:teuthology.orchestra.run.smithi027.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T13:28:27.030 INFO:teuthology.orchestra.run.smithi045.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.2 MB/s | 169 kB 00:00 2024-04-25T13:28:27.038 INFO:teuthology.orchestra.run.smithi027.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 582 kB 00:00 2024-04-25T13:28:27.046 INFO:teuthology.orchestra.run.smithi122.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-04-25T13:28:27.064 INFO:teuthology.orchestra.run.smithi027.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 443 kB/s | 25 kB 00:00 2024-04-25T13:28:27.071 INFO:teuthology.orchestra.run.smithi122.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-25T13:28:27.089 INFO:teuthology.orchestra.run.smithi027.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-25T13:28:27.097 INFO:teuthology.orchestra.run.smithi045.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 513 kB/s | 34 kB 00:00 2024-04-25T13:28:27.114 INFO:teuthology.orchestra.run.smithi027.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T13:28:27.140 INFO:teuthology.orchestra.run.smithi027.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T13:28:27.164 INFO:teuthology.orchestra.run.smithi045.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.1 MB/s | 142 kB 00:00 2024-04-25T13:28:27.165 INFO:teuthology.orchestra.run.smithi027.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 708 kB/s | 35 kB 00:00 2024-04-25T13:28:27.230 INFO:teuthology.orchestra.run.smithi122.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-04-25T13:28:27.256 INFO:teuthology.orchestra.run.smithi122.stdout:(148/152): re2-20211101-3.el9.x86_64.rpm 739 kB/s | 192 kB 00:00 2024-04-25T13:28:27.265 INFO:teuthology.orchestra.run.smithi027.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-04-25T13:28:27.339 INFO:teuthology.orchestra.run.smithi122.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 16 MB/s | 1.3 MB 00:00 2024-04-25T13:28:27.340 INFO:teuthology.orchestra.run.smithi045.stdout:(30/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 5.2 MB/s | 3.0 MB 00:00 2024-04-25T13:28:27.340 INFO:teuthology.orchestra.run.smithi027.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 26 MB/s | 4.4 MB 00:00 2024-04-25T13:28:27.382 INFO:teuthology.orchestra.run.smithi027.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-25T13:28:27.416 INFO:teuthology.orchestra.run.smithi027.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-25T13:28:27.431 INFO:teuthology.orchestra.run.smithi122.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-04-25T13:28:27.444 INFO:teuthology.orchestra.run.smithi027.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-25T13:28:27.565 INFO:teuthology.orchestra.run.smithi122.stdout:(151/152): librados2-19.0.0-3291.g994e2f20.el9. 6.7 MB/s | 3.3 MB 00:00 2024-04-25T13:28:27.577 INFO:teuthology.orchestra.run.smithi027.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 36 MB/s | 5.8 MB 00:00 2024-04-25T13:28:27.619 INFO:teuthology.orchestra.run.smithi027.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.9 MB/s | 1.0 MB 00:00 2024-04-25T13:28:27.644 INFO:teuthology.orchestra.run.smithi027.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-25T13:28:27.757 INFO:teuthology.orchestra.run.smithi122.stdout:(152/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 5.9 MB/s | 3.1 MB 00:00 2024-04-25T13:28:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 42 MB/s | 6.0 MB 00:00 2024-04-25T13:28:27.762 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-25T13:28:27.762 INFO:teuthology.orchestra.run.smithi122.stdout:Total 15 MB/s | 214 MB 00:13 2024-04-25T13:28:27.787 INFO:teuthology.orchestra.run.smithi027.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.7 MB/s | 244 kB 00:00 2024-04-25T13:28:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-25T13:28:27.837 INFO:teuthology.orchestra.run.smithi027.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-04-25T13:28:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-25T13:28:27.888 INFO:teuthology.orchestra.run.smithi027.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-25T13:28:27.913 INFO:teuthology.orchestra.run.smithi027.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 698 kB/s | 35 kB 00:00 2024-04-25T13:28:28.013 INFO:teuthology.orchestra.run.smithi027.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-04-25T13:28:28.047 INFO:teuthology.orchestra.run.smithi027.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 250 kB 00:00 2024-04-25T13:28:28.072 INFO:teuthology.orchestra.run.smithi027.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-25T13:28:28.089 INFO:teuthology.orchestra.run.smithi027.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-04-25T13:28:28.098 INFO:teuthology.orchestra.run.smithi045.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 3.8 MB/s | 3.5 MB 00:00 2024-04-25T13:28:28.114 INFO:teuthology.orchestra.run.smithi027.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-25T13:28:28.140 INFO:teuthology.orchestra.run.smithi027.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-25T13:28:28.165 INFO:teuthology.orchestra.run.smithi027.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T13:28:28.190 INFO:teuthology.orchestra.run.smithi027.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T13:28:28.199 INFO:teuthology.orchestra.run.smithi045.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.4 MB/s | 249 kB 00:00 2024-04-25T13:28:28.215 INFO:teuthology.orchestra.run.smithi027.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T13:28:28.241 INFO:teuthology.orchestra.run.smithi027.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T13:28:28.265 INFO:teuthology.orchestra.run.smithi045.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 741 kB/s | 49 kB 00:00 2024-04-25T13:28:28.316 INFO:teuthology.orchestra.run.smithi045.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 328 kB/s | 16 kB 00:00 2024-04-25T13:28:28.424 INFO:teuthology.orchestra.run.smithi045.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 2.5 MB/s | 273 kB 00:00 2024-04-25T13:28:28.550 INFO:teuthology.orchestra.run.smithi045.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 6.1 MB/s | 7.4 MB 00:01 2024-04-25T13:28:28.583 INFO:teuthology.orchestra.run.smithi045.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 552 kB/s | 18 kB 00:00 2024-04-25T13:28:28.658 INFO:teuthology.orchestra.run.smithi045.stdout:(38/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 3.2 MB/s | 762 kB 00:00 2024-04-25T13:28:28.666 INFO:teuthology.orchestra.run.smithi027.stdout:(91/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 7.2 MB/s | 43 MB 00:06 2024-04-25T13:28:28.717 INFO:teuthology.orchestra.run.smithi045.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 80 kB 00:00 2024-04-25T13:28:28.784 INFO:teuthology.orchestra.run.smithi045.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-04-25T13:28:28.791 INFO:teuthology.orchestra.run.smithi027.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 76 kB/s | 42 kB 00:00 2024-04-25T13:28:28.809 INFO:teuthology.orchestra.run.smithi045.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-25T13:28:28.834 INFO:teuthology.orchestra.run.smithi045.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T13:28:28.855 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-25T13:28:28.876 INFO:teuthology.orchestra.run.smithi045.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 19 MB/s | 1.2 MB 00:00 2024-04-25T13:28:28.902 INFO:teuthology.orchestra.run.smithi045.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-25T13:28:28.927 INFO:teuthology.orchestra.run.smithi045.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T13:28:28.944 INFO:teuthology.orchestra.run.smithi045.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T13:28:28.969 INFO:teuthology.orchestra.run.smithi045.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.1 MB/s | 215 kB 00:00 2024-04-25T13:28:28.994 INFO:teuthology.orchestra.run.smithi045.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 14 MB/s | 689 kB 00:00 2024-04-25T13:28:28.996 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-25T13:28:28.997 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-25T13:28:29.017 INFO:teuthology.orchestra.run.smithi027.stdout:(93/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-25T13:28:29.028 INFO:teuthology.orchestra.run.smithi045.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T13:28:29.042 INFO:teuthology.orchestra.run.smithi027.stdout:(94/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 119 kB/s | 44 kB 00:00 2024-04-25T13:28:29.053 INFO:teuthology.orchestra.run.smithi045.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T13:28:29.067 INFO:teuthology.orchestra.run.smithi027.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 808 kB/s | 221 kB 00:00 2024-04-25T13:28:29.079 INFO:teuthology.orchestra.run.smithi045.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-25T13:28:29.093 INFO:teuthology.orchestra.run.smithi027.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 724 kB/s | 36 kB 00:00 2024-04-25T13:28:29.104 INFO:teuthology.orchestra.run.smithi045.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T13:28:29.109 INFO:teuthology.orchestra.run.smithi027.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 715 kB/s | 47 kB 00:00 2024-04-25T13:28:29.129 INFO:teuthology.orchestra.run.smithi045.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 662 kB/s | 33 kB 00:00 2024-04-25T13:28:29.135 INFO:teuthology.orchestra.run.smithi027.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 305 kB 00:00 2024-04-25T13:28:29.152 INFO:teuthology.orchestra.run.smithi027.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-04-25T13:28:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 475 kB/s | 18 kB 00:00 2024-04-25T13:28:29.178 INFO:teuthology.orchestra.run.smithi027.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-04-25T13:28:29.194 INFO:teuthology.orchestra.run.smithi027.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T13:28:29.211 INFO:teuthology.orchestra.run.smithi027.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-25T13:28:29.222 INFO:teuthology.orchestra.run.smithi045.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-04-25T13:28:29.228 INFO:teuthology.orchestra.run.smithi027.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 448 kB/s | 22 kB 00:00 2024-04-25T13:28:29.247 INFO:teuthology.orchestra.run.smithi045.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 150 kB/s | 11 kB 00:00 2024-04-25T13:28:29.270 INFO:teuthology.orchestra.run.smithi027.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-25T13:28:29.272 INFO:teuthology.orchestra.run.smithi045.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T13:28:29.297 INFO:teuthology.orchestra.run.smithi045.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-04-25T13:28:29.331 INFO:teuthology.orchestra.run.smithi045.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T13:28:29.356 INFO:teuthology.orchestra.run.smithi045.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 582 kB 00:00 2024-04-25T13:28:29.381 INFO:teuthology.orchestra.run.smithi045.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 518 kB/s | 25 kB 00:00 2024-04-25T13:28:29.406 INFO:teuthology.orchestra.run.smithi045.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-04-25T13:28:29.432 INFO:teuthology.orchestra.run.smithi045.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T13:28:29.457 INFO:teuthology.orchestra.run.smithi045.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T13:28:29.482 INFO:teuthology.orchestra.run.smithi045.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 711 kB/s | 35 kB 00:00 2024-04-25T13:28:29.579 INFO:teuthology.orchestra.run.smithi027.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-04-25T13:28:29.591 INFO:teuthology.orchestra.run.smithi045.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-04-25T13:28:29.667 INFO:teuthology.orchestra.run.smithi045.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-04-25T13:28:29.708 INFO:teuthology.orchestra.run.smithi045.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-25T13:28:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 630 kB/s | 309 kB 00:00 2024-04-25T13:28:29.746 INFO:teuthology.orchestra.run.smithi027.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-25T13:28:29.750 INFO:teuthology.orchestra.run.smithi045.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-25T13:28:29.775 INFO:teuthology.orchestra.run.smithi045.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-25T13:28:29.788 INFO:teuthology.orchestra.run.smithi027.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-04-25T13:28:29.813 INFO:teuthology.orchestra.run.smithi027.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 724 kB/s | 49 kB 00:00 2024-04-25T13:28:29.855 INFO:teuthology.orchestra.run.smithi027.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-25T13:28:29.897 INFO:teuthology.orchestra.run.smithi027.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-25T13:28:29.926 INFO:teuthology.orchestra.run.smithi045.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.8 MB 00:00 2024-04-25T13:28:29.959 INFO:teuthology.orchestra.run.smithi045.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.6 MB/s | 1.0 MB 00:00 2024-04-25T13:28:29.984 INFO:teuthology.orchestra.run.smithi045.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-25T13:28:30.005 INFO:teuthology.orchestra.run.smithi027.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-04-25T13:28:30.026 INFO:teuthology.orchestra.run.smithi045.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 6.1 MB/s | 244 kB 00:00 2024-04-25T13:28:30.039 INFO:teuthology.orchestra.run.smithi027.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 838 kB 00:00 2024-04-25T13:28:30.106 INFO:teuthology.orchestra.run.smithi027.stdout:(114/152): libarrow-9.0.0-10.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-04-25T13:28:30.110 INFO:teuthology.orchestra.run.smithi045.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 40 MB/s | 6.0 MB 00:00 2024-04-25T13:28:30.131 INFO:teuthology.orchestra.run.smithi027.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 346 kB/s | 43 kB 00:00 2024-04-25T13:28:30.135 INFO:teuthology.orchestra.run.smithi045.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.3 MB/s | 248 kB 00:00 2024-04-25T13:28:30.156 INFO:teuthology.orchestra.run.smithi027.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-04-25T13:28:30.160 INFO:teuthology.orchestra.run.smithi045.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 517 kB/s | 26 kB 00:00 2024-04-25T13:28:30.182 INFO:teuthology.orchestra.run.smithi027.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-04-25T13:28:30.186 INFO:teuthology.orchestra.run.smithi045.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-04-25T13:28:30.211 INFO:teuthology.orchestra.run.smithi045.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-25T13:28:30.232 INFO:teuthology.orchestra.run.smithi027.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-25T13:28:30.237 INFO:teuthology.orchestra.run.smithi045.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 698 kB/s | 35 kB 00:00 2024-04-25T13:28:30.257 INFO:teuthology.orchestra.run.smithi027.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-25T13:28:30.282 INFO:teuthology.orchestra.run.smithi027.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-25T13:28:30.286 INFO:teuthology.orchestra.run.smithi045.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.2 MB/s | 250 kB 00:00 2024-04-25T13:28:30.308 INFO:teuthology.orchestra.run.smithi027.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 467 kB/s | 35 kB 00:00 2024-04-25T13:28:30.341 INFO:teuthology.orchestra.run.smithi027.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-25T13:28:30.362 INFO:teuthology.orchestra.run.smithi045.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-04-25T13:28:30.387 INFO:teuthology.orchestra.run.smithi045.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-25T13:28:30.391 INFO:teuthology.orchestra.run.smithi027.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-04-25T13:28:30.412 INFO:teuthology.orchestra.run.smithi045.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T13:28:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-25T13:28:30.439 INFO:teuthology.orchestra.run.smithi045.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 842 kB/s | 42 kB 00:00 2024-04-25T13:28:30.442 INFO:teuthology.orchestra.run.smithi027.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-25T13:28:30.463 INFO:teuthology.orchestra.run.smithi045.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-25T13:28:30.467 INFO:teuthology.orchestra.run.smithi027.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-04-25T13:28:30.488 INFO:teuthology.orchestra.run.smithi045.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T13:28:30.493 INFO:teuthology.orchestra.run.smithi027.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 306 kB/s | 23 kB 00:00 2024-04-25T13:28:30.513 INFO:teuthology.orchestra.run.smithi045.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T13:28:30.518 INFO:teuthology.orchestra.run.smithi027.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-25T13:28:30.539 INFO:teuthology.orchestra.run.smithi045.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T13:28:30.543 INFO:teuthology.orchestra.run.smithi027.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-25T13:28:30.564 INFO:teuthology.orchestra.run.smithi045.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T13:28:30.585 INFO:teuthology.orchestra.run.smithi027.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-25T13:28:30.611 INFO:teuthology.orchestra.run.smithi027.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 502 kB/s | 46 kB 00:00 2024-04-25T13:28:30.628 INFO:teuthology.orchestra.run.smithi027.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 939 kB/s | 79 kB 00:00 2024-04-25T13:28:30.661 INFO:teuthology.orchestra.run.smithi027.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 765 kB/s | 58 kB 00:00 2024-04-25T13:28:30.672 INFO:teuthology.orchestra.run.smithi045.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 392 kB/s | 42 kB 00:00 2024-04-25T13:28:30.695 INFO:teuthology.orchestra.run.smithi027.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-25T13:28:30.720 INFO:teuthology.orchestra.run.smithi027.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-04-25T13:28:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-25T13:28:30.771 INFO:teuthology.orchestra.run.smithi027.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-25T13:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-25T13:28:30.773 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-25T13:28:30.796 INFO:teuthology.orchestra.run.smithi027.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-25T13:28:30.822 INFO:teuthology.orchestra.run.smithi027.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-25T13:28:30.847 INFO:teuthology.orchestra.run.smithi027.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-04-25T13:28:30.872 INFO:teuthology.orchestra.run.smithi027.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-25T13:28:30.897 INFO:teuthology.orchestra.run.smithi027.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-04-25T13:28:30.931 INFO:teuthology.orchestra.run.smithi027.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-25T13:28:30.956 INFO:teuthology.orchestra.run.smithi027.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-04-25T13:28:30.973 INFO:teuthology.orchestra.run.smithi045.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-25T13:28:30.981 INFO:teuthology.orchestra.run.smithi027.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-25T13:28:30.998 INFO:teuthology.orchestra.run.smithi045.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 137 kB/s | 44 kB 00:00 2024-04-25T13:28:31.007 INFO:teuthology.orchestra.run.smithi027.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-25T13:28:31.032 INFO:teuthology.orchestra.run.smithi027.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-25T13:28:31.049 INFO:teuthology.orchestra.run.smithi045.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-25T13:28:31.074 INFO:teuthology.orchestra.run.smithi045.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 478 kB/s | 36 kB 00:00 2024-04-25T13:28:31.082 INFO:teuthology.orchestra.run.smithi027.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T13:28:31.099 INFO:teuthology.orchestra.run.smithi045.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-25T13:28:31.133 INFO:teuthology.orchestra.run.smithi045.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-25T13:28:31.157 INFO:teuthology.orchestra.run.smithi027.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 18 MB/s | 1.3 MB 00:00 2024-04-25T13:28:31.158 INFO:teuthology.orchestra.run.smithi045.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 989 kB/s | 58 kB 00:00 2024-04-25T13:28:31.183 INFO:teuthology.orchestra.run.smithi045.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T13:28:31.209 INFO:teuthology.orchestra.run.smithi045.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T13:28:31.234 INFO:teuthology.orchestra.run.smithi045.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-04-25T13:28:31.241 INFO:teuthology.orchestra.run.smithi027.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-25T13:28:31.259 INFO:teuthology.orchestra.run.smithi045.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-04-25T13:28:31.301 INFO:teuthology.orchestra.run.smithi045.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 13 MB/s | 863 kB 00:00 2024-04-25T13:28:31.424 INFO:teuthology.orchestra.run.smithi027.stdout:(151/152): librados2-19.0.0-3291.g994e2f20.el9. 8.0 MB/s | 3.3 MB 00:00 2024-04-25T13:28:31.550 INFO:teuthology.orchestra.run.smithi027.stdout:(152/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-04-25T13:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-25T13:28:31.557 INFO:teuthology.orchestra.run.smithi027.stdout:Total 13 MB/s | 214 MB 00:16 2024-04-25T13:28:31.702 INFO:teuthology.orchestra.run.smithi045.stdout:(104/152): ceph-test-19.0.0-3291.g994e2f20.el9. 5.3 MB/s | 43 MB 00:08 2024-04-25T13:28:31.802 INFO:teuthology.orchestra.run.smithi045.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 195 kB/s | 19 kB 00:00 2024-04-25T13:28:31.868 INFO:teuthology.orchestra.run.smithi045.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 545 kB/s | 309 kB 00:00 2024-04-25T13:28:31.919 INFO:teuthology.orchestra.run.smithi045.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-04-25T13:28:31.961 INFO:teuthology.orchestra.run.smithi045.stdout:(108/152): abseil-cpp-20211102.0-3.el9.x86_64.r 783 kB/s | 548 kB 00:00 2024-04-25T13:28:31.986 INFO:teuthology.orchestra.run.smithi045.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 727 kB/s | 49 kB 00:00 2024-04-25T13:28:32.028 INFO:teuthology.orchestra.run.smithi045.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-25T13:28:32.053 INFO:teuthology.orchestra.run.smithi045.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-25T13:28:32.137 INFO:teuthology.orchestra.run.smithi045.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-25T13:28:32.162 INFO:teuthology.orchestra.run.smithi045.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-04-25T13:28:32.187 INFO:teuthology.orchestra.run.smithi045.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 869 kB/s | 43 kB 00:00 2024-04-25T13:28:32.204 INFO:teuthology.orchestra.run.smithi045.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 772 kB/s | 32 kB 00:00 2024-04-25T13:28:32.229 INFO:teuthology.orchestra.run.smithi045.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 358 kB/s | 15 kB 00:00 2024-04-25T13:28:32.255 INFO:teuthology.orchestra.run.smithi045.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-25T13:28:32.280 INFO:teuthology.orchestra.run.smithi045.stdout:(118/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-04-25T13:28:32.305 INFO:teuthology.orchestra.run.smithi045.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-25T13:28:32.330 INFO:teuthology.orchestra.run.smithi045.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-25T13:28:32.347 INFO:teuthology.orchestra.run.smithi045.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-04-25T13:28:32.372 INFO:teuthology.orchestra.run.smithi045.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-25T13:28:32.423 INFO:teuthology.orchestra.run.smithi045.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 215 kB/s | 11 kB 00:00 2024-04-25T13:28:32.448 INFO:teuthology.orchestra.run.smithi045.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-25T13:28:32.473 INFO:teuthology.orchestra.run.smithi045.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-04-25T13:28:32.515 INFO:teuthology.orchestra.run.smithi045.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-25T13:28:32.541 INFO:teuthology.orchestra.run.smithi045.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-04-25T13:28:32.566 INFO:teuthology.orchestra.run.smithi045.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-25T13:28:32.591 INFO:teuthology.orchestra.run.smithi045.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-04-25T13:28:32.625 INFO:teuthology.orchestra.run.smithi045.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-25T13:28:32.650 INFO:teuthology.orchestra.run.smithi045.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 553 kB/s | 46 kB 00:00 2024-04-25T13:28:32.675 INFO:teuthology.orchestra.run.smithi045.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 942 kB/s | 79 kB 00:00 2024-04-25T13:28:32.700 INFO:teuthology.orchestra.run.smithi045.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 767 kB/s | 58 kB 00:00 2024-04-25T13:28:32.708 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-25T13:28:32.726 INFO:teuthology.orchestra.run.smithi045.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 329 kB/s | 16 kB 00:00 2024-04-25T13:28:32.751 INFO:teuthology.orchestra.run.smithi045.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-25T13:28:32.776 INFO:teuthology.orchestra.run.smithi045.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-04-25T13:28:32.801 INFO:teuthology.orchestra.run.smithi045.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-04-25T13:28:32.827 INFO:teuthology.orchestra.run.smithi045.stdout:(138/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 714 kB/s | 90 kB 00:00 2024-04-25T13:28:32.852 INFO:teuthology.orchestra.run.smithi045.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-25T13:28:32.868 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-25T13:28:32.869 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-25T13:28:32.877 INFO:teuthology.orchestra.run.smithi045.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-04-25T13:28:32.894 INFO:teuthology.orchestra.run.smithi045.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 535 kB/s | 36 kB 00:00 2024-04-25T13:28:32.919 INFO:teuthology.orchestra.run.smithi045.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 763 kB/s | 51 kB 00:00 2024-04-25T13:28:32.944 INFO:teuthology.orchestra.run.smithi045.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-25T13:28:32.978 INFO:teuthology.orchestra.run.smithi045.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-04-25T13:28:33.003 INFO:teuthology.orchestra.run.smithi045.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-04-25T13:28:33.028 INFO:teuthology.orchestra.run.smithi045.stdout:(146/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-25T13:28:33.054 INFO:teuthology.orchestra.run.smithi045.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-25T13:28:33.104 INFO:teuthology.orchestra.run.smithi045.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T13:28:33.163 INFO:teuthology.orchestra.run.smithi045.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T13:28:33.263 INFO:teuthology.orchestra.run.smithi045.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-04-25T13:28:33.521 INFO:teuthology.orchestra.run.smithi045.stdout:(151/152): librados2-19.0.0-3291.g994e2f20.el9. 6.7 MB/s | 3.3 MB 00:00 2024-04-25T13:28:33.622 INFO:teuthology.orchestra.run.smithi045.stdout:(152/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 5.4 MB/s | 3.1 MB 00:00 2024-04-25T13:28:33.625 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-04-25T13:28:33.626 INFO:teuthology.orchestra.run.smithi045.stdout:Total 12 MB/s | 214 MB 00:18 2024-04-25T13:28:34.711 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-25T13:28:34.711 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-25T13:28:34.778 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-04-25T13:28:34.938 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-04-25T13:28:34.938 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-04-25T13:28:36.670 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-25T13:28:36.761 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T13:28:36.775 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-04-25T13:28:36.775 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-04-25T13:28:36.808 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T13:28:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T13:28:37.054 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T13:28:37.088 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T13:28:37.131 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T13:28:37.478 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T13:28:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:28:37.731 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:28:37.757 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:28:37.857 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:28:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T13:28:38.039 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T13:28:38.092 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T13:28:38.129 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:28:38.234 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:28:38.272 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T13:28:38.334 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T13:28:38.356 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T13:28:38.380 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T13:28:38.573 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T13:28:38.606 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T13:28:38.712 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T13:28:38.808 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T13:28:38.864 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T13:28:38.899 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T13:28:38.964 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T13:28:39.007 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T13:28:39.051 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T13:28:39.083 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T13:28:39.111 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T13:28:39.181 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T13:28:39.234 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T13:28:39.265 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T13:28:39.317 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T13:28:39.394 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T13:28:39.428 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T13:28:39.470 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T13:28:39.499 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T13:28:39.569 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T13:28:39.615 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T13:28:39.687 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T13:28:39.819 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T13:28:39.884 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T13:28:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T13:28:40.488 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T13:28:40.526 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T13:28:40.578 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T13:28:40.615 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T13:28:40.657 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T13:28:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T13:28:40.868 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-25T13:28:40.894 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T13:28:40.948 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T13:28:40.979 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T13:28:41.021 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T13:28:41.053 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T13:28:41.090 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T13:28:41.136 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T13:28:41.181 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T13:28:41.182 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T13:28:41.232 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T13:28:41.248 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T13:28:41.283 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T13:28:41.289 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T13:28:41.314 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T13:28:41.366 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T13:28:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T13:28:41.463 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T13:28:41.494 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T13:28:41.535 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T13:28:41.628 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T13:28:41.661 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T13:28:41.701 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T13:28:41.717 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T13:28:41.735 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T13:28:41.768 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T13:28:41.839 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:28:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T13:28:41.968 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:28:41.968 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T13:28:41.999 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:28:42.009 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T13:28:42.038 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T13:28:42.072 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T13:28:42.091 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:28:42.111 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T13:28:42.128 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T13:28:42.147 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T13:28:42.173 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T13:28:42.207 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T13:28:42.241 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T13:28:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T13:28:42.293 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T13:28:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T13:28:42.395 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T13:28:42.395 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:28:42.411 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T13:28:42.485 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:28:42.512 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T13:28:42.575 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T13:28:42.597 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T13:28:42.620 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T13:28:42.654 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T13:28:42.697 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T13:28:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T13:28:42.835 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T13:28:42.838 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-04-25T13:28:42.862 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T13:28:42.863 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T13:28:42.915 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T13:28:42.926 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T13:28:42.964 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T13:28:42.975 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T13:28:43.073 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T13:28:43.111 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T13:28:43.112 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T13:28:43.148 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T13:28:43.226 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T13:28:43.272 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T13:28:43.291 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T13:28:43.315 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T13:28:43.319 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T13:28:43.348 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T13:28:43.365 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T13:28:43.375 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T13:28:43.448 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T13:28:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T13:28:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T13:28:43.563 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T13:28:43.631 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T13:28:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T13:28:43.673 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T13:28:43.678 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T13:28:43.712 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T13:28:43.715 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T13:28:43.717 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T13:28:43.745 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T13:28:43.811 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T13:28:43.819 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:28:43.821 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T13:28:43.863 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T13:28:43.869 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T13:28:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:28:43.949 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T13:28:43.949 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T13:28:43.982 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T13:28:43.987 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:28:44.094 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T13:28:44.096 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:28:44.125 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T13:28:44.148 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T13:28:44.158 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T13:28:44.180 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T13:28:44.266 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T13:28:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T13:28:44.380 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T13:28:44.408 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:28:44.515 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:28:44.542 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T13:28:44.605 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T13:28:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T13:28:44.651 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T13:28:44.710 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T13:28:44.795 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T13:28:44.829 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T13:28:44.852 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T13:28:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T13:28:44.884 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T13:28:44.908 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T13:28:44.910 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T13:28:44.961 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T13:28:44.994 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T13:28:45.028 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T13:28:45.091 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T13:28:45.117 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T13:28:45.124 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T13:28:45.168 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T13:28:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T13:28:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T13:28:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T13:28:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T13:28:45.302 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T13:28:45.327 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T13:28:45.347 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T13:28:45.359 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T13:28:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T13:28:45.387 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T13:28:45.431 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T13:28:45.459 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T13:28:45.466 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T13:28:45.510 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T13:28:45.510 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T13:28:45.539 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T13:28:45.552 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T13:28:45.575 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T13:28:45.653 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T13:28:45.685 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T13:28:45.698 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T13:28:45.729 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T13:28:45.749 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T13:28:45.757 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T13:28:45.779 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T13:28:45.820 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T13:28:45.829 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T13:28:45.872 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T13:28:45.920 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T13:28:45.959 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T13:28:45.961 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T13:28:46.002 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T13:28:46.027 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T13:28:46.051 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T13:28:46.104 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T13:28:46.131 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T13:28:46.159 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T13:28:46.240 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T13:28:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T13:28:46.320 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T13:28:46.347 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T13:28:46.381 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T13:28:46.393 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T13:28:46.421 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T13:28:46.455 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T13:28:46.495 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T13:28:46.534 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T13:28:46.564 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T13:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T13:28:46.632 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T13:28:46.675 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T13:28:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T13:28:46.708 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T13:28:46.718 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T13:28:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T13:28:46.802 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T13:28:46.805 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T13:28:46.840 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T13:28:46.842 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T13:28:46.848 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T13:28:46.874 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T13:28:46.883 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T13:28:46.886 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T13:28:46.910 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T13:28:46.921 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T13:28:46.935 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T13:28:46.963 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T13:28:47.048 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T13:28:47.098 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T13:28:47.121 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T13:28:47.182 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T13:28:47.239 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T13:28:47.313 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T13:28:47.349 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T13:28:47.386 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T13:28:47.401 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T13:28:47.431 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T13:28:47.433 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T13:28:47.468 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T13:28:47.502 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T13:28:47.512 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T13:28:47.524 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T13:28:47.555 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T13:28:47.707 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T13:28:47.750 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T13:28:47.780 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T13:28:47.823 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T13:28:47.879 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T13:28:47.920 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T13:28:47.927 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T13:28:47.955 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T13:28:47.961 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T13:28:47.995 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T13:28:48.028 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T13:28:48.053 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T13:28:48.063 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T13:28:48.120 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T13:28:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T13:28:48.212 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T13:28:48.246 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T13:28:48.268 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T13:28:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T13:28:48.338 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T13:28:48.373 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T13:28:48.412 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T13:28:48.431 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T13:28:48.452 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T13:28:48.460 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T13:28:48.481 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T13:28:48.516 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T13:28:48.541 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T13:28:48.549 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T13:28:48.591 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T13:28:48.625 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T13:28:48.716 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T13:28:48.772 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T13:28:48.803 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T13:28:48.851 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T13:28:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T13:28:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T13:28:49.239 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T13:28:49.355 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T13:28:49.775 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:49.778 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T13:28:49.813 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:49.826 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T13:28:49.859 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T13:28:49.963 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T13:28:50.018 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T13:28:50.108 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T13:28:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T13:28:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T13:28:50.350 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T13:28:50.390 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:50.425 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T13:28:50.430 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T13:28:50.525 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T13:28:50.525 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T13:28:50.525 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T13:28:50.525 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:28:50.554 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T13:28:50.679 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T13:28:50.758 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T13:28:50.793 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T13:28:50.826 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T13:28:51.117 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T13:28:51.163 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T13:28:51.205 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T13:28:51.236 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T13:28:51.280 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T13:28:51.310 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T13:28:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T13:28:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T13:28:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T13:28:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T13:28:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T13:28:52.541 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T13:28:52.620 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T13:28:52.648 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T13:28:52.681 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T13:28:53.021 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T13:28:53.059 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T13:28:53.152 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T13:28:53.181 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T13:28:53.218 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T13:28:53.736 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T13:28:53.762 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T13:28:53.834 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T13:28:53.864 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T13:28:54.307 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:54.339 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:54.930 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:54.968 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T13:28:55.059 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T13:28:55.059 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T13:28:55.059 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T13:28:55.060 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:28:55.095 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T13:28:56.207 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:56.243 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:56.832 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T13:28:56.879 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T13:28:56.974 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T13:28:56.974 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T13:28:56.975 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T13:28:56.975 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:28:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T13:29:02.126 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T13:29:02.126 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-04-25T13:29:02.126 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-04-25T13:29:02.126 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-04-25T13:29:02.126 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-04-25T13:29:02.126 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-04-25T13:29:02.127 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-04-25T13:29:02.127 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-04-25T13:29:02.127 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T13:29:02.317 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T13:29:02.317 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:02.317 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T13:29:02.317 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:03.898 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T13:29:03.923 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T13:29:03.923 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:03.923 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T13:29:03.924 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T13:29:03.924 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T13:29:03.924 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:04.157 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T13:29:04.179 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T13:29:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T13:29:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T13:29:04.179 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T13:29:04.179 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:04.641 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T13:29:04.663 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T13:29:04.663 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:04.663 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T13:29:04.663 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T13:29:04.663 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T13:29:04.663 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:04.771 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T13:29:04.820 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T13:29:04.875 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T13:29:05.060 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T13:29:05.117 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T13:29:05.203 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T13:29:05.300 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T13:29:05.439 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T13:29:05.476 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T13:29:05.508 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T13:29:05.536 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T13:29:05.980 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T13:29:06.010 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T13:29:06.042 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T13:29:06.103 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T13:29:06.142 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T13:29:07.172 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T13:29:07.211 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T13:29:07.393 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T13:29:07.393 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-04-25T13:29:07.393 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-04-25T13:29:07.393 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-04-25T13:29:07.394 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-04-25T13:29:07.394 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-04-25T13:29:07.394 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-04-25T13:29:07.394 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-04-25T13:29:07.394 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:07.582 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T13:29:07.599 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T13:29:07.599 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:07.599 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T13:29:07.600 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T13:29:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T13:29:08.295 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T13:29:08.471 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T13:29:08.496 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T13:29:08.519 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T13:29:08.519 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:08.519 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T13:29:08.519 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T13:29:08.520 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T13:29:08.520 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T13:29:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T13:29:08.674 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T13:29:08.721 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T13:29:08.941 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T13:29:08.941 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-04-25T13:29:08.941 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-04-25T13:29:08.941 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-04-25T13:29:08.941 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-04-25T13:29:08.941 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-04-25T13:29:08.942 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-04-25T13:29:08.942 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-04-25T13:29:08.942 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:09.116 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T13:29:09.133 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T13:29:09.134 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:09.134 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T13:29:09.134 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:09.236 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T13:29:09.263 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T13:29:09.263 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:09.263 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T13:29:09.264 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T13:29:09.264 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T13:29:09.264 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:09.503 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T13:29:09.527 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T13:29:09.527 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:09.527 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T13:29:09.527 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T13:29:09.527 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T13:29:09.527 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:09.659 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T13:29:09.692 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T13:29:09.714 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T13:29:09.714 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:09.714 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T13:29:09.714 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T13:29:09.715 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T13:29:09.715 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:09.759 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T13:29:09.781 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T13:29:09.781 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:09.781 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T13:29:09.781 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:09.827 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T13:29:09.848 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T13:29:09.848 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:09.849 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T13:29:09.849 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T13:29:09.849 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T13:29:09.849 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:10.012 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T13:29:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T13:29:10.038 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:10.038 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T13:29:10.038 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T13:29:10.038 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T13:29:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:10.152 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T13:29:10.161 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T13:29:10.183 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T13:29:10.183 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:10.183 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T13:29:10.183 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T13:29:10.183 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T13:29:10.183 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:10.209 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T13:29:10.273 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T13:29:10.469 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T13:29:10.514 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T13:29:10.594 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T13:29:10.693 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T13:29:10.777 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T13:29:10.805 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T13:29:10.805 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:10.805 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T13:29:10.805 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T13:29:10.806 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T13:29:10.806 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:10.851 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T13:29:10.888 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T13:29:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T13:29:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T13:29:11.044 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T13:29:11.068 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T13:29:11.069 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:11.069 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T13:29:11.069 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T13:29:11.069 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T13:29:11.069 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:11.354 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T13:29:11.388 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T13:29:11.421 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T13:29:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T13:29:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T13:29:11.553 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T13:29:11.577 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T13:29:11.578 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:11.578 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T13:29:11.578 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T13:29:11.578 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T13:29:11.578 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:11.684 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T13:29:11.733 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T13:29:11.797 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T13:29:11.995 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T13:29:12.038 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T13:29:12.110 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T13:29:12.217 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T13:29:12.397 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T13:29:12.437 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T13:29:12.469 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T13:29:12.488 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T13:29:12.638 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T13:29:12.671 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T13:29:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T13:29:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T13:29:12.985 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T13:29:13.049 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T13:29:13.090 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T13:29:13.701 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T13:29:13.730 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T13:29:13.839 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T13:29:14.015 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T13:29:14.040 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T13:29:14.065 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T13:29:14.065 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:14.065 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T13:29:14.065 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T13:29:14.065 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T13:29:14.065 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:14.116 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T13:29:14.129 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T13:29:14.181 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T13:29:14.211 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T13:29:14.230 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T13:29:14.282 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T13:29:14.474 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T13:29:14.515 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T13:29:14.555 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T13:29:14.588 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T13:29:14.615 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T13:29:14.797 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T13:29:14.845 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T13:29:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T13:29:15.234 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T13:29:15.261 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T13:29:15.277 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T13:29:15.310 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T13:29:15.334 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T13:29:15.335 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:15.335 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T13:29:15.335 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T13:29:15.335 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T13:29:15.335 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:15.369 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T13:29:15.378 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T13:29:15.401 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T13:29:15.401 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:15.401 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T13:29:15.402 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:15.453 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T13:29:15.477 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T13:29:15.478 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:15.478 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T13:29:15.478 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T13:29:15.478 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T13:29:15.478 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:15.488 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:15.532 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:15.551 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:15.552 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T13:29:15.580 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T13:29:15.605 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T13:29:15.605 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:15.605 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T13:29:15.605 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T13:29:15.605 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T13:29:15.606 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:15.655 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T13:29:15.667 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T13:29:15.778 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T13:29:15.804 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T13:29:15.828 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T13:29:15.828 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:15.828 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T13:29:15.828 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T13:29:15.828 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T13:29:15.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:15.831 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T13:29:16.380 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T13:29:16.455 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T13:29:16.496 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T13:29:16.579 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T13:29:16.618 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T13:29:16.618 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T13:29:16.704 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T13:29:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T13:29:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T13:29:16.802 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T13:29:16.838 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T13:29:16.866 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T13:29:16.869 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T13:29:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T13:29:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T13:29:16.891 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T13:29:16.891 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:16.942 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T13:29:16.966 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T13:29:16.966 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:16.966 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T13:29:16.966 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:17.026 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T13:29:17.050 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T13:29:17.050 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:17.050 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T13:29:17.050 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T13:29:17.050 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T13:29:17.050 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:17.385 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T13:29:17.409 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T13:29:17.409 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T13:29:17.409 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T13:29:17.409 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T13:29:17.409 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T13:29:17.410 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:20.375 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T13:29:20.417 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T13:29:20.449 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T13:29:20.483 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T13:29:20.508 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T13:29:20.701 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T13:29:20.746 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T13:29:20.775 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T13:29:21.368 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:21.406 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:21.427 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:21.931 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T13:29:21.972 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T13:29:22.014 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T13:29:22.045 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T13:29:22.072 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T13:29:22.277 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T13:29:22.318 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T13:29:22.342 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T13:29:22.347 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T13:29:22.436 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T13:29:22.478 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T13:29:22.564 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T13:29:22.599 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T13:29:22.599 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T13:29:22.691 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T13:29:22.724 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T13:29:22.724 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T13:29:22.817 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T13:29:22.884 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:22.916 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:22.961 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:22.982 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T13:29:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T13:29:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T13:29:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T13:29:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T13:29:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T13:29:23.012 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T13:29:23.013 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T13:29:23.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T13:29:23.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T13:29:23.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T13:29:23.017 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T13:29:23.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T13:29:23.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T13:29:23.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T13:29:23.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T13:29:23.899 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T13:29:23.990 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T13:29:24.032 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T13:29:24.126 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T13:29:24.163 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T13:29:24.163 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T13:29:24.247 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T13:29:24.280 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T13:29:24.280 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T13:29:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T13:29:24.439 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:25.064 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:25.064 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.065 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.066 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T13:29:25.067 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T13:29:25.068 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T13:29:25.069 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T13:29:25.070 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T13:29:25.071 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T13:29:25.072 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T13:29:25.073 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T13:29:25.074 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T13:29:25.075 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T13:29:25.076 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T13:29:25.076 INFO:teuthology.orchestra.run.smithi122.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T13:29:25.076 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T13:29:25.076 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T13:29:25.076 INFO:teuthology.orchestra.run.smithi122.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T13:29:25.076 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:29:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-25T13:29:25.516 DEBUG:teuthology.parallel:result is None 2024-04-25T13:29:28.546 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:28.546 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T13:29:28.546 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T13:29:28.546 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T13:29:28.546 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T13:29:28.546 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T13:29:28.546 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T13:29:28.546 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T13:29:28.547 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T13:29:28.548 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T13:29:28.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T13:29:28.550 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T13:29:28.551 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T13:29:28.552 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T13:29:28.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T13:29:28.554 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T13:29:28.555 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T13:29:28.556 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T13:29:28.557 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T13:29:28.557 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T13:29:28.557 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T13:29:28.557 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T13:29:28.557 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T13:29:28.557 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T13:29:28.557 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T13:29:30.419 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T13:29:30.420 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T13:29:30.421 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T13:29:30.422 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T13:29:30.423 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T13:29:30.424 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T13:29:30.425 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T13:29:30.426 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T13:29:30.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T13:29:30.428 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T13:29:30.429 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T13:29:30.430 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T13:29:30.430 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T13:29:30.430 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T13:29:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-04-25T13:29:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.716 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.718 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T13:29:30.719 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T13:29:30.720 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T13:29:30.721 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.722 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T13:29:30.725 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T13:29:30.726 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T13:29:30.727 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.728 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T13:29:30.729 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T13:29:30.730 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T13:29:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T13:29:30.731 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:29:30.731 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-25T13:29:31.100 DEBUG:teuthology.parallel:result is None 2024-04-25T13:29:32.561 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T13:29:32.561 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:32.561 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-04-25T13:29:32.561 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.565 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.565 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T13:29:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T13:29:32.567 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T13:29:32.568 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.569 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T13:29:32.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T13:29:32.571 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T13:29:32.572 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T13:29:32.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T13:29:32.574 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T13:29:32.575 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T13:29:32.575 INFO:teuthology.orchestra.run.smithi045.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T13:29:32.575 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T13:29:32.575 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T13:29:32.575 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T13:29:32.575 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-25T13:29:32.575 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-04-25T13:29:32.955 DEBUG:teuthology.parallel:result is None 2024-04-25T13:29:32.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:29:33.099 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T13:29:33.121 INFO:teuthology.orchestra.run.smithi027.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T13:29:33.122 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T13:29:33.122 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T13:29:33.124 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:29:33.343 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T13:29:33.366 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T13:29:33.366 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T13:29:33.366 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T13:29:33.368 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:29:33.511 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T13:29:33.536 INFO:teuthology.orchestra.run.smithi122.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T13:29:33.537 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T13:29:33.537 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T13:29:33.539 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T13:29:33.539 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:29:33.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T13:29:33.567 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:29:33.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T13:29:33.594 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:29:33.594 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T13:29:33.624 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T13:29:33.625 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:29:33.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T13:29:33.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T13:29:33.717 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:29:33.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T13:29:33.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T13:29:33.811 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:29:33.811 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T13:29:33.839 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T13:29:33.907 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T13:29:33.907 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:29:33.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T13:29:33.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T13:29:33.999 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:29:34.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T13:29:34.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T13:29:34.091 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:29:34.092 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T13:29:34.120 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T13:29:34.187 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T13:29:34.188 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:29:34.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T13:29:34.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T13:29:34.278 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:29:34.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T13:29:34.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T13:29:34.367 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:29:34.367 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T13:29:34.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T13:29:34.462 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T13:29:34.576 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T13:29:34.576 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:29:34.577 INFO:tasks.cephadm:Cluster fsid is db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:29:34.577 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T13:29:34.577 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.27', 'mon.b': '172.21.15.45', 'mon.c': '172.21.15.122'} 2024-04-25T13:29:34.577 INFO:tasks.cephadm:First mon is mon.a on smithi027 2024-04-25T13:29:34.577 INFO:tasks.cephadm:First mgr is x 2024-04-25T13:29:34.577 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T13:29:34.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-04-25T13:29:34.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2024-04-25T13:29:34.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2024-04-25T13:29:34.655 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T13:29:34.655 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:29:34.869 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78872/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3291-g994e2f20', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3291.g994e2f20'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-25 04:02:18.990013', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240425.015853-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-25T13:29:35.017 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-pdonnell-testing-20240425.015853-debug, sha1 994e2f20102a20fa9a27966c14b625cc01884df8 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:29:35.018 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm 2024-04-25T13:29:35.019 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:36.011 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 13:29 /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:36.012 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:36.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 13:29 /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:36.628 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:37.216 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 13:29 /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:37.217 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:37.237 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:37.257 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T13:29:37.294 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 on all hosts... 2024-04-25T13:29:37.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T13:29:37.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T13:29:37.300 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T13:29:38.894 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T13:29:38.979 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T13:29:39.019 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T13:30:10.939 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-25T13:30:10.939 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T13:30:10.939 INFO:teuthology.orchestra.run.smithi045.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T13:30:10.940 INFO:teuthology.orchestra.run.smithi045.stdout: "repo_digests": [ 2024-04-25T13:30:10.940 INFO:teuthology.orchestra.run.smithi045.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T13:30:10.940 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-25T13:30:10.940 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-25T13:30:11.622 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:30:11.622 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T13:30:11.622 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T13:30:11.622 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-04-25T13:30:11.622 INFO:teuthology.orchestra.run.smithi027.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T13:30:11.622 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:30:11.622 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:30:14.748 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-04-25T13:30:14.748 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T13:30:14.748 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T13:30:14.748 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2024-04-25T13:30:14.749 INFO:teuthology.orchestra.run.smithi122.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T13:30:14.749 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-04-25T13:30:14.749 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-04-25T13:30:14.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-04-25T13:30:14.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2024-04-25T13:30:14.837 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2024-04-25T13:30:14.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-04-25T13:30:14.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2024-04-25T13:30:14.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2024-04-25T13:30:14.944 INFO:tasks.cephadm:Writing seed config... 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T13:30:14.947 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T13:30:14.948 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T13:30:14.949 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T13:30:14.950 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T13:30:14.950 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T13:30:14.950 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:30:14.950 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T13:30:14.968 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = db01fc44-0307-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T13:30:14.968 DEBUG:teuthology.orchestra.run.smithi027:mon.a> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T13:30:15.010 DEBUG:teuthology.orchestra.run.smithi027:mgr.x> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T13:30:15.054 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T13:30:15.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 -v bootstrap --fsid db01fc44-0307-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:30:15.253 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-25T13:30:15.254 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8', '-v', 'bootstrap', '--fsid', 'db01fc44-0307-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.27', '--skip-admin-label'] 2024-04-25T13:30:15.254 INFO:teuthology.orchestra.run.smithi027.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T13:30:15.254 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying podman|docker is present... 2024-04-25T13:30:15.281 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T13:30:15.281 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying lvm2 is present... 2024-04-25T13:30:15.281 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying time synchronization is in place... 2024-04-25T13:30:15.288 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T13:30:15.288 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T13:30:15.294 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T13:30:15.294 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T13:30:15.301 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-04-25T13:30:15.307 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-04-25T13:30:15.307 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-04-25T13:30:15.307 INFO:teuthology.orchestra.run.smithi027.stdout:Repeating the final host check... 2024-04-25T13:30:15.331 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T13:30:15.332 INFO:teuthology.orchestra.run.smithi027.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T13:30:15.332 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl is present 2024-04-25T13:30:15.332 INFO:teuthology.orchestra.run.smithi027.stdout:lvcreate is present 2024-04-25T13:30:15.338 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T13:30:15.338 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T13:30:15.344 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T13:30:15.344 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T13:30:15.350 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:Host looks OK 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:Cluster fsid: db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:Acquiring lock 139639172365472 on /run/cephadm/db01fc44-0307-11ef-bc93-c7b262605968.lock 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:Lock 139639172365472 acquired on /run/cephadm/db01fc44-0307-11ef-bc93-c7b262605968.lock 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 3300 ... 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 6789 ... 2024-04-25T13:30:15.359 INFO:teuthology.orchestra.run.smithi027.stdout:Base mon IP(s) is [172.21.15.27:3300, 172.21.15.27:6789], mon addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-04-25T13:30:15.362 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2024-04-25T13:30:15.362 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2024-04-25T13:30:15.364 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T13:30:15.364 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T13:30:15.364 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T13:30:15.367 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T13:30:15.367 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T13:30:15.367 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T13:30:15.367 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T13:30:15.367 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-25T13:30:15.367 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T13:30:15.368 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-04-25T13:30:15.368 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-04-25T13:30:15.368 INFO:teuthology.orchestra.run.smithi027.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T13:30:15.368 INFO:teuthology.orchestra.run.smithi027.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T13:30:15.369 INFO:teuthology.orchestra.run.smithi027.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T13:30:15.848 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T13:30:15.848 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T13:30:15.849 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T13:30:15.849 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:665339d96aad80d083ba9988a62422bcef0a42210fb2b28340e33ab98589f7b8 2024-04-25T13:30:15.849 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T13:30:15.849 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying config sha256:d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T13:30:15.849 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T13:30:16.777 INFO:teuthology.orchestra.run.smithi027.stdout:ceph: stdout ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T13:30:16.778 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph version: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T13:30:16.778 INFO:teuthology.orchestra.run.smithi027.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T13:30:17.726 INFO:teuthology.orchestra.run.smithi027.stdout:stat: stdout 167 167 2024-04-25T13:30:17.726 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial keys... 2024-04-25T13:30:18.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQDqWipmtHuKAxAAZPabrBVUh9UoGeBuvPOLkA== 2024-04-25T13:30:19.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQDrWipmcMXUARAApkxvrEVCZAvUrVDsDS8BGw== 2024-04-25T13:30:20.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQDrWipmDDVEOBAAZqKlwI22L7D4afUGyyq33A== 2024-04-25T13:30:20.547 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial monmap... 2024-04-25T13:30:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T13:30:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T13:30:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:21.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T13:30:21.431 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool for a [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T13:30:21.431 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-04-25T13:30:21.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: set fsid to db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:21.431 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T13:30:21.431 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:30:21.432 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mon... 2024-04-25T13:30:30.420 INFO:teuthology.orchestra.run.smithi027.stdout:create mon.a on 2024-04-25T13:30:30.905 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T13:30:31.249 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T13:30:31.693 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-db01fc44-0307-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-db01fc44-0307-11ef-bc93-c7b262605968.target. 2024-04-25T13:30:31.693 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-db01fc44-0307-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-db01fc44-0307-11ef-bc93-c7b262605968.target. 2024-04-25T13:30:32.521 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.a 2024-04-25T13:30:32.521 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T13:30:32.770 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-db01fc44-0307-11ef-bc93-c7b262605968.target.wants/ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-db01fc44-0307-11ef-bc93-c7b262605968@.service. 2024-04-25T13:30:33.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:33 smithi027 podman[28744]: 2024-04-25 13:30:32.914303592 +0000 UTC m=+0.018852997 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:30:33.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:33 smithi027 podman[28744]: 2024-04-25 13:30:33.366233354 +0000 UTC m=+0.470782747 container create e729794600e29f560fc802aecf919a141e54d9b350aff7abb71f55a029e91113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD) 2024-04-25T13:30:33.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:33 smithi027 podman[28744]: 2024-04-25 13:30:33.560275849 +0000 UTC m=+0.664825258 container init e729794600e29f560fc802aecf919a141e54d9b350aff7abb71f55a029e91113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T13:30:33.623 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:33 smithi027 podman[28744]: 2024-04-25 13:30:33.563707868 +0000 UTC m=+0.668257260 container start e729794600e29f560fc802aecf919a141e54d9b350aff7abb71f55a029e91113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T13:30:33.658 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T13:30:33.658 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T13:30:33.665 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T13:30:33.665 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T13:30:33.666 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T13:30:33.666 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T13:30:33.666 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon to start... 2024-04-25T13:30:33.666 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon... 2024-04-25T13:30:33.888 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:33 smithi027 ceph-mon[28762]: mkfs db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:33.888 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:33 smithi027 ceph-mon[28762]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T13:30:33.888 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:33 smithi027 bash[28744]: e729794600e29f560fc802aecf919a141e54d9b350aff7abb71f55a029e91113 2024-04-25T13:30:33.888 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:33 smithi027 systemd[1]: Started Ceph mon.a for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:30:34.719 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T13:30:34.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout id: db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:34.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T13:30:34.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:34.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout services: 2024-04-25T13:30:34.725 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.547653s) 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout data: 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.083+0000 7f805cca2700 1 Processor -- start 2024-04-25T13:30:34.726 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.083+0000 7f805cca2700 1 -- start start 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.084+0000 7f805cca2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f8058108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.084+0000 7f805cca2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f80581092a0 con 0x7f80581088a0 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.084+0000 7f805659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f8058108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.084+0000 7f805659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f8058108cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60152/0 (socket says 172.21.15.27:60152) 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.084+0000 7f805659c700 1 -- 172.21.15.27:0/2612156699 learned_addr learned my addr 172.21.15.27:0/2612156699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.084+0000 7f805659c700 1 -- 172.21.15.27:0/2612156699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80581099d0 con 0x7f80581088a0 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.085+0000 7f805659c700 1 --2- 172.21.15.27:0/2612156699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f8058108cb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f804c0098a0 tx=0x7f804c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bf6941f232909b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.085+0000 7f805559a700 1 -- 172.21.15.27:0/2612156699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f804c004160 con 0x7f80581088a0 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.085+0000 7f805559a700 1 -- 172.21.15.27:0/2612156699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f804c004300 con 0x7f80581088a0 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.085+0000 7f805559a700 1 -- 172.21.15.27:0/2612156699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f804c0036a0 con 0x7f80581088a0 2024-04-25T13:30:34.727 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.086+0000 7f805cca2700 1 -- 172.21.15.27:0/2612156699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 msgr2=0x7f8058108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.086+0000 7f805cca2700 1 --2- 172.21.15.27:0/2612156699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f8058108cb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f804c0098a0 tx=0x7f804c009bb0 comp rx=0 tx=0).stop 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.086+0000 7f805cca2700 1 -- 172.21.15.27:0/2612156699 shutdown_connections 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.086+0000 7f805cca2700 1 --2- 172.21.15.27:0/2612156699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f8058108cb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.086+0000 7f805cca2700 1 -- 172.21.15.27:0/2612156699 >> 172.21.15.27:0/2612156699 conn(0x7f8058103f80 msgr2=0x7f80581063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.086+0000 7f805cca2700 1 -- 172.21.15.27:0/2612156699 shutdown_connections 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.086+0000 7f805cca2700 1 -- 172.21.15.27:0/2612156699 wait complete. 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805cca2700 1 Processor -- start 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805cca2700 1 -- start start 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805cca2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f805819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805cca2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8058109f00 con 0x7f80581088a0 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f805819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f805819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60164/0 (socket says 172.21.15.27:60164) 2024-04-25T13:30:34.728 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805659c700 1 -- 172.21.15.27:0/2420142899 learned_addr learned my addr 172.21.15.27:0/2420142899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805659c700 1 -- 172.21.15.27:0/2420142899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805819d980 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.087+0000 7f805659c700 1 --2- 172.21.15.27:0/2420142899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f805819d440 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f804c000c00 tx=0x7f804c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.088+0000 7f80477fe700 1 -- 172.21.15.27:0/2420142899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f804c003db0 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.088+0000 7f80477fe700 1 -- 172.21.15.27:0/2420142899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f804c011630 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.088+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805819dc10 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.088+0000 7f80477fe700 1 -- 172.21.15.27:0/2420142899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f804c011950 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.088+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80581a0820 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.088+0000 7f80477fe700 1 -- 172.21.15.27:0/2420142899 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f804c011af0 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.088+0000 7f80477fe700 1 -- 172.21.15.27:0/2420142899 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f804c01c450 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.089+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8058064cf0 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.091+0000 7f80477fe700 1 -- 172.21.15.27:0/2420142899 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f804c004260 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.166+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f805802e810 con 0x7f80581088a0 2024-04-25T13:30:34.729 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.166+0000 7f80477fe700 1 -- 172.21.15.27:0/2420142899 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f804c01b450 con 0x7f80581088a0 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.167+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 msgr2=0x7f805819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.167+0000 7f805cca2700 1 --2- 172.21.15.27:0/2420142899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f805819d440 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f804c000c00 tx=0x7f804c004750 comp rx=0 tx=0).stop 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.167+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 shutdown_connections 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.167+0000 7f805cca2700 1 --2- 172.21.15.27:0/2420142899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80581088a0 0x7f805819d440 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.167+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 >> 172.21.15.27:0/2420142899 conn(0x7f8058103f80 msgr2=0x7f8058193b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.167+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 shutdown_connections 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:34.167+0000 7f805cca2700 1 -- 172.21.15.27:0/2420142899 wait complete. 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:mon is available 2024-04-25T13:30:34.730 INFO:teuthology.orchestra.run.smithi027.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: monmap epoch 1 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: last_changed 2024-04-25T13:30:20.883827+0000 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: min_mon_release 19 (squid) 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: election_strategy: 1 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:30:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: fsmap 2024-04-25T13:30:34.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T13:30:34.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: mgrmap e1: no daemons active 2024-04-25T13:30:34.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:34 smithi027 ceph-mon[28762]: from='client.? 172.21.15.27:0/2420142899' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:35.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.141+0000 7f2f5d215700 1 Processor -- start 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.141+0000 7f2f5d215700 1 -- start start 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.141+0000 7f2f5d215700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f58108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.141+0000 7f2f5d215700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f581092a0 con 0x7f2f581088a0 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.142+0000 7f2f56d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f58108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:35.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.142+0000 7f2f56d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f58108cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60172/0 (socket says 172.21.15.27:60172) 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.142+0000 7f2f56d9d700 1 -- 172.21.15.27:0/867971386 learned_addr learned my addr 172.21.15.27:0/867971386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.142+0000 7f2f56d9d700 1 -- 172.21.15.27:0/867971386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f581099d0 con 0x7f2f581088a0 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.142+0000 7f2f56d9d700 1 --2- 172.21.15.27:0/867971386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f58108cb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0098a0 tx=0x7f2f4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ca0f13aa220948 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.142+0000 7f2f55d9b700 1 -- 172.21.15.27:0/867971386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f4c004160 con 0x7f2f581088a0 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.142+0000 7f2f55d9b700 1 -- 172.21.15.27:0/867971386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f4c004300 con 0x7f2f581088a0 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.142+0000 7f2f55d9b700 1 -- 172.21.15.27:0/867971386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f4c0036a0 con 0x7f2f581088a0 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.143+0000 7f2f5d215700 1 -- 172.21.15.27:0/867971386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 msgr2=0x7f2f58108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.143+0000 7f2f5d215700 1 --2- 172.21.15.27:0/867971386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f58108cb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0098a0 tx=0x7f2f4c009bb0 comp rx=0 tx=0).stop 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.143+0000 7f2f5d215700 1 -- 172.21.15.27:0/867971386 shutdown_connections 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.143+0000 7f2f5d215700 1 --2- 172.21.15.27:0/867971386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f58108cb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.143+0000 7f2f5d215700 1 -- 172.21.15.27:0/867971386 >> 172.21.15.27:0/867971386 conn(0x7f2f58103f80 msgr2=0x7f2f581063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.143+0000 7f2f5d215700 1 -- 172.21.15.27:0/867971386 shutdown_connections 2024-04-25T13:30:35.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.143+0000 7f2f5d215700 1 -- 172.21.15.27:0/867971386 wait complete. 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.144+0000 7f2f5d215700 1 Processor -- start 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.144+0000 7f2f5d215700 1 -- start start 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.144+0000 7f2f5d215700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f5807c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.144+0000 7f2f5d215700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58109f00 con 0x7f2f581088a0 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.144+0000 7f2f56d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f5807c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f56d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f5807c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60176/0 (socket says 172.21.15.27:60176) 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f56d9d700 1 -- 172.21.15.27:0/565623382 learned_addr learned my addr 172.21.15.27:0/565623382 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f56d9d700 1 -- 172.21.15.27:0/565623382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5807ce40 con 0x7f2f581088a0 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f56d9d700 1 --2- 172.21.15.27:0/565623382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f5807c900 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c000c00 tx=0x7f2f4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f3ffff700 1 -- 172.21.15.27:0/565623382 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f4c003db0 con 0x7f2f581088a0 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f3ffff700 1 -- 172.21.15.27:0/565623382 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f4c011630 con 0x7f2f581088a0 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f581a01d0 con 0x7f2f581088a0 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f3ffff700 1 -- 172.21.15.27:0/565623382 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f4c011950 con 0x7f2f581088a0 2024-04-25T13:30:35.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.145+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f581a0650 con 0x7f2f581088a0 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.146+0000 7f2f3ffff700 1 -- 172.21.15.27:0/565623382 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2f4c011af0 con 0x7f2f581088a0 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.146+0000 7f2f3ffff700 1 -- 172.21.15.27:0/565623382 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f4c01c450 con 0x7f2f581088a0 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.146+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f58064cf0 con 0x7f2f581088a0 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.149+0000 7f2f3ffff700 1 -- 172.21.15.27:0/565623382 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2f4c032020 con 0x7f2f581088a0 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.219+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2f5802db60 con 0x7f2f581088a0 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.223+0000 7f2f3ffff700 1 -- 172.21.15.27:0/565623382 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f2f4c004260 con 0x7f2f581088a0 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.224+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 msgr2=0x7f2f5807c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.224+0000 7f2f5d215700 1 --2- 172.21.15.27:0/565623382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f5807c900 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c000c00 tx=0x7f2f4c004750 comp rx=0 tx=0).stop 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.224+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 shutdown_connections 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.224+0000 7f2f5d215700 1 --2- 172.21.15.27:0/565623382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f581088a0 0x7f2f5807c900 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.224+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 >> 172.21.15.27:0/565623382 conn(0x7f2f58103f80 msgr2=0x7f2f581929f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.224+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 shutdown_connections 2024-04-25T13:30:35.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:35.224+0000 7f2f5d215700 1 -- 172.21.15.27:0/565623382 wait complete. 2024-04-25T13:30:35.740 INFO:teuthology.orchestra.run.smithi027.stdout:Generating new minimal ceph.conf... 2024-04-25T13:30:35.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:35 smithi027 ceph-mon[28762]: from='client.? 172.21.15.27:0/565623382' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T13:30:35.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:35 smithi027 ceph-mon[28762]: from='client.? 172.21.15.27:0/565623382' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T13:30:36.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:36 smithi027 ceph-mon[28762]: from='client.? 172.21.15.27:0/75451744' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:30:38.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.158+0000 7faab4df4700 1 Processor -- start 2024-04-25T13:30:38.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.158+0000 7faab4df4700 1 -- start start 2024-04-25T13:30:38.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.158+0000 7faab4df4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab010e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:38.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.158+0000 7faab4df4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faab010ee60 con 0x7faab007d4b0 2024-04-25T13:30:38.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.159+0000 7faaae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab010e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:38.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.159+0000 7faaae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab010e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60180/0 (socket says 172.21.15.27:60180) 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.159+0000 7faaae59c700 1 -- 172.21.15.27:0/1305703595 learned_addr learned my addr 172.21.15.27:0/1305703595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.159+0000 7faaae59c700 1 -- 172.21.15.27:0/1305703595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab010efe0 con 0x7faab007d4b0 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.159+0000 7faaae59c700 1 --2- 172.21.15.27:0/1305703595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab010e920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7faaa0009e80 tx=0x7faaa000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bff4778f95e54d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.159+0000 7faaad59a700 1 -- 172.21.15.27:0/1305703595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaa0017070 con 0x7faab007d4b0 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.159+0000 7faaad59a700 1 -- 172.21.15.27:0/1305703595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faaa000b8c0 con 0x7faab007d4b0 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.159+0000 7faaad59a700 1 -- 172.21.15.27:0/1305703595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaa0004490 con 0x7faab007d4b0 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.160+0000 7faab4df4700 1 -- 172.21.15.27:0/1305703595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 msgr2=0x7faab010e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.160+0000 7faab4df4700 1 --2- 172.21.15.27:0/1305703595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab010e920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7faaa0009e80 tx=0x7faaa000b330 comp rx=0 tx=0).stop 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.160+0000 7faab4df4700 1 -- 172.21.15.27:0/1305703595 shutdown_connections 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.160+0000 7faab4df4700 1 --2- 172.21.15.27:0/1305703595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab010e920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.160+0000 7faab4df4700 1 -- 172.21.15.27:0/1305703595 >> 172.21.15.27:0/1305703595 conn(0x7faab0103f10 msgr2=0x7faab01063a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.160+0000 7faab4df4700 1 -- 172.21.15.27:0/1305703595 shutdown_connections 2024-04-25T13:30:38.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.160+0000 7faab4df4700 1 -- 172.21.15.27:0/1305703595 wait complete. 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.161+0000 7faab4df4700 1 Processor -- start 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.161+0000 7faab4df4700 1 -- start start 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.161+0000 7faab4df4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab019fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.161+0000 7faab4df4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faab010fef0 con 0x7faab007d4b0 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.161+0000 7faaae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab019fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.161+0000 7faaae59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab019fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60188/0 (socket says 172.21.15.27:60188) 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.161+0000 7faaae59c700 1 -- 172.21.15.27:0/75451744 learned_addr learned my addr 172.21.15.27:0/75451744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faaae59c700 1 -- 172.21.15.27:0/75451744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab01a0430 con 0x7faab007d4b0 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faaae59c700 1 --2- 172.21.15.27:0/75451744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab019fef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faaa0004220 tx=0x7faaa000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faaa77fe700 1 -- 172.21.15.27:0/75451744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaa0020020 con 0x7faab007d4b0 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab01a06c0 con 0x7faab007d4b0 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faaa77fe700 1 -- 172.21.15.27:0/75451744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faaa0012430 con 0x7faab007d4b0 2024-04-25T13:30:38.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faaa77fe700 1 -- 172.21.15.27:0/75451744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaa0017040 con 0x7faab007d4b0 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab01a32d0 con 0x7faab007d4b0 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faaa77fe700 1 -- 172.21.15.27:0/75451744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faaa0023020 con 0x7faab007d4b0 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.162+0000 7faaa77fe700 1 -- 172.21.15.27:0/75451744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7faaa000f050 con 0x7faab007d4b0 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.163+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab0064cf0 con 0x7faab007d4b0 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.165+0000 7faaa77fe700 1 -- 172.21.15.27:0/75451744 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7faaa002a020 con 0x7faab007d4b0 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.236+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7faab01a34a0 con 0x7faab007d4b0 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.236+0000 7faaa77fe700 1 -- 172.21.15.27:0/75451744 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7faaa000b860 con 0x7faab007d4b0 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.237+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 msgr2=0x7faab019fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.237+0000 7faab4df4700 1 --2- 172.21.15.27:0/75451744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab019fef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faaa0004220 tx=0x7faaa000bdc0 comp rx=0 tx=0).stop 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.237+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 shutdown_connections 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.237+0000 7faab4df4700 1 --2- 172.21.15.27:0/75451744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab007d4b0 0x7faab019fef0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.237+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 >> 172.21.15.27:0/75451744 conn(0x7faab0103f10 msgr2=0x7faab01047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.237+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 shutdown_connections 2024-04-25T13:30:38.739 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:36.237+0000 7faab4df4700 1 -- 172.21.15.27:0/75451744 wait complete. 2024-04-25T13:30:38.740 INFO:teuthology.orchestra.run.smithi027.stdout:Restarting the monitor... 2024-04-25T13:30:39.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:38 smithi027 systemd[1]: Stopping Ceph mon.a for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:30:39.787 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:39 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a[28758]: 2024-04-25T13:30:39.453+0000 7f723e5a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:30:39.787 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:39 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a[28758]: 2024-04-25T13:30:39.453+0000 7f723e5a2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T13:30:39.787 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:39 smithi027 podman[29010]: 2024-04-25 13:30:39.536331637 +0000 UTC m=+0.619768130 container died e729794600e29f560fc802aecf919a141e54d9b350aff7abb71f55a029e91113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-25T13:30:40.145 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:39 smithi027 podman[29010]: 2024-04-25 13:30:39.88659791 +0000 UTC m=+0.970034403 container cleanup e729794600e29f560fc802aecf919a141e54d9b350aff7abb71f55a029e91113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:30:40.146 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:39 smithi027 bash[29010]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a 2024-04-25T13:30:40.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 podman[29024]: 2024-04-25 13:30:40.145780466 +0000 UTC m=+0.607935902 container remove e729794600e29f560fc802aecf919a141e54d9b350aff7abb71f55a029e91113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:30:40.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 systemd[1]: ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T13:30:40.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 systemd[1]: Stopped Ceph mon.a for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:30:40.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 systemd[1]: Starting Ceph mon.a for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:30:40.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 podman[29121]: 2024-04-25 13:30:40.369323266 +0000 UTC m=+0.018222749 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:30:40.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 podman[29121]: 2024-04-25 13:30:40.488504068 +0000 UTC m=+0.137403539 container create d3ac28c1a0c9b3cb569a0c68e9252c201386d0464c1ab1d7d9b17c9c3dc12225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:30:40.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 podman[29121]: 2024-04-25 13:30:40.69067696 +0000 UTC m=+0.339576431 container init d3ac28c1a0c9b3cb569a0c68e9252c201386d0464c1ab1d7d9b17c9c3dc12225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T13:30:40.727 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 podman[29121]: 2024-04-25 13:30:40.693713199 +0000 UTC m=+0.342612672 container start d3ac28c1a0c9b3cb569a0c68e9252c201386d0464c1ab1d7d9b17c9c3dc12225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T13:30:40.762 INFO:teuthology.orchestra.run.smithi027.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T13:30:40.990 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T13:30:40.990 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: pidfile_write: ignore empty --pid-file 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: load: jerasure load: lrc 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: RocksDB version: 7.9.2 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Git sha 0 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: DB SUMMARY 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: DB Session ID: QZQTGCX9IEYLPGS4KHNH 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: CURRENT file: CURRENT 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T13:30:40.991 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104072 ; 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.error_if_exists: 0 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.create_if_missing: 0 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.paranoid_checks: 1 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.env: 0x562f94a4a500 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.info_log: 0x562f95afbfc0 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.statistics: (nil) 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.use_fsync: 0 2024-04-25T13:30:40.992 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_log_file_size: 0 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.allow_fallocate: 1 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.use_direct_reads: 0 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.db_log_dir: 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.wal_dir: 2024-04-25T13:30:40.993 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T13:30:40.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T13:30:40.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T13:30:40.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T13:30:40.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T13:30:40.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T13:30:40.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T13:30:40.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.write_buffer_manager: 0x562f95acc8c0 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.unordered_write: 0 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T13:30:40.995 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.row_cache: None 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.wal_filter: None 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.two_write_queues: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.wal_compression: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.atomic_flush: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.log_readahead_size: 0 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T13:30:40.996 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T13:30:40.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T13:30:40.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T13:30:40.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T13:30:40.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T13:30:40.997 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T13:30:40.998 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_background_jobs: 2 2024-04-25T13:30:40.998 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_background_compactions: -1 2024-04-25T13:30:40.998 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_subcompactions: 1 2024-04-25T13:30:40.998 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T13:30:40.998 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_open_files: -1 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_background_flushes: -1 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Compression algorithms supported: 2024-04-25T13:30:40.999 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: kZSTD supported: 0 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: kXpressCompression supported: 0 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: kZlibCompression supported: 1 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: kSnappyCompression supported: 1 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: kLZ4Compression supported: 1 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: kBZip2Compression supported: 0 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.merge_operator: 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_filter: None 2024-04-25T13:30:41.000 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562f95c021a0) 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_type: 0 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_index_type: 0 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_shortening: 1 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: checksum: 4 2024-04-25T13:30:41.001 INFO:journalctl@ceph.mon.a.smithi027.stdout: no_block_cache: 0 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache: 0x562f95aff1f0 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_options: 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: capacity : 536870912 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_shard_bits : 4 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: strict_capacity_limit : 0 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_cache_compressed: (nil) 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: persistent_cache: (nil) 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size: 4096 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_size_deviation: 10 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_restart_interval: 16 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: index_block_restart_interval: 1 2024-04-25T13:30:41.002 INFO:journalctl@ceph.mon.a.smithi027.stdout: metadata_block_size: 4096 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: partition_filters: 0 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: use_delta_encoding: 1 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: filter_policy: bloomfilter 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: whole_key_filtering: 1 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: verify_compression: 0 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: format_version: 5 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_index_compression: 1 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: block_align: 0 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: max_auto_readahead_size: 262144 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: prepopulate_block_cache: 0 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: initial_auto_readahead_size: 8192 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T13:30:41.003 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression: NoCompression 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.num_levels: 7 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T13:30:41.004 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T13:30:41.005 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T13:30:41.006 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T13:30:41.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.inplace_update_support: 0 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.bloom_locality: 0 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.max_successive_merges: 0 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T13:30:41.008 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.ttl: 2592000 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.enable_blob_files: false 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.min_blob_size: 0 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T13:30:41.009 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1d1e767d-726a-4498-a181-128d70fb5be2 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051840734193, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051840735471, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99300, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97246, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13917, "raw_average_key_size": 49, "raw_value_size": 89640, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714051840, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1d1e767d-726a-4498-a181-128d70fb5be2", "db_session_id": "QZQTGCX9IEYLPGS4KHNH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051840735555, "job": 1, "event": "recovery_finished"} 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562f95ae3c00 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: DB pointer 0x562f9a8d6000 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T13:30:41.010 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** DB Stats ** 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: L0 2/0 98.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Sum 2/0 98.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T13:30:41.011 INFO:journalctl@ceph.mon.a.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T13:30:41.012 INFO:journalctl@ceph.mon.a.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: Cumulative compaction: 0.00 GB write, 28.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: Interval compaction: 0.00 GB write, 28.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache BinnedLRUCache@0x562f95aff1f0#2 capacity: 512.00 MB usage: 1.41 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.91 KB,0.000172853%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: starting mon.a rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a@-1(???) e1 preinit fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a@-1(???).mds e1 new map 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a@-1(???).mds e1 print_map 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: e1 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T13:30:41.013 INFO:journalctl@ceph.mon.a.smithi027.stdout: legacy client fscid: -1 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout: 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout: No filesystems configured 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 bash[29121]: d3ac28c1a0c9b3cb569a0c68e9252c201386d0464c1ab1d7d9b17c9c3dc12225 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 systemd[1]: Started Ceph mon.a for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: monmap epoch 1 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: last_changed 2024-04-25T13:30:20.883827+0000 2024-04-25T13:30:41.014 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:30:41.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: min_mon_release 19 (squid) 2024-04-25T13:30:41.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: election_strategy: 1 2024-04-25T13:30:41.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:30:41.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: fsmap 2024-04-25T13:30:41.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T13:30:41.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:40 smithi027 ceph-mon[29140]: mgrmap e1: no daemons active 2024-04-25T13:30:41.871 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.207+0000 7f5f10f6a700 1 Processor -- start 2024-04-25T13:30:41.871 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.208+0000 7f5f10f6a700 1 -- start start 2024-04-25T13:30:41.871 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.208+0000 7f5f10f6a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c10c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:41.871 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.208+0000 7f5f10f6a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c10cb30 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.871 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.208+0000 7f5f0a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c10c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.208+0000 7f5f0a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c10c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60202/0 (socket says 172.21.15.27:60202) 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.208+0000 7f5f0a59c700 1 -- 172.21.15.27:0/174217362 learned_addr learned my addr 172.21.15.27:0/174217362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.209+0000 7f5f0a59c700 1 -- 172.21.15.27:0/174217362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c10ccb0 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.209+0000 7f5f0a59c700 1 --2- 172.21.15.27:0/174217362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c10c540 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0098a0 tx=0x7f5efc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=544a90298554458e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.209+0000 7f5f0959a700 1 -- 172.21.15.27:0/174217362 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5efc004160 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.209+0000 7f5f0959a700 1 -- 172.21.15.27:0/174217362 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5efc004300 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.209+0000 7f5f0959a700 1 -- 172.21.15.27:0/174217362 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5efc011af0 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.210+0000 7f5f10f6a700 1 -- 172.21.15.27:0/174217362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 msgr2=0x7f5f0c10c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.210+0000 7f5f10f6a700 1 --2- 172.21.15.27:0/174217362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c10c540 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0098a0 tx=0x7f5efc009bb0 comp rx=0 tx=0).stop 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.210+0000 7f5f10f6a700 1 -- 172.21.15.27:0/174217362 shutdown_connections 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.210+0000 7f5f10f6a700 1 --2- 172.21.15.27:0/174217362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c10c540 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.210+0000 7f5f10f6a700 1 -- 172.21.15.27:0/174217362 >> 172.21.15.27:0/174217362 conn(0x7f5f0c103f50 msgr2=0x7f5f0c1063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:41.872 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.210+0000 7f5f10f6a700 1 -- 172.21.15.27:0/174217362 shutdown_connections 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.210+0000 7f5f10f6a700 1 -- 172.21.15.27:0/174217362 wait complete. 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.211+0000 7f5f10f6a700 1 Processor -- start 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.211+0000 7f5f10f6a700 1 -- start start 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.211+0000 7f5f10f6a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.211+0000 7f5f10f6a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c10dd30 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.211+0000 7f5f0a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.211+0000 7f5f0a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60204/0 (socket says 172.21.15.27:60204) 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.211+0000 7f5f0a59c700 1 -- 172.21.15.27:0/2508158108 learned_addr learned my addr 172.21.15.27:0/2508158108 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.211+0000 7f5f0a59c700 1 -- 172.21.15.27:0/2508158108 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c19e250 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.212+0000 7f5f0a59c700 1 --2- 172.21.15.27:0/2508158108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c19dd10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009fb0 tx=0x7f5efc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.212+0000 7f5efb7fe700 1 -- 172.21.15.27:0/2508158108 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5efc0036a0 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.212+0000 7f5efb7fe700 1 -- 172.21.15.27:0/2508158108 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5efc01a430 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.212+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0c19e4e0 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.212+0000 7f5efb7fe700 1 -- 172.21.15.27:0/2508158108 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5efc019550 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.873 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.212+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f0c1a10e0 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.212+0000 7f5efb7fe700 1 -- 172.21.15.27:0/2508158108 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5efc019be0 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.212+0000 7f5efb7fe700 1 -- 172.21.15.27:0/2508158108 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5efc024730 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.213+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0c064cf0 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.215+0000 7f5efb7fe700 1 -- 172.21.15.27:0/2508158108 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5efc043e50 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.285+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f5f0c1a12b0 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.287+0000 7f5efb7fe700 1 -- 172.21.15.27:0/2508158108 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f5efc032020 con 0x7f5f0c07d4d0 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.288+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 msgr2=0x7f5f0c19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.288+0000 7f5f10f6a700 1 --2- 172.21.15.27:0/2508158108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c19dd10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009fb0 tx=0x7f5efc011940 comp rx=0 tx=0).stop 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.288+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 shutdown_connections 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.288+0000 7f5f10f6a700 1 --2- 172.21.15.27:0/2508158108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f0c07d4d0 0x7f5f0c19dd10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.288+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 >> 172.21.15.27:0/2508158108 conn(0x7f5f0c103f50 msgr2=0x7f5f0c104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.288+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 shutdown_connections 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:41.288+0000 7f5f10f6a700 1 -- 172.21.15.27:0/2508158108 wait complete. 2024-04-25T13:30:41.874 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T13:30:41.896 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:30:41.896 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mgr... 2024-04-25T13:30:41.897 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T13:30:41.897 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T13:30:42.154 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.x 2024-04-25T13:30:42.154 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T13:30:42.311 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-db01fc44-0307-11ef-bc93-c7b262605968.target.wants/ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-db01fc44-0307-11ef-bc93-c7b262605968@.service. 2024-04-25T13:30:42.550 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:42 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2508158108' entity='client.admin' 2024-04-25T13:30:42.550 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:42 smithi027 systemd[1]: Starting Ceph mgr.x for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:30:42.802 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:42 smithi027 podman[29364]: 2024-04-25 13:30:42.453519511 +0000 UTC m=+0.018587940 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:30:42.802 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:42 smithi027 podman[29364]: 2024-04-25 13:30:42.557452778 +0000 UTC m=+0.122521206 container create e2e0c7abc52382931cf5e11ed4a1e221850cfe21c41e0a6c098882ff870501a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T13:30:42.802 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:42 smithi027 podman[29364]: 2024-04-25 13:30:42.80184983 +0000 UTC m=+0.366918259 container init e2e0c7abc52382931cf5e11ed4a1e221850cfe21c41e0a6c098882ff870501a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T13:30:42.876 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T13:30:42.876 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T13:30:42.885 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T13:30:42.885 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T13:30:42.885 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T13:30:42.886 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T13:30:42.896 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T13:30:42.896 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T13:30:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T13:30:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T13:30:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T13:30:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T13:30:42.905 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr to start... 2024-04-25T13:30:42.906 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr... 2024-04-25T13:30:43.096 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:42 smithi027 podman[29364]: 2024-04-25 13:30:42.804697318 +0000 UTC m=+0.369765754 container start e2e0c7abc52382931cf5e11ed4a1e221850cfe21c41e0a6c098882ff870501a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T13:30:43.096 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:42 smithi027 bash[29364]: e2e0c7abc52382931cf5e11ed4a1e221850cfe21c41e0a6c098882ff870501a3 2024-04-25T13:30:43.096 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:42 smithi027 systemd[1]: Started Ceph mgr.x for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:30:43.096 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:42 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:42.982+0000 7f72f6c43200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T13:30:43.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:43 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:43.180+0000 7f72f6c43200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:30:43.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:43 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:43.289+0000 7f72f6c43200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T13:30:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:43 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1339568902' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T13:30:43.653 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:43 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:43.618+0000 7f72f6c43200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T13:30:44.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:43 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:43.706+0000 7f72f6c43200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T13:30:44.528 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:44.528 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "db01fc44-0307-11ef-bc93-c7b262605968", 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T13:30:44.529 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T13:30:44.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T13:30:44.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:44.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T13:30:33.621260+0000", 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.317+0000 7f73b459b700 1 Processor -- start 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.317+0000 7f73b459b700 1 -- start start 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.317+0000 7f73b459b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac10c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.317+0000 7f73b459b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac10cae0 con 0x7f73ac07cc40 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.318+0000 7f73b2337700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac10c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:44.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.318+0000 7f73b2337700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac10c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60230/0 (socket says 172.21.15.27:60230) 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.318+0000 7f73b2337700 1 -- 172.21.15.27:0/4241235225 learned_addr learned my addr 172.21.15.27:0/4241235225 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.318+0000 7f73b2337700 1 -- 172.21.15.27:0/4241235225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac10cc60 con 0x7f73ac07cc40 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.318+0000 7f73b2337700 1 --2- 172.21.15.27:0/4241235225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac10c4f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f73a80098a0 tx=0x7f73a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391bfce1547f3b90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.319+0000 7f73b1335700 1 -- 172.21.15.27:0/4241235225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73a8004160 con 0x7f73ac07cc40 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.319+0000 7f73b1335700 1 -- 172.21.15.27:0/4241235225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f73a8004300 con 0x7f73ac07cc40 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.319+0000 7f73b1335700 1 -- 172.21.15.27:0/4241235225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73a8011af0 con 0x7f73ac07cc40 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.319+0000 7f73b459b700 1 -- 172.21.15.27:0/4241235225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 msgr2=0x7f73ac10c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.319+0000 7f73b459b700 1 --2- 172.21.15.27:0/4241235225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac10c4f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f73a80098a0 tx=0x7f73a8009bb0 comp rx=0 tx=0).stop 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.320+0000 7f73b459b700 1 -- 172.21.15.27:0/4241235225 shutdown_connections 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.320+0000 7f73b459b700 1 --2- 172.21.15.27:0/4241235225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac10c4f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.320+0000 7f73b459b700 1 -- 172.21.15.27:0/4241235225 >> 172.21.15.27:0/4241235225 conn(0x7f73ac103f80 msgr2=0x7f73ac1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.320+0000 7f73b459b700 1 -- 172.21.15.27:0/4241235225 shutdown_connections 2024-04-25T13:30:44.534 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.320+0000 7f73b459b700 1 -- 172.21.15.27:0/4241235225 wait complete. 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.320+0000 7f73b459b700 1 Processor -- start 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.320+0000 7f73b459b700 1 -- start start 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73b459b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac109c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73b459b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac10dce0 con 0x7f73ac07cc40 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73b2337700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73b2337700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac109c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60240/0 (socket says 172.21.15.27:60240) 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73b2337700 1 -- 172.21.15.27:0/1339568902 learned_addr learned my addr 172.21.15.27:0/1339568902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73b2337700 1 -- 172.21.15.27:0/1339568902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac10a1a0 con 0x7f73ac07cc40 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73b2337700 1 --2- 172.21.15.27:0/1339568902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac109c60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009fb0 tx=0x7f73a8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73a37fe700 1 -- 172.21.15.27:0/1339568902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73a80036a0 con 0x7f73ac07cc40 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73a37fe700 1 -- 172.21.15.27:0/1339568902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f73a801a430 con 0x7f73ac07cc40 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.321+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ac1083f0 con 0x7f73ac07cc40 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.322+0000 7f73a37fe700 1 -- 172.21.15.27:0/1339568902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73a8019550 con 0x7f73ac07cc40 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.322+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ac1088d0 con 0x7f73ac07cc40 2024-04-25T13:30:44.535 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.322+0000 7f73a37fe700 1 -- 172.21.15.27:0/1339568902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f73a8019b90 con 0x7f73ac07cc40 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.322+0000 7f73a37fe700 1 -- 172.21.15.27:0/1339568902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f73a8024730 con 0x7f73ac07cc40 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.323+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ac064cf0 con 0x7f73ac07cc40 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.325+0000 7f73a37fe700 1 -- 172.21.15.27:0/1339568902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f73a8043e50 con 0x7f73ac07cc40 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.400+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f73ac108b50 con 0x7f73ac07cc40 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.400+0000 7f73a37fe700 1 -- 172.21.15.27:0/1339568902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f73a8032020 con 0x7f73ac07cc40 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.402+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 msgr2=0x7f73ac109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.402+0000 7f73b459b700 1 --2- 172.21.15.27:0/1339568902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac109c60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009fb0 tx=0x7f73a8011940 comp rx=0 tx=0).stop 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.402+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 shutdown_connections 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.402+0000 7f73b459b700 1 --2- 172.21.15.27:0/1339568902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73ac07cc40 0x7f73ac109c60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.402+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 >> 172.21.15.27:0/1339568902 conn(0x7f73ac103f80 msgr2=0x7f73ac105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.402+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 shutdown_connections 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:43.402+0000 7f73b459b700 1 -- 172.21.15.27:0/1339568902 wait complete. 2024-04-25T13:30:44.536 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (1/15)... 2024-04-25T13:30:45.318 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:44 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:44.966+0000 7f72f6c43200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T13:30:45.319 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:45 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:45.141+0000 7f72f6c43200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T13:30:45.319 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:45 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:45.318+0000 7f72f6c43200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T13:30:45.903 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:45 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:45.614+0000 7f72f6c43200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T13:30:45.903 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:45 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:45.709+0000 7f72f6c43200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T13:30:46.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:45 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:45.919+0000 7f72f6c43200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T13:30:46.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:46 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:46.115+0000 7f72f6c43200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:30:46.901 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:46 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:46.749+0000 7f72f6c43200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T13:30:47.304 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:47 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/525456333' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T13:30:47.304 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:47.204+0000 7f72f6c43200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T13:30:47.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:47.303+0000 7f72f6c43200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T13:30:47.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:47.511+0000 7f72f6c43200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T13:30:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T13:30:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "db01fc44-0307-11ef-bc93-c7b262605968", 2024-04-25T13:30:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T13:30:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T13:30:47.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T13:30:47.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:47.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T13:30:47.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:47.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T13:30:47.738 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T13:30:47.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T13:30:47.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T13:30:47.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T13:30:47.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T13:30:47.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:47.746 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T13:30:47.747 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T13:30:33.621260+0000", 2024-04-25T13:30:47.748 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.989+0000 7f2d7320a700 1 Processor -- start 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.989+0000 7f2d7320a700 1 -- start start 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.989+0000 7f2d7320a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.990+0000 7f2d7320a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c1092a0 con 0x7f2d6c1088a0 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.990+0000 7f2d70fa6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.990+0000 7f2d70fa6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c108cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54610/0 (socket says 172.21.15.27:54610) 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.990+0000 7f2d70fa6700 1 -- 172.21.15.27:0/4275799624 learned_addr learned my addr 172.21.15.27:0/4275799624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.990+0000 7f2d70fa6700 1 -- 172.21.15.27:0/4275799624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6c1099d0 con 0x7f2d6c1088a0 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.990+0000 7f2d70fa6700 1 --2- 172.21.15.27:0/4275799624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c108cb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098a0 tx=0x7f2d60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df94fadbdd772b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:47.749 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.990+0000 7f2d6b7fe700 1 -- 172.21.15.27:0/4275799624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d60004160 con 0x7f2d6c1088a0 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.991+0000 7f2d6b7fe700 1 -- 172.21.15.27:0/4275799624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2d60004300 con 0x7f2d6c1088a0 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.991+0000 7f2d6b7fe700 1 -- 172.21.15.27:0/4275799624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d60011af0 con 0x7f2d6c1088a0 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.992+0000 7f2d7320a700 1 -- 172.21.15.27:0/4275799624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 msgr2=0x7f2d6c108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.992+0000 7f2d7320a700 1 --2- 172.21.15.27:0/4275799624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c108cb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098a0 tx=0x7f2d60009bb0 comp rx=0 tx=0).stop 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.992+0000 7f2d7320a700 1 -- 172.21.15.27:0/4275799624 shutdown_connections 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.992+0000 7f2d7320a700 1 --2- 172.21.15.27:0/4275799624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c108cb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.992+0000 7f2d7320a700 1 -- 172.21.15.27:0/4275799624 >> 172.21.15.27:0/4275799624 conn(0x7f2d6c103f80 msgr2=0x7f2d6c1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.992+0000 7f2d7320a700 1 -- 172.21.15.27:0/4275799624 shutdown_connections 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.992+0000 7f2d7320a700 1 -- 172.21.15.27:0/4275799624 wait complete. 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.993+0000 7f2d7320a700 1 Processor -- start 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.993+0000 7f2d7320a700 1 -- start start 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.993+0000 7f2d7320a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c07c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:47.750 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.993+0000 7f2d7320a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c10cd80 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.993+0000 7f2d70fa6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c07c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.993+0000 7f2d70fa6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c07c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54626/0 (socket says 172.21.15.27:54626) 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.993+0000 7f2d70fa6700 1 -- 172.21.15.27:0/525456333 learned_addr learned my addr 172.21.15.27:0/525456333 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.993+0000 7f2d70fa6700 1 -- 172.21.15.27:0/525456333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6c07ce40 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.994+0000 7f2d70fa6700 1 --2- 172.21.15.27:0/525456333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c07c900 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d60011890 tx=0x7f2d60011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.994+0000 7f2d69ffb700 1 -- 172.21.15.27:0/525456333 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d600039e0 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.994+0000 7f2d69ffb700 1 -- 172.21.15.27:0/525456333 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2d60003b80 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.994+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6c1a0af0 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.994+0000 7f2d69ffb700 1 -- 172.21.15.27:0/525456333 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d600246e0 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.994+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6c1a0f70 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.994+0000 7f2d69ffb700 1 -- 172.21.15.27:0/525456333 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d60022020 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.994+0000 7f2d69ffb700 1 -- 172.21.15.27:0/525456333 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2d6002dc40 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.995+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d6c064cf0 con 0x7f2d6c1088a0 2024-04-25T13:30:47.751 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:46.998+0000 7f2d69ffb700 1 -- 172.21.15.27:0/525456333 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2d60043e50 con 0x7f2d6c1088a0 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.072+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2d6c1a1310 con 0x7f2d6c1088a0 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.073+0000 7f2d69ffb700 1 -- 172.21.15.27:0/525456333 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2d60032020 con 0x7f2d6c1088a0 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.074+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 msgr2=0x7f2d6c07c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.074+0000 7f2d7320a700 1 --2- 172.21.15.27:0/525456333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c07c900 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d60011890 tx=0x7f2d60011970 comp rx=0 tx=0).stop 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.074+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 shutdown_connections 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.074+0000 7f2d7320a700 1 --2- 172.21.15.27:0/525456333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d6c1088a0 0x7f2d6c07c900 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.074+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 >> 172.21.15.27:0/525456333 conn(0x7f2d6c103f80 msgr2=0x7f2d6c105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.074+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 shutdown_connections 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:47.074+0000 7f2d7320a700 1 -- 172.21.15.27:0/525456333 wait complete. 2024-04-25T13:30:47.752 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (2/15)... 2024-04-25T13:30:48.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:48.056+0000 7f72f6c43200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T13:30:48.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:48.151+0000 7f72f6c43200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T13:30:48.754 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:48.400+0000 7f72f6c43200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T13:30:48.755 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:48.538+0000 7f72f6c43200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T13:30:48.755 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:48.634+0000 7f72f6c43200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T13:30:48.755 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:48.753+0000 7f72f6c43200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T13:30:49.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:49.066+0000 7f72f6c43200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T13:30:49.968 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:49.686+0000 7f72f6c43200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T13:30:49.968 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:49.911+0000 7f72f6c43200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T13:30:50.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: Activating manager daemon x 2024-04-25T13:30:50.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: mgrmap e2: x(active, starting, since 0.00384971s) 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: Manager daemon x is now available 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' 2024-04-25T13:30:50.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:49 smithi027 ceph-mon[29140]: from='mgr.14100 172.21.15.27:0/1630610120' entity='mgr.x' 2024-04-25T13:30:50.846 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:50.846 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T13:30:50.846 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "db01fc44-0307-11ef-bc93-c7b262605968", 2024-04-25T13:30:50.846 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T13:30:50.846 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T13:30:50.846 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T13:30:50.846 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:50.847 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T13:30:50.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:50.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T13:30:50.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T13:30:50.848 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T13:30:50.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T13:30:50.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:50.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T13:30:33.621260+0000", 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.200+0000 7f5deb0de700 1 Processor -- start 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.200+0000 7f5deb0de700 1 -- start start 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5deb0de700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de410e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5deb0de700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5de410eea0 con 0x7f5de407cc00 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5de8e7a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de410e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:50.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5de8e7a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de410e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54710/0 (socket says 172.21.15.27:54710) 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5de8e7a700 1 -- 172.21.15.27:0/2173963211 learned_addr learned my addr 172.21.15.27:0/2173963211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5de8e7a700 1 -- 172.21.15.27:0/2173963211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de410f020 con 0x7f5de407cc00 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5de8e7a700 1 --2- 172.21.15.27:0/2173963211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de410e960 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009e80 tx=0x7f5ddc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9998cb20961ec1fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5de37fe700 1 -- 172.21.15.27:0/2173963211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ddc017070 con 0x7f5de407cc00 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5de37fe700 1 -- 172.21.15.27:0/2173963211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5ddc00b8c0 con 0x7f5de407cc00 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.201+0000 7f5de37fe700 1 -- 172.21.15.27:0/2173963211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ddc004490 con 0x7f5de407cc00 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.202+0000 7f5deb0de700 1 -- 172.21.15.27:0/2173963211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 msgr2=0x7f5de410e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.202+0000 7f5deb0de700 1 --2- 172.21.15.27:0/2173963211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de410e960 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009e80 tx=0x7f5ddc00b330 comp rx=0 tx=0).stop 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.202+0000 7f5deb0de700 1 -- 172.21.15.27:0/2173963211 shutdown_connections 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.202+0000 7f5deb0de700 1 --2- 172.21.15.27:0/2173963211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de410e960 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.202+0000 7f5deb0de700 1 -- 172.21.15.27:0/2173963211 >> 172.21.15.27:0/2173963211 conn(0x7f5de4103fb0 msgr2=0x7f5de4106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.202+0000 7f5deb0de700 1 -- 172.21.15.27:0/2173963211 shutdown_connections 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.202+0000 7f5deb0de700 1 -- 172.21.15.27:0/2173963211 wait complete. 2024-04-25T13:30:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.203+0000 7f5deb0de700 1 Processor -- start 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.203+0000 7f5deb0de700 1 -- start start 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.203+0000 7f5deb0de700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de419fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.203+0000 7f5deb0de700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5de410ff30 con 0x7f5de407cc00 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.203+0000 7f5de8e7a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de419fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.203+0000 7f5de8e7a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de419fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54716/0 (socket says 172.21.15.27:54716) 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.203+0000 7f5de8e7a700 1 -- 172.21.15.27:0/2477603790 learned_addr learned my addr 172.21.15.27:0/2477603790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.204+0000 7f5de8e7a700 1 -- 172.21.15.27:0/2477603790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de41a0410 con 0x7f5de407cc00 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.204+0000 7f5de8e7a700 1 --2- 172.21.15.27:0/2477603790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de419fed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00b7b0 tx=0x7f5ddc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.204+0000 7f5de1ffb700 1 -- 172.21.15.27:0/2477603790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ddc020020 con 0x7f5de407cc00 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.204+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de41a06a0 con 0x7f5de407cc00 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.204+0000 7f5de1ffb700 1 -- 172.21.15.27:0/2477603790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5ddc012430 con 0x7f5de407cc00 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.204+0000 7f5de1ffb700 1 -- 172.21.15.27:0/2477603790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ddc017040 con 0x7f5de407cc00 2024-04-25T13:30:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.204+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de41a32b0 con 0x7f5de407cc00 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.205+0000 7f5de1ffb700 1 -- 172.21.15.27:0/2477603790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 2) ==== 49272+0+0 (secure 0 0 0) 0x7f5ddc023020 con 0x7f5de407cc00 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.205+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de4064cf0 con 0x7f5de407cc00 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.205+0000 7f5de1ffb700 1 -- 172.21.15.27:0/2477603790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5ddc00f050 con 0x7f5de407cc00 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.208+0000 7f5de1ffb700 1 -- 172.21.15.27:0/2477603790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5ddc02a020 con 0x7f5de407cc00 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.280+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5de41a3930 con 0x7f5de407cc00 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.281+0000 7f5de1ffb700 1 -- 172.21.15.27:0/2477603790 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5ddc02dd60 con 0x7f5de407cc00 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.282+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 msgr2=0x7f5de419fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.282+0000 7f5deb0de700 1 --2- 172.21.15.27:0/2477603790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de419fed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00b7b0 tx=0x7f5ddc00bd50 comp rx=0 tx=0).stop 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.282+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 shutdown_connections 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.282+0000 7f5deb0de700 1 --2- 172.21.15.27:0/2477603790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5de407cc00 0x7f5de419fed0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.282+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 >> 172.21.15.27:0/2477603790 conn(0x7f5de4103fb0 msgr2=0x7f5de41058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.282+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 shutdown_connections 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:50.282+0000 7f5deb0de700 1 -- 172.21.15.27:0/2477603790 wait complete. 2024-04-25T13:30:50.861 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (3/15)... 2024-04-25T13:30:51.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:50 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2477603790' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T13:30:51.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:50 smithi027 ceph-mon[29140]: mgrmap e3: x(active, since 1.00768s) 2024-04-25T13:30:53.231 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:52 smithi027 ceph-mon[29140]: mgrmap e4: x(active, since 2s) 2024-04-25T13:30:54.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:53 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2304530384' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T13:30:54.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:54.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T13:30:54.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "db01fc44-0307-11ef-bc93-c7b262605968", 2024-04-25T13:30:54.164 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T13:30:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:54.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T13:30:54.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T13:30:54.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:54.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T13:30:54.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:54.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T13:30:54.166 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T13:30:54.175 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T13:30:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T13:30:33.621260+0000", 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.271+0000 7fc2016f2700 1 Processor -- start 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.271+0000 7fc2016f2700 1 -- start start 2024-04-25T13:30:54.177 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.271+0000 7fc2016f2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc109e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.271+0000 7fc2016f2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc10a3f0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.271+0000 7fc1faffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.271+0000 7fc1faffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc109e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54730/0 (socket says 172.21.15.27:54730) 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.271+0000 7fc1faffd700 1 -- 172.21.15.27:0/347085810 learned_addr learned my addr 172.21.15.27:0/347085810 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.272+0000 7fc1faffd700 1 -- 172.21.15.27:0/347085810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1fc10aad0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.272+0000 7fc1faffd700 1 --2- 172.21.15.27:0/347085810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc109e00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec0098a0 tx=0x7fc1ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad47a76bc1a1ed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.272+0000 7fc1f9ffb700 1 -- 172.21.15.27:0/347085810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1ec004160 con 0x7fc1fc1099f0 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.272+0000 7fc1f9ffb700 1 -- 172.21.15.27:0/347085810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc1ec004300 con 0x7fc1fc1099f0 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.272+0000 7fc1f9ffb700 1 -- 172.21.15.27:0/347085810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1ec011af0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.273+0000 7fc2016f2700 1 -- 172.21.15.27:0/347085810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 msgr2=0x7fc1fc109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.273+0000 7fc2016f2700 1 --2- 172.21.15.27:0/347085810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc109e00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec0098a0 tx=0x7fc1ec009bb0 comp rx=0 tx=0).stop 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.273+0000 7fc2016f2700 1 -- 172.21.15.27:0/347085810 shutdown_connections 2024-04-25T13:30:54.178 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.273+0000 7fc2016f2700 1 --2- 172.21.15.27:0/347085810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc109e00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.273+0000 7fc2016f2700 1 -- 172.21.15.27:0/347085810 >> 172.21.15.27:0/347085810 conn(0x7fc1fc105070 msgr2=0x7fc1fc1074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.273+0000 7fc2016f2700 1 -- 172.21.15.27:0/347085810 shutdown_connections 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.273+0000 7fc2016f2700 1 -- 172.21.15.27:0/347085810 wait complete. 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.273+0000 7fc2016f2700 1 Processor -- start 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.274+0000 7fc2016f2700 1 -- start start 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.274+0000 7fc2016f2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc19dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.274+0000 7fc2016f2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc10de80 con 0x7fc1fc1099f0 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.274+0000 7fc1faffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc19dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.274+0000 7fc1faffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc19dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54744/0 (socket says 172.21.15.27:54744) 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.274+0000 7fc1faffd700 1 -- 172.21.15.27:0/2304530384 learned_addr learned my addr 172.21.15.27:0/2304530384 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.274+0000 7fc1faffd700 1 -- 172.21.15.27:0/2304530384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1fc19e2c0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.274+0000 7fc1faffd700 1 --2- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc19dd80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec011890 tx=0x7fc1ec011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.275+0000 7fc1e3fff700 1 -- 172.21.15.27:0/2304530384 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1ec0039e0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.275+0000 7fc1e3fff700 1 -- 172.21.15.27:0/2304530384 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc1ec003b80 con 0x7fc1fc1099f0 2024-04-25T13:30:54.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.275+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1fc19e550 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.275+0000 7fc1e3fff700 1 -- 172.21.15.27:0/2304530384 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1ec0246e0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.275+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1fc1a1160 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.276+0000 7fc1e3fff700 1 -- 172.21.15.27:0/2304530384 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7fc1ec022020 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.276+0000 7fc1e3fff700 1 --2- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc1e403d570 0x7fc1e403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.276+0000 7fc1e3fff700 1 -- 172.21.15.27:0/2304530384 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc1ec0530d0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.276+0000 7fc1fa7fc700 1 --2- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc1e403d570 0x7fc1e403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.276+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1fc064cf0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.276+0000 7fc1fa7fc700 1 --2- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc1e403d570 0x7fc1e403fa80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0006fd0 tx=0x7fc1f0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.282+0000 7fc1e3fff700 1 -- 172.21.15.27:0/2304530384 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1ec0172b0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.577+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc1fc1a1680 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.578+0000 7fc1e3fff700 1 -- 172.21.15.27:0/2304530384 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc1ec00fdd0 con 0x7fc1fc1099f0 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc1e403d570 msgr2=0x7fc1e403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:54.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 --2- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc1e403d570 0x7fc1e403fa80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0006fd0 tx=0x7fc1f0006e40 comp rx=0 tx=0).stop 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 msgr2=0x7fc1fc19dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 --2- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc19dd80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec011890 tx=0x7fc1ec011970 comp rx=0 tx=0).stop 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 shutdown_connections 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 --2- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc1e403d570 0x7fc1e403fa80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 --2- 172.21.15.27:0/2304530384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1fc1099f0 0x7fc1fc19dd80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 >> 172.21.15.27:0/2304530384 conn(0x7fc1fc105070 msgr2=0x7fc1fc105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 shutdown_connections 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:53.580+0000 7fc2016f2700 1 -- 172.21.15.27:0/2304530384 wait complete. 2024-04-25T13:30:54.181 INFO:teuthology.orchestra.run.smithi027.stdout:mgr is available 2024-04-25T13:30:55.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1382170228' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T13:30:55.399 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T13:30:55.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.617+0000 7f31d909a700 1 Processor -- start 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.617+0000 7f31d909a700 1 -- start start 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.617+0000 7f31d909a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d410c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.617+0000 7f31d909a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31d410cb30 con 0x7f31d407cc00 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.617+0000 7f31d2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d410c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.617+0000 7f31d2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d410c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58614/0 (socket says 172.21.15.27:58614) 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.617+0000 7f31d2d9d700 1 -- 172.21.15.27:0/567086264 learned_addr learned my addr 172.21.15.27:0/567086264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:55.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.618+0000 7f31d2d9d700 1 -- 172.21.15.27:0/567086264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d410ccb0 con 0x7f31d407cc00 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.618+0000 7f31d2d9d700 1 --2- 172.21.15.27:0/567086264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d410c560 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0098a0 tx=0x7f31cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13fa5190de4eb5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.618+0000 7f31d1d9b700 1 -- 172.21.15.27:0/567086264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31cc004160 con 0x7f31d407cc00 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.618+0000 7f31d1d9b700 1 -- 172.21.15.27:0/567086264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f31cc004300 con 0x7f31d407cc00 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.618+0000 7f31d1d9b700 1 -- 172.21.15.27:0/567086264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31cc011af0 con 0x7f31d407cc00 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.619+0000 7f31d909a700 1 -- 172.21.15.27:0/567086264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 msgr2=0x7f31d410c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.619+0000 7f31d909a700 1 --2- 172.21.15.27:0/567086264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d410c560 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0098a0 tx=0x7f31cc009bb0 comp rx=0 tx=0).stop 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.619+0000 7f31d909a700 1 -- 172.21.15.27:0/567086264 shutdown_connections 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.619+0000 7f31d909a700 1 --2- 172.21.15.27:0/567086264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d410c560 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.619+0000 7f31d909a700 1 -- 172.21.15.27:0/567086264 >> 172.21.15.27:0/567086264 conn(0x7f31d4103fb0 msgr2=0x7f31d4106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.619+0000 7f31d909a700 1 -- 172.21.15.27:0/567086264 shutdown_connections 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.619+0000 7f31d909a700 1 -- 172.21.15.27:0/567086264 wait complete. 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.620+0000 7f31d909a700 1 Processor -- start 2024-04-25T13:30:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.620+0000 7f31d909a700 1 -- start start 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.620+0000 7f31d909a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.620+0000 7f31d909a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31d410dd30 con 0x7f31d407cc00 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.620+0000 7f31d2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.620+0000 7f31d2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d419dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58616/0 (socket says 172.21.15.27:58616) 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.620+0000 7f31d2d9d700 1 -- 172.21.15.27:0/1382170228 learned_addr learned my addr 172.21.15.27:0/1382170228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.621+0000 7f31d2d9d700 1 -- 172.21.15.27:0/1382170228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d419e250 con 0x7f31d407cc00 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.621+0000 7f31d2d9d700 1 --2- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d419dd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0117b0 tx=0x7f31cc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.621+0000 7f31cbfff700 1 -- 172.21.15.27:0/1382170228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31cc003af0 con 0x7f31d407cc00 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.621+0000 7f31cbfff700 1 -- 172.21.15.27:0/1382170228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f31cc020070 con 0x7f31d407cc00 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.621+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d419e4e0 con 0x7f31d407cc00 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.621+0000 7f31cbfff700 1 -- 172.21.15.27:0/1382170228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31cc019630 con 0x7f31d407cc00 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.621+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d41a10e0 con 0x7f31d407cc00 2024-04-25T13:30:55.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.622+0000 7f31cbfff700 1 -- 172.21.15.27:0/1382170228 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f31cc0198a0 con 0x7f31d407cc00 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.622+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d4064cf0 con 0x7f31d407cc00 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.623+0000 7f31cbfff700 1 --2- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f31c003d570 0x7f31c003fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.623+0000 7f31cbfff700 1 -- 172.21.15.27:0/1382170228 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f31cc0528e0 con 0x7f31d407cc00 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.623+0000 7f31d259c700 1 --2- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f31c003d570 0x7f31c003fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.623+0000 7f31d259c700 1 --2- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f31c003d570 0x7f31c003fa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f31bc006fd0 tx=0x7f31bc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.628+0000 7f31cbfff700 1 -- 172.21.15.27:0/1382170228 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31cc02bde0 con 0x7f31d407cc00 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.835+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f31d41a28e0 con 0x7f31d407cc00 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.837+0000 7f31cbfff700 1 -- 172.21.15.27:0/1382170228 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f31cc02ade0 con 0x7f31d407cc00 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f31c003d570 msgr2=0x7f31c003fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 --2- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f31c003d570 0x7f31c003fa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f31bc006fd0 tx=0x7f31bc006e40 comp rx=0 tx=0).stop 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 msgr2=0x7f31d419dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 --2- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d419dd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0117b0 tx=0x7f31cc011b70 comp rx=0 tx=0).stop 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 shutdown_connections 2024-04-25T13:30:55.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 --2- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f31c003d570 0x7f31c003fa80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:55.405 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 --2- 172.21.15.27:0/1382170228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d407cc00 0x7f31d419dd10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:55.405 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 >> 172.21.15.27:0/1382170228 conn(0x7f31d4103fb0 msgr2=0x7f31d4105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:55.405 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 shutdown_connections 2024-04-25T13:30:55.405 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:54.840+0000 7f31d909a700 1 -- 172.21.15.27:0/1382170228 wait complete. 2024-04-25T13:30:55.405 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling cephadm module... 2024-04-25T13:30:56.364 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:56 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3053737910' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T13:30:56.364 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: ignoring --setuser ceph since I am not root 2024-04-25T13:30:56.364 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: ignoring --setgroup ceph since I am not root 2024-04-25T13:30:56.364 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:56.363+0000 7fb077313200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T13:30:56.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:56.558+0000 7fb077313200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:30:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.831+0000 7fc4e3d5b700 1 Processor -- start 2024-04-25T13:30:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.831+0000 7fc4e3d5b700 1 -- start start 2024-04-25T13:30:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e3d5b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc10a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e3d5b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc10a7e0 con 0x7fc4dc109de0 2024-04-25T13:30:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e1af7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc10a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e1af7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc10a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58618/0 (socket says 172.21.15.27:58618) 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e1af7700 1 -- 172.21.15.27:0/3671142885 learned_addr learned my addr 172.21.15.27:0/3671142885 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e1af7700 1 -- 172.21.15.27:0/3671142885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc108b00 con 0x7fc4dc109de0 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e1af7700 1 --2- 172.21.15.27:0/3671142885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc10a1f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009e80 tx=0x7fc4cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92b0219088537f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e0af5700 1 -- 172.21.15.27:0/3671142885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4cc017070 con 0x7fc4dc109de0 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e0af5700 1 -- 172.21.15.27:0/3671142885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc4cc00b8c0 con 0x7fc4dc109de0 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.832+0000 7fc4e0af5700 1 -- 172.21.15.27:0/3671142885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4cc004490 con 0x7fc4dc109de0 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.833+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3671142885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 msgr2=0x7fc4dc10a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.833+0000 7fc4e3d5b700 1 --2- 172.21.15.27:0/3671142885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc10a1f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009e80 tx=0x7fc4cc00b330 comp rx=0 tx=0).stop 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.833+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3671142885 shutdown_connections 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.833+0000 7fc4e3d5b700 1 --2- 172.21.15.27:0/3671142885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc10a1f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.833+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3671142885 >> 172.21.15.27:0/3671142885 conn(0x7fc4dc1036e0 msgr2=0x7fc4dc105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.833+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3671142885 shutdown_connections 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.833+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3671142885 wait complete. 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.834+0000 7fc4e3d5b700 1 Processor -- start 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.834+0000 7fc4e3d5b700 1 -- start start 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.834+0000 7fc4e3d5b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.834+0000 7fc4e3d5b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc1079f0 con 0x7fc4dc109de0 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.834+0000 7fc4e1af7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.834+0000 7fc4e1af7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc19bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58626/0 (socket says 172.21.15.27:58626) 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.834+0000 7fc4e1af7700 1 -- 172.21.15.27:0/3053737910 learned_addr learned my addr 172.21.15.27:0/3053737910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.834+0000 7fc4e1af7700 1 -- 172.21.15.27:0/3053737910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc19bff0 con 0x7fc4dc109de0 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.835+0000 7fc4e1af7700 1 --2- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc19bab0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00bca0 tx=0x7fc4cc00bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.835+0000 7fc4daffd700 1 -- 172.21.15.27:0/3053737910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4cc020020 con 0x7fc4dc109de0 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.835+0000 7fc4daffd700 1 -- 172.21.15.27:0/3053737910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc4cc012430 con 0x7fc4dc109de0 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.835+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4dc19c280 con 0x7fc4dc109de0 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.835+0000 7fc4daffd700 1 -- 172.21.15.27:0/3053737910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4cc017040 con 0x7fc4dc109de0 2024-04-25T13:30:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.835+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4dc19ee90 con 0x7fc4dc109de0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.836+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4dc064cf0 con 0x7fc4dc109de0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.836+0000 7fc4daffd700 1 -- 172.21.15.27:0/3053737910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7fc4cc023020 con 0x7fc4dc109de0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.836+0000 7fc4daffd700 1 --2- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc4c403f7d0 0x7fc4c4041ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.836+0000 7fc4daffd700 1 -- 172.21.15.27:0/3053737910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc4cc00f050 con 0x7fc4dc109de0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.836+0000 7fc4e12f6700 1 --2- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc4c403f7d0 0x7fc4c4041ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.837+0000 7fc4e12f6700 1 --2- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc4c403f7d0 0x7fc4c4041ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0006fd0 tx=0x7fc4d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:55.842+0000 7fc4daffd700 1 -- 172.21.15.27:0/3053737910 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4cc02a020 con 0x7fc4dc109de0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.107+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fc4dc194d10 con 0x7fc4dc109de0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.159+0000 7fc4daffd700 1 -- 172.21.15.27:0/3053737910 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fc4dc194d10 con 0x7fc4dc109de0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.161+0000 7fc4daffd700 1 -- 172.21.15.27:0/3053737910 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7fc4cc0125e0 con 0x7fc4dc109de0 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.162+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc4c403f7d0 msgr2=0x7fc4c4041ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.162+0000 7fc4e3d5b700 1 --2- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc4c403f7d0 0x7fc4c4041ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0006fd0 tx=0x7fc4d0006e40 comp rx=0 tx=0).stop 2024-04-25T13:30:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.162+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 msgr2=0x7fc4dc19bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.162+0000 7fc4e3d5b700 1 --2- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc19bab0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00bca0 tx=0x7fc4cc00bd80 comp rx=0 tx=0).stop 2024-04-25T13:30:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.163+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 shutdown_connections 2024-04-25T13:30:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.163+0000 7fc4e3d5b700 1 --2- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fc4c403f7d0 0x7fc4c4041ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.163+0000 7fc4e3d5b700 1 --2- 172.21.15.27:0/3053737910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc109de0 0x7fc4dc19bab0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.163+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 >> 172.21.15.27:0/3053737910 conn(0x7fc4dc1036e0 msgr2=0x7fc4dc191dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.163+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 shutdown_connections 2024-04-25T13:30:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:56.163+0000 7fc4e3d5b700 1 -- 172.21.15.27:0/3053737910 wait complete. 2024-04-25T13:30:56.974 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:56.666+0000 7fb077313200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T13:30:57.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:57 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3053737910' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T13:30:57.225 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:57 smithi027 ceph-mon[29140]: mgrmap e5: x(active, since 6s) 2024-04-25T13:30:57.225 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:57 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:57.004+0000 7fb077313200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T13:30:57.225 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:57 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:57.094+0000 7fb077313200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T13:30:58.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T13:30:58.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T13:30:58.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T13:30:58.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T13:30:58.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T13:30:58.128 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.224+0000 7fb79f265700 1 Processor -- start 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.224+0000 7fb79f265700 1 -- start start 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.224+0000 7fb79f265700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb798108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.224+0000 7fb79f265700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb7981092a0 con 0x7fb7981088a0 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.225+0000 7fb79d001700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb798108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.225+0000 7fb79d001700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb798108cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58652/0 (socket says 172.21.15.27:58652) 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.225+0000 7fb79d001700 1 -- 172.21.15.27:0/2562472727 learned_addr learned my addr 172.21.15.27:0/2562472727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.225+0000 7fb79d001700 1 -- 172.21.15.27:0/2562472727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7981099d0 con 0x7fb7981088a0 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.225+0000 7fb79d001700 1 --2- 172.21.15.27:0/2562472727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb798108cb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb7940098a0 tx=0x7fb794009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea48096495e0657 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.226+0000 7fb78ffff700 1 -- 172.21.15.27:0/2562472727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb794004160 con 0x7fb7981088a0 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.226+0000 7fb78ffff700 1 -- 172.21.15.27:0/2562472727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb794004300 con 0x7fb7981088a0 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.226+0000 7fb78ffff700 1 -- 172.21.15.27:0/2562472727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb794011af0 con 0x7fb7981088a0 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.226+0000 7fb79f265700 1 -- 172.21.15.27:0/2562472727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 msgr2=0x7fb798108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:58.129 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.226+0000 7fb79f265700 1 --2- 172.21.15.27:0/2562472727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb798108cb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb7940098a0 tx=0x7fb794009bb0 comp rx=0 tx=0).stop 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.227+0000 7fb79f265700 1 -- 172.21.15.27:0/2562472727 shutdown_connections 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.227+0000 7fb79f265700 1 --2- 172.21.15.27:0/2562472727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb798108cb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.227+0000 7fb79f265700 1 -- 172.21.15.27:0/2562472727 >> 172.21.15.27:0/2562472727 conn(0x7fb798103f80 msgr2=0x7fb7981063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.227+0000 7fb79f265700 1 -- 172.21.15.27:0/2562472727 shutdown_connections 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.227+0000 7fb79f265700 1 -- 172.21.15.27:0/2562472727 wait complete. 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.228+0000 7fb79f265700 1 Processor -- start 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.228+0000 7fb79f265700 1 -- start start 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.228+0000 7fb79f265700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb79807c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.228+0000 7fb79f265700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb79810cd80 con 0x7fb7981088a0 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.228+0000 7fb79d001700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb79807c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.228+0000 7fb79d001700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb79807c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58656/0 (socket says 172.21.15.27:58656) 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.228+0000 7fb79d001700 1 -- 172.21.15.27:0/454371398 learned_addr learned my addr 172.21.15.27:0/454371398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:30:58.130 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.228+0000 7fb79d001700 1 -- 172.21.15.27:0/454371398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb79807ce40 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.229+0000 7fb79d001700 1 --2- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb79807c900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb794009fb0 tx=0x7fb794011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.229+0000 7fb78e7fc700 1 -- 172.21.15.27:0/454371398 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7940036a0 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.229+0000 7fb78e7fc700 1 -- 172.21.15.27:0/454371398 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb794003cd0 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.229+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7981a0af0 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.229+0000 7fb78e7fc700 1 -- 172.21.15.27:0/454371398 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb79401a430 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.229+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7981a0f70 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.230+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb798064cf0 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.231+0000 7fb78e7fc700 1 -- 172.21.15.27:0/454371398 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7fb794024440 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.231+0000 7fb78e7fc700 1 --2- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fb78403d570 0x7fb78403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.231+0000 7fb78e7fc700 1 -- 172.21.15.27:0/454371398 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb794052430 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.231+0000 7fb79c800700 1 -- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fb78403d570 msgr2=0x7fb78403fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4149968369 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.231+0000 7fb79c800700 1 --2- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fb78403d570 0x7fb78403fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.236+0000 7fb78e7fc700 1 -- 172.21.15.27:0/454371398 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb794051e50 con 0x7fb7981088a0 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.432+0000 7fb79c800700 1 -- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fb78403d570 msgr2=0x7fb78403fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4149968369 2024-04-25T13:30:58.131 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.432+0000 7fb79c800700 1 --2- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fb78403d570 0x7fb78403fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.537+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fb7981a1890 con 0x7fb7981088a0 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.538+0000 7fb78e7fc700 1 -- 172.21.15.27:0/454371398 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fb794017e00 con 0x7fb7981088a0 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.542+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fb78403d570 msgr2=0x7fb78403fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.542+0000 7fb79f265700 1 --2- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fb78403d570 0x7fb78403fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.542+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 msgr2=0x7fb79807c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.542+0000 7fb79f265700 1 --2- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb79807c900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb794009fb0 tx=0x7fb794011890 comp rx=0 tx=0).stop 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.542+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 shutdown_connections 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.542+0000 7fb79f265700 1 --2- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7fb78403d570 0x7fb78403fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.542+0000 7fb79f265700 1 --2- 172.21.15.27:0/454371398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7981088a0 0x7fb79807c900 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb794009fb0 tx=0x7fb794011890 comp rx=0 tx=0).stop 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.542+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 >> 172.21.15.27:0/454371398 conn(0x7fb798103f80 msgr2=0x7fb798105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.543+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 shutdown_connections 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:57.543+0000 7fb79f265700 1 -- 172.21.15.27:0/454371398 wait complete. 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-04-25T13:30:58.132 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 5... 2024-04-25T13:30:58.377 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:30:58 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/454371398' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T13:30:58.378 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:58.360+0000 7fb077313200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T13:30:58.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:58.545+0000 7fb077313200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T13:30:59.031 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:58.726+0000 7fb077313200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T13:30:59.339 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:59.030+0000 7fb077313200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T13:30:59.339 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:59.127+0000 7fb077313200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T13:30:59.340 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:59.339+0000 7fb077313200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T13:30:59.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:30:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:30:59.541+0000 7fb077313200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:31:00.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:00.171+0000 7fb077313200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T13:31:00.901 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:00.615+0000 7fb077313200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T13:31:00.901 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:00.716+0000 7fb077313200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T13:31:01.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:00.927+0000 7fb077313200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T13:31:01.828 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:01.474+0000 7fb077313200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T13:31:01.828 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:01.571+0000 7fb077313200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T13:31:02.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:01.827+0000 7fb077313200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T13:31:02.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:01.967+0000 7fb077313200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T13:31:02.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:02.066+0000 7fb077313200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T13:31:02.501 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:02.187+0000 7fb077313200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T13:31:02.901 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:02.501+0000 7fb077313200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T13:31:03.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:03.130+0000 7fb077313200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T13:31:03.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:03.357+0000 7fb077313200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T13:31:03.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: Active manager daemon x restarted 2024-04-25T13:31:03.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: Activating manager daemon x 2024-04-25T13:31:03.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T13:31:03.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: mgrmap e6: x(active, starting, since 0.00387828s) 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: Manager daemon x is now available 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:03.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:03 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:04.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:04 smithi027 ceph-mon[29140]: Found migration_current of "None". Setting to last migration. 2024-04-25T13:31:04.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:04 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T13:31:04.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:04 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T13:31:04.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:04 smithi027 ceph-mon[29140]: mgrmap e7: x(active, since 1.00629s) 2024-04-25T13:31:04.945 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T13:31:04.945 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T13:31:04.945 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T13:31:04.945 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T13:31:04.945 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.595+0000 7f97ea656700 1 Processor -- start 2024-04-25T13:31:04.945 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.595+0000 7f97ea656700 1 -- start start 2024-04-25T13:31:04.945 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.595+0000 7f97ea656700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e410c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.595+0000 7f97ea656700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97e410cb10 con 0x7f97e407cc60 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.595+0000 7f97e3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e410c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.595+0000 7f97e3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e410c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58668/0 (socket says 172.21.15.27:58668) 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.595+0000 7f97e3fff700 1 -- 172.21.15.27:0/1421009073 learned_addr learned my addr 172.21.15.27:0/1421009073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.595+0000 7f97e3fff700 1 -- 172.21.15.27:0/1421009073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e410cc90 con 0x7f97e407cc60 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.596+0000 7f97e3fff700 1 --2- 172.21.15.27:0/1421009073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e410c500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97d40098a0 tx=0x7f97d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffec59816c22fc95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.596+0000 7f97e2ffd700 1 -- 172.21.15.27:0/1421009073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97d4004160 con 0x7f97e407cc60 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.596+0000 7f97e2ffd700 1 -- 172.21.15.27:0/1421009073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f97d4004300 con 0x7f97e407cc60 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.596+0000 7f97e2ffd700 1 -- 172.21.15.27:0/1421009073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97d4011af0 con 0x7f97e407cc60 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.597+0000 7f97ea656700 1 -- 172.21.15.27:0/1421009073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 msgr2=0x7f97e410c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.597+0000 7f97ea656700 1 --2- 172.21.15.27:0/1421009073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e410c500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97d40098a0 tx=0x7f97d4009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.597+0000 7f97ea656700 1 -- 172.21.15.27:0/1421009073 shutdown_connections 2024-04-25T13:31:04.946 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.597+0000 7f97ea656700 1 --2- 172.21.15.27:0/1421009073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e410c500 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.597+0000 7f97ea656700 1 -- 172.21.15.27:0/1421009073 >> 172.21.15.27:0/1421009073 conn(0x7f97e4103fb0 msgr2=0x7f97e4106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.597+0000 7f97ea656700 1 -- 172.21.15.27:0/1421009073 shutdown_connections 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.597+0000 7f97ea656700 1 -- 172.21.15.27:0/1421009073 wait complete. 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.598+0000 7f97ea656700 1 Processor -- start 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.598+0000 7f97ea656700 1 -- start start 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.598+0000 7f97ea656700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.598+0000 7f97ea656700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97e410dd10 con 0x7f97e407cc60 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.598+0000 7f97e3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.598+0000 7f97e3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e419dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58676/0 (socket says 172.21.15.27:58676) 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.598+0000 7f97e3fff700 1 -- 172.21.15.27:0/3582440555 learned_addr learned my addr 172.21.15.27:0/3582440555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.598+0000 7f97e3fff700 1 -- 172.21.15.27:0/3582440555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e419e270 con 0x7f97e407cc60 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.599+0000 7f97e3fff700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e419dd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f97d40117b0 tx=0x7f97d4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.599+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97d4003af0 con 0x7f97e407cc60 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.599+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e419e500 con 0x7f97e407cc60 2024-04-25T13:31:04.947 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.599+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f97d4020070 con 0x7f97e407cc60 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.599+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97d4019630 con 0x7f97e407cc60 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.599+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e41a1100 con 0x7f97e407cc60 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.600+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f97d40198f0 con 0x7f97e407cc60 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.600+0000 7f97e17fa700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 0x7f97cc03fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.600+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 --> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f97cc040190 con 0x7f97cc03d570 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.600+0000 7f97e37fe700 1 -- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 msgr2=0x7f97cc03fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4149968369 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.600+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f97d4052a60 con 0x7f97e407cc60 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.600+0000 7f97e37fe700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 0x7f97cc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.801+0000 7f97e37fe700 1 -- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 msgr2=0x7f97cc03fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4149968369 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:58.801+0000 7f97e37fe700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 0x7f97cc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:59.202+0000 7f97e37fe700 1 -- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 msgr2=0x7f97cc03fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4149968369 2024-04-25T13:31:04.948 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:30:59.202+0000 7f97e37fe700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 0x7f97cc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:00.003+0000 7f97e37fe700 1 -- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 msgr2=0x7f97cc03fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4149968369 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:00.003+0000 7f97e37fe700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 0x7f97cc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:01.604+0000 7f97e37fe700 1 -- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 msgr2=0x7f97cc03fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/4149968369 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:01.604+0000 7f97e37fe700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 0x7f97cc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:03.361+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 6) ==== 49283+0+0 (secure 0 0 0) 0x7f97d402dcf0 con 0x7f97e407cc60 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:03.361+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 msgr2=0x7f97cc03fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:03.361+0000 7f97e17fa700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 0x7f97cc03fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.362+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7f97d40541b0 con 0x7f97e407cc60 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.362+0000 7f97e17fa700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f97cc040eb0 0x7f97cc0432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.362+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f97d401ab90 con 0x7f97cc040eb0 2024-04-25T13:31:04.949 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.363+0000 7f97e37fe700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f97cc040eb0 0x7f97cc0432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.364+0000 7f97e37fe700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f97cc040eb0 0x7f97cc0432d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f97d8003a10 tx=0x7f97d80092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.365+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mgr.14120 v2:172.21.15.27:6800/1748672481 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f97d401ab90 con 0x7f97cc040eb0 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.373+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f97e4064cf0 con 0x7f97cc040eb0 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.373+0000 7f97e17fa700 1 -- 172.21.15.27:0/3582440555 <== mgr.14120 v2:172.21.15.27:6800/1748672481 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f97e4064cf0 con 0x7f97cc040eb0 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f97cc040eb0 msgr2=0x7f97cc0432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f97cc040eb0 0x7f97cc0432d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f97d8003a10 tx=0x7f97d80092b0 comp rx=0 tx=0).stop 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 msgr2=0x7f97e419dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e419dd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f97d40117b0 tx=0x7f97d4011b70 comp rx=0 tx=0).stop 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 shutdown_connections 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f97cc040eb0 0x7f97cc0432d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97e407cc60 0x7f97e419dd30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 --2- 172.21.15.27:0/3582440555 >> [v2:172.21.15.27:6800/4149968369,v1:172.21.15.27:6801/4149968369] conn(0x7f97cc03d570 0x7f97cc03fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:04.950 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 >> 172.21.15.27:0/3582440555 conn(0x7f97e4103fb0 msgr2=0x7f97e41058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:04.951 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 shutdown_connections 2024-04-25T13:31:04.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:04.374+0000 7f97ea656700 1 -- 172.21.15.27:0/3582440555 wait complete. 2024-04-25T13:31:04.954 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 5 is available 2024-04-25T13:31:04.954 INFO:teuthology.orchestra.run.smithi027.stdout:Setting orchestrator backend to cephadm... 2024-04-25T13:31:06.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.365+0000 7fea1f12d700 1 Processor -- start 2024-04-25T13:31:06.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.365+0000 7fea1f12d700 1 -- start start 2024-04-25T13:31:06.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.365+0000 7fea1f12d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1810c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:06.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.365+0000 7fea1f12d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea1810cae0 con 0x7fea1807cc40 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.365+0000 7fea1cec9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1810c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.365+0000 7fea1cec9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1810c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43052/0 (socket says 172.21.15.27:43052) 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.365+0000 7fea1cec9700 1 -- 172.21.15.27:0/749231325 learned_addr learned my addr 172.21.15.27:0/749231325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.365+0000 7fea1cec9700 1 -- 172.21.15.27:0/749231325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1810cc60 con 0x7fea1807cc40 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.366+0000 7fea1cec9700 1 --2- 172.21.15.27:0/749231325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1810c4f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0098a0 tx=0x7fea0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3a8703bc0388c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.366+0000 7fea177fe700 1 -- 172.21.15.27:0/749231325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea0c004160 con 0x7fea1807cc40 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.366+0000 7fea177fe700 1 -- 172.21.15.27:0/749231325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fea0c004300 con 0x7fea1807cc40 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.366+0000 7fea177fe700 1 -- 172.21.15.27:0/749231325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea0c011af0 con 0x7fea1807cc40 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.366+0000 7fea1f12d700 1 -- 172.21.15.27:0/749231325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 msgr2=0x7fea1810c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.366+0000 7fea1f12d700 1 --2- 172.21.15.27:0/749231325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1810c4f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0098a0 tx=0x7fea0c009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.367+0000 7fea1f12d700 1 -- 172.21.15.27:0/749231325 shutdown_connections 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.367+0000 7fea1f12d700 1 --2- 172.21.15.27:0/749231325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1810c4f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:06.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.367+0000 7fea1f12d700 1 -- 172.21.15.27:0/749231325 >> 172.21.15.27:0/749231325 conn(0x7fea18103f80 msgr2=0x7fea181063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.367+0000 7fea1f12d700 1 -- 172.21.15.27:0/749231325 shutdown_connections 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.367+0000 7fea1f12d700 1 -- 172.21.15.27:0/749231325 wait complete. 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.367+0000 7fea1f12d700 1 Processor -- start 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.367+0000 7fea1f12d700 1 -- start start 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea1f12d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea1f12d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fea1810dce0 con 0x7fea1807cc40 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea1cec9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea1cec9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1819dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43062/0 (socket says 172.21.15.27:43062) 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea1cec9700 1 -- 172.21.15.27:0/2558951660 learned_addr learned my addr 172.21.15.27:0/2558951660 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea1cec9700 1 -- 172.21.15.27:0/2558951660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1819e220 con 0x7fea1807cc40 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea1cec9700 1 --2- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1819dce0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009fb0 tx=0x7fea0c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea15ffb700 1 -- 172.21.15.27:0/2558951660 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea0c0036a0 con 0x7fea1807cc40 2024-04-25T13:31:06.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea15ffb700 1 -- 172.21.15.27:0/2558951660 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fea0c01a430 con 0x7fea1807cc40 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1819e4b0 con 0x7fea1807cc40 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.368+0000 7fea15ffb700 1 -- 172.21.15.27:0/2558951660 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea0c019550 con 0x7fea1807cc40 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.369+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea181a10c0 con 0x7fea1807cc40 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.370+0000 7fea15ffb700 1 -- 172.21.15.27:0/2558951660 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7fea0c019850 con 0x7fea1807cc40 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.370+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea18064cf0 con 0x7fea1807cc40 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.370+0000 7fea15ffb700 1 --2- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fea0803d3f0 0x7fea0803f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.370+0000 7fea15ffb700 1 -- 172.21.15.27:0/2558951660 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fea0c0524f0 con 0x7fea1807cc40 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.370+0000 7fea17fff700 1 --2- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fea0803d3f0 0x7fea0803f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.370+0000 7fea17fff700 1 --2- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fea0803d3f0 0x7fea0803f900 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea04006fd0 tx=0x7fea04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.376+0000 7fea15ffb700 1 -- 172.21.15.27:0/2558951660 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea0c017e90 con 0x7fea1807cc40 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.599+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fea181060a0 con 0x7fea0803d3f0 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.607+0000 7fea15ffb700 1 -- 172.21.15.27:0/2558951660 <== mgr.14120 v2:172.21.15.27:6800/1748672481 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fea181060a0 con 0x7fea0803d3f0 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fea0803d3f0 msgr2=0x7fea0803f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:06.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 --2- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fea0803d3f0 0x7fea0803f900 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea04006fd0 tx=0x7fea04006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:06.143 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 msgr2=0x7fea1819dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:06.143 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 --2- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1819dce0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009fb0 tx=0x7fea0c011940 comp rx=0 tx=0).stop 2024-04-25T13:31:06.143 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 shutdown_connections 2024-04-25T13:31:06.143 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 --2- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fea0803d3f0 0x7fea0803f900 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:06.143 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 --2- 172.21.15.27:0/2558951660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1807cc40 0x7fea1819dce0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:06.143 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 >> 172.21.15.27:0/2558951660 conn(0x7fea18103f80 msgr2=0x7fea18105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:06.143 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 shutdown_connections 2024-04-25T13:31:06.143 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:05.609+0000 7fea1f12d700 1 -- 172.21.15.27:0/2558951660 wait complete. 2024-04-25T13:31:06.257 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:06 smithi027 ceph-mon[29140]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T13:31:06.257 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:06 smithi027 ceph-mon[29140]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T13:31:06.257 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:06 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:06.257 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:06 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:06.258 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:06 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:06.258 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:06 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:06.258 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:06 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:07.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:07 smithi027 ceph-mon[29140]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:07.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:07 smithi027 ceph-mon[29140]: [25/Apr/2024:13:31:05] ENGINE Bus STARTING 2024-04-25T13:31:07.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:07 smithi027 ceph-mon[29140]: [25/Apr/2024:13:31:05] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T13:31:07.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:07 smithi027 ceph-mon[29140]: [25/Apr/2024:13:31:05] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T13:31:07.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:07 smithi027 ceph-mon[29140]: [25/Apr/2024:13:31:05] ENGINE Bus STARTED 2024-04-25T13:31:07.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:07 smithi027 ceph-mon[29140]: mgrmap e8: x(active, since 2s) 2024-04-25T13:31:07.454 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T13:31:07.454 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.638+0000 7fe8d5d94700 1 Processor -- start 2024-04-25T13:31:07.454 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.638+0000 7fe8d5d94700 1 -- start start 2024-04-25T13:31:07.454 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.638+0000 7fe8d5d94700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d010c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:07.454 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.638+0000 7fe8d5d94700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d007d600 con 0x7fe8d010a490 2024-04-25T13:31:07.454 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.638+0000 7fe8cf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d010c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:07.454 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.638+0000 7fe8cf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d010c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43076/0 (socket says 172.21.15.27:43076) 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.638+0000 7fe8cf7fe700 1 -- 172.21.15.27:0/40457838 learned_addr learned my addr 172.21.15.27:0/40457838 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.639+0000 7fe8cf7fe700 1 -- 172.21.15.27:0/40457838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d010cdf0 con 0x7fe8d010a490 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.639+0000 7fe8cf7fe700 1 --2- 172.21.15.27:0/40457838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d010c8b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009e80 tx=0x7fe8c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d329bb46e4e8e09d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.639+0000 7fe8ce7fc700 1 -- 172.21.15.27:0/40457838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8c8017070 con 0x7fe8d010a490 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.639+0000 7fe8ce7fc700 1 -- 172.21.15.27:0/40457838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe8c800b8c0 con 0x7fe8d010a490 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.639+0000 7fe8ce7fc700 1 -- 172.21.15.27:0/40457838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8c8004490 con 0x7fe8d010a490 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.640+0000 7fe8d5d94700 1 -- 172.21.15.27:0/40457838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 msgr2=0x7fe8d010c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.640+0000 7fe8d5d94700 1 --2- 172.21.15.27:0/40457838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d010c8b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009e80 tx=0x7fe8c800b330 comp rx=0 tx=0).stop 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.640+0000 7fe8d5d94700 1 -- 172.21.15.27:0/40457838 shutdown_connections 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.640+0000 7fe8d5d94700 1 --2- 172.21.15.27:0/40457838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d010c8b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.640+0000 7fe8d5d94700 1 -- 172.21.15.27:0/40457838 >> 172.21.15.27:0/40457838 conn(0x7fe8d0103f30 msgr2=0x7fe8d01063a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.640+0000 7fe8d5d94700 1 -- 172.21.15.27:0/40457838 shutdown_connections 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.640+0000 7fe8d5d94700 1 -- 172.21.15.27:0/40457838 wait complete. 2024-04-25T13:31:07.455 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.641+0000 7fe8d5d94700 1 Processor -- start 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.641+0000 7fe8d5d94700 1 -- start start 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.641+0000 7fe8d5d94700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d019dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.641+0000 7fe8d5d94700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d010dcf0 con 0x7fe8d010a490 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.641+0000 7fe8cf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d019dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.641+0000 7fe8cf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d019dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43084/0 (socket says 172.21.15.27:43084) 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.641+0000 7fe8cf7fe700 1 -- 172.21.15.27:0/2493768051 learned_addr learned my addr 172.21.15.27:0/2493768051 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.641+0000 7fe8cf7fe700 1 -- 172.21.15.27:0/2493768051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d019e1d0 con 0x7fe8d010a490 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.642+0000 7fe8cf7fe700 1 --2- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d019dc90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800b7b0 tx=0x7fe8c800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.642+0000 7fe8ccff9700 1 -- 172.21.15.27:0/2493768051 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8c8020020 con 0x7fe8d010a490 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.642+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d019e460 con 0x7fe8d010a490 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.642+0000 7fe8ccff9700 1 -- 172.21.15.27:0/2493768051 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe8c8012430 con 0x7fe8d010a490 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.642+0000 7fe8ccff9700 1 -- 172.21.15.27:0/2493768051 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8c8017040 con 0x7fe8d010a490 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.642+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d01a1070 con 0x7fe8d010a490 2024-04-25T13:31:07.456 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.643+0000 7fe8ccff9700 1 -- 172.21.15.27:0/2493768051 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fe8c8023020 con 0x7fe8d010a490 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.643+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d0064cf0 con 0x7fe8d010a490 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.643+0000 7fe8ccff9700 1 --2- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fe8bc03d570 0x7fe8bc03fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.643+0000 7fe8ccff9700 1 -- 172.21.15.27:0/2493768051 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe8c800f050 con 0x7fe8d010a490 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.643+0000 7fe8ceffd700 1 --2- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fe8bc03d570 0x7fe8bc03fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.644+0000 7fe8ceffd700 1 --2- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fe8bc03d570 0x7fe8bc03fa80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8006fd0 tx=0x7fe8b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.649+0000 7fe8ccff9700 1 -- 172.21.15.27:0/2493768051 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8c802a020 con 0x7fe8d010a490 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.855+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fe8d0104f00 con 0x7fe8bc03d570 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.856+0000 7fe8ccff9700 1 -- 172.21.15.27:0/2493768051 <== mgr.14120 v2:172.21.15.27:6800/1748672481 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fe8d0104f00 con 0x7fe8bc03d570 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fe8bc03d570 msgr2=0x7fe8bc03fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 --2- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fe8bc03d570 0x7fe8bc03fa80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8006fd0 tx=0x7fe8b8006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 msgr2=0x7fe8d019dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 --2- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d019dc90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800b7b0 tx=0x7fe8c800bd50 comp rx=0 tx=0).stop 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 shutdown_connections 2024-04-25T13:31:07.457 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 --2- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fe8bc03d570 0x7fe8bc03fa80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:07.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 --2- 172.21.15.27:0/2493768051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8d010a490 0x7fe8d019dc90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:07.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 >> 172.21.15.27:0/2493768051 conn(0x7fe8d0103f30 msgr2=0x7fe8d01047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:07.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.858+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 shutdown_connections 2024-04-25T13:31:07.458 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:06.859+0000 7fe8d5d94700 1 -- 172.21.15.27:0/2493768051 wait complete. 2024-04-25T13:31:07.458 INFO:teuthology.orchestra.run.smithi027.stdout:Generating ssh key... 2024-04-25T13:31:08.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-mon[29140]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:08.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: Generating public/private rsa key pair. 2024-04-25T13:31:08.401 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: Your identification has been saved in /tmp/tmpnc5pkdcm/key. 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: Your public key has been saved in /tmp/tmpnc5pkdcm/key.pub. 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: The key fingerprint is: 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: SHA256:WEt9o8o9ttoJ15bTvk97J+7E1HzLsfo6HAqN1TbL8GA ceph-db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: The key's randomart image is: 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: +---[RSA 3072]----+ 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | | 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | . | 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | o . + | 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | + . E = o | 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | . S * * +.+| 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | . = o X. =| 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | + * B =+.| 2024-04-25T13:31:08.402 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | = * *o.+| 2024-04-25T13:31:08.403 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: | ..+ =O*+| 2024-04-25T13:31:08.403 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: +----[SHA256]-----+ 2024-04-25T13:31:08.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.872+0000 7fd8989a8700 1 Processor -- start 2024-04-25T13:31:08.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.872+0000 7fd8989a8700 1 -- start start 2024-04-25T13:31:08.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.873+0000 7fd8989a8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89410c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:08.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.873+0000 7fd8989a8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd89410cb50 con 0x7fd89407d4d0 2024-04-25T13:31:08.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.873+0000 7fd89259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89410c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.873+0000 7fd89259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89410c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43092/0 (socket says 172.21.15.27:43092) 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.873+0000 7fd89259c700 1 -- 172.21.15.27:0/3181412492 learned_addr learned my addr 172.21.15.27:0/3181412492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.873+0000 7fd89259c700 1 -- 172.21.15.27:0/3181412492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89410ccd0 con 0x7fd89407d4d0 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.874+0000 7fd89259c700 1 --2- 172.21.15.27:0/3181412492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89410c560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd8880098a0 tx=0x7fd888009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a6ca4d05be9964 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.874+0000 7fd89159a700 1 -- 172.21.15.27:0/3181412492 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd888004160 con 0x7fd89407d4d0 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.874+0000 7fd89159a700 1 -- 172.21.15.27:0/3181412492 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd888004300 con 0x7fd89407d4d0 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.874+0000 7fd89159a700 1 -- 172.21.15.27:0/3181412492 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd888011af0 con 0x7fd89407d4d0 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.874+0000 7fd8989a8700 1 -- 172.21.15.27:0/3181412492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 msgr2=0x7fd89410c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.874+0000 7fd8989a8700 1 --2- 172.21.15.27:0/3181412492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89410c560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd8880098a0 tx=0x7fd888009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.875+0000 7fd8989a8700 1 -- 172.21.15.27:0/3181412492 shutdown_connections 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.875+0000 7fd8989a8700 1 --2- 172.21.15.27:0/3181412492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89410c560 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.875+0000 7fd8989a8700 1 -- 172.21.15.27:0/3181412492 >> 172.21.15.27:0/3181412492 conn(0x7fd894103f90 msgr2=0x7fd894106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:08.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.875+0000 7fd8989a8700 1 -- 172.21.15.27:0/3181412492 shutdown_connections 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.875+0000 7fd8989a8700 1 -- 172.21.15.27:0/3181412492 wait complete. 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.875+0000 7fd8989a8700 1 Processor -- start 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.876+0000 7fd8989a8700 1 -- start start 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.876+0000 7fd8989a8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.876+0000 7fd8989a8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd89410dd50 con 0x7fd89407d4d0 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.876+0000 7fd89259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.876+0000 7fd89259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89419dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43108/0 (socket says 172.21.15.27:43108) 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.876+0000 7fd89259c700 1 -- 172.21.15.27:0/504232358 learned_addr learned my addr 172.21.15.27:0/504232358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.876+0000 7fd89259c700 1 -- 172.21.15.27:0/504232358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89419e270 con 0x7fd89407d4d0 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.876+0000 7fd89259c700 1 --2- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89419dd30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8880117b0 tx=0x7fd888011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.877+0000 7fd8837fe700 1 -- 172.21.15.27:0/504232358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd888003af0 con 0x7fd89407d4d0 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.877+0000 7fd8837fe700 1 -- 172.21.15.27:0/504232358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd888020070 con 0x7fd89407d4d0 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.877+0000 7fd8837fe700 1 -- 172.21.15.27:0/504232358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd888019630 con 0x7fd89407d4d0 2024-04-25T13:31:08.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.877+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89419e500 con 0x7fd89407d4d0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.877+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8941a1100 con 0x7fd89407d4d0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.878+0000 7fd8837fe700 1 -- 172.21.15.27:0/504232358 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fd8880198a0 con 0x7fd89407d4d0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.878+0000 7fd8837fe700 1 --2- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd87c03d5c0 0x7fd87c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.878+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd894064cf0 con 0x7fd89407d4d0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.878+0000 7fd8837fe700 1 -- 172.21.15.27:0/504232358 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd88801a9c0 con 0x7fd89407d4d0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.878+0000 7fd891d9b700 1 --2- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd87c03d5c0 0x7fd87c03fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.878+0000 7fd891d9b700 1 --2- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd87c03d5c0 0x7fd87c03fad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd884006fd0 tx=0x7fd884006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:07.883+0000 7fd8837fe700 1 -- 172.21.15.27:0/504232358 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd888016260 con 0x7fd89407d4d0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.092+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fd894104f90 con 0x7fd87c03d5c0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.270+0000 7fd8837fe700 1 -- 172.21.15.27:0/504232358 <== mgr.14120 v2:172.21.15.27:6800/1748672481 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd894104f90 con 0x7fd87c03d5c0 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd87c03d5c0 msgr2=0x7fd87c03fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 --2- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd87c03d5c0 0x7fd87c03fad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd884006fd0 tx=0x7fd884006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:08.797 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 msgr2=0x7fd89419dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:08.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 --2- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89419dd30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8880117b0 tx=0x7fd888011b70 comp rx=0 tx=0).stop 2024-04-25T13:31:08.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 shutdown_connections 2024-04-25T13:31:08.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 --2- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd87c03d5c0 0x7fd87c03fad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:08.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 --2- 172.21.15.27:0/504232358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd89407d4d0 0x7fd89419dd30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:08.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 >> 172.21.15.27:0/504232358 conn(0x7fd894103f90 msgr2=0x7fd894104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:08.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 shutdown_connections 2024-04-25T13:31:08.798 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:08.272+0000 7fd8989a8700 1 -- 172.21.15.27:0/504232358 wait complete. 2024-04-25T13:31:09.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:09 smithi027 ceph-mon[29140]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:09.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:09 smithi027 ceph-mon[29140]: Generating ssh key... 2024-04-25T13:31:09.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:09 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:09.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:09 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:10.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoDQ9BjclXvbBtDzMsXR/cQEqOazPcwXazPCUnOM2Z7aFIhCIBdXrIo8fDNqmeGRtrihaUBw+Kc4GbM5Qs6eD1FmDzncWoRgWi9ZpTU1Dzc64GVJzwGnMyJx3XP+BKr+N386F05nqO1Kk/XdVbnohYXZXY1DRZ2E3P5hmvDKbBZmbXUWPE1bn7fCfiBX5GnQnRuh0hmaoAuwGoYyiC7bcjVwuq27ROL/6Gdzaeg9HAJgF4FhHqLaDeRK70T+UtyKLs/rXwEhQJqeoPkIkcj71YQOe4c9ILtDKOgrGNhgqg4Zev1rjS1L7+Mz85yw6JRaacuXKOZKiVWG9YsDjqZiPoxlr9/3OYERX6XE0GY/vvb7rkubCNEUoJI9+qubqx72psGBuJLHi0+PLa34dwcazM1U67Y7uBO98SKJyjksLzs6o5fZVqgYGWEAY4g1okcEEcBhRwP4+33//m+f/F3JAtN9pmIPHUwdBs6/rJrfxMVsuNPqvhLVN8WO+tXTe20tk= ceph-db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:31:10.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.214+0000 7fd6edbe4700 1 Processor -- start 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.214+0000 7fd6edbe4700 1 -- start start 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.214+0000 7fd6edbe4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e810c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.214+0000 7fd6edbe4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e810cb10 con 0x7fd6e807cc40 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.215+0000 7fd6e77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e810c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.215+0000 7fd6e77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e810c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43120/0 (socket says 172.21.15.27:43120) 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.215+0000 7fd6e77fe700 1 -- 172.21.15.27:0/300714243 learned_addr learned my addr 172.21.15.27:0/300714243 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.215+0000 7fd6e77fe700 1 -- 172.21.15.27:0/300714243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e810cc90 con 0x7fd6e807cc40 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.215+0000 7fd6e77fe700 1 --2- 172.21.15.27:0/300714243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e810c520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00098a0 tx=0x7fd6d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcbf3bffab3ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.215+0000 7fd6e67fc700 1 -- 172.21.15.27:0/300714243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6d0004160 con 0x7fd6e807cc40 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.215+0000 7fd6e67fc700 1 -- 172.21.15.27:0/300714243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd6d0004300 con 0x7fd6e807cc40 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.215+0000 7fd6e67fc700 1 -- 172.21.15.27:0/300714243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6d0011af0 con 0x7fd6e807cc40 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.216+0000 7fd6edbe4700 1 -- 172.21.15.27:0/300714243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 msgr2=0x7fd6e810c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:10.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.216+0000 7fd6edbe4700 1 --2- 172.21.15.27:0/300714243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e810c520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00098a0 tx=0x7fd6d0009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.216+0000 7fd6edbe4700 1 -- 172.21.15.27:0/300714243 shutdown_connections 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.216+0000 7fd6edbe4700 1 --2- 172.21.15.27:0/300714243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e810c520 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.216+0000 7fd6edbe4700 1 -- 172.21.15.27:0/300714243 >> 172.21.15.27:0/300714243 conn(0x7fd6e8103fb0 msgr2=0x7fd6e8106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.216+0000 7fd6edbe4700 1 -- 172.21.15.27:0/300714243 shutdown_connections 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.216+0000 7fd6edbe4700 1 -- 172.21.15.27:0/300714243 wait complete. 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.217+0000 7fd6edbe4700 1 Processor -- start 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.217+0000 7fd6edbe4700 1 -- start start 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.217+0000 7fd6edbe4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e8109c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.217+0000 7fd6edbe4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e810dd10 con 0x7fd6e807cc40 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.217+0000 7fd6e77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e8109c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.217+0000 7fd6e77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e8109c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43126/0 (socket says 172.21.15.27:43126) 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.217+0000 7fd6e77fe700 1 -- 172.21.15.27:0/1439249167 learned_addr learned my addr 172.21.15.27:0/1439249167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.218+0000 7fd6e77fe700 1 -- 172.21.15.27:0/1439249167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e810a1d0 con 0x7fd6e807cc40 2024-04-25T13:31:10.062 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.218+0000 7fd6e77fe700 1 --2- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e8109c90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009fb0 tx=0x7fd6d0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.218+0000 7fd6e4ff9700 1 -- 172.21.15.27:0/1439249167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6d00036a0 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.218+0000 7fd6e4ff9700 1 -- 172.21.15.27:0/1439249167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd6d001a430 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.218+0000 7fd6e4ff9700 1 -- 172.21.15.27:0/1439249167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6d0019550 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.218+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e8108420 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.218+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e8108900 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.219+0000 7fd6e4ff9700 1 -- 172.21.15.27:0/1439249167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fd6d0019850 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.219+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e8064cf0 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.219+0000 7fd6e4ff9700 1 --2- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd6d403d520 0x7fd6d403fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.219+0000 7fd6e4ff9700 1 -- 172.21.15.27:0/1439249167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd6d00526b0 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.219+0000 7fd6e6ffd700 1 --2- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd6d403d520 0x7fd6d403fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.220+0000 7fd6e6ffd700 1 --2- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd6d403d520 0x7fd6d403fa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006fd0 tx=0x7fd6d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.225+0000 7fd6e4ff9700 1 -- 172.21.15.27:0/1439249167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd6d0017e90 con 0x7fd6e807cc40 2024-04-25T13:31:10.063 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.435+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fd6e81090b0 con 0x7fd6d403d520 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.436+0000 7fd6e4ff9700 1 -- 172.21.15.27:0/1439249167 <== mgr.14120 v2:172.21.15.27:6800/1748672481 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fd6e81090b0 con 0x7fd6d403d520 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd6d403d520 msgr2=0x7fd6d403fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 --2- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd6d403d520 0x7fd6d403fa30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006fd0 tx=0x7fd6d8006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 msgr2=0x7fd6e8109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 --2- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e8109c90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009fb0 tx=0x7fd6d0011940 comp rx=0 tx=0).stop 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 shutdown_connections 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 --2- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fd6d403d520 0x7fd6d403fa30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 --2- 172.21.15.27:0/1439249167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6e807cc40 0x7fd6e8109c90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 >> 172.21.15.27:0/1439249167 conn(0x7fd6e8103fb0 msgr2=0x7fd6e8105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 shutdown_connections 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:09.439+0000 7fd6edbe4700 1 -- 172.21.15.27:0/1439249167 wait complete. 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T13:31:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:Adding host smithi027... 2024-04-25T13:31:10.370 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:10 smithi027 ceph-mon[29140]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:11.540 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:11 smithi027 ceph-mon[29140]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:12.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:12 smithi027 ceph-mon[29140]: Deploying cephadm binary to smithi027 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Added host 'smithi027' with addr '172.21.15.27' 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.489+0000 7f678bd63700 1 Processor -- start 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.489+0000 7f678bd63700 1 -- start start 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.489+0000 7f678bd63700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678410c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.489+0000 7f678bd63700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f678410cb10 con 0x7f678407cc60 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.490+0000 7f6789aff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678410c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.490+0000 7f6789aff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678410c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43138/0 (socket says 172.21.15.27:43138) 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.490+0000 7f6789aff700 1 -- 172.21.15.27:0/1609236322 learned_addr learned my addr 172.21.15.27:0/1609236322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.490+0000 7f6789aff700 1 -- 172.21.15.27:0/1609236322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678410cc90 con 0x7f678407cc60 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.490+0000 7f6789aff700 1 --2- 172.21.15.27:0/1609236322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678410c500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f67800098a0 tx=0x7f6780009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3e7e60d0bdf0ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.490+0000 7f6788afd700 1 -- 172.21.15.27:0/1609236322 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6780004160 con 0x7f678407cc60 2024-04-25T13:31:15.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.490+0000 7f6788afd700 1 -- 172.21.15.27:0/1609236322 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6780004300 con 0x7f678407cc60 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.490+0000 7f6788afd700 1 -- 172.21.15.27:0/1609236322 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6780011af0 con 0x7f678407cc60 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.491+0000 7f678bd63700 1 -- 172.21.15.27:0/1609236322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 msgr2=0x7f678410c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.491+0000 7f678bd63700 1 --2- 172.21.15.27:0/1609236322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678410c500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f67800098a0 tx=0x7f6780009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.491+0000 7f678bd63700 1 -- 172.21.15.27:0/1609236322 shutdown_connections 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.491+0000 7f678bd63700 1 --2- 172.21.15.27:0/1609236322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678410c500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.491+0000 7f678bd63700 1 -- 172.21.15.27:0/1609236322 >> 172.21.15.27:0/1609236322 conn(0x7f6784103fb0 msgr2=0x7f6784106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.491+0000 7f678bd63700 1 -- 172.21.15.27:0/1609236322 shutdown_connections 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.491+0000 7f678bd63700 1 -- 172.21.15.27:0/1609236322 wait complete. 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.492+0000 7f678bd63700 1 Processor -- start 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.492+0000 7f678bd63700 1 -- start start 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.492+0000 7f678bd63700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.492+0000 7f678bd63700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f678410dd10 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.492+0000 7f6789aff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.492+0000 7f6789aff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678419dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43154/0 (socket says 172.21.15.27:43154) 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.492+0000 7f6789aff700 1 -- 172.21.15.27:0/2183361700 learned_addr learned my addr 172.21.15.27:0/2183361700 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.492+0000 7f6789aff700 1 -- 172.21.15.27:0/2183361700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678419e270 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.493+0000 7f6789aff700 1 --2- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678419dd30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6780009fb0 tx=0x7f6780011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.493+0000 7f677affd700 1 -- 172.21.15.27:0/2183361700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67800036a0 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.493+0000 7f677affd700 1 -- 172.21.15.27:0/2183361700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f678001a430 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.493+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f678419e500 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.493+0000 7f677affd700 1 -- 172.21.15.27:0/2183361700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6780019550 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.493+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67841a1100 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.494+0000 7f677affd700 1 -- 172.21.15.27:0/2183361700 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f6780019800 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.494+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6784064cf0 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.494+0000 7f677affd700 1 --2- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f677003d570 0x7f677003fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.494+0000 7f677affd700 1 -- 172.21.15.27:0/2183361700 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6780052780 con 0x7f678407cc60 2024-04-25T13:31:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.494+0000 7f67892fe700 1 --2- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f677003d570 0x7f677003fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.495+0000 7f67892fe700 1 --2- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f677003d570 0x7f677003fa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6774006fd0 tx=0x7f6774006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.499+0000 7f677affd700 1 -- 172.21.15.27:0/2183361700 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6780023390 con 0x7f678407cc60 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:10.718+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}) -- 0x7f6784106040 con 0x7f677003d570 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.994+0000 7f677affd700 1 -- 172.21.15.27:0/2183361700 <== mgr.14120 v2:172.21.15.27:6800/1748672481 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f6784106040 con 0x7f677003d570 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f677003d570 msgr2=0x7f677003fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 --2- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f677003d570 0x7f677003fa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6774006fd0 tx=0x7f6774006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 msgr2=0x7f678419dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 --2- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678419dd30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6780009fb0 tx=0x7f6780011940 comp rx=0 tx=0).stop 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 shutdown_connections 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 --2- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f677003d570 0x7f677003fa80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 --2- 172.21.15.27:0/2183361700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f678407cc60 0x7f678419dd30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 >> 172.21.15.27:0/2183361700 conn(0x7f6784103fb0 msgr2=0x7f67841058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 shutdown_connections 2024-04-25T13:31:15.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:14.997+0000 7f678bd63700 1 -- 172.21.15.27:0/2183361700 wait complete. 2024-04-25T13:31:15.631 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mon service... 2024-04-25T13:31:16.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:15 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:16.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:15 smithi027 ceph-mon[29140]: Added host smithi027 2024-04-25T13:31:16.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:15 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:17.075 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.121+0000 7fb28e363700 1 Processor -- start 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.121+0000 7fb28e363700 1 -- start start 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.122+0000 7fb28e363700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb288109e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.122+0000 7fb28e363700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb28810a450 con 0x7fb288109a50 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.122+0000 7fb287fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb288109e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.122+0000 7fb287fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb288109e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57412/0 (socket says 172.21.15.27:57412) 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.122+0000 7fb287fff700 1 -- 172.21.15.27:0/1949765671 learned_addr learned my addr 172.21.15.27:0/1949765671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.122+0000 7fb287fff700 1 -- 172.21.15.27:0/1949765671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28810ab30 con 0x7fb288109a50 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.123+0000 7fb287fff700 1 --2- 172.21.15.27:0/1949765671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb288109e60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb2780098a0 tx=0x7fb278009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2dea8b024c5568 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.123+0000 7fb286ffd700 1 -- 172.21.15.27:0/1949765671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb278004160 con 0x7fb288109a50 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.123+0000 7fb286ffd700 1 -- 172.21.15.27:0/1949765671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb278004300 con 0x7fb288109a50 2024-04-25T13:31:17.076 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.123+0000 7fb286ffd700 1 -- 172.21.15.27:0/1949765671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb278011af0 con 0x7fb288109a50 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.124+0000 7fb28e363700 1 -- 172.21.15.27:0/1949765671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 msgr2=0x7fb288109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.124+0000 7fb28e363700 1 --2- 172.21.15.27:0/1949765671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb288109e60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb2780098a0 tx=0x7fb278009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.124+0000 7fb28e363700 1 -- 172.21.15.27:0/1949765671 shutdown_connections 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.124+0000 7fb28e363700 1 --2- 172.21.15.27:0/1949765671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb288109e60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.124+0000 7fb28e363700 1 -- 172.21.15.27:0/1949765671 >> 172.21.15.27:0/1949765671 conn(0x7fb2881050b0 msgr2=0x7fb288107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.124+0000 7fb28e363700 1 -- 172.21.15.27:0/1949765671 shutdown_connections 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.124+0000 7fb28e363700 1 -- 172.21.15.27:0/1949765671 wait complete. 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.125+0000 7fb28e363700 1 Processor -- start 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.125+0000 7fb28e363700 1 -- start start 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.125+0000 7fb28e363700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb28807ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.125+0000 7fb28e363700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb28810dee0 con 0x7fb288109a50 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.125+0000 7fb287fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb28807ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.125+0000 7fb287fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb28807ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57420/0 (socket says 172.21.15.27:57420) 2024-04-25T13:31:17.077 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.125+0000 7fb287fff700 1 -- 172.21.15.27:0/1065197509 learned_addr learned my addr 172.21.15.27:0/1065197509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.126+0000 7fb287fff700 1 -- 172.21.15.27:0/1065197509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28807ef40 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.126+0000 7fb287fff700 1 --2- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb28807ea00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb278011890 tx=0x7fb278011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.126+0000 7fb2857fa700 1 -- 172.21.15.27:0/1065197509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2780039e0 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.126+0000 7fb2857fa700 1 -- 172.21.15.27:0/1065197509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb278003b80 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.126+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb28807d190 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.126+0000 7fb2857fa700 1 -- 172.21.15.27:0/1065197509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2780246e0 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.126+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb28807d670 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.127+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb288064cf0 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.127+0000 7fb2857fa700 1 -- 172.21.15.27:0/1065197509 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fb278022020 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.127+0000 7fb2857fa700 1 --2- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb27003d5c0 0x7fb27003fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.128+0000 7fb2857fa700 1 -- 172.21.15.27:0/1065197509 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb278052560 con 0x7fb288109a50 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.128+0000 7fb2877fe700 1 --2- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb27003d5c0 0x7fb27003fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:17.078 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.128+0000 7fb2877fe700 1 --2- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb27003d5c0 0x7fb27003fad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb27c006fd0 tx=0x7fb27c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.133+0000 7fb2857fa700 1 -- 172.21.15.27:0/1065197509 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb27802c350 con 0x7fb288109a50 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.380+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fb28807de30 con 0x7fb27003d5c0 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.388+0000 7fb2857fa700 1 -- 172.21.15.27:0/1065197509 <== mgr.14120 v2:172.21.15.27:6800/1748672481 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb28807de30 con 0x7fb27003d5c0 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb27003d5c0 msgr2=0x7fb27003fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 --2- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb27003d5c0 0x7fb27003fad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb27c006fd0 tx=0x7fb27c006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 msgr2=0x7fb28807ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 --2- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb28807ea00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb278011890 tx=0x7fb278011970 comp rx=0 tx=0).stop 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 shutdown_connections 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 --2- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb27003d5c0 0x7fb27003fad0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 --2- 172.21.15.27:0/1065197509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb288109a50 0x7fb28807ea00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 >> 172.21.15.27:0/1065197509 conn(0x7fb2881050b0 msgr2=0x7fb288105940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.390+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 shutdown_connections 2024-04-25T13:31:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:16.391+0000 7fb28e363700 1 -- 172.21.15.27:0/1065197509 wait complete. 2024-04-25T13:31:17.080 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying unmanaged mgr service... 2024-04-25T13:31:17.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:17 smithi027 ceph-mon[29140]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:17.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:17 smithi027 ceph-mon[29140]: Saving service mon spec with placement count:5 2024-04-25T13:31:17.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:17 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.700+0000 7f58ab0a8700 1 Processor -- start 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.700+0000 7f58ab0a8700 1 -- start start 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.700+0000 7f58ab0a8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a410c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.700+0000 7f58ab0a8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f58a410cb50 con 0x7f58a407d4b0 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.701+0000 7f58a8e44700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a410c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.701+0000 7f58a8e44700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a410c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57430/0 (socket says 172.21.15.27:57430) 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.701+0000 7f58a8e44700 1 -- 172.21.15.27:0/2550481277 learned_addr learned my addr 172.21.15.27:0/2550481277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:18.962 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.701+0000 7f58a8e44700 1 -- 172.21.15.27:0/2550481277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a410ccd0 con 0x7f58a407d4b0 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.701+0000 7f58a8e44700 1 --2- 172.21.15.27:0/2550481277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a410c580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f589c009e80 tx=0x7f589c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571db011dfaf7b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.701+0000 7f58a37fe700 1 -- 172.21.15.27:0/2550481277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f589c017070 con 0x7f58a407d4b0 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58a37fe700 1 -- 172.21.15.27:0/2550481277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f589c00b8c0 con 0x7f58a407d4b0 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58a37fe700 1 -- 172.21.15.27:0/2550481277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f589c004490 con 0x7f58a407d4b0 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2550481277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 msgr2=0x7f58a410c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58ab0a8700 1 --2- 172.21.15.27:0/2550481277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a410c580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f589c009e80 tx=0x7f589c00b330 comp rx=0 tx=0).stop 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2550481277 shutdown_connections 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58ab0a8700 1 --2- 172.21.15.27:0/2550481277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a410c580 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2550481277 >> 172.21.15.27:0/2550481277 conn(0x7f58a4103f90 msgr2=0x7f58a4106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2550481277 shutdown_connections 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.702+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2550481277 wait complete. 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.703+0000 7f58ab0a8700 1 Processor -- start 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.703+0000 7f58ab0a8700 1 -- start start 2024-04-25T13:31:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.703+0000 7f58ab0a8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.703+0000 7f58ab0a8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f58a410dd50 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.703+0000 7f58a8e44700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.703+0000 7f58a8e44700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a419dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57442/0 (socket says 172.21.15.27:57442) 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.703+0000 7f58a8e44700 1 -- 172.21.15.27:0/2837269794 learned_addr learned my addr 172.21.15.27:0/2837269794 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.704+0000 7f58a8e44700 1 -- 172.21.15.27:0/2837269794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a419e230 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.704+0000 7f58a8e44700 1 --2- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a419dcf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f589c004220 tx=0x7f589c00bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.704+0000 7f58a1ffb700 1 -- 172.21.15.27:0/2837269794 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f589c020020 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.704+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a419e4c0 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.704+0000 7f58a1ffb700 1 -- 172.21.15.27:0/2837269794 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f589c012430 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.704+0000 7f58a1ffb700 1 -- 172.21.15.27:0/2837269794 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f589c017040 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.704+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a41a10d0 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.705+0000 7f58a1ffb700 1 -- 172.21.15.27:0/2837269794 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f589c023020 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.705+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a4064cf0 con 0x7f58a407d4b0 2024-04-25T13:31:18.964 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.705+0000 7f58a1ffb700 1 --2- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f588c03d570 0x7f588c03fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.706+0000 7f58a1ffb700 1 -- 172.21.15.27:0/2837269794 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f589c00f050 con 0x7f58a407d4b0 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.706+0000 7f58a3fff700 1 --2- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f588c03d570 0x7f588c03fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.706+0000 7f58a3fff700 1 --2- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f588c03d570 0x7f588c03fa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5898006fd0 tx=0x7f5898006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.711+0000 7f58a1ffb700 1 -- 172.21.15.27:0/2837269794 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f589c02a020 con 0x7f58a407d4b0 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.935+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f58a4104f70 con 0x7f588c03d570 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.941+0000 7f58a1ffb700 1 -- 172.21.15.27:0/2837269794 <== mgr.14120 v2:172.21.15.27:6800/1748672481 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f58a4104f70 con 0x7f588c03d570 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f588c03d570 msgr2=0x7f588c03fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 --2- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f588c03d570 0x7f588c03fa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5898006fd0 tx=0x7f5898006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 msgr2=0x7f58a419dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 --2- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a419dcf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f589c004220 tx=0x7f589c00bdc0 comp rx=0 tx=0).stop 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 shutdown_connections 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 --2- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f588c03d570 0x7f588c03fa80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 --2- 172.21.15.27:0/2837269794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58a407d4b0 0x7f58a419dcf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:18.965 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 >> 172.21.15.27:0/2837269794 conn(0x7f58a4103f90 msgr2=0x7f58a4104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:18.966 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 shutdown_connections 2024-04-25T13:31:18.966 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:17.943+0000 7f58ab0a8700 1 -- 172.21.15.27:0/2837269794 wait complete. 2024-04-25T13:31:19.082 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:18 smithi027 ceph-mon[29140]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:19.083 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:18 smithi027 ceph-mon[29140]: Saving service mgr spec with placement count:2 2024-04-25T13:31:19.083 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:18 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:19.083 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:18 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:20.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:20 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:20.236 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:20 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1938078397' entity='client.admin' 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.433+0000 7f5cc4b14700 1 Processor -- start 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.433+0000 7f5cc4b14700 1 -- start start 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.433+0000 7f5cc4b14700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc010b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.433+0000 7f5cc4b14700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc010b900 con 0x7f5cc010af20 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.433+0000 7f5cbe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc010b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.433+0000 7f5cbe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc010b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57448/0 (socket says 172.21.15.27:57448) 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.433+0000 7f5cbe59c700 1 -- 172.21.15.27:0/1164460392 learned_addr learned my addr 172.21.15.27:0/1164460392 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.434+0000 7f5cbe59c700 1 -- 172.21.15.27:0/1164460392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc010ba80 con 0x7f5cc010af20 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.434+0000 7f5cbe59c700 1 --2- 172.21.15.27:0/1164460392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc010b330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40098a0 tx=0x7f5cb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4660946bc4077b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.434+0000 7f5cbd59a700 1 -- 172.21.15.27:0/1164460392 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cb4004160 con 0x7f5cc010af20 2024-04-25T13:31:20.244 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.434+0000 7f5cbd59a700 1 -- 172.21.15.27:0/1164460392 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb4004300 con 0x7f5cc010af20 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.434+0000 7f5cbd59a700 1 -- 172.21.15.27:0/1164460392 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cb4011a20 con 0x7f5cc010af20 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.435+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1164460392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 msgr2=0x7f5cc010b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.435+0000 7f5cc4b14700 1 --2- 172.21.15.27:0/1164460392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc010b330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40098a0 tx=0x7f5cb4009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.435+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1164460392 shutdown_connections 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.435+0000 7f5cc4b14700 1 --2- 172.21.15.27:0/1164460392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc010b330 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.435+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1164460392 >> 172.21.15.27:0/1164460392 conn(0x7f5cc01036e0 msgr2=0x7f5cc0105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.435+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1164460392 shutdown_connections 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.435+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1164460392 wait complete. 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.436+0000 7f5cc4b14700 1 Processor -- start 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.436+0000 7f5cc4b14700 1 -- start start 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.436+0000 7f5cc4b14700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.436+0000 7f5cc4b14700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc010dec0 con 0x7f5cc010af20 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.436+0000 7f5cbe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:20.245 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.436+0000 7f5cbe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc019dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57460/0 (socket says 172.21.15.27:57460) 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.436+0000 7f5cbe59c700 1 -- 172.21.15.27:0/1938078397 learned_addr learned my addr 172.21.15.27:0/1938078397 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.437+0000 7f5cbe59c700 1 -- 172.21.15.27:0/1938078397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc019e200 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.437+0000 7f5cbe59c700 1 --2- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc019dcc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40119f0 tx=0x7f5cb4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.437+0000 7f5caf7fe700 1 -- 172.21.15.27:0/1938078397 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cb40039e0 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.437+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc019e490 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.437+0000 7f5caf7fe700 1 -- 172.21.15.27:0/1938078397 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb401b8f0 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.437+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc01a10a0 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.437+0000 7f5caf7fe700 1 -- 172.21.15.27:0/1938078397 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cb40237a0 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.438+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ca0005360 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.438+0000 7f5caf7fe700 1 -- 172.21.15.27:0/1938078397 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f5cb402c3f0 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.438+0000 7f5caf7fe700 1 --2- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f5ca803d520 0x7f5ca803fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.438+0000 7f5caf7fe700 1 -- 172.21.15.27:0/1938078397 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5cb4052bb0 con 0x7f5cc010af20 2024-04-25T13:31:20.246 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.439+0000 7f5cbdd9b700 1 --2- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f5ca803d520 0x7f5ca803fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.439+0000 7f5cbdd9b700 1 --2- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f5ca803d520 0x7f5ca803fa30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0006fd0 tx=0x7f5cb0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.444+0000 7f5caf7fe700 1 -- 172.21.15.27:0/1938078397 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5cb401f030 con 0x7f5cc010af20 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.660+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f5ca0005650 con 0x7f5cc010af20 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.663+0000 7f5caf7fe700 1 -- 172.21.15.27:0/1938078397 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f5cb40172b0 con 0x7f5cc010af20 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.665+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f5ca803d520 msgr2=0x7f5ca803fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.665+0000 7f5cc4b14700 1 --2- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f5ca803d520 0x7f5ca803fa30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0006fd0 tx=0x7f5cb0006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.665+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 msgr2=0x7f5cc019dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.665+0000 7f5cc4b14700 1 --2- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc019dcc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40119f0 tx=0x7f5cb4011ad0 comp rx=0 tx=0).stop 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.666+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 shutdown_connections 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.666+0000 7f5cc4b14700 1 --2- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f5ca803d520 0x7f5ca803fa30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.666+0000 7f5cc4b14700 1 --2- 172.21.15.27:0/1938078397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc010af20 0x7f5cc019dcc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.666+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 >> 172.21.15.27:0/1938078397 conn(0x7f5cc01036e0 msgr2=0x7f5cc0104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:20.247 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.666+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 shutdown_connections 2024-04-25T13:31:20.248 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:19.666+0000 7f5cc4b14700 1 -- 172.21.15.27:0/1938078397 wait complete. 2024-04-25T13:31:21.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.684+0000 7f4b6d7f2700 1 Processor -- start 2024-04-25T13:31:21.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.684+0000 7f4b6d7f2700 1 -- start start 2024-04-25T13:31:21.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.685+0000 7f4b6d7f2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6807d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:21.544 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.685+0000 7f4b6d7f2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6807d710 con 0x7f4b6807ecb0 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.685+0000 7f4b66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6807d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.685+0000 7f4b66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6807d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57474/0 (socket says 172.21.15.27:57474) 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.685+0000 7f4b66ffd700 1 -- 172.21.15.27:0/2607551523 learned_addr learned my addr 172.21.15.27:0/2607551523 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.685+0000 7f4b66ffd700 1 -- 172.21.15.27:0/2607551523 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6807d890 con 0x7f4b6807ecb0 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.685+0000 7f4b66ffd700 1 --2- 172.21.15.27:0/2607551523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6807d1d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b580098a0 tx=0x7f4b58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0e90b38e43688c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.685+0000 7f4b667fc700 1 -- 172.21.15.27:0/2607551523 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b58004160 con 0x7f4b6807ecb0 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b667fc700 1 -- 172.21.15.27:0/2607551523 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b58004300 con 0x7f4b6807ecb0 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b667fc700 1 -- 172.21.15.27:0/2607551523 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b58011a20 con 0x7f4b6807ecb0 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2607551523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 msgr2=0x7f4b6807d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b6d7f2700 1 --2- 172.21.15.27:0/2607551523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6807d1d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b580098a0 tx=0x7f4b58009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2607551523 shutdown_connections 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b6d7f2700 1 --2- 172.21.15.27:0/2607551523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6807d1d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:21.545 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2607551523 >> 172.21.15.27:0/2607551523 conn(0x7f4b681050b0 msgr2=0x7f4b68107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2607551523 shutdown_connections 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.686+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2607551523 wait complete. 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.687+0000 7f4b6d7f2700 1 Processor -- start 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.687+0000 7f4b6d7f2700 1 -- start start 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.687+0000 7f4b6d7f2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6819ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.687+0000 7f4b6d7f2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6810ffb0 con 0x7f4b6807ecb0 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.687+0000 7f4b66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6819ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.687+0000 7f4b66ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6819ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57476/0 (socket says 172.21.15.27:57476) 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.687+0000 7f4b66ffd700 1 -- 172.21.15.27:0/2754527589 learned_addr learned my addr 172.21.15.27:0/2754527589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.688+0000 7f4b66ffd700 1 -- 172.21.15.27:0/2754527589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b681a0440 con 0x7f4b6807ecb0 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.688+0000 7f4b66ffd700 1 --2- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6819ff00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4b58000c00 tx=0x7f4b58011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.688+0000 7f4b64ff9700 1 -- 172.21.15.27:0/2754527589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b58003aa0 con 0x7f4b6807ecb0 2024-04-25T13:31:21.546 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.688+0000 7f4b64ff9700 1 -- 172.21.15.27:0/2754527589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b5801b8f0 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.688+0000 7f4b64ff9700 1 -- 172.21.15.27:0/2754527589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b58023890 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.688+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b681a06d0 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.688+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b681a0af0 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.689+0000 7f4b64ff9700 1 -- 172.21.15.27:0/2754527589 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f4b5802c3f0 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.689+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b68064cf0 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.690+0000 7f4b64ff9700 1 --2- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f4b5403d5c0 0x7f4b5403fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.690+0000 7f4b64ff9700 1 -- 172.21.15.27:0/2754527589 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4b58052d70 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.690+0000 7f4b5ffff700 1 --2- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f4b5403d5c0 0x7f4b5403fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.690+0000 7f4b5ffff700 1 --2- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f4b5403d5c0 0x7f4b5403fad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4b50006fd0 tx=0x7f4b50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.695+0000 7f4b64ff9700 1 -- 172.21.15.27:0/2754527589 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b58057070 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.903+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f4b681a34c0 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.906+0000 7f4b64ff9700 1 -- 172.21.15.27:0/2754527589 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f4b5801f030 con 0x7f4b6807ecb0 2024-04-25T13:31:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f4b5403d5c0 msgr2=0x7f4b5403fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 --2- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f4b5403d5c0 0x7f4b5403fad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4b50006fd0 tx=0x7f4b50006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 msgr2=0x7f4b6819ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 --2- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6819ff00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4b58000c00 tx=0x7f4b58011aa0 comp rx=0 tx=0).stop 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 shutdown_connections 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 --2- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7f4b5403d5c0 0x7f4b5403fad0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 --2- 172.21.15.27:0/2754527589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b6807ecb0 0x7f4b6819ff00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 >> 172.21.15.27:0/2754527589 conn(0x7f4b681050b0 msgr2=0x7f4b68105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 shutdown_connections 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:20.909+0000 7f4b6d7f2700 1 -- 172.21.15.27:0/2754527589 wait complete. 2024-04-25T13:31:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling the dashboard module... 2024-04-25T13:31:22.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:21 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2754527589' entity='client.admin' 2024-04-25T13:31:23.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:22 smithi027 ceph-mon[29140]: from='mgr.14120 172.21.15.27:0/3096706268' entity='mgr.x' 2024-04-25T13:31:23.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:22 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/251021789' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T13:31:23.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: ignoring --setuser ceph since I am not root 2024-04-25T13:31:23.152 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: ignoring --setgroup ceph since I am not root 2024-04-25T13:31:23.489 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:23.181+0000 7f1089b1b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T13:31:23.489 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:23.379+0000 7f1089b1b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:31:23.489 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:23.488+0000 7f1089b1b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.505+0000 7fc613bd4700 1 Processor -- start 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.506+0000 7fc613bd4700 1 -- start start 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.506+0000 7fc613bd4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c108ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.506+0000 7fc613bd4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c1092d0 con 0x7fc60c1088d0 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.506+0000 7fc611970700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c108ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.506+0000 7fc611970700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c108ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57492/0 (socket says 172.21.15.27:57492) 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.506+0000 7fc611970700 1 -- 172.21.15.27:0/1667067965 learned_addr learned my addr 172.21.15.27:0/1667067965 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.506+0000 7fc611970700 1 -- 172.21.15.27:0/1667067965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c109a00 con 0x7fc60c1088d0 2024-04-25T13:31:23.595 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.506+0000 7fc611970700 1 --2- 172.21.15.27:0/1667067965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c108ce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc6080098a0 tx=0x7fc608009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe1fd06bbd0dd1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.507+0000 7fc61096e700 1 -- 172.21.15.27:0/1667067965 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc608004160 con 0x7fc60c1088d0 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.507+0000 7fc61096e700 1 -- 172.21.15.27:0/1667067965 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc608004300 con 0x7fc60c1088d0 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.507+0000 7fc61096e700 1 -- 172.21.15.27:0/1667067965 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc608011a20 con 0x7fc60c1088d0 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.507+0000 7fc613bd4700 1 -- 172.21.15.27:0/1667067965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 msgr2=0x7fc60c108ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.507+0000 7fc613bd4700 1 --2- 172.21.15.27:0/1667067965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c108ce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc6080098a0 tx=0x7fc608009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.507+0000 7fc613bd4700 1 -- 172.21.15.27:0/1667067965 shutdown_connections 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.507+0000 7fc613bd4700 1 --2- 172.21.15.27:0/1667067965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c108ce0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.508+0000 7fc613bd4700 1 -- 172.21.15.27:0/1667067965 >> 172.21.15.27:0/1667067965 conn(0x7fc60c103f10 msgr2=0x7fc60c1063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.508+0000 7fc613bd4700 1 -- 172.21.15.27:0/1667067965 shutdown_connections 2024-04-25T13:31:23.596 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.508+0000 7fc613bd4700 1 -- 172.21.15.27:0/1667067965 wait complete. 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.508+0000 7fc613bd4700 1 Processor -- start 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.508+0000 7fc613bd4700 1 -- start start 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc613bd4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c19dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc613bd4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c10cdb0 con 0x7fc60c1088d0 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc611970700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c19dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc611970700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c19dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57498/0 (socket says 172.21.15.27:57498) 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc611970700 1 -- 172.21.15.27:0/251021789 learned_addr learned my addr 172.21.15.27:0/251021789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc611970700 1 -- 172.21.15.27:0/251021789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c19e270 con 0x7fc60c1088d0 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc611970700 1 --2- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c19dd30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc608003e70 tx=0x7fc608011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc602ffd700 1 -- 172.21.15.27:0/251021789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc608003af0 con 0x7fc60c1088d0 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60c19e500 con 0x7fc60c1088d0 2024-04-25T13:31:23.597 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc602ffd700 1 -- 172.21.15.27:0/251021789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc60801a8e0 con 0x7fc60c1088d0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc602ffd700 1 -- 172.21.15.27:0/251021789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc608024870 con 0x7fc60c1088d0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.509+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60c1a1100 con 0x7fc60c1088d0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.510+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60c064cf0 con 0x7fc60c1088d0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.510+0000 7fc602ffd700 1 -- 172.21.15.27:0/251021789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fc608022020 con 0x7fc60c1088d0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.511+0000 7fc602ffd700 1 --2- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fc5f803d520 0x7fc5f803fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.511+0000 7fc602ffd700 1 -- 172.21.15.27:0/251021789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc6080527b0 con 0x7fc60c1088d0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.511+0000 7fc61116f700 1 --2- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fc5f803d520 0x7fc5f803fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.511+0000 7fc61116f700 1 --2- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fc5f803d520 0x7fc5f803fa30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc006fd0 tx=0x7fc5fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.516+0000 7fc602ffd700 1 -- 172.21.15.27:0/251021789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc60800e260 con 0x7fc60c1088d0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.802+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fc60c1a12d0 con 0x7fc60c1088d0 2024-04-25T13:31:23.598 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.939+0000 7fc602ffd700 1 -- 172.21.15.27:0/251021789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fc608052140 con 0x7fc60c1088d0 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.941+0000 7fc602ffd700 1 -- 172.21.15.27:0/251021789 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7fc60802dc90 con 0x7fc60c1088d0 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fc5f803d520 msgr2=0x7fc5f803fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 --2- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fc5f803d520 0x7fc5f803fa30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc006fd0 tx=0x7fc5fc006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 msgr2=0x7fc60c19dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 --2- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c19dd30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc608003e70 tx=0x7fc608011aa0 comp rx=0 tx=0).stop 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 shutdown_connections 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 --2- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fc5f803d520 0x7fc5f803fa30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 --2- 172.21.15.27:0/251021789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc60c1088d0 0x7fc60c19dd30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 >> 172.21.15.27:0/251021789 conn(0x7fc60c103f10 msgr2=0x7fc60c104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 shutdown_connections 2024-04-25T13:31:23.599 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:22.942+0000 7fc613bd4700 1 -- 172.21.15.27:0/251021789 wait complete. 2024-04-25T13:31:24.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/251021789' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T13:31:24.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-mon[29140]: mgrmap e9: x(active, since 19s) 2024-04-25T13:31:24.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:23.817+0000 7f1089b1b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T13:31:24.152 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:23.911+0000 7f1089b1b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.035+0000 7fb94049b700 1 Processor -- start 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.036+0000 7fb94049b700 1 -- start start 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.036+0000 7fb94049b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93810c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:24.927 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.036+0000 7fb94049b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb93810cae0 con 0x7fb93807cc40 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.036+0000 7fb93e237700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93810c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.036+0000 7fb93e237700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93810c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42232/0 (socket says 172.21.15.27:42232) 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.036+0000 7fb93e237700 1 -- 172.21.15.27:0/264496115 learned_addr learned my addr 172.21.15.27:0/264496115 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.037+0000 7fb93e237700 1 -- 172.21.15.27:0/264496115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93810cc60 con 0x7fb93807cc40 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.037+0000 7fb93e237700 1 --2- 172.21.15.27:0/264496115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93810c4f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9340098a0 tx=0x7fb934009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3266a7f1199d95a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.037+0000 7fb93d235700 1 -- 172.21.15.27:0/264496115 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb934004160 con 0x7fb93807cc40 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.037+0000 7fb93d235700 1 -- 172.21.15.27:0/264496115 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb934004300 con 0x7fb93807cc40 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.037+0000 7fb93d235700 1 -- 172.21.15.27:0/264496115 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb934011a20 con 0x7fb93807cc40 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.038+0000 7fb94049b700 1 -- 172.21.15.27:0/264496115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 msgr2=0x7fb93810c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.038+0000 7fb94049b700 1 --2- 172.21.15.27:0/264496115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93810c4f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9340098a0 tx=0x7fb934009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.038+0000 7fb94049b700 1 -- 172.21.15.27:0/264496115 shutdown_connections 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.038+0000 7fb94049b700 1 --2- 172.21.15.27:0/264496115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93810c4f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.038+0000 7fb94049b700 1 -- 172.21.15.27:0/264496115 >> 172.21.15.27:0/264496115 conn(0x7fb938103f80 msgr2=0x7fb9381063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.038+0000 7fb94049b700 1 -- 172.21.15.27:0/264496115 shutdown_connections 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.038+0000 7fb94049b700 1 -- 172.21.15.27:0/264496115 wait complete. 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb94049b700 1 Processor -- start 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb94049b700 1 -- start start 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb94049b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93819dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb94049b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb93810dd20 con 0x7fb93807cc40 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb93e237700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93819dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb93e237700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93819dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42246/0 (socket says 172.21.15.27:42246) 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb93e237700 1 -- 172.21.15.27:0/796803141 learned_addr learned my addr 172.21.15.27:0/796803141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb93e237700 1 -- 172.21.15.27:0/796803141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93819e1e0 con 0x7fb93807cc40 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.039+0000 7fb93e237700 1 --2- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93819dca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb9340119f0 tx=0x7fb934011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.040+0000 7fb92f7fe700 1 -- 172.21.15.27:0/796803141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9340039e0 con 0x7fb93807cc40 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.040+0000 7fb92f7fe700 1 -- 172.21.15.27:0/796803141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb93401b8f0 con 0x7fb93807cc40 2024-04-25T13:31:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.040+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb93819e470 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.040+0000 7fb92f7fe700 1 -- 172.21.15.27:0/796803141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9340237a0 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.040+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9381a1080 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.041+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb938064cf0 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.041+0000 7fb92f7fe700 1 -- 172.21.15.27:0/796803141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7fb9340193d0 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.041+0000 7fb92f7fe700 1 --2- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb92403d5d0 0x7fb92403fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.041+0000 7fb92f7fe700 1 -- 172.21.15.27:0/796803141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb934052d90 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.041+0000 7fb93da36700 1 -- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb92403d5d0 msgr2=0x7fb92403fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1748672481 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.041+0000 7fb93da36700 1 --2- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb92403d5d0 0x7fb92403fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.046+0000 7fb92f7fe700 1 -- 172.21.15.27:0/796803141 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb93401f030 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.242+0000 7fb93da36700 1 -- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb92403d5d0 msgr2=0x7fb92403fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1748672481 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.242+0000 7fb93da36700 1 --2- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb92403d5d0 0x7fb92403fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.324+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fb9381a1250 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.327+0000 7fb92f7fe700 1 -- 172.21.15.27:0/796803141 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fb934018e60 con 0x7fb93807cc40 2024-04-25T13:31:24.930 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb92403d5d0 msgr2=0x7fb92403fae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 --2- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb92403d5d0 0x7fb92403fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 msgr2=0x7fb93819dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 --2- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93819dca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb9340119f0 tx=0x7fb934011ad0 comp rx=0 tx=0).stop 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 shutdown_connections 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 --2- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7fb92403d5d0 0x7fb92403fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 --2- 172.21.15.27:0/796803141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb93807cc40 0x7fb93819dca0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 >> 172.21.15.27:0/796803141 conn(0x7fb938103f80 msgr2=0x7fb9381058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 shutdown_connections 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:24.331+0000 7fb94049b700 1 -- 172.21.15.27:0/796803141 wait complete. 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-04-25T13:31:24.931 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 9... 2024-04-25T13:31:25.047 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:24 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/796803141' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T13:31:25.298 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:25.186+0000 7f1089b1b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T13:31:25.553 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:25.372+0000 7f1089b1b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T13:31:25.553 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:25.552+0000 7f1089b1b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T13:31:26.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:25.860+0000 7f1089b1b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T13:31:26.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:25.959+0000 7f1089b1b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T13:31:26.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:26 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:26.171+0000 7f1089b1b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T13:31:26.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:26 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:26.375+0000 7f1089b1b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:31:27.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:27.010+0000 7f1089b1b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T13:31:27.777 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:27.460+0000 7f1089b1b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T13:31:27.778 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:27.562+0000 7f1089b1b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T13:31:27.778 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:27.777+0000 7f1089b1b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T13:31:28.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:28.327+0000 7f1089b1b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T13:31:28.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:28.426+0000 7f1089b1b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T13:31:29.051 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:28.685+0000 7f1089b1b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T13:31:29.051 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:28.826+0000 7f1089b1b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T13:31:29.052 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:28.927+0000 7f1089b1b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T13:31:29.052 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:29 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:29.050+0000 7f1089b1b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T13:31:29.651 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:29 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:29.370+0000 7f1089b1b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T13:31:30.287 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:30.005+0000 7f1089b1b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T13:31:30.288 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:31:30.233+0000 7f1089b1b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T13:31:30.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: Active manager daemon x restarted 2024-04-25T13:31:30.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: Activating manager daemon x 2024-04-25T13:31:30.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T13:31:30.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: mgrmap e10: x(active, starting, since 0.00484052s) 2024-04-25T13:31:30.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:31:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T13:31:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T13:31:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T13:31:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T13:31:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: Manager daemon x is now available 2024-04-25T13:31:30.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:31.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T13:31:31.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T13:31:31.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:31 smithi027 ceph-mon[29140]: [25/Apr/2024:13:31:30] ENGINE Bus STARTING 2024-04-25T13:31:31.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:31 smithi027 ceph-mon[29140]: [25/Apr/2024:13:31:30] ENGINE Serving on https://172.21.15.27:7150 2024-04-25T13:31:31.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:31 smithi027 ceph-mon[29140]: [25/Apr/2024:13:31:30] ENGINE Serving on http://172.21.15.27:8765 2024-04-25T13:31:31.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:31 smithi027 ceph-mon[29140]: [25/Apr/2024:13:31:30] ENGINE Bus STARTED 2024-04-25T13:31:31.559 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:31 smithi027 ceph-mon[29140]: mgrmap e11: x(active, since 1.00687s) 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.408+0000 7faf0e09b700 1 Processor -- start 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.408+0000 7faf0e09b700 1 -- start start 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.408+0000 7faf0e09b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0810e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.408+0000 7faf0e09b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810ee80 con 0x7faf0807cc20 2024-04-25T13:31:31.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.408+0000 7faf077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0810e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.408+0000 7faf077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0810e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42248/0 (socket says 172.21.15.27:42248) 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.408+0000 7faf077fe700 1 -- 172.21.15.27:0/1953786421 learned_addr learned my addr 172.21.15.27:0/1953786421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.408+0000 7faf077fe700 1 -- 172.21.15.27:0/1953786421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0810f000 con 0x7faf0807cc20 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.409+0000 7faf077fe700 1 --2- 172.21.15.27:0/1953786421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0810e940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faef80098a0 tx=0x7faef8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6261e25c87227166 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.409+0000 7faf067fc700 1 -- 172.21.15.27:0/1953786421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faef8004160 con 0x7faf0807cc20 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.409+0000 7faf067fc700 1 -- 172.21.15.27:0/1953786421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef8004300 con 0x7faf0807cc20 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.409+0000 7faf067fc700 1 -- 172.21.15.27:0/1953786421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faef8011a20 con 0x7faf0807cc20 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.410+0000 7faf0e09b700 1 -- 172.21.15.27:0/1953786421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 msgr2=0x7faf0810e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.410+0000 7faf0e09b700 1 --2- 172.21.15.27:0/1953786421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0810e940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faef80098a0 tx=0x7faef8009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.410+0000 7faf0e09b700 1 -- 172.21.15.27:0/1953786421 shutdown_connections 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.410+0000 7faf0e09b700 1 --2- 172.21.15.27:0/1953786421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0810e940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:31.854 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.410+0000 7faf0e09b700 1 -- 172.21.15.27:0/1953786421 >> 172.21.15.27:0/1953786421 conn(0x7faf08103fb0 msgr2=0x7faf08106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.410+0000 7faf0e09b700 1 -- 172.21.15.27:0/1953786421 shutdown_connections 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.410+0000 7faf0e09b700 1 -- 172.21.15.27:0/1953786421 wait complete. 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.410+0000 7faf0e09b700 1 Processor -- start 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.411+0000 7faf0e09b700 1 -- start start 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.411+0000 7faf0e09b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0819fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.411+0000 7faf0e09b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810ff50 con 0x7faf0807cc20 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.411+0000 7faf077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0819fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.411+0000 7faf077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0819fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42254/0 (socket says 172.21.15.27:42254) 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.411+0000 7faf077fe700 1 -- 172.21.15.27:0/4022601664 learned_addr learned my addr 172.21.15.27:0/4022601664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.411+0000 7faf077fe700 1 -- 172.21.15.27:0/4022601664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf081a0410 con 0x7faf0807cc20 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.411+0000 7faf077fe700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0819fed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faef8004660 tx=0x7faef8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.412+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faef8003aa0 con 0x7faf0807cc20 2024-04-25T13:31:31.855 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.412+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef801b8f0 con 0x7faf0807cc20 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.412+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf081a06a0 con 0x7faf0807cc20 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.412+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faef8023900 con 0x7faf0807cc20 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.412+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf081a32b0 con 0x7faf0807cc20 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.413+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7faef8023bc0 con 0x7faf0807cc20 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.413+0000 7faf04ff9700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 0x7faef003fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.413+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 --> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7faef00401e0 con 0x7faef003d5c0 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.413+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faef8052f90 con 0x7faf0807cc20 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.413+0000 7faf06ffd700 1 -- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 msgr2=0x7faef003fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1748672481 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.413+0000 7faf06ffd700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 0x7faef003fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.614+0000 7faf06ffd700 1 -- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 msgr2=0x7faef003fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1748672481 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:25.614+0000 7faf06ffd700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 0x7faef003fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T13:31:31.856 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:26.015+0000 7faf06ffd700 1 -- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 msgr2=0x7faef003fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1748672481 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:26.015+0000 7faf06ffd700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 0x7faef003fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:26.816+0000 7faf06ffd700 1 -- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 msgr2=0x7faef003fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1748672481 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:26.816+0000 7faf06ffd700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 0x7faef003fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:28.417+0000 7faf06ffd700 1 -- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 msgr2=0x7faef003fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1748672481 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:28.417+0000 7faf06ffd700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 0x7faef003fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:30.237+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 10) ==== 49296+0+0 (secure 0 0 0) 0x7faef801baa0 con 0x7faf0807cc20 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:30.238+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 msgr2=0x7faef003fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:30.238+0000 7faf04ff9700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 0x7faef003fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.238+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7faef8054730 con 0x7faf0807cc20 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.239+0000 7faf04ff9700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faef8052b80 0x7faef0043000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.239+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7faef00401e0 con 0x7faef8052b80 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.239+0000 7faf06ffd700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faef8052b80 0x7faef0043000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.240+0000 7faf06ffd700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faef8052b80 0x7faef0043000 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faefc003a10 tx=0x7faefc0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:31.857 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.241+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7faef00401e0 con 0x7faef8052b80 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7faf08064cf0 con 0x7faef8052b80 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf04ff9700 1 -- 172.21.15.27:0/4022601664 <== mgr.14152 v2:172.21.15.27:6800/1716439151 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7faf08064cf0 con 0x7faef8052b80 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faef8052b80 msgr2=0x7faef0043000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faef8052b80 0x7faef0043000 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faefc003a10 tx=0x7faefc0092b0 comp rx=0 tx=0).stop 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 msgr2=0x7faf0819fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0819fed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faef8004660 tx=0x7faef8011aa0 comp rx=0 tx=0).stop 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 shutdown_connections 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faef8052b80 0x7faef0043000 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf0807cc20 0x7faf0819fed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 --2- 172.21.15.27:0/4022601664 >> [v2:172.21.15.27:6800/1748672481,v1:172.21.15.27:6801/1748672481] conn(0x7faef003d5c0 0x7faef003fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.257+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 >> 172.21.15.27:0/4022601664 conn(0x7faf08103fb0 msgr2=0x7faf08105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.258+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 shutdown_connections 2024-04-25T13:31:31.858 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:31.258+0000 7faf0e09b700 1 -- 172.21.15.27:0/4022601664 wait complete. 2024-04-25T13:31:31.859 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 9 is available 2024-04-25T13:31:31.859 INFO:teuthology.orchestra.run.smithi027.stdout:Generating a dashboard self-signed certificate... 2024-04-25T13:31:32.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:32 smithi027 ceph-mon[29140]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T13:31:32.500 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:32 smithi027 ceph-mon[29140]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T13:31:33.149 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T13:31:33.149 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.272+0000 7f3d204e2700 1 Processor -- start 2024-04-25T13:31:33.149 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.272+0000 7f3d204e2700 1 -- start start 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.273+0000 7f3d204e2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1810c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.273+0000 7f3d204e2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1807d600 con 0x7f3d1810a4a0 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.273+0000 7f3d1e27e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1810c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.273+0000 7f3d1e27e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1810c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42330/0 (socket says 172.21.15.27:42330) 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.273+0000 7f3d1e27e700 1 -- 172.21.15.27:0/938282841 learned_addr learned my addr 172.21.15.27:0/938282841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.273+0000 7f3d1e27e700 1 -- 172.21.15.27:0/938282841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1810ce00 con 0x7f3d1810a4a0 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.273+0000 7f3d1e27e700 1 --2- 172.21.15.27:0/938282841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1810c8c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3d080098a0 tx=0x7f3d08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5523ac5bdd844486 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.274+0000 7f3d1d27c700 1 -- 172.21.15.27:0/938282841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d08004160 con 0x7f3d1810a4a0 2024-04-25T13:31:33.150 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.274+0000 7f3d1d27c700 1 -- 172.21.15.27:0/938282841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d08004300 con 0x7f3d1810a4a0 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.274+0000 7f3d1d27c700 1 -- 172.21.15.27:0/938282841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d08011a20 con 0x7f3d1810a4a0 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.274+0000 7f3d204e2700 1 -- 172.21.15.27:0/938282841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 msgr2=0x7f3d1810c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.274+0000 7f3d204e2700 1 --2- 172.21.15.27:0/938282841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1810c8c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3d080098a0 tx=0x7f3d08009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.274+0000 7f3d204e2700 1 -- 172.21.15.27:0/938282841 shutdown_connections 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.274+0000 7f3d204e2700 1 --2- 172.21.15.27:0/938282841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1810c8c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.274+0000 7f3d204e2700 1 -- 172.21.15.27:0/938282841 >> 172.21.15.27:0/938282841 conn(0x7f3d18103f00 msgr2=0x7f3d181063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.275+0000 7f3d204e2700 1 -- 172.21.15.27:0/938282841 shutdown_connections 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.275+0000 7f3d204e2700 1 -- 172.21.15.27:0/938282841 wait complete. 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.275+0000 7f3d204e2700 1 Processor -- start 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.275+0000 7f3d204e2700 1 -- start start 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.275+0000 7f3d204e2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1819dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.275+0000 7f3d204e2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1810dd40 con 0x7f3d1810a4a0 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d1e27e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1819dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:33.151 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d1e27e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1819dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42336/0 (socket says 172.21.15.27:42336) 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d1e27e700 1 -- 172.21.15.27:0/1204753241 learned_addr learned my addr 172.21.15.27:0/1204753241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d1e27e700 1 -- 172.21.15.27:0/1204753241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1819e150 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d1e27e700 1 --2- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1819dc10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3d08000c00 tx=0x7f3d08011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d0f7fe700 1 -- 172.21.15.27:0/1204753241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d08003aa0 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d0f7fe700 1 -- 172.21.15.27:0/1204753241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d0801b8f0 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1819e3e0 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d0f7fe700 1 -- 172.21.15.27:0/1204753241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d08023890 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.276+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d181a0ff0 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.277+0000 7f3d0f7fe700 1 -- 172.21.15.27:0/1204753241 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7f3d0802c3f0 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.277+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d18064cf0 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.277+0000 7f3d0f7fe700 1 --2- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3d0403d560 0x7f3d0403fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.278+0000 7f3d0f7fe700 1 -- 172.21.15.27:0/1204753241 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3d08052ce0 con 0x7f3d1810a4a0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.278+0000 7f3d1da7d700 1 --2- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3d0403d560 0x7f3d0403fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:33.152 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.278+0000 7f3d1da7d700 1 --2- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3d0403d560 0x7f3d0403fa70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d14006fd0 tx=0x7f3d14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.283+0000 7f3d0f7fe700 1 -- 172.21.15.27:0/1204753241 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d08057070 con 0x7f3d1810a4a0 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.290+0000 7f3d0f7fe700 1 -- 172.21.15.27:0/1204753241 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f3d0802cb00 con 0x7f3d1810a4a0 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.499+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f3d18104e10 con 0x7f3d0403d560 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.550+0000 7f3d0f7fe700 1 -- 172.21.15.27:0/1204753241 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f3d18104e10 con 0x7f3d0403d560 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3d0403d560 msgr2=0x7f3d0403fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 --2- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3d0403d560 0x7f3d0403fa70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d14006fd0 tx=0x7f3d14006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 msgr2=0x7f3d1819dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 --2- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1819dc10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3d08000c00 tx=0x7f3d08011aa0 comp rx=0 tx=0).stop 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 shutdown_connections 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 --2- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3d0403d560 0x7f3d0403fa70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 --2- 172.21.15.27:0/1204753241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d1810a4a0 0x7f3d1819dc10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 >> 172.21.15.27:0/1204753241 conn(0x7f3d18103f00 msgr2=0x7f3d18104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.552+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 shutdown_connections 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:32.553+0000 7f3d204e2700 1 -- 172.21.15.27:0/1204753241 wait complete. 2024-04-25T13:31:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial admin user... 2024-04-25T13:31:33.396 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:33 smithi027 ceph-mon[29140]: mgrmap e12: x(active, since 2s) 2024-04-25T13:31:33.396 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:33 smithi027 ceph-mon[29140]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:33.396 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:33.396 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:33.396 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$iQJXe/MLwVX5JlU6.4GgU.InbG0IsEkSJXFiq8QIF8W56ukOaWh9O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714051894, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.605+0000 7f66e232b700 1 Processor -- start 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.605+0000 7f66e232b700 1 -- start start 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.605+0000 7f66e232b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc10c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.605+0000 7f66e232b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc10cb40 con 0x7f66dc07cbc0 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.605+0000 7f66dbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc10c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.605+0000 7f66dbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc10c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42342/0 (socket says 172.21.15.27:42342) 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.605+0000 7f66dbfff700 1 -- 172.21.15.27:0/2903753487 learned_addr learned my addr 172.21.15.27:0/2903753487 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.606+0000 7f66dbfff700 1 -- 172.21.15.27:0/2903753487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66dc10ccc0 con 0x7f66dc07cbc0 2024-04-25T13:31:34.648 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.606+0000 7f66dbfff700 1 --2- 172.21.15.27:0/2903753487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc10c570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66c40098a0 tx=0x7f66c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212cf55198884363 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.606+0000 7f66daffd700 1 -- 172.21.15.27:0/2903753487 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66c4004160 con 0x7f66dc07cbc0 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.606+0000 7f66daffd700 1 -- 172.21.15.27:0/2903753487 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66c4004300 con 0x7f66dc07cbc0 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.606+0000 7f66daffd700 1 -- 172.21.15.27:0/2903753487 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66c4011a20 con 0x7f66dc07cbc0 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.607+0000 7f66e232b700 1 -- 172.21.15.27:0/2903753487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 msgr2=0x7f66dc10c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.607+0000 7f66e232b700 1 --2- 172.21.15.27:0/2903753487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc10c570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66c40098a0 tx=0x7f66c4009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.607+0000 7f66e232b700 1 -- 172.21.15.27:0/2903753487 shutdown_connections 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.607+0000 7f66e232b700 1 --2- 172.21.15.27:0/2903753487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc10c570 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.607+0000 7f66e232b700 1 -- 172.21.15.27:0/2903753487 >> 172.21.15.27:0/2903753487 conn(0x7f66dc103f80 msgr2=0x7f66dc1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.607+0000 7f66e232b700 1 -- 172.21.15.27:0/2903753487 shutdown_connections 2024-04-25T13:31:34.649 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.607+0000 7f66e232b700 1 -- 172.21.15.27:0/2903753487 wait complete. 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.608+0000 7f66e232b700 1 Processor -- start 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.608+0000 7f66e232b700 1 -- start start 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.608+0000 7f66e232b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc19dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.608+0000 7f66e232b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc10dd20 con 0x7f66dc07cbc0 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.608+0000 7f66dbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc19dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.608+0000 7f66dbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc19dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42348/0 (socket says 172.21.15.27:42348) 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.608+0000 7f66dbfff700 1 -- 172.21.15.27:0/112361417 learned_addr learned my addr 172.21.15.27:0/112361417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.609+0000 7f66dbfff700 1 -- 172.21.15.27:0/112361417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66dc19e200 con 0x7f66dc07cbc0 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.609+0000 7f66dbfff700 1 --2- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc19dcc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66c4000c00 tx=0x7f66c4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.609+0000 7f66d97fa700 1 -- 172.21.15.27:0/112361417 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66c4003aa0 con 0x7f66dc07cbc0 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.609+0000 7f66d97fa700 1 -- 172.21.15.27:0/112361417 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66c401b8f0 con 0x7f66dc07cbc0 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.609+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66dc19e490 con 0x7f66dc07cbc0 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.609+0000 7f66d97fa700 1 -- 172.21.15.27:0/112361417 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66c4023890 con 0x7f66dc07cbc0 2024-04-25T13:31:34.650 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.609+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66dc1a10a0 con 0x7f66dc07cbc0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.610+0000 7f66d97fa700 1 -- 172.21.15.27:0/112361417 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f66c402c3f0 con 0x7f66dc07cbc0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.610+0000 7f66d97fa700 1 --2- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66c803d5c0 0x7f66c803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.610+0000 7f66d97fa700 1 -- 172.21.15.27:0/112361417 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f66c4053d30 con 0x7f66dc07cbc0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.610+0000 7f66db7fe700 1 --2- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66c803d5c0 0x7f66c803fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.610+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66dc064cf0 con 0x7f66dc07cbc0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.611+0000 7f66db7fe700 1 --2- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66c803d5c0 0x7f66c803fad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66cc006fd0 tx=0x7f66cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.616+0000 7f66d97fa700 1 -- 172.21.15.27:0/112361417 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f66c401f030 con 0x7f66dc07cbc0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:33.844+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f66dc105fd0 con 0x7f66c803d5c0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.092+0000 7f66d97fa700 1 -- 172.21.15.27:0/112361417 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f66dc105fd0 con 0x7f66c803d5c0 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66c803d5c0 msgr2=0x7f66c803fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 --2- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66c803d5c0 0x7f66c803fad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66cc006fd0 tx=0x7f66cc006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 msgr2=0x7f66dc19dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 --2- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc19dcc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66c4000c00 tx=0x7f66c4011aa0 comp rx=0 tx=0).stop 2024-04-25T13:31:34.651 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 shutdown_connections 2024-04-25T13:31:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 --2- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66c803d5c0 0x7f66c803fad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 --2- 172.21.15.27:0/112361417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc07cbc0 0x7f66dc19dcc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 >> 172.21.15.27:0/112361417 conn(0x7f66dc103f80 msgr2=0x7f66dc1058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 shutdown_connections 2024-04-25T13:31:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:34.094+0000 7f66e232b700 1 -- 172.21.15.27:0/112361417 wait complete. 2024-04-25T13:31:34.652 INFO:teuthology.orchestra.run.smithi027.stdout:Fetching dashboard port number... 2024-04-25T13:31:35.018 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:35.018 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:34 smithi027 ceph-mon[29140]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:35.018 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:36.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:35 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/322890439' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T13:31:36.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T13:31:36.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.134+0000 7f6fed1ee700 1 Processor -- start 2024-04-25T13:31:36.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.134+0000 7f6fed1ee700 1 -- start start 2024-04-25T13:31:36.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.134+0000 7f6fed1ee700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe810c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:36.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.134+0000 7f6fed1ee700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe810cb70 con 0x7f6fe807cbc0 2024-04-25T13:31:36.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.134+0000 7f6fe6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe810c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:36.400 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.135+0000 7f6fe6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe810c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43554/0 (socket says 172.21.15.27:43554) 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.135+0000 7f6fe6d9d700 1 -- 172.21.15.27:0/1139383421 learned_addr learned my addr 172.21.15.27:0/1139383421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.135+0000 7f6fe6d9d700 1 -- 172.21.15.27:0/1139383421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe810ccf0 con 0x7f6fe807cbc0 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.135+0000 7f6fe6d9d700 1 --2- 172.21.15.27:0/1139383421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe810c5a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00098a0 tx=0x7f6fe0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e4e7106e6aade0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.135+0000 7f6fe5d9b700 1 -- 172.21.15.27:0/1139383421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fe0004160 con 0x7f6fe807cbc0 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.135+0000 7f6fe5d9b700 1 -- 172.21.15.27:0/1139383421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fe0004300 con 0x7f6fe807cbc0 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.135+0000 7f6fe5d9b700 1 -- 172.21.15.27:0/1139383421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fe0011a20 con 0x7f6fe807cbc0 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.136+0000 7f6fed1ee700 1 -- 172.21.15.27:0/1139383421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 msgr2=0x7f6fe810c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.136+0000 7f6fed1ee700 1 --2- 172.21.15.27:0/1139383421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe810c5a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00098a0 tx=0x7f6fe0009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.136+0000 7f6fed1ee700 1 -- 172.21.15.27:0/1139383421 shutdown_connections 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.136+0000 7f6fed1ee700 1 --2- 172.21.15.27:0/1139383421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe810c5a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.136+0000 7f6fed1ee700 1 -- 172.21.15.27:0/1139383421 >> 172.21.15.27:0/1139383421 conn(0x7f6fe8103fb0 msgr2=0x7f6fe8106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.136+0000 7f6fed1ee700 1 -- 172.21.15.27:0/1139383421 shutdown_connections 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.136+0000 7f6fed1ee700 1 -- 172.21.15.27:0/1139383421 wait complete. 2024-04-25T13:31:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.137+0000 7f6fed1ee700 1 Processor -- start 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.137+0000 7f6fed1ee700 1 -- start start 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.137+0000 7f6fed1ee700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe8109d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.137+0000 7f6fed1ee700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe810dd50 con 0x7f6fe807cbc0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.137+0000 7f6fe6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe8109d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.137+0000 7f6fe6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe8109d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43558/0 (socket says 172.21.15.27:43558) 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.137+0000 7f6fe6d9d700 1 -- 172.21.15.27:0/322890439 learned_addr learned my addr 172.21.15.27:0/322890439 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.138+0000 7f6fe6d9d700 1 -- 172.21.15.27:0/322890439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe810a250 con 0x7f6fe807cbc0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.138+0000 7f6fe6d9d700 1 --2- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe8109d10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0000c00 tx=0x7f6fe0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.138+0000 7f6fdffff700 1 -- 172.21.15.27:0/322890439 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fe0003aa0 con 0x7f6fe807cbc0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.138+0000 7f6fdffff700 1 -- 172.21.15.27:0/322890439 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fe001b8f0 con 0x7f6fe807cbc0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.138+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe81084a0 con 0x7f6fe807cbc0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.138+0000 7f6fdffff700 1 -- 172.21.15.27:0/322890439 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fe0023890 con 0x7f6fe807cbc0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.138+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe8108980 con 0x7f6fe807cbc0 2024-04-25T13:31:36.402 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.139+0000 7f6fdffff700 1 -- 172.21.15.27:0/322890439 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f6fe002c3f0 con 0x7f6fe807cbc0 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.139+0000 7f6fdffff700 1 --2- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6fd403d5c0 0x7f6fd403fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.139+0000 7f6fdffff700 1 -- 172.21.15.27:0/322890439 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6fe0053db0 con 0x7f6fe807cbc0 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.139+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe8064cf0 con 0x7f6fe807cbc0 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.139+0000 7f6fe659c700 1 --2- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6fd403d5c0 0x7f6fd403fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.140+0000 7f6fe659c700 1 --2- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6fd403d5c0 0x7f6fd403fad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0006fd0 tx=0x7f6fd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.145+0000 7f6fdffff700 1 -- 172.21.15.27:0/322890439 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6fe001f030 con 0x7f6fe807cbc0 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.359+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f6fe8108c00 con 0x7f6fe807cbc0 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.360+0000 7f6fdffff700 1 -- 172.21.15.27:0/322890439 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f6fe8108c00 con 0x7f6fe807cbc0 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.362+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6fd403d5c0 msgr2=0x7f6fd403fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.362+0000 7f6fed1ee700 1 --2- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6fd403d5c0 0x7f6fd403fad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0006fd0 tx=0x7f6fd0006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.362+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 msgr2=0x7f6fe8109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.362+0000 7f6fed1ee700 1 --2- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe8109d10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0000c00 tx=0x7f6fe0011aa0 comp rx=0 tx=0).stop 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.362+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 shutdown_connections 2024-04-25T13:31:36.403 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.362+0000 7f6fed1ee700 1 --2- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6fd403d5c0 0x7f6fd403fad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:36.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.362+0000 7f6fed1ee700 1 --2- 172.21.15.27:0/322890439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe807cbc0 0x7f6fe8109d10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:36.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.362+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 >> 172.21.15.27:0/322890439 conn(0x7f6fe8103fb0 msgr2=0x7f6fe81058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:36.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.363+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 shutdown_connections 2024-04-25T13:31:36.404 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:35.363+0000 7f6fed1ee700 1 -- 172.21.15.27:0/322890439 wait complete. 2024-04-25T13:31:36.408 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T13:31:36.408 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-25T13:31:36.415 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T13:31:36.416 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-25T13:31:36.416 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-25T13:31:36.416 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T13:31:36.417 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph Dashboard is now available at: 2024-04-25T13:31:36.417 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:36.417 INFO:teuthology.orchestra.run.smithi027.stdout: URL: https://smithi027:8443/ 2024-04-25T13:31:36.418 INFO:teuthology.orchestra.run.smithi027.stdout: User: admin 2024-04-25T13:31:36.418 INFO:teuthology.orchestra.run.smithi027.stdout: Password: 2fbzarsnin 2024-04-25T13:31:36.418 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:36.418 INFO:teuthology.orchestra.run.smithi027.stdout:Saving cluster configuration to /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config directory 2024-04-25T13:31:36.418 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling autotune for osd_memory_target 2024-04-25T13:31:37.842 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:37 smithi027 ceph-mon[29140]: mgrmap e13: x(active, since 6s) 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.821+0000 7f4e5ff71700 1 Processor -- start 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.821+0000 7f4e5ff71700 1 -- start start 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.821+0000 7f4e5ff71700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5810e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.821+0000 7f4e5ff71700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5810ee50 con 0x7f4e5807cc20 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.822+0000 7f4e5dd0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5810e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.822+0000 7f4e5dd0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5810e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43568/0 (socket says 172.21.15.27:43568) 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.822+0000 7f4e5dd0d700 1 -- 172.21.15.27:0/3888254009 learned_addr learned my addr 172.21.15.27:0/3888254009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.822+0000 7f4e5dd0d700 1 -- 172.21.15.27:0/3888254009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5810efd0 con 0x7f4e5807cc20 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.822+0000 7f4e5dd0d700 1 --2- 172.21.15.27:0/3888254009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5810e910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e540098a0 tx=0x7f4e54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa881af0f15cc6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.822+0000 7f4e5cd0b700 1 -- 172.21.15.27:0/3888254009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e54004160 con 0x7f4e5807cc20 2024-04-25T13:31:37.850 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.822+0000 7f4e5cd0b700 1 -- 172.21.15.27:0/3888254009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e54004300 con 0x7f4e5807cc20 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.822+0000 7f4e5cd0b700 1 -- 172.21.15.27:0/3888254009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e54011a20 con 0x7f4e5807cc20 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.823+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3888254009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 msgr2=0x7f4e5810e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.823+0000 7f4e5ff71700 1 --2- 172.21.15.27:0/3888254009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5810e910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e540098a0 tx=0x7f4e54009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.823+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3888254009 shutdown_connections 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.823+0000 7f4e5ff71700 1 --2- 172.21.15.27:0/3888254009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5810e910 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.823+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3888254009 >> 172.21.15.27:0/3888254009 conn(0x7f4e58103f80 msgr2=0x7f4e581063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.823+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3888254009 shutdown_connections 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.823+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3888254009 wait complete. 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.824+0000 7f4e5ff71700 1 Processor -- start 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.824+0000 7f4e5ff71700 1 -- start start 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.824+0000 7f4e5ff71700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5819fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.824+0000 7f4e5ff71700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5810ff20 con 0x7f4e5807cc20 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.824+0000 7f4e5dd0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5819fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:37.851 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.824+0000 7f4e5dd0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5819fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43574/0 (socket says 172.21.15.27:43574) 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.824+0000 7f4e5dd0d700 1 -- 172.21.15.27:0/3128257128 learned_addr learned my addr 172.21.15.27:0/3128257128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.824+0000 7f4e5dd0d700 1 -- 172.21.15.27:0/3128257128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e581a03b0 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.825+0000 7f4e5dd0d700 1 --2- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5819fe70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e54006af0 tx=0x7f4e54011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.825+0000 7f4e4effd700 1 -- 172.21.15.27:0/3128257128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e54003b20 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.825+0000 7f4e4effd700 1 -- 172.21.15.27:0/3128257128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e5401b8f0 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.825+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e581a0640 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.825+0000 7f4e4effd700 1 -- 172.21.15.27:0/3128257128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e540238c0 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.825+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e581a3250 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.826+0000 7f4e4effd700 1 -- 172.21.15.27:0/3128257128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f4e54023a80 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.826+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e58064cf0 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.826+0000 7f4e4effd700 1 --2- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4e4403d5d0 0x7f4e4403fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.826+0000 7f4e4effd700 1 -- 172.21.15.27:0/3128257128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4e54053060 con 0x7f4e5807cc20 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.826+0000 7f4e5d50c700 1 --2- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4e4403d5d0 0x7f4e4403fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:37.852 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.827+0000 7f4e5d50c700 1 --2- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4e4403d5d0 0x7f4e4403fae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4e48006fd0 tx=0x7f4e48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.832+0000 7f4e4effd700 1 -- 172.21.15.27:0/3128257128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e5401f030 con 0x7f4e5807cc20 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:36.834+0000 7f4e4effd700 1 -- 172.21.15.27:0/3128257128 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f4e5402c3f0 con 0x7f4e5807cc20 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.040+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f4e581a3560 con 0x7f4e5807cc20 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.041+0000 7f4e4effd700 1 -- 172.21.15.27:0/3128257128 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f4e540172b0 con 0x7f4e5807cc20 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4e4403d5d0 msgr2=0x7f4e4403fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 --2- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4e4403d5d0 0x7f4e4403fae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4e48006fd0 tx=0x7f4e48006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 msgr2=0x7f4e5819fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 --2- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5819fe70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e54006af0 tx=0x7f4e54011aa0 comp rx=0 tx=0).stop 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 shutdown_connections 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 --2- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4e4403d5d0 0x7f4e4403fae0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 --2- 172.21.15.27:0/3128257128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5807cc20 0x7f4e5819fe70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 >> 172.21.15.27:0/3128257128 conn(0x7f4e58103f80 msgr2=0x7f4e58105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 shutdown_connections 2024-04-25T13:31:37.853 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:37.043+0000 7f4e5ff71700 1 -- 172.21.15.27:0/3128257128 wait complete. 2024-04-25T13:31:39.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.301+0000 7ff9a5b84700 1 Processor -- start 2024-04-25T13:31:39.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.302+0000 7ff9a5b84700 1 -- start start 2024-04-25T13:31:39.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.302+0000 7ff9a5b84700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a0108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.302+0000 7ff9a5b84700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a01092a0 con 0x7ff9a01088a0 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.302+0000 7ff99f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a0108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.302+0000 7ff99f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a0108cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43578/0 (socket says 172.21.15.27:43578) 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.302+0000 7ff99f7fe700 1 -- 172.21.15.27:0/1422967362 learned_addr learned my addr 172.21.15.27:0/1422967362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.302+0000 7ff99f7fe700 1 -- 172.21.15.27:0/1422967362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a01099d0 con 0x7ff9a01088a0 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.302+0000 7ff99f7fe700 1 --2- 172.21.15.27:0/1422967362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a0108cb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9880098a0 tx=0x7ff988009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20114d779fa52f3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff99e7fc700 1 -- 172.21.15.27:0/1422967362 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff988004160 con 0x7ff9a01088a0 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff99e7fc700 1 -- 172.21.15.27:0/1422967362 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff988004300 con 0x7ff9a01088a0 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff99e7fc700 1 -- 172.21.15.27:0/1422967362 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff988011a20 con 0x7ff9a01088a0 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff9a5b84700 1 -- 172.21.15.27:0/1422967362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 msgr2=0x7ff9a0108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff9a5b84700 1 --2- 172.21.15.27:0/1422967362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a0108cb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9880098a0 tx=0x7ff988009bb0 comp rx=0 tx=0).stop 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff9a5b84700 1 -- 172.21.15.27:0/1422967362 shutdown_connections 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff9a5b84700 1 --2- 172.21.15.27:0/1422967362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a0108cb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff9a5b84700 1 -- 172.21.15.27:0/1422967362 >> 172.21.15.27:0/1422967362 conn(0x7ff9a0103f80 msgr2=0x7ff9a01063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff9a5b84700 1 -- 172.21.15.27:0/1422967362 shutdown_connections 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.303+0000 7ff9a5b84700 1 -- 172.21.15.27:0/1422967362 wait complete. 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.304+0000 7ff9a5b84700 1 Processor -- start 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.304+0000 7ff9a5b84700 1 -- start start 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.304+0000 7ff9a5b84700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a007c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.304+0000 7ff9a5b84700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a010cd80 con 0x7ff9a01088a0 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff99f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a007c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff99f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a007c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43592/0 (socket says 172.21.15.27:43592) 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff99f7fe700 1 -- 172.21.15.27:0/2751830985 learned_addr learned my addr 172.21.15.27:0/2751830985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff99f7fe700 1 -- 172.21.15.27:0/2751830985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a007ce40 con 0x7ff9a01088a0 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff99f7fe700 1 --2- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a007c900 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff988003e70 tx=0x7ff988011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff99cff9700 1 -- 172.21.15.27:0/2751830985 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff988003af0 con 0x7ff9a01088a0 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff99cff9700 1 -- 172.21.15.27:0/2751830985 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff98801a8e0 con 0x7ff9a01088a0 2024-04-25T13:31:39.200 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a01a0aa0 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff99cff9700 1 -- 172.21.15.27:0/2751830985 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff988024870 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.305+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a01a0f20 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.306+0000 7ff99cff9700 1 -- 172.21.15.27:0/2751830985 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7ff988022020 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.306+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a0064cf0 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.306+0000 7ff99cff9700 1 --2- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff98c03d670 0x7ff98c03fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.307+0000 7ff99cff9700 1 -- 172.21.15.27:0/2751830985 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff988052920 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.307+0000 7ff99effd700 1 --2- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff98c03d670 0x7ff98c03fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.307+0000 7ff99effd700 1 --2- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff98c03d670 0x7ff98c03fb80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff990006fd0 tx=0x7ff990006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.312+0000 7ff99cff9700 1 -- 172.21.15.27:0/2751830985 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff98801a430 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.634+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7ff9a019c880 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.637+0000 7ff99cff9700 1 -- 172.21.15.27:0/2751830985 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7ff98802b300 con 0x7ff9a01088a0 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff98c03d670 msgr2=0x7ff98c03fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 --2- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff98c03d670 0x7ff98c03fb80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff990006fd0 tx=0x7ff990006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:39.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 msgr2=0x7ff9a007c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 --2- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a007c900 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff988003e70 tx=0x7ff988011aa0 comp rx=0 tx=0).stop 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 shutdown_connections 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 --2- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff98c03d670 0x7ff98c03fb80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 --2- 172.21.15.27:0/2751830985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a01088a0 0x7ff9a007c900 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 >> 172.21.15.27:0/2751830985 conn(0x7ff9a0103f80 msgr2=0x7ff9a0105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 shutdown_connections 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-25T13:31:38.639+0000 7ff9a5b84700 1 -- 172.21.15.27:0/2751830985 wait complete. 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T13:31:39.202 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout: ceph telemetry on 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout:For more information see: 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:39.203 INFO:teuthology.orchestra.run.smithi027.stdout:Bootstrap complete. 2024-04-25T13:31:39.238 INFO:tasks.cephadm:Fetching config... 2024-04-25T13:31:39.238 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:31:39.238 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T13:31:39.255 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T13:31:39.256 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:31:39.256 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T13:31:39.311 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T13:31:39.311 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:31:39.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T13:31:39.376 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T13:31:39.377 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:31:39.377 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T13:31:39.432 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T13:31:39.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoDQ9BjclXvbBtDzMsXR/cQEqOazPcwXazPCUnOM2Z7aFIhCIBdXrIo8fDNqmeGRtrihaUBw+Kc4GbM5Qs6eD1FmDzncWoRgWi9ZpTU1Dzc64GVJzwGnMyJx3XP+BKr+N386F05nqO1Kk/XdVbnohYXZXY1DRZ2E3P5hmvDKbBZmbXUWPE1bn7fCfiBX5GnQnRuh0hmaoAuwGoYyiC7bcjVwuq27ROL/6Gdzaeg9HAJgF4FhHqLaDeRK70T+UtyKLs/rXwEhQJqeoPkIkcj71YQOe4c9ILtDKOgrGNhgqg4Zev1rjS1L7+Mz85yw6JRaacuXKOZKiVWG9YsDjqZiPoxlr9/3OYERX6XE0GY/vvb7rkubCNEUoJI9+qubqx72psGBuJLHi0+PLa34dwcazM1U67Y7uBO98SKJyjksLzs6o5fZVqgYGWEAY4g1okcEEcBhRwP4+33//m+f/F3JAtN9pmIPHUwdBs6/rJrfxMVsuNPqvhLVN8WO+tXTe20tk= ceph-db01fc44-0307-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T13:31:39.505 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoDQ9BjclXvbBtDzMsXR/cQEqOazPcwXazPCUnOM2Z7aFIhCIBdXrIo8fDNqmeGRtrihaUBw+Kc4GbM5Qs6eD1FmDzncWoRgWi9ZpTU1Dzc64GVJzwGnMyJx3XP+BKr+N386F05nqO1Kk/XdVbnohYXZXY1DRZ2E3P5hmvDKbBZmbXUWPE1bn7fCfiBX5GnQnRuh0hmaoAuwGoYyiC7bcjVwuq27ROL/6Gdzaeg9HAJgF4FhHqLaDeRK70T+UtyKLs/rXwEhQJqeoPkIkcj71YQOe4c9ILtDKOgrGNhgqg4Zev1rjS1L7+Mz85yw6JRaacuXKOZKiVWG9YsDjqZiPoxlr9/3OYERX6XE0GY/vvb7rkubCNEUoJI9+qubqx72psGBuJLHi0+PLa34dwcazM1U67Y7uBO98SKJyjksLzs6o5fZVqgYGWEAY4g1okcEEcBhRwP4+33//m+f/F3JAtN9pmIPHUwdBs6/rJrfxMVsuNPqvhLVN8WO+tXTe20tk= ceph-db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:31:39.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoDQ9BjclXvbBtDzMsXR/cQEqOazPcwXazPCUnOM2Z7aFIhCIBdXrIo8fDNqmeGRtrihaUBw+Kc4GbM5Qs6eD1FmDzncWoRgWi9ZpTU1Dzc64GVJzwGnMyJx3XP+BKr+N386F05nqO1Kk/XdVbnohYXZXY1DRZ2E3P5hmvDKbBZmbXUWPE1bn7fCfiBX5GnQnRuh0hmaoAuwGoYyiC7bcjVwuq27ROL/6Gdzaeg9HAJgF4FhHqLaDeRK70T+UtyKLs/rXwEhQJqeoPkIkcj71YQOe4c9ILtDKOgrGNhgqg4Zev1rjS1L7+Mz85yw6JRaacuXKOZKiVWG9YsDjqZiPoxlr9/3OYERX6XE0GY/vvb7rkubCNEUoJI9+qubqx72psGBuJLHi0+PLa34dwcazM1U67Y7uBO98SKJyjksLzs6o5fZVqgYGWEAY4g1okcEEcBhRwP4+33//m+f/F3JAtN9pmIPHUwdBs6/rJrfxMVsuNPqvhLVN8WO+tXTe20tk= ceph-db01fc44-0307-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T13:31:39.557 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoDQ9BjclXvbBtDzMsXR/cQEqOazPcwXazPCUnOM2Z7aFIhCIBdXrIo8fDNqmeGRtrihaUBw+Kc4GbM5Qs6eD1FmDzncWoRgWi9ZpTU1Dzc64GVJzwGnMyJx3XP+BKr+N386F05nqO1Kk/XdVbnohYXZXY1DRZ2E3P5hmvDKbBZmbXUWPE1bn7fCfiBX5GnQnRuh0hmaoAuwGoYyiC7bcjVwuq27ROL/6Gdzaeg9HAJgF4FhHqLaDeRK70T+UtyKLs/rXwEhQJqeoPkIkcj71YQOe4c9ILtDKOgrGNhgqg4Zev1rjS1L7+Mz85yw6JRaacuXKOZKiVWG9YsDjqZiPoxlr9/3OYERX6XE0GY/vvb7rkubCNEUoJI9+qubqx72psGBuJLHi0+PLa34dwcazM1U67Y7uBO98SKJyjksLzs6o5fZVqgYGWEAY4g1okcEEcBhRwP4+33//m+f/F3JAtN9pmIPHUwdBs6/rJrfxMVsuNPqvhLVN8WO+tXTe20tk= ceph-db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:31:39.569 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoDQ9BjclXvbBtDzMsXR/cQEqOazPcwXazPCUnOM2Z7aFIhCIBdXrIo8fDNqmeGRtrihaUBw+Kc4GbM5Qs6eD1FmDzncWoRgWi9ZpTU1Dzc64GVJzwGnMyJx3XP+BKr+N386F05nqO1Kk/XdVbnohYXZXY1DRZ2E3P5hmvDKbBZmbXUWPE1bn7fCfiBX5GnQnRuh0hmaoAuwGoYyiC7bcjVwuq27ROL/6Gdzaeg9HAJgF4FhHqLaDeRK70T+UtyKLs/rXwEhQJqeoPkIkcj71YQOe4c9ILtDKOgrGNhgqg4Zev1rjS1L7+Mz85yw6JRaacuXKOZKiVWG9YsDjqZiPoxlr9/3OYERX6XE0GY/vvb7rkubCNEUoJI9+qubqx72psGBuJLHi0+PLa34dwcazM1U67Y7uBO98SKJyjksLzs6o5fZVqgYGWEAY4g1okcEEcBhRwP4+33//m+f/F3JAtN9pmIPHUwdBs6/rJrfxMVsuNPqvhLVN8WO+tXTe20tk= ceph-db01fc44-0307-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T13:31:39.607 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoDQ9BjclXvbBtDzMsXR/cQEqOazPcwXazPCUnOM2Z7aFIhCIBdXrIo8fDNqmeGRtrihaUBw+Kc4GbM5Qs6eD1FmDzncWoRgWi9ZpTU1Dzc64GVJzwGnMyJx3XP+BKr+N386F05nqO1Kk/XdVbnohYXZXY1DRZ2E3P5hmvDKbBZmbXUWPE1bn7fCfiBX5GnQnRuh0hmaoAuwGoYyiC7bcjVwuq27ROL/6Gdzaeg9HAJgF4FhHqLaDeRK70T+UtyKLs/rXwEhQJqeoPkIkcj71YQOe4c9ILtDKOgrGNhgqg4Zev1rjS1L7+Mz85yw6JRaacuXKOZKiVWG9YsDjqZiPoxlr9/3OYERX6XE0GY/vvb7rkubCNEUoJI9+qubqx72psGBuJLHi0+PLa34dwcazM1U67Y7uBO98SKJyjksLzs6o5fZVqgYGWEAY4g1okcEEcBhRwP4+33//m+f/F3JAtN9pmIPHUwdBs6/rJrfxMVsuNPqvhLVN8WO+tXTe20tk= ceph-db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:31:39.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T13:31:39.874 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:31:39.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:39 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2751830985' entity='client.admin' 2024-04-25T13:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.160+0000 7f33f539f700 1 -- 172.21.15.27:0/2632313658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 msgr2=0x7f33f0103cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.160+0000 7f33f539f700 1 --2- 172.21.15.27:0/2632313658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 0x7f33f0103cd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009910 tx=0x7f33e0009c20 comp rx=0 tx=0).stop 2024-04-25T13:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.160+0000 7f33f539f700 1 -- 172.21.15.27:0/2632313658 shutdown_connections 2024-04-25T13:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.160+0000 7f33f539f700 1 --2- 172.21.15.27:0/2632313658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 0x7f33f0103cd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.160+0000 7f33f539f700 1 -- 172.21.15.27:0/2632313658 >> 172.21.15.27:0/2632313658 conn(0x7f33f00fef60 msgr2=0x7f33f01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.160+0000 7f33f539f700 1 -- 172.21.15.27:0/2632313658 shutdown_connections 2024-04-25T13:31:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.160+0000 7f33f539f700 1 -- 172.21.15.27:0/2632313658 wait complete. 2024-04-25T13:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.161+0000 7f33f539f700 1 Processor -- start 2024-04-25T13:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.161+0000 7f33f539f700 1 -- start start 2024-04-25T13:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.161+0000 7f33f539f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 0x7f33f0199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.161+0000 7f33f539f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33f0109880 con 0x7f33f01038c0 2024-04-25T13:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.161+0000 7f33effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 0x7f33f0199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.161+0000 7f33effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 0x7f33f0199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43598/0 (socket says 172.21.15.27:43598) 2024-04-25T13:31:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.161+0000 7f33effff700 1 -- 172.21.15.27:0/820852537 learned_addr learned my addr 172.21.15.27:0/820852537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:42.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.162+0000 7f33effff700 1 -- 172.21.15.27:0/820852537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f0199940 con 0x7f33f01038c0 2024-04-25T13:31:42.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.162+0000 7f33effff700 1 --2- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 0x7f33f0199400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33e00057d0 tx=0x7f33e0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.162+0000 7f33ed7fa700 1 -- 172.21.15.27:0/820852537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33e0027070 con 0x7f33f01038c0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.162+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f0199bd0 con 0x7f33f01038c0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.162+0000 7f33ed7fa700 1 -- 172.21.15.27:0/820852537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e000be10 con 0x7f33f01038c0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.162+0000 7f33ed7fa700 1 -- 172.21.15.27:0/820852537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33e0022dd0 con 0x7f33f01038c0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.162+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f019c7e0 con 0x7f33f01038c0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.163+0000 7f33ed7fa700 1 -- 172.21.15.27:0/820852537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f33e002a070 con 0x7f33f01038c0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.163+0000 7f33ed7fa700 1 --2- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33d803d5c0 0x7f33d803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.163+0000 7f33ed7fa700 1 -- 172.21.15.27:0/820852537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f33e001e030 con 0x7f33f01038c0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.163+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33f010a000 con 0x7f33f01038c0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.163+0000 7f33ef7fe700 1 --2- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33d803d5c0 0x7f33d803fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.164+0000 7f33ef7fe700 1 --2- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33d803d5c0 0x7f33d803fad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f33e4006fd0 tx=0x7f33e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:42.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.169+0000 7f33ed7fa700 1 -- 172.21.15.27:0/820852537 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33e0013070 con 0x7f33f01038c0 2024-04-25T13:31:42.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.378+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f33f002e580 con 0x7f33f01038c0 2024-04-25T13:31:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.381+0000 7f33ed7fa700 1 -- 172.21.15.27:0/820852537 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f33e00350c0 con 0x7f33f01038c0 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33d803d5c0 msgr2=0x7f33d803fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 --2- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33d803d5c0 0x7f33d803fad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f33e4006fd0 tx=0x7f33e4006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 msgr2=0x7f33f0199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 --2- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 0x7f33f0199400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33e00057d0 tx=0x7f33e0005800 comp rx=0 tx=0).stop 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 shutdown_connections 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 --2- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33d803d5c0 0x7f33d803fad0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 --2- 172.21.15.27:0/820852537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33f01038c0 0x7f33f0199400 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 >> 172.21.15.27:0/820852537 conn(0x7f33f00fef60 msgr2=0x7f33f00ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 shutdown_connections 2024-04-25T13:31:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:42.384+0000 7f33f539f700 1 -- 172.21.15.27:0/820852537 wait complete. 2024-04-25T13:31:42.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:42.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:42.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:31:42.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:42.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:42.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:43.000 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T13:31:43.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T13:31:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:31:43.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:43 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/820852537' entity='client.admin' 2024-04-25T13:31:43.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:43.649 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.773+0000 7f4f1c357700 1 -- 172.21.15.27:0/1971475656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 msgr2=0x7f4f14102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.773+0000 7f4f1c357700 1 --2- 172.21.15.27:0/1971475656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 0x7f4f14102dc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4f10009910 tx=0x7f4f10009c20 comp rx=0 tx=0).stop 2024-04-25T13:31:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.773+0000 7f4f1c357700 1 -- 172.21.15.27:0/1971475656 shutdown_connections 2024-04-25T13:31:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.773+0000 7f4f1c357700 1 --2- 172.21.15.27:0/1971475656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 0x7f4f14102dc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.773+0000 7f4f1c357700 1 -- 172.21.15.27:0/1971475656 >> 172.21.15.27:0/1971475656 conn(0x7f4f140fe010 msgr2=0x7f4f141004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.773+0000 7f4f1c357700 1 -- 172.21.15.27:0/1971475656 shutdown_connections 2024-04-25T13:31:44.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.773+0000 7f4f1c357700 1 -- 172.21.15.27:0/1971475656 wait complete. 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.774+0000 7f4f1c357700 1 Processor -- start 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.774+0000 7f4f1c357700 1 -- start start 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.774+0000 7f4f1c357700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 0x7f4f1406c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.774+0000 7f4f1c357700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4f14108970 con 0x7f4f141029b0 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.774+0000 7f4f1a0f3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 0x7f4f1406c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.774+0000 7f4f1a0f3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 0x7f4f1406c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43254/0 (socket says 172.21.15.27:43254) 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.774+0000 7f4f1a0f3700 1 -- 172.21.15.27:0/3737338481 learned_addr learned my addr 172.21.15.27:0/3737338481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.775+0000 7f4f1a0f3700 1 -- 172.21.15.27:0/3737338481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1406c930 con 0x7f4f141029b0 2024-04-25T13:31:44.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.775+0000 7f4f1a0f3700 1 --2- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 0x7f4f1406c3f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f100057d0 tx=0x7f4f10005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:44.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.775+0000 7f4f0b7fe700 1 -- 172.21.15.27:0/3737338481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f10027070 con 0x7f4f141029b0 2024-04-25T13:31:44.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.775+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f1419c320 con 0x7f4f141029b0 2024-04-25T13:31:44.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.775+0000 7f4f0b7fe700 1 -- 172.21.15.27:0/3737338481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f1000be10 con 0x7f4f141029b0 2024-04-25T13:31:44.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.775+0000 7f4f0b7fe700 1 -- 172.21.15.27:0/3737338481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f10022dd0 con 0x7f4f141029b0 2024-04-25T13:31:44.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.775+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f1419c800 con 0x7f4f141029b0 2024-04-25T13:31:44.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.776+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f141090f0 con 0x7f4f141029b0 2024-04-25T13:31:44.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.776+0000 7f4f0b7fe700 1 -- 172.21.15.27:0/3737338481 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f4f1002a070 con 0x7f4f141029b0 2024-04-25T13:31:44.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.776+0000 7f4f0b7fe700 1 --2- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f0003d5c0 0x7f4f0003fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:44.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.776+0000 7f4f0b7fe700 1 -- 172.21.15.27:0/3737338481 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4f1001e030 con 0x7f4f141029b0 2024-04-25T13:31:44.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.777+0000 7f4f198f2700 1 --2- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f0003d5c0 0x7f4f0003fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:44.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.777+0000 7f4f198f2700 1 --2- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f0003d5c0 0x7f4f0003fad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4f04006fd0 tx=0x7f4f04006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:44.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.782+0000 7f4f0b7fe700 1 -- 172.21.15.27:0/3737338481 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f10013070 con 0x7f4f141029b0 2024-04-25T13:31:44.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.992+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f4f1402e740 con 0x7f4f0003d5c0 2024-04-25T13:31:44.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.996+0000 7f4f0b7fe700 1 -- 172.21.15.27:0/3737338481 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4f1402e740 con 0x7f4f0003d5c0 2024-04-25T13:31:44.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f0003d5c0 msgr2=0x7f4f0003fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:44.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 --2- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f0003d5c0 0x7f4f0003fad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4f04006fd0 tx=0x7f4f04006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:45.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 msgr2=0x7f4f1406c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:45.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 --2- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 0x7f4f1406c3f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f100057d0 tx=0x7f4f10005800 comp rx=0 tx=0).stop 2024-04-25T13:31:45.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 shutdown_connections 2024-04-25T13:31:45.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 --2- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f0003d5c0 0x7f4f0003fad0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:45.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 --2- 172.21.15.27:0/3737338481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f141029b0 0x7f4f1406c3f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:45.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 >> 172.21.15.27:0/3737338481 conn(0x7f4f140fe010 msgr2=0x7f4f140fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:45.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 shutdown_connections 2024-04-25T13:31:45.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:44.999+0000 7f4f1c357700 1 -- 172.21.15.27:0/3737338481 wait complete. 2024-04-25T13:31:45.778 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi045 2024-04-25T13:31:45.778 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:31:45.778 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/etc/ceph/ceph.conf 2024-04-25T13:31:45.799 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:31:45.799 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:31:45.856 INFO:tasks.cephadm:Adding host smithi045 to orchestrator... 2024-04-25T13:31:45.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch host add smithi045 2024-04-25T13:31:46.100 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:31:46.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:46.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:46.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:46.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:31:46.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:31:46.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T13:31:46.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:46.336 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:46.337 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:47.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:46 smithi027 ceph-mon[29140]: Updating smithi027:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:31:47.276 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:46 smithi027 ceph-mon[29140]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:31:47.277 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:46 smithi027 ceph-mon[29140]: Updating smithi027:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T13:31:47.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.631+0000 7f7a08306700 1 -- 172.21.15.27:0/189165398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 msgr2=0x7f7a00109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.631+0000 7f7a08306700 1 --2- 172.21.15.27:0/189165398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 0x7f7a00109e20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f79f8009910 tx=0x7f79f8009c20 comp rx=0 tx=0).stop 2024-04-25T13:31:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.631+0000 7f7a08306700 1 -- 172.21.15.27:0/189165398 shutdown_connections 2024-04-25T13:31:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.631+0000 7f7a08306700 1 --2- 172.21.15.27:0/189165398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 0x7f7a00109e20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.631+0000 7f7a08306700 1 -- 172.21.15.27:0/189165398 >> 172.21.15.27:0/189165398 conn(0x7f7a000ff560 msgr2=0x7f7a001019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.631+0000 7f7a08306700 1 -- 172.21.15.27:0/189165398 shutdown_connections 2024-04-25T13:31:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.631+0000 7f7a08306700 1 -- 172.21.15.27:0/189165398 wait complete. 2024-04-25T13:31:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.632+0000 7f7a08306700 1 Processor -- start 2024-04-25T13:31:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.632+0000 7f7a08306700 1 -- start start 2024-04-25T13:31:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.632+0000 7f7a08306700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 0x7f7a001997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.632+0000 7f7a08306700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0010df60 con 0x7f7a00109a10 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f7a060a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 0x7f7a001997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f7a060a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 0x7f7a001997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43266/0 (socket says 172.21.15.27:43266) 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f7a060a2700 1 -- 172.21.15.27:0/675914983 learned_addr learned my addr 172.21.15.27:0/675914983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f7a060a2700 1 -- 172.21.15.27:0/675914983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a00199d20 con 0x7f7a00109a10 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f7a060a2700 1 --2- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 0x7f7a001997e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f79f80056d0 tx=0x7f79f80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f79ff7fe700 1 -- 172.21.15.27:0/675914983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79f8027070 con 0x7f7a00109a10 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f79ff7fe700 1 -- 172.21.15.27:0/675914983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f800be10 con 0x7f7a00109a10 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a0019a010 con 0x7f7a00109a10 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f79ff7fe700 1 -- 172.21.15.27:0/675914983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79f8022d20 con 0x7f7a00109a10 2024-04-25T13:31:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.633+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a0019e010 con 0x7f7a00109a10 2024-04-25T13:31:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.634+0000 7f79ff7fe700 1 -- 172.21.15.27:0/675914983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f79f802a070 con 0x7f7a00109a10 2024-04-25T13:31:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.634+0000 7f79ff7fe700 1 --2- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79e803d610 0x7f79e803fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.634+0000 7f79ff7fe700 1 -- 172.21.15.27:0/675914983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f79f801e030 con 0x7f7a00109a10 2024-04-25T13:31:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.635+0000 7f7a058a1700 1 --2- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79e803d610 0x7f79e803fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.635+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a00105120 con 0x7f7a00109a10 2024-04-25T13:31:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.635+0000 7f7a058a1700 1 --2- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79e803d610 0x7f79e803fb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f79f4006fd0 tx=0x7f79f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.640+0000 7f79ff7fe700 1 -- 172.21.15.27:0/675914983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79f8013070 con 0x7f7a00109a10 2024-04-25T13:31:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:47.854+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi045", "target": ["mon-mgr", ""]}) -- 0x7f7a0019ab40 con 0x7f79e803d610 2024-04-25T13:31:48.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:48 smithi027 ceph-mon[29140]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:49.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:49 smithi027 ceph-mon[29140]: Deploying cephadm binary to smithi045 2024-04-25T13:31:51.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:51 smithi027 ceph-mon[29140]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:31:52.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.168+0000 7f79ff7fe700 1 -- 172.21.15.27:0/675914983 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f7a0019ab40 con 0x7f79e803d610 2024-04-25T13:31:52.169 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi045' with addr '172.21.15.45' 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79e803d610 msgr2=0x7f79e803fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 --2- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79e803d610 0x7f79e803fb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f79f4006fd0 tx=0x7f79f4006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 msgr2=0x7f7a001997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 --2- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 0x7f7a001997e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f79f80056d0 tx=0x7f79f80057b0 comp rx=0 tx=0).stop 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 shutdown_connections 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 --2- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79e803d610 0x7f79e803fb20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 --2- 172.21.15.27:0/675914983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a00109a10 0x7f7a001997e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 >> 172.21.15.27:0/675914983 conn(0x7f7a000ff560 msgr2=0x7f7a00101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 shutdown_connections 2024-04-25T13:31:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:52.171+0000 7f7a08306700 1 -- 172.21.15.27:0/675914983 wait complete. 2024-04-25T13:31:52.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T13:31:53.031 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:31:53.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:53.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:53 smithi027 ceph-mon[29140]: Added host smithi045 2024-04-25T13:31:53.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:31:53.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:53 smithi027 ceph-mon[29140]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:31:53.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:54.342 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:54 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 -- 172.21.15.27:0/3382950021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 msgr2=0x7fd2a4102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 --2- 172.21.15.27:0/3382950021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 0x7fd2a4102730 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd29800aa20 tx=0x7fd29800ad30 comp rx=0 tx=0).stop 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 -- 172.21.15.27:0/3382950021 shutdown_connections 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 --2- 172.21.15.27:0/3382950021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 0x7fd2a4102730 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 -- 172.21.15.27:0/3382950021 >> 172.21.15.27:0/3382950021 conn(0x7fd2a40fdd00 msgr2=0x7fd2a4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 -- 172.21.15.27:0/3382950021 shutdown_connections 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 -- 172.21.15.27:0/3382950021 wait complete. 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 Processor -- start 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.580+0000 7fd2aaad6700 1 -- start start 2024-04-25T13:31:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2aaad6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 0x7fd2a4107f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:54.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2aaad6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a410bd70 con 0x7fd2a4102320 2024-04-25T13:31:54.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2a8872700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 0x7fd2a4107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:54.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2a8872700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 0x7fd2a4107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37822/0 (socket says 172.21.15.27:37822) 2024-04-25T13:31:54.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2a8872700 1 -- 172.21.15.27:0/1116982615 learned_addr learned my addr 172.21.15.27:0/1116982615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:54.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2a8872700 1 -- 172.21.15.27:0/1116982615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a4109d70 con 0x7fd2a4102320 2024-04-25T13:31:54.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2a8872700 1 --2- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 0x7fd2a4107f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd298006010 tx=0x7fd298005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:54.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2a1ffb700 1 -- 172.21.15.27:0/1116982615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd298005400 con 0x7fd2a4102320 2024-04-25T13:31:54.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.581+0000 7fd2a1ffb700 1 -- 172.21.15.27:0/1116982615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2980055a0 con 0x7fd2a4102320 2024-04-25T13:31:54.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.582+0000 7fd2a1ffb700 1 -- 172.21.15.27:0/1116982615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd29800e700 con 0x7fd2a4102320 2024-04-25T13:31:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.582+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a4108460 con 0x7fd2a4102320 2024-04-25T13:31:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.582+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a41088c0 con 0x7fd2a4102320 2024-04-25T13:31:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.583+0000 7fd2a1ffb700 1 -- 172.21.15.27:0/1116982615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fd29800e8a0 con 0x7fd2a4102320 2024-04-25T13:31:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.583+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a41037d0 con 0x7fd2a4102320 2024-04-25T13:31:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.583+0000 7fd2a1ffb700 1 --2- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd29403d670 0x7fd29403fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.583+0000 7fd2a1ffb700 1 -- 172.21.15.27:0/1116982615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd29801e070 con 0x7fd2a4102320 2024-04-25T13:31:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.583+0000 7fd2a3fff700 1 --2- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd29403d670 0x7fd29403fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.583+0000 7fd2a3fff700 1 --2- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd29403d670 0x7fd29403fb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd290006fd0 tx=0x7fd290006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:54.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.588+0000 7fd2a1ffb700 1 -- 172.21.15.27:0/1116982615 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd29800d3e0 con 0x7fd2a4102320 2024-04-25T13:31:54.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.800+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2a4109450 con 0x7fd29403d670 2024-04-25T13:31:54.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.802+0000 7fd2a1ffb700 1 -- 172.21.15.27:0/1116982615 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7fd2a4109450 con 0x7fd29403d670 2024-04-25T13:31:54.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:31:54.802 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}] 2024-04-25T13:31:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd29403d670 msgr2=0x7fd29403fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 --2- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd29403d670 0x7fd29403fb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd290006fd0 tx=0x7fd290006e40 comp rx=0 tx=0).stop 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 msgr2=0x7fd2a4107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 --2- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 0x7fd2a4107f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd298006010 tx=0x7fd298005050 comp rx=0 tx=0).stop 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 shutdown_connections 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 --2- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd29403d670 0x7fd29403fb80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 --2- 172.21.15.27:0/1116982615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2a4102320 0x7fd2a4107f20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 >> 172.21.15.27:0/1116982615 conn(0x7fd2a40fdd00 msgr2=0x7fd2a4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 shutdown_connections 2024-04-25T13:31:54.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:54.804+0000 7fd2aaad6700 1 -- 172.21.15.27:0/1116982615 wait complete. 2024-04-25T13:31:55.415 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2024-04-25T13:31:55.415 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:31:55.415 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2024-04-25T13:31:55.434 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:31:55.434 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:31:55.492 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2024-04-25T13:31:55.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch host add smithi122 2024-04-25T13:31:55.738 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:31:56.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:55 smithi027 ceph-mon[29140]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:31:56.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:55 smithi027 ceph-mon[29140]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:31:56.131 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:57.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.201+0000 7f5f74c5f700 1 -- 172.21.15.27:0/2801540297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 msgr2=0x7f5f70108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:31:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.201+0000 7f5f74c5f700 1 --2- 172.21.15.27:0/2801540297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 0x7f5f70108d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5f60009910 tx=0x7f5f60009c20 comp rx=0 tx=0).stop 2024-04-25T13:31:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.202+0000 7f5f74c5f700 1 -- 172.21.15.27:0/2801540297 shutdown_connections 2024-04-25T13:31:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.202+0000 7f5f74c5f700 1 --2- 172.21.15.27:0/2801540297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 0x7f5f70108d20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:31:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.202+0000 7f5f74c5f700 1 -- 172.21.15.27:0/2801540297 >> 172.21.15.27:0/2801540297 conn(0x7f5f700fe420 msgr2=0x7f5f70100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:31:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.202+0000 7f5f74c5f700 1 -- 172.21.15.27:0/2801540297 shutdown_connections 2024-04-25T13:31:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.202+0000 7f5f74c5f700 1 -- 172.21.15.27:0/2801540297 wait complete. 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.202+0000 7f5f74c5f700 1 Processor -- start 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.202+0000 7f5f74c5f700 1 -- start start 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f74c5f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 0x7f5f70198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f74c5f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7010ce60 con 0x7f5f70108910 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f6e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 0x7f5f70198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f6e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 0x7f5f70198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37832/0 (socket says 172.21.15.27:37832) 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f6e59c700 1 -- 172.21.15.27:0/431646060 learned_addr learned my addr 172.21.15.27:0/431646060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f6e59c700 1 -- 172.21.15.27:0/431646060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f70198c40 con 0x7f5f70108910 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f6e59c700 1 --2- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 0x7f5f70198700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5f600092e0 tx=0x7f5f60005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f5f7fe700 1 -- 172.21.15.27:0/431646060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f60026070 con 0x7f5f70108910 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f5f7fe700 1 -- 172.21.15.27:0/431646060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f6000b6f0 con 0x7f5f70108910 2024-04-25T13:31:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f70198f30 con 0x7f5f70108910 2024-04-25T13:31:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.203+0000 7f5f5f7fe700 1 -- 172.21.15.27:0/431646060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5f600037f0 con 0x7f5f70108910 2024-04-25T13:31:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.204+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f7019cf30 con 0x7f5f70108910 2024-04-25T13:31:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.204+0000 7f5f5f7fe700 1 -- 172.21.15.27:0/431646060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f5f60029070 con 0x7f5f70108910 2024-04-25T13:31:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.204+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f70104020 con 0x7f5f70108910 2024-04-25T13:31:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.205+0000 7f5f5f7fe700 1 --2- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f5803d670 0x7f5f5803fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:31:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.205+0000 7f5f5f7fe700 1 -- 172.21.15.27:0/431646060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5f6001e070 con 0x7f5f70108910 2024-04-25T13:31:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.205+0000 7f5f6dd9b700 1 --2- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f5803d670 0x7f5f5803fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:31:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.205+0000 7f5f6dd9b700 1 --2- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f5803d670 0x7f5f5803fb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5f64006fd0 tx=0x7f5f64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:31:57.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.210+0000 7f5f5f7fe700 1 -- 172.21.15.27:0/431646060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f60013070 con 0x7f5f70108910 2024-04-25T13:31:57.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:31:57.423+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}) -- 0x7f5f70199a60 con 0x7f5f5803d670 2024-04-25T13:31:58.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:57 smithi027 ceph-mon[29140]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:31:59.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:31:59.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: Deploying cephadm binary to smithi122 2024-04-25T13:31:59.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:59.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:59.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:59.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:31:59.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:31:59.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:31:59.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:00.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:59 smithi027 ceph-mon[29140]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-25T13:32:00.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:59 smithi027 ceph-mon[29140]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:00.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:59 smithi027 ceph-mon[29140]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:00.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:31:59 smithi027 ceph-mon[29140]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:01.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.691+0000 7f5f5f7fe700 1 -- 172.21.15.27:0/431646060 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f5f70199a60 con 0x7f5f5803d670 2024-04-25T13:32:01.692 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi122' with addr '172.21.15.122' 2024-04-25T13:32:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f5803d670 msgr2=0x7f5f5803fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 --2- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f5803d670 0x7f5f5803fb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5f64006fd0 tx=0x7f5f64006e40 comp rx=0 tx=0).stop 2024-04-25T13:32:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 msgr2=0x7f5f70198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 --2- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 0x7f5f70198700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5f600092e0 tx=0x7f5f60005780 comp rx=0 tx=0).stop 2024-04-25T13:32:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 shutdown_connections 2024-04-25T13:32:01.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 --2- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f5803d670 0x7f5f5803fb80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 --2- 172.21.15.27:0/431646060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f70108910 0x7f5f70198700 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 >> 172.21.15.27:0/431646060 conn(0x7f5f700fe420 msgr2=0x7f5f700ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 shutdown_connections 2024-04-25T13:32:01.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:01.693+0000 7f5f74c5f700 1 -- 172.21.15.27:0/431646060 wait complete. 2024-04-25T13:32:01.998 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:01 smithi027 ceph-mon[29140]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:02.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:02.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:02.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:02.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:02.015 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:02.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T13:32:02.496 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:32:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:02 smithi027 ceph-mon[29140]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T13:32:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:02 smithi027 ceph-mon[29140]: Added host smithi122 2024-04-25T13:32:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:03.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:03 smithi027 ceph-mon[29140]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.981+0000 7f71a33e4700 1 -- 172.21.15.27:0/4017503348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 msgr2=0x7f719c105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.981+0000 7f71a33e4700 1 --2- 172.21.15.27:0/4017503348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 0x7f719c105010 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7190009910 tx=0x7f7190009c20 comp rx=0 tx=0).stop 2024-04-25T13:32:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.981+0000 7f71a33e4700 1 -- 172.21.15.27:0/4017503348 shutdown_connections 2024-04-25T13:32:03.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.981+0000 7f71a33e4700 1 --2- 172.21.15.27:0/4017503348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 0x7f719c105010 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.981+0000 7f71a33e4700 1 -- 172.21.15.27:0/4017503348 >> 172.21.15.27:0/4017503348 conn(0x7f719c078460 msgr2=0x7f719c0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.981+0000 7f71a33e4700 1 -- 172.21.15.27:0/4017503348 shutdown_connections 2024-04-25T13:32:03.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.981+0000 7f71a33e4700 1 -- 172.21.15.27:0/4017503348 wait complete. 2024-04-25T13:32:03.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.982+0000 7f71a33e4700 1 Processor -- start 2024-04-25T13:32:03.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.982+0000 7f71a33e4700 1 -- start start 2024-04-25T13:32:03.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.982+0000 7f71a33e4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 0x7f719c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:03.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.982+0000 7f71a33e4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f719c10dff0 con 0x7f719c104c00 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.982+0000 7f71a1180700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 0x7f719c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.982+0000 7f71a1180700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 0x7f719c1997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34970/0 (socket says 172.21.15.27:34970) 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.982+0000 7f71a1180700 1 -- 172.21.15.27:0/1375081295 learned_addr learned my addr 172.21.15.27:0/1375081295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.982+0000 7f71a1180700 1 -- 172.21.15.27:0/1375081295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719c199d30 con 0x7f719c104c00 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.983+0000 7f71a1180700 1 --2- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 0x7f719c1997f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7190006010 tx=0x7f7190005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.983+0000 7f718e7fc700 1 -- 172.21.15.27:0/1375081295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7190026070 con 0x7f719c104c00 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.983+0000 7f718e7fc700 1 -- 172.21.15.27:0/1375081295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719000bc00 con 0x7f719c104c00 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.983+0000 7f718e7fc700 1 -- 172.21.15.27:0/1375081295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7190003910 con 0x7f719c104c00 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.983+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719c199fc0 con 0x7f719c104c00 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.983+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719c19a420 con 0x7f719c104c00 2024-04-25T13:32:03.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.984+0000 7f718e7fc700 1 -- 172.21.15.27:0/1375081295 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f7190029070 con 0x7f719c104c00 2024-04-25T13:32:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.984+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719c06b5a0 con 0x7f719c104c00 2024-04-25T13:32:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.984+0000 7f718e7fc700 1 --2- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f718803d670 0x7f718803fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.984+0000 7f718e7fc700 1 -- 172.21.15.27:0/1375081295 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f719001e070 con 0x7f719c104c00 2024-04-25T13:32:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.984+0000 7f71a097f700 1 --2- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f718803d670 0x7f718803fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:03.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.985+0000 7f71a097f700 1 --2- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f718803d670 0x7f718803fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7198006fd0 tx=0x7f7198006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:03.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:03.989+0000 7f718e7fc700 1 -- 172.21.15.27:0/1375081295 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7190013070 con 0x7f719c104c00 2024-04-25T13:32:04.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.204+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f719c19abc0 con 0x7f718803d670 2024-04-25T13:32:04.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.205+0000 7f718e7fc700 1 -- 172.21.15.27:0/1375081295 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f719c19abc0 con 0x7f718803d670 2024-04-25T13:32:04.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:32:04.206 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2024-04-25T13:32:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f718803d670 msgr2=0x7f718803fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 --2- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f718803d670 0x7f718803fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7198006fd0 tx=0x7f7198006e40 comp rx=0 tx=0).stop 2024-04-25T13:32:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 msgr2=0x7f719c1997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 --2- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 0x7f719c1997f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7190006010 tx=0x7f7190005780 comp rx=0 tx=0).stop 2024-04-25T13:32:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 shutdown_connections 2024-04-25T13:32:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 --2- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f718803d670 0x7f718803fb80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 --2- 172.21.15.27:0/1375081295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f719c104c00 0x7f719c1997f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 >> 172.21.15.27:0/1375081295 conn(0x7f719c078460 msgr2=0x7f719c105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 shutdown_connections 2024-04-25T13:32:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:04.207+0000 7f71a33e4700 1 -- 172.21.15.27:0/1375081295 wait complete. 2024-04-25T13:32:04.809 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T13:32:04.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T13:32:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:32:05.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:05 smithi027 ceph-mon[29140]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:32:05.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:05 smithi027 ceph-mon[29140]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:05.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.486+0000 7f532bd32700 1 -- 172.21.15.27:0/3340577034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 msgr2=0x7f5324108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.486+0000 7f532bd32700 1 --2- 172.21.15.27:0/3340577034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 0x7f5324108d40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5320009910 tx=0x7f5320009c20 comp rx=0 tx=0).stop 2024-04-25T13:32:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.486+0000 7f532bd32700 1 -- 172.21.15.27:0/3340577034 shutdown_connections 2024-04-25T13:32:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.486+0000 7f532bd32700 1 --2- 172.21.15.27:0/3340577034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 0x7f5324108d40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.486+0000 7f532bd32700 1 -- 172.21.15.27:0/3340577034 >> 172.21.15.27:0/3340577034 conn(0x7f53240fe440 msgr2=0x7f5324100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.486+0000 7f532bd32700 1 -- 172.21.15.27:0/3340577034 shutdown_connections 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.486+0000 7f532bd32700 1 -- 172.21.15.27:0/3340577034 wait complete. 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f532bd32700 1 Processor -- start 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f532bd32700 1 -- start start 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f532bd32700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 0x7f5324198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f532bd32700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f532410ce80 con 0x7f5324108930 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f5329ace700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 0x7f5324198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f5329ace700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 0x7f5324198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34998/0 (socket says 172.21.15.27:34998) 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f5329ace700 1 -- 172.21.15.27:0/103199239 learned_addr learned my addr 172.21.15.27:0/103199239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f5329ace700 1 -- 172.21.15.27:0/103199239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5324198c50 con 0x7f5324108930 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.487+0000 7f5329ace700 1 --2- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 0x7f5324198710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53200092e0 tx=0x7f5320005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.488+0000 7f531affd700 1 -- 172.21.15.27:0/103199239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5320026070 con 0x7f5324108930 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.488+0000 7f531affd700 1 -- 172.21.15.27:0/103199239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f532000b6f0 con 0x7f5324108930 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.488+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5324198f40 con 0x7f5324108930 2024-04-25T13:32:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.488+0000 7f531affd700 1 -- 172.21.15.27:0/103199239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53200037f0 con 0x7f5324108930 2024-04-25T13:32:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.488+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f532419cf40 con 0x7f5324108930 2024-04-25T13:32:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.489+0000 7f531affd700 1 -- 172.21.15.27:0/103199239 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f5320029070 con 0x7f5324108930 2024-04-25T13:32:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.489+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5324104040 con 0x7f5324108930 2024-04-25T13:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.489+0000 7f531affd700 1 --2- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f531003d610 0x7f531003fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.489+0000 7f531affd700 1 -- 172.21.15.27:0/103199239 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f532001e070 con 0x7f5324108930 2024-04-25T13:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.489+0000 7f53292cd700 1 --2- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f531003d610 0x7f531003fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.490+0000 7f53292cd700 1 --2- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f531003d610 0x7f531003fb20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5314006fd0 tx=0x7f5314006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.494+0000 7f531affd700 1 -- 172.21.15.27:0/103199239 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5320013070 con 0x7f5324108930 2024-04-25T13:32:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:06.706+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f5324199a70 con 0x7f5324108930 2024-04-25T13:32:07.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.285+0000 7f531affd700 1 -- 172.21.15.27:0/103199239 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f5320030090 con 0x7f5324108930 2024-04-25T13:32:07.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f531003d610 msgr2=0x7f531003fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:07.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 --2- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f531003d610 0x7f531003fb20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5314006fd0 tx=0x7f5314006e40 comp rx=0 tx=0).stop 2024-04-25T13:32:07.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 msgr2=0x7f5324198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:07.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 --2- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 0x7f5324198710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53200092e0 tx=0x7f5320005780 comp rx=0 tx=0).stop 2024-04-25T13:32:07.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 shutdown_connections 2024-04-25T13:32:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 --2- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f531003d610 0x7f531003fb20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 --2- 172.21.15.27:0/103199239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5324108930 0x7f5324198710 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 >> 172.21.15.27:0/103199239 conn(0x7f53240fe440 msgr2=0x7f53240ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 shutdown_connections 2024-04-25T13:32:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:07.287+0000 7f532bd32700 1 -- 172.21.15.27:0/103199239 wait complete. 2024-04-25T13:32:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-04-25T13:32:07.647 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:07 smithi027 ceph-mon[29140]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:07.648 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:07 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/103199239' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T13:32:07.951 INFO:tasks.cephadm:Adding mon.a on smithi027 2024-04-25T13:32:07.951 INFO:tasks.cephadm:Adding mon.b on smithi045 2024-04-25T13:32:07.951 INFO:tasks.cephadm:Adding mon.c on smithi122 2024-04-25T13:32:07.952 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c' 2024-04-25T13:32:08.185 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:08.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/103199239' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T13:32:08.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:08.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:08.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:08.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.522+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/2115043732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 msgr2=0x7f7cc01036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.522+0000 7f7cc5ce7700 1 --2- 172.21.15.122:0/2115043732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 0x7f7cc01036c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009910 tx=0x7f7cb0009c20 comp rx=0 tx=0).stop 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.522+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/2115043732 shutdown_connections 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.522+0000 7f7cc5ce7700 1 --2- 172.21.15.122:0/2115043732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 0x7f7cc01036c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.522+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/2115043732 >> 172.21.15.122:0/2115043732 conn(0x7f7cc00fdd00 msgr2=0x7f7cc0100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.522+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/2115043732 shutdown_connections 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.522+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/2115043732 wait complete. 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.522+0000 7f7cc5ce7700 1 Processor -- start 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.523+0000 7f7cc5ce7700 1 -- start start 2024-04-25T13:32:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.523+0000 7f7cc5ce7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 0x7f7cc019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:09.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.523+0000 7f7cc5ce7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc010d080 con 0x7f7cc01032b0 2024-04-25T13:32:09.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.523+0000 7f7cbf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 0x7f7cc019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.523+0000 7f7cbf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 0x7f7cc019a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:58704/0 (socket says 172.21.15.122:58704) 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.523+0000 7f7cbf7fe700 1 -- 172.21.15.122:0/4095167544 learned_addr learned my addr 172.21.15.122:0/4095167544 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.524+0000 7f7cbf7fe700 1 -- 172.21.15.122:0/4095167544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc0102e30 con 0x7f7cc01032b0 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.524+0000 7f7cbf7fe700 1 --2- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 0x7f7cc019a930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00056d0 tx=0x7f7cb00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.524+0000 7f7cbd7fa700 1 -- 172.21.15.122:0/4095167544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cb0026070 con 0x7f7cc01032b0 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.524+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc019af30 con 0x7f7cc01032b0 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.524+0000 7f7cbd7fa700 1 -- 172.21.15.122:0/4095167544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cb000bc00 con 0x7f7cc01032b0 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.524+0000 7f7cbd7fa700 1 -- 172.21.15.122:0/4095167544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cb0003920 con 0x7f7cc01032b0 2024-04-25T13:32:09.525 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.524+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc019b3f0 con 0x7f7cc01032b0 2024-04-25T13:32:09.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.525+0000 7f7cbd7fa700 1 -- 172.21.15.122:0/4095167544 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f7cb0029070 con 0x7f7cc01032b0 2024-04-25T13:32:09.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.525+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc01042d0 con 0x7f7cc01032b0 2024-04-25T13:32:09.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.525+0000 7f7cbd7fa700 1 --2- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7cac03d610 0x7f7cac03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:09.526 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.525+0000 7f7cbd7fa700 1 -- 172.21.15.122:0/4095167544 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7cb001e070 con 0x7f7cc01032b0 2024-04-25T13:32:09.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.526+0000 7f7cb6ffd700 1 --2- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7cac03d610 0x7f7cac03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:09.527 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.526+0000 7f7cb6ffd700 1 --2- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7cac03d610 0x7f7cac03fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8006fd0 tx=0x7f7ca8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:09.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.531+0000 7f7cbd7fa700 1 -- 172.21.15.122:0/4095167544 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7cb0013070 con 0x7f7cc01032b0 2024-04-25T13:32:09.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:09 smithi027 ceph-mon[29140]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:09.740 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.739+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}) -- 0x7f7cc006aaa0 con 0x7f7cac03d610 2024-04-25T13:32:09.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.746+0000 7f7cbd7fa700 1 -- 172.21.15.122:0/4095167544 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7cc006aaa0 con 0x7f7cac03d610 2024-04-25T13:32:09.748 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mon update... 2024-04-25T13:32:09.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7cac03d610 msgr2=0x7f7cac03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:09.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 --2- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7cac03d610 0x7f7cac03fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8006fd0 tx=0x7f7ca8006e40 comp rx=0 tx=0).stop 2024-04-25T13:32:09.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 msgr2=0x7f7cc019a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:09.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 --2- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 0x7f7cc019a930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00056d0 tx=0x7f7cb00057b0 comp rx=0 tx=0).stop 2024-04-25T13:32:09.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 shutdown_connections 2024-04-25T13:32:09.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 --2- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7cac03d610 0x7f7cac03fb20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:09.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 --2- 172.21.15.122:0/4095167544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cc01032b0 0x7f7cc019a930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:09.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 >> 172.21.15.122:0/4095167544 conn(0x7f7cc00fdd00 msgr2=0x7f7cc0109d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:09.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 shutdown_connections 2024-04-25T13:32:09.751 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:09.748+0000 7f7cc5ce7700 1 -- 172.21.15.122:0/4095167544 wait complete. 2024-04-25T13:32:10.309 DEBUG:teuthology.orchestra.run.smithi045:mon.b> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T13:32:10.312 DEBUG:teuthology.orchestra.run.smithi122:mon.c> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T13:32:10.315 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T13:32:10.316 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T13:32:11.076 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:11.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c;count:3 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:11.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:10 smithi027 ceph-mon[29140]: Deploying daemon mon.c on smithi122 2024-04-25T13:32:12.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:11 smithi027 ceph-mon[29140]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:13.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.838+0000 7f4dadd5a700 1 -- 172.21.15.122:0/1474694075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 msgr2=0x7f4da8105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:13.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.838+0000 7f4dadd5a700 1 --2- 172.21.15.122:0/1474694075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 0x7f4da8105030 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009910 tx=0x7f4d98009c20 comp rx=0 tx=0).stop 2024-04-25T13:32:13.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.838+0000 7f4dadd5a700 1 -- 172.21.15.122:0/1474694075 shutdown_connections 2024-04-25T13:32:13.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.838+0000 7f4dadd5a700 1 --2- 172.21.15.122:0/1474694075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 0x7f4da8105030 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:13.841 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.838+0000 7f4dadd5a700 1 -- 172.21.15.122:0/1474694075 >> 172.21.15.122:0/1474694075 conn(0x7f4da8078480 msgr2=0x7f4da80788e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.838+0000 7f4dadd5a700 1 -- 172.21.15.122:0/1474694075 shutdown_connections 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.839+0000 7f4dadd5a700 1 -- 172.21.15.122:0/1474694075 wait complete. 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.839+0000 7f4dadd5a700 1 Processor -- start 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.839+0000 7f4dadd5a700 1 -- start start 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.840+0000 7f4dadd5a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 0x7f4da819f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.840+0000 7f4dadd5a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4da810e010 con 0x7f4da8104c20 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.840+0000 7f4da77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 0x7f4da819f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.840+0000 7f4da77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 0x7f4da819f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:58718/0 (socket says 172.21.15.122:58718) 2024-04-25T13:32:13.842 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.840+0000 7f4da77fe700 1 -- 172.21.15.122:0/2508842562 learned_addr learned my addr 172.21.15.122:0/2508842562 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:32:13.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.841+0000 7f4da77fe700 1 -- 172.21.15.122:0/2508842562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da819fa40 con 0x7f4da8104c20 2024-04-25T13:32:13.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.841+0000 7f4da77fe700 1 --2- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 0x7f4da819f500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d980056d0 tx=0x7f4d980057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:13.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.841+0000 7f4da4ff9700 1 -- 172.21.15.122:0/2508842562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d98026070 con 0x7f4da8104c20 2024-04-25T13:32:13.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.841+0000 7f4da4ff9700 1 -- 172.21.15.122:0/2508842562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d98004370 con 0x7f4da8104c20 2024-04-25T13:32:13.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.841+0000 7f4da4ff9700 1 -- 172.21.15.122:0/2508842562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d980039b0 con 0x7f4da8104c20 2024-04-25T13:32:13.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.841+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da819fcd0 con 0x7f4da8104c20 2024-04-25T13:32:13.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.841+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da8199690 con 0x7f4da8104c20 2024-04-25T13:32:13.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.842+0000 7f4da4ff9700 1 -- 172.21.15.122:0/2508842562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f4d98029070 con 0x7f4da8104c20 2024-04-25T13:32:13.844 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.843+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da806b5a0 con 0x7f4da8104c20 2024-04-25T13:32:13.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.843+0000 7f4da4ff9700 1 --2- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4d9003d670 0x7f4d9003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:13.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.843+0000 7f4da4ff9700 1 -- 172.21.15.122:0/2508842562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4d9801e070 con 0x7f4da8104c20 2024-04-25T13:32:13.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.843+0000 7f4da6ffd700 1 --2- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4d9003d670 0x7f4d9003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:13.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.844+0000 7f4da6ffd700 1 --2- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4d9003d670 0x7f4d9003fb80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c006fd0 tx=0x7f4d9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:13.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:13.849+0000 7f4da4ff9700 1 -- 172.21.15.122:0/2508842562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d98013070 con 0x7f4da8104c20 2024-04-25T13:32:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:13 smithi027 ceph-mon[29140]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 systemd[1]: Starting Ceph mon.c for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:32:14.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.157+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4da8064cf0 con 0x7f4da8104c20 2024-04-25T13:32:14.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.157+0000 7f4da4ff9700 1 -- 172.21.15.122:0/2508842562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f4d98030300 con 0x7f4da8104c20 2024-04-25T13:32:14.159 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:32:14.159 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","modified":"2024-04-25T13:30:20.883827Z","created":"2024-04-25T13:30:20.883827Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T13:32:14.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4d9003d670 msgr2=0x7f4d9003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:14.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 --2- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4d9003d670 0x7f4d9003fb80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c006fd0 tx=0x7f4d9c006e40 comp rx=0 tx=0).stop 2024-04-25T13:32:14.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 msgr2=0x7f4da819f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:14.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 --2- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 0x7f4da819f500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d980056d0 tx=0x7f4d980057b0 comp rx=0 tx=0).stop 2024-04-25T13:32:14.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 shutdown_connections 2024-04-25T13:32:14.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 --2- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4d9003d670 0x7f4d9003fb80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:14.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 --2- 172.21.15.122:0/2508842562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4da8104c20 0x7f4da819f500 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:14.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 >> 172.21.15.122:0/2508842562 conn(0x7f4da8078480 msgr2=0x7f4da8105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:14.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 shutdown_connections 2024-04-25T13:32:14.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:14.160+0000 7f4dadd5a700 1 -- 172.21.15.122:0/2508842562 wait complete. 2024-04-25T13:32:14.162 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-25T13:32:14.459 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 podman[32809]: 2024-04-25 13:32:14.134912791 +0000 UTC m=+0.017650781 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:32:14.460 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 podman[32809]: 2024-04-25 13:32:14.257711327 +0000 UTC m=+0.140449358 container create 3705a47103b402683d0da385748395bc402db88327a2f94f38865eb38cc0e678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T13:32:14.460 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 podman[32809]: 2024-04-25 13:32:14.459813084 +0000 UTC m=+0.342551073 container init 3705a47103b402683d0da385748395bc402db88327a2f94f38865eb38cc0e678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T13:32:14.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 podman[32809]: 2024-04-25 13:32:14.462833669 +0000 UTC m=+0.345571659 container start 3705a47103b402683d0da385748395bc402db88327a2f94f38865eb38cc0e678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pidfile_write: ignore empty --pid-file 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: load: jerasure load: lrc 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: RocksDB version: 7.9.2 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Git sha 0 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: DB SUMMARY 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: DB Session ID: 59JT07NDHL83JJU2LJPC 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: CURRENT file: CURRENT 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T13:32:14.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T13:32:14.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-25T13:32:14.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-25T13:32:14.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.error_if_exists: 0 2024-04-25T13:32:14.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.create_if_missing: 0 2024-04-25T13:32:14.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.paranoid_checks: 1 2024-04-25T13:32:14.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.env: 0x55bd21bcb500 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.info_log: 0x55bd23fd9fc0 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.statistics: (nil) 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.use_fsync: 0 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_log_file_size: 0 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T13:32:14.813 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.allow_fallocate: 1 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.use_direct_reads: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.db_log_dir: 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.wal_dir: 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T13:32:14.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.write_buffer_manager: 0x55bd23faa8c0 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T13:32:14.815 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.unordered_write: 0 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.row_cache: None 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.wal_filter: None 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.two_write_queues: 0 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.wal_compression: 0 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.atomic_flush: 0 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T13:32:14.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.log_readahead_size: 0 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_background_jobs: 2 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_background_compactions: -1 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_subcompactions: 1 2024-04-25T13:32:14.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T13:32:14.820 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T13:32:14.820 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T13:32:14.820 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T13:32:14.820 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T13:32:14.820 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T13:32:14.820 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T13:32:14.820 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T13:32:14.820 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_open_files: -1 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_background_flushes: -1 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Compression algorithms supported: 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: kZSTD supported: 0 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: kXpressCompression supported: 0 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: kZlibCompression supported: 1 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: kSnappyCompression supported: 1 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: kLZ4Compression supported: 1 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: kBZip2Compression supported: 0 2024-04-25T13:32:14.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.merge_operator: 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_filter: None 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bd240e01a0) 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T13:32:14.822 INFO:journalctl@ceph.mon.c.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_type: 0 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: data_block_index_type: 0 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_shortening: 1 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: checksum: 4 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: no_block_cache: 0 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache: 0x55bd23fdd1f0 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_options: 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: capacity : 536870912 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: num_shard_bits : 4 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: strict_capacity_limit : 0 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-04-25T13:32:14.823 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_compressed: (nil) 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: persistent_cache: (nil) 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_size: 4096 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_size_deviation: 10 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_restart_interval: 16 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_block_restart_interval: 1 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: metadata_block_size: 4096 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: partition_filters: 0 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: use_delta_encoding: 1 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: filter_policy: bloomfilter 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: whole_key_filtering: 1 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: verify_compression: 0 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: format_version: 5 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: enable_index_compression: 1 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_align: 0 2024-04-25T13:32:14.824 INFO:journalctl@ceph.mon.c.smithi122.stdout: max_auto_readahead_size: 262144 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout: prepopulate_block_cache: 0 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout: initial_auto_readahead_size: 8192 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression: NoCompression 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.num_levels: 7 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T13:32:14.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T13:32:14.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T13:32:14.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T13:32:14.828 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T13:32:14.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T13:32:14.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T13:32:14.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T13:32:14.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T13:32:14.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T13:32:14.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T13:32:14.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T13:32:14.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T13:32:14.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T13:32:14.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T13:32:14.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T13:32:14.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T13:32:14.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T13:32:14.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T13:32:14.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T13:32:14.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T13:32:14.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T13:32:14.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T13:32:14.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T13:32:14.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T13:32:14.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T13:32:14.832 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T13:32:14.832 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T13:32:14.832 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.inplace_update_support: 0 2024-04-25T13:32:14.832 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T13:32:14.832 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T13:32:14.832 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T13:32:14.832 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.bloom_locality: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.max_successive_merges: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.ttl: 2592000 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.enable_blob_files: false 2024-04-25T13:32:14.833 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.min_blob_size: 0 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9752a1f3-1843-4f72-8a70-a047920b2b64 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051934503014, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T13:32:14.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051934503555, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714051934, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9752a1f3-1843-4f72-8a70-a047920b2b64", "db_session_id": "59JT07NDHL83JJU2LJPC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051934503651, "job": 1, "event": "recovery_finished"} 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bd23fc1c00 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: DB pointer 0x55bd28db4000 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** DB Stats ** 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T13:32:14.835 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T13:32:14.836 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T13:32:14.837 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T13:32:14.838 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T13:32:14.838 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T13:32:14.838 INFO:journalctl@ceph.mon.c.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T13:32:14.838 INFO:journalctl@ceph.mon.c.smithi122.stdout: Block cache BinnedLRUCache@0x55bd23fdd1f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-25T13:32:14.838 INFO:journalctl@ceph.mon.c.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T13:32:14.838 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-04-25T13:32:14.838 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T13:32:14.839 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-25T13:32:14.839 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: using public_addr v2:172.21.15.122:0/0 -> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-04-25T13:32:14.839 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: starting mon.c rank -1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:14.839 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(???) e0 preinit fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:14.839 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).mds e1 new map 2024-04-25T13:32:14.839 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-25T13:32:14.839 INFO:journalctl@ceph.mon.c.smithi122.stdout: e1 2024-04-25T13:32:14.839 INFO:journalctl@ceph.mon.c.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout: legacy client fscid: -1 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout: No filesystems configured 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:32:14.840 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Deploying cephadm binary to smithi122 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:14.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Added host smithi122 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/103199239' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/103199239' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:14.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c;count:3 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:14.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:14.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:14.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:14.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: Deploying daemon mon.c on smithi122 2024-04-25T13:32:14.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:14.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 ceph-mon[32843]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T13:32:14.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 bash[32809]: 3705a47103b402683d0da385748395bc402db88327a2f94f38865eb38cc0e678 2024-04-25T13:32:14.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:14 smithi122 systemd[1]: Started Ceph mon.c for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:32:16.168 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T13:32:16.169 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T13:32:16.408 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: mon.a calling monitor election 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: mon.c calling monitor election 2024-04-25T13:32:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: monmap epoch 2 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: last_changed 2024-04-25T13:32:14.529890+0000 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: min_mon_release 19 (squid) 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: election_strategy: 1 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: fsmap 2024-04-25T13:32:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:19.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: mgrmap e13: x(active, since 49s) 2024-04-25T13:32:19.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T13:32:19.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:19.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:19.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:19.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:19.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:32:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: mon.a calling monitor election 2024-04-25T13:32:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: mon.c calling monitor election 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: monmap epoch 2 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: last_changed 2024-04-25T13:32:14.529890+0000 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: min_mon_release 19 (squid) 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: election_strategy: 1 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:20.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: fsmap 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: mgrmap e13: x(active, since 49s) 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: overall HEALTH_OK 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:20.061 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:20.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.871+0000 7fc2d1c48700 1 -- 172.21.15.122:0/3295843646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 msgr2=0x7fc2b4005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:20.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.871+0000 7fc2d1c48700 1 --2- 172.21.15.122:0/3295843646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 0x7fc2b4005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005fa0 tx=0x7fc2bc00ffc0 comp rx=0 tx=0).stop 2024-04-25T13:32:20.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.871+0000 7fc2d1c48700 1 -- 172.21.15.122:0/3295843646 shutdown_connections 2024-04-25T13:32:20.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.871+0000 7fc2d1c48700 1 --2- 172.21.15.122:0/3295843646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 0x7fc2b4005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:20.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.871+0000 7fc2d1c48700 1 -- 172.21.15.122:0/3295843646 >> 172.21.15.122:0/3295843646 conn(0x7fc2cc079950 msgr2=0x7fc2cc079d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:20.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.871+0000 7fc2d1c48700 1 -- 172.21.15.122:0/3295843646 shutdown_connections 2024-04-25T13:32:20.873 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.871+0000 7fc2d1c48700 1 -- 172.21.15.122:0/3295843646 wait complete. 2024-04-25T13:32:20.874 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:20 smithi122 ceph-mon[32843]: Deploying daemon mon.b on smithi045 2024-04-25T13:32:20.874 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:20.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.872+0000 7fc2d1c48700 1 Processor -- start 2024-04-25T13:32:20.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2d1c48700 1 -- start start 2024-04-25T13:32:20.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2d1c48700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 0x7fc2cc1a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:20.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2d1c48700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 0x7fc2cc19b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:20.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2d1c48700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc10bac0 con 0x7fc2cc104ce0 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2d1c48700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc10b940 con 0x7fc2cc1a1d00 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2caffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 0x7fc2cc19b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2caffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 0x7fc2cc19b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:45080/0 (socket says 172.21.15.122:45080) 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2caffd700 1 -- 172.21.15.122:0/38357809 learned_addr learned my addr 172.21.15.122:0/38357809 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2caffd700 1 -- 172.21.15.122:0/38357809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 msgr2=0x7fc2cc19b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2cb7fe700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 0x7fc2cc1a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2caffd700 1 -- 172.21.15.122:0/38357809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 msgr2=0x7fc2cc19b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2caffd700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 0x7fc2cc19b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.873+0000 7fc2caffd700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 0x7fc2cc19b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2cb7fe700 1 -- 172.21.15.122:0/38357809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 msgr2=0x7fc2cc19b9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2cb7fe700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 0x7fc2cc19b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2cb7fe700 1 -- 172.21.15.122:0/38357809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2cc1a22d0 con 0x7fc2cc104ce0 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2cb7fe700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 0x7fc2cc1a17c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005fa0 tx=0x7fc2bc004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:20.876 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2c8ff9700 1 -- 172.21.15.122:0/38357809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc2bc005230 con 0x7fc2cc104ce0 2024-04-25T13:32:20.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2c8ff9700 1 -- 172.21.15.122:0/38357809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2bc01b430 con 0x7fc2cc104ce0 2024-04-25T13:32:20.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2cc19bf80 con 0x7fc2cc104ce0 2024-04-25T13:32:20.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2c8ff9700 1 -- 172.21.15.122:0/38357809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc2bc011670 con 0x7fc2cc104ce0 2024-04-25T13:32:20.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.874+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2cc19c510 con 0x7fc2cc104ce0 2024-04-25T13:32:20.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.875+0000 7fc2c8ff9700 1 -- 172.21.15.122:0/38357809 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fc2bc011920 con 0x7fc2cc104ce0 2024-04-25T13:32:20.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.875+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2cc06c490 con 0x7fc2cc104ce0 2024-04-25T13:32:20.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.876+0000 7fc2c8ff9700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2b803d420 0x7fc2b803f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:20.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.876+0000 7fc2c8ff9700 1 -- 172.21.15.122:0/38357809 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc2bc053ac0 con 0x7fc2cc104ce0 2024-04-25T13:32:20.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.876+0000 7fc2caffd700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2b803d420 0x7fc2b803f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:20.878 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.877+0000 7fc2caffd700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2b803d420 0x7fc2b803f930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc104610 tx=0x7fc2c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:20.883 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:20.882+0000 7fc2c8ff9700 1 -- 172.21.15.122:0/38357809 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2bc055e20 con 0x7fc2cc104ce0 2024-04-25T13:32:20.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:20 smithi027 ceph-mon[29140]: Deploying daemon mon.b on smithi045 2024-04-25T13:32:20.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:21.167 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.166+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc2cc19cf10 con 0x7fc2cc104ce0 2024-04-25T13:32:21.168 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.167+0000 7fc2c8ff9700 1 -- 172.21.15.122:0/38357809 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fc2bc022070 con 0x7fc2cc104ce0 2024-04-25T13:32:21.168 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:32:21.168 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","modified":"2024-04-25T13:32:14.529890Z","created":"2024-04-25T13:30:20.883827Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T13:32:21.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2b803d420 msgr2=0x7fc2b803f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:21.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2b803d420 0x7fc2b803f930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc104610 tx=0x7fc2c0008040 comp rx=0 tx=0).stop 2024-04-25T13:32:21.171 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 msgr2=0x7fc2cc1a17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 0x7fc2cc1a17c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005fa0 tx=0x7fc2bc004d60 comp rx=0 tx=0).stop 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 shutdown_connections 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2b803d420 0x7fc2b803f930 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc104ce0 0x7fc2cc1a17c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 --2- 172.21.15.122:0/38357809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2cc1a1d00 0x7fc2cc19b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 >> 172.21.15.122:0/38357809 conn(0x7fc2cc079950 msgr2=0x7fc2cc101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 shutdown_connections 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:21.170+0000 7fc2d1c48700 1 -- 172.21.15.122:0/38357809 wait complete. 2024-04-25T13:32:21.172 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2024-04-25T13:32:21.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:21 smithi122 ceph-mon[32843]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:21.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:21 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/38357809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T13:32:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:21 smithi027 ceph-mon[29140]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:21 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/38357809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T13:32:22.758 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T13:32:22.758 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T13:32:23.006 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:32:23.436 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 podman[32712]: 2024-04-25 13:32:23.401866319 +0000 UTC m=+0.360266864 container start b396fa368b634dc01c985307591b7e6d34acdaaa0b60b9e2306900c82027f840 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T13:32:23.832 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T13:32:23.832 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T13:32:23.832 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pidfile_write: ignore empty --pid-file 2024-04-25T13:32:23.832 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: load: jerasure load: lrc 2024-04-25T13:32:23.832 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: RocksDB version: 7.9.2 2024-04-25T13:32:23.832 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Git sha 0 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: DB SUMMARY 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: DB Session ID: JCUQNBILOZBQZ57ZAFUJ 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: CURRENT file: CURRENT 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.error_if_exists: 0 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.create_if_missing: 0 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.paranoid_checks: 1 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T13:32:23.833 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.env: 0x55991cbbb500 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.info_log: 0x55991f0b7fc0 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.statistics: (nil) 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.use_fsync: 0 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_log_file_size: 0 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.allow_fallocate: 1 2024-04-25T13:32:23.834 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.use_direct_reads: 0 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.db_log_dir: 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.wal_dir: 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T13:32:23.835 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.write_buffer_manager: 0x55991f0888c0 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.unordered_write: 0 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T13:32:23.836 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.row_cache: None 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.wal_filter: None 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.two_write_queues: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.wal_compression: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.atomic_flush: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.log_readahead_size: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T13:32:23.837 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_background_jobs: 2 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_background_compactions: -1 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_subcompactions: 1 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T13:32:23.838 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_open_files: -1 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_background_flushes: -1 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Compression algorithms supported: 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: kZSTD supported: 0 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: kXpressCompression supported: 0 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T13:32:23.839 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: kZlibCompression supported: 1 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: kSnappyCompression supported: 1 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: kLZ4Compression supported: 1 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: kBZip2Compression supported: 0 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.merge_operator: 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_filter: None 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T13:32:23.840 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55991f1be1a0) 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: cache_index_and_filter_blocks: 1 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: pin_top_level_index_and_filter: 1 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: index_type: 0 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: data_block_index_type: 0 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: index_shortening: 1 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: checksum: 4 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: no_block_cache: 0 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: block_cache: 0x55991f0bb1f0 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: block_cache_name: BinnedLRUCache 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: block_cache_options: 2024-04-25T13:32:23.841 INFO:journalctl@ceph.mon.b.smithi045.stdout: capacity : 536870912 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: num_shard_bits : 4 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: strict_capacity_limit : 0 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: high_pri_pool_ratio: 0.000 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: block_cache_compressed: (nil) 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: persistent_cache: (nil) 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: block_size: 4096 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: block_size_deviation: 10 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: block_restart_interval: 16 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: index_block_restart_interval: 1 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: metadata_block_size: 4096 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: partition_filters: 0 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: use_delta_encoding: 1 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: filter_policy: bloomfilter 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: whole_key_filtering: 1 2024-04-25T13:32:23.842 INFO:journalctl@ceph.mon.b.smithi045.stdout: verify_compression: 0 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout: read_amp_bytes_per_bit: 0 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout: format_version: 5 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout: enable_index_compression: 1 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout: block_align: 0 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout: max_auto_readahead_size: 262144 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout: prepopulate_block_cache: 0 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout: initial_auto_readahead_size: 8192 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression: NoCompression 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T13:32:23.843 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.num_levels: 7 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T13:32:23.844 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T13:32:23.845 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T13:32:23.846 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.inplace_update_support: 0 2024-04-25T13:32:23.847 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.bloom_locality: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.max_successive_merges: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.ttl: 2592000 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T13:32:23.848 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.enable_blob_files: false 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.min_blob_size: 0 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c6952d38-ce46-4918-921f-1de7255c1beb 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051943442753, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T13:32:23.849 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051943443190, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714051943, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c6952d38-ce46-4918-921f-1de7255c1beb", "db_session_id": "JCUQNBILOZBQZ57ZAFUJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714051943443273, "job": 1, "event": "recovery_finished"} 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55991f09fc00 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: DB pointer 0x559923e92000 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout: ** DB Stats ** 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T13:32:23.850 INFO:journalctl@ceph.mon.b.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: ** Compaction Stats [default] ** 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: ** Compaction Stats [default] ** 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T13:32:23.851 INFO:journalctl@ceph.mon.b.smithi045.stdout: 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: Block cache BinnedLRUCache@0x55991f0bb1f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T13:32:23.852 INFO:journalctl@ceph.mon.b.smithi045.stdout: 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: using public_addr v2:172.21.15.45:0/0 -> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: starting mon.b rank -1 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(???) e0 preinit fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout: e1 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout: legacy client fscid: -1 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout: 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout: No filesystems configured 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T13:32:23.853 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:23.854 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Deploying cephadm binary to smithi122 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.855 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Added host smithi122 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:32:23.856 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/103199239' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/103199239' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-25T13:32:23.857 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Saving service mon spec with placement smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c;count:3 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.858 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Deploying daemon mon.c on smithi122 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.a calling monitor election 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.859 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.c calling monitor election 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: monmap epoch 2 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: last_changed 2024-04-25T13:32:14.529890+0000 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: min_mon_release 19 (squid) 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: election_strategy: 1 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:23.860 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: fsmap 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mgrmap e13: x(active, since 49s) 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: overall HEALTH_OK 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: Deploying daemon mon.b on smithi045 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/38357809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 ceph-mon[32731]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 bash[32712]: b396fa368b634dc01c985307591b7e6d34acdaaa0b60b9e2306900c82027f840 2024-04-25T13:32:23.861 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:23 smithi045 systemd[1]: Started Ceph mon.b for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:32:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:32:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: mon.c calling monitor election 2024-04-25T13:32:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: mon.a calling monitor election 2024-04-25T13:32:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: monmap epoch 3 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: last_changed 2024-04-25T13:32:23.475463+0000 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: min_mon_release 19 (squid) 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: election_strategy: 1 2024-04-25T13:32:28.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: 2: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: fsmap 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: mgrmap e13: x(active, since 58s) 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: overall HEALTH_OK 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:28.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.852+0000 7f48d7733700 1 -- 172.21.15.122:0/1373679914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0108a20 msgr2=0x7f48bc005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.852+0000 7f48d7733700 1 --2- 172.21.15.122:0/1373679914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0108a20 0x7f48bc005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48c4005fd0 tx=0x7f48c400ff70 comp rx=0 tx=0).stop 2024-04-25T13:32:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.853+0000 7f48d7733700 1 -- 172.21.15.122:0/1373679914 shutdown_connections 2024-04-25T13:32:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.853+0000 7f48d7733700 1 --2- 172.21.15.122:0/1373679914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0108a20 0x7f48bc005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.853+0000 7f48d7733700 1 -- 172.21.15.122:0/1373679914 >> 172.21.15.122:0/1373679914 conn(0x7f48d00fe530 msgr2=0x7f48d0100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.853+0000 7f48d7733700 1 -- 172.21.15.122:0/1373679914 shutdown_connections 2024-04-25T13:32:28.854 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.853+0000 7f48d7733700 1 -- 172.21.15.122:0/1373679914 wait complete. 2024-04-25T13:32:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.854+0000 7f48d7733700 1 Processor -- start 2024-04-25T13:32:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.854+0000 7f48d7733700 1 -- start start 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.854+0000 7f48d7733700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 0x7f48d019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.854+0000 7f48d7733700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 0x7f48d01987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.854+0000 7f48d7733700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0198d20 0x7f48d0199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.854+0000 7f48d7733700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f48d010a9a0 con 0x7f48d0198d20 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.854+0000 7f48d7733700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f48d010a820 con 0x7f48d019eae0 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.854+0000 7f48d7733700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f48d010ab20 con 0x7f48d0108a20 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d54cf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 0x7f48d019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d54cf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 0x7f48d019e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37324/0 (socket says 172.21.15.122:37324) 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d54cf700 1 -- 172.21.15.122:0/2695543416 learned_addr learned my addr 172.21.15.122:0/2695543416 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:32:28.857 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d5cd0700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0198d20 0x7f48d0199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d4cce700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 0x7f48d01987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d54cf700 1 -- 172.21.15.122:0/2695543416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 msgr2=0x7f48d019e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d54cf700 1 -- 172.21.15.122:0/2695543416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 msgr2=0x7f48d019e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d54cf700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 0x7f48d019e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d54cf700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 0x7f48d019e5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d4cce700 1 -- 172.21.15.122:0/2695543416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 msgr2=0x7f48d01987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d4cce700 1 -- 172.21.15.122:0/2695543416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 msgr2=0x7f48d01987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d4cce700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 0x7f48d01987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d4cce700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 0x7f48d01987e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d5cd0700 1 -- 172.21.15.122:0/2695543416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 msgr2=0x7f48d019e5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d5cd0700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 0x7f48d019e5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d5cd0700 1 -- 172.21.15.122:0/2695543416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 msgr2=0x7f48d01987e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:32:28.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d5cd0700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 0x7f48d01987e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:28.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d5cd0700 1 -- 172.21.15.122:0/2695543416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d01a5400 con 0x7f48d0198d20 2024-04-25T13:32:28.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48d5cd0700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0198d20 0x7f48d0199130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f48c800bd50 tx=0x7f48c800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:28.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48c27fc700 1 -- 172.21.15.122:0/2695543416 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8019070 con 0x7f48d0198d20 2024-04-25T13:32:28.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.855+0000 7f48c27fc700 1 -- 172.21.15.122:0/2695543416 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48c800cd10 con 0x7f48d0198d20 2024-04-25T13:32:28.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.856+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d01a56f0 con 0x7f48d0198d20 2024-04-25T13:32:28.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.856+0000 7f48c27fc700 1 -- 172.21.15.122:0/2695543416 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8012700 con 0x7f48d0198d20 2024-04-25T13:32:28.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.856+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d01a5c00 con 0x7f48d0198d20 2024-04-25T13:32:28.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.856+0000 7f48c27fc700 1 -- 172.21.15.122:0/2695543416 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f48c801c070 con 0x7f48d0198d20 2024-04-25T13:32:28.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.857+0000 7f48c27fc700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48b403daa0 0x7f48b403ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:28.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.857+0000 7f48c27fc700 1 -- 172.21.15.122:0/2695543416 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f48c8050f20 con 0x7f48d0198d20 2024-04-25T13:32:28.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.857+0000 7f48d54cf700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48b403daa0 0x7f48b403ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:28.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.857+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d0104130 con 0x7f48d0198d20 2024-04-25T13:32:28.860 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.858+0000 7f48d54cf700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48b403daa0 0x7f48b403ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f48c400fad0 tx=0x7f48c4005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:28.864 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:28.863+0000 7f48c27fc700 1 -- 172.21.15.122:0/2695543416 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48c800f160 con 0x7f48d0198d20 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: mon.c calling monitor election 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: mon.a calling monitor election 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: monmap epoch 3 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: last_changed 2024-04-25T13:32:23.475463+0000 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: min_mon_release 19 (squid) 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: election_strategy: 1 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: 2: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: fsmap 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: mgrmap e13: x(active, since 58s) 2024-04-25T13:32:28.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T13:32:28.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:28.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:28.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:28.904 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:29.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.158+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f48d002e510 con 0x7f48d0198d20 2024-04-25T13:32:29.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.159+0000 7f48c27fc700 1 -- 172.21.15.122:0/2695543416 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f48c8014a10 con 0x7f48d0198d20 2024-04-25T13:32:29.160 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-25T13:32:29.160 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":3,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","modified":"2024-04-25T13:32:23.475463Z","created":"2024-04-25T13:30:20.883827Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T13:32:29.162 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48b403daa0 msgr2=0x7f48b403ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48b403daa0 0x7f48b403ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f48c400fad0 tx=0x7f48c4005c00 comp rx=0 tx=0).stop 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0198d20 msgr2=0x7f48d0199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0198d20 0x7f48d0199130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f48c800bd50 tx=0x7f48c800be30 comp rx=0 tx=0).stop 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 shutdown_connections 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48b403daa0 0x7f48b403ffb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48d0108a20 0x7f48d019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48d0198d20 0x7f48d0199130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 --2- 172.21.15.122:0/2695543416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48d019eae0 0x7f48d01987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.161+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 >> 172.21.15.122:0/2695543416 conn(0x7f48d00fe530 msgr2=0x7f48d0106260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.162+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 shutdown_connections 2024-04-25T13:32:29.163 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:29.162+0000 7f48d7733700 1 -- 172.21.15.122:0/2695543416 wait complete. 2024-04-25T13:32:29.164 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 3 2024-04-25T13:32:29.718 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:29 smithi122 ceph-mon[32843]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:29.718 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:29.718 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:29 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/2695543416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T13:32:29.718 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:29.746 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T13:32:29.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T13:32:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:29 smithi027 ceph-mon[29140]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:29 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/2695543416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T13:32:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:30.003 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:32:30.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:32:30.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: mon.c calling monitor election 2024-04-25T13:32:30.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: mon.a calling monitor election 2024-04-25T13:32:30.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:30.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:32:30.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:30.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: monmap epoch 3 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: last_changed 2024-04-25T13:32:23.475463+0000 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: min_mon_release 19 (squid) 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: election_strategy: 1 2024-04-25T13:32:30.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: 2: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: fsmap 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: mgrmap e13: x(active, since 58s) 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: overall HEALTH_OK 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:30.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/2695543416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T13:32:30.832 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.650+0000 7feef4803700 1 -- 172.21.15.27:0/86253076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 msgr2=0x7feef0105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.650+0000 7feef4803700 1 --2- 172.21.15.27:0/86253076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 0x7feef0105010 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feee0009910 tx=0x7feee0009c20 comp rx=0 tx=0).stop 2024-04-25T13:32:31.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.650+0000 7feef4803700 1 -- 172.21.15.27:0/86253076 shutdown_connections 2024-04-25T13:32:31.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.650+0000 7feef4803700 1 --2- 172.21.15.27:0/86253076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 0x7feef0105010 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:31.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.650+0000 7feef4803700 1 -- 172.21.15.27:0/86253076 >> 172.21.15.27:0/86253076 conn(0x7feef0078460 msgr2=0x7feef00788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:31.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.650+0000 7feef4803700 1 -- 172.21.15.27:0/86253076 shutdown_connections 2024-04-25T13:32:31.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.651+0000 7feef4803700 1 -- 172.21.15.27:0/86253076 wait complete. 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feef4803700 1 Processor -- start 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feef4803700 1 -- start start 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feef4803700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 0x7feef019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feef4803700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feef019fc30 0x7feef0199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feef4803700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef0199e50 0x7feef019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feef4803700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feef010dff0 con 0x7feef0104c00 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feef4803700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feef010de70 con 0x7feef019fc30 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feef4803700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feef010e170 con 0x7feef0199e50 2024-04-25T13:32:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feeee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 0x7feef019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feeee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 0x7feef019f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57264/0 (socket says 172.21.15.27:57264) 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feeee59c700 1 -- 172.21.15.27:0/1557439899 learned_addr learned my addr 172.21.15.27:0/1557439899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.652+0000 7feeedd9b700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feef019fc30 0x7feef0199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feeeed9d700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef0199e50 0x7feef019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feeee59c700 1 -- 172.21.15.27:0/1557439899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef0199e50 msgr2=0x7feef019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feeee59c700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef0199e50 0x7feef019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feeee59c700 1 -- 172.21.15.27:0/1557439899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feef019fc30 msgr2=0x7feef0199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feeee59c700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feef019fc30 0x7feef0199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feeee59c700 1 -- 172.21.15.27:0/1557439899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef01a6580 con 0x7feef0104c00 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feeee59c700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 0x7feef019f6f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feee0005b40 tx=0x7feee0004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feedf7fe700 1 -- 172.21.15.27:0/1557439899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0026070 con 0x7feef0104c00 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feedf7fe700 1 -- 172.21.15.27:0/1557439899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feee000b750 con 0x7feef0104c00 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef01a67b0 con 0x7feef0104c00 2024-04-25T13:32:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feedf7fe700 1 -- 172.21.15.27:0/1557439899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee00038e0 con 0x7feef0104c00 2024-04-25T13:32:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.653+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef01a6cb0 con 0x7feef0104c00 2024-04-25T13:32:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.654+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef0064cf0 con 0x7feef0104c00 2024-04-25T13:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.654+0000 7feedf7fe700 1 -- 172.21.15.27:0/1557439899 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7feee0029070 con 0x7feef0104c00 2024-04-25T13:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.655+0000 7feedf7fe700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feed803daa0 0x7feed803ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.655+0000 7feedf7fe700 1 -- 172.21.15.27:0/1557439899 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7feee0052320 con 0x7feef0104c00 2024-04-25T13:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.655+0000 7feeedd9b700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feed803daa0 0x7feed803ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.656+0000 7feeedd9b700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feed803daa0 0x7feed803ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feef0106e70 tx=0x7feee4009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:31.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.660+0000 7feedf7fe700 1 -- 172.21.15.27:0/1557439899 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feee0013070 con 0x7feef0104c00 2024-04-25T13:32:31.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: mon.b calling monitor election 2024-04-25T13:32:31.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:31.781 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: mon.b calling monitor election 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: mon.c calling monitor election 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: mon.a calling monitor election 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: monmap epoch 3 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: last_changed 2024-04-25T13:32:23.475463+0000 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: min_mon_release 19 (squid) 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: election_strategy: 1 2024-04-25T13:32:31.782 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: 2: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: fsmap 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: mgrmap e13: x(active, since 60s) 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: Updating smithi027:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.783 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:31.784 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: mon.b calling monitor election 2024-04-25T13:32:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: mon.b calling monitor election 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: mon.c calling monitor election 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: mon.a calling monitor election 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: monmap epoch 3 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: last_changed 2024-04-25T13:32:23.475463+0000 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: min_mon_release 19 (squid) 2024-04-25T13:32:31.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: election_strategy: 1 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: 2: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: fsmap 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: mgrmap e13: x(active, since 60s) 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: overall HEALTH_OK 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: Updating smithi027:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:31.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: mon.b calling monitor election 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: Updating smithi045:/etc/ceph/ceph.conf 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: mon.b calling monitor election 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: mon.c calling monitor election 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: mon.a calling monitor election 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T13:32:31.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: monmap epoch 3 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: last_changed 2024-04-25T13:32:23.475463+0000 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: created 2024-04-25T13:30:20.883827+0000 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: min_mon_release 19 (squid) 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: election_strategy: 1 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: 2: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: fsmap 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: mgrmap e13: x(active, since 60s) 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: overall HEALTH_OK 2024-04-25T13:32:31.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: Updating smithi027:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: Updating smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: Updating smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:31.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:31.832 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:32:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.871+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7feef01a6f10 con 0x7feef0104c00 2024-04-25T13:32:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.871+0000 7feedf7fe700 1 -- 172.21.15.27:0/1557439899 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7feee0029e10 con 0x7feef0104c00 2024-04-25T13:32:31.872 INFO:teuthology.orchestra.run.smithi027.stdout:# minimal ceph.conf for db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:31.873 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-04-25T13:32:31.873 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:32:31.873 INFO:teuthology.orchestra.run.smithi027.stdout: mon_host = [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.873+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feed803daa0 msgr2=0x7feed803ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.873+0000 7feef4803700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feed803daa0 0x7feed803ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feef0106e70 tx=0x7feee4009210 comp rx=0 tx=0).stop 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.873+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 msgr2=0x7feef019f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.873+0000 7feef4803700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 0x7feef019f6f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feee0005b40 tx=0x7feee0004ed0 comp rx=0 tx=0).stop 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.874+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 shutdown_connections 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.874+0000 7feef4803700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feed803daa0 0x7feed803ffb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.874+0000 7feef4803700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feef0199e50 0x7feef019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.874+0000 7feef4803700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feef0104c00 0x7feef019f6f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.874+0000 7feef4803700 1 --2- 172.21.15.27:0/1557439899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feef019fc30 0x7feef0199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.874+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 >> 172.21.15.27:0/1557439899 conn(0x7feef0078460 msgr2=0x7feef0109fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.874+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 shutdown_connections 2024-04-25T13:32:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:31.874+0000 7feef4803700 1 -- 172.21.15.27:0/1557439899 wait complete. 2024-04-25T13:32:32.535 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T13:32:32.535 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:32:32.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T13:32:32.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:32 smithi027 ceph-mon[29140]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T13:32:32.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:32 smithi027 ceph-mon[29140]: Reconfiguring daemon mon.a on smithi027 2024-04-25T13:32:32.561 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:32 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1557439899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:32.563 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:32:32.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:32.628 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:32:32.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T13:32:32.662 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:32:32.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:32.729 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:32:32.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T13:32:32.755 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:32 smithi122 ceph-mon[32843]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T13:32:32.756 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:32 smithi122 ceph-mon[32843]: Reconfiguring daemon mon.a on smithi027 2024-04-25T13:32:32.756 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:32 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1557439899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:32.761 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:32:32.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T13:32:32.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:32 smithi045 ceph-mon[32731]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T13:32:32.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:32 smithi045 ceph-mon[32731]: Reconfiguring daemon mon.a on smithi027 2024-04-25T13:32:32.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:32 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1557439899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:32.829 INFO:tasks.cephadm:Adding mgr.x on smithi027 2024-04-25T13:32:32.830 INFO:tasks.cephadm:Adding mgr.y on smithi045 2024-04-25T13:32:32.830 INFO:tasks.cephadm:Adding mgr.z on smithi122 2024-04-25T13:32:32.830 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi027=x;smithi045=y;smithi122=z' 2024-04-25T13:32:33.102 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:32:33.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:33 smithi027 ceph-mon[29140]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:33.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:33.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:33.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:33.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:33.800 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:33.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:33 smithi122 ceph-mon[32843]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:33.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:33.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:33.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:33.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:33.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:33.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:33 smithi045 ceph-mon[32731]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:33.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:33.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:33.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:33.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:33.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1126360222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc103ba0 msgr2=0x7fcccc103fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 --2- 172.21.15.122:0/1126360222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc103ba0 0x7fcccc103fb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009910 tx=0x7fccbc009c20 comp rx=0 tx=0).stop 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1126360222 shutdown_connections 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 --2- 172.21.15.122:0/1126360222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc103ba0 0x7fcccc103fb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1126360222 >> 172.21.15.122:0/1126360222 conn(0x7fcccc0ff200 msgr2=0x7fcccc101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1126360222 shutdown_connections 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1126360222 wait complete. 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 Processor -- start 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.616+0000 7fccd3c4b700 1 -- start start 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd3c4b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcccc103ba0 0x7fcccc077330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:34.619 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd3c4b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0759e0 0x7fcccc075df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd3c4b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc076330 0x7fcccc1a3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd3c4b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc109b60 con 0x7fcccc076330 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd3c4b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1099e0 con 0x7fcccc0759e0 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd3c4b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc109ce0 con 0x7fcccc103ba0 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcccc103ba0 0x7fcccc077330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcccc103ba0 0x7fcccc077330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:37328/0 (socket says 172.21.15.122:37328) 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 -- 172.21.15.122:0/1399152714 learned_addr learned my addr 172.21.15.122:0/1399152714 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd11e6700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0759e0 0x7fcccc075df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd21e8700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc076330 0x7fcccc1a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 -- 172.21.15.122:0/1399152714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0759e0 msgr2=0x7fcccc075df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0759e0 0x7fcccc075df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 -- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc076330 msgr2=0x7fcccc1a3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc076330 0x7fcccc1a3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:34.620 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 -- 172.21.15.122:0/1399152714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcccc1a3e60 con 0x7fcccc103ba0 2024-04-25T13:32:34.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd21e8700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc076330 0x7fcccc1a3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:32:34.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd19e7700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcccc103ba0 0x7fcccc077330 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fccbc005850 tx=0x7fccbc004ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:34.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccc2ffd700 1 -- 172.21.15.122:0/1399152714 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc026070 con 0x7fcccc103ba0 2024-04-25T13:32:34.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccc2ffd700 1 -- 172.21.15.122:0/1399152714 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccbc00b750 con 0x7fcccc103ba0 2024-04-25T13:32:34.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcccc1a4090 con 0x7fcccc103ba0 2024-04-25T13:32:34.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.617+0000 7fccc2ffd700 1 -- 172.21.15.122:0/1399152714 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc003970 con 0x7fcccc103ba0 2024-04-25T13:32:34.621 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.618+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcccc1a45c0 con 0x7fcccc103ba0 2024-04-25T13:32:34.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.618+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcccc064cf0 con 0x7fcccc103ba0 2024-04-25T13:32:34.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.618+0000 7fccc2ffd700 1 -- 172.21.15.122:0/1399152714 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fccbc029070 con 0x7fcccc103ba0 2024-04-25T13:32:34.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.619+0000 7fccc2ffd700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fccb803daa0 0x7fccb803ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:34.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.619+0000 7fccc2ffd700 1 -- 172.21.15.122:0/1399152714 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fccbc052240 con 0x7fcccc103ba0 2024-04-25T13:32:34.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.619+0000 7fccd11e6700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fccb803daa0 0x7fccb803ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:34.622 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.619+0000 7fccd11e6700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fccb803daa0 0x7fccb803ffb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcccc076ce0 tx=0x7fccc8009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:34.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.624+0000 7fccc2ffd700 1 -- 172.21.15.122:0/1399152714 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fccbc013070 con 0x7fcccc103ba0 2024-04-25T13:32:34.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:34 smithi122 ceph-mon[32843]: Reconfiguring mon.b (monmap changed)... 2024-04-25T13:32:34.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:34 smithi122 ceph-mon[32843]: Reconfiguring daemon mon.b on smithi045 2024-04-25T13:32:34.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:34 smithi045 ceph-mon[32731]: Reconfiguring mon.b (monmap changed)... 2024-04-25T13:32:34.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:34 smithi045 ceph-mon[32731]: Reconfiguring daemon mon.b on smithi045 2024-04-25T13:32:34.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.839+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi045=y;smithi122=z", "target": ["mon-mgr", ""]}) -- 0x7fcccc101050 con 0x7fccb803daa0 2024-04-25T13:32:34.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.845+0000 7fccc2ffd700 1 -- 172.21.15.122:0/1399152714 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcccc101050 con 0x7fccb803daa0 2024-04-25T13:32:34.847 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mgr update... 2024-04-25T13:32:34.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.847+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fccb803daa0 msgr2=0x7fccb803ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:34.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.847+0000 7fccd3c4b700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fccb803daa0 0x7fccb803ffb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcccc076ce0 tx=0x7fccc8009210 comp rx=0 tx=0).stop 2024-04-25T13:32:34.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcccc103ba0 msgr2=0x7fcccc077330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:34.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcccc103ba0 0x7fcccc077330 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fccbc005850 tx=0x7fccbc004ed0 comp rx=0 tx=0).stop 2024-04-25T13:32:34.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 shutdown_connections 2024-04-25T13:32:34.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fccb803daa0 0x7fccb803ffb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:34.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcccc103ba0 0x7fcccc077330 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:34.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcccc076330 0x7fcccc1a3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:34.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 --2- 172.21.15.122:0/1399152714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0759e0 0x7fcccc075df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:34.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 >> 172.21.15.122:0/1399152714 conn(0x7fcccc0ff200 msgr2=0x7fcccc100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:34.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 shutdown_connections 2024-04-25T13:32:34.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:32:34.848+0000 7fccd3c4b700 1 -- 172.21.15.122:0/1399152714 wait complete. 2024-04-25T13:32:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:34 smithi027 ceph-mon[29140]: Reconfiguring mon.b (monmap changed)... 2024-04-25T13:32:34.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:34 smithi027 ceph-mon[29140]: Reconfiguring daemon mon.b on smithi045 2024-04-25T13:32:35.436 DEBUG:teuthology.orchestra.run.smithi045:mgr.y> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T13:32:35.439 DEBUG:teuthology.orchestra.run.smithi122:mgr.z> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T13:32:35.441 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T13:32:35.441 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:32:35.441 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T13:32:35.461 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T13:32:35.462 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 13:31:40.181476210 +0000 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 13:27:15.918340388 +0000 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 13:27:15.918340388 +0000 2024-04-25T13:32:35.520 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 13:27:15.918340388 +0000 2024-04-25T13:32:35.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T13:32:35.588 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T13:32:35.588 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T13:32:35.588 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000225663 s, 2.3 MB/s 2024-04-25T13:32:35.590 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T13:32:35.647 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 13:31:40.218475260 +0000 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 13:27:16.280330321 +0000 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 13:27:16.280330321 +0000 2024-04-25T13:32:35.704 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 13:27:16.280330321 +0000 2024-04-25T13:32:35.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T13:32:35.769 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T13:32:35.769 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T13:32:35.769 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000202692 s, 2.5 MB/s 2024-04-25T13:32:35.770 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T13:32:35.827 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-04-25T13:32:35.883 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T13:32:35.883 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:35.884 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T13:32:35.884 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:35.884 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:35.884 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 13:31:40.252474388 +0000 2024-04-25T13:32:35.884 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 13:27:16.630320588 +0000 2024-04-25T13:32:35.884 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 13:27:16.630320588 +0000 2024-04-25T13:32:35.884 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 13:27:16.630320588 +0000 2024-04-25T13:32:35.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi045=y;smithi122=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: Saving service mgr spec with placement smithi027=x;smithi045=y;smithi122=z;count:3 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: Reconfiguring mon.c (monmap changed)... 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:35.904 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:35 smithi122 ceph-mon[32843]: Reconfiguring daemon mon.c on smithi122 2024-04-25T13:32:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T13:32:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T13:32:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00020111 s, 2.5 MB/s 2024-04-25T13:32:35.948 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T13:32:36.005 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-04-25T13:32:36.061 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T13:32:36.061 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:36.061 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T13:32:36.061 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:36.061 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:36.061 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-25 13:31:40.289473438 +0000 2024-04-25T13:32:36.061 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-25 13:27:16.986310688 +0000 2024-04-25T13:32:36.062 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-25 13:27:16.987310660 +0000 2024-04-25T13:32:36.062 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-25 13:27:16.986310688 +0000 2024-04-25T13:32:36.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi045=y;smithi122=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: Saving service mgr spec with placement smithi027=x;smithi045=y;smithi122=z;count:3 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: Reconfiguring mon.c (monmap changed)... 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:36.123 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:35 smithi027 ceph-mon[29140]: Reconfiguring daemon mon.c on smithi122 2024-04-25T13:32:36.125 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-25T13:32:36.125 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-25T13:32:36.125 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000193259 s, 2.6 MB/s 2024-04-25T13:32:36.126 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T13:32:36.182 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-25T13:32:36.183 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T13:32:36.202 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T13:32:36.202 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-04-25T13:32:36.260 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T13:32:36.260 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:36.261 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T13:32:36.261 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:36.261 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:36.261 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-25 13:32:22.846794207 +0000 2024-04-25T13:32:36.261 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-25 13:27:34.943941958 +0000 2024-04-25T13:32:36.261 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-25 13:27:34.943941958 +0000 2024-04-25T13:32:36.261 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-25 13:27:34.943941958 +0000 2024-04-25T13:32:36.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi027=x;smithi045=y;smithi122=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: Saving service mgr spec with placement smithi027=x;smithi045=y;smithi122=z;count:3 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: Reconfiguring mon.c (monmap changed)... 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T13:32:36.325 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T13:32:36.326 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:36.326 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:35 smithi045 ceph-mon[32731]: Reconfiguring daemon mon.c on smithi122 2024-04-25T13:32:36.327 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-25T13:32:36.327 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-25T13:32:36.327 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000109215 s, 4.7 MB/s 2024-04-25T13:32:36.329 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T13:32:36.388 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-04-25T13:32:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T13:32:36.446 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:36.446 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T13:32:36.446 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:36.446 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:36.446 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-25 13:32:22.846794207 +0000 2024-04-25T13:32:36.446 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-25 13:27:35.275936785 +0000 2024-04-25T13:32:36.447 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-25 13:27:35.275936785 +0000 2024-04-25T13:32:36.447 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-25 13:27:35.275936785 +0000 2024-04-25T13:32:36.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T13:32:36.513 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-25T13:32:36.513 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-25T13:32:36.513 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000120944 s, 4.2 MB/s 2024-04-25T13:32:36.514 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T13:32:36.572 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-04-25T13:32:36.629 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T13:32:36.629 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:36.629 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-25T13:32:36.629 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:36.630 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:36.630 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-25 13:32:22.847794192 +0000 2024-04-25T13:32:36.630 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-25 13:27:35.630931252 +0000 2024-04-25T13:32:36.630 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-25 13:27:35.630931252 +0000 2024-04-25T13:32:36.630 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-25 13:27:35.630931252 +0000 2024-04-25T13:32:36.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T13:32:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-25T13:32:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-25T13:32:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000120447 s, 4.3 MB/s 2024-04-25T13:32:36.696 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T13:32:36.753 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-25 13:32:22.847794192 +0000 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-25 13:27:35.978925829 +0000 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-25 13:27:35.978925829 +0000 2024-04-25T13:32:36.810 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-04-25 13:27:35.978925829 +0000 2024-04-25T13:32:36.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T13:32:36.875 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-25T13:32:36.875 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-25T13:32:36.875 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000117367 s, 4.4 MB/s 2024-04-25T13:32:36.876 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T13:32:36.934 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-25T13:32:36.934 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T13:32:36.950 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T13:32:36.950 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-25 13:32:13.925907214 +0000 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-25 13:27:27.098603915 +0000 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-25 13:27:27.098603915 +0000 2024-04-25T13:32:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-25 13:27:27.098603915 +0000 2024-04-25T13:32:37.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T13:32:37.068 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-25T13:32:37.068 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-25T13:32:37.068 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000271893 s, 1.9 MB/s 2024-04-25T13:32:37.069 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T13:32:37.125 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-04-25T13:32:37.180 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T13:32:37.180 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:37.180 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T13:32:37.180 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:37.180 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:37.180 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-25 13:32:13.925907214 +0000 2024-04-25T13:32:37.181 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-25 13:27:27.441596155 +0000 2024-04-25T13:32:37.181 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-25 13:27:27.441596155 +0000 2024-04-25T13:32:37.181 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-25 13:27:27.441596155 +0000 2024-04-25T13:32:37.181 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T13:32:37.243 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-25T13:32:37.243 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-25T13:32:37.243 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000198068 s, 2.6 MB/s 2024-04-25T13:32:37.244 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T13:32:37.300 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-25 13:32:13.926907192 +0000 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-25 13:27:27.780588486 +0000 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-25 13:27:27.780588486 +0000 2024-04-25T13:32:37.354 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-25 13:27:27.780588486 +0000 2024-04-25T13:32:37.355 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T13:32:37.416 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-25T13:32:37.416 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-25T13:32:37.417 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000198509 s, 2.6 MB/s 2024-04-25T13:32:37.418 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T13:32:37.473 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-04-25T13:32:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T13:32:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T13:32:37.528 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T13:32:37.528 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T13:32:37.528 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T13:32:37.528 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-25 13:32:13.926907192 +0000 2024-04-25T13:32:37.528 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-25 13:27:28.111580998 +0000 2024-04-25T13:32:37.528 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-25 13:27:28.111580998 +0000 2024-04-25T13:32:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-25 13:27:28.111580998 +0000 2024-04-25T13:32:37.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T13:32:37.590 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-25T13:32:37.590 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-25T13:32:37.590 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000178258 s, 2.9 MB/s 2024-04-25T13:32:37.591 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T13:32:37.647 INFO:tasks.cephadm:Deploying osd.0 on smithi027 with /dev/vg_nvme/lv_4... 2024-04-25T13:32:37.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T13:32:37.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T13:32:37.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:37.860 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:37.885 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:32:37.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:37.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:37.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:37.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:37.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:37.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:37.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:37.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:37.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T13:32:37.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:37.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:38.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:38.893 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:38 smithi027 ceph-mon[29140]: Deploying daemon mgr.z on smithi122 2024-04-25T13:32:38.920 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:38 smithi122 ceph-mon[32843]: Deploying daemon mgr.z on smithi122 2024-04-25T13:32:39.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:38 smithi045 ceph-mon[32731]: Deploying daemon mgr.z on smithi122 2024-04-25T13:32:39.559 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:39 smithi122 podman[34674]: 2024-04-25 13:32:39.515636893 +0000 UTC m=+0.126015872 container create 2068ea4a75cf2b1415b87d71dff2cd339c1b37c67ab103db4f01c77f7528cda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T13:32:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T13:32:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T13:32:39.706 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T13:32:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T13:32:39.706 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110545 s, 949 MB/s 2024-04-25T13:32:39.706 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T13:32:39.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:39 smithi122 ceph-mon[32843]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:39.901 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:39 smithi122 podman[34674]: 2024-04-25 13:32:39.734971111 +0000 UTC m=+0.345350090 container init 2068ea4a75cf2b1415b87d71dff2cd339c1b37c67ab103db4f01c77f7528cda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T13:32:39.901 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:39 smithi122 podman[34674]: 2024-04-25 13:32:39.737633096 +0000 UTC m=+0.348012075 container start 2068ea4a75cf2b1415b87d71dff2cd339c1b37c67ab103db4f01c77f7528cda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:32:39.901 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:39 smithi122 ceph-mgr[34693]: -- 172.21.15.122:0/3674775804 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c6ed145a0 con 0x564c73a9e900 2024-04-25T13:32:39.902 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:39 smithi122 bash[34674]: 2068ea4a75cf2b1415b87d71dff2cd339c1b37c67ab103db4f01c77f7528cda5 2024-04-25T13:32:39.902 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:39 smithi122 systemd[1]: Started Ceph mgr.z for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:32:39.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:39 smithi027 ceph-mon[29140]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:40.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:39 smithi045 ceph-mon[32731]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:40.184 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:39.900+0000 7f1323712200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T13:32:40.184 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:40.083+0000 7f1323712200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:32:40.185 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:40.184+0000 7f1323712200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T13:32:40.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_4 2024-04-25T13:32:40.545 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:32:40.809 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:40.490+0000 7f1323712200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T13:32:40.809 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:40.574+0000 7f1323712200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T13:32:40.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:40.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:40.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:40.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:40.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T13:32:40.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:40.985 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:40.986 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:40 smithi027 ceph-mon[29140]: Deploying daemon mgr.y on smithi045 2024-04-25T13:32:41.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:41.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:41.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:41.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:41.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T13:32:41.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:41.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:41.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:40 smithi122 ceph-mon[32843]: Deploying daemon mgr.y on smithi045 2024-04-25T13:32:41.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:41.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:41.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:41.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:41.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T13:32:41.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:41.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:41.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:40 smithi045 ceph-mon[32731]: Deploying daemon mgr.y on smithi045 2024-04-25T13:32:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 -- 172.21.15.27:0/3281354966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68104240 msgr2=0x7f6a68104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 --2- 172.21.15.27:0/3281354966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68104240 0x7f6a68104650 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009990 tx=0x7f6a64009ca0 comp rx=0 tx=0).stop 2024-04-25T13:32:41.994 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:41 smithi027 ceph-mon[29140]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:41.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 -- 172.21.15.27:0/3281354966 shutdown_connections 2024-04-25T13:32:41.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 --2- 172.21.15.27:0/3281354966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a68103110 0x7f6a68103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:41.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 --2- 172.21.15.27:0/3281354966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a68104be0 0x7f6a6810b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:41.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 --2- 172.21.15.27:0/3281354966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68104240 0x7f6a68104650 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:41.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 -- 172.21.15.27:0/3281354966 >> 172.21.15.27:0/3281354966 conn(0x7f6a680fe750 msgr2=0x7f6a68100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 -- 172.21.15.27:0/3281354966 shutdown_connections 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.993+0000 7f6a702d5700 1 -- 172.21.15.27:0/3281354966 wait complete. 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a702d5700 1 Processor -- start 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a702d5700 1 -- start start 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a702d5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68103110 0x7f6a68199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a702d5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a68104240 0x7f6a68199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a702d5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a68104be0 0x7f6a681a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a702d5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6810ec00 con 0x7f6a68103110 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a702d5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6810ea80 con 0x7f6a68104be0 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a702d5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6810ed80 con 0x7f6a68104240 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a6e071700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68103110 0x7f6a68199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a6e071700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68103110 0x7f6a68199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33026/0 (socket says 172.21.15.27:33026) 2024-04-25T13:32:41.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a6e071700 1 -- 172.21.15.27:0/2788037244 learned_addr learned my addr 172.21.15.27:0/2788037244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.994+0000 7f6a6d870700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a68104240 0x7f6a68199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a6e872700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a68104be0 0x7f6a681a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a6e071700 1 -- 172.21.15.27:0/2788037244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a68104240 msgr2=0x7f6a68199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a6e071700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a68104240 0x7f6a68199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a6e071700 1 -- 172.21.15.27:0/2788037244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a68104be0 msgr2=0x7f6a681a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a6e071700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a68104be0 0x7f6a681a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a6e071700 1 -- 172.21.15.27:0/2788037244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a681a12a0 con 0x7f6a68103110 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a6e071700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68103110 0x7f6a68199740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00d950 tx=0x7f6a5c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c017070 con 0x7f6a68103110 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a5c00f690 con 0x7f6a68103110 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c003aa0 con 0x7f6a68103110 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a681a1590 con 0x7f6a68103110 2024-04-25T13:32:41.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.995+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a681a1b00 con 0x7f6a68103110 2024-04-25T13:32:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.996+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f6a5c003c40 con 0x7f6a68103110 2024-04-25T13:32:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.996+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a68051190 con 0x7f6a68103110 2024-04-25T13:32:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.997+0000 7f6a5b7fe700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a5403d850 0x7f6a5403fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:32:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.997+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6a5c0512f0 con 0x7f6a68103110 2024-04-25T13:32:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.997+0000 7f6a6d870700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a5403d850 0x7f6a5403fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:32:42.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:41.997+0000 7f6a6d870700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a5403d850 0x7f6a5403fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400b5c0 tx=0x7f6a6401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:32:42.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:42.002+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a5c020030 con 0x7f6a68103110 2024-04-25T13:32:42.037 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:41 smithi045 ceph-mon[32731]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:42.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:41 smithi122 ceph-mon[32843]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:42.059 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:41 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:41.754+0000 7f1323712200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T13:32:42.059 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:41 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:41.923+0000 7f1323712200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T13:32:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:42.219+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6a68063ad0 con 0x7f6a5403d850 2024-04-25T13:32:42.288 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:42 smithi045 podman[34099]: 2024-04-25 13:32:42.252648511 +0000 UTC m=+0.326549525 container start d464e8b57aed8a456891f62e48f8cfbb434f178f6fa6eac35de0dbeb3a8ef1be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T13:32:42.288 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mgr[34119]: -- 172.21.15.45:0/1149790110 <== mon.2 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b5442625a0 con 0x55b548fec480 2024-04-25T13:32:42.400 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:42.098+0000 7f1323712200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T13:32:42.400 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:42.399+0000 7f1323712200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T13:32:42.578 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:42.418+0000 7f40bba4d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T13:32:42.579 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:42 smithi045 bash[34099]: d464e8b57aed8a456891f62e48f8cfbb434f178f6fa6eac35de0dbeb3a8ef1be 2024-04-25T13:32:42.579 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:42 smithi045 systemd[1]: Started Ceph mgr.y for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:32:42.702 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:42.494+0000 7f1323712200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T13:32:42.702 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:42.701+0000 7f1323712200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T13:32:42.898 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:42.622+0000 7f40bba4d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:32:42.898 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:42.734+0000 7f40bba4d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T13:32:43.058 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:42.917+0000 7f1323712200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:32:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:32:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:32:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:43.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:32:43.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:32:43.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:43.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:43.160 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:43 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:43.068+0000 7f40bba4d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T13:32:43.160 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:32:43.160 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:32:43.160 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:43.160 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.160 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.160 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.160 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:43.161 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:43.429 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:43 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:43.159+0000 7f40bba4d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T13:32:43.809 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:43.542+0000 7f1323712200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T13:32:44.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:43 smithi045 ceph-mon[32731]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:44.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:43 smithi045 ceph-mon[32731]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:44.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:43 smithi027 ceph-mon[29140]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:44.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:43 smithi027 ceph-mon[29140]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:44.286 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:43 smithi122 ceph-mon[32843]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:32:44.286 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:43 smithi122 ceph-mon[32843]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:44.287 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:43.980+0000 7f1323712200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T13:32:44.287 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:44 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:44.086+0000 7f1323712200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T13:32:44.287 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:44 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:44.286+0000 7f1323712200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T13:32:44.810 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:44 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:44.458+0000 7f40bba4d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T13:32:44.811 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:44 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:44.634+0000 7f40bba4d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T13:32:44.811 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:44 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:44.809+0000 7f40bba4d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T13:32:45.058 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:44 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:44.803+0000 7f1323712200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T13:32:45.059 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:44 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:44.895+0000 7f1323712200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T13:32:45.363 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:45.136+0000 7f1323712200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T13:32:45.363 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:45.268+0000 7f1323712200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T13:32:45.363 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:45.362+0000 7f1323712200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T13:32:45.363 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2942940940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24cfe666-151a-4e3b-b9eb-448622218f5e"}]: dispatch 2024-04-25T13:32:45.364 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2942940940' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24cfe666-151a-4e3b-b9eb-448622218f5e"}]': finished 2024-04-25T13:32:45.365 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T13:32:45.365 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T13:32:45.381 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:45.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:45.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:45.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T13:32:45.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2942940940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24cfe666-151a-4e3b-b9eb-448622218f5e"}]: dispatch 2024-04-25T13:32:45.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2942940940' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24cfe666-151a-4e3b-b9eb-448622218f5e"}]': finished 2024-04-25T13:32:45.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T13:32:45.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:32:45.412 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:45.107+0000 7f40bba4d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T13:32:45.412 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:45.201+0000 7f40bba4d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T13:32:45.412 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:45.411+0000 7f40bba4d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T13:32:45.412 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.412 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.412 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.412 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.412 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: Reconfiguring daemon mgr.x on smithi027 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2942940940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24cfe666-151a-4e3b-b9eb-448622218f5e"}]: dispatch 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2942940940' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24cfe666-151a-4e3b-b9eb-448622218f5e"}]': finished 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T13:32:45.413 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:32:45.795 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:45.477+0000 7f1323712200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T13:32:45.795 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:45 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:45.795+0000 7f1323712200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T13:32:45.828 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:45 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:45.608+0000 7f40bba4d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T13:32:46.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:46 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:46.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:46 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1103688239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:32:46.559 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:46 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:46.387+0000 7f1323712200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T13:32:46.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:46 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:46.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:46 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1103688239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:32:46.578 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:46.254+0000 7f40bba4d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T13:32:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:46 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:46 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1103688239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:32:46.998 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:46.692+0000 7f40bba4d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T13:32:46.999 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:46.790+0000 7f40bba4d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T13:32:46.999 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:46.997+0000 7f40bba4d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T13:32:47.059 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:32:46 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z[34689]: 2024-04-25T13:32:46.600+0000 7f1323712200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T13:32:47.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:47.315+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 14) ==== 98518+0+0 (secure 0 0 0) 0x7f6a5c01acd0 con 0x7f6a68103110 2024-04-25T13:32:47.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:47 smithi122 ceph-mon[32843]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:47.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:47 smithi122 ceph-mon[32843]: Standby manager daemon z started 2024-04-25T13:32:47.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:47 smithi122 ceph-mon[32843]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T13:32:47.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:47 smithi122 ceph-mon[32843]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T13:32:47.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:47 smithi122 ceph-mon[32843]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T13:32:47.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:47 smithi122 ceph-mon[32843]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T13:32:47.562 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-mon[32731]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:47.562 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-mon[32731]: Standby manager daemon z started 2024-04-25T13:32:47.562 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-mon[32731]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T13:32:47.562 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-mon[32731]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T13:32:47.562 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-mon[32731]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T13:32:47.562 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-mon[32731]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T13:32:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:47 smithi027 ceph-mon[29140]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:47 smithi027 ceph-mon[29140]: Standby manager daemon z started 2024-04-25T13:32:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:47 smithi027 ceph-mon[29140]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T13:32:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:47 smithi027 ceph-mon[29140]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T13:32:47.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:47 smithi027 ceph-mon[29140]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T13:32:47.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:47 smithi027 ceph-mon[29140]: from='mgr.? 172.21.15.122:0/3581552715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T13:32:47.828 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:47.561+0000 7f40bba4d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T13:32:47.828 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:47.656+0000 7f40bba4d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T13:32:48.266 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:47 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:47.909+0000 7f40bba4d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T13:32:48.267 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:48 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:48.046+0000 7f40bba4d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T13:32:48.267 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:48 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:48.145+0000 7f40bba4d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T13:32:48.267 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:48 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:48.266+0000 7f40bba4d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T13:32:48.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:48 smithi045 ceph-mon[32731]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T13:32:48.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T13:32:48.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:48 smithi027 ceph-mon[29140]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T13:32:48.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T13:32:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:48 smithi122 ceph-mon[32843]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T13:32:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T13:32:49.078 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:48 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:48.589+0000 7f40bba4d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T13:32:49.355 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:49 smithi027 ceph-mon[29140]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:49.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:49 smithi045 ceph-mon[32731]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:49.579 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:49 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:49.216+0000 7f40bba4d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T13:32:49.579 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:32:49 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y[34114]: 2024-04-25T13:32:49.437+0000 7f40bba4d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T13:32:49.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:49 smithi122 ceph-mon[32843]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:50.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:32:50.328+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f6a5c064040 con 0x7f6a68103110 2024-04-25T13:32:50.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:50 smithi027 ceph-mon[29140]: Standby manager daemon y started 2024-04-25T13:32:50.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:50 smithi027 ceph-mon[29140]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T13:32:50.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:50 smithi027 ceph-mon[29140]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T13:32:50.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:50 smithi027 ceph-mon[29140]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T13:32:50.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:50 smithi027 ceph-mon[29140]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T13:32:50.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:50 smithi045 ceph-mon[32731]: Standby manager daemon y started 2024-04-25T13:32:50.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:50 smithi045 ceph-mon[32731]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T13:32:50.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:50 smithi045 ceph-mon[32731]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T13:32:50.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:50 smithi045 ceph-mon[32731]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T13:32:50.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:50 smithi045 ceph-mon[32731]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T13:32:50.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:50 smithi122 ceph-mon[32843]: Standby manager daemon y started 2024-04-25T13:32:50.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:50 smithi122 ceph-mon[32843]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T13:32:50.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:50 smithi122 ceph-mon[32843]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T13:32:50.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:50 smithi122 ceph-mon[32843]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T13:32:50.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:50 smithi122 ceph-mon[32843]: from='mgr.? 172.21.15.45:0/388397749' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:51.609 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:51.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T13:32:51.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T13:32:51.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:51.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:51.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:51.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:51.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:51.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:32:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:53.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:53 smithi122 ceph-mon[32843]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:53.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:53 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T13:32:53.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:53 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:53.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:53 smithi122 ceph-mon[32843]: Deploying daemon osd.0 on smithi027 2024-04-25T13:32:53.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:53 smithi027 ceph-mon[29140]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:53.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T13:32:53.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:53.593 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:53 smithi027 ceph-mon[29140]: Deploying daemon osd.0 on smithi027 2024-04-25T13:32:53.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:53 smithi045 ceph-mon[32731]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:53.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:53 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T13:32:53.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:53 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:32:53.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:53 smithi045 ceph-mon[32731]: Deploying daemon osd.0 on smithi027 2024-04-25T13:32:55.596 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:55 smithi027 ceph-mon[29140]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:55.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:55 smithi122 ceph-mon[32843]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:55.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:55 smithi045 ceph-mon[32731]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:57.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:57 smithi027 ceph-mon[29140]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:57.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:57 smithi122 ceph-mon[32843]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:57.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:57 smithi045 ceph-mon[32731]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:59.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:59 smithi027 ceph-mon[29140]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:59.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:59 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:59.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:59 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:59.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:32:59 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:59.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:59 smithi122 ceph-mon[32843]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:59.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:59 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:59.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:59 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:59.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:32:59 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:32:59.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:59 smithi045 ceph-mon[32731]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:32:59.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:59 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:59.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:59 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:32:59.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:32:59 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:01.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:01 smithi027 ceph-mon[29140]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:01.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:01.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:01.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:01 smithi122 ceph-mon[32843]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:01.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:01.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:01.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:01 smithi045 ceph-mon[32731]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:01.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:01.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:01.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:01.975 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 0 on host 'smithi027' 2024-04-25T13:33:01.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.974+0000 7f6a5b7fe700 1 -- 172.21.15.27:0/2788037244 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6a68063ad0 con 0x7f6a5403d850 2024-04-25T13:33:01.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a5403d850 msgr2=0x7f6a5403fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:01.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a5403d850 0x7f6a5403fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400b5c0 tx=0x7f6a6401a040 comp rx=0 tx=0).stop 2024-04-25T13:33:01.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68103110 msgr2=0x7f6a68199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:01.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68103110 0x7f6a68199740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00d950 tx=0x7f6a5c00dc60 comp rx=0 tx=0).stop 2024-04-25T13:33:01.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 shutdown_connections 2024-04-25T13:33:01.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a5403d850 0x7f6a5403fd60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:01.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a68104be0 0x7f6a681a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:01.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a68103110 0x7f6a68199740 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:01.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 --2- 172.21.15.27:0/2788037244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a68104240 0x7f6a68199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:01.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 >> 172.21.15.27:0/2788037244 conn(0x7f6a680fe750 msgr2=0x7f6a68107470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:33:01.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 shutdown_connections 2024-04-25T13:33:01.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:01.976+0000 7f6a702d5700 1 -- 172.21.15.27:0/2788037244 wait complete. 2024-04-25T13:33:02.576 DEBUG:teuthology.orchestra.run.smithi027:osd.0> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T13:33:02.579 INFO:tasks.cephadm:Deploying osd.1 on smithi045 with /dev/vg_nvme/lv_4... 2024-04-25T13:33:02.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T13:33:02.835 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T13:33:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:03.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:02 smithi027 ceph-mon[29140]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T13:33:03.267 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:02 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:03.267 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:02 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:03.267 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:02 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:03.267 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:02 smithi045 ceph-mon[32731]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T13:33:03.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:02 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:03.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:02 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:03.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:02 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:03.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:02 smithi122 ceph-mon[32843]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T13:33:04.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:03 smithi027 ceph-mon[29140]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:04.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:03 smithi027 ceph-mon[29140]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T13:33:04.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:03 smithi027 ceph-mon[29140]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T13:33:04.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:03 smithi027 ceph-mon[29140]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:33:04.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:04.153 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0[41693]: 2024-04-25T13:33:03.973+0000 7fdae79e1700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T13:33:04.154 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0[41693]: 2024-04-25T13:33:03.976+0000 7fdae3796700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:33:04.174 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:03 smithi045 ceph-mon[32731]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:04.175 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:03 smithi045 ceph-mon[32731]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T13:33:04.175 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:03 smithi045 ceph-mon[32731]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T13:33:04.175 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:03 smithi045 ceph-mon[32731]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:33:04.175 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:04.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:03 smithi122 ceph-mon[32843]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:04.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:03 smithi122 ceph-mon[32843]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T13:33:04.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:03 smithi122 ceph-mon[32843]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T13:33:04.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:03 smithi122 ceph-mon[32843]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:33:04.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:04.837 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T13:33:04.837 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T13:33:04.837 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-04-25T13:33:04.837 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-04-25T13:33:04.837 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00840079 s, 1.2 GB/s 2024-04-25T13:33:04.837 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-04-25T13:33:05.101 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:04 smithi045 ceph-mon[32731]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:33:05.101 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:04 smithi045 ceph-mon[32731]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T13:33:05.101 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:04 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:05.102 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:04 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:05.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:04 smithi122 ceph-mon[32843]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:33:05.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:04 smithi122 ceph-mon[32843]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T13:33:05.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:04 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:05.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:04 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:05.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:04 smithi027 ceph-mon[29140]: from='osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:33:05.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:04 smithi027 ceph-mon[29140]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T13:33:05.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:04 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:05.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:04 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:05.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2024-04-25T13:33:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] boot 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:06.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:06.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] boot 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:33:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:33:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T13:33:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: osd.0 [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] boot 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:07.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:06 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi027 2024-04-25T13:33:07.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:06 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T13:33:07.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:06 smithi045 ceph-mon[32731]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T13:33:07.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:06 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi027 2024-04-25T13:33:07.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:06 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T13:33:07.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:06 smithi122 ceph-mon[32843]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 -- 172.21.15.45:0/1417628293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 msgr2=0x7f9654106d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1417628293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 0x7f9654106d80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9644009990 tx=0x7f9644009ca0 comp rx=0 tx=0).stop 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 -- 172.21.15.45:0/1417628293 shutdown_connections 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1417628293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9654109ec0 0x7f965410c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1417628293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 0x7f9654106d80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1417628293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9654107350 0x7f9654109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 -- 172.21.15.45:0/1417628293 >> 172.21.15.45:0/1417628293 conn(0x7f96540fe750 msgr2=0x7f9654100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 -- 172.21.15.45:0/1417628293 shutdown_connections 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.320+0000 7f965a0b0700 1 -- 172.21.15.45:0/1417628293 wait complete. 2024-04-25T13:33:07.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.321+0000 7f965a0b0700 1 Processor -- start 2024-04-25T13:33:07.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.321+0000 7f965a0b0700 1 -- start start 2024-04-25T13:33:07.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.321+0000 7f965a0b0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 0x7f965419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.321+0000 7f965a0b0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9654107350 0x7f965419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.321+0000 7f965a0b0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9654109ec0 0x7f96541a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.321+0000 7f965a0b0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f965410fc60 con 0x7f9654107350 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f965a0b0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f965410fae0 con 0x7f965406c1a0 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f965a0b0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f965410fde0 con 0x7f9654109ec0 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 0x7f965419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 0x7f965419a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42644/0 (socket says 172.21.15.45:42644) 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 -- 172.21.15.45:0/1660866922 learned_addr learned my addr 172.21.15.45:0/1660866922 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f9652ffd700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9654107350 0x7f965419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f9653fff700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9654109ec0 0x7f96541a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 -- 172.21.15.45:0/1660866922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9654109ec0 msgr2=0x7f96541a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9654109ec0 0x7f96541a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:07.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 -- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9654107350 msgr2=0x7f965419acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9654107350 0x7f965419acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 -- 172.21.15.45:0/1660866922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96541a22e0 con 0x7f965406c1a0 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f96537fe700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 0x7f965419a780 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9644006010 tx=0x7f96440047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f9650ff9700 1 -- 172.21.15.45:0/1660866922 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644027070 con 0x7f965406c1a0 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f9650ff9700 1 -- 172.21.15.45:0/1660866922 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96440040c0 con 0x7f965406c1a0 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f9650ff9700 1 -- 172.21.15.45:0/1660866922 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644022d90 con 0x7f965406c1a0 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96541a2570 con 0x7f965406c1a0 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.322+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96541a29c0 con 0x7f965406c1a0 2024-04-25T13:33:07.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.323+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9654051190 con 0x7f965406c1a0 2024-04-25T13:33:07.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.325+0000 7f9650ff9700 1 -- 172.21.15.45:0/1660866922 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f964402a070 con 0x7f965406c1a0 2024-04-25T13:33:07.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.325+0000 7f9650ff9700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f963c0afb70 0x7f963c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:07.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.326+0000 7f9650ff9700 1 -- 172.21.15.45:0/1660866922 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f96440de1b0 con 0x7f965406c1a0 2024-04-25T13:33:07.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.326+0000 7f9652ffd700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f963c0afb70 0x7f963c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:07.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.326+0000 7f9652ffd700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f963c0afb70 0x7f963c0b2080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f965419b830 tx=0x7f964800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:33:07.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.330+0000 7f9650ff9700 1 -- 172.21.15.45:0/1660866922 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9644013070 con 0x7f965406c1a0 2024-04-25T13:33:07.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:06 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi027 2024-04-25T13:33:07.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:06 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi027 to 17088M 2024-04-25T13:33:07.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:06 smithi027 ceph-mon[29140]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T13:33:07.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:07.546+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9654063ad0 con 0x7f963c0afb70 2024-04-25T13:33:08.279 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:07 smithi045 ceph-mon[32731]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:08.279 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:08.279 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:08.279 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:08.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:07 smithi122 ceph-mon[32843]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:08.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:08.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:08.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:08.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:07 smithi027 ceph-mon[29140]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:08.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:08.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:08.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:09.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:08 smithi122 ceph-mon[32843]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:09.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:08 smithi045 ceph-mon[32731]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:09.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:08 smithi027 ceph-mon[29140]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:10.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:09 smithi122 ceph-mon[32843]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:10.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:09 smithi045 ceph-mon[32731]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:10.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:09 smithi027 ceph-mon[29140]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:11.303 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:10 smithi027 ceph-mon[29140]: from='client.? 172.21.15.45:0/4225891572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]: dispatch 2024-04-25T13:33:11.303 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:10 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]: dispatch 2024-04-25T13:33:11.303 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:10 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]': finished 2024-04-25T13:33:11.303 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:10 smithi027 ceph-mon[29140]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T13:33:11.303 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:11.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:10 smithi122 ceph-mon[32843]: from='client.? 172.21.15.45:0/4225891572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]: dispatch 2024-04-25T13:33:11.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:10 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]: dispatch 2024-04-25T13:33:11.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:10 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]': finished 2024-04-25T13:33:11.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:10 smithi122 ceph-mon[32843]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T13:33:11.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:10 smithi045 ceph-mon[32731]: from='client.? 172.21.15.45:0/4225891572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]: dispatch 2024-04-25T13:33:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:10 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]: dispatch 2024-04-25T13:33:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:10 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f666f7a-74de-468f-8d21-f9aaeab6d18e"}]': finished 2024-04-25T13:33:11.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:10 smithi045 ceph-mon[32731]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T13:33:11.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:12.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:11 smithi122 ceph-mon[32843]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:12.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:11 smithi122 ceph-mon[32843]: from='client.? 172.21.15.45:0/2431402932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:33:12.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:11 smithi045 ceph-mon[32731]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:12.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:11 smithi045 ceph-mon[32731]: from='client.? 172.21.15.45:0/2431402932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:33:12.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:11 smithi027 ceph-mon[29140]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:12.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:11 smithi027 ceph-mon[29140]: from='client.? 172.21.15.45:0/2431402932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:33:14.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:14 smithi122 ceph-mon[32843]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:14.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:14 smithi045 ceph-mon[32731]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:14.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:14 smithi027 ceph-mon[29140]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:16.285 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:16 smithi045 ceph-mon[32731]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:16 smithi122 ceph-mon[32843]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:16 smithi027 ceph-mon[29140]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:18.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:18 smithi122 ceph-mon[32843]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:18.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:18 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T13:33:18.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:18 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:18.310 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:18 smithi045 ceph-mon[32731]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:18.310 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:18 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T13:33:18.310 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:18 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:18 smithi027 ceph-mon[29140]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:18 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T13:33:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:18 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:19.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:19 smithi122 ceph-mon[32843]: Deploying daemon osd.1 on smithi045 2024-04-25T13:33:19.313 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:19 smithi045 ceph-mon[32731]: Deploying daemon osd.1 on smithi045 2024-04-25T13:33:19.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:19 smithi027 ceph-mon[29140]: Deploying daemon osd.1 on smithi045 2024-04-25T13:33:20.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:20 smithi122 ceph-mon[32843]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:20.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:20 smithi045 ceph-mon[32731]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:20.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:20 smithi027 ceph-mon[29140]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:22.264 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:22 smithi045 ceph-mon[32731]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:22.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:22 smithi122 ceph-mon[32843]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:22.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:22 smithi027 ceph-mon[29140]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:24.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:24 smithi045 ceph-mon[32731]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:24.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:24.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:24.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:24 smithi122 ceph-mon[32843]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:24 smithi027 ceph-mon[29140]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:24.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:26 smithi122 ceph-mon[32843]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:26 smithi045 ceph-mon[32731]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:26.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:26.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:26.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:26 smithi027 ceph-mon[29140]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:26.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:26.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:26.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:26.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.864+0000 7f9650ff9700 1 -- 172.21.15.45:0/1660866922 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9654063ad0 con 0x7f963c0afb70 2024-04-25T13:33:26.866 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 1 on host 'smithi045' 2024-04-25T13:33:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f963c0afb70 msgr2=0x7f963c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f963c0afb70 0x7f963c0b2080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f965419b830 tx=0x7f964800b410 comp rx=0 tx=0).stop 2024-04-25T13:33:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 msgr2=0x7f965419a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 0x7f965419a780 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9644006010 tx=0x7f96440047b0 comp rx=0 tx=0).stop 2024-04-25T13:33:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 shutdown_connections 2024-04-25T13:33:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f963c0afb70 0x7f963c0b2080 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:26.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9654109ec0 0x7f96541a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965406c1a0 0x7f965419a780 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 --2- 172.21.15.45:0/1660866922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9654107350 0x7f965419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 >> 172.21.15.45:0/1660866922 conn(0x7f96540fe750 msgr2=0x7f9654105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:33:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 shutdown_connections 2024-04-25T13:33:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:33:26.867+0000 7f965a0b0700 1 -- 172.21.15.45:0/1660866922 wait complete. 2024-04-25T13:33:27.512 DEBUG:teuthology.orchestra.run.smithi045:osd.1> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T13:33:27.514 INFO:tasks.cephadm:Deploying osd.2 on smithi122 with /dev/vg_nvme/lv_4... 2024-04-25T13:33:27.514 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T13:33:27.760 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:33:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:27 smithi045 ceph-mon[32731]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:27 smithi045 ceph-mon[32731]: from='osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T13:33:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:27 smithi045 ceph-mon[32731]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T13:33:28.121 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:27 smithi122 ceph-mon[32843]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:28.122 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:28.122 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:28.122 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:28.122 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:27 smithi122 ceph-mon[32843]: from='osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T13:33:28.122 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:27 smithi122 ceph-mon[32843]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T13:33:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:27 smithi027 ceph-mon[29140]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:27 smithi027 ceph-mon[29140]: from='osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T13:33:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:27 smithi027 ceph-mon[29140]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T13:33:29.209 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:29 smithi122 ceph-mon[32843]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T13:33:29.209 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:29 smithi122 ceph-mon[32843]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T13:33:29.209 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:29.210 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:29 smithi122 ceph-mon[32843]: from='osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:33:29.210 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:29 smithi122 ceph-mon[32843]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:33:29.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:29 smithi045 ceph-mon[32731]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T13:33:29.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:29 smithi045 ceph-mon[32731]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T13:33:29.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:29.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:29 smithi045 ceph-mon[32731]: from='osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:33:29.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:29 smithi045 ceph-mon[32731]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:33:29.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:29 smithi027 ceph-mon[29140]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T13:33:29.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:29 smithi027 ceph-mon[29140]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T13:33:29.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:29.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:29 smithi027 ceph-mon[29140]: from='osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:33:29.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:29 smithi027 ceph-mon[29140]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:33:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T13:33:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T13:33:29.783 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-25T13:33:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-25T13:33:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109335 s, 959 MB/s 2024-04-25T13:33:29.783 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 2024-04-25T13:33:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-25T13:33:29.806 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:29 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1[37314]: 2024-04-25T13:33:29.526+0000 7fbe9f24e700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T13:33:29.806 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:29 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1[37314]: 2024-04-25T13:33:29.529+0000 7fbe96ffb700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:33:30.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:30 smithi045 ceph-mon[32731]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:30 smithi045 ceph-mon[32731]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:33:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:30 smithi045 ceph-mon[32731]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T13:33:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:30 smithi045 ceph-mon[32731]: from='osd.1 ' entity='osd.1' 2024-04-25T13:33:30.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:30 smithi122 ceph-mon[32843]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:30.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:30 smithi122 ceph-mon[32843]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:33:30.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:30 smithi122 ceph-mon[32843]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T13:33:30.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:30.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:30.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:30 smithi122 ceph-mon[32843]: from='osd.1 ' entity='osd.1' 2024-04-25T13:33:30.357 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2024-04-25T13:33:30.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:30 smithi027 ceph-mon[29140]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:30.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:30 smithi027 ceph-mon[29140]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:33:30.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:30 smithi027 ceph-mon[29140]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T13:33:30.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:30.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:30.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:30 smithi027 ceph-mon[29140]: from='osd.1 ' entity='osd.1' 2024-04-25T13:33:30.598 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:33:31.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:33:31.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:33:31.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:31.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] boot 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] boot 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:33:31.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: osd.1 [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] boot 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T13:33:31.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:32.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.185+0000 7f2bd7456700 1 -- 172.21.15.122:0/2850539291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd0104210 msgr2=0x7f2bd0104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:32.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.185+0000 7f2bd7456700 1 --2- 172.21.15.122:0/2850539291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd0104210 0x7f2bd0104620 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009990 tx=0x7f2bc0009ca0 comp rx=0 tx=0).stop 2024-04-25T13:33:32.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 -- 172.21.15.122:0/2850539291 shutdown_connections 2024-04-25T13:33:32.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 --2- 172.21.15.122:0/2850539291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2bd0104bb0 0x7f2bd010b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:32.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 --2- 172.21.15.122:0/2850539291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bd01030e0 0x7f2bd01034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:32.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 --2- 172.21.15.122:0/2850539291 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd0104210 0x7f2bd0104620 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:32.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 -- 172.21.15.122:0/2850539291 >> 172.21.15.122:0/2850539291 conn(0x7f2bd00fe740 msgr2=0x7f2bd0100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 -- 172.21.15.122:0/2850539291 shutdown_connections 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 -- 172.21.15.122:0/2850539291 wait complete. 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 Processor -- start 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.186+0000 7f2bd7456700 1 -- start start 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd7456700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd01030e0 0x7f2bd019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd7456700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2bd0104210 0x7f2bd019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd7456700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bd0104bb0 0x7f2bd01a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd7456700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd010eb60 con 0x7f2bd0104bb0 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd7456700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd010e9e0 con 0x7f2bd0104210 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd7456700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd010ece0 con 0x7f2bd01030e0 2024-04-25T13:33:32.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd01030e0 0x7f2bd019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd01030e0 0x7f2bd019a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:60992/0 (socket says 172.21.15.122:60992) 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 -- 172.21.15.122:0/298157946 learned_addr learned my addr 172.21.15.122:0/298157946 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd59f3700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bd0104bb0 0x7f2bd01a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd49f1700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2bd0104210 0x7f2bd019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 -- 172.21.15.122:0/298157946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2bd0104210 msgr2=0x7f2bd019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2bd0104210 0x7f2bd019ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 -- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bd0104bb0 msgr2=0x7f2bd01a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bd0104bb0 0x7f2bd01a1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 -- 172.21.15.122:0/298157946 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd01a2320 con 0x7f2bd01030e0 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd59f3700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bd0104bb0 0x7f2bd01a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd51f2700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd01030e0 0x7f2bd019a7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc007f10 tx=0x7f2bcc00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:33:32.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bc67fc700 1 -- 172.21.15.122:0/298157946 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc00dbd0 con 0x7f2bd01030e0 2024-04-25T13:33:32.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bc67fc700 1 -- 172.21.15.122:0/298157946 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bcc00dd70 con 0x7f2bd01030e0 2024-04-25T13:33:32.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bc67fc700 1 -- 172.21.15.122:0/298157946 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc0133f0 con 0x7f2bd01030e0 2024-04-25T13:33:32.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.187+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd01a2610 con 0x7f2bd01030e0 2024-04-25T13:33:32.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.188+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd01a2ba0 con 0x7f2bd01030e0 2024-04-25T13:33:32.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.189+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bd0051190 con 0x7f2bd01030e0 2024-04-25T13:33:32.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.190+0000 7f2bc67fc700 1 -- 172.21.15.122:0/298157946 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f2bcc01a020 con 0x7f2bd01030e0 2024-04-25T13:33:32.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.190+0000 7f2bc67fc700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2bbc0afb70 0x7f2bbc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:32.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.191+0000 7f2bc67fc700 1 -- 172.21.15.122:0/298157946 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f2bcc0dcba0 con 0x7f2bd01030e0 2024-04-25T13:33:32.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.191+0000 7f2bd49f1700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2bbc0afb70 0x7f2bbc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:32.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.191+0000 7f2bd49f1700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2bbc0afb70 0x7f2bbc0b2080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0006010 tx=0x7f2bc000b580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:33:32.195 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.195+0000 7f2bc67fc700 1 -- 172.21.15.122:0/298157946 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bcc010070 con 0x7f2bd01030e0 2024-04-25T13:33:32.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:32 smithi122 ceph-mon[32843]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:32.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:32 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi045 2024-04-25T13:33:32.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:32 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi045 to 17093M 2024-04-25T13:33:32.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:32 smithi045 ceph-mon[32731]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:32.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:32 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi045 2024-04-25T13:33:32.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:32 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi045 to 17093M 2024-04-25T13:33:32.399 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:32.399+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2bd006bbf0 con 0x7f2bbc0afb70 2024-04-25T13:33:32.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:32 smithi027 ceph-mon[29140]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T13:33:32.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:32 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi045 2024-04-25T13:33:32.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:32 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi045 to 17093M 2024-04-25T13:33:33.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:33 smithi122 ceph-mon[32843]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T13:33:33.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:33.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:33.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:33 smithi045 ceph-mon[32731]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T13:33:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:33 smithi027 ceph-mon[29140]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T13:33:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:34.181 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:34 smithi122 ceph-mon[32843]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:34.181 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:34 smithi122 ceph-mon[32843]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:34 smithi045 ceph-mon[32731]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:34.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:34 smithi045 ceph-mon[32731]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:34.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:34 smithi027 ceph-mon[29140]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:34.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:34 smithi027 ceph-mon[29140]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:36.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:36 smithi045 ceph-mon[32731]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:36.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:36 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/1447553311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]: dispatch 2024-04-25T13:33:36.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:36 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]: dispatch 2024-04-25T13:33:36.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:36 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]': finished 2024-04-25T13:33:36.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:36 smithi045 ceph-mon[32731]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T13:33:36.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:36 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:36.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:36 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/578525950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:33:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:36 smithi027 ceph-mon[29140]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:36 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/1447553311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]: dispatch 2024-04-25T13:33:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:36 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]: dispatch 2024-04-25T13:33:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:36 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]': finished 2024-04-25T13:33:36.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:36 smithi027 ceph-mon[29140]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T13:33:36.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:36 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:36.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:36 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/578525950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:33:36.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:36 smithi122 ceph-mon[32843]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:36.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:36 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/1447553311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]: dispatch 2024-04-25T13:33:36.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:36 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]: dispatch 2024-04-25T13:33:36.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:36 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9efabc5-f820-4cd2-9e55-956219ccb103"}]': finished 2024-04-25T13:33:36.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:36 smithi122 ceph-mon[32843]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T13:33:36.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:36 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:36.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:36 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/578525950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:33:38.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:38 smithi045 ceph-mon[32731]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:38.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:38 smithi027 ceph-mon[29140]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:38.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:38 smithi122 ceph-mon[32843]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:40.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:40 smithi045 ceph-mon[32731]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:40.356 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:40 smithi122 ceph-mon[32843]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:40.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:40 smithi027 ceph-mon[29140]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:42.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:42 smithi045 ceph-mon[32731]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:42.366 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:42 smithi122 ceph-mon[32843]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:42.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:42 smithi027 ceph-mon[29140]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:43.269 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:43 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T13:33:43.269 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:43 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:43.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:43 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T13:33:43.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:43 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:43.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T13:33:43.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:44.208 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:44 smithi122 ceph-mon[32843]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:44.208 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:44 smithi122 ceph-mon[32843]: Deploying daemon osd.2 on smithi122 2024-04-25T13:33:44.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:44 smithi045 ceph-mon[32731]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:44.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:44 smithi045 ceph-mon[32731]: Deploying daemon osd.2 on smithi122 2024-04-25T13:33:44.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:44 smithi027 ceph-mon[29140]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:44.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:44 smithi027 ceph-mon[29140]: Deploying daemon osd.2 on smithi122 2024-04-25T13:33:45.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:45 smithi027 ceph-mon[29140]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:45.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:45 smithi122 ceph-mon[32843]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:45.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:45 smithi045 ceph-mon[32731]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:47.531 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:47 smithi122 ceph-mon[32843]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:47.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:47 smithi045 ceph-mon[32731]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:47 smithi027 ceph-mon[29140]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:49.028 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:49.028 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:49.029 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:49.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:49.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:49.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:49.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:49.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:49.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:50.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:49 smithi122 ceph-mon[32843]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:50.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:49 smithi045 ceph-mon[32731]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:50.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:49 smithi027 ceph-mon[29140]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:51.506 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:51 smithi122 ceph-mon[32843]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:51.506 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:51.506 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:51.506 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:51.506 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:51.506 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:51.574 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.572+0000 7f2bc67fc700 1 -- 172.21.15.122:0/298157946 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2bd006bbf0 con 0x7f2bbc0afb70 2024-04-25T13:33:51.574 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 2 on host 'smithi122' 2024-04-25T13:33:51.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2bbc0afb70 msgr2=0x7f2bbc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:51.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2bbc0afb70 0x7f2bbc0b2080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0006010 tx=0x7f2bc000b580 comp rx=0 tx=0).stop 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd01030e0 msgr2=0x7f2bd019a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd01030e0 0x7f2bd019a7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc007f10 tx=0x7f2bcc00d3b0 comp rx=0 tx=0).stop 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 shutdown_connections 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2bbc0afb70 0x7f2bbc0b2080 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2bd01030e0 0x7f2bd019a7c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bd0104bb0 0x7f2bd01a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 --2- 172.21.15.122:0/298157946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2bd0104210 0x7f2bd019ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 >> 172.21.15.122:0/298157946 conn(0x7f2bd00fe740 msgr2=0x7f2bd006b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 shutdown_connections 2024-04-25T13:33:51.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:33:51.574+0000 7f2bd7456700 1 -- 172.21.15.122:0/298157946 wait complete. 2024-04-25T13:33:51.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:51 smithi045 ceph-mon[32731]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:51.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:51.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:51.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:51.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:51.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:51.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:51 smithi027 ceph-mon[29140]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:51.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:51.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:51.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:51.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:51.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:52.294 DEBUG:teuthology.orchestra.run.smithi122:osd.2> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T13:33:52.297 INFO:tasks.cephadm:Deploying osd.3 on smithi027 with /dev/vg_nvme/lv_3... 2024-04-25T13:33:52.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T13:33:52.556 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:33:52.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:52 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:52.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:52 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:52.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:52 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:52.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:52 smithi027 ceph-mon[29140]: from='osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T13:33:52.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:52 smithi027 ceph-mon[29140]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T13:33:52.795 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:52 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:52.795 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:52 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:52.796 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:52 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:52.796 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:52 smithi122 ceph-mon[32843]: from='osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T13:33:52.796 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:52 smithi122 ceph-mon[32843]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T13:33:52.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:52 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:52.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:52 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:52.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:52 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:33:52.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:52 smithi045 ceph-mon[32731]: from='osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T13:33:52.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:52 smithi045 ceph-mon[32731]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T13:33:53.764 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:53 smithi122 ceph-mon[32843]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:53.765 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:53 smithi122 ceph-mon[32843]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T13:33:53.765 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:53 smithi122 ceph-mon[32843]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T13:33:53.765 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:53 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:53.765 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:53 smithi122 ceph-mon[32843]: from='osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:33:53.765 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:53 smithi122 ceph-mon[32843]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:33:53.765 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:53 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2[37980]: 2024-04-25T13:33:53.562+0000 7f9520aae700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T13:33:53.765 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:53 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2[37980]: 2024-04-25T13:33:53.566+0000 7f951c863700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:33:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:53 smithi045 ceph-mon[32731]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:53 smithi045 ceph-mon[32731]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T13:33:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:53 smithi045 ceph-mon[32731]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T13:33:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:53 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:53.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:53 smithi045 ceph-mon[32731]: from='osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:33:53.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:53 smithi045 ceph-mon[32731]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:33:53.836 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:53 smithi027 ceph-mon[29140]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:53.836 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:53 smithi027 ceph-mon[29140]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T13:33:53.836 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:53 smithi027 ceph-mon[29140]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T13:33:53.836 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:53.836 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:53 smithi027 ceph-mon[29140]: from='osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:33:53.836 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:53 smithi027 ceph-mon[29140]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:33:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T13:33:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T13:33:54.557 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T13:33:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T13:33:54.557 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111416 s, 941 MB/s 2024-04-25T13:33:54.557 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T13:33:54.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:54 smithi122 ceph-mon[32843]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:33:54.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:54 smithi122 ceph-mon[32843]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T13:33:54.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:54 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:54.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:54 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:54.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:54 smithi045 ceph-mon[32731]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:33:54.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:54 smithi045 ceph-mon[32731]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T13:33:54.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:54 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:54.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:54 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:54.872 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:54 smithi027 ceph-mon[29140]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:33:54.872 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:54 smithi027 ceph-mon[29140]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T13:33:54.872 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:54 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:54.872 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:54 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:55.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_3 2024-04-25T13:33:55.543 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:33:55.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] boot 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:55.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] boot 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:55.855 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: osd.2 [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] boot 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:56.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:33:56.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:56.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:56.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:33:56.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:56.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:56 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi122 2024-04-25T13:33:56.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:56 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi122 to 17088M 2024-04-25T13:33:56.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:56 smithi045 ceph-mon[32731]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T13:33:56.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:56 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:33:56.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:56 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi122 2024-04-25T13:33:56.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:56 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi122 to 17088M 2024-04-25T13:33:56.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:56 smithi027 ceph-mon[29140]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T13:33:56.881 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:56 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:33:57.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:56 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi122 2024-04-25T13:33:57.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:56 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi122 to 17088M 2024-04-25T13:33:57.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:56 smithi122 ceph-mon[32843]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T13:33:57.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:56 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:33:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.098+0000 7f554528e700 1 -- 172.21.15.27:0/382225648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540104210 msgr2=0x7f5540104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.098+0000 7f554528e700 1 --2- 172.21.15.27:0/382225648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540104210 0x7f5540104620 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5530009990 tx=0x7f5530009ca0 comp rx=0 tx=0).stop 2024-04-25T13:33:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 -- 172.21.15.27:0/382225648 shutdown_connections 2024-04-25T13:33:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 --2- 172.21.15.27:0/382225648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5540105ce0 0x7f554010c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 --2- 172.21.15.27:0/382225648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540104210 0x7f5540104620 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:57.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 --2- 172.21.15.27:0/382225648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5540105340 0x7f5540105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 -- 172.21.15.27:0/382225648 >> 172.21.15.27:0/382225648 conn(0x7f55400ff870 msgr2=0x7f5540101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:33:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 -- 172.21.15.27:0/382225648 shutdown_connections 2024-04-25T13:33:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 -- 172.21.15.27:0/382225648 wait complete. 2024-04-25T13:33:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 Processor -- start 2024-04-25T13:33:57.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.099+0000 7f554528e700 1 -- start start 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f554528e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5540104210 0x7f554019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f554528e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540105340 0x7f554019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f554528e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5540105ce0 0x7f55401a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f554528e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f554010fc80 con 0x7f5540105340 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f554528e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f554010fb00 con 0x7f5540104210 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f554528e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f554010fe00 con 0x7f5540105ce0 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540105340 0x7f554019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540105340 0x7f554019ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47962/0 (socket says 172.21.15.27:47962) 2024-04-25T13:33:57.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 -- 172.21.15.27:0/2475275634 learned_addr learned my addr 172.21.15.27:0/2475275634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553f7fe700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5540105ce0 0x7f55401a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553effd700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5540104210 0x7f554019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 -- 172.21.15.27:0/2475275634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5540105ce0 msgr2=0x7f55401a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5540105ce0 0x7f55401a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 -- 172.21.15.27:0/2475275634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5540104210 msgr2=0x7f554019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5540104210 0x7f554019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 -- 172.21.15.27:0/2475275634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55401a2330 con 0x7f5540105340 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553f7fe700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5540105ce0 0x7f55401a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.100+0000 7f553e7fc700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540105340 0x7f554019ad10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f553400c920 tx=0x7f553400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:33:57.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.101+0000 7f5527fff700 1 -- 172.21.15.27:0/2475275634 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534007b00 con 0x7f5540105340 2024-04-25T13:33:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.101+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55401a2620 con 0x7f5540105340 2024-04-25T13:33:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.101+0000 7f5527fff700 1 -- 172.21.15.27:0/2475275634 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5534004ce0 con 0x7f5540105340 2024-04-25T13:33:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.101+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55401a2bb0 con 0x7f5540105340 2024-04-25T13:33:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.101+0000 7f5527fff700 1 -- 172.21.15.27:0/2475275634 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534005650 con 0x7f5540105340 2024-04-25T13:33:57.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.102+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5540051190 con 0x7f5540105340 2024-04-25T13:33:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.103+0000 7f5527fff700 1 -- 172.21.15.27:0/2475275634 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f5534007ca0 con 0x7f5540105340 2024-04-25T13:33:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.104+0000 7f5527fff700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55280afaf0 0x7f55280b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:33:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.104+0000 7f553effd700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55280afaf0 0x7f55280b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:33:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.104+0000 7f5527fff700 1 -- 172.21.15.27:0/2475275634 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7f55340dd6d0 con 0x7f5540105340 2024-04-25T13:33:57.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.105+0000 7f553effd700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55280afaf0 0x7f55280b2000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f55300038a0 tx=0x7f553001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:33:57.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.108+0000 7f5527fff700 1 -- 172.21.15.27:0/2475275634 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5534014030 con 0x7f5540105340 2024-04-25T13:33:57.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:57.325+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f554002e960 con 0x7f55280afaf0 2024-04-25T13:33:57.849 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46177]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T13:33:57.849 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46177]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:57.849 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46177]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:57.849 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46177]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:57.849 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46180]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T13:33:57.849 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46180]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:57.849 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46180]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:57.849 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46180]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:57.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 ceph-mon[29140]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:33:57.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T13:33:57.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 ceph-mon[29140]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T13:33:57.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:33:57.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:57.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:57.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:57.891 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 ceph-mon[32843]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:33:57.891 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T13:33:57.891 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 ceph-mon[32843]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T13:33:57.891 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:33:57.891 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:57.891 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:57.891 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:57.891 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41755]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T13:33:57.891 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41755]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:57.891 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41755]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:57.892 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41755]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:57.892 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41758]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T13:33:57.892 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41758]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:57.892 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41758]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:57.892 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41758]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:57.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41450]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T13:33:57.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41450]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:57.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41450]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:57.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41450]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:57.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41453]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T13:33:57.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41453]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:57.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41453]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:57.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41453]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:57.930 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 ceph-mon[32731]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:33:57.930 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T13:33:57.930 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 ceph-mon[32731]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T13:33:57.930 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:33:57.930 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:33:57.930 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:33:57.930 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:33:57.930 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41456]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T13:33:58.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46218]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T13:33:58.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46218]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:58.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46218]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:58.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46218]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:58.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46235]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T13:33:58.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46235]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:58.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46235]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:58.114 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:57 smithi027 sudo[46235]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:58.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41761]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T13:33:58.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41761]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:58.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41761]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:58.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41761]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:58.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41764]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T13:33:58.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41764]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:58.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41764]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:58.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:57 smithi122 sudo[41764]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:58.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41456]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:58.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41456]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:58.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41456]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:58.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41459]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T13:33:58.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41459]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T13:33:58.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41459]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T13:33:58.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:57 smithi045 sudo[41459]: pam_unix(sudo:session): session closed for user root 2024-04-25T13:33:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:33:58.847+0000 7f5527fff700 1 -- 172.21.15.27:0/2475275634 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f55340a3300 con 0x7f5540105340 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:58.849 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:58.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:58.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:58.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:58.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:58.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:58.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:58.850 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:58 smithi027 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:59.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:59.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T13:33:59.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T13:33:59.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:59.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:59.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:59.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:59.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:58 smithi122 ceph-mon[32843]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:59.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T13:33:59.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:58 smithi045 ceph-mon[32731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T13:33:59.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:59 smithi027 ceph-mon[29140]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:33:59.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:59 smithi027 ceph-mon[29140]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T13:33:59.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:33:59 smithi027 ceph-mon[29140]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T13:34:00.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:59 smithi122 ceph-mon[32843]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:00.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:59 smithi122 ceph-mon[32843]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T13:34:00.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:33:59 smithi122 ceph-mon[32843]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T13:34:00.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:59 smithi045 ceph-mon[32731]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:00.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:59 smithi045 ceph-mon[32731]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T13:34:00.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:33:59 smithi045 ceph-mon[32731]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T13:34:00.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:00 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2920427056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0b3dc7e-9e5d-4cee-8395-0a83219ee52d"}]: dispatch 2024-04-25T13:34:00.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:00 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2920427056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0b3dc7e-9e5d-4cee-8395-0a83219ee52d"}]': finished 2024-04-25T13:34:00.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:00 smithi027 ceph-mon[29140]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T13:34:00.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:00 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:01.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:00 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2920427056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0b3dc7e-9e5d-4cee-8395-0a83219ee52d"}]: dispatch 2024-04-25T13:34:01.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:00 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2920427056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0b3dc7e-9e5d-4cee-8395-0a83219ee52d"}]': finished 2024-04-25T13:34:01.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:00 smithi122 ceph-mon[32843]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T13:34:01.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:00 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:01.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:00 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2920427056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0b3dc7e-9e5d-4cee-8395-0a83219ee52d"}]: dispatch 2024-04-25T13:34:01.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:00 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2920427056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0b3dc7e-9e5d-4cee-8395-0a83219ee52d"}]': finished 2024-04-25T13:34:01.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:00 smithi045 ceph-mon[32731]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T13:34:01.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:00 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:01.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:01 smithi027 ceph-mon[29140]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:01.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:01 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/496624202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:02.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:01 smithi122 ceph-mon[32843]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:02.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:01 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/496624202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:01 smithi045 ceph-mon[32731]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:01 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/496624202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:03.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:03 smithi027 ceph-mon[29140]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:04.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:03 smithi122 ceph-mon[32843]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:04.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:03 smithi045 ceph-mon[32731]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:05.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:05 smithi027 ceph-mon[29140]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:06.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:05 smithi122 ceph-mon[32843]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:06.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:05 smithi045 ceph-mon[32731]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:07.735 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:07 smithi027 ceph-mon[29140]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:07 smithi122 ceph-mon[32843]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:08.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:07 smithi045 ceph-mon[32731]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:08.662 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T13:34:08.662 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:08 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:08.663 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:08 smithi027 ceph-mon[29140]: Deploying daemon osd.3 on smithi027 2024-04-25T13:34:09.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:08 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T13:34:09.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:08 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:09.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:08 smithi122 ceph-mon[32843]: Deploying daemon osd.3 on smithi027 2024-04-25T13:34:09.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:08 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T13:34:09.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:08 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:09.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:08 smithi045 ceph-mon[32731]: Deploying daemon osd.3 on smithi027 2024-04-25T13:34:09.709 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:09 smithi027 ceph-mon[29140]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:09.960 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:09 smithi122 ceph-mon[32843]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:10.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:09 smithi045 ceph-mon[32731]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:11.769 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:11 smithi027 ceph-mon[29140]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:11.881 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:11 smithi045 ceph-mon[32731]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:12.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:11 smithi122 ceph-mon[32843]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:13.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:13 smithi027 ceph-mon[29140]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:13.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:13.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:13 smithi122 ceph-mon[32843]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:13 smithi045 ceph-mon[32731]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:14.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:14 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:15.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:14 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:15.819 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:15 smithi027 ceph-mon[29140]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:16.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:15 smithi122 ceph-mon[32843]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:16.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:15 smithi045 ceph-mon[32731]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.587+0000 7f5527fff700 1 -- 172.21.15.27:0/2475275634 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f554002e960 con 0x7f55280afaf0 2024-04-25T13:34:17.589 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 3 on host 'smithi027' 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55280afaf0 msgr2=0x7f55280b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55280afaf0 0x7f55280b2000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f55300038a0 tx=0x7f553001a040 comp rx=0 tx=0).stop 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540105340 msgr2=0x7f554019ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540105340 0x7f554019ad10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f553400c920 tx=0x7f553400cce0 comp rx=0 tx=0).stop 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 shutdown_connections 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55280afaf0 0x7f55280b2000 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5540105ce0 0x7f55401a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5540104210 0x7f554019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 --2- 172.21.15.27:0/2475275634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5540105340 0x7f554019ad10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 >> 172.21.15.27:0/2475275634 conn(0x7f55400ff870 msgr2=0x7f5540108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 shutdown_connections 2024-04-25T13:34:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:34:17.590+0000 7f554528e700 1 -- 172.21.15.27:0/2475275634 wait complete. 2024-04-25T13:34:17.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:17.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:17.830 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:17.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T13:34:17.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.831 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:17.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:17.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:17.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T13:34:17.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:17.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:18.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:18.266 DEBUG:teuthology.orchestra.run.smithi027:osd.3> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T13:34:18.269 INFO:tasks.cephadm:Deploying osd.4 on smithi045 with /dev/vg_nvme/lv_3... 2024-04-25T13:34:18.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T13:34:18.524 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T13:34:18.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:18 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:18.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:18 smithi045 ceph-mon[32731]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T13:34:18.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:18 smithi045 ceph-mon[32731]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T13:34:18.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:18 smithi045 ceph-mon[32731]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:34:18.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:18 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:18.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:18 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:18.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:18 smithi027 ceph-mon[29140]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T13:34:18.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:18 smithi027 ceph-mon[29140]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T13:34:18.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:18 smithi027 ceph-mon[29140]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:34:18.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:18 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:19.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:18 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:19.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:18 smithi122 ceph-mon[32843]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T13:34:19.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:18 smithi122 ceph-mon[32843]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T13:34:19.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:18 smithi122 ceph-mon[32843]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:34:19.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:18 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:19.402 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:34:19 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3[48395]: 2024-04-25T13:34:19.073+0000 7f7f816b8700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T13:34:19.402 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:34:19 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3[48395]: 2024-04-25T13:34:19.079+0000 7f7f78c64700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:34:19.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:19 smithi027 ceph-mon[29140]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:19.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:19 smithi027 ceph-mon[29140]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:34:19.777 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:19 smithi027 ceph-mon[29140]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T13:34:19.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:19.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:19.778 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:19 smithi027 ceph-mon[29140]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' 2024-04-25T13:34:19.899 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:19 smithi045 ceph-mon[32731]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:19.899 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:19 smithi045 ceph-mon[32731]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:34:19.899 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:19 smithi045 ceph-mon[32731]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T13:34:19.899 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:19.899 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:19.899 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:19 smithi045 ceph-mon[32731]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' 2024-04-25T13:34:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:19 smithi122 ceph-mon[32843]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:19 smithi122 ceph-mon[32843]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:34:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:19 smithi122 ceph-mon[32843]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T13:34:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:19 smithi122 ceph-mon[32843]: from='osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500]' entity='osd.3' 2024-04-25T13:34:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T13:34:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T13:34:20.496 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-04-25T13:34:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-04-25T13:34:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00854335 s, 1.2 GB/s 2024-04-25T13:34:20.496 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-04-25T13:34:20.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:20 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:34:20.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:20 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:34:20.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:20.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:20 smithi027 ceph-mon[29140]: osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] boot 2024-04-25T13:34:20.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:20 smithi027 ceph-mon[29140]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T13:34:20.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:20.903 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:20 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:34:20.904 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:20 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:34:20.904 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:20 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:20.904 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:20 smithi045 ceph-mon[32731]: osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] boot 2024-04-25T13:34:20.904 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:20 smithi045 ceph-mon[32731]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T13:34:20.904 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:20 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:21.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:20 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:34:21.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:20 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:34:21.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:21.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:20 smithi122 ceph-mon[32843]: osd.3 [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] boot 2024-04-25T13:34:21.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:20 smithi122 ceph-mon[32843]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T13:34:21.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T13:34:21.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2024-04-25T13:34:21.482 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T13:34:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T13:34:21.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:21.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:21.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:21.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:21.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:21.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:21.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:21.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:21.941 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:21.941 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T13:34:21.941 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:21.942 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:21.942 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:21.942 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:21.942 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:21.942 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:21.942 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:21.942 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:22.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T13:34:22.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T13:34:22.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:22.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:22.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:22.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:22.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:22.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:22.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:22.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:22.898 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:22 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi027 2024-04-25T13:34:22.898 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:22 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T13:34:22.898 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:22 smithi045 ceph-mon[32731]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T13:34:23.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:22 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi027 2024-04-25T13:34:23.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:22 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T13:34:23.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:22 smithi122 ceph-mon[32843]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T13:34:23.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:22 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi027 2024-04-25T13:34:23.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:22 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi027 to 8544M 2024-04-25T13:34:23.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:22 smithi027 ceph-mon[29140]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3743580500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 msgr2=0x7f4ca01029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3743580500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 0x7f4ca01029c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009990 tx=0x7f4c90009ca0 comp rx=0 tx=0).stop 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3743580500 shutdown_connections 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3743580500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ca0109fc0 0x7f4ca010c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3743580500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 0x7f4ca01029c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3743580500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca0107660 0x7f4ca0109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3743580500 >> 172.21.15.45:0/3743580500 conn(0x7f4ca00fded0 msgr2=0x7f4ca0100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3743580500 shutdown_connections 2024-04-25T13:34:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.162+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3743580500 wait complete. 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca85d7700 1 Processor -- start 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca85d7700 1 -- start start 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca85d7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 0x7f4ca019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca85d7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ca0107660 0x7f4ca019ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca85d7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca0109fc0 0x7f4ca01a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca85d7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca010fd80 con 0x7f4ca0109fc0 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca85d7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca010fc00 con 0x7f4ca0102570 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca85d7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca010ff00 con 0x7f4ca0107660 2024-04-25T13:34:23.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca6373700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 0x7f4ca019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca6373700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 0x7f4ca019a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58542/0 (socket says 172.21.15.45:58542) 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca6373700 1 -- 172.21.15.45:0/3909925657 learned_addr learned my addr 172.21.15.45:0/3909925657 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca5b72700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ca0107660 0x7f4ca019ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.163+0000 7f4ca6b74700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca0109fc0 0x7f4ca01a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.164+0000 7f4ca6373700 1 -- 172.21.15.45:0/3909925657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ca0107660 msgr2=0x7f4ca019ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.164+0000 7f4ca6373700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ca0107660 0x7f4ca019ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.164+0000 7f4ca6373700 1 -- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca0109fc0 msgr2=0x7f4ca01a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.164+0000 7f4ca6373700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca0109fc0 0x7f4ca01a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.164+0000 7f4ca6373700 1 -- 172.21.15.45:0/3909925657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca01a2460 con 0x7f4ca0102570 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.164+0000 7f4ca6373700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 0x7f4ca019a900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4c90006010 tx=0x7f4c900047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.164+0000 7f4c977fe700 1 -- 172.21.15.45:0/3909925657 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90027070 con 0x7f4ca0102570 2024-04-25T13:34:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.164+0000 7f4c977fe700 1 -- 172.21.15.45:0/3909925657 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c900041c0 con 0x7f4ca0102570 2024-04-25T13:34:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.165+0000 7f4c977fe700 1 -- 172.21.15.45:0/3909925657 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9000f460 con 0x7f4ca0102570 2024-04-25T13:34:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.165+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca01a26f0 con 0x7f4ca0102570 2024-04-25T13:34:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.165+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca01a2bc0 con 0x7f4ca0102570 2024-04-25T13:34:23.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.166+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca0051190 con 0x7f4ca0102570 2024-04-25T13:34:23.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.167+0000 7f4c977fe700 1 -- 172.21.15.45:0/3909925657 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4c9002a070 con 0x7f4ca0102570 2024-04-25T13:34:23.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.168+0000 7f4c977fe700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4c8c0afb80 0x7f4c8c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:34:23.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.168+0000 7f4c977fe700 1 -- 172.21.15.45:0/3909925657 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f4c900df020 con 0x7f4ca0102570 2024-04-25T13:34:23.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.168+0000 7f4ca5b72700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4c8c0afb80 0x7f4c8c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:34:23.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.169+0000 7f4ca5b72700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4c8c0afb80 0x7f4c8c0b2090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ca019b9b0 tx=0x7f4c9c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:34:23.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.172+0000 7f4c977fe700 1 -- 172.21.15.45:0/3909925657 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c90013070 con 0x7f4ca0102570 2024-04-25T13:34:23.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:23.391+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4ca0063ad0 con 0x7f4c8c0afb80 2024-04-25T13:34:24.028 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:23 smithi045 ceph-mon[32731]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:24.028 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:34:24.028 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:34:24.028 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:23 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:24.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:23 smithi122 ceph-mon[32843]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:24.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:23 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:34:24.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:23 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:34:24.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:23 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:24.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:23 smithi027 ceph-mon[29140]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:24.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:23 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:34:24.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:23 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:34:24.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:23 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:25.031 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:24 smithi045 ceph-mon[32731]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:34:25.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:24 smithi122 ceph-mon[32843]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:34:25.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:24 smithi027 ceph-mon[29140]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:34:26.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:25 smithi122 ceph-mon[32843]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:26.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:25 smithi045 ceph-mon[32731]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:26.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:25 smithi027 ceph-mon[29140]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:27.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:26 smithi122 ceph-mon[32843]: from='client.? 172.21.15.45:0/2825735903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]: dispatch 2024-04-25T13:34:27.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:26 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]: dispatch 2024-04-25T13:34:27.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:26 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]': finished 2024-04-25T13:34:27.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:26 smithi122 ceph-mon[32843]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T13:34:27.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:27.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:26 smithi045 ceph-mon[32731]: from='client.? 172.21.15.45:0/2825735903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]: dispatch 2024-04-25T13:34:27.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:26 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]: dispatch 2024-04-25T13:34:27.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:26 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]': finished 2024-04-25T13:34:27.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:26 smithi045 ceph-mon[32731]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T13:34:27.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:26 smithi027 ceph-mon[29140]: from='client.? 172.21.15.45:0/2825735903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]: dispatch 2024-04-25T13:34:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:26 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]: dispatch 2024-04-25T13:34:27.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:26 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75840f44-2190-409f-802e-0479ec705f0a"}]': finished 2024-04-25T13:34:27.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:26 smithi027 ceph-mon[29140]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T13:34:27.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:27 smithi122 ceph-mon[32843]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T13:34:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:27 smithi122 ceph-mon[32843]: from='client.? 172.21.15.45:0/2054517919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:27 smithi045 ceph-mon[32731]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T13:34:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:27 smithi045 ceph-mon[32731]: from='client.? 172.21.15.45:0/2054517919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:27 smithi027 ceph-mon[29140]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T13:34:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:27 smithi027 ceph-mon[29140]: from='client.? 172.21.15.45:0/2054517919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:29.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:29 smithi045 ceph-mon[32731]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T13:34:30.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:29 smithi122 ceph-mon[32843]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T13:34:30.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:29 smithi027 ceph-mon[29140]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T13:34:32.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:31 smithi122 ceph-mon[32843]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T13:34:32.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:31 smithi045 ceph-mon[32731]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T13:34:32.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:31 smithi027 ceph-mon[29140]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T13:34:34.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:33 smithi122 ceph-mon[32843]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:34:34.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:33 smithi045 ceph-mon[32731]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:34:34.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:33 smithi027 ceph-mon[29140]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:34:35.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T13:34:35.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:35.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:34 smithi122 ceph-mon[32843]: Deploying daemon osd.4 on smithi045 2024-04-25T13:34:35.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T13:34:35.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:35.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:34 smithi045 ceph-mon[32731]: Deploying daemon osd.4 on smithi045 2024-04-25T13:34:35.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T13:34:35.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:35.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:34 smithi027 ceph-mon[29140]: Deploying daemon osd.4 on smithi045 2024-04-25T13:34:36.025 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:35 smithi045 ceph-mon[32731]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:34:36.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:35 smithi122 ceph-mon[32843]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:34:36.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:35 smithi027 ceph-mon[29140]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:34:38.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:37 smithi122 ceph-mon[32843]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:38.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:37 smithi045 ceph-mon[32731]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:38.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:37 smithi027 ceph-mon[29140]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:40.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:39 smithi122 ceph-mon[32843]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:40.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:39 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:40.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:39 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:40.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:39 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:40.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:39 smithi045 ceph-mon[32731]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:40.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:39 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:40.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:39 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:40.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:39 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:40.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:39 smithi027 ceph-mon[29140]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:40.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:39 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:40.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:39 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:40.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:39 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:41.968 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:41 smithi045 ceph-mon[32731]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:42.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:41 smithi122 ceph-mon[32843]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:42.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:41 smithi027 ceph-mon[29140]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:43.475 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:43 smithi045 ceph-mon[32731]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:43.475 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:43 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:43.475 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:43 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:43.475 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:43 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:43.476 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:43 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:43.476 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:43 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:43.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:43 smithi122 ceph-mon[32843]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:43.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:43 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:43.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:43 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:43.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:43 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:43.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:43 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:43.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:43 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.565+0000 7f4c977fe700 1 -- 172.21.15.45:0/3909925657 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4ca0063ad0 con 0x7f4c8c0afb80 2024-04-25T13:34:43.566 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 4 on host 'smithi045' 2024-04-25T13:34:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4c8c0afb80 msgr2=0x7f4c8c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4c8c0afb80 0x7f4c8c0b2090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ca019b9b0 tx=0x7f4c9c009190 comp rx=0 tx=0).stop 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 msgr2=0x7f4ca019a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 0x7f4ca019a900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4c90006010 tx=0x7f4c900047b0 comp rx=0 tx=0).stop 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 shutdown_connections 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4c8c0afb80 0x7f4c8c0b2090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca0109fc0 0x7f4ca01a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca0102570 0x7f4ca019a900 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 --2- 172.21.15.45:0/3909925657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ca0107660 0x7f4ca019ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 >> 172.21.15.45:0/3909925657 conn(0x7f4ca00fded0 msgr2=0x7f4ca0108600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 shutdown_connections 2024-04-25T13:34:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:34:43.568+0000 7f4ca85d7700 1 -- 172.21.15.45:0/3909925657 wait complete. 2024-04-25T13:34:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:43 smithi027 ceph-mon[29140]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:43 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:44.264 DEBUG:teuthology.orchestra.run.smithi045:osd.4> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T13:34:44.266 INFO:tasks.cephadm:Deploying osd.5 on smithi122 with /dev/vg_nvme/lv_3... 2024-04-25T13:34:44.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T13:34:44.519 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:34:44.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:44.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:44.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:44.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:44 smithi122 ceph-mon[32843]: from='osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T13:34:44.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:44 smithi122 ceph-mon[32843]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T13:34:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:44.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:44.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:44.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:44 smithi045 ceph-mon[32731]: from='osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T13:34:44.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:44 smithi045 ceph-mon[32731]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T13:34:44.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:44.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:44.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:34:44.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:44 smithi027 ceph-mon[29140]: from='osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T13:34:44.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:44 smithi027 ceph-mon[29140]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T13:34:45.657 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:45 smithi045 ceph-mon[32731]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:45.657 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:45 smithi045 ceph-mon[32731]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T13:34:45.657 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:45 smithi045 ceph-mon[32731]: from='osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:34:45.657 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:45 smithi045 ceph-mon[32731]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T13:34:45.657 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:45.657 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:45 smithi045 ceph-mon[32731]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:34:45.802 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:45 smithi122 ceph-mon[32843]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:45.802 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:45 smithi122 ceph-mon[32843]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T13:34:45.802 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:45 smithi122 ceph-mon[32843]: from='osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:34:45.802 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:45 smithi122 ceph-mon[32843]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T13:34:45.802 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:45.803 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:45 smithi122 ceph-mon[32843]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:34:45.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:45 smithi027 ceph-mon[29140]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:45.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:45 smithi027 ceph-mon[29140]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T13:34:45.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:45 smithi027 ceph-mon[29140]: from='osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:34:45.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:45 smithi027 ceph-mon[29140]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T13:34:45.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:45.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:45 smithi027 ceph-mon[29140]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:34:46.329 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:34:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4[44261]: 2024-04-25T13:34:46.026+0000 7fbb54bcd700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T13:34:46.329 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:34:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4[44261]: 2024-04-25T13:34:46.035+0000 7fbb4c97a700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:34:46.498 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T13:34:46.498 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T13:34:46.498 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-25T13:34:46.498 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-25T13:34:46.498 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109825 s, 955 MB/s 2024-04-25T13:34:46.498 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 2024-04-25T13:34:46.498 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-25T13:34:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:46 smithi122 ceph-mon[32843]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:34:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:46 smithi122 ceph-mon[32843]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T13:34:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:46 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:46 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:46 smithi122 ceph-mon[32843]: from='osd.4 ' entity='osd.4' 2024-04-25T13:34:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:46 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:46 smithi045 ceph-mon[32731]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:34:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:46 smithi045 ceph-mon[32731]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T13:34:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:46 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:46 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:46.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:46 smithi045 ceph-mon[32731]: from='osd.4 ' entity='osd.4' 2024-04-25T13:34:46.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:46 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:46.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:46 smithi027 ceph-mon[29140]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:34:46.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:46 smithi027 ceph-mon[29140]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T13:34:46.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:46 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:46.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:46 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:46.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:46 smithi027 ceph-mon[29140]: from='osd.4 ' entity='osd.4' 2024-04-25T13:34:46.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:46 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:47.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2024-04-25T13:34:47.397 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:34:47.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:47 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:34:47.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:47 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:34:47.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:47 smithi045 ceph-mon[32731]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:47.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:47 smithi045 ceph-mon[32731]: osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] boot 2024-04-25T13:34:47.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:47 smithi045 ceph-mon[32731]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T13:34:47.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:47 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:47.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:47 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:34:47.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:47 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:34:47.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:47 smithi122 ceph-mon[32843]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:47.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:47 smithi122 ceph-mon[32843]: osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] boot 2024-04-25T13:34:47.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:47 smithi122 ceph-mon[32843]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T13:34:47.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:47 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:47.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:47 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:34:47.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:47 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:34:47.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:47 smithi027 ceph-mon[29140]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-25T13:34:47.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:47 smithi027 ceph-mon[29140]: osd.4 [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] boot 2024-04-25T13:34:47.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:47 smithi027 ceph-mon[29140]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T13:34:47.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:47 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T13:34:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi045 2024-04-25T13:34:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi045 to 8546M 2024-04-25T13:34:48.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:48.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:48.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:48.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:48 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:48.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi045 2024-04-25T13:34:48.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:48.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:48.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:48.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:48.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi045 to 8546M 2024-04-25T13:34:48.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:48.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:48.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:48.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:48 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:49.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 -- 172.21.15.122:0/210500146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038109e90 msgr2=0x7f003810c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:49.058 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 --2- 172.21.15.122:0/210500146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038109e90 0x7f003810c3a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f003000b0a0 tx=0x7f003000b3b0 comp rx=0 tx=0).stop 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 -- 172.21.15.122:0/210500146 shutdown_connections 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 --2- 172.21.15.122:0/210500146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038109e90 0x7f003810c3a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 --2- 172.21.15.122:0/210500146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0038076580 0x7f0038106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 --2- 172.21.15.122:0/210500146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0038107320 0x7f0038109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 -- 172.21.15.122:0/210500146 >> 172.21.15.122:0/210500146 conn(0x7f00380fe740 msgr2=0x7f0038100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 -- 172.21.15.122:0/210500146 shutdown_connections 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.055+0000 7f003cdea700 1 -- 172.21.15.122:0/210500146 wait complete. 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003cdea700 1 Processor -- start 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003cdea700 1 -- start start 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003cdea700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038076580 0x7f003819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003cdea700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0038107320 0x7f003819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:34:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003cdea700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0038109e90 0x7f00381a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003cdea700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f003810fc30 con 0x7f0038109e90 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003cdea700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f003810fab0 con 0x7f0038107320 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003cdea700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f003810fdb0 con 0x7f0038076580 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003659c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038076580 0x7f003819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003659c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038076580 0x7f003819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:48882/0 (socket says 172.21.15.122:48882) 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003659c700 1 -- 172.21.15.122:0/277499763 learned_addr learned my addr 172.21.15.122:0/277499763 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f0035d9b700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0038107320 0x7f003819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f0036d9d700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0038109e90 0x7f00381a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003659c700 1 -- 172.21.15.122:0/277499763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0038107320 msgr2=0x7f003819ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003659c700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0038107320 0x7f003819ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.056+0000 7f003659c700 1 -- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0038109e90 msgr2=0x7f00381a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.057+0000 7f003659c700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0038109e90 0x7f00381a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.057+0000 7f003659c700 1 -- 172.21.15.122:0/277499763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00381a2230 con 0x7f0038076580 2024-04-25T13:34:49.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.057+0000 7f003659c700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038076580 0x7f003819a6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f002800e9a0 tx=0x7f002800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:34:49.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.057+0000 7f00277fe700 1 -- 172.21.15.122:0/277499763 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002800ca10 con 0x7f0038076580 2024-04-25T13:34:49.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.057+0000 7f00277fe700 1 -- 172.21.15.122:0/277499763 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f002800cbb0 con 0x7f0038076580 2024-04-25T13:34:49.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.057+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00381a2520 con 0x7f0038076580 2024-04-25T13:34:49.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.057+0000 7f00277fe700 1 -- 172.21.15.122:0/277499763 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028018680 con 0x7f0038076580 2024-04-25T13:34:49.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.057+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00381a2a60 con 0x7f0038076580 2024-04-25T13:34:49.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.058+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0038051190 con 0x7f0038076580 2024-04-25T13:34:49.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.059+0000 7f00277fe700 1 -- 172.21.15.122:0/277499763 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0028010430 con 0x7f0038076580 2024-04-25T13:34:49.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.060+0000 7f00277fe700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f00200afb80 0x7f00200b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:34:49.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.060+0000 7f00277fe700 1 -- 172.21.15.122:0/277499763 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f00280ddd60 con 0x7f0038076580 2024-04-25T13:34:49.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.060+0000 7f0035d9b700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f00200afb80 0x7f00200b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:34:49.062 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.061+0000 7f0035d9b700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f00200afb80 0x7f00200b2090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f003819b780 tx=0x7f002c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:34:49.068 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.067+0000 7f00277fe700 1 -- 172.21.15.122:0/277499763 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f00280a79b0 con 0x7f0038076580 2024-04-25T13:34:49.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi045 2024-04-25T13:34:49.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:49.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:49.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:49.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:34:49.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi045 to 8546M 2024-04-25T13:34:49.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:49.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:49.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:34:49.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:48 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:34:49.271 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:34:49.270+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0038063ad0 con 0x7f00200afb80 2024-04-25T13:34:49.881 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:49 smithi122 ceph-mon[32843]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:49.881 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:49 smithi122 ceph-mon[32843]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T13:34:49.881 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:49 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:34:49.881 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:49 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:34:49.881 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:49 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:49.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:49 smithi027 ceph-mon[29140]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:49.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:49 smithi027 ceph-mon[29140]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T13:34:49.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:49 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:34:49.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:49 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:34:49.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:49 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:50.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:49 smithi045 ceph-mon[32731]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:50.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:49 smithi045 ceph-mon[32731]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T13:34:50.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:49 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:34:50.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:49 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:34:50.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:49 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:34:50.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:50 smithi027 ceph-mon[29140]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:34:50.908 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:50 smithi122 ceph-mon[32843]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:34:51.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:50 smithi045 ceph-mon[32731]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:34:51.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:51 smithi027 ceph-mon[29140]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:52.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:51 smithi122 ceph-mon[32843]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:52.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:51 smithi045 ceph-mon[32731]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:52.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/911990807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]: dispatch 2024-04-25T13:34:52.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:52 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]: dispatch 2024-04-25T13:34:52.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:52 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]': finished 2024-04-25T13:34:52.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:52 smithi122 ceph-mon[32843]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T13:34:52.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:52 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:34:52.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/911990807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]: dispatch 2024-04-25T13:34:52.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:52 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]: dispatch 2024-04-25T13:34:52.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:52 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]': finished 2024-04-25T13:34:52.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:52 smithi027 ceph-mon[29140]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T13:34:52.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:52 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:34:53.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/911990807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]: dispatch 2024-04-25T13:34:53.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:52 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]: dispatch 2024-04-25T13:34:53.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:52 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e2af772-371f-40d5-8ff8-541551a67e38"}]': finished 2024-04-25T13:34:53.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:52 smithi045 ceph-mon[32731]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T13:34:53.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:52 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:34:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:53 smithi045 ceph-mon[32731]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:53 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/254908705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:53.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:53 smithi027 ceph-mon[29140]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:53.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:53 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/254908705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:54.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:53 smithi122 ceph-mon[32843]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:54.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:53 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/254908705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:34:55.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:55 smithi027 ceph-mon[29140]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:56.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:55 smithi122 ceph-mon[32843]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:56.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:55 smithi045 ceph-mon[32731]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:57.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:57 smithi027 ceph-mon[29140]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:58.025 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:57 smithi122 ceph-mon[32843]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:58.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:57 smithi045 ceph-mon[32731]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:59.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:59 smithi027 ceph-mon[29140]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:34:59.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:59 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T13:34:59.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:34:59 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:00.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:59 smithi122 ceph-mon[32843]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:00.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:59 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T13:35:00.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:34:59 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:00.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:59 smithi045 ceph-mon[32731]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:00.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:59 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T13:35:00.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:34:59 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:00.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:00 smithi027 ceph-mon[29140]: Deploying daemon osd.5 on smithi122 2024-04-25T13:35:01.046 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:00 smithi122 ceph-mon[32843]: Deploying daemon osd.5 on smithi122 2024-04-25T13:35:01.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:00 smithi045 ceph-mon[32731]: Deploying daemon osd.5 on smithi122 2024-04-25T13:35:01.834 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:01 smithi122 ceph-mon[32843]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:01.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:01 smithi027 ceph-mon[29140]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:01 smithi045 ceph-mon[32731]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:03.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:03 smithi027 ceph-mon[29140]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:04.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:03 smithi122 ceph-mon[32843]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:04.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:03 smithi045 ceph-mon[32731]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:05.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:05 smithi027 ceph-mon[29140]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:05.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:05.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:05.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:05.986 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:05 smithi122 ceph-mon[32843]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:05.987 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:05.987 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:05.987 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:06.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:05 smithi045 ceph-mon[32731]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:06.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:06.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:06.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:07.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:07 smithi027 ceph-mon[29140]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:07 smithi122 ceph-mon[32843]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:08.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:07 smithi045 ceph-mon[32731]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:09.340 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.340+0000 7f00277fe700 1 -- 172.21.15.122:0/277499763 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0038063ad0 con 0x7f00200afb80 2024-04-25T13:35:09.340 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 5 on host 'smithi122' 2024-04-25T13:35:09.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f00200afb80 msgr2=0x7f00200b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:09.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f00200afb80 0x7f00200b2090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f003819b780 tx=0x7f002c009260 comp rx=0 tx=0).stop 2024-04-25T13:35:09.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038076580 msgr2=0x7f003819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:09.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038076580 0x7f003819a6d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f002800e9a0 tx=0x7f002800ecb0 comp rx=0 tx=0).stop 2024-04-25T13:35:09.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 shutdown_connections 2024-04-25T13:35:09.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f00200afb80 0x7f00200b2090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:09.346 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0038076580 0x7f003819a6d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:09.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0038109e90 0x7f00381a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:09.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 --2- 172.21.15.122:0/277499763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0038107320 0x7f003819ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:09.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 >> 172.21.15.122:0/277499763 conn(0x7f00380fe740 msgr2=0x7f0038105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:35:09.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 shutdown_connections 2024-04-25T13:35:09.347 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:35:09.343+0000 7f003cdea700 1 -- 172.21.15.122:0/277499763 wait complete. 2024-04-25T13:35:09.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:09.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:09.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:09.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:09.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T13:35:09.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T13:35:09.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:09.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:09.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:10.062 DEBUG:teuthology.orchestra.run.smithi122:osd.5> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T13:35:10.064 INFO:tasks.cephadm:Deploying osd.6 on smithi027 with /dev/vg_nvme/lv_2... 2024-04-25T13:35:10.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T13:35:10.320 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:35:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:10 smithi027 ceph-mon[29140]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T13:35:10.771 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:10 smithi027 ceph-mon[29140]: from='osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:35:10.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:10 smithi027 ceph-mon[29140]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T13:35:10.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:10.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:10 smithi027 ceph-mon[29140]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:35:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:10 smithi122 ceph-mon[32843]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T13:35:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:10 smithi122 ceph-mon[32843]: from='osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:35:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:10 smithi122 ceph-mon[32843]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T13:35:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:10 smithi122 ceph-mon[32843]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:35:11.059 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:35:10 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:35:10.659+0000 7ff70e302700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T13:35:11.059 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:35:10 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:35:10.667+0000 7ff70a0b7700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:35:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:10 smithi045 ceph-mon[32731]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T13:35:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:10 smithi045 ceph-mon[32731]: from='osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:35:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:10 smithi045 ceph-mon[32731]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T13:35:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:10 smithi045 ceph-mon[32731]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:35:11.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:11 smithi027 ceph-mon[29140]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:11.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:11 smithi027 ceph-mon[29140]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:35:11.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:11 smithi027 ceph-mon[29140]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T13:35:11.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:11 smithi027 ceph-mon[29140]: osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] boot 2024-04-25T13:35:11.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:11 smithi027 ceph-mon[29140]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T13:35:11.824 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:11 smithi122 ceph-mon[32843]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:11.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:11 smithi122 ceph-mon[32843]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:35:11.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:11 smithi122 ceph-mon[32843]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T13:35:11.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:11 smithi122 ceph-mon[32843]: osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] boot 2024-04-25T13:35:11.895 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:11 smithi122 ceph-mon[32843]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T13:35:11.895 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:11 smithi045 ceph-mon[32731]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T13:35:11.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:11 smithi045 ceph-mon[32731]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:35:11.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:11 smithi045 ceph-mon[32731]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T13:35:11.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:11.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:11 smithi045 ceph-mon[32731]: osd.5 [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] boot 2024-04-25T13:35:11.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:11 smithi045 ceph-mon[32731]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T13:35:11.980 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T13:35:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T13:35:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T13:35:12.604 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T13:35:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T13:35:12.604 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131421 s, 798 MB/s 2024-04-25T13:35:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T13:35:12.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:12 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:35:12.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:12 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:35:12.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:12 smithi122 ceph-mon[32843]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T13:35:12.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:12 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:35:12.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:12 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:35:12.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:12 smithi027 ceph-mon[29140]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T13:35:13.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:12 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:35:13.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:12 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:35:13.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:12 smithi045 ceph-mon[32731]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T13:35:13.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_2 2024-04-25T13:35:13.561 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:35:13.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:13.941 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T13:35:13.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:13.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:13.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:13.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:13.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:13.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:13.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:13.942 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:14.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:14.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T13:35:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:14.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:14.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:14.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:15.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:14 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi122 2024-04-25T13:35:15.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:14 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi122 to 8544M 2024-04-25T13:35:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:14 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi122 2024-04-25T13:35:15.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:14 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi122 to 8544M 2024-04-25T13:35:15.084 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:14 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi122 2024-04-25T13:35:15.084 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:14 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi122 to 8544M 2024-04-25T13:35:15.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 -- 172.21.15.27:0/4284048485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 msgr2=0x7f4968107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:15.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 --2- 172.21.15.27:0/4284048485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 0x7f4968107130 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4958009990 tx=0x7f4958009ca0 comp rx=0 tx=0).stop 2024-04-25T13:35:15.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 -- 172.21.15.27:0/4284048485 shutdown_connections 2024-04-25T13:35:15.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 --2- 172.21.15.27:0/4284048485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4968109fd0 0x7f496810c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:15.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 --2- 172.21.15.27:0/4284048485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 0x7f4968107130 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:15.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 --2- 172.21.15.27:0/4284048485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4968107670 0x7f4968109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:15.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 -- 172.21.15.27:0/4284048485 >> 172.21.15.27:0/4284048485 conn(0x7f49680fe750 msgr2=0x7f4968100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 -- 172.21.15.27:0/4284048485 shutdown_connections 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.332+0000 7f496c836700 1 -- 172.21.15.27:0/4284048485 wait complete. 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.333+0000 7f496c836700 1 Processor -- start 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.333+0000 7f496c836700 1 -- start start 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496c836700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 0x7f496819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496c836700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4968107670 0x7f496819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496c836700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4968109fd0 0x7f49681a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496c836700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f496810fc50 con 0x7f4968104d10 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496c836700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f496810fad0 con 0x7f4968107670 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496c836700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f496810fdd0 con 0x7f4968109fd0 2024-04-25T13:35:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 0x7f496819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 0x7f496819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42648/0 (socket says 172.21.15.27:42648) 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 -- 172.21.15.27:0/3833575467 learned_addr learned my addr 172.21.15.27:0/3833575467 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f4966d9d700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4968109fd0 0x7f49681a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f4965d9b700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4968107670 0x7f496819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 -- 172.21.15.27:0/3833575467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4968109fd0 msgr2=0x7f49681a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4968109fd0 0x7f49681a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 -- 172.21.15.27:0/3833575467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4968107670 msgr2=0x7f496819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4968107670 0x7f496819ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 -- 172.21.15.27:0/3833575467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49681a22a0 con 0x7f4968104d10 2024-04-25T13:35:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f4965d9b700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4968107670 0x7f496819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:35:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f496659c700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 0x7f496819a7d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4958006010 tx=0x7f49580057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:35:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.334+0000 7f49577fe700 1 -- 172.21.15.27:0/3833575467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4958027070 con 0x7f4968104d10 2024-04-25T13:35:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.335+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49681a2530 con 0x7f4968104d10 2024-04-25T13:35:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.335+0000 7f49577fe700 1 -- 172.21.15.27:0/3833575467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49580040c0 con 0x7f4968104d10 2024-04-25T13:35:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.335+0000 7f49577fe700 1 -- 172.21.15.27:0/3833575467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4958022d60 con 0x7f4968104d10 2024-04-25T13:35:15.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.335+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49681a29e0 con 0x7f4968104d10 2024-04-25T13:35:15.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.336+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4968051190 con 0x7f4968104d10 2024-04-25T13:35:15.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.337+0000 7f49577fe700 1 -- 172.21.15.27:0/3833575467 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f495802a070 con 0x7f4968104d10 2024-04-25T13:35:15.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.338+0000 7f49577fe700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afb80 0x7f49500b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:35:15.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.338+0000 7f4965d9b700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afb80 0x7f49500b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:35:15.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.338+0000 7f49577fe700 1 -- 172.21.15.27:0/3833575467 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4533+0+0 (secure 0 0 0) 0x7f49580df4c0 con 0x7f4968104d10 2024-04-25T13:35:15.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.338+0000 7f4965d9b700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afb80 0x7f49500b2090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f496819b880 tx=0x7f495c00a680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:35:15.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.341+0000 7f49577fe700 1 -- 172.21.15.27:0/3833575467 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4958031080 con 0x7f4968104d10 2024-04-25T13:35:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:15.559+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4968063ad0 con 0x7f49500afb80 2024-04-25T13:35:16.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:15 smithi122 ceph-mon[32843]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:16.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:35:16.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:35:16.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:16.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:15 smithi045 ceph-mon[32731]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:16.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:35:16.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:35:16.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:16.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:15 smithi027 ceph-mon[29140]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:16.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:35:16.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:35:16.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:17.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:16 smithi045 ceph-mon[32731]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:35:17.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:16 smithi027 ceph-mon[29140]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:35:17.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:16 smithi122 ceph-mon[32843]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:35:18.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:17 smithi045 ceph-mon[32731]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-25T13:35:18.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:17 smithi027 ceph-mon[29140]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-25T13:35:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:17 smithi122 ceph-mon[32843]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-25T13:35:19.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:18 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1579512688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e18badd-2f57-42b1-a367-e88ecaa3c0e8"}]: dispatch 2024-04-25T13:35:19.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:18 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1579512688' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e18badd-2f57-42b1-a367-e88ecaa3c0e8"}]': finished 2024-04-25T13:35:19.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:18 smithi045 ceph-mon[32731]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T13:35:19.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:18 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:19.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:18 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1579512688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e18badd-2f57-42b1-a367-e88ecaa3c0e8"}]: dispatch 2024-04-25T13:35:19.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:18 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1579512688' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e18badd-2f57-42b1-a367-e88ecaa3c0e8"}]': finished 2024-04-25T13:35:19.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:18 smithi027 ceph-mon[29140]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T13:35:19.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:18 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:19.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:18 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1579512688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e18badd-2f57-42b1-a367-e88ecaa3c0e8"}]: dispatch 2024-04-25T13:35:19.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:18 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1579512688' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e18badd-2f57-42b1-a367-e88ecaa3c0e8"}]': finished 2024-04-25T13:35:19.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:18 smithi122 ceph-mon[32843]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T13:35:19.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:18 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:19 smithi045 ceph-mon[32731]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T13:35:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:19 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/426180163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:35:20.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:19 smithi027 ceph-mon[29140]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T13:35:20.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:19 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/426180163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:35:20.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:19 smithi122 ceph-mon[32843]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T13:35:20.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:19 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/426180163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:35:22.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:21 smithi045 ceph-mon[32731]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T13:35:22.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:21 smithi027 ceph-mon[29140]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T13:35:22.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:21 smithi122 ceph-mon[32843]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T13:35:23.975 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:23 smithi027 ceph-mon[29140]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T13:35:24.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:23 smithi045 ceph-mon[32731]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T13:35:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:23 smithi122 ceph-mon[32843]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T13:35:26.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:25 smithi045 ceph-mon[32731]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:35:26.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:25 smithi027 ceph-mon[29140]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:35:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:25 smithi122 ceph-mon[32843]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:35:27.064 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T13:35:27.065 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:27.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T13:35:27.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:27.330 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T13:35:27.330 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:28.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:27 smithi027 ceph-mon[29140]: Deploying daemon osd.6 on smithi027 2024-04-25T13:35:28.133 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:27 smithi027 ceph-mon[29140]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:28.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:27 smithi122 ceph-mon[32843]: Deploying daemon osd.6 on smithi027 2024-04-25T13:35:28.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:27 smithi122 ceph-mon[32843]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:28.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:27 smithi045 ceph-mon[32731]: Deploying daemon osd.6 on smithi027 2024-04-25T13:35:28.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:27 smithi045 ceph-mon[32731]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:30.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:29 smithi027 ceph-mon[29140]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:30.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:29 smithi122 ceph-mon[32843]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:30.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:29 smithi045 ceph-mon[32731]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:32.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:31 smithi027 ceph-mon[29140]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:32.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:31 smithi122 ceph-mon[32843]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:32.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:31 smithi045 ceph-mon[32731]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:33 smithi027 ceph-mon[29140]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:33.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:33.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:33.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:33.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:33 smithi122 ceph-mon[32843]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:33.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:33.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:33.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:33.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:33 smithi045 ceph-mon[32731]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:35.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:35 smithi027 ceph-mon[29140]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:35.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:35.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:35.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:35.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:35.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:35.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:35 smithi122 ceph-mon[32843]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:35.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:35.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:35.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:35.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:35.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:35.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:35 smithi045 ceph-mon[32731]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:35.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:35.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:35.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:35.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:35.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:36.323 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 6 on host 'smithi027' 2024-04-25T13:35:36.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.322+0000 7f49577fe700 1 -- 172.21.15.27:0/3833575467 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4968063ad0 con 0x7f49500afb80 2024-04-25T13:35:36.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afb80 msgr2=0x7f49500b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afb80 0x7f49500b2090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f496819b880 tx=0x7f495c00a680 comp rx=0 tx=0).stop 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 msgr2=0x7f496819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 0x7f496819a7d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4958006010 tx=0x7f49580057c0 comp rx=0 tx=0).stop 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 shutdown_connections 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afb80 0x7f49500b2090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4968109fd0 0x7f49681a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4968104d10 0x7f496819a7d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 --2- 172.21.15.27:0/3833575467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4968107670 0x7f496819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 >> 172.21.15.27:0/3833575467 conn(0x7f49680fe750 msgr2=0x7f4968105940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 shutdown_connections 2024-04-25T13:35:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:35:36.325+0000 7f496c836700 1 -- 172.21.15.27:0/3833575467 wait complete. 2024-04-25T13:35:36.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:36 smithi027 ceph-mon[29140]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T13:35:36.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:36 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:36.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:36 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:36.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:36 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:36.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:36 smithi122 ceph-mon[32843]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T13:35:36.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:36 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:36.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:36 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:36.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:36 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:36.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:36 smithi045 ceph-mon[32731]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T13:35:36.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:36 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:36.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:36 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:36.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:36 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:35:37.084 DEBUG:teuthology.orchestra.run.smithi027:osd.6> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T13:35:37.086 INFO:tasks.cephadm:Deploying osd.7 on smithi045 with /dev/vg_nvme/lv_2... 2024-04-25T13:35:37.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T13:35:37.342 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T13:35:37.466 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:37 smithi045 ceph-mon[32731]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:37.466 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:37 smithi045 ceph-mon[32731]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T13:35:37.466 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:37 smithi045 ceph-mon[32731]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T13:35:37.466 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:37 smithi045 ceph-mon[32731]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:35:37.466 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:37 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:37.466 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:37 smithi045 ceph-mon[32731]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:35:37.616 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:37 smithi027 ceph-mon[29140]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:37.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:37 smithi027 ceph-mon[29140]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T13:35:37.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:37 smithi027 ceph-mon[29140]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T13:35:37.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:37 smithi027 ceph-mon[29140]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:35:37.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:37 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:37.617 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:37 smithi027 ceph-mon[29140]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:35:37.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:37 smithi122 ceph-mon[32843]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:37.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:37 smithi122 ceph-mon[32843]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T13:35:37.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:37 smithi122 ceph-mon[32843]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T13:35:37.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:37 smithi122 ceph-mon[32843]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:35:37.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:37 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:37.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:37 smithi122 ceph-mon[32843]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:35:37.901 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:35:37 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:35:37.833+0000 7f1cec7bc700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T13:35:37.901 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:35:37 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:35:37.843+0000 7f1ce4569700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:35:38.588 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:38 smithi027 ceph-mon[29140]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T13:35:38.588 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:38 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:38.588 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:38 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:38.588 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:38 smithi027 ceph-mon[29140]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' 2024-04-25T13:35:38.755 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:38 smithi045 ceph-mon[32731]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T13:35:38.755 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:38 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:38.755 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:38 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:38.755 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:38 smithi045 ceph-mon[32731]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' 2024-04-25T13:35:38.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:38 smithi122 ceph-mon[32843]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T13:35:38.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:38 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:38.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:38 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:38.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:38 smithi122 ceph-mon[32843]: from='osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961]' entity='osd.6' 2024-04-25T13:35:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T13:35:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T13:35:39.468 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-04-25T13:35:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-04-25T13:35:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00846777 s, 1.2 GB/s 2024-04-25T13:35:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-04-25T13:35:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:39 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:35:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:39 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:35:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:39 smithi027 ceph-mon[29140]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:39 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:39 smithi027 ceph-mon[29140]: osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] boot 2024-04-25T13:35:39.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:39 smithi027 ceph-mon[29140]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T13:35:39.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:39 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:39.710 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:39 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:35:39.710 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:39 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:35:39.710 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:39 smithi122 ceph-mon[32843]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:39.710 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:39 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:39.710 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:39 smithi122 ceph-mon[32843]: osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] boot 2024-04-25T13:35:39.711 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:39 smithi122 ceph-mon[32843]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T13:35:39.711 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:39 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:39.775 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:39 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:35:39.775 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:39 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:35:39.775 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:39 smithi045 ceph-mon[32731]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-25T13:35:39.775 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:39 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:39.775 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:39 smithi045 ceph-mon[32731]: osd.6 [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] boot 2024-04-25T13:35:39.775 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:39 smithi045 ceph-mon[32731]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T13:35:39.775 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:39 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T13:35:40.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2024-04-25T13:35:40.404 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:41.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T13:35:41.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:41.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:41.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T13:35:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:35:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:41.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:41.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:35:41.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:35:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 -- 172.21.15.45:0/3411295126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104250 msgr2=0x7fc308104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/3411295126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104250 0x7fc308104660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009990 tx=0x7fc2f0009ca0 comp rx=0 tx=0).stop 2024-04-25T13:35:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 -- 172.21.15.45:0/3411295126 shutdown_connections 2024-04-25T13:35:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/3411295126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc308104bf0 0x7fc30810b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/3411295126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308103120 0x7fc308103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/3411295126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104250 0x7fc308104660 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 -- 172.21.15.45:0/3411295126 >> 172.21.15.45:0/3411295126 conn(0x7fc3080fe760 msgr2=0x7fc308100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:35:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 -- 172.21.15.45:0/3411295126 shutdown_connections 2024-04-25T13:35:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.109+0000 7fc30e1ee700 1 -- 172.21.15.45:0/3411295126 wait complete. 2024-04-25T13:35:42.111 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:41 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi027 2024-04-25T13:35:42.111 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:41 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T13:35:42.111 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:41 smithi045 ceph-mon[32731]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:42.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc30e1ee700 1 Processor -- start 2024-04-25T13:35:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc30e1ee700 1 -- start start 2024-04-25T13:35:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc30e1ee700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc308103120 0x7fc308076320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:35:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc30e1ee700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308104250 0x7fc30819f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:35:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc30e1ee700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104bf0 0x7fc3081a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:35:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc30e1ee700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc30810eb90 con 0x7fc308104250 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc30e1ee700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc30810ea10 con 0x7fc308104bf0 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc30e1ee700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc30810ed10 con 0x7fc308103120 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc307fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104bf0 0x7fc3081a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc307fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104bf0 0x7fc3081a1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53924/0 (socket says 172.21.15.45:53924) 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc307fff700 1 -- 172.21.15.45:0/1109734738 learned_addr learned my addr 172.21.15.45:0/1109734738 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.110+0000 7fc3077fe700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc308103120 0x7fc308076320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc306ffd700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308104250 0x7fc30819f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc307fff700 1 -- 172.21.15.45:0/1109734738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc308103120 msgr2=0x7fc308076320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc307fff700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc308103120 0x7fc308076320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc307fff700 1 -- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308104250 msgr2=0x7fc30819f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc307fff700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308104250 0x7fc30819f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc307fff700 1 -- 172.21.15.45:0/1109734738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3081a2360 con 0x7fc308104bf0 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc307fff700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104bf0 0x7fc3081a1b30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc300007ee0 tx=0x7fc30000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:35:42.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc304ff9700 1 -- 172.21.15.45:0/1109734738 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30000dd20 con 0x7fc308104bf0 2024-04-25T13:35:42.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3081a2650 con 0x7fc308104bf0 2024-04-25T13:35:42.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc304ff9700 1 -- 172.21.15.45:0/1109734738 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc300004020 con 0x7fc308104bf0 2024-04-25T13:35:42.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc304ff9700 1 -- 172.21.15.45:0/1109734738 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300015660 con 0x7fc308104bf0 2024-04-25T13:35:42.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.111+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3081a2b90 con 0x7fc308104bf0 2024-04-25T13:35:42.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.112+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc308051190 con 0x7fc308104bf0 2024-04-25T13:35:42.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:41 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi027 2024-04-25T13:35:42.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:41 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T13:35:42.217 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:41 smithi027 ceph-mon[29140]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:42.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.114+0000 7fc304ff9700 1 -- 172.21.15.45:0/1109734738 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc3000041d0 con 0x7fc308104bf0 2024-04-25T13:35:42.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.115+0000 7fc304ff9700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2f40afc00 0x7fc2f40b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:35:42.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.115+0000 7fc304ff9700 1 -- 172.21.15.45:0/1109734738 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7fc3000de110 con 0x7fc308104bf0 2024-04-25T13:35:42.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.115+0000 7fc3077fe700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2f40afc00 0x7fc2f40b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:35:42.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.116+0000 7fc3077fe700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2f40afc00 0x7fc2f40b2110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc308104150 tx=0x7fc2f800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:35:42.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.119+0000 7fc304ff9700 1 -- 172.21.15.45:0/1109734738 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc300010070 con 0x7fc308104bf0 2024-04-25T13:35:42.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:41 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi027 2024-04-25T13:35:42.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:41 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi027 to 5696M 2024-04-25T13:35:42.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:41 smithi122 ceph-mon[32843]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:42.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:35:42.336+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc308063ad0 con 0x7fc2f40afc00 2024-04-25T13:35:43.109 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:35:43.110 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:35:43.110 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:43.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:35:43.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:35:43.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:43.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:35:43.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:35:43.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:44.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:43 smithi027 ceph-mon[29140]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:44.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:43 smithi027 ceph-mon[29140]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:35:44.173 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:43 smithi045 ceph-mon[32731]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:44.173 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:43 smithi045 ceph-mon[32731]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:35:44.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:43 smithi122 ceph-mon[32843]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:44.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:43 smithi122 ceph-mon[32843]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:35:46.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:45 smithi045 ceph-mon[32731]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:46.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:45 smithi045 ceph-mon[32731]: from='client.? 172.21.15.45:0/1465284799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]: dispatch 2024-04-25T13:35:46.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:45 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]: dispatch 2024-04-25T13:35:46.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:45 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]': finished 2024-04-25T13:35:46.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:45 smithi045 ceph-mon[32731]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T13:35:46.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:45 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:35:46.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:45 smithi027 ceph-mon[29140]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:46.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:45 smithi027 ceph-mon[29140]: from='client.? 172.21.15.45:0/1465284799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]: dispatch 2024-04-25T13:35:46.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:45 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]: dispatch 2024-04-25T13:35:46.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:45 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]': finished 2024-04-25T13:35:46.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:45 smithi027 ceph-mon[29140]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T13:35:46.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:45 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:35:46.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:45 smithi122 ceph-mon[32843]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:46.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:45 smithi122 ceph-mon[32843]: from='client.? 172.21.15.45:0/1465284799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]: dispatch 2024-04-25T13:35:46.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:45 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]: dispatch 2024-04-25T13:35:46.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:45 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30034bf9-dc47-49be-a9c5-e8ff56a32d8e"}]': finished 2024-04-25T13:35:46.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:45 smithi122 ceph-mon[32843]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T13:35:46.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:45 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:35:47.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:46 smithi027 ceph-mon[29140]: from='client.? 172.21.15.45:0/2376936963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:35:47.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:46 smithi122 ceph-mon[32843]: from='client.? 172.21.15.45:0/2376936963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:35:47.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:46 smithi045 ceph-mon[32731]: from='client.? 172.21.15.45:0/2376936963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:35:48.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:47 smithi027 ceph-mon[29140]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:47 smithi122 ceph-mon[32843]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:48.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:47 smithi045 ceph-mon[32731]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:50.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:49 smithi045 ceph-mon[32731]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:50.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:49 smithi027 ceph-mon[29140]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:50.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:49 smithi122 ceph-mon[32843]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:52.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:51 smithi122 ceph-mon[32843]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:52.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:51 smithi045 ceph-mon[32731]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:52.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:51 smithi027 ceph-mon[29140]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:54.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:53 smithi027 ceph-mon[29140]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:54.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T13:35:54.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:54.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:53 smithi027 ceph-mon[29140]: Deploying daemon osd.7 on smithi045 2024-04-25T13:35:54.306 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:53 smithi045 ceph-mon[32731]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:54.307 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:53 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T13:35:54.307 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:53 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:54.307 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:53 smithi045 ceph-mon[32731]: Deploying daemon osd.7 on smithi045 2024-04-25T13:35:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:53 smithi122 ceph-mon[32843]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:53 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T13:35:54.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:53 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:35:54.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:53 smithi122 ceph-mon[32843]: Deploying daemon osd.7 on smithi045 2024-04-25T13:35:56.121 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:55 smithi045 ceph-mon[32731]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:56.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:55 smithi027 ceph-mon[29140]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:56.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:55 smithi122 ceph-mon[32843]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:58.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:35:57 smithi027 ceph-mon[29140]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:58.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:35:57 smithi122 ceph-mon[32843]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:35:58.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:35:57 smithi045 ceph-mon[32731]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:00 smithi122 ceph-mon[32843]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:00.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:00 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:00.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:00 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:00.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:00 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:00.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:00 smithi045 ceph-mon[32731]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:00.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:00 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:00.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:00 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:00.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:00 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:00.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:00 smithi027 ceph-mon[29140]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:00.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:00 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:00.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:00 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:00.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:00 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:02.241 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:02 smithi045 ceph-mon[32731]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:02.241 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:02 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:02.241 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:02 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:02.241 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:02 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:02.241 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:02 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:02.241 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:02 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:02.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:02 smithi122 ceph-mon[32843]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:02.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:02 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:02.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:02 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:02.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:02 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:02.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:02 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:02.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:02 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:02.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:02 smithi027 ceph-mon[29140]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:02.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:02.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:02.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:02.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:02.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:02 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:03.013 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 7 on host 'smithi045' 2024-04-25T13:36:03.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.012+0000 7fc304ff9700 1 -- 172.21.15.45:0/1109734738 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc308063ad0 con 0x7fc2f40afc00 2024-04-25T13:36:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2f40afc00 msgr2=0x7fc2f40b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2f40afc00 0x7fc2f40b2110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc308104150 tx=0x7fc2f800a380 comp rx=0 tx=0).stop 2024-04-25T13:36:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104bf0 msgr2=0x7fc3081a1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104bf0 0x7fc3081a1b30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc300007ee0 tx=0x7fc30000d3b0 comp rx=0 tx=0).stop 2024-04-25T13:36:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 shutdown_connections 2024-04-25T13:36:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc2f40afc00 0x7fc2f40b2110 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc308103120 0x7fc308076320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc308104bf0 0x7fc3081a1b30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 --2- 172.21.15.45:0/1109734738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc308104250 0x7fc30819f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.014+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 >> 172.21.15.45:0/1109734738 conn(0x7fc3080fe760 msgr2=0x7fc308107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:36:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.015+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 shutdown_connections 2024-04-25T13:36:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:36:03.015+0000 7fc30e1ee700 1 -- 172.21.15.45:0/1109734738 wait complete. 2024-04-25T13:36:03.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:03 smithi122 ceph-mon[32843]: from='osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T13:36:03.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:03 smithi122 ceph-mon[32843]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T13:36:03.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:03.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:03.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:03.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:03 smithi045 ceph-mon[32731]: from='osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T13:36:03.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:03 smithi045 ceph-mon[32731]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T13:36:03.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:03.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:03.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:03.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:03 smithi027 ceph-mon[29140]: from='osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T13:36:03.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:03 smithi027 ceph-mon[29140]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T13:36:03.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:03.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:03.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:03.677 DEBUG:teuthology.orchestra.run.smithi045:osd.7> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T13:36:03.679 INFO:tasks.cephadm:Deploying osd.8 on smithi122 with /dev/vg_nvme/lv_2... 2024-04-25T13:36:03.679 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T13:36:03.921 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:36:04.283 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:04 smithi122 ceph-mon[32843]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:04.284 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:04 smithi122 ceph-mon[32843]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T13:36:04.284 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:04 smithi122 ceph-mon[32843]: from='osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:36:04.284 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:04 smithi122 ceph-mon[32843]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T13:36:04.284 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:04 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:04.284 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:04 smithi122 ceph-mon[32843]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:36:04.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:04 smithi045 ceph-mon[32731]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:04.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:04 smithi045 ceph-mon[32731]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T13:36:04.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:04 smithi045 ceph-mon[32731]: from='osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:36:04.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:04 smithi045 ceph-mon[32731]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T13:36:04.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:04 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:04.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:04 smithi045 ceph-mon[32731]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:36:04.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:04 smithi027 ceph-mon[29140]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:04.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:04 smithi027 ceph-mon[29140]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T13:36:04.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:04 smithi027 ceph-mon[29140]: from='osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:36:04.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:04 smithi027 ceph-mon[29140]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T13:36:04.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:04 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:04.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:04 smithi027 ceph-mon[29140]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:36:04.828 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:36:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:36:04.513+0000 7f19f8e80700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T13:36:04.828 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:36:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:36:04.525+0000 7f19f042c700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:36:05.195 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:05 smithi045 ceph-mon[32731]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:36:05.196 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:05 smithi045 ceph-mon[32731]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T13:36:05.196 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:05.196 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:05 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:05.196 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:05 smithi045 ceph-mon[32731]: from='osd.7 ' entity='osd.7' 2024-04-25T13:36:05.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:05 smithi122 ceph-mon[32843]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:36:05.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:05 smithi122 ceph-mon[32843]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T13:36:05.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:05.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:05 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:05.231 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:05 smithi122 ceph-mon[32843]: from='osd.7 ' entity='osd.7' 2024-04-25T13:36:05.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:05 smithi027 ceph-mon[29140]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:36:05.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:05 smithi027 ceph-mon[29140]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T13:36:05.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:05.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:05 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:05.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:05 smithi027 ceph-mon[29140]: from='osd.7 ' entity='osd.7' 2024-04-25T13:36:06.145 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T13:36:06.145 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T13:36:06.145 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-25T13:36:06.145 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-25T13:36:06.145 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128751 s, 814 MB/s 2024-04-25T13:36:06.145 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-25T13:36:06.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:06 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:36:06.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:06 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:36:06.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:06 smithi122 ceph-mon[32843]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:06.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:06 smithi122 ceph-mon[32843]: osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] boot 2024-04-25T13:36:06.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:06 smithi122 ceph-mon[32843]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T13:36:06.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:06 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:06.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:06 smithi122 ceph-mon[32843]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T13:36:06.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:06 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:36:06.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:06 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:36:06.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:06 smithi045 ceph-mon[32731]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:06 smithi045 ceph-mon[32731]: osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] boot 2024-04-25T13:36:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:06 smithi045 ceph-mon[32731]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T13:36:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:06 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:06.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:06 smithi045 ceph-mon[32731]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T13:36:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:06 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:36:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:06 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:36:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:06 smithi027 ceph-mon[29140]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-04-25T13:36:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:06 smithi027 ceph-mon[29140]: osd.7 [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] boot 2024-04-25T13:36:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:06 smithi027 ceph-mon[29140]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T13:36:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:06 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T13:36:06.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:06 smithi027 ceph-mon[29140]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T13:36:06.891 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2024-04-25T13:36:07.132 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:36:07.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:07.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi045 2024-04-25T13:36:07.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:07.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:07.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:07.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:07.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:07.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi045 to 5697M 2024-04-25T13:36:07.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:07.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:07.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:07.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:07.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:07 smithi122 ceph-mon[32843]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T13:36:08.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:08.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi045 2024-04-25T13:36:08.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:08.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:08.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:08.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:08.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:08.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi045 to 5697M 2024-04-25T13:36:08.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:08.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:08.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:08.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:08.080 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:07 smithi045 ceph-mon[32731]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T13:36:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi045 2024-04-25T13:36:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi045 to 5697M 2024-04-25T13:36:08.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:08.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:08.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:08.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:08.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:07 smithi027 ceph-mon[29140]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T13:36:08.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 -- 172.21.15.122:0/1114799166 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 msgr2=0x7fbb10105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:08.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/1114799166 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 0x7fbb10105750 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009990 tx=0x7fbb00009ca0 comp rx=0 tx=0).stop 2024-04-25T13:36:08.881 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 -- 172.21.15.122:0/1114799166 shutdown_connections 2024-04-25T13:36:08.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/1114799166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb10105ce0 0x7fbb1010c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:08.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/1114799166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb10104210 0x7fbb10104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:08.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/1114799166 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 0x7fbb10105750 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:08.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 -- 172.21.15.122:0/1114799166 >> 172.21.15.122:0/1114799166 conn(0x7fbb100ff870 msgr2=0x7fbb10101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:36:08.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 -- 172.21.15.122:0/1114799166 shutdown_connections 2024-04-25T13:36:08.882 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.880+0000 7fbb17dc5700 1 -- 172.21.15.122:0/1114799166 wait complete. 2024-04-25T13:36:08.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb17dc5700 1 Processor -- start 2024-04-25T13:36:08.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb17dc5700 1 -- start start 2024-04-25T13:36:08.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb17dc5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb10104210 0x7fbb1019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:36:08.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb17dc5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 0x7fbb1019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:36:08.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb17dc5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb10105ce0 0x7fbb101a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:36:08.884 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb17dc5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1010fc90 con 0x7fbb10104210 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb17dc5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1010fb10 con 0x7fbb10105ce0 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb17dc5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1010fe10 con 0x7fbb10105340 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb15360700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 0x7fbb1019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb15360700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 0x7fbb1019ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:33086/0 (socket says 172.21.15.122:33086) 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb15360700 1 -- 172.21.15.122:0/970944173 learned_addr learned my addr 172.21.15.122:0/970944173 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.881+0000 7fbb15b61700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb10104210 0x7fbb1019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb16362700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb10105ce0 0x7fbb101a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb15360700 1 -- 172.21.15.122:0/970944173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb10105ce0 msgr2=0x7fbb101a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb15360700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb10105ce0 0x7fbb101a1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb15360700 1 -- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb10104210 msgr2=0x7fbb1019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb15360700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb10104210 0x7fbb1019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:08.885 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb15360700 1 -- 172.21.15.122:0/970944173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb101a2210 con 0x7fbb10105340 2024-04-25T13:36:08.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb15360700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 0x7fbb1019ac80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb000098e0 tx=0x7fbb000047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:36:08.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb06ffd700 1 -- 172.21.15.122:0/970944173 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00026070 con 0x7fbb10105340 2024-04-25T13:36:08.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb101a24a0 con 0x7fbb10105340 2024-04-25T13:36:08.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb06ffd700 1 -- 172.21.15.122:0/970944173 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb00021410 con 0x7fbb10105340 2024-04-25T13:36:08.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb06ffd700 1 -- 172.21.15.122:0/970944173 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00017650 con 0x7fbb10105340 2024-04-25T13:36:08.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.882+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb101a28f0 con 0x7fbb10105340 2024-04-25T13:36:08.886 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.883+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb10051190 con 0x7fbb10105340 2024-04-25T13:36:08.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.885+0000 7fbb06ffd700 1 -- 172.21.15.122:0/970944173 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbb00029070 con 0x7fbb10105340 2024-04-25T13:36:08.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.885+0000 7fbb06ffd700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbafc0afb80 0x7fbafc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:36:08.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.886+0000 7fbb06ffd700 1 -- 172.21.15.122:0/970944173 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7fbb000defd0 con 0x7fbb10105340 2024-04-25T13:36:08.887 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.886+0000 7fbb15b61700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbafc0afb80 0x7fbafc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:36:08.888 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.886+0000 7fbb15b61700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbafc0afb80 0x7fbafc0b2090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbb10105240 tx=0x7fbb0c007a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:36:08.891 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:08.890+0000 7fbb06ffd700 1 -- 172.21.15.122:0/970944173 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb00013070 con 0x7fbb10105340 2024-04-25T13:36:09.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:09.094+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbb10063b90 con 0x7fbafc0afb80 2024-04-25T13:36:09.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:09 smithi122 ceph-mon[32843]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:09.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:09 smithi122 ceph-mon[32843]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:36:09.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:36:09.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:36:09.994 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:10.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:09 smithi045 ceph-mon[32731]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:10.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:09 smithi045 ceph-mon[32731]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:36:10.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:36:10.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:36:10.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:10.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:09 smithi027 ceph-mon[29140]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:10.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:09 smithi027 ceph-mon[29140]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:36:10.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:36:10.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:36:10.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:12.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:11 smithi122 ceph-mon[32843]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T13:36:12.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:11 smithi045 ceph-mon[32731]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T13:36:12.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:11 smithi027 ceph-mon[29140]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T13:36:13.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:12 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/997792567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]: dispatch 2024-04-25T13:36:13.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:12 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]: dispatch 2024-04-25T13:36:13.052 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:12 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]': finished 2024-04-25T13:36:13.052 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:12 smithi027 ceph-mon[29140]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T13:36:13.052 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:13.052 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:12 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/2264291870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:36:13.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:12 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/997792567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]: dispatch 2024-04-25T13:36:13.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:12 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]: dispatch 2024-04-25T13:36:13.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:12 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]': finished 2024-04-25T13:36:13.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:12 smithi122 ceph-mon[32843]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T13:36:13.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:13.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:12 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/2264291870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:36:13.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:12 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/997792567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]: dispatch 2024-04-25T13:36:13.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:12 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]: dispatch 2024-04-25T13:36:13.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:12 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e"}]': finished 2024-04-25T13:36:13.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:12 smithi045 ceph-mon[32731]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T13:36:13.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:13.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:12 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/2264291870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:36:14.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:14 smithi122 ceph-mon[32843]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T13:36:14.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:14 smithi045 ceph-mon[32731]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T13:36:14.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:14 smithi027 ceph-mon[29140]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T13:36:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:16 smithi122 ceph-mon[32843]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T13:36:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:16 smithi045 ceph-mon[32731]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T13:36:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:16 smithi027 ceph-mon[29140]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T13:36:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:18 smithi122 ceph-mon[32843]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T13:36:18.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:18 smithi045 ceph-mon[32731]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T13:36:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:18 smithi027 ceph-mon[29140]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T13:36:20.323 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:20 smithi122 ceph-mon[32843]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:36:20.323 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T13:36:20.323 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:20.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:20 smithi045 ceph-mon[32731]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:36:20.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:20 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T13:36:20.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:20 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:20.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:20 smithi027 ceph-mon[29140]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:36:20.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T13:36:20.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:21.326 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:21 smithi122 ceph-mon[32843]: Deploying daemon osd.8 on smithi122 2024-04-25T13:36:21.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:21 smithi045 ceph-mon[32731]: Deploying daemon osd.8 on smithi122 2024-04-25T13:36:21.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:21 smithi027 ceph-mon[29140]: Deploying daemon osd.8 on smithi122 2024-04-25T13:36:22.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:22 smithi122 ceph-mon[32843]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:22.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:22 smithi045 ceph-mon[32731]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:22.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:22 smithi027 ceph-mon[29140]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:24.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:24 smithi045 ceph-mon[32731]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:24.334 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:24 smithi122 ceph-mon[32843]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:24 smithi027 ceph-mon[29140]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:26.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:26 smithi045 ceph-mon[32731]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:26.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:26.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:26.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:26.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:26 smithi027 ceph-mon[29140]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:26.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:26.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:26.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:26.423 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:26 smithi122 ceph-mon[32843]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:26.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:26.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:26.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:28.388 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:28 smithi122 ceph-mon[32843]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:28.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:28 smithi027 ceph-mon[29140]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:28.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:28 smithi045 ceph-mon[32731]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:29.768 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.768+0000 7fbb06ffd700 1 -- 172.21.15.122:0/970944173 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbb10063b90 con 0x7fbafc0afb80 2024-04-25T13:36:29.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:29 smithi122 ceph-mon[32843]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:29.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:29.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:29.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:29.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:29.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:29.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:29 smithi122 ceph-mon[32843]: from='osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T13:36:29.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:29 smithi122 ceph-mon[32843]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T13:36:29.769 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 8 on host 'smithi122' 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbafc0afb80 msgr2=0x7fbafc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbafc0afb80 0x7fbafc0b2090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbb10105240 tx=0x7fbb0c007a30 comp rx=0 tx=0).stop 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 msgr2=0x7fbb1019ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 0x7fbb1019ac80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb000098e0 tx=0x7fbb000047b0 comp rx=0 tx=0).stop 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 shutdown_connections 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbafc0afb80 0x7fbafc0b2090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb10105ce0 0x7fbb101a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb10104210 0x7fbb1019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 --2- 172.21.15.122:0/970944173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb10105340 0x7fbb1019ac80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 >> 172.21.15.122:0/970944173 conn(0x7fbb100ff870 msgr2=0x7fbb10108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 shutdown_connections 2024-04-25T13:36:29.771 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:36:29.771+0000 7fbb17dc5700 1 -- 172.21.15.122:0/970944173 wait complete. 2024-04-25T13:36:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:29 smithi027 ceph-mon[29140]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:29 smithi027 ceph-mon[29140]: from='osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T13:36:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:29 smithi027 ceph-mon[29140]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T13:36:30.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:29 smithi045 ceph-mon[32731]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:30.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:30.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:30.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:29 smithi045 ceph-mon[32731]: from='osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T13:36:30.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:29 smithi045 ceph-mon[32731]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T13:36:30.456 DEBUG:teuthology.orchestra.run.smithi122:osd.8> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T13:36:30.458 INFO:tasks.cephadm:Deploying osd.9 on smithi027 with /dev/vg_nvme/lv_1... 2024-04-25T13:36:30.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T13:36:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:36:30.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:30.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:30.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:30.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:30 smithi027 ceph-mon[29140]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T13:36:30.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:30 smithi027 ceph-mon[29140]: from='osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:36:30.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:30 smithi027 ceph-mon[29140]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T13:36:30.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:30.839 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:30 smithi027 ceph-mon[29140]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:36:30.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:30.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:30.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:30.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:30 smithi122 ceph-mon[32843]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T13:36:30.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:30 smithi122 ceph-mon[32843]: from='osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:36:30.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:30 smithi122 ceph-mon[32843]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T13:36:30.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:30.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:30 smithi122 ceph-mon[32843]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:36:31.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:31.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:31.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:31.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:30 smithi045 ceph-mon[32731]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T13:36:31.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:30 smithi045 ceph-mon[32731]: from='osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:36:31.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:30 smithi045 ceph-mon[32731]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T13:36:31.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:31.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:30 smithi045 ceph-mon[32731]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:36:31.809 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:36:31 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:36:31.511+0000 7f0f4889d700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T13:36:31.809 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:36:31 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:36:31.522+0000 7f0f4064a700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:36:32.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:32 smithi027 ceph-mon[29140]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:32.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:32 smithi027 ceph-mon[29140]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:36:32.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:32 smithi027 ceph-mon[29140]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T13:36:32.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:32 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:32.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:32 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:32.130 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:32 smithi027 ceph-mon[29140]: from='osd.8 ' entity='osd.8' 2024-04-25T13:36:32.142 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:32 smithi122 ceph-mon[32843]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:32.142 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:32 smithi122 ceph-mon[32843]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:36:32.142 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:32 smithi122 ceph-mon[32843]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T13:36:32.142 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:32 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:32.142 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:32 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:32.143 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:32 smithi122 ceph-mon[32843]: from='osd.8 ' entity='osd.8' 2024-04-25T13:36:32.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:32 smithi045 ceph-mon[32731]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:32.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:32 smithi045 ceph-mon[32731]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:36:32.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:32 smithi045 ceph-mon[32731]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T13:36:32.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:32 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:32.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:32 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:32.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:32 smithi045 ceph-mon[32731]: from='osd.8 ' entity='osd.8' 2024-04-25T13:36:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T13:36:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T13:36:32.836 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-25T13:36:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-25T13:36:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128025 s, 819 MB/s 2024-04-25T13:36:32.836 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-25T13:36:33.245 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:33 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:36:33.246 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:33 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:36:33.246 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:33 smithi122 ceph-mon[32843]: osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] boot 2024-04-25T13:36:33.246 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:33 smithi122 ceph-mon[32843]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T13:36:33.246 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:33 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:36:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:33 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:36:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:33 smithi027 ceph-mon[29140]: osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] boot 2024-04-25T13:36:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:33 smithi027 ceph-mon[29140]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T13:36:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:33.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:33 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:36:33.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:33 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:36:33.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:33 smithi045 ceph-mon[32731]: osd.8 [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] boot 2024-04-25T13:36:33.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:33 smithi045 ceph-mon[32731]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T13:36:33.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T13:36:33.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_1 2024-04-25T13:36:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:36:34.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:34.289 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:34 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:34.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:34 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-25T13:36:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T13:36:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:36:34.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:34.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:34.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:34.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:34 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:35.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:35 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi122 2024-04-25T13:36:35.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:35 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi122 to 5696M 2024-04-25T13:36:35.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:35 smithi027 ceph-mon[29140]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T13:36:35.175 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:35 smithi027 ceph-mon[29140]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:35.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:35 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi122 2024-04-25T13:36:35.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:35 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi122 to 5696M 2024-04-25T13:36:35.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:35 smithi122 ceph-mon[32843]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T13:36:35.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:35 smithi122 ceph-mon[32843]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:35.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:35 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi122 2024-04-25T13:36:35.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:35 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi122 to 5696M 2024-04-25T13:36:35.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:35 smithi045 ceph-mon[32731]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T13:36:35.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:35 smithi045 ceph-mon[32731]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:35.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.804+0000 7fa385cab700 1 -- 172.21.15.27:0/3377671714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 msgr2=0x7fa38010b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:35.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.804+0000 7fa385cab700 1 --2- 172.21.15.27:0/3377671714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 0x7fa38010b460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa37800b0a0 tx=0x7fa37800b3b0 comp rx=0 tx=0).stop 2024-04-25T13:36:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.804+0000 7fa385cab700 1 -- 172.21.15.27:0/3377671714 shutdown_connections 2024-04-25T13:36:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.804+0000 7fa385cab700 1 --2- 172.21.15.27:0/3377671714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3801030e0 0x7fa3801034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.804+0000 7fa385cab700 1 --2- 172.21.15.27:0/3377671714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 0x7fa38010b460 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.804+0000 7fa385cab700 1 --2- 172.21.15.27:0/3377671714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa380104210 0x7fa380104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.805+0000 7fa385cab700 1 -- 172.21.15.27:0/3377671714 >> 172.21.15.27:0/3377671714 conn(0x7fa3800fe740 msgr2=0x7fa380100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:36:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.805+0000 7fa385cab700 1 -- 172.21.15.27:0/3377671714 shutdown_connections 2024-04-25T13:36:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.805+0000 7fa385cab700 1 -- 172.21.15.27:0/3377671714 wait complete. 2024-04-25T13:36:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.805+0000 7fa385cab700 1 Processor -- start 2024-04-25T13:36:35.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.805+0000 7fa385cab700 1 -- start start 2024-04-25T13:36:35.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa385cab700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3801030e0 0x7fa38019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:36:35.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa385cab700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa380104210 0x7fa38019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:36:35.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa385cab700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 0x7fa3801a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:36:35.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa385cab700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa38010fc40 con 0x7fa380104bb0 2024-04-25T13:36:35.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa385cab700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa38010fac0 con 0x7fa380104210 2024-04-25T13:36:35.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa385cab700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa38010fdc0 con 0x7fa3801030e0 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 0x7fa3801a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 0x7fa3801a1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46984/0 (socket says 172.21.15.27:46984) 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 -- 172.21.15.27:0/1462817649 learned_addr learned my addr 172.21.15.27:0/1462817649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37effd700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa380104210 0x7fa38019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37f7fe700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3801030e0 0x7fa38019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 -- 172.21.15.27:0/1462817649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3801030e0 msgr2=0x7fa38019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3801030e0 0x7fa38019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 -- 172.21.15.27:0/1462817649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa380104210 msgr2=0x7fa38019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa380104210 0x7fa38019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 -- 172.21.15.27:0/1462817649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3801a2200 con 0x7fa380104bb0 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37effd700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa380104210 0x7fa38019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.806+0000 7fa37ffff700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 0x7fa3801a1c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3780062a0 tx=0x7fa378004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:36:35.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.807+0000 7fa37cff9700 1 -- 172.21.15.27:0/1462817649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378021070 con 0x7fa380104bb0 2024-04-25T13:36:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.807+0000 7fa37cff9700 1 -- 172.21.15.27:0/1462817649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa378007c80 con 0x7fa380104bb0 2024-04-25T13:36:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.807+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3801a2490 con 0x7fa380104bb0 2024-04-25T13:36:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.807+0000 7fa37cff9700 1 -- 172.21.15.27:0/1462817649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378004850 con 0x7fa380104bb0 2024-04-25T13:36:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.807+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3801a29d0 con 0x7fa380104bb0 2024-04-25T13:36:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.808+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa380051190 con 0x7fa380104bb0 2024-04-25T13:36:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.809+0000 7fa37cff9700 1 -- 172.21.15.27:0/1462817649 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa378003a00 con 0x7fa380104bb0 2024-04-25T13:36:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.810+0000 7fa37cff9700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3680afb80 0x7fa3680b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:36:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.810+0000 7fa37f7fe700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3680afb80 0x7fa3680b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:36:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.810+0000 7fa37cff9700 1 -- 172.21.15.27:0/1462817649 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7fa37800f020 con 0x7fa380104bb0 2024-04-25T13:36:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.810+0000 7fa37f7fe700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3680afb80 0x7fa3680b2090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa370005fd0 tx=0x7fa370005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:36:35.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:35.814+0000 7fa37cff9700 1 -- 172.21.15.27:0/1462817649 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa37802a020 con 0x7fa380104bb0 2024-04-25T13:36:36.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:36.031+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa380063ad0 con 0x7fa3680afb80 2024-04-25T13:36:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:36 smithi027 ceph-mon[29140]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:36:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:36 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:36:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:36 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:36:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:36 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:36.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:36 smithi122 ceph-mon[32843]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:36:36.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:36 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:36:36.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:36 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:36:36.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:36 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:36.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:36 smithi045 ceph-mon[32731]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:36:36.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:36 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:36:36.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:36 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:36:36.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:36 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:37.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:37 smithi027 ceph-mon[29140]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-25T13:36:37.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:37 smithi122 ceph-mon[32843]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-25T13:36:37.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:37 smithi045 ceph-mon[32731]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-25T13:36:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:39 smithi027 ceph-mon[29140]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T13:36:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:39 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/392730056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43d7183-1100-4873-9968-72ba295cef58"}]: dispatch 2024-04-25T13:36:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:39 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/392730056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b43d7183-1100-4873-9968-72ba295cef58"}]': finished 2024-04-25T13:36:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:39 smithi027 ceph-mon[29140]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T13:36:39.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:39 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:39 smithi122 ceph-mon[32843]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T13:36:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:39 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/392730056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43d7183-1100-4873-9968-72ba295cef58"}]: dispatch 2024-04-25T13:36:39.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:39 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/392730056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b43d7183-1100-4873-9968-72ba295cef58"}]': finished 2024-04-25T13:36:39.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:39 smithi122 ceph-mon[32843]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T13:36:39.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:39 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:39 smithi045 ceph-mon[32731]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T13:36:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:39 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/392730056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b43d7183-1100-4873-9968-72ba295cef58"}]: dispatch 2024-04-25T13:36:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:39 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/392730056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b43d7183-1100-4873-9968-72ba295cef58"}]': finished 2024-04-25T13:36:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:39 smithi045 ceph-mon[32731]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T13:36:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:39 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:40.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:40 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/634246712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:36:40.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:40 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/634246712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:36:40.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:40 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/634246712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:36:41.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:41 smithi027 ceph-mon[29140]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T13:36:41.728 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:41 smithi045 ceph-mon[32731]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T13:36:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:41 smithi122 ceph-mon[32843]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T13:36:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:43 smithi027 ceph-mon[29140]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-25T13:36:43.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:43 smithi122 ceph-mon[32843]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-25T13:36:43.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:43 smithi045 ceph-mon[32731]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-25T13:36:45.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:45 smithi027 ceph-mon[29140]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:36:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:45 smithi122 ceph-mon[32843]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:36:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:45 smithi045 ceph-mon[32731]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:36:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:47 smithi027 ceph-mon[29140]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:47 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T13:36:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:47 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:47 smithi027 ceph-mon[29140]: Deploying daemon osd.9 on smithi027 2024-04-25T13:36:47.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:47 smithi122 ceph-mon[32843]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:47.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:47 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T13:36:47.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:47 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:47.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:47 smithi122 ceph-mon[32843]: Deploying daemon osd.9 on smithi027 2024-04-25T13:36:47.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:47 smithi045 ceph-mon[32731]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:47.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:47 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T13:36:47.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:47 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:47.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:47 smithi045 ceph-mon[32731]: Deploying daemon osd.9 on smithi027 2024-04-25T13:36:49.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:49 smithi027 ceph-mon[29140]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:49.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:49 smithi122 ceph-mon[32843]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:49.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:49 smithi045 ceph-mon[32731]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:51.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:51 smithi027 ceph-mon[29140]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:51.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:51 smithi122 ceph-mon[32843]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:51.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:51 smithi045 ceph-mon[32731]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:54 smithi122 ceph-mon[32843]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:54 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:54 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:54 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:54 smithi045 ceph-mon[32731]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:54 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:54 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:54 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:54.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:54 smithi027 ceph-mon[29140]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:54.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:54 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:54.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:54 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:54.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:54 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:56.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:56 smithi122 ceph-mon[32843]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:56 smithi045 ceph-mon[32731]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:56 smithi027 ceph-mon[29140]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:57.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.244+0000 7fa37cff9700 1 -- 172.21.15.27:0/1462817649 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa380063ad0 con 0x7fa3680afb80 2024-04-25T13:36:57.245 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 9 on host 'smithi027' 2024-04-25T13:36:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3680afb80 msgr2=0x7fa3680b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3680afb80 0x7fa3680b2090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa370005fd0 tx=0x7fa370005e50 comp rx=0 tx=0).stop 2024-04-25T13:36:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 msgr2=0x7fa3801a1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:36:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 0x7fa3801a1c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3780062a0 tx=0x7fa378004e40 comp rx=0 tx=0).stop 2024-04-25T13:36:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 shutdown_connections 2024-04-25T13:36:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3680afb80 0x7fa3680b2090 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3801030e0 0x7fa38019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa380104bb0 0x7fa3801a1c10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 --2- 172.21.15.27:0/1462817649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa380104210 0x7fa38019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:36:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 >> 172.21.15.27:0/1462817649 conn(0x7fa3800fe740 msgr2=0x7fa380107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:36:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 shutdown_connections 2024-04-25T13:36:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:36:57.246+0000 7fa385cab700 1 -- 172.21.15.27:0/1462817649 wait complete. 2024-04-25T13:36:57.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:57 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.576 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:57 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:36:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:57 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:36:57.937 DEBUG:teuthology.orchestra.run.smithi027:osd.9> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T13:36:57.939 INFO:tasks.cephadm:Deploying osd.10 on smithi045 with /dev/vg_nvme/lv_1... 2024-04-25T13:36:57.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T13:36:58.196 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T13:36:58.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:58 smithi027 ceph-mon[29140]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T13:36:58.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:58 smithi027 ceph-mon[29140]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T13:36:58.438 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:58 smithi027 ceph-mon[29140]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:36:58.439 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:58 smithi122 ceph-mon[32843]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T13:36:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:58 smithi122 ceph-mon[32843]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T13:36:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:58 smithi122 ceph-mon[32843]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:36:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:58.567 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:58 smithi045 ceph-mon[32731]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T13:36:58.567 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:58 smithi045 ceph-mon[32731]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T13:36:58.567 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:58 smithi045 ceph-mon[32731]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-25T13:36:58.568 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:59.151 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:36:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:36:58.723+0000 7f98c4a0a700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T13:36:59.151 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:36:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:36:58.735+0000 7f98bc7b7700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:36:59.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:59 smithi027 ceph-mon[29140]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:36:59.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:59 smithi027 ceph-mon[29140]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T13:36:59.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:59 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:59.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:59 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:59.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:59 smithi027 ceph-mon[29140]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:59.535 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:36:59 smithi027 ceph-mon[29140]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' 2024-04-25T13:36:59.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:59 smithi122 ceph-mon[32843]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:36:59.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:59 smithi122 ceph-mon[32843]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T13:36:59.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:59 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:59.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:59 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:59.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:59 smithi122 ceph-mon[32843]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:59.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:36:59 smithi122 ceph-mon[32843]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' 2024-04-25T13:36:59.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:59 smithi045 ceph-mon[32731]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-25T13:36:59.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:59 smithi045 ceph-mon[32731]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T13:36:59.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:59 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:59.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:59 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:36:59.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:59 smithi045 ceph-mon[32731]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T13:36:59.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:36:59 smithi045 ceph-mon[32731]: from='osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727]' entity='osd.9' 2024-04-25T13:37:00.333 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T13:37:00.333 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T13:37:00.333 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-04-25T13:37:00.333 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-04-25T13:37:00.333 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00856986 s, 1.2 GB/s 2024-04-25T13:37:00.333 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-04-25T13:37:00.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:00 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:37:00.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:00 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:37:00.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:00 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:37:00.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:00 smithi122 ceph-mon[32843]: osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] boot 2024-04-25T13:37:00.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:00 smithi122 ceph-mon[32843]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T13:37:00.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:00 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:37:00.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:00 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:37:00.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:00 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:37:00.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:00 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:37:00.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:00 smithi045 ceph-mon[32731]: osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] boot 2024-04-25T13:37:00.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:00 smithi045 ceph-mon[32731]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T13:37:00.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:00 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:37:00.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:00 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:37:00.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:00 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:37:00.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:00 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:37:00.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:00 smithi027 ceph-mon[29140]: osd.9 [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] boot 2024-04-25T13:37:00.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:00 smithi027 ceph-mon[29140]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T13:37:00.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:00 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T13:37:00.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi045:vg_nvme/lv_1 2024-04-25T13:37:01.242 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T13:37:02.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:02.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T13:37:02.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:02.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:02.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:01 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T13:37:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:02.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:02.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:01 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:02.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:02.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:02.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:01 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:02.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 -- 172.21.15.45:0/2011622528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c70104f00 msgr2=0x7f6c7010b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:02.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 --2- 172.21.15.45:0/2011622528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c70104f00 0x7f6c7010b800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c600099c0 tx=0x7f6c60009cd0 comp rx=0 tx=0).stop 2024-04-25T13:37:02.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 -- 172.21.15.45:0/2011622528 shutdown_connections 2024-04-25T13:37:02.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 --2- 172.21.15.45:0/2011622528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c70104f00 0x7f6c7010b800 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:02.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 --2- 172.21.15.45:0/2011622528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c701036b0 0x7f6c70103ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:02.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 --2- 172.21.15.45:0/2011622528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c701045b0 0x7f6c701049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:02.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 -- 172.21.15.45:0/2011622528 >> 172.21.15.45:0/2011622528 conn(0x7f6c700fde80 msgr2=0x7f6c70100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:37:02.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 -- 172.21.15.45:0/2011622528 shutdown_connections 2024-04-25T13:37:02.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.980+0000 7f6c7547e700 1 -- 172.21.15.45:0/2011622528 wait complete. 2024-04-25T13:37:02.982 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:02 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi027 2024-04-25T13:37:02.982 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:02 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T13:37:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.981+0000 7f6c7547e700 1 Processor -- start 2024-04-25T13:37:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.981+0000 7f6c7547e700 1 -- start start 2024-04-25T13:37:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c7547e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c701036b0 0x7f6c70102d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c7547e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c701045b0 0x7f6c7019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c7547e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c70104f00 0x7f6c701a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c7547e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7010ed80 con 0x7f6c70104f00 2024-04-25T13:37:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c7547e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7010ec00 con 0x7f6c701036b0 2024-04-25T13:37:02.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c7547e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7010ef00 con 0x7f6c701045b0 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c701036b0 0x7f6c70102d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c701036b0 0x7f6c70102d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59944/0 (socket says 172.21.15.45:59944) 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 -- 172.21.15.45:0/1917738569 learned_addr learned my addr 172.21.15.45:0/1917738569 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6e7fc700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c701045b0 0x7f6c7019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6f7fe700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c70104f00 0x7f6c701a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 -- 172.21.15.45:0/1917738569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c701045b0 msgr2=0x7f6c7019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c701045b0 0x7f6c7019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 -- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c70104f00 msgr2=0x7f6c701a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c70104f00 0x7f6c701a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 -- 172.21.15.45:0/1917738569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c701a2490 con 0x7f6c701036b0 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.982+0000 7f6c6effd700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c701036b0 0x7f6c70102d70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800ada0 tx=0x7f6c6800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.983+0000 7f6c67fff700 1 -- 172.21.15.45:0/1917738569 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68017020 con 0x7f6c701036b0 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.983+0000 7f6c67fff700 1 -- 172.21.15.45:0/1917738569 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c6800cd80 con 0x7f6c701036b0 2024-04-25T13:37:02.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.983+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c701a2780 con 0x7f6c701036b0 2024-04-25T13:37:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.983+0000 7f6c67fff700 1 -- 172.21.15.45:0/1917738569 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68005260 con 0x7f6c701036b0 2024-04-25T13:37:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.983+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c701a2cc0 con 0x7f6c701036b0 2024-04-25T13:37:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.984+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c70051190 con 0x7f6c701036b0 2024-04-25T13:37:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.985+0000 7f6c67fff700 1 -- 172.21.15.45:0/1917738569 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6c6800c8d0 con 0x7f6c701036b0 2024-04-25T13:37:02.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.986+0000 7f6c67fff700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6c5c0afb80 0x7f6c5c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.986+0000 7f6c67fff700 1 -- 172.21.15.45:0/1917738569 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7f6c680ddb40 con 0x7f6c701036b0 2024-04-25T13:37:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.986+0000 7f6c6e7fc700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6c5c0afb80 0x7f6c5c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:02.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.987+0000 7f6c6e7fc700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6c5c0afb80 0x7f6c5c0b2090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c70102720 tx=0x7f6c5800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:37:02.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:02.991+0000 7f6c67fff700 1 -- 172.21.15.45:0/1917738569 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c68021020 con 0x7f6c701036b0 2024-04-25T13:37:03.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:02 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi027 2024-04-25T13:37:03.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:02 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T13:37:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:02 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi027 2024-04-25T13:37:03.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:02 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi027 to 4272M 2024-04-25T13:37:03.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:03.208+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6c7002e840 con 0x7f6c5c0afb80 2024-04-25T13:37:03.991 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:03 smithi045 ceph-mon[32731]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:03.991 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:37:03.991 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:37:03.991 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:04.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:03 smithi122 ceph-mon[32843]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:04.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:37:04.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:37:04.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:04.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:03 smithi027 ceph-mon[29140]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:04.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:37:04.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:37:04.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:04.901 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:04 smithi045 ceph-mon[32731]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:37:05.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:04 smithi122 ceph-mon[32843]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:37:05.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:04 smithi027 ceph-mon[29140]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:37:06.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:05 smithi122 ceph-mon[32843]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:06.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:05 smithi045 ceph-mon[32731]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:06.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:05 smithi027 ceph-mon[29140]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:06.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:06 smithi045 ceph-mon[32731]: from='client.? 172.21.15.45:0/1992920369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]: dispatch 2024-04-25T13:37:06.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:06 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]: dispatch 2024-04-25T13:37:06.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:06 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]': finished 2024-04-25T13:37:06.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:06 smithi045 ceph-mon[32731]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T13:37:06.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:06 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:07.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:06 smithi122 ceph-mon[32843]: from='client.? 172.21.15.45:0/1992920369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]: dispatch 2024-04-25T13:37:07.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:06 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]: dispatch 2024-04-25T13:37:07.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:06 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]': finished 2024-04-25T13:37:07.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:06 smithi122 ceph-mon[32843]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T13:37:07.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:06 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:07.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:06 smithi027 ceph-mon[29140]: from='client.? 172.21.15.45:0/1992920369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]: dispatch 2024-04-25T13:37:07.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:06 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]: dispatch 2024-04-25T13:37:07.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:06 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8438652f-3c76-4b50-8995-cd190d70f017"}]': finished 2024-04-25T13:37:07.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:06 smithi027 ceph-mon[29140]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T13:37:07.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:06 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:08.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:07 smithi122 ceph-mon[32843]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:08.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:07 smithi122 ceph-mon[32843]: from='client.? 172.21.15.45:0/3274158647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:37:08.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:07 smithi045 ceph-mon[32731]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:08.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:07 smithi045 ceph-mon[32731]: from='client.? 172.21.15.45:0/3274158647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:37:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:07 smithi027 ceph-mon[29140]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:07 smithi027 ceph-mon[29140]: from='client.? 172.21.15.45:0/3274158647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:37:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:09 smithi122 ceph-mon[32843]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:10.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:09 smithi045 ceph-mon[32731]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:10.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:09 smithi027 ceph-mon[29140]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:12.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:11 smithi122 ceph-mon[32843]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:12.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:11 smithi045 ceph-mon[32731]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:12.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:11 smithi027 ceph-mon[29140]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:14.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:13 smithi122 ceph-mon[32843]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:14.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:13 smithi045 ceph-mon[32731]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:13 smithi027 ceph-mon[29140]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:15.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T13:37:15.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:15.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:14 smithi122 ceph-mon[32843]: Deploying daemon osd.10 on smithi045 2024-04-25T13:37:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:14 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T13:37:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:14 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:14 smithi045 ceph-mon[32731]: Deploying daemon osd.10 on smithi045 2024-04-25T13:37:15.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:14 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T13:37:15.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:14 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:15.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:14 smithi027 ceph-mon[29140]: Deploying daemon osd.10 on smithi045 2024-04-25T13:37:16.024 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:15 smithi045 ceph-mon[32731]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:16.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:15 smithi122 ceph-mon[32843]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:16.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:15 smithi027 ceph-mon[29140]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:18.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:17 smithi122 ceph-mon[32843]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:18.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:17 smithi045 ceph-mon[32731]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:18.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:17 smithi027 ceph-mon[29140]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:19 smithi122 ceph-mon[32843]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:19 smithi045 ceph-mon[32731]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:20.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:19 smithi027 ceph-mon[29140]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:21.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:21.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:21.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:21.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:20 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:21.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:20 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:21.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:20 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:21.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:21.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:21.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:22.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:21 smithi122 ceph-mon[32843]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:22.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:21 smithi045 ceph-mon[32731]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:22.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:21 smithi027 ceph-mon[29140]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:23.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.928+0000 7f6c67fff700 1 -- 172.21.15.45:0/1917738569 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6c7002e840 con 0x7f6c5c0afb80 2024-04-25T13:37:23.929 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 10 on host 'smithi045' 2024-04-25T13:37:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.930+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6c5c0afb80 msgr2=0x7f6c5c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.930+0000 7f6c7547e700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6c5c0afb80 0x7f6c5c0b2090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c70102720 tx=0x7f6c5800b410 comp rx=0 tx=0).stop 2024-04-25T13:37:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c701036b0 msgr2=0x7f6c70102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c701036b0 0x7f6c70102d70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800ada0 tx=0x7f6c6800c330 comp rx=0 tx=0).stop 2024-04-25T13:37:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 shutdown_connections 2024-04-25T13:37:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6c5c0afb80 0x7f6c5c0b2090 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:23.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c70104f00 0x7f6c701a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:23.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c701036b0 0x7f6c70102d70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:23.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 --2- 172.21.15.45:0/1917738569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c701045b0 0x7f6c7019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:23.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 >> 172.21.15.45:0/1917738569 conn(0x7f6c700fde80 msgr2=0x7f6c701077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:37:23.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 shutdown_connections 2024-04-25T13:37:23.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-25T13:37:23.931+0000 7f6c7547e700 1 -- 172.21.15.45:0/1917738569 wait complete. 2024-04-25T13:37:24.267 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:24.267 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.267 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.268 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:24.268 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:24.268 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.268 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T13:37:24.268 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T13:37:24.268 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.268 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.268 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:24 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T13:37:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T13:37:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:24 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:24.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:24 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:24.614 DEBUG:teuthology.orchestra.run.smithi045:osd.10> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T13:37:24.617 INFO:tasks.cephadm:Deploying osd.11 on smithi122 with /dev/vg_nvme/lv_1... 2024-04-25T13:37:24.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T13:37:24.868 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:37:25.172 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:25 smithi045 ceph-mon[32731]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T13:37:25.172 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:25 smithi045 ceph-mon[32731]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T13:37:25.172 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:25.172 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:25 smithi045 ceph-mon[32731]: from='osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:37:25.172 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:25 smithi045 ceph-mon[32731]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:37:25.173 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:37:25 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:37:25.038+0000 7f393ca65700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T13:37:25.173 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:37:25 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:37:25.053+0000 7f393881a700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:37:25.247 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:25 smithi122 ceph-mon[32843]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T13:37:25.247 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:25 smithi122 ceph-mon[32843]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T13:37:25.247 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:25.248 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:25 smithi122 ceph-mon[32843]: from='osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:37:25.248 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:25 smithi122 ceph-mon[32843]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:37:25.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:25 smithi027 ceph-mon[29140]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T13:37:25.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:25 smithi027 ceph-mon[29140]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T13:37:25.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:25.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:25 smithi027 ceph-mon[29140]: from='osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:37:25.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:25 smithi027 ceph-mon[29140]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-04-25T13:37:26.153 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:26 smithi045 ceph-mon[32731]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:26.153 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:26 smithi045 ceph-mon[32731]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:37:26.153 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:26 smithi045 ceph-mon[32731]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T13:37:26.153 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:26.153 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:26.153 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:26 smithi045 ceph-mon[32731]: osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] boot 2024-04-25T13:37:26.153 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:26 smithi045 ceph-mon[32731]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T13:37:26.153 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:26.287 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:26 smithi027 ceph-mon[29140]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:26.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:26 smithi027 ceph-mon[29140]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:37:26.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:26 smithi027 ceph-mon[29140]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T13:37:26.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:26.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:26.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:26 smithi027 ceph-mon[29140]: osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] boot 2024-04-25T13:37:26.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:26 smithi027 ceph-mon[29140]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T13:37:26.288 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:26 smithi122 ceph-mon[32843]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-25T13:37:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:26 smithi122 ceph-mon[32843]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-04-25T13:37:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:26 smithi122 ceph-mon[32843]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T13:37:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:26 smithi122 ceph-mon[32843]: osd.10 [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] boot 2024-04-25T13:37:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:26 smithi122 ceph-mon[32843]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T13:37:26.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T13:37:27.033 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T13:37:27.033 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T13:37:27.033 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-25T13:37:27.033 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-25T13:37:27.033 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108948 s, 962 MB/s 2024-04-25T13:37:27.033 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-25T13:37:27.723 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_1 2024-04-25T13:37:27.969 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:37:28.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:37:28.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:37:28.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:28.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:37:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:37:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T13:37:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:28.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:37:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:37:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: pgmap v233: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T13:37:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:28.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:29.127 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:28 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi045 2024-04-25T13:37:29.128 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:28 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi045 to 4273M 2024-04-25T13:37:29.128 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:28 smithi122 ceph-mon[32843]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T13:37:29.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:28 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi045 2024-04-25T13:37:29.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:28 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi045 to 4273M 2024-04-25T13:37:29.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:28 smithi027 ceph-mon[29140]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T13:37:29.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:28 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi045 2024-04-25T13:37:29.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:28 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi045 to 4273M 2024-04-25T13:37:29.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:28 smithi045 ceph-mon[32731]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T13:37:29.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 -- 172.21.15.122:0/4223579453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 msgr2=0x7f0f90105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:29.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 --2- 172.21.15.122:0/4223579453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 0x7f0f90105750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009990 tx=0x7f0f84009ca0 comp rx=0 tx=0).stop 2024-04-25T13:37:29.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 -- 172.21.15.122:0/4223579453 shutdown_connections 2024-04-25T13:37:29.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 --2- 172.21.15.122:0/4223579453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f90105ce0 0x7f0f9010c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:29.703 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 --2- 172.21.15.122:0/4223579453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f90104210 0x7f0f90104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:29.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 --2- 172.21.15.122:0/4223579453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 0x7f0f90105750 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:29.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 -- 172.21.15.122:0/4223579453 >> 172.21.15.122:0/4223579453 conn(0x7f0f900ff870 msgr2=0x7f0f90101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:37:29.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 -- 172.21.15.122:0/4223579453 shutdown_connections 2024-04-25T13:37:29.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 -- 172.21.15.122:0/4223579453 wait complete. 2024-04-25T13:37:29.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.703+0000 7f0f960c3700 1 Processor -- start 2024-04-25T13:37:29.704 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f960c3700 1 -- start start 2024-04-25T13:37:29.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f960c3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f90104210 0x7f0f9019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:29.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f960c3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 0x7f0f9019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:29.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f960c3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f90105ce0 0x7f0f901a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:29.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f960c3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9010fc70 con 0x7f0f90104210 2024-04-25T13:37:29.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f960c3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9010faf0 con 0x7f0f90105ce0 2024-04-25T13:37:29.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f960c3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9010fdf0 con 0x7f0f90105340 2024-04-25T13:37:29.706 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f90104210 0x7f0f9019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f90104210 0x7f0f9019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:38578/0 (socket says 172.21.15.122:38578) 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8f7fe700 1 -- 172.21.15.122:0/190543341 learned_addr learned my addr 172.21.15.122:0/190543341 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8effd700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 0x7f0f9019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8ffff700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f90105ce0 0x7f0f901a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8f7fe700 1 -- 172.21.15.122:0/190543341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 msgr2=0x7f0f9019ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8f7fe700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 0x7f0f9019ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8f7fe700 1 -- 172.21.15.122:0/190543341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f90105ce0 msgr2=0x7f0f901a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8f7fe700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f90105ce0 0x7f0f901a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8f7fe700 1 -- 172.21.15.122:0/190543341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f901a22b0 con 0x7f0f90104210 2024-04-25T13:37:29.707 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.704+0000 7f0f8effd700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 0x7f0f9019ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:37:29.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.705+0000 7f0f8f7fe700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f90104210 0x7f0f9019a750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000d950 tx=0x7f0f8000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:37:29.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.705+0000 7f0f8cff9700 1 -- 172.21.15.122:0/190543341 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80017070 con 0x7f0f90104210 2024-04-25T13:37:29.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.705+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f901a25a0 con 0x7f0f90104210 2024-04-25T13:37:29.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.705+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f901a2b30 con 0x7f0f90104210 2024-04-25T13:37:29.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.705+0000 7f0f8cff9700 1 -- 172.21.15.122:0/190543341 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f8000f690 con 0x7f0f90104210 2024-04-25T13:37:29.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.705+0000 7f0f8cff9700 1 -- 172.21.15.122:0/190543341 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f800039e0 con 0x7f0f90104210 2024-04-25T13:37:29.708 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.706+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7c005360 con 0x7f0f90104210 2024-04-25T13:37:29.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.708+0000 7f0f8cff9700 1 -- 172.21.15.122:0/190543341 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0f80003b80 con 0x7f0f90104210 2024-04-25T13:37:29.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.708+0000 7f0f8cff9700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f780afb80 0x7f0f780b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:29.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.708+0000 7f0f8cff9700 1 -- 172.21.15.122:0/190543341 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7f0f800de7b0 con 0x7f0f90104210 2024-04-25T13:37:29.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.708+0000 7f0f8effd700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f780afb80 0x7f0f780b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:29.709 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.709+0000 7f0f8effd700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f780afb80 0x7f0f780b2090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f9019b800 tx=0x7f0f8400b720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:37:29.712 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.712+0000 7f0f8cff9700 1 -- 172.21.15.122:0/190543341 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f80020030 con 0x7f0f90104210 2024-04-25T13:37:29.920 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:29.920+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0f7c000bf0 con 0x7f0f780afb80 2024-04-25T13:37:30.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:29 smithi122 ceph-mon[32843]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:30.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:29 smithi027 ceph-mon[29140]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:30.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:29 smithi045 ceph-mon[32731]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:31.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:30 smithi027 ceph-mon[29140]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:37:31.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:37:31.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:37:31.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:31.212 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:30 smithi122 ceph-mon[32843]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:37:31.212 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:37:31.212 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:37:31.213 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:30 smithi045 ceph-mon[32731]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:37:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T13:37:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T13:37:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:31.959 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:31 smithi122 ceph-mon[32843]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-25T13:37:32.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:31 smithi027 ceph-mon[29140]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-25T13:37:32.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:31 smithi045 ceph-mon[32731]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-25T13:37:33.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:32 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/4274769505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]: dispatch 2024-04-25T13:37:33.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:32 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]: dispatch 2024-04-25T13:37:33.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:32 smithi027 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]': finished 2024-04-25T13:37:33.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:32 smithi027 ceph-mon[29140]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T13:37:33.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:32 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:33.213 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:32 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/4274769505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]: dispatch 2024-04-25T13:37:33.214 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:32 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]: dispatch 2024-04-25T13:37:33.214 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:32 smithi122 ceph-mon[32843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]': finished 2024-04-25T13:37:33.214 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:32 smithi122 ceph-mon[32843]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T13:37:33.214 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:32 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:32 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/4274769505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]: dispatch 2024-04-25T13:37:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:32 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]: dispatch 2024-04-25T13:37:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:32 smithi045 ceph-mon[32731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23f3a823-363d-42ca-b03c-64cca2ba75d8"}]': finished 2024-04-25T13:37:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:32 smithi045 ceph-mon[32731]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T13:37:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:32 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:34.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:33 smithi027 ceph-mon[29140]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T13:37:34.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:33 smithi027 ceph-mon[29140]: from='client.? 172.21.15.122:0/584696635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:37:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:33 smithi122 ceph-mon[32843]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T13:37:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:33 smithi122 ceph-mon[32843]: from='client.? 172.21.15.122:0/584696635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:37:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:33 smithi045 ceph-mon[32731]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T13:37:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:33 smithi045 ceph-mon[32731]: from='client.? 172.21.15.122:0/584696635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T13:37:36.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:35 smithi027 ceph-mon[29140]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T13:37:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:35 smithi122 ceph-mon[32843]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T13:37:36.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:35 smithi045 ceph-mon[32731]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T13:37:38.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:37 smithi122 ceph-mon[32843]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T13:37:38.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:37 smithi027 ceph-mon[29140]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T13:37:38.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:37 smithi045 ceph-mon[32731]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T13:37:40.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:39 smithi122 ceph-mon[32843]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:37:40.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:39 smithi027 ceph-mon[29140]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:37:40.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:39 smithi045 ceph-mon[32731]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T13:37:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T13:37:41.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:40 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:41.221 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T13:37:41.221 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:40 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:41.229 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T13:37:41.229 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:40 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:42.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:41 smithi027 ceph-mon[29140]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:42.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:41 smithi027 ceph-mon[29140]: Deploying daemon osd.11 on smithi122 2024-04-25T13:37:42.249 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:41 smithi122 ceph-mon[32843]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:42.249 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:41 smithi122 ceph-mon[32843]: Deploying daemon osd.11 on smithi122 2024-04-25T13:37:42.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:41 smithi045 ceph-mon[32731]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:42.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:41 smithi045 ceph-mon[32731]: Deploying daemon osd.11 on smithi122 2024-04-25T13:37:44.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:43 smithi122 ceph-mon[32843]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:44.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:43 smithi027 ceph-mon[29140]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:44.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:43 smithi045 ceph-mon[32731]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:46.132 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:45 smithi122 ceph-mon[32843]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:46.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:45 smithi027 ceph-mon[29140]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:46.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:45 smithi045 ceph-mon[32731]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:48.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:47 smithi027 ceph-mon[29140]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:48.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:47 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:48.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:47 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:48.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:47 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:47 smithi122 ceph-mon[32843]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:47 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:47 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:47 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:48.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:47 smithi045 ceph-mon[32731]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:48.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:47 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:48.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:47 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:48.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:47 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:50.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:49 smithi027 ceph-mon[29140]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:50.203 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:49 smithi122 ceph-mon[32843]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:50.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:49 smithi045 ceph-mon[32731]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:50.897 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.896+0000 7f0f8cff9700 1 -- 172.21.15.122:0/190543341 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f0f7c000bf0 con 0x7f0f780afb80 2024-04-25T13:37:50.897 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 11 on host 'smithi122' 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f780afb80 msgr2=0x7f0f780b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f780afb80 0x7f0f780b2090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0f9019b800 tx=0x7f0f8400b720 comp rx=0 tx=0).stop 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f90104210 msgr2=0x7f0f9019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f90104210 0x7f0f9019a750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000d950 tx=0x7f0f8000dc60 comp rx=0 tx=0).stop 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 shutdown_connections 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f780afb80 0x7f0f780b2090 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f90105ce0 0x7f0f901a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f90104210 0x7f0f9019a750 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:50.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 --2- 172.21.15.122:0/190543341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f90105340 0x7f0f9019ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:50.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 >> 172.21.15.122:0/190543341 conn(0x7f0f900ff870 msgr2=0x7f0f90108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:37:50.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 shutdown_connections 2024-04-25T13:37:50.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:37:50.898+0000 7f0f960c3700 1 -- 172.21.15.122:0/190543341 wait complete. 2024-04-25T13:37:51.270 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.271 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:50 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:50 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:51.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:50 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:37:51.622 DEBUG:teuthology.orchestra.run.smithi122:osd.11> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T13:37:51.624 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T13:37:51.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T13:37:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:37:52.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:51 smithi122 ceph-mon[32843]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:52.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:51 smithi122 ceph-mon[32843]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T13:37:52.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:51 smithi122 ceph-mon[32843]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T13:37:52.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:51 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:52.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:51 smithi122 ceph-mon[32843]: from='osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:37:52.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:51 smithi122 ceph-mon[32843]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:37:52.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:51 smithi027 ceph-mon[29140]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:52.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:51 smithi027 ceph-mon[29140]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T13:37:52.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:51 smithi027 ceph-mon[29140]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T13:37:52.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:51 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:52.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:51 smithi027 ceph-mon[29140]: from='osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:37:52.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:51 smithi027 ceph-mon[29140]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:37:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:51 smithi045 ceph-mon[32731]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:51 smithi045 ceph-mon[32731]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T13:37:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:51 smithi045 ceph-mon[32731]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T13:37:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:51 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:51 smithi045 ceph-mon[32731]: from='osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:37:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:51 smithi045 ceph-mon[32731]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-25T13:37:52.809 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:37:52 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:37:52.392+0000 7fc9f3f11700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T13:37:52.809 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:37:52 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:37:52.411+0000 7fc9ebcbe700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T13:37:53.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:52 smithi027 ceph-mon[29140]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:37:53.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:52 smithi027 ceph-mon[29140]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T13:37:53.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:52 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:53.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:52 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:53.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:52 smithi027 ceph-mon[29140]: from='osd.11 ' entity='osd.11' 2024-04-25T13:37:53.177 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:52 smithi122 ceph-mon[32843]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:37:53.177 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:52 smithi122 ceph-mon[32843]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T13:37:53.177 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:52 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:53.177 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:52 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:53.177 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:52 smithi122 ceph-mon[32843]: from='osd.11 ' entity='osd.11' 2024-04-25T13:37:53.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:52 smithi045 ceph-mon[32731]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-25T13:37:53.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:52 smithi045 ceph-mon[32731]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T13:37:53.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:52 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:53.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:52 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:53.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:52 smithi045 ceph-mon[32731]: from='osd.11 ' entity='osd.11' 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.813+0000 7fa1218c2700 1 -- 172.21.15.27:0/3263119483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 msgr2=0x7fa11c10c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.813+0000 7fa1218c2700 1 --2- 172.21.15.27:0/3263119483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 0x7fa11c10c3a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa11400b0a0 tx=0x7fa11400b3b0 comp rx=0 tx=0).stop 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 -- 172.21.15.27:0/3263119483 shutdown_connections 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 --2- 172.21.15.27:0/3263119483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 0x7fa11c10c3a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 --2- 172.21.15.27:0/3263119483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa11c06c1a0 0x7fa11c106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 --2- 172.21.15.27:0/3263119483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa11c107320 0x7fa11c109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 -- 172.21.15.27:0/3263119483 >> 172.21.15.27:0/3263119483 conn(0x7fa11c0fe740 msgr2=0x7fa11c100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 -- 172.21.15.27:0/3263119483 shutdown_connections 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 -- 172.21.15.27:0/3263119483 wait complete. 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 Processor -- start 2024-04-25T13:37:53.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.814+0000 7fa1218c2700 1 -- start start 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa1218c2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa11c06c1a0 0x7fa11c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa1218c2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa11c107320 0x7fa11c19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa1218c2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 0x7fa11c1a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa1218c2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c10fc30 con 0x7fa11c109e90 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa1218c2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c10fab0 con 0x7fa11c107320 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa1218c2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c10fdb0 con 0x7fa11c06c1a0 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 0x7fa11c1a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11affd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa11c06c1a0 0x7fa11c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11affd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa11c06c1a0 0x7fa11c19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:48452/0 (socket says 172.21.15.27:48452) 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11affd700 1 -- 172.21.15.27:0/670503074 learned_addr learned my addr 172.21.15.27:0/670503074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11a7fc700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa11c107320 0x7fa11c19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11b7fe700 1 -- 172.21.15.27:0/670503074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa11c06c1a0 msgr2=0x7fa11c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11b7fe700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa11c06c1a0 0x7fa11c19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11b7fe700 1 -- 172.21.15.27:0/670503074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa11c107320 msgr2=0x7fa11c19ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11b7fe700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa11c107320 0x7fa11c19ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11b7fe700 1 -- 172.21.15.27:0/670503074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c1a2260 con 0x7fa11c109e90 2024-04-25T13:37:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.815+0000 7fa11b7fe700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 0x7fa11c1a1be0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa11400b070 tx=0x7fa114004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:37:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.816+0000 7fa1208c0700 1 -- 172.21.15.27:0/670503074 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114021070 con 0x7fa11c109e90 2024-04-25T13:37:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.816+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c1a24f0 con 0x7fa11c109e90 2024-04-25T13:37:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.816+0000 7fa1208c0700 1 -- 172.21.15.27:0/670503074 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa11401d450 con 0x7fa11c109e90 2024-04-25T13:37:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.816+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11c1a2a80 con 0x7fa11c109e90 2024-04-25T13:37:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.816+0000 7fa1208c0700 1 -- 172.21.15.27:0/670503074 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114005280 con 0x7fa11c109e90 2024-04-25T13:37:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.817+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa108005360 con 0x7fa11c109e90 2024-04-25T13:37:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.818+0000 7fa1208c0700 1 -- 172.21.15.27:0/670503074 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa114024070 con 0x7fa11c109e90 2024-04-25T13:37:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.819+0000 7fa1208c0700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1040afc00 0x7fa1040b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.819+0000 7fa11affd700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1040afc00 0x7fa1040b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.819+0000 7fa1208c0700 1 -- 172.21.15.27:0/670503074 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fa11400f020 con 0x7fa11c109e90 2024-04-25T13:37:53.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.820+0000 7fa11affd700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1040afc00 0x7fa1040b2110 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa10c007d10 tx=0x7fa10c007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:37:53.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:53.823+0000 7fa1208c0700 1 -- 172.21.15.27:0/670503074 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa11402b020 con 0x7fa11c109e90 2024-04-25T13:37:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.031+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fa108005920 con 0x7fa11c109e90 2024-04-25T13:37:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.031+0000 7fa1208c0700 1 -- 172.21.15.27:0/670503074 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7fa1140a9010 con 0x7fa11c109e90 2024-04-25T13:37:54.032 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:37:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1040afc00 msgr2=0x7fa1040b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1040afc00 0x7fa1040b2110 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa10c007d10 tx=0x7fa10c007480 comp rx=0 tx=0).stop 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 msgr2=0x7fa11c1a1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 0x7fa11c1a1be0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa11400b070 tx=0x7fa114004ce0 comp rx=0 tx=0).stop 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 shutdown_connections 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1040afc00 0x7fa1040b2110 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa11c06c1a0 0x7fa11c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c109e90 0x7fa11c1a1be0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 --2- 172.21.15.27:0/670503074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa11c107320 0x7fa11c19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 >> 172.21.15.27:0/670503074 conn(0x7fa11c0fe740 msgr2=0x7fa11c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 shutdown_connections 2024-04-25T13:37:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:54.034+0000 7fa1218c2700 1 -- 172.21.15.27:0/670503074 wait complete. 2024-04-25T13:37:54.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:53 smithi122 ceph-mon[32843]: purged_snaps scrub starts 2024-04-25T13:37:54.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:53 smithi122 ceph-mon[32843]: purged_snaps scrub ok 2024-04-25T13:37:54.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:53 smithi122 ceph-mon[32843]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:54.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:53 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:54.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:53 smithi122 ceph-mon[32843]: osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] boot 2024-04-25T13:37:54.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:53 smithi122 ceph-mon[32843]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T13:37:54.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:53 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:53 smithi045 ceph-mon[32731]: purged_snaps scrub starts 2024-04-25T13:37:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:53 smithi045 ceph-mon[32731]: purged_snaps scrub ok 2024-04-25T13:37:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:53 smithi045 ceph-mon[32731]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:53 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:53 smithi045 ceph-mon[32731]: osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] boot 2024-04-25T13:37:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:53 smithi045 ceph-mon[32731]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T13:37:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:53 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:54.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:53 smithi027 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T13:37:54.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:53 smithi027 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T13:37:54.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:53 smithi027 ceph-mon[29140]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T13:37:54.339 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:54.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:53 smithi027 ceph-mon[29140]: osd.11 [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] boot 2024-04-25T13:37:54.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:53 smithi027 ceph-mon[29140]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T13:37:54.340 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:53 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T13:37:54.828 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"num_osds":12,"num_up_osds":12,"osd_up_since":1714052273,"num_in_osds":12,"osd_in_since":1714052252,"num_remapped_pgs":0} 2024-04-25T13:37:54.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T13:37:55.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:54 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/670503074' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T13:37:55.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:54 smithi122 ceph-mon[32843]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T13:37:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:37:55.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/670503074' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T13:37:55.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:54 smithi027 ceph-mon[29140]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T13:37:55.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:54 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/670503074' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T13:37:55.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:54 smithi045 ceph-mon[32731]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T13:37:56.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:56.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:55 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:56.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:55 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:37:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:37:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:56.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:37:56.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:37:56.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:55 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 -- 172.21.15.27:0/1554772958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105370 msgr2=0x7fa618105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 --2- 172.21.15.27:0/1554772958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105370 0x7fa618105780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa614009990 tx=0x7fa614009ca0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 -- 172.21.15.27:0/1554772958 shutdown_connections 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 --2- 172.21.15.27:0/1554772958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa61810e7a0 0x7fa618110c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 --2- 172.21.15.27:0/1554772958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105370 0x7fa618105780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 --2- 172.21.15.27:0/1554772958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa618105cc0 0x7fa61810e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 -- 172.21.15.27:0/1554772958 >> 172.21.15.27:0/1554772958 conn(0x7fa6180fec30 msgr2=0x7fa618101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 -- 172.21.15.27:0/1554772958 shutdown_connections 2024-04-25T13:37:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.028+0000 7fa61fdda700 1 -- 172.21.15.27:0/1554772958 wait complete. 2024-04-25T13:37:57.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:56 smithi027 ceph-mon[29140]: Detected new or changed devices on smithi122 2024-04-25T13:37:57.030 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:56 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi122 to 4272M 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61fdda700 1 Processor -- start 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61fdda700 1 -- start start 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61fdda700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa618105370 0x7fa618103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61fdda700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105cc0 0x7fa6181035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61fdda700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa61810e7a0 0x7fa618103ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61fdda700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa6181142e0 con 0x7fa618105cc0 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61fdda700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa618114160 con 0x7fa61810e7a0 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61fdda700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa618114460 con 0x7fa618105370 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61d375700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105cc0 0x7fa6181035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61d375700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105cc0 0x7fa6181035a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36544/0 (socket says 172.21.15.27:36544) 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.029+0000 7fa61d375700 1 -- 172.21.15.27:0/2869216251 learned_addr learned my addr 172.21.15.27:0/2869216251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61db76700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa618105370 0x7fa618103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61e377700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa61810e7a0 0x7fa618103ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61d375700 1 -- 172.21.15.27:0/2869216251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa618105370 msgr2=0x7fa618103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61d375700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa618105370 0x7fa618103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61d375700 1 -- 172.21.15.27:0/2869216251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa61810e7a0 msgr2=0x7fa618103ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61d375700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa61810e7a0 0x7fa618103ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61d375700 1 -- 172.21.15.27:0/2869216251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6181a9050 con 0x7fa618105cc0 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61db76700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa618105370 0x7fa618103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61d375700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105cc0 0x7fa6181035a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa60800d980 tx=0x7fa60800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa60effd700 1 -- 172.21.15.27:0/2869216251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608017070 con 0x7fa618105cc0 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6181a9340 con 0x7fa618105cc0 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa60effd700 1 -- 172.21.15.27:0/2869216251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa608004830 con 0x7fa618105cc0 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa60effd700 1 -- 172.21.15.27:0/2869216251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608005440 con 0x7fa618105cc0 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.030+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6181a9880 con 0x7fa618105cc0 2024-04-25T13:37:57.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.032+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa618051190 con 0x7fa618105cc0 2024-04-25T13:37:57.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.033+0000 7fa60effd700 1 -- 172.21.15.27:0/2869216251 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa6080055e0 con 0x7fa618105cc0 2024-04-25T13:37:57.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.033+0000 7fa60effd700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa6040afc00 0x7fa6040b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:37:57.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.034+0000 7fa61db76700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa6040afc00 0x7fa6040b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:37:57.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.034+0000 7fa60effd700 1 -- 172.21.15.27:0/2869216251 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa6080de6b0 con 0x7fa618105cc0 2024-04-25T13:37:57.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.034+0000 7fa61db76700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa6040afc00 0x7fa6040b2110 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa614003800 tx=0x7fa61401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:37:57.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.038+0000 7fa60effd700 1 -- 172.21.15.27:0/2869216251 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa608020020 con 0x7fa618105cc0 2024-04-25T13:37:57.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.245+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa61802e510 con 0x7fa618105cc0 2024-04-25T13:37:57.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.247+0000 7fa60effd700 1 -- 172.21.15.27:0/2869216251 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18410 (secure 0 0 0) 0x7fa6080a7730 con 0x7fa618105cc0 2024-04-25T13:37:57.247 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:37:57.247 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:37:54.399799+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:37:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa6040afc00 msgr2=0x7fa6040b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa6040afc00 0x7fa6040b2110 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa614003800 tx=0x7fa61401a040 comp rx=0 tx=0).stop 2024-04-25T13:37:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105cc0 msgr2=0x7fa6181035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:37:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105cc0 0x7fa6181035a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa60800d980 tx=0x7fa60800dd40 comp rx=0 tx=0).stop 2024-04-25T13:37:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 shutdown_connections 2024-04-25T13:37:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa6040afc00 0x7fa6040b2110 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa618105370 0x7fa618103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa61810e7a0 0x7fa618103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 --2- 172.21.15.27:0/2869216251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa618105cc0 0x7fa6181035a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:37:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 >> 172.21.15.27:0/2869216251 conn(0x7fa6180fec30 msgr2=0x7fa6180ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:37:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 shutdown_connections 2024-04-25T13:37:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:37:57.249+0000 7fa61fdda700 1 -- 172.21.15.27:0/2869216251 wait complete. 2024-04-25T13:37:57.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:56 smithi122 ceph-mon[32843]: Detected new or changed devices on smithi122 2024-04-25T13:37:57.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:56 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi122 to 4272M 2024-04-25T13:37:57.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:56 smithi045 ceph-mon[32731]: Detected new or changed devices on smithi122 2024-04-25T13:37:57.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:56 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi122 to 4272M 2024-04-25T13:37:58.048 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:37:58.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T13:37:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:37:58.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:57 smithi122 ceph-mon[32843]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:37:58.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:57 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2869216251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:37:58.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:57 smithi045 ceph-mon[32731]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:37:58.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:57 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2869216251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:37:58.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:57 smithi027 ceph-mon[29140]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:37:58.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:57 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2869216251' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 -- 172.21.15.27:0/1758885931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8108630 msgr2=0x7f53a810ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 --2- 172.21.15.27:0/1758885931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8108630 0x7f53a810ff90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f53a00099c0 tx=0x7f53a0009cd0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 -- 172.21.15.27:0/1758885931 shutdown_connections 2024-04-25T13:38:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 --2- 172.21.15.27:0/1758885931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8108630 0x7f53a810ff90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 --2- 172.21.15.27:0/1758885931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53a81033b0 0x7f53a81037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 --2- 172.21.15.27:0/1758885931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a8107ce0 0x7f53a81080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 -- 172.21.15.27:0/1758885931 >> 172.21.15.27:0/1758885931 conn(0x7f53a80fee20 msgr2=0x7f53a8101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:00.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 -- 172.21.15.27:0/1758885931 shutdown_connections 2024-04-25T13:38:00.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.218+0000 7f53ae228700 1 -- 172.21.15.27:0/1758885931 wait complete. 2024-04-25T13:38:00.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.219+0000 7f53ae228700 1 Processor -- start 2024-04-25T13:38:00.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.219+0000 7f53ae228700 1 -- start start 2024-04-25T13:38:00.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53ae228700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a81033b0 0x7f53a80ff300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53ae228700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8107ce0 0x7f53a80ff840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53ae228700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53a8108630 0x7f53a80ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53ae228700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53a8113320 con 0x7f53a8107ce0 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53ae228700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81131a0 con 0x7f53a8108630 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53ae228700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f53a81134a0 con 0x7f53a81033b0 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8107ce0 0x7f53a80ff840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8107ce0 0x7f53a80ff840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36562/0 (socket says 172.21.15.27:36562) 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 -- 172.21.15.27:0/2332930324 learned_addr learned my addr 172.21.15.27:0/2332930324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a7fff700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53a8108630 0x7f53a80ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a77fe700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a81033b0 0x7f53a80ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 -- 172.21.15.27:0/2332930324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a81033b0 msgr2=0x7f53a80ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a81033b0 0x7f53a80ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 -- 172.21.15.27:0/2332930324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53a8108630 msgr2=0x7f53a80ffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:00.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53a8108630 0x7f53a80ffd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 -- 172.21.15.27:0/2332930324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a81a90e0 con 0x7f53a8107ce0 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a6ffd700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8107ce0 0x7f53a80ff840 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f539000b500 tx=0x7f539000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.220+0000 7f53a4ff9700 1 -- 172.21.15.27:0/2332930324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390015020 con 0x7f53a8107ce0 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.221+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a81a93d0 con 0x7f53a8107ce0 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.221+0000 7f53a4ff9700 1 -- 172.21.15.27:0/2332930324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5390011ce0 con 0x7f53a8107ce0 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.221+0000 7f53a4ff9700 1 -- 172.21.15.27:0/2332930324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390010bd0 con 0x7f53a8107ce0 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.221+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a81a9960 con 0x7f53a8107ce0 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.222+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a8051190 con 0x7f53a8107ce0 2024-04-25T13:38:00.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.223+0000 7f53a4ff9700 1 -- 172.21.15.27:0/2332930324 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5390018020 con 0x7f53a8107ce0 2024-04-25T13:38:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.224+0000 7f53a4ff9700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53940afc00 0x7f53940b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.224+0000 7f53a77fe700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53940afc00 0x7f53940b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:00.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.224+0000 7f53a4ff9700 1 -- 172.21.15.27:0/2332930324 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f53900de280 con 0x7f53a8107ce0 2024-04-25T13:38:00.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.225+0000 7f53a77fe700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53940afc00 0x7f53940b2110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f53980060b0 tx=0x7f539800c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:00.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.228+0000 7f53a4ff9700 1 -- 172.21.15.27:0/2332930324 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f539001f020 con 0x7f53a8107ce0 2024-04-25T13:38:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:37:59 smithi122 ceph-mon[32843]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:00.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:37:59 smithi045 ceph-mon[32731]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:00.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:37:59 smithi027 ceph-mon[29140]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:00.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.437+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53a8100830 con 0x7f53a8107ce0 2024-04-25T13:38:00.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.438+0000 7f53a4ff9700 1 -- 172.21.15.27:0/2332930324 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f53900a72e0 con 0x7f53a8107ce0 2024-04-25T13:38:00.438 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:38:00.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53940afc00 msgr2=0x7f53940b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:00.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53940afc00 0x7f53940b2110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f53980060b0 tx=0x7f539800c040 comp rx=0 tx=0).stop 2024-04-25T13:38:00.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8107ce0 msgr2=0x7f53a80ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:00.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8107ce0 0x7f53a80ff840 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f539000b500 tx=0x7f539000b810 comp rx=0 tx=0).stop 2024-04-25T13:38:00.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 shutdown_connections 2024-04-25T13:38:00.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53940afc00 0x7f53940b2110 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a81033b0 0x7f53a80ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53a8108630 0x7f53a80ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 --2- 172.21.15.27:0/2332930324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a8107ce0 0x7f53a80ff840 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:00.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 >> 172.21.15.27:0/2332930324 conn(0x7f53a80fee20 msgr2=0x7f53a8075b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:00.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 shutdown_connections 2024-04-25T13:38:00.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:00.440+0000 7f53ae228700 1 -- 172.21.15.27:0/2332930324 wait complete. 2024-04-25T13:38:01.070 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:00 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2332930324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:01.098 INFO:tasks.cephadm:Adding mds.a on smithi027 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.d on smithi027 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.g on smithi027 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.j on smithi027 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.b on smithi045 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.e on smithi045 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.h on smithi045 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.k on smithi045 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.c on smithi122 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.f on smithi122 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.i on smithi122 2024-04-25T13:38:01.099 INFO:tasks.cephadm:Adding mds.l on smithi122 2024-04-25T13:38:01.099 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l' 2024-04-25T13:38:01.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:00 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2332930324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:01.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:00 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2332930324' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:01.347 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T13:38:02.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:01 smithi122 ceph-mon[32843]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:02.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:01 smithi045 ceph-mon[32731]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:02.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:01 smithi027 ceph-mon[29140]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 -- 172.21.15.122:0/420760955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 msgr2=0x7fa008109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/420760955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 0x7fa008109830 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0009990 tx=0x7f9ff0009ca0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 -- 172.21.15.122:0/420760955 shutdown_connections 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/420760955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008109ec0 0x7fa00810c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/420760955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa00806c1a0 0x7fa008106d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/420760955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 0x7fa008109830 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 -- 172.21.15.122:0/420760955 >> 172.21.15.122:0/420760955 conn(0x7fa0080fe750 msgr2=0x7fa008100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 -- 172.21.15.122:0/420760955 shutdown_connections 2024-04-25T13:38:03.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.042+0000 7fa00d5fd700 1 -- 172.21.15.122:0/420760955 wait complete. 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.043+0000 7fa00d5fd700 1 Processor -- start 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.043+0000 7fa00d5fd700 1 -- start start 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.043+0000 7fa00d5fd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa00806c1a0 0x7fa00819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.043+0000 7fa00d5fd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 0x7fa00819ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.043+0000 7fa00d5fd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008109ec0 0x7fa0081a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.043+0000 7fa00d5fd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa00810fc60 con 0x7fa00806c1a0 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.043+0000 7fa00d5fd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa00810fae0 con 0x7fa008109ec0 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.043+0000 7fa00d5fd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa00810fde0 con 0x7fa008107350 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 0x7fa00819ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 0x7fa00819ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:44218/0 (socket says 172.21.15.122:44218) 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 -- 172.21.15.122:0/3132436119 learned_addr learned my addr 172.21.15.122:0/3132436119 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7fa006ffd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa00806c1a0 0x7fa00819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 -- 172.21.15.122:0/3132436119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008109ec0 msgr2=0x7fa0081a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008109ec0 0x7fa0081a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 -- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa00806c1a0 msgr2=0x7fa00819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:03.045 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa00806c1a0 0x7fa00819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 -- 172.21.15.122:0/3132436119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0081a2290 con 0x7fa008107350 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7fa006ffd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa00806c1a0 0x7fa00819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7f9ffe7fc700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 0x7fa00819ad00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0006010 tx=0x7f9ff0005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7fa004ff9700 1 -- 172.21.15.122:0/3132436119 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0027070 con 0x7fa008107350 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0081a2520 con 0x7fa008107350 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7fa004ff9700 1 -- 172.21.15.122:0/3132436119 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ff0004100 con 0x7fa008107350 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7fa004ff9700 1 -- 172.21.15.122:0/3132436119 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0022de0 con 0x7fa008107350 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.044+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0081a2970 con 0x7fa008107350 2024-04-25T13:38:03.046 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.045+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa008051190 con 0x7fa008107350 2024-04-25T13:38:03.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.047+0000 7fa004ff9700 1 -- 172.21.15.122:0/3132436119 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9ff002a070 con 0x7fa008107350 2024-04-25T13:38:03.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.048+0000 7fa004ff9700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ff40afb80 0x7f9ff40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:03.047 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.048+0000 7fa004ff9700 1 -- 172.21.15.122:0/3132436119 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f9ff00dff00 con 0x7fa008107350 2024-04-25T13:38:03.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.048+0000 7fa006ffd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ff40afb80 0x7f9ff40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:03.048 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.049+0000 7fa006ffd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ff40afb80 0x7f9ff40b2090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8005fd0 tx=0x7f9ff8005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:03.051 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.052+0000 7fa004ff9700 1 -- 172.21.15.122:0/3132436119 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ff0031080 con 0x7fa008107350 2024-04-25T13:38:03.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.259+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l", "target": ["mon-mgr", ""]}) -- 0x7fa008063d90 con 0x7f9ff40afb80 2024-04-25T13:38:03.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.268+0000 7fa004ff9700 1 -- 172.21.15.122:0/3132436119 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fa008063d90 con 0x7f9ff40afb80 2024-04-25T13:38:03.267 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mds.all update... 2024-04-25T13:38:03.269 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ff40afb80 msgr2=0x7f9ff40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ff40afb80 0x7f9ff40b2090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8005fd0 tx=0x7f9ff8005dc0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 msgr2=0x7fa00819ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 0x7fa00819ad00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0006010 tx=0x7f9ff0005e80 comp rx=0 tx=0).stop 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 shutdown_connections 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ff40afb80 0x7f9ff40b2090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa008109ec0 0x7fa0081a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa00806c1a0 0x7fa00819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 --2- 172.21.15.122:0/3132436119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa008107350 0x7fa00819ad00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 >> 172.21.15.122:0/3132436119 conn(0x7fa0080fe750 msgr2=0x7fa0081057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 shutdown_connections 2024-04-25T13:38:03.270 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-25T13:38:03.270+0000 7fa00d5fd700 1 -- 172.21.15.122:0/3132436119 wait complete. 2024-04-25T13:38:03.999 DEBUG:teuthology.orchestra.run.smithi027:mds.a> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T13:38:04.002 DEBUG:teuthology.orchestra.run.smithi027:mds.d> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T13:38:04.045 DEBUG:teuthology.orchestra.run.smithi027:mds.g> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T13:38:04.088 DEBUG:teuthology.orchestra.run.smithi027:mds.j> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T13:38:04.131 DEBUG:teuthology.orchestra.run.smithi045:mds.b> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T13:38:04.133 DEBUG:teuthology.orchestra.run.smithi045:mds.e> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T13:38:04.176 DEBUG:teuthology.orchestra.run.smithi045:mds.h> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T13:38:04.219 DEBUG:teuthology.orchestra.run.smithi045:mds.k> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T13:38:04.262 DEBUG:teuthology.orchestra.run.smithi122:mds.c> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T13:38:04.264 DEBUG:teuthology.orchestra.run.smithi122:mds.f> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T13:38:04.280 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:04.280 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:04.280 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:04.280 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:04.280 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:04.281 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:04.281 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:04.281 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:04.281 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:03 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:04.284 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:04.284 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:04.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:04.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:04.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:04.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:04.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:04.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:04.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:03 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:04.286 DEBUG:teuthology.orchestra.run.smithi122:mds.i> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T13:38:04.328 DEBUG:teuthology.orchestra.run.smithi122:mds.l> sudo journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T13:38:04.371 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T13:38:04.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:04.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:03 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 -- 172.21.15.27:0/3780598341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c1029e0 msgr2=0x7f246c1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 --2- 172.21.15.27:0/3780598341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c1029e0 0x7f246c1036e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f245c009900 tx=0x7f245c02a410 comp rx=0 tx=0).stop 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 -- 172.21.15.27:0/3780598341 shutdown_connections 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 --2- 172.21.15.27:0/3780598341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f246c108920 0x7f246c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 --2- 172.21.15.27:0/3780598341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c103c20 0x7f246c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 --2- 172.21.15.27:0/3780598341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c1029e0 0x7f246c1036e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 -- 172.21.15.27:0/3780598341 >> 172.21.15.27:0/3780598341 conn(0x7f246c0fdbd0 msgr2=0x7f246c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 -- 172.21.15.27:0/3780598341 shutdown_connections 2024-04-25T13:38:04.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.502+0000 7f2471b06640 1 -- 172.21.15.27:0/3780598341 wait complete. 2024-04-25T13:38:04.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.503+0000 7f2471b06640 1 Processor -- start 2024-04-25T13:38:04.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.503+0000 7f2471b06640 1 -- start start 2024-04-25T13:38:04.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.503+0000 7f2471b06640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f246c1029e0 0x7f246c19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.503+0000 7f2471b06640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103c20 0x7f246c19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.503+0000 7f2471b06640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c108920 0x7f246c1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.503+0000 7f2471b06640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f246c10f5d0 con 0x7f246c103c20 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f2471b06640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f246c10f450 con 0x7f246c108920 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f2471b06640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f246c10f750 con 0x7f246c1029e0 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103c20 0x7f246c19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103c20 0x7f246c19a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60648/0 (socket says 172.21.15.27:60648) 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 -- 172.21.15.27:0/2736961749 learned_addr learned my addr 172.21.15.27:0/2736961749 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246bfff640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c108920 0x7f246c1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246b7fe640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f246c1029e0 0x7f246c19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 -- 172.21.15.27:0/2736961749 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f246c1029e0 msgr2=0x7f246c19a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:04.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f246c1029e0 0x7f246c19a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 -- 172.21.15.27:0/2736961749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c108920 msgr2=0x7f246c1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c108920 0x7f246c1a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 -- 172.21.15.27:0/2736961749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246c1a1e70 con 0x7f246c103c20 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f246affd640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103c20 0x7f246c19a870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f245400d950 tx=0x7f245400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f2468ff9640 1 -- 172.21.15.27:0/2736961749 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454017070 con 0x7f246c103c20 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f2468ff9640 1 -- 172.21.15.27:0/2736961749 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2454009c30 con 0x7f246c103c20 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246c1a2160 con 0x7f246c103c20 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f2468ff9640 1 -- 172.21.15.27:0/2736961749 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245400f620 con 0x7f246c103c20 2024-04-25T13:38:04.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.504+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f246c1a26b0 con 0x7f246c103c20 2024-04-25T13:38:04.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.506+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f246c107730 con 0x7f246c103c20 2024-04-25T13:38:04.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.507+0000 7f2468ff9640 1 -- 172.21.15.27:0/2736961749 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2454010460 con 0x7f246c103c20 2024-04-25T13:38:04.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.507+0000 7f2468ff9640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24580afaf0 0x7f24580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:04.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.508+0000 7f246b7fe640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24580afaf0 0x7f24580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:04.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.508+0000 7f2468ff9640 1 -- 172.21.15.27:0/2736961749 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f24540de340 con 0x7f246c103c20 2024-04-25T13:38:04.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.508+0000 7f246b7fe640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24580afaf0 0x7f24580b2000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f245c005ab0 tx=0x7f245c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:04.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.512+0000 7f2468ff9640 1 -- 172.21.15.27:0/2736961749 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2454009de0 con 0x7f246c103c20 2024-04-25T13:38:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.698+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f246c02dd20 con 0x7f246c103c20 2024-04-25T13:38:04.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.699+0000 7f2468ff9640 1 -- 172.21.15.27:0/2736961749 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18410 (secure 0 0 0) 0x7f24540a6d70 con 0x7f246c103c20 2024-04-25T13:38:04.700 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:04.700 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:37:54.399799+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:38:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24580afaf0 msgr2=0x7f24580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24580afaf0 0x7f24580b2000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f245c005ab0 tx=0x7f245c031400 comp rx=0 tx=0).stop 2024-04-25T13:38:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103c20 msgr2=0x7f246c19a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103c20 0x7f246c19a870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f245400d950 tx=0x7f245400de10 comp rx=0 tx=0).stop 2024-04-25T13:38:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 shutdown_connections 2024-04-25T13:38:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24580afaf0 0x7f24580b2000 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c108920 0x7f246c1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f246c103c20 0x7f246c19a870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 --2- 172.21.15.27:0/2736961749 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f246c1029e0 0x7f246c19a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.701+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 >> 172.21.15.27:0/2736961749 conn(0x7f246c0fdbd0 msgr2=0x7f246c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.702+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 shutdown_connections 2024-04-25T13:38:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.702+0000 7f2471b06640 1 -- 172.21.15.27:0/2736961749 wait complete. 2024-04-25T13:38:04.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:38:04.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:38:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 -- 172.21.15.27:0/472853005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960109890 msgr2=0x7f596010bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 --2- 172.21.15.27:0/472853005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960109890 0x7f596010bcb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5950009ca0 tx=0x7f595002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 -- 172.21.15.27:0/472853005 shutdown_connections 2024-04-25T13:38:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 --2- 172.21.15.27:0/472853005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960109890 0x7f596010bcb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 --2- 172.21.15.27:0/472853005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5960106f30 0x7f5960109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 --2- 172.21.15.27:0/472853005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59601045d0 0x7f59601069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 -- 172.21.15.27:0/472853005 >> 172.21.15.27:0/472853005 conn(0x7f59600fdbd0 msgr2=0x7f5960100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 -- 172.21.15.27:0/472853005 shutdown_connections 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 -- 172.21.15.27:0/472853005 wait complete. 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 Processor -- start 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.869+0000 7f59685b2640 1 -- start start 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f59685b2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59601045d0 0x7f596019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f59685b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960106f30 0x7f596019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f59685b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5960109890 0x7f59601a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f59685b2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f596010f610 con 0x7f5960106f30 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f59685b2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f596010f490 con 0x7f5960109890 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f59685b2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f596010f790 con 0x7f59601045d0 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960106f30 0x7f596019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f596634e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59601045d0 0x7f596019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960106f30 0x7f596019aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60668/0 (socket says 172.21.15.27:60668) 2024-04-25T13:38:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 -- 172.21.15.27:0/3687554377 learned_addr learned my addr 172.21.15.27:0/3687554377 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f596634e640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59601045d0 0x7f596019a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42116/0 (socket says 172.21.15.27:42116) 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5966b4f640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5960109890 0x7f59601a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 -- 172.21.15.27:0/3687554377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59601045d0 msgr2=0x7f596019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59601045d0 0x7f596019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 -- 172.21.15.27:0/3687554377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5960109890 msgr2=0x7f59601a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5960109890 0x7f59601a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 -- 172.21.15.27:0/3687554377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59601a2080 con 0x7f5960106f30 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5966b4f640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5960109890 0x7f59601a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f596634e640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59601045d0 0x7f596019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f5965b4d640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960106f30 0x7f596019aa80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f595400e9f0 tx=0x7f595400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.870+0000 7f595f7fe640 1 -- 172.21.15.27:0/3687554377 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595400cd80 con 0x7f5960106f30 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.871+0000 7f595f7fe640 1 -- 172.21.15.27:0/3687554377 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59540054d0 con 0x7f5960106f30 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.871+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59601a2370 con 0x7f5960106f30 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.871+0000 7f595f7fe640 1 -- 172.21.15.27:0/3687554377 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954010690 con 0x7f5960106f30 2024-04-25T13:38:04.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.871+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59601a2870 con 0x7f5960106f30 2024-04-25T13:38:04.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.872+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f596006c2a0 con 0x7f5960106f30 2024-04-25T13:38:04.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.873+0000 7f595f7fe640 1 -- 172.21.15.27:0/3687554377 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5954004e30 con 0x7f5960106f30 2024-04-25T13:38:04.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.874+0000 7f595f7fe640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f59480afaf0 0x7f59480b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:04.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.874+0000 7f596634e640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f59480afaf0 0x7f59480b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.874+0000 7f595f7fe640 1 -- 172.21.15.27:0/3687554377 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f59540dee20 con 0x7f5960106f30 2024-04-25T13:38:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.874+0000 7f596634e640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f59480afaf0 0x7f59480b2000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f596006a8b0 tx=0x7f595800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:04.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:04.877+0000 7f595f7fe640 1 -- 172.21.15.27:0/3687554377 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5954053ff0 con 0x7f5960106f30 2024-04-25T13:38:05.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.067+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59601a2b30 con 0x7f5960106f30 2024-04-25T13:38:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.068+0000 7f595f7fe640 1 -- 172.21.15.27:0/3687554377 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f59540a7ea0 con 0x7f5960106f30 2024-04-25T13:38:05.068 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:38:05.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f59480afaf0 msgr2=0x7f59480b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:05.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f59480afaf0 0x7f59480b2000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f596006a8b0 tx=0x7f595800b390 comp rx=0 tx=0).stop 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960106f30 msgr2=0x7f596019aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960106f30 0x7f596019aa80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f595400e9f0 tx=0x7f595400edf0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 shutdown_connections 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f59480afaf0 0x7f59480b2000 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5960109890 0x7f59601a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5960106f30 0x7f596019aa80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 --2- 172.21.15.27:0/3687554377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59601045d0 0x7f596019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 >> 172.21.15.27:0/3687554377 conn(0x7f59600fdbd0 msgr2=0x7f5960105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 shutdown_connections 2024-04-25T13:38:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.070+0000 7f59685b2640 1 -- 172.21.15.27:0/3687554377 wait complete. 2024-04-25T13:38:05.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 -- 172.21.15.27:0/3511321539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 msgr2=0x7fa458104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/3511321539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 0x7fa458104030 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa440009900 tx=0x7fa44002a410 comp rx=0 tx=0).stop 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 -- 172.21.15.27:0/3511321539 shutdown_connections 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/3511321539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa458108920 0x7fa45810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/3511321539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 0x7fa458104030 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/3511321539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4581029e0 0x7fa4581036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 -- 172.21.15.27:0/3511321539 >> 172.21.15.27:0/3511321539 conn(0x7fa4580fdbd0 msgr2=0x7fa458100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 -- 172.21.15.27:0/3511321539 shutdown_connections 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.193+0000 7fa45e0b6640 1 -- 172.21.15.27:0/3511321539 wait complete. 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa45e0b6640 1 Processor -- start 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa45e0b6640 1 -- start start 2024-04-25T13:38:05.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa45e0b6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4581029e0 0x7fa4580ffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa45e0b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 0x7fa4580fe360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa45e0b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa458108920 0x7fa4580fe8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa45e0b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa45810e500 con 0x7fa458103c20 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa45e0b6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa45810e380 con 0x7fa458108920 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa45e0b6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa45810e680 con 0x7fa4581029e0 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa456ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 0x7fa4580fe360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa456ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 0x7fa4580fe360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60676/0 (socket says 172.21.15.27:60676) 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.194+0000 7fa456ffd640 1 -- 172.21.15.27:0/537733578 learned_addr learned my addr 172.21.15.27:0/537733578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa4577fe640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4581029e0 0x7fa4580ffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa457fff640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa458108920 0x7fa4580fe8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa456ffd640 1 -- 172.21.15.27:0/537733578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4581029e0 msgr2=0x7fa4580ffcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa456ffd640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4581029e0 0x7fa4580ffcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa456ffd640 1 -- 172.21.15.27:0/537733578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa458108920 msgr2=0x7fa4580fe8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa456ffd640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa458108920 0x7fa4580fe8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa456ffd640 1 -- 172.21.15.27:0/537733578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4580fefd0 con 0x7fa458103c20 2024-04-25T13:38:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa457fff640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa458108920 0x7fa4580fe8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa456ffd640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 0x7fa4580fe360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa440006010 tx=0x7fa440004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa454ff9640 1 -- 172.21.15.27:0/537733578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44003f030 con 0x7fa458103c20 2024-04-25T13:38:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa454ff9640 1 -- 172.21.15.27:0/537733578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa440003cb0 con 0x7fa458103c20 2024-04-25T13:38:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4581a4370 con 0x7fa458103c20 2024-04-25T13:38:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa454ff9640 1 -- 172.21.15.27:0/537733578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440035070 con 0x7fa458103c20 2024-04-25T13:38:05.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.195+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4581a4770 con 0x7fa458103c20 2024-04-25T13:38:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.196+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa45806a190 con 0x7fa458103c20 2024-04-25T13:38:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.198+0000 7fa454ff9640 1 -- 172.21.15.27:0/537733578 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa440042030 con 0x7fa458103c20 2024-04-25T13:38:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.198+0000 7fa454ff9640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4440afaf0 0x7fa4440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.198+0000 7fa4577fe640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4440afaf0 0x7fa4440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.198+0000 7fa454ff9640 1 -- 172.21.15.27:0/537733578 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa4400ff480 con 0x7fa458103c20 2024-04-25T13:38:05.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.199+0000 7fa4577fe640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4440afaf0 0x7fa4440b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa448005fd0 tx=0x7fa448005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:05.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.202+0000 7fa454ff9640 1 -- 172.21.15.27:0/537733578 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa4400c8500 con 0x7fa458103c20 2024-04-25T13:38:05.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:05 smithi122 ceph-mon[32843]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:05.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:05 smithi122 ceph-mon[32843]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l;count:12 2024-04-25T13:38:05.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:05 smithi122 ceph-mon[32843]: Deploying daemon mds.l on smithi122 2024-04-25T13:38:05.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:05 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2736961749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:05.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:05 smithi045 ceph-mon[32731]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:05.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:05 smithi045 ceph-mon[32731]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l;count:12 2024-04-25T13:38:05.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:05 smithi045 ceph-mon[32731]: Deploying daemon mds.l on smithi122 2024-04-25T13:38:05.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:05 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2736961749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:05.399 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:05 smithi027 ceph-mon[29140]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:05.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:05 smithi027 ceph-mon[29140]: Saving service mds.all spec with placement smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l;count:12 2024-04-25T13:38:05.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:05 smithi027 ceph-mon[29140]: Deploying daemon mds.l on smithi122 2024-04-25T13:38:05.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:05 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2736961749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:05.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.427+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4581a49d0 con 0x7fa458103c20 2024-04-25T13:38:05.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.427+0000 7fa454ff9640 1 -- 172.21.15.27:0/537733578 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fa4400c7e00 con 0x7fa458103c20 2024-04-25T13:38:05.427 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:05.428 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T13:38:05.428 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 1 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4440afaf0 msgr2=0x7fa4440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4440afaf0 0x7fa4440b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa448005fd0 tx=0x7fa448005850 comp rx=0 tx=0).stop 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 msgr2=0x7fa4580fe360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 0x7fa4580fe360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa440006010 tx=0x7fa440004340 comp rx=0 tx=0).stop 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 shutdown_connections 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4440afaf0 0x7fa4440b2000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa458108920 0x7fa4580fe8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa458103c20 0x7fa4580fe360 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 --2- 172.21.15.27:0/537733578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4581029e0 0x7fa4580ffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 >> 172.21.15.27:0/537733578 conn(0x7fa4580fdbd0 msgr2=0x7fa458074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 shutdown_connections 2024-04-25T13:38:05.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:05.429+0000 7fa45e0b6640 1 -- 172.21.15.27:0/537733578 wait complete. 2024-04-25T13:38:06.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:06 smithi122 ceph-mon[32843]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:06 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3687554377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:06.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:06 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/537733578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:06.309 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 systemd[1]: Starting Ceph mds.l for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:06.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:06 smithi045 ceph-mon[32731]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:06.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:06 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3687554377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:06.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:06 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/537733578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:06 smithi027 ceph-mon[29140]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:06 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3687554377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:06.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:06 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/537733578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:06.676 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 podman[63344]: 2024-04-25 13:38:06.300117795 +0000 UTC m=+0.017808727 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:06.676 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 podman[63344]: 2024-04-25 13:38:06.41446146 +0000 UTC m=+0.132152364 container create d2d867b1f50e9a8eab65322b7ed510fc02ebd6aa429a638a261f4a7ec0823f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T13:38:06.676 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 podman[63344]: 2024-04-25 13:38:06.676908743 +0000 UTC m=+0.394599648 container init d2d867b1f50e9a8eab65322b7ed510fc02ebd6aa429a638a261f4a7ec0823f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T13:38:07.058 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 podman[63344]: 2024-04-25 13:38:06.679509768 +0000 UTC m=+0.397200676 container start d2d867b1f50e9a8eab65322b7ed510fc02ebd6aa429a638a261f4a7ec0823f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:38:07.058 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 ceph-mds[63363]: -- 172.21.15.122:0/4194322746 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c33bef3a40 con 0x55c33bf3b180 2024-04-25T13:38:07.059 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l[63359]: starting mds.l at 2024-04-25T13:38:07.059 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 bash[63344]: d2d867b1f50e9a8eab65322b7ed510fc02ebd6aa429a638a261f4a7ec0823f4f 2024-04-25T13:38:07.059 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:38:06 smithi122 systemd[1]: Started Ceph mds.l for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:07.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:07.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 -- 172.21.15.27:0/2360733264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 msgr2=0x7ff15010ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2360733264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 0x7ff15010ad20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff148009ca0 tx=0x7ff14802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 -- 172.21.15.27:0/2360733264 shutdown_connections 2024-04-25T13:38:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2360733264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 0x7ff15010ad20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2360733264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150103c00 0x7ff150104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2360733264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1501029c0 0x7ff1501036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 -- 172.21.15.27:0/2360733264 >> 172.21.15.27:0/2360733264 conn(0x7ff1500fdbd0 msgr2=0x7ff150100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 -- 172.21.15.27:0/2360733264 shutdown_connections 2024-04-25T13:38:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.569+0000 7ff1582e7640 1 -- 172.21.15.27:0/2360733264 wait complete. 2024-04-25T13:38:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff1582e7640 1 Processor -- start 2024-04-25T13:38:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff1582e7640 1 -- start start 2024-04-25T13:38:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff1582e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1501029c0 0x7ff15006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff1582e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150103c00 0x7ff15019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff1582e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 0x7ff1501a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff1582e7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010e4f0 con 0x7ff150108900 2024-04-25T13:38:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff1582e7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010e370 con 0x7ff150103c00 2024-04-25T13:38:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff1582e7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010e670 con 0x7ff1501029c0 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156884640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 0x7ff1501a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156884640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 0x7ff1501a14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60696/0 (socket says 172.21.15.27:60696) 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156884640 1 -- 172.21.15.27:0/2683482449 learned_addr learned my addr 172.21.15.27:0/2683482449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156083640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1501029c0 0x7ff15006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff155882640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150103c00 0x7ff15019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156884640 1 -- 172.21.15.27:0/2683482449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1501029c0 msgr2=0x7ff15006a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156884640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1501029c0 0x7ff15006a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156884640 1 -- 172.21.15.27:0/2683482449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150103c00 msgr2=0x7ff15019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156884640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150103c00 0x7ff15019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.570+0000 7ff156884640 1 -- 172.21.15.27:0/2683482449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1501a1ce0 con 0x7ff150108900 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.571+0000 7ff156884640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 0x7ff1501a14c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff1480097b0 tx=0x7ff148031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.571+0000 7ff1437fe640 1 -- 172.21.15.27:0/2683482449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148003bf0 con 0x7ff150108900 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.571+0000 7ff1437fe640 1 -- 172.21.15.27:0/2683482449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff148003d90 con 0x7ff150108900 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.571+0000 7ff1437fe640 1 -- 172.21.15.27:0/2683482449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14802fc30 con 0x7ff150108900 2024-04-25T13:38:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.571+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1501a1f70 con 0x7ff150108900 2024-04-25T13:38:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.571+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1501a23d0 con 0x7ff150108900 2024-04-25T13:38:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.572+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15006afa0 con 0x7ff150108900 2024-04-25T13:38:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.573+0000 7ff1437fe640 1 -- 172.21.15.27:0/2683482449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff148004040 con 0x7ff150108900 2024-04-25T13:38:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.574+0000 7ff1437fe640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff13c0afaf0 0x7ff13c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.574+0000 7ff156083640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff13c0afaf0 0x7ff13c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.574+0000 7ff1437fe640 1 -- 172.21.15.27:0/2683482449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff1480fff90 con 0x7ff150108900 2024-04-25T13:38:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.575+0000 7ff156083640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff13c0afaf0 0x7ff13c0b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff144005fd0 tx=0x7ff14400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:07.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.579+0000 7ff1437fe640 1 -- 172.21.15.27:0/2683482449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1480c9010 con 0x7ff150108900 2024-04-25T13:38:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.804+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff15002dc10 con 0x7ff150108900 2024-04-25T13:38:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.804+0000 7ff1437fe640 1 -- 172.21.15.27:0/2683482449 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7ff1480c8910 con 0x7ff150108900 2024-04-25T13:38:07.805 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:07.805 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T13:38:07.805 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 2 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff13c0afaf0 msgr2=0x7ff13c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff13c0afaf0 0x7ff13c0b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff144005fd0 tx=0x7ff14400b390 comp rx=0 tx=0).stop 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 msgr2=0x7ff1501a14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 0x7ff1501a14c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff1480097b0 tx=0x7ff148031840 comp rx=0 tx=0).stop 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 shutdown_connections 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff13c0afaf0 0x7ff13c0b2000 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff150108900 0x7ff1501a14c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff150103c00 0x7ff15019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 --2- 172.21.15.27:0/2683482449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1501029c0 0x7ff15006a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 >> 172.21.15.27:0/2683482449 conn(0x7ff1500fdbd0 msgr2=0x7ff1501048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 shutdown_connections 2024-04-25T13:38:07.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:07.806+0000 7ff1582e7640 1 -- 172.21.15.27:0/2683482449 wait complete. 2024-04-25T13:38:07.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:07.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.122:6832/3322601046,v1:172.21.15.122:6833/3322601046] up:boot 2024-04-25T13:38:07.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: fsmap 1 up:standby 2024-04-25T13:38:07.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T13:38:07.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:07.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:07.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:07.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:07.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:07.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:07.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:07 smithi045 ceph-mon[32731]: Deploying daemon mds.k on smithi045 2024-04-25T13:38:08.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:08.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.122:6832/3322601046,v1:172.21.15.122:6833/3322601046] up:boot 2024-04-25T13:38:08.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: fsmap 1 up:standby 2024-04-25T13:38:08.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T13:38:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:08.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:07 smithi122 ceph-mon[32843]: Deploying daemon mds.k on smithi045 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.122:6832/3322601046,v1:172.21.15.122:6833/3322601046] up:boot 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: fsmap 1 up:standby 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:08.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:08.152 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:07 smithi027 ceph-mon[29140]: Deploying daemon mds.k on smithi045 2024-04-25T13:38:08.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:08 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2683482449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:09.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:08 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2683482449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:09.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:08 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2683482449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:09.328 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 systemd[1]: Starting Ceph mds.k for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:09.676 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 podman[62641]: 2024-04-25 13:38:09.31996656 +0000 UTC m=+0.018512772 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:09.677 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 podman[62641]: 2024-04-25 13:38:09.424167261 +0000 UTC m=+0.122713460 container create 4235b5a6ead6b3f98817da1e7dd606c8c366e064ab08b4b3a2d758926176b2a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T13:38:09.677 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 podman[62641]: 2024-04-25 13:38:09.677214939 +0000 UTC m=+0.375761139 container init 4235b5a6ead6b3f98817da1e7dd606c8c366e064ab08b4b3a2d758926176b2a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:38:09.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.949+0000 7f674c1ff640 1 -- 172.21.15.27:0/1734202074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6744106f10 msgr2=0x7f6744109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.949+0000 7f674c1ff640 1 --2- 172.21.15.27:0/1734202074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6744106f10 0x7f6744109330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6734009900 tx=0x7f673402a410 comp rx=0 tx=0).stop 2024-04-25T13:38:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.949+0000 7f674c1ff640 1 -- 172.21.15.27:0/1734202074 shutdown_connections 2024-04-25T13:38:09.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.949+0000 7f674c1ff640 1 --2- 172.21.15.27:0/1734202074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6744109870 0x7f674410bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.949+0000 7f674c1ff640 1 --2- 172.21.15.27:0/1734202074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6744106f10 0x7f6744109330 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.949+0000 7f674c1ff640 1 --2- 172.21.15.27:0/1734202074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f67441045b0 0x7f67441069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.949+0000 7f674c1ff640 1 -- 172.21.15.27:0/1734202074 >> 172.21.15.27:0/1734202074 conn(0x7f67440fdbd0 msgr2=0x7f6744100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.950+0000 7f674c1ff640 1 -- 172.21.15.27:0/1734202074 shutdown_connections 2024-04-25T13:38:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.950+0000 7f674c1ff640 1 -- 172.21.15.27:0/1734202074 wait complete. 2024-04-25T13:38:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.950+0000 7f674c1ff640 1 Processor -- start 2024-04-25T13:38:09.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.950+0000 7f674c1ff640 1 -- start start 2024-04-25T13:38:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.950+0000 7f674c1ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67441045b0 0x7f6744103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.950+0000 7f674c1ff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6744106f10 0x7f67441020a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.950+0000 7f674c1ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6744109870 0x7f67441025e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f674c1ff640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f674410f5d0 con 0x7f67441045b0 2024-04-25T13:38:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f674c1ff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f674410f450 con 0x7f6744109870 2024-04-25T13:38:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f674c1ff640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f674410f750 con 0x7f6744106f10 2024-04-25T13:38:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67441045b0 0x7f6744103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:09.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67441045b0 0x7f6744103a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60712/0 (socket says 172.21.15.27:60712) 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 -- 172.21.15.27:0/639070409 learned_addr learned my addr 172.21.15.27:0/639070409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f674979a640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6744106f10 0x7f67441020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f674a79c640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6744109870 0x7f67441025e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 -- 172.21.15.27:0/639070409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6744106f10 msgr2=0x7f67441020a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6744106f10 0x7f67441020a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 -- 172.21.15.27:0/639070409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6744109870 msgr2=0x7f67441025e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6744109870 0x7f67441025e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 -- 172.21.15.27:0/639070409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6744102d10 con 0x7f67441045b0 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f6749f9b640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67441045b0 0x7f6744103a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f674000d770 tx=0x7f674000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.951+0000 7f673affd640 1 -- 172.21.15.27:0/639070409 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6740017070 con 0x7f67441045b0 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.952+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67441a42d0 con 0x7f67441045b0 2024-04-25T13:38:09.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.952+0000 7f673affd640 1 -- 172.21.15.27:0/639070409 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6740009c30 con 0x7f67441045b0 2024-04-25T13:38:09.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.952+0000 7f673affd640 1 -- 172.21.15.27:0/639070409 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674000f620 con 0x7f67441045b0 2024-04-25T13:38:09.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.952+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67441a47c0 con 0x7f67441045b0 2024-04-25T13:38:09.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.953+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67440ff6f0 con 0x7f67441045b0 2024-04-25T13:38:09.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.954+0000 7f673affd640 1 -- 172.21.15.27:0/639070409 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f674001a070 con 0x7f67441045b0 2024-04-25T13:38:09.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.955+0000 7f673affd640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67300afaf0 0x7f67300b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:09.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.955+0000 7f674979a640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67300afaf0 0x7f67300b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:09.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.955+0000 7f673affd640 1 -- 172.21.15.27:0/639070409 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f67400de1d0 con 0x7f67441045b0 2024-04-25T13:38:09.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.956+0000 7f674979a640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67300afaf0 0x7f67300b2000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6734009230 tx=0x7f6734031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:09.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:09.958+0000 7f673affd640 1 -- 172.21.15.27:0/639070409 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6740009de0 con 0x7f67441045b0 2024-04-25T13:38:09.982 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:09 smithi122 ceph-mon[32843]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:10.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:09 smithi045 ceph-mon[32731]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:10.078 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 podman[62641]: 2024-04-25 13:38:09.680256624 +0000 UTC m=+0.378802824 container start 4235b5a6ead6b3f98817da1e7dd606c8c366e064ab08b4b3a2d758926176b2a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T13:38:10.078 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 ceph-mds[62660]: -- 172.21.15.45:0/1274198618 <== mon.2 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556aef691a40 con 0x556aef6d9180 2024-04-25T13:38:10.078 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k[62656]: starting mds.k at 2024-04-25T13:38:10.078 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 bash[62641]: 4235b5a6ead6b3f98817da1e7dd606c8c366e064ab08b4b3a2d758926176b2a8 2024-04-25T13:38:10.078 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:38:09 smithi045 systemd[1]: Started Ceph mds.k for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:10.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:09 smithi027 ceph-mon[29140]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.186+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f674402db50 con 0x7f67441045b0 2024-04-25T13:38:10.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.186+0000 7f673affd640 1 -- 172.21.15.27:0/639070409 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f67400a6c00 con 0x7f67441045b0 2024-04-25T13:38:10.187 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:10.187 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T13:38:10.187 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-04-25T13:38:10.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67300afaf0 msgr2=0x7f67300b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67300afaf0 0x7f67300b2000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6734009230 tx=0x7f6734031400 comp rx=0 tx=0).stop 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67441045b0 msgr2=0x7f6744103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67441045b0 0x7f6744103a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f674000d770 tx=0x7f674000dc30 comp rx=0 tx=0).stop 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 shutdown_connections 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67300afaf0 0x7f67300b2000 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6744109870 0x7f67441025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6744106f10 0x7f67441020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 --2- 172.21.15.27:0/639070409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67441045b0 0x7f6744103a00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 >> 172.21.15.27:0/639070409 conn(0x7f67440fdbd0 msgr2=0x7f67441051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 shutdown_connections 2024-04-25T13:38:10.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:10.189+0000 7f674c1ff640 1 -- 172.21.15.27:0/639070409 wait complete. 2024-04-25T13:38:10.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.45:6832/3792078836,v1:172.21.15.45:6833/3792078836] up:boot 2024-04-25T13:38:10.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: fsmap 2 up:standby 2024-04-25T13:38:10.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T13:38:10.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:10.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:10.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:10.801 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:10.802 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:10.802 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:10.802 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: Deploying daemon mds.j on smithi027 2024-04-25T13:38:10.802 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:10 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/639070409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:11.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.45:6832/3792078836,v1:172.21.15.45:6833/3792078836] up:boot 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: fsmap 2 up:standby 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: Deploying daemon mds.j on smithi027 2024-04-25T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:10 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/639070409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.45:6832/3792078836,v1:172.21.15.45:6833/3792078836] up:boot 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: fsmap 2 up:standby 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: Deploying daemon mds.j on smithi027 2024-04-25T13:38:11.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:10 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/639070409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:12.122 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:11 smithi045 ceph-mon[32731]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:12.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:11 smithi027 ceph-mon[29140]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:12.151 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 systemd[1]: Starting Ceph mds.j for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:12.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:12.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:11 smithi122 ceph-mon[32843]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:12.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 -- 172.21.15.27:0/2209663149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b01029e0 msgr2=0x7f79b01036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:12.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2209663149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b01029e0 0x7f79b01036e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009ca0 tx=0x7f79a002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 -- 172.21.15.27:0/2209663149 shutdown_connections 2024-04-25T13:38:12.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2209663149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79b0108920 0x7f79b010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2209663149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79b0103c20 0x7f79b0104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2209663149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b01029e0 0x7f79b01036e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 -- 172.21.15.27:0/2209663149 >> 172.21.15.27:0/2209663149 conn(0x7f79b00fdbd0 msgr2=0x7f79b0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:12.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 -- 172.21.15.27:0/2209663149 shutdown_connections 2024-04-25T13:38:12.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.327+0000 7f79b4a67640 1 -- 172.21.15.27:0/2209663149 wait complete. 2024-04-25T13:38:12.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79b4a67640 1 Processor -- start 2024-04-25T13:38:12.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79b4a67640 1 -- start start 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79b4a67640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79b01029e0 0x7f79b0199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79b4a67640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b0103c20 0x7f79b01996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79b4a67640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79b0108920 0x7f79b01a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79b4a67640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f79b010e4c0 con 0x7f79b0103c20 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79b4a67640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79b010e340 con 0x7f79b01029e0 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79b4a67640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f79b010e640 con 0x7f79b0108920 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79add9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b0103c20 0x7f79b01996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79add9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b0103c20 0x7f79b01996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60732/0 (socket says 172.21.15.27:60732) 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79add9b640 1 -- 172.21.15.27:0/2285270942 learned_addr learned my addr 172.21.15.27:0/2285270942 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79ae59c640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79b01029e0 0x7f79b0199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.328+0000 7f79aed9d640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79b0108920 0x7f79b01a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79add9b640 1 -- 172.21.15.27:0/2285270942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79b0108920 msgr2=0x7f79b01a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79add9b640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79b0108920 0x7f79b01a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79add9b640 1 -- 172.21.15.27:0/2285270942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79b01029e0 msgr2=0x7f79b0199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79add9b640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79b01029e0 0x7f79b0199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79add9b640 1 -- 172.21.15.27:0/2285270942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b01a0cc0 con 0x7f79b0103c20 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79add9b640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b0103c20 0x7f79b01996c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f799800c890 tx=0x7f799800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79a77fe640 1 -- 172.21.15.27:0/2285270942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998004220 con 0x7f79b0103c20 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79a77fe640 1 -- 172.21.15.27:0/2285270942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79980043c0 con 0x7f79b0103c20 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b0074e30 con 0x7f79b0103c20 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79a77fe640 1 -- 172.21.15.27:0/2285270942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998004a00 con 0x7f79b0103c20 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.329+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b0075380 con 0x7f79b0103c20 2024-04-25T13:38:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.330+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b006a190 con 0x7f79b0103c20 2024-04-25T13:38:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.331+0000 7f79a77fe640 1 -- 172.21.15.27:0/2285270942 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f799801e020 con 0x7f79b0103c20 2024-04-25T13:38:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.332+0000 7f79a77fe640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f799c0afaf0 0x7f799c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.332+0000 7f79ae59c640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f799c0afaf0 0x7f799c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.332+0000 7f79a77fe640 1 -- 172.21.15.27:0/2285270942 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f79980e0850 con 0x7f79b0103c20 2024-04-25T13:38:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.333+0000 7f79ae59c640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f799c0afaf0 0x7f799c0b2000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f79a0000f80 tx=0x7f79a0009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.336+0000 7f79a77fe640 1 -- 172.21.15.27:0/2285270942 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7998025070 con 0x7f79b0103c20 2024-04-25T13:38:12.495 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 podman[67081]: 2024-04-25 13:38:12.266570127 +0000 UTC m=+0.018649610 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:12.495 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 podman[67081]: 2024-04-25 13:38:12.391547425 +0000 UTC m=+0.143626900 container create 54604d426a1e208e050f1642c75655b09d1b899c2606ac938054d1ecf46f0df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T13:38:12.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.570+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79b01071a0 con 0x7f79b0103c20 2024-04-25T13:38:12.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.570+0000 7f79a77fe640 1 -- 172.21.15.27:0/2285270942 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f79980a98d0 con 0x7f79b0103c20 2024-04-25T13:38:12.571 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:12.571 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T13:38:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-04-25T13:38:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f799c0afaf0 msgr2=0x7f799c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f799c0afaf0 0x7f799c0b2000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f79a0000f80 tx=0x7f79a0009360 comp rx=0 tx=0).stop 2024-04-25T13:38:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b0103c20 msgr2=0x7f79b01996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b0103c20 0x7f79b01996c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f799800c890 tx=0x7f799800cd50 comp rx=0 tx=0).stop 2024-04-25T13:38:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 shutdown_connections 2024-04-25T13:38:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f799c0afaf0 0x7f799c0b2000 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79b0108920 0x7f79b01a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79b0103c20 0x7f79b01996c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 --2- 172.21.15.27:0/2285270942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79b01029e0 0x7f79b0199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 >> 172.21.15.27:0/2285270942 conn(0x7f79b00fdbd0 msgr2=0x7f79b0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 shutdown_connections 2024-04-25T13:38:12.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:12.573+0000 7f79b4a67640 1 -- 172.21.15.27:0/2285270942 wait complete. 2024-04-25T13:38:12.897 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2285270942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:12.897 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:12.898 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 podman[67081]: 2024-04-25 13:38:12.627560343 +0000 UTC m=+0.379639808 container init 54604d426a1e208e050f1642c75655b09d1b899c2606ac938054d1ecf46f0df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T13:38:12.898 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 podman[67081]: 2024-04-25 13:38:12.63076291 +0000 UTC m=+0.382842386 container start 54604d426a1e208e050f1642c75655b09d1b899c2606ac938054d1ecf46f0df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T13:38:12.898 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-mds[67121]: -- 172.21.15.27:0/3959825059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56401a431a40 con 0x56401a479180 2024-04-25T13:38:12.898 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j[67115]: starting mds.j at 2024-04-25T13:38:12.898 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 bash[67081]: 54604d426a1e208e050f1642c75655b09d1b899c2606ac938054d1ecf46f0df4 2024-04-25T13:38:12.898 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:38:12 smithi027 systemd[1]: Started Ceph mds.j for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:13.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:13.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:13.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:13.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:13.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:13.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:12 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2285270942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:13.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:13.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:13.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:13.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:13.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:13.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:13.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:12 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2285270942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:13.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:13.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:13.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:13.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:13.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:13.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:13 smithi027 ceph-mon[29140]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:13 smithi027 ceph-mon[29140]: Deploying daemon mds.d on smithi027 2024-04-25T13:38:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:13 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.27:6834/3166244728,v1:172.21.15.27:6835/3166244728] up:boot 2024-04-25T13:38:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:13 smithi027 ceph-mon[29140]: fsmap 3 up:standby 2024-04-25T13:38:14.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T13:38:14.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:13 smithi122 ceph-mon[32843]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:14.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:13 smithi122 ceph-mon[32843]: Deploying daemon mds.d on smithi027 2024-04-25T13:38:14.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:13 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.27:6834/3166244728,v1:172.21.15.27:6835/3166244728] up:boot 2024-04-25T13:38:14.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:13 smithi122 ceph-mon[32843]: fsmap 3 up:standby 2024-04-25T13:38:14.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T13:38:14.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:13 smithi045 ceph-mon[32731]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:14.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:13 smithi045 ceph-mon[32731]: Deploying daemon mds.d on smithi027 2024-04-25T13:38:14.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:13 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.27:6834/3166244728,v1:172.21.15.27:6835/3166244728] up:boot 2024-04-25T13:38:14.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:13 smithi045 ceph-mon[32731]: fsmap 3 up:standby 2024-04-25T13:38:14.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T13:38:14.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:14.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 -- 172.21.15.27:0/1441342599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 msgr2=0x7fc164103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:14.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 --2- 172.21.15.27:0/1441342599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 0x7fc164103540 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc154009900 tx=0x7fc15402a410 comp rx=0 tx=0).stop 2024-04-25T13:38:14.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 -- 172.21.15.27:0/1441342599 shutdown_connections 2024-04-25T13:38:14.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 --2- 172.21.15.27:0/1441342599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc164108780 0x7fc16410aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 --2- 172.21.15.27:0/1441342599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc164103a80 0x7fc164103e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 --2- 172.21.15.27:0/1441342599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 0x7fc164103540 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 -- 172.21.15.27:0/1441342599 >> 172.21.15.27:0/1441342599 conn(0x7fc1640fda30 msgr2=0x7fc1640ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:14.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 -- 172.21.15.27:0/1441342599 shutdown_connections 2024-04-25T13:38:14.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.735+0000 7fc169001640 1 -- 172.21.15.27:0/1441342599 wait complete. 2024-04-25T13:38:14.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.736+0000 7fc169001640 1 Processor -- start 2024-04-25T13:38:14.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.736+0000 7fc169001640 1 -- start start 2024-04-25T13:38:14.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc169001640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 0x7fc16406beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:14.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc169001640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc164103a80 0x7fc16419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:14.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc169001640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc164108780 0x7fc1641a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc169001640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc16410e340 con 0x7fc164102840 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc169001640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc16410e1c0 con 0x7fc164103a80 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc169001640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc16410e4c0 con 0x7fc164108780 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 0x7fc16406beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 0x7fc16406beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60566/0 (socket says 172.21.15.27:60566) 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 -- 172.21.15.27:0/2957492244 learned_addr learned my addr 172.21.15.27:0/2957492244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc16259c640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc164103a80 0x7fc16419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc16359e640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc164108780 0x7fc1641a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 -- 172.21.15.27:0/2957492244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc164108780 msgr2=0x7fc1641a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc164108780 0x7fc1641a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 -- 172.21.15.27:0/2957492244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc164103a80 msgr2=0x7fc16419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc164103a80 0x7fc16419ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 -- 172.21.15.27:0/2957492244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1641a1bd0 con 0x7fc164102840 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.737+0000 7fc162d9d640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 0x7fc16406beb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc154006010 tx=0x7fc154004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.738+0000 7fc14bfff640 1 -- 172.21.15.27:0/2957492244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15403f030 con 0x7fc164102840 2024-04-25T13:38:14.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.738+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1641a1e60 con 0x7fc164102840 2024-04-25T13:38:14.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.738+0000 7fc14bfff640 1 -- 172.21.15.27:0/2957492244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc154003aa0 con 0x7fc164102840 2024-04-25T13:38:14.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.738+0000 7fc14bfff640 1 -- 172.21.15.27:0/2957492244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154035070 con 0x7fc164102840 2024-04-25T13:38:14.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.738+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1641a22c0 con 0x7fc164102840 2024-04-25T13:38:14.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.739+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc164107590 con 0x7fc164102840 2024-04-25T13:38:14.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.741+0000 7fc14bfff640 1 -- 172.21.15.27:0/2957492244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc154042030 con 0x7fc164102840 2024-04-25T13:38:14.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.742+0000 7fc14bfff640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc14c0afae0 0x7fc14c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:14.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.742+0000 7fc16259c640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc14c0afae0 0x7fc14c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:14.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.742+0000 7fc14bfff640 1 -- 172.21.15.27:0/2957492244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc1540ff100 con 0x7fc164102840 2024-04-25T13:38:14.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.743+0000 7fc16259c640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc14c0afae0 0x7fc14c0b1ff0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc16406b850 tx=0x7fc158005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:14.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.750+0000 7fc14bfff640 1 -- 172.21.15.27:0/2957492244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1540c7b30 con 0x7fc164102840 2024-04-25T13:38:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.991+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc16402db50 con 0x7fc164102840 2024-04-25T13:38:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.992+0000 7fc14bfff640 1 -- 172.21.15.27:0/2957492244 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fc1540d0070 con 0x7fc164102840 2024-04-25T13:38:14.992 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:14.993 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T13:38:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 4 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc14c0afae0 msgr2=0x7fc14c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc14c0afae0 0x7fc14c0b1ff0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc16406b850 tx=0x7fc158005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 msgr2=0x7fc16406beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 0x7fc16406beb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc154006010 tx=0x7fc154004270 comp rx=0 tx=0).stop 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 shutdown_connections 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc14c0afae0 0x7fc14c0b1ff0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc164108780 0x7fc1641a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc164103a80 0x7fc16419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 --2- 172.21.15.27:0/2957492244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc164102840 0x7fc16406beb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 >> 172.21.15.27:0/2957492244 conn(0x7fc1640fda30 msgr2=0x7fc164104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 shutdown_connections 2024-04-25T13:38:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:14.994+0000 7fc169001640 1 -- 172.21.15.27:0/2957492244 wait complete. 2024-04-25T13:38:15.577 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:38:15 smithi027 podman[67457]: 2024-04-25 13:38:15.543586258 +0000 UTC m=+0.350720510 container start 1ee835dc342cae2854fe494609282c24e52ea9b067670606382dde9310b5af4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T13:38:15.901 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mds[67477]: -- 172.21.15.27:0/66501084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563901ed7a40 con 0x563901f1fa80 2024-04-25T13:38:15.901 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-d[67473]: starting mds.d at 2024-04-25T13:38:15.901 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:38:15 smithi027 bash[67457]: 1ee835dc342cae2854fe494609282c24e52ea9b067670606382dde9310b5af4b 2024-04-25T13:38:15.901 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:38:15 smithi027 systemd[1]: Started Ceph mds.d for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:15 smithi122 ceph-mon[32843]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:15 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2957492244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:16.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:16.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:15 smithi045 ceph-mon[32731]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:15 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2957492244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mon[29140]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2957492244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:16.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:17.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 -- 172.21.15.27:0/3471897187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 msgr2=0x7f7bf01037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3471897187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 0x7f7bf01037d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009900 tx=0x7f7bd802a410 comp rx=0 tx=0).stop 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 -- 172.21.15.27:0/3471897187 shutdown_connections 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3471897187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf01080c0 0x7f7bf010a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3471897187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 0x7f7bf01037d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3471897187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0102180 0x7f7bf0102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 -- 172.21.15.27:0/3471897187 >> 172.21.15.27:0/3471897187 conn(0x7f7bf00fd3b0 msgr2=0x7f7bf00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 -- 172.21.15.27:0/3471897187 shutdown_connections 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 -- 172.21.15.27:0/3471897187 wait complete. 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 Processor -- start 2024-04-25T13:38:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.138+0000 7f7bf6538640 1 -- start start 2024-04-25T13:38:17.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bf6538640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf0102180 0x7f7bf0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:17.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bf6538640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 0x7f7bf0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:17.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bf6538640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf01080c0 0x7f7bf019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:17.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bf6538640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf010d410 con 0x7f7bf01033c0 2024-04-25T13:38:17.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bf6538640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf010d290 con 0x7f7bf01080c0 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bf6538640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf010d590 con 0x7f7bf0102180 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 0x7f7bf0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bef7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 0x7f7bf0198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60594/0 (socket says 172.21.15.27:60594) 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bef7fe640 1 -- 172.21.15.27:0/3363650783 learned_addr learned my addr 172.21.15.27:0/3363650783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7beffff640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf0102180 0x7f7bf0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bf4ad5640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf01080c0 0x7f7bf019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bef7fe640 1 -- 172.21.15.27:0/3363650783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf0102180 msgr2=0x7f7bf0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bef7fe640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf0102180 0x7f7bf0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bef7fe640 1 -- 172.21.15.27:0/3363650783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf01080c0 msgr2=0x7f7bf019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bef7fe640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf01080c0 0x7f7bf019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.139+0000 7f7bef7fe640 1 -- 172.21.15.27:0/3363650783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf019fb20 con 0x7f7bf01033c0 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.140+0000 7f7bef7fe640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 0x7f7bf0198520 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8006010 tx=0x7f7bd8004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:17.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.140+0000 7f7bed7fa640 1 -- 172.21.15.27:0/3363650783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd803f030 con 0x7f7bf01033c0 2024-04-25T13:38:17.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.140+0000 7f7bed7fa640 1 -- 172.21.15.27:0/3363650783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bd8003cb0 con 0x7f7bf01033c0 2024-04-25T13:38:17.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.140+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf019fdb0 con 0x7f7bf01033c0 2024-04-25T13:38:17.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.140+0000 7f7bed7fa640 1 -- 172.21.15.27:0/3363650783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8035070 con 0x7f7bf01033c0 2024-04-25T13:38:17.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.140+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf01a01e0 con 0x7f7bf01033c0 2024-04-25T13:38:17.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.141+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf0106ed0 con 0x7f7bf01033c0 2024-04-25T13:38:17.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.142+0000 7f7bed7fa640 1 -- 172.21.15.27:0/3363650783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f7bd8042030 con 0x7f7bf01033c0 2024-04-25T13:38:17.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.143+0000 7f7bed7fa640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7bdc0afaf0 0x7f7bdc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:17.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.143+0000 7f7bed7fa640 1 -- 172.21.15.27:0/3363650783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f7bd80ff3c0 con 0x7f7bf01033c0 2024-04-25T13:38:17.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.143+0000 7f7beffff640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7bdc0afaf0 0x7f7bdc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:17.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.144+0000 7f7beffff640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7bdc0afaf0 0x7f7bdc0b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0102d80 tx=0x7f7be0010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:17.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.147+0000 7f7bed7fa640 1 -- 172.21.15.27:0/3363650783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7bd8003e60 con 0x7f7bf01033c0 2024-04-25T13:38:17.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:16 smithi122 ceph-mon[32843]: Deploying daemon mds.e on smithi045 2024-04-25T13:38:17.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:16 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.27:6836/481277630,v1:172.21.15.27:6837/481277630] up:boot 2024-04-25T13:38:17.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:16 smithi122 ceph-mon[32843]: fsmap 4 up:standby 2024-04-25T13:38:17.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:16 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T13:38:17.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:16 smithi045 ceph-mon[32731]: Deploying daemon mds.e on smithi045 2024-04-25T13:38:17.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:16 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.27:6836/481277630,v1:172.21.15.27:6837/481277630] up:boot 2024-04-25T13:38:17.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:16 smithi045 ceph-mon[32731]: fsmap 4 up:standby 2024-04-25T13:38:17.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:16 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T13:38:17.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.373+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bf002db50 con 0x7f7bf01033c0 2024-04-25T13:38:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.374+0000 7f7bed7fa640 1 -- 172.21.15.27:0/3363650783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f7bd80c7df0 con 0x7f7bf01033c0 2024-04-25T13:38:17.374 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:17.374 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T13:38:17.375 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 5 2024-04-25T13:38:17.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7bdc0afaf0 msgr2=0x7f7bdc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:17.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7bdc0afaf0 0x7f7bdc0b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0102d80 tx=0x7f7be0010040 comp rx=0 tx=0).stop 2024-04-25T13:38:17.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 msgr2=0x7f7bf0198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:17.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 0x7f7bf0198520 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8006010 tx=0x7f7bd8004340 comp rx=0 tx=0).stop 2024-04-25T13:38:17.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 shutdown_connections 2024-04-25T13:38:17.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7bdc0afaf0 0x7f7bdc0b2000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf01080c0 0x7f7bf019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7bf01033c0 0x7f7bf0198520 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 --2- 172.21.15.27:0/3363650783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7bf0102180 0x7f7bf0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 >> 172.21.15.27:0/3363650783 conn(0x7f7bf00fd3b0 msgr2=0x7f7bf01040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 shutdown_connections 2024-04-25T13:38:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:17.376+0000 7f7bf6538640 1 -- 172.21.15.27:0/3363650783 wait complete. 2024-04-25T13:38:17.386 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:16 smithi027 ceph-mon[29140]: Deploying daemon mds.e on smithi045 2024-04-25T13:38:17.386 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:16 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.27:6836/481277630,v1:172.21.15.27:6837/481277630] up:boot 2024-04-25T13:38:17.386 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:16 smithi027 ceph-mon[29140]: fsmap 4 up:standby 2024-04-25T13:38:17.386 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:16 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T13:38:18.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:17 smithi045 ceph-mon[32731]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:18.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:17 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3363650783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:18.079 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:17 smithi045 systemd[1]: Starting Ceph mds.e for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:17 smithi122 ceph-mon[32843]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:17 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3363650783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:17 smithi027 ceph-mon[29140]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:17 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3363650783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:18.439 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:18 smithi045 podman[62964]: 2024-04-25 13:38:18.071293804 +0000 UTC m=+0.017973025 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:18.439 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:18 smithi045 podman[62964]: 2024-04-25 13:38:18.196057073 +0000 UTC m=+0.142736283 container create fae0190ee89d5d4c646ac917326de0ba5a044940d935b9fe2165a0177ad433b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T13:38:18.828 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:18 smithi045 podman[62964]: 2024-04-25 13:38:18.440255876 +0000 UTC m=+0.386935082 container init fae0190ee89d5d4c646ac917326de0ba5a044940d935b9fe2165a0177ad433b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-25T13:38:18.828 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:18 smithi045 podman[62964]: 2024-04-25 13:38:18.442967633 +0000 UTC m=+0.389646849 container start fae0190ee89d5d4c646ac917326de0ba5a044940d935b9fe2165a0177ad433b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:38:18.828 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:18 smithi045 ceph-mds[62983]: -- 172.21.15.45:0/2089021264 <== mon.2 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5644cc2a3a40 con 0x5644cc2eba80 2024-04-25T13:38:18.829 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e[62979]: starting mds.e at 2024-04-25T13:38:18.829 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:18 smithi045 bash[62964]: fae0190ee89d5d4c646ac917326de0ba5a044940d935b9fe2165a0177ad433b3 2024-04-25T13:38:18.829 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:38:18 smithi045 systemd[1]: Started Ceph mds.e for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:19.388 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:19.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 -- 172.21.15.27:0/2872874300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438103b00 msgr2=0x7fc438104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 --2- 172.21.15.27:0/2872874300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438103b00 0x7fc438104800 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc428009900 tx=0x7fc42802a410 comp rx=0 tx=0).stop 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 -- 172.21.15.27:0/2872874300 shutdown_connections 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 --2- 172.21.15.27:0/2872874300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc438109a40 0x7fc43810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 --2- 172.21.15.27:0/2872874300 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc438104d40 0x7fc438105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 --2- 172.21.15.27:0/2872874300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438103b00 0x7fc438104800 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 -- 172.21.15.27:0/2872874300 >> 172.21.15.27:0/2872874300 conn(0x7fc4380fed30 msgr2=0x7fc4381011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 -- 172.21.15.27:0/2872874300 shutdown_connections 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.514+0000 7fc43e896640 1 -- 172.21.15.27:0/2872874300 wait complete. 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43e896640 1 Processor -- start 2024-04-25T13:38:19.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43e896640 1 -- start start 2024-04-25T13:38:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43e896640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc438103b00 0x7fc43819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43e896640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438104d40 0x7fc43819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43e896640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc438109a40 0x7fc4381a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:19.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43e896640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc43810f660 con 0x7fc438104d40 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43e896640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc43810f4e0 con 0x7fc438103b00 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43e896640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc43810f7e0 con 0x7fc438109a40 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc4377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438104d40 0x7fc43819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc4377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438104d40 0x7fc43819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60612/0 (socket says 172.21.15.27:60612) 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc4377fe640 1 -- 172.21.15.27:0/3888050467 learned_addr learned my addr 172.21.15.27:0/3888050467 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc437fff640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc438103b00 0x7fc43819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc43ce33640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc438109a40 0x7fc4381a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc4377fe640 1 -- 172.21.15.27:0/3888050467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc438109a40 msgr2=0x7fc4381a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.515+0000 7fc4377fe640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc438109a40 0x7fc4381a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc4377fe640 1 -- 172.21.15.27:0/3888050467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc438103b00 msgr2=0x7fc43819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc4377fe640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc438103b00 0x7fc43819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc4377fe640 1 -- 172.21.15.27:0/3888050467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4381a1dd0 con 0x7fc438104d40 2024-04-25T13:38:19.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc4377fe640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438104d40 0x7fc43819a7d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00b4a0 tx=0x7fc42c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc4357fa640 1 -- 172.21.15.27:0/3888050467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c014070 con 0x7fc438104d40 2024-04-25T13:38:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc4357fa640 1 -- 172.21.15.27:0/3888050467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc42c010ac0 con 0x7fc438104d40 2024-04-25T13:38:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4381a20c0 con 0x7fc438104d40 2024-04-25T13:38:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc4357fa640 1 -- 172.21.15.27:0/3888050467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c017c30 con 0x7fc438104d40 2024-04-25T13:38:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.516+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4381a2690 con 0x7fc438104d40 2024-04-25T13:38:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.517+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc438108850 con 0x7fc438104d40 2024-04-25T13:38:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.519+0000 7fc4357fa640 1 -- 172.21.15.27:0/3888050467 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc42c017070 con 0x7fc438104d40 2024-04-25T13:38:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.520+0000 7fc4357fa640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4200afaf0 0x7fc4200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.520+0000 7fc437fff640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4200afaf0 0x7fc4200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:19.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.520+0000 7fc4357fa640 1 -- 172.21.15.27:0/3888050467 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc42c0dddd0 con 0x7fc438104d40 2024-04-25T13:38:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.521+0000 7fc437fff640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4200afaf0 0x7fc4200b2000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc428005ab0 tx=0x7fc428031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.524+0000 7fc4357fa640 1 -- 172.21.15.27:0/3888050467 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc42c0a6800 con 0x7fc438104d40 2024-04-25T13:38:19.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.758+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc43802db50 con 0x7fc438104d40 2024-04-25T13:38:19.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.759+0000 7fc4357fa640 1 -- 172.21.15.27:0/3888050467 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7fc42c0aab00 con 0x7fc438104d40 2024-04-25T13:38:19.759 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:19.759 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T13:38:19.760 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 6 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4200afaf0 msgr2=0x7fc4200b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4200afaf0 0x7fc4200b2000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc428005ab0 tx=0x7fc428031400 comp rx=0 tx=0).stop 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438104d40 msgr2=0x7fc43819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438104d40 0x7fc43819a7d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00b4a0 tx=0x7fc42c00b960 comp rx=0 tx=0).stop 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 shutdown_connections 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4200afaf0 0x7fc4200b2000 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc438109a40 0x7fc4381a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc438104d40 0x7fc43819a7d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 --2- 172.21.15.27:0/3888050467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc438103b00 0x7fc43819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:19.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.761+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 >> 172.21.15.27:0/3888050467 conn(0x7fc4380fed30 msgr2=0x7fc438105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:19.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.762+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 shutdown_connections 2024-04-25T13:38:19.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:19.762+0000 7fc43e896640 1 -- 172.21.15.27:0/3888050467 wait complete. 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: Deploying daemon mds.f on smithi122 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.45:6834/4225728799,v1:172.21.15.45:6835/4225728799] up:boot 2024-04-25T13:38:19.772 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: fsmap 5 up:standby 2024-04-25T13:38:19.773 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T13:38:19.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:19.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:19.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:19.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:19.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:19.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:19.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:19.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: Deploying daemon mds.f on smithi122 2024-04-25T13:38:19.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.45:6834/4225728799,v1:172.21.15.45:6835/4225728799] up:boot 2024-04-25T13:38:19.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: fsmap 5 up:standby 2024-04-25T13:38:19.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T13:38:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: Deploying daemon mds.f on smithi122 2024-04-25T13:38:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.45:6834/4225728799,v1:172.21.15.45:6835/4225728799] up:boot 2024-04-25T13:38:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: fsmap 5 up:standby 2024-04-25T13:38:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T13:38:20.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:20 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3888050467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:20.990 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:20 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3888050467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:20.991 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:20 smithi122 systemd[1]: Starting Ceph mds.f for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:21.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:20 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3888050467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:21.267 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:20 smithi122 podman[63657]: 2024-04-25 13:38:20.893036882 +0000 UTC m=+0.017946105 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:21.267 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:21 smithi122 podman[63657]: 2024-04-25 13:38:21.004401075 +0000 UTC m=+0.129310298 container create f8e414b7cf09bd3642fe92273573d9110cce4682d81d104d2f80f427e3c8215a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T13:38:21.267 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:21 smithi122 podman[63657]: 2024-04-25 13:38:21.232133826 +0000 UTC m=+0.357043049 container init f8e414b7cf09bd3642fe92273573d9110cce4682d81d104d2f80f427e3c8215a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-25T13:38:21.267 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:21 smithi122 podman[63657]: 2024-04-25 13:38:21.234681486 +0000 UTC m=+0.359590710 container start f8e414b7cf09bd3642fe92273573d9110cce4682d81d104d2f80f427e3c8215a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T13:38:21.267 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-mds[63677]: -- 172.21.15.122:0/1637035542 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5587507fda40 con 0x558750845a80 2024-04-25T13:38:21.558 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f[63673]: starting mds.f at 2024-04-25T13:38:21.558 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:21 smithi122 bash[63657]: f8e414b7cf09bd3642fe92273573d9110cce4682d81d104d2f80f427e3c8215a 2024-04-25T13:38:21.558 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:38:21 smithi122 systemd[1]: Started Ceph mds.f for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:21.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:21.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:21 smithi027 ceph-mon[29140]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:21.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.900+0000 7f0466b72640 1 -- 172.21.15.27:0/118531089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460109a50 msgr2=0x7f046010be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.900+0000 7f0466b72640 1 --2- 172.21.15.27:0/118531089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460109a50 0x7f046010be70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0458009ca0 tx=0x7f045802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.900+0000 7f0466b72640 1 -- 172.21.15.27:0/118531089 shutdown_connections 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.900+0000 7f0466b72640 1 --2- 172.21.15.27:0/118531089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460109a50 0x7f046010be70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.901+0000 7f0466b72640 1 --2- 172.21.15.27:0/118531089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0460104d50 0x7f0460105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.901+0000 7f0466b72640 1 --2- 172.21.15.27:0/118531089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0460103b10 0x7f0460104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.901+0000 7f0466b72640 1 -- 172.21.15.27:0/118531089 >> 172.21.15.27:0/118531089 conn(0x7f04600fed40 msgr2=0x7f04601011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.901+0000 7f0466b72640 1 -- 172.21.15.27:0/118531089 shutdown_connections 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.901+0000 7f0466b72640 1 -- 172.21.15.27:0/118531089 wait complete. 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.901+0000 7f0466b72640 1 Processor -- start 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.901+0000 7f0466b72640 1 -- start start 2024-04-25T13:38:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f0466b72640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0460103b10 0x7f046019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f0466b72640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460104d50 0x7f046019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f0466b72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0460109a50 0x7f04601a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f0466b72640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f046010f620 con 0x7f0460104d50 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f0466b72640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f046010f4a0 con 0x7f0460109a50 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f0466b72640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f046010f7a0 con 0x7f0460103b10 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460104d50 0x7f046019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460104d50 0x7f046019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60622/0 (socket says 172.21.15.27:60622) 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 -- 172.21.15.27:0/917313099 learned_addr learned my addr 172.21.15.27:0/917313099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f046490e640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0460103b10 0x7f046019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f046510f640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0460109a50 0x7f04601a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 -- 172.21.15.27:0/917313099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0460103b10 msgr2=0x7f046019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0460103b10 0x7f046019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 -- 172.21.15.27:0/917313099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0460109a50 msgr2=0x7f04601a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0460109a50 0x7f04601a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 -- 172.21.15.27:0/917313099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04601a1d30 con 0x7f0460104d50 2024-04-25T13:38:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f046510f640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0460109a50 0x7f04601a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045ffff640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460104d50 0x7f046019a730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f044c00e9f0 tx=0x7f044c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045dffb640 1 -- 172.21.15.27:0/917313099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c00cd80 con 0x7f0460104d50 2024-04-25T13:38:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045dffb640 1 -- 172.21.15.27:0/917313099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f044c0108e0 con 0x7f0460104d50 2024-04-25T13:38:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04601a1fc0 con 0x7f0460104d50 2024-04-25T13:38:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.902+0000 7f045dffb640 1 -- 172.21.15.27:0/917313099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c018980 con 0x7f0460104d50 2024-04-25T13:38:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.903+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04601a24b0 con 0x7f0460104d50 2024-04-25T13:38:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.904+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0460108860 con 0x7f0460104d50 2024-04-25T13:38:21.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.905+0000 7f045dffb640 1 -- 172.21.15.27:0/917313099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f044c010430 con 0x7f0460104d50 2024-04-25T13:38:21.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.906+0000 7f045dffb640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04500afaf0 0x7f04500b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:21.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.906+0000 7f046490e640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04500afaf0 0x7f04500b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:21.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.906+0000 7f045dffb640 1 -- 172.21.15.27:0/917313099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f044c0defe0 con 0x7f0460104d50 2024-04-25T13:38:21.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.906+0000 7f046490e640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04500afaf0 0x7f04500b2000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f04540095f0 tx=0x7f0454005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:21.909+0000 7f045dffb640 1 -- 172.21.15.27:0/917313099 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f044c0a8060 con 0x7f0460104d50 2024-04-25T13:38:22.040 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-mon[32843]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:22.040 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:22.040 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:22.040 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:22.040 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:22.040 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:22.040 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:22.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:21 smithi045 ceph-mon[32731]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:22.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:22.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:22.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:22.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:22.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:22.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.139+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f046002db50 con 0x7f0460104d50 2024-04-25T13:38:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.140+0000 7f045dffb640 1 -- 172.21.15.27:0/917313099 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f044c0b00e0 con 0x7f0460104d50 2024-04-25T13:38:22.140 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:22.140 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T13:38:22.141 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-04-25T13:38:22.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04500afaf0 msgr2=0x7f04500b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:22.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04500afaf0 0x7f04500b2000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f04540095f0 tx=0x7f0454005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:22.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460104d50 msgr2=0x7f046019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:22.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460104d50 0x7f046019a730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f044c00e9f0 tx=0x7f044c00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:38:22.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 shutdown_connections 2024-04-25T13:38:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04500afaf0 0x7f04500b2000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0460109a50 0x7f04601a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0460104d50 0x7f046019a730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 --2- 172.21.15.27:0/917313099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0460103b10 0x7f046019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 >> 172.21.15.27:0/917313099 conn(0x7f04600fed40 msgr2=0x7f0460105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 shutdown_connections 2024-04-25T13:38:22.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:22.142+0000 7f0466b72640 1 -- 172.21.15.27:0/917313099 wait complete. 2024-04-25T13:38:22.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:22 smithi122 ceph-mon[32843]: Deploying daemon mds.c on smithi122 2024-04-25T13:38:22.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:22 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.122:6834/1340027477,v1:172.21.15.122:6835/1340027477] up:boot 2024-04-25T13:38:22.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:22 smithi122 ceph-mon[32843]: fsmap 6 up:standby 2024-04-25T13:38:22.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:22 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T13:38:22.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:22 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/917313099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:22.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:22 smithi027 ceph-mon[29140]: Deploying daemon mds.c on smithi122 2024-04-25T13:38:22.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:22 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.122:6834/1340027477,v1:172.21.15.122:6835/1340027477] up:boot 2024-04-25T13:38:22.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:22 smithi027 ceph-mon[29140]: fsmap 6 up:standby 2024-04-25T13:38:22.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:22 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T13:38:22.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:22 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/917313099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:23.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:22 smithi045 ceph-mon[32731]: Deploying daemon mds.c on smithi122 2024-04-25T13:38:23.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:22 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.122:6834/1340027477,v1:172.21.15.122:6835/1340027477] up:boot 2024-04-25T13:38:23.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:22 smithi045 ceph-mon[32731]: fsmap 6 up:standby 2024-04-25T13:38:23.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:22 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T13:38:23.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:22 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/917313099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:23.617 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:23 smithi122 systemd[1]: Starting Ceph mds.c for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:23.874 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:23 smithi122 ceph-mon[32843]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:23.875 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:23 smithi122 podman[63969]: 2024-04-25 13:38:23.649967188 +0000 UTC m=+0.017327750 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:23.875 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:23 smithi122 podman[63969]: 2024-04-25 13:38:23.753456327 +0000 UTC m=+0.120816945 container create 83de7562972e04926abfe00a192555ad3cfe64f09bac17d5a317b332dda3e083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T13:38:23.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:23 smithi027 ceph-mon[29140]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:24.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:23 smithi045 ceph-mon[32731]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:24.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:24.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 -- 172.21.15.27:0/255252972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c103c00 msgr2=0x7f7c5c104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 --2- 172.21.15.27:0/255252972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c103c00 0x7f7c5c104010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009900 tx=0x7f7c4c02a410 comp rx=0 tx=0).stop 2024-04-25T13:38:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 -- 172.21.15.27:0/255252972 shutdown_connections 2024-04-25T13:38:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 --2- 172.21.15.27:0/255252972 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c108900 0x7f7c5c10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 --2- 172.21.15.27:0/255252972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c103c00 0x7f7c5c104010 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 --2- 172.21.15.27:0/255252972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c5c1029c0 0x7f7c5c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 -- 172.21.15.27:0/255252972 >> 172.21.15.27:0/255252972 conn(0x7f7c5c0fdbd0 msgr2=0x7f7c5c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 -- 172.21.15.27:0/255252972 shutdown_connections 2024-04-25T13:38:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.283+0000 7f7c64297640 1 -- 172.21.15.27:0/255252972 wait complete. 2024-04-25T13:38:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c64297640 1 Processor -- start 2024-04-25T13:38:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c64297640 1 -- start start 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c64297640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1029c0 0x7f7c5c199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c64297640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c5c103c00 0x7f7c5c199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c64297640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c108900 0x7f7c5c1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c64297640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c10e4e0 con 0x7f7c5c1029c0 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c64297640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c10e360 con 0x7f7c5c108900 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c64297640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5c10e660 con 0x7f7c5c103c00 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c62033640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1029c0 0x7f7c5c199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c62033640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1029c0 0x7f7c5c199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45888/0 (socket says 172.21.15.27:45888) 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c62033640 1 -- 172.21.15.27:0/1434844022 learned_addr learned my addr 172.21.15.27:0/1434844022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c62834640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c108900 0x7f7c5c1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.284+0000 7f7c61832640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c5c103c00 0x7f7c5c199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c62033640 1 -- 172.21.15.27:0/1434844022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c5c103c00 msgr2=0x7f7c5c199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c62033640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c5c103c00 0x7f7c5c199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c62033640 1 -- 172.21.15.27:0/1434844022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c108900 msgr2=0x7f7c5c1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c62033640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c108900 0x7f7c5c1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c62033640 1 -- 172.21.15.27:0/1434844022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5c1a0c60 con 0x7f7c5c1029c0 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c61832640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c5c103c00 0x7f7c5c199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c62033640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1029c0 0x7f7c5c199120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800eb50 tx=0x7f7c5800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c537fe640 1 -- 172.21.15.27:0/1434844022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58003b70 con 0x7f7c5c1029c0 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c537fe640 1 -- 172.21.15.27:0/1434844022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c5800ee50 con 0x7f7c5c1029c0 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c537fe640 1 -- 172.21.15.27:0/1434844022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58010790 con 0x7f7c5c1029c0 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5c1a0f50 con 0x7f7c5c1029c0 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.285+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c5c1a13b0 con 0x7f7c5c1029c0 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.286+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c5c06afa0 con 0x7f7c5c1029c0 2024-04-25T13:38:24.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.289+0000 7f7c537fe640 1 -- 172.21.15.27:0/1434844022 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f7c58004a40 con 0x7f7c5c1029c0 2024-04-25T13:38:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.289+0000 7f7c537fe640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7c480afb70 0x7f7c480b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.289+0000 7f7c61832640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7c480afb70 0x7f7c480b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:24.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.289+0000 7f7c537fe640 1 -- 172.21.15.27:0/1434844022 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f7c580def70 con 0x7f7c5c1029c0 2024-04-25T13:38:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.290+0000 7f7c61832640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7c480afb70 0x7f7c480b2080 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c000c00 tx=0x7f7c4c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:24.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.293+0000 7f7c537fe640 1 -- 172.21.15.27:0/1434844022 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c580a7ff0 con 0x7f7c5c1029c0 2024-04-25T13:38:24.308 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:24 smithi122 podman[63969]: 2024-04-25 13:38:24.00653458 +0000 UTC m=+0.373895143 container init 83de7562972e04926abfe00a192555ad3cfe64f09bac17d5a317b332dda3e083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T13:38:24.308 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:24 smithi122 podman[63969]: 2024-04-25 13:38:24.009147416 +0000 UTC m=+0.376507981 container start 83de7562972e04926abfe00a192555ad3cfe64f09bac17d5a317b332dda3e083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T13:38:24.308 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:24 smithi122 ceph-mds[63988]: -- 172.21.15.122:0/897762444 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560858af5a40 con 0x560858b3d180 2024-04-25T13:38:24.308 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:24 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c[63984]: starting mds.c at 2024-04-25T13:38:24.309 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:24 smithi122 bash[63969]: 83de7562972e04926abfe00a192555ad3cfe64f09bac17d5a317b332dda3e083 2024-04-25T13:38:24.309 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:38:24 smithi122 systemd[1]: Started Ceph mds.c for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.531+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c5c107710 con 0x7f7c5c1029c0 2024-04-25T13:38:24.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.531+0000 7f7c537fe640 1 -- 172.21.15.27:0/1434844022 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f7c580a78f0 con 0x7f7c5c1029c0 2024-04-25T13:38:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:24.532 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T13:38:24.532 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2024-04-25T13:38:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.533+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7c480afb70 msgr2=0x7f7c480b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.533+0000 7f7c64297640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7c480afb70 0x7f7c480b2080 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c000c00 tx=0x7f7c4c005d50 comp rx=0 tx=0).stop 2024-04-25T13:38:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.533+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1029c0 msgr2=0x7f7c5c199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.533+0000 7f7c64297640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1029c0 0x7f7c5c199120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800eb50 tx=0x7f7c5800c6a0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.533+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 shutdown_connections 2024-04-25T13:38:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.533+0000 7f7c64297640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7c480afb70 0x7f7c480b2080 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.533+0000 7f7c64297640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c5c108900 0x7f7c5c1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.533+0000 7f7c64297640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c5c103c00 0x7f7c5c199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.534+0000 7f7c64297640 1 --2- 172.21.15.27:0/1434844022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c1029c0 0x7f7c5c199120 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.534+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 >> 172.21.15.27:0/1434844022 conn(0x7f7c5c0fdbd0 msgr2=0x7f7c5c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.534+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 shutdown_connections 2024-04-25T13:38:24.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:24.534+0000 7f7c64297640 1 -- 172.21.15.27:0/1434844022 wait complete. 2024-04-25T13:38:25.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: Deploying daemon mds.h on smithi045 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1434844022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.122:6836/1997914176,v1:172.21.15.122:6837/1997914176] up:boot 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: fsmap 7 up:standby 2024-04-25T13:38:25.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: Deploying daemon mds.h on smithi045 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:25.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1434844022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:25.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.122:6836/1997914176,v1:172.21.15.122:6837/1997914176] up:boot 2024-04-25T13:38:25.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: fsmap 7 up:standby 2024-04-25T13:38:25.579 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: Deploying daemon mds.h on smithi045 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1434844022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.122:6836/1997914176,v1:172.21.15.122:6837/1997914176] up:boot 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: fsmap 7 up:standby 2024-04-25T13:38:25.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T13:38:26.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/389643068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac103c00 msgr2=0x7f2aac104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/389643068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac103c00 0x7f2aac104010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009ca0 tx=0x7f2a9c02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/389643068 shutdown_connections 2024-04-25T13:38:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/389643068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2aac108900 0x7f2aac10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/389643068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac103c00 0x7f2aac104010 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/389643068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2aac1029c0 0x7f2aac1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/389643068 >> 172.21.15.27:0/389643068 conn(0x7f2aac0fdbd0 msgr2=0x7f2aac100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/389643068 shutdown_connections 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.673+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/389643068 wait complete. 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab2bc5640 1 Processor -- start 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab2bc5640 1 -- start start 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab2bc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac1029c0 0x7f2aac199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab2bc5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2aac103c00 0x7f2aac199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab2bc5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2aac108900 0x7f2aac1a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab2bc5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac10e4c0 con 0x7f2aac1029c0 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab2bc5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac10e340 con 0x7f2aac103c00 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab2bc5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac10e640 con 0x7f2aac108900 2024-04-25T13:38:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab0961640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac1029c0 0x7f2aac199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab0961640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac1029c0 0x7f2aac199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45900/0 (socket says 172.21.15.27:45900) 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab0961640 1 -- 172.21.15.27:0/3062291497 learned_addr learned my addr 172.21.15.27:0/3062291497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2aabfff640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2aac103c00 0x7f2aac199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab1162640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2aac108900 0x7f2aac1a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab0961640 1 -- 172.21.15.27:0/3062291497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2aac108900 msgr2=0x7f2aac1a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab0961640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2aac108900 0x7f2aac1a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab0961640 1 -- 172.21.15.27:0/3062291497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2aac103c00 msgr2=0x7f2aac199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab0961640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2aac103c00 0x7f2aac199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.674+0000 7f2ab0961640 1 -- 172.21.15.27:0/3062291497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aac1a0b90 con 0x7f2aac1029c0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.675+0000 7f2ab0961640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac1029c0 0x7f2aac199100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400c970 tx=0x7f2aa400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.675+0000 7f2aa9ffb640 1 -- 172.21.15.27:0/3062291497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4007c70 con 0x7f2aac1029c0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.675+0000 7f2aa9ffb640 1 -- 172.21.15.27:0/3062291497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2aa4007e10 con 0x7f2aac1029c0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.675+0000 7f2aa9ffb640 1 -- 172.21.15.27:0/3062291497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40187b0 con 0x7f2aac1029c0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.675+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aac1a0e80 con 0x7f2aac1029c0 2024-04-25T13:38:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.675+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aac1a1340 con 0x7f2aac1029c0 2024-04-25T13:38:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.676+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aac107710 con 0x7f2aac1029c0 2024-04-25T13:38:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.677+0000 7f2aa9ffb640 1 -- 172.21.15.27:0/3062291497 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2aa400fab0 con 0x7f2aac1029c0 2024-04-25T13:38:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.678+0000 7f2aa9ffb640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a980afb70 0x7f2a980b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.678+0000 7f2aabfff640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a980afb70 0x7f2a980b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.678+0000 7f2aa9ffb640 1 -- 172.21.15.27:0/3062291497 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f2aa40de670 con 0x7f2aac1029c0 2024-04-25T13:38:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.679+0000 7f2aabfff640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a980afb70 0x7f2a980b2080 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0094e0 tx=0x7f2a9c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.683+0000 7f2aa9ffb640 1 -- 172.21.15.27:0/3062291497 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2aa4014030 con 0x7f2aac1029c0 2024-04-25T13:38:26.765 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:26 smithi045 systemd[1]: Starting Ceph mds.h for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.909+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2aac02db50 con 0x7f2aac1029c0 2024-04-25T13:38:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.909+0000 7f2aa9ffb640 1 -- 172.21.15.27:0/3062291497 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f2aa40a76f0 con 0x7f2aac1029c0 2024-04-25T13:38:26.909 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:26.910 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T13:38:26.910 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-04-25T13:38:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a980afb70 msgr2=0x7f2a980b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a980afb70 0x7f2a980b2080 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0094e0 tx=0x7f2a9c004a40 comp rx=0 tx=0).stop 2024-04-25T13:38:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac1029c0 msgr2=0x7f2aac199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac1029c0 0x7f2aac199100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400c970 tx=0x7f2aa400ce30 comp rx=0 tx=0).stop 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 shutdown_connections 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a980afb70 0x7f2a980b2080 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2aac108900 0x7f2aac1a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2aac103c00 0x7f2aac199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 --2- 172.21.15.27:0/3062291497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aac1029c0 0x7f2aac199100 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 >> 172.21.15.27:0/3062291497 conn(0x7f2aac0fdbd0 msgr2=0x7f2aac1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 shutdown_connections 2024-04-25T13:38:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:26.911+0000 7f2ab2bc5640 1 -- 172.21.15.27:0/3062291497 wait complete. 2024-04-25T13:38:27.019 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:26 smithi045 podman[63275]: 2024-04-25 13:38:26.667496012 +0000 UTC m=+0.018573665 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:27.019 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:26 smithi045 podman[63275]: 2024-04-25 13:38:26.78441994 +0000 UTC m=+0.135497579 container create e41878edd41fb86eced882d80a5f53d0415673491ca19e75c4f7b80f3c3e690b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-25T13:38:27.328 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:27 smithi045 podman[63275]: 2024-04-25 13:38:27.020439628 +0000 UTC m=+0.371517269 container init e41878edd41fb86eced882d80a5f53d0415673491ca19e75c4f7b80f3c3e690b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T13:38:27.328 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:27 smithi045 podman[63275]: 2024-04-25 13:38:27.023221334 +0000 UTC m=+0.374298973 container start e41878edd41fb86eced882d80a5f53d0415673491ca19e75c4f7b80f3c3e690b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux ) 2024-04-25T13:38:27.328 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mds[63294]: -- 172.21.15.45:0/1755895769 <== mon.2 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556eb4af9a40 con 0x556eb4b41a80 2024-04-25T13:38:27.328 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h[63290]: starting mds.h at 2024-04-25T13:38:27.328 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:27 smithi045 bash[63275]: e41878edd41fb86eced882d80a5f53d0415673491ca19e75c4f7b80f3c3e690b 2024-04-25T13:38:27.328 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:38:27 smithi045 systemd[1]: Started Ceph mds.h for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:27 smithi027 ceph-mon[29140]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:27 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3062291497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:27 smithi122 ceph-mon[32843]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:27.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:27 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3062291497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:27.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:27.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:27.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mon[32731]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3062291497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:27.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:27.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:28.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:28 smithi122 ceph-mon[32843]: Deploying daemon mds.i on smithi122 2024-04-25T13:38:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:28 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.45:6836/4218693179,v1:172.21.15.45:6837/4218693179] up:boot 2024-04-25T13:38:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:28 smithi122 ceph-mon[32843]: fsmap 8 up:standby 2024-04-25T13:38:28.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T13:38:28.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:28 smithi045 ceph-mon[32731]: Deploying daemon mds.i on smithi122 2024-04-25T13:38:28.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:28 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.45:6836/4218693179,v1:172.21.15.45:6837/4218693179] up:boot 2024-04-25T13:38:28.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:28 smithi045 ceph-mon[32731]: fsmap 8 up:standby 2024-04-25T13:38:28.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:28 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T13:38:28.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:28 smithi027 ceph-mon[29140]: Deploying daemon mds.i on smithi122 2024-04-25T13:38:28.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:28 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.45:6836/4218693179,v1:172.21.15.45:6837/4218693179] up:boot 2024-04-25T13:38:28.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:28 smithi027 ceph-mon[29140]: fsmap 8 up:standby 2024-04-25T13:38:28.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T13:38:28.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 -- 172.21.15.27:0/1289209025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10108920 msgr2=0x7eff1010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 --2- 172.21.15.27:0/1289209025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10108920 0x7eff1010ad40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7eff00009900 tx=0x7eff0002a410 comp rx=0 tx=0).stop 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 -- 172.21.15.27:0/1289209025 shutdown_connections 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 --2- 172.21.15.27:0/1289209025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10108920 0x7eff1010ad40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 --2- 172.21.15.27:0/1289209025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff10103c20 0x7eff10104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 --2- 172.21.15.27:0/1289209025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff101029e0 0x7eff101036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 -- 172.21.15.27:0/1289209025 >> 172.21.15.27:0/1289209025 conn(0x7eff100fdbd0 msgr2=0x7eff10100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 -- 172.21.15.27:0/1289209025 shutdown_connections 2024-04-25T13:38:29.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.047+0000 7eff14f0d640 1 -- 172.21.15.27:0/1289209025 wait complete. 2024-04-25T13:38:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.048+0000 7eff14f0d640 1 Processor -- start 2024-04-25T13:38:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.048+0000 7eff14f0d640 1 -- start start 2024-04-25T13:38:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff14f0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff101029e0 0x7eff1010d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff14f0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10103c20 0x7eff1010b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff14f0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff10108920 0x7eff1010be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff14f0d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7eff1010f610 con 0x7eff10103c20 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff14f0d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7eff1010f490 con 0x7eff10108920 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff14f0d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7eff1010f790 con 0x7eff101029e0 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10103c20 0x7eff1010b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10103c20 0x7eff1010b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45918/0 (socket says 172.21.15.27:45918) 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 -- 172.21.15.27:0/2482096881 learned_addr learned my addr 172.21.15.27:0/2482096881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0e59c640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff101029e0 0x7eff1010d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0ed9d640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff10108920 0x7eff1010be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 -- 172.21.15.27:0/2482096881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff101029e0 msgr2=0x7eff1010d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff101029e0 0x7eff1010d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 -- 172.21.15.27:0/2482096881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff10108920 msgr2=0x7eff1010be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff10108920 0x7eff1010be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 -- 172.21.15.27:0/2482096881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1010c5c0 con 0x7eff10103c20 2024-04-25T13:38:29.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff0dd9b640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10103c20 0x7eff1010b950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efef800d950 tx=0x7efef800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:29.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.049+0000 7eff077fe640 1 -- 172.21.15.27:0/2482096881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8017030 con 0x7eff10103c20 2024-04-25T13:38:29.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.050+0000 7eff077fe640 1 -- 172.21.15.27:0/2482096881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efef8004020 con 0x7eff10103c20 2024-04-25T13:38:29.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.050+0000 7eff077fe640 1 -- 172.21.15.27:0/2482096881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef80047a0 con 0x7eff10103c20 2024-04-25T13:38:29.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.050+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff101a42e0 con 0x7eff10103c20 2024-04-25T13:38:29.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.050+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff101a4750 con 0x7eff10103c20 2024-04-25T13:38:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.051+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff10107730 con 0x7eff10103c20 2024-04-25T13:38:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.052+0000 7eff077fe640 1 -- 172.21.15.27:0/2482096881 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7efef8009a30 con 0x7eff10103c20 2024-04-25T13:38:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.053+0000 7eff077fe640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efefc0afaf0 0x7efefc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.053+0000 7eff0e59c640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efefc0afaf0 0x7efefc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.053+0000 7eff077fe640 1 -- 172.21.15.27:0/2482096881 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7efef80de7c0 con 0x7eff10103c20 2024-04-25T13:38:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.053+0000 7eff0e59c640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efefc0afaf0 0x7efefc0b2000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7eff101035e0 tx=0x7eff08009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:29.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.056+0000 7eff077fe640 1 -- 172.21.15.27:0/2482096881 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efef80a7840 con 0x7eff10103c20 2024-04-25T13:38:29.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.283+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff1002db50 con 0x7eff10103c20 2024-04-25T13:38:29.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.283+0000 7eff077fe640 1 -- 172.21.15.27:0/2482096881 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7efef80a7140 con 0x7eff10103c20 2024-04-25T13:38:29.285 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:29.285 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T13:38:29.285 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 9 2024-04-25T13:38:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efefc0afaf0 msgr2=0x7efefc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efefc0afaf0 0x7efefc0b2000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7eff101035e0 tx=0x7eff08009670 comp rx=0 tx=0).stop 2024-04-25T13:38:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10103c20 msgr2=0x7eff1010b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10103c20 0x7eff1010b950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efef800d950 tx=0x7efef800de10 comp rx=0 tx=0).stop 2024-04-25T13:38:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 shutdown_connections 2024-04-25T13:38:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efefc0afaf0 0x7efefc0b2000 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff10108920 0x7eff1010be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10103c20 0x7eff1010b950 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 --2- 172.21.15.27:0/2482096881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff101029e0 0x7eff1010d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 >> 172.21.15.27:0/2482096881 conn(0x7eff100fdbd0 msgr2=0x7eff10104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 shutdown_connections 2024-04-25T13:38:29.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:29.286+0000 7eff14f0d640 1 -- 172.21.15.27:0/2482096881 wait complete. 2024-04-25T13:38:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:29 smithi122 ceph-mon[32843]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:29.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:29 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2482096881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:29.559 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:29 smithi122 systemd[1]: Starting Ceph mds.i for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:29.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:29 smithi027 ceph-mon[29140]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:29.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:29 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2482096881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:29.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:29 smithi045 ceph-mon[32731]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:29.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:29 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2482096881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:29.913 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:29 smithi122 podman[64277]: 2024-04-25 13:38:29.561775673 +0000 UTC m=+0.018178464 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:29.913 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:29 smithi122 podman[64277]: 2024-04-25 13:38:29.682001749 +0000 UTC m=+0.138404542 container create 115a3ae05213681ad8840899f72b34901c9903e2e03d04e35d61900d9f1f98ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T13:38:29.913 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:29 smithi122 podman[64277]: 2024-04-25 13:38:29.914118204 +0000 UTC m=+0.370520992 container init 115a3ae05213681ad8840899f72b34901c9903e2e03d04e35d61900d9f1f98ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T13:38:30.308 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:29 smithi122 podman[64277]: 2024-04-25 13:38:29.916678849 +0000 UTC m=+0.373081646 container start 115a3ae05213681ad8840899f72b34901c9903e2e03d04e35d61900d9f1f98ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:38:30.308 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:29 smithi122 ceph-mds[64296]: -- 172.21.15.122:0/3851274356 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e53aee7a40 con 0x55e53af2f600 2024-04-25T13:38:30.308 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:29 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i[64292]: starting mds.i at 2024-04-25T13:38:30.309 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:30 smithi122 bash[64277]: 115a3ae05213681ad8840899f72b34901c9903e2e03d04e35d61900d9f1f98ae 2024-04-25T13:38:30.309 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:38:30 smithi122 systemd[1]: Started Ceph mds.i for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:31.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:31.323 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.323 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:31.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:31.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:31.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.122:6838/3397333136,v1:172.21.15.122:6839/3397333136] up:boot 2024-04-25T13:38:31.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: fsmap 9 up:standby 2024-04-25T13:38:31.324 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T13:38:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:31.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.122:6838/3397333136,v1:172.21.15.122:6839/3397333136] up:boot 2024-04-25T13:38:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: fsmap 9 up:standby 2024-04-25T13:38:31.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.415+0000 7fb5381b2640 1 -- 172.21.15.27:0/1427779528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 msgr2=0x7fb53010bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.415+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1427779528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 0x7fb53010bcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb528009ca0 tx=0x7fb52802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.415+0000 7fb5381b2640 1 -- 172.21.15.27:0/1427779528 shutdown_connections 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.415+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1427779528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 0x7fb53010bcd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.415+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1427779528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb530106f50 0x7fb530109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.415+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1427779528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5301045f0 0x7fb530106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.415+0000 7fb5381b2640 1 -- 172.21.15.27:0/1427779528 >> 172.21.15.27:0/1427779528 conn(0x7fb5300fdbf0 msgr2=0x7fb5301000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.415+0000 7fb5381b2640 1 -- 172.21.15.27:0/1427779528 shutdown_connections 2024-04-25T13:38:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 -- 172.21.15.27:0/1427779528 wait complete. 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 Processor -- start 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 -- start start 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5301045f0 0x7fb530103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb530106f50 0x7fb5301020e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 0x7fb530102620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010f630 con 0x7fb5301098b0 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010f4b0 con 0x7fb5301045f0 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb5381b2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010f7b0 con 0x7fb530106f50 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.416+0000 7fb53674f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 0x7fb530102620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53674f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 0x7fb530102620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45944/0 (socket says 172.21.15.27:45944) 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53674f640 1 -- 172.21.15.27:0/1504691191 learned_addr learned my addr 172.21.15.27:0/1504691191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb535f4e640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5301045f0 0x7fb530103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53574d640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb530106f50 0x7fb5301020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53674f640 1 -- 172.21.15.27:0/1504691191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb530106f50 msgr2=0x7fb5301020e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53674f640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb530106f50 0x7fb5301020e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53674f640 1 -- 172.21.15.27:0/1504691191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5301045f0 msgr2=0x7fb530103a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53674f640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5301045f0 0x7fb530103a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53674f640 1 -- 172.21.15.27:0/1504691191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb530102d50 con 0x7fb5301098b0 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb535f4e640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5301045f0 0x7fb530103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb53674f640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 0x7fb530102620 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb528009430 tx=0x7fb528003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb522ffd640 1 -- 172.21.15.27:0/1504691191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52802f380 con 0x7fb5301098b0 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb522ffd640 1 -- 172.21.15.27:0/1504691191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5280317c0 con 0x7fb5301098b0 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5301a4260 con 0x7fb5301098b0 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb522ffd640 1 -- 172.21.15.27:0/1504691191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52802fcd0 con 0x7fb5301098b0 2024-04-25T13:38:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.417+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5301a46c0 con 0x7fb5301098b0 2024-04-25T13:38:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.418+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53006a190 con 0x7fb5301098b0 2024-04-25T13:38:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.420+0000 7fb522ffd640 1 -- 172.21.15.27:0/1504691191 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fb528004b40 con 0x7fb5301098b0 2024-04-25T13:38:31.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.421+0000 7fb522ffd640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb51c0afc90 0x7fb51c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:31.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.421+0000 7fb535f4e640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb51c0afc90 0x7fb51c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:31.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.421+0000 7fb522ffd640 1 -- 172.21.15.27:0/1504691191 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fb5280ffff0 con 0x7fb5301098b0 2024-04-25T13:38:31.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.421+0000 7fb535f4e640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb51c0afc90 0x7fb51c0b21a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb52c005d00 tx=0x7fb52c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:31.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.427+0000 7fb522ffd640 1 -- 172.21.15.27:0/1504691191 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5280098b0 con 0x7fb5301098b0 2024-04-25T13:38:31.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:31.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:31.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:31.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:31.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.122:6838/3397333136,v1:172.21.15.122:6839/3397333136] up:boot 2024-04-25T13:38:31.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: fsmap 9 up:standby 2024-04-25T13:38:31.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T13:38:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.654+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5300ff710 con 0x7fb5301098b0 2024-04-25T13:38:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.654+0000 7fb522ffd640 1 -- 172.21.15.27:0/1504691191 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7fb5280c9070 con 0x7fb5301098b0 2024-04-25T13:38:31.655 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:31.655 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T13:38:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 10 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb51c0afc90 msgr2=0x7fb51c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb51c0afc90 0x7fb51c0b21a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb52c005d00 tx=0x7fb52c005c70 comp rx=0 tx=0).stop 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 msgr2=0x7fb530102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 0x7fb530102620 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb528009430 tx=0x7fb528003ba0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 shutdown_connections 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb51c0afc90 0x7fb51c0b21a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5301098b0 0x7fb530102620 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb530106f50 0x7fb5301020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 --2- 172.21.15.27:0/1504691191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5301045f0 0x7fb530103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.656+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 >> 172.21.15.27:0/1504691191 conn(0x7fb5300fdbf0 msgr2=0x7fb5301050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.657+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 shutdown_connections 2024-04-25T13:38:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:31.657+0000 7fb5381b2640 1 -- 172.21.15.27:0/1504691191 wait complete. 2024-04-25T13:38:32.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:32 smithi027 ceph-mon[29140]: Deploying daemon mds.g on smithi027 2024-04-25T13:38:32.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:32 smithi027 ceph-mon[29140]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:32.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:32 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1504691191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:32.401 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 systemd[1]: Starting Ceph mds.g for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:32.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:32 smithi122 ceph-mon[32843]: Deploying daemon mds.g on smithi027 2024-04-25T13:38:32.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:32 smithi122 ceph-mon[32843]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:32.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:32 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1504691191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:32.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:32 smithi045 ceph-mon[32731]: Deploying daemon mds.g on smithi027 2024-04-25T13:38:32.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:32 smithi045 ceph-mon[32731]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:32.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:32 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1504691191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:32.803 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 podman[68056]: 2024-04-25 13:38:32.399495428 +0000 UTC m=+0.018848185 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:32.803 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 podman[68056]: 2024-04-25 13:38:32.533880326 +0000 UTC m=+0.153233071 container create c73ebae03d8ef50df78c404e0fe16d955d3eb4d003767c21592c4418f54d8dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424) 2024-04-25T13:38:32.803 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 podman[68056]: 2024-04-25 13:38:32.80399619 +0000 UTC m=+0.423348937 container init c73ebae03d8ef50df78c404e0fe16d955d3eb4d003767c21592c4418f54d8dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T13:38:33.151 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 podman[68056]: 2024-04-25 13:38:32.807048805 +0000 UTC m=+0.426401556 container start c73ebae03d8ef50df78c404e0fe16d955d3eb4d003767c21592c4418f54d8dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T13:38:33.151 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 ceph-mds[68075]: -- 172.21.15.27:0/1939468779 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560fba303a40 con 0x560fba34b180 2024-04-25T13:38:33.151 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g[68071]: starting mds.g at 2024-04-25T13:38:33.151 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 bash[68056]: c73ebae03d8ef50df78c404e0fe16d955d3eb4d003767c21592c4418f54d8dfe 2024-04-25T13:38:33.151 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:38:32 smithi027 systemd[1]: Started Ceph mds.g for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:33.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:33.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 -- 172.21.15.27:0/1549583017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 msgr2=0x7f97f8103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:33.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 --2- 172.21.15.27:0/1549583017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 0x7f97f8103140 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009930 tx=0x7f97e402a420 comp rx=0 tx=0).stop 2024-04-25T13:38:33.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 -- 172.21.15.27:0/1549583017 shutdown_connections 2024-04-25T13:38:33.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 --2- 172.21.15.27:0/1549583017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f8103680 0x7f97f8103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:33.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 --2- 172.21.15.27:0/1549583017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 0x7f97f8103140 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:33.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 --2- 172.21.15.27:0/1549583017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f81023e0 0x7f97f81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:33.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 -- 172.21.15.27:0/1549583017 >> 172.21.15.27:0/1549583017 conn(0x7f97f80fdbf0 msgr2=0x7f97f81000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:33.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 -- 172.21.15.27:0/1549583017 shutdown_connections 2024-04-25T13:38:33.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.785+0000 7f97fef34640 1 -- 172.21.15.27:0/1549583017 wait complete. 2024-04-25T13:38:33.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fef34640 1 Processor -- start 2024-04-25T13:38:33.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fef34640 1 -- start start 2024-04-25T13:38:33.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fef34640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f81023e0 0x7f97f819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:33.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fef34640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 0x7f97f819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:33.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fef34640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f8103680 0x7f97f81a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:33.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fef34640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97f8111840 con 0x7f97f8102d30 2024-04-25T13:38:33.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fef34640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97f81116c0 con 0x7f97f8103680 2024-04-25T13:38:33.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fef34640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f97f81119c0 con 0x7f97f81023e0 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 0x7f97f819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 0x7f97f819c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54486/0 (socket says 172.21.15.27:54486) 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97f7fff640 1 -- 172.21.15.27:0/3304976036 learned_addr learned my addr 172.21.15.27:0/3304976036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fccd0640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f81023e0 0x7f97f819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97f7fff640 1 -- 172.21.15.27:0/3304976036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f81023e0 msgr2=0x7f97f819c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97f7fff640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f81023e0 0x7f97f819c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97f7fff640 1 -- 172.21.15.27:0/3304976036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f8103680 msgr2=0x7f97f81a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97fd4d1640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f8103680 0x7f97f81a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97f7fff640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f8103680 0x7f97f81a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.786+0000 7f97f7fff640 1 -- 172.21.15.27:0/3304976036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f81a3fb0 con 0x7f97f8102d30 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.787+0000 7f97f7fff640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 0x7f97f819c9b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009230 tx=0x7f97e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.787+0000 7f97f5ffb640 1 -- 172.21.15.27:0/3304976036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e403f030 con 0x7f97f8102d30 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.787+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f81a4240 con 0x7f97f8102d30 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.787+0000 7f97f5ffb640 1 -- 172.21.15.27:0/3304976036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97e4003cb0 con 0x7f97f8102d30 2024-04-25T13:38:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.787+0000 7f97f5ffb640 1 -- 172.21.15.27:0/3304976036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4035070 con 0x7f97f8102d30 2024-04-25T13:38:33.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.787+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f81a4730 con 0x7f97f8102d30 2024-04-25T13:38:33.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.788+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f810d420 con 0x7f97f8102d30 2024-04-25T13:38:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.789+0000 7f97f5ffb640 1 -- 172.21.15.27:0/3304976036 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f97e4042030 con 0x7f97f8102d30 2024-04-25T13:38:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.790+0000 7f97f5ffb640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97e80afaf0 0x7f97e80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.790+0000 7f97fccd0640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97e80afaf0 0x7f97e80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.790+0000 7f97f5ffb640 1 -- 172.21.15.27:0/3304976036 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f97e40ff2b0 con 0x7f97f8102d30 2024-04-25T13:38:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.791+0000 7f97fccd0640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97e80afaf0 0x7f97e80b2000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f97f80757f0 tx=0x7f97ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:33.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:33.794+0000 7f97f5ffb640 1 -- 172.21.15.27:0/3304976036 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97e40c8330 con 0x7f97f8102d30 2024-04-25T13:38:34.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.023+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97f802db50 con 0x7f97f8102d30 2024-04-25T13:38:34.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.024+0000 7f97f5ffb640 1 -- 172.21.15.27:0/3304976036 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f97e40c7c30 con 0x7f97f8102d30 2024-04-25T13:38:34.024 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:34.024 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T13:38:34.025 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 11 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97e80afaf0 msgr2=0x7f97e80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97e80afaf0 0x7f97e80b2000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f97f80757f0 tx=0x7f97ec005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 msgr2=0x7f97f819c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 0x7f97f819c9b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009230 tx=0x7f97e40042f0 comp rx=0 tx=0).stop 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 shutdown_connections 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97e80afaf0 0x7f97e80b2000 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f8103680 0x7f97f81a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f8102d30 0x7f97f819c9b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 --2- 172.21.15.27:0/3304976036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f81023e0 0x7f97f819c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.026+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 >> 172.21.15.27:0/3304976036 conn(0x7f97f80fdbf0 msgr2=0x7f97f810a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.027+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 shutdown_connections 2024-04-25T13:38:34.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:34.027+0000 7f97fef34640 1 -- 172.21.15.27:0/3304976036 wait complete. 2024-04-25T13:38:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:34.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:34.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: Deploying daemon mds.a on smithi027 2024-04-25T13:38:34.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.27:6838/2432690809,v1:172.21.15.27:6839/2432690809] up:boot 2024-04-25T13:38:34.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: fsmap 10 up:standby 2024-04-25T13:38:34.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: Deploying daemon mds.a on smithi027 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.27:6838/2432690809,v1:172.21.15.27:6839/2432690809] up:boot 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: fsmap 10 up:standby 2024-04-25T13:38:34.319 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T13:38:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:34.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:34.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: Deploying daemon mds.a on smithi027 2024-04-25T13:38:34.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.27:6838/2432690809,v1:172.21.15.27:6839/2432690809] up:boot 2024-04-25T13:38:34.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: fsmap 10 up:standby 2024-04-25T13:38:34.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T13:38:35.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:34 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3304976036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:35.325 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:34 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3304976036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:35.325 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 systemd[1]: Starting Ceph mds.a for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:35.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:34 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3304976036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:35.628 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 podman[68414]: 2024-04-25 13:38:35.233990138 +0000 UTC m=+0.018991883 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:35.628 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 podman[68414]: 2024-04-25 13:38:35.366974797 +0000 UTC m=+0.151976532 container create e939bfeeb51a5280087e0df981dd8231554dd90f76b7d0e55513a736a5caecf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T13:38:35.628 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 podman[68414]: 2024-04-25 13:38:35.628677176 +0000 UTC m=+0.413678908 container init e939bfeeb51a5280087e0df981dd8231554dd90f76b7d0e55513a736a5caecf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T13:38:35.901 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 podman[68414]: 2024-04-25 13:38:35.631541357 +0000 UTC m=+0.416543099 container start e939bfeeb51a5280087e0df981dd8231554dd90f76b7d0e55513a736a5caecf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-25T13:38:35.901 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-mds[68434]: -- 172.21.15.27:0/2787909824 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560d4b51a40 con 0x5560d4b99600 2024-04-25T13:38:35.901 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a[68430]: starting mds.a at 2024-04-25T13:38:35.901 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 bash[68414]: e939bfeeb51a5280087e0df981dd8231554dd90f76b7d0e55513a736a5caecf4 2024-04-25T13:38:35.901 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:38:35 smithi027 systemd[1]: Started Ceph mds.a for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:36.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/616027137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 msgr2=0x7f6bec104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/616027137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 0x7f6bec104010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009900 tx=0x7f6bd402a410 comp rx=0 tx=0).stop 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/616027137 shutdown_connections 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/616027137 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bec108900 0x7f6bec10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/616027137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 0x7f6bec104010 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/616027137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bec1029c0 0x7f6bec1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/616027137 >> 172.21.15.27:0/616027137 conn(0x7f6bec0fdbd0 msgr2=0x7f6bec100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/616027137 shutdown_connections 2024-04-25T13:38:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.153+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/616027137 wait complete. 2024-04-25T13:38:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.154+0000 7f6bf0d1f640 1 Processor -- start 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.154+0000 7f6bf0d1f640 1 -- start start 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.154+0000 7f6bf0d1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bec1029c0 0x7f6bec0ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.154+0000 7f6bf0d1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 0x7f6bec0fe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.154+0000 7f6bf0d1f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bec108900 0x7f6bec0fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.154+0000 7f6bf0d1f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10e4f0 con 0x7f6bec103c00 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.154+0000 7f6bf0d1f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10e370 con 0x7f6bec1029c0 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.154+0000 7f6bf0d1f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10e670 con 0x7f6bec108900 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 0x7f6bec0fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 0x7f6bec0fe100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54520/0 (socket says 172.21.15.27:54520) 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 -- 172.21.15.27:0/1145298078 learned_addr learned my addr 172.21.15.27:0/1145298078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6bea59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bec1029c0 0x7f6bec0ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6bead9d640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bec108900 0x7f6bec0fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 -- 172.21.15.27:0/1145298078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bec108900 msgr2=0x7f6bec0fe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bec108900 0x7f6bec0fe640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 -- 172.21.15.27:0/1145298078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bec1029c0 msgr2=0x7f6bec0ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bec1029c0 0x7f6bec0ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 -- 172.21.15.27:0/1145298078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec0fed70 con 0x7f6bec103c00 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6bea59c640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bec1029c0 0x7f6bec0ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be1d9b640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 0x7f6bec0fe100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4006010 tx=0x7f6bd4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be3fff640 1 -- 172.21.15.27:0/1145298078 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd403f030 con 0x7f6bec103c00 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be3fff640 1 -- 172.21.15.27:0/1145298078 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd4003aa0 con 0x7f6bec103c00 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec1a42d0 con 0x7f6bec103c00 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6be3fff640 1 -- 172.21.15.27:0/1145298078 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4035070 con 0x7f6bec103c00 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.155+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec1a46d0 con 0x7f6bec103c00 2024-04-25T13:38:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.156+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec06afa0 con 0x7f6bec103c00 2024-04-25T13:38:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.159+0000 7f6be3fff640 1 -- 172.21.15.27:0/1145298078 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6bd4042030 con 0x7f6bec103c00 2024-04-25T13:38:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.159+0000 7f6be3fff640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bd80afaf0 0x7f6bd80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.160+0000 7f6bea59c640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bd80afaf0 0x7f6bd80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.160+0000 7f6be3fff640 1 -- 172.21.15.27:0/1145298078 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f6bd40ff380 con 0x7f6bec103c00 2024-04-25T13:38:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.160+0000 7f6bea59c640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bd80afaf0 0x7f6bd80b2000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6bec1035c0 tx=0x7f6bdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:36.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.163+0000 7f6be3fff640 1 -- 172.21.15.27:0/1145298078 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bd40c8400 con 0x7f6bec103c00 2024-04-25T13:38:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:35 smithi122 ceph-mon[32843]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:36.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:35 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:36.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:35 smithi045 ceph-mon[32731]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:36.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:36.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:36.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:35 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.387+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bec1a4930 con 0x7f6bec103c00 2024-04-25T13:38:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.388+0000 7f6be3fff640 1 -- 172.21.15.27:0/1145298078 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f6bd40c7d00 con 0x7f6bec103c00 2024-04-25T13:38:36.388 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:36.388 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T13:38:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 12 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.389+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bd80afaf0 msgr2=0x7f6bd80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.389+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bd80afaf0 0x7f6bd80b2000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6bec1035c0 tx=0x7f6bdc005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.389+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 msgr2=0x7f6bec0fe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.389+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 0x7f6bec0fe100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4006010 tx=0x7f6bd4031a50 comp rx=0 tx=0).stop 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.390+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 shutdown_connections 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.390+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bd80afaf0 0x7f6bd80b2000 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.390+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bec108900 0x7f6bec0fe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.390+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bec103c00 0x7f6bec0fe100 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.390+0000 7f6bf0d1f640 1 --2- 172.21.15.27:0/1145298078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bec1029c0 0x7f6bec0ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.390+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 >> 172.21.15.27:0/1145298078 conn(0x7f6bec0fdbd0 msgr2=0x7f6bec109d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.390+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 shutdown_connections 2024-04-25T13:38:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:36.390+0000 7f6bf0d1f640 1 -- 172.21.15.27:0/1145298078 wait complete. 2024-04-25T13:38:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-mon[29140]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T13:38:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T13:38:36.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:35 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:37.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:36 smithi045 ceph-mon[32731]: Deploying daemon mds.b on smithi045 2024-04-25T13:38:37.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:36 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.27:6840/1048885133,v1:172.21.15.27:6841/1048885133] up:boot 2024-04-25T13:38:37.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:36 smithi045 ceph-mon[32731]: fsmap 11 up:standby 2024-04-25T13:38:37.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:36 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T13:38:37.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:36 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1145298078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:37.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:36 smithi122 ceph-mon[32843]: Deploying daemon mds.b on smithi045 2024-04-25T13:38:37.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:36 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.27:6840/1048885133,v1:172.21.15.27:6841/1048885133] up:boot 2024-04-25T13:38:37.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:36 smithi122 ceph-mon[32843]: fsmap 11 up:standby 2024-04-25T13:38:37.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:36 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T13:38:37.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:36 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1145298078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:37.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:36 smithi027 ceph-mon[29140]: Deploying daemon mds.b on smithi045 2024-04-25T13:38:37.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:36 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.27:6840/1048885133,v1:172.21.15.27:6841/1048885133] up:boot 2024-04-25T13:38:37.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:36 smithi027 ceph-mon[29140]: fsmap 11 up:standby 2024-04-25T13:38:37.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:36 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T13:38:37.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:36 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1145298078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:38.066 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:37 smithi045 ceph-mon[32731]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:38.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:37 smithi122 ceph-mon[32843]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:38.328 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 systemd[1]: Starting Ceph mds.b for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:38:38.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:37 smithi027 ceph-mon[29140]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:38.402 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.528+0000 7fbd642d1640 1 -- 172.21.15.27:0/1858211394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c106f50 msgr2=0x7fbd5c109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.528+0000 7fbd642d1640 1 --2- 172.21.15.27:0/1858211394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c106f50 0x7fbd5c109370 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009900 tx=0x7fbd4c02a410 comp rx=0 tx=0).stop 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.528+0000 7fbd642d1640 1 -- 172.21.15.27:0/1858211394 shutdown_connections 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.528+0000 7fbd642d1640 1 --2- 172.21.15.27:0/1858211394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd5c1098b0 0x7fbd5c10bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.528+0000 7fbd642d1640 1 --2- 172.21.15.27:0/1858211394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c106f50 0x7fbd5c109370 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.528+0000 7fbd642d1640 1 --2- 172.21.15.27:0/1858211394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd5c1045f0 0x7fbd5c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.528+0000 7fbd642d1640 1 -- 172.21.15.27:0/1858211394 >> 172.21.15.27:0/1858211394 conn(0x7fbd5c0fdbf0 msgr2=0x7fbd5c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.529+0000 7fbd642d1640 1 -- 172.21.15.27:0/1858211394 shutdown_connections 2024-04-25T13:38:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.529+0000 7fbd642d1640 1 -- 172.21.15.27:0/1858211394 wait complete. 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.529+0000 7fbd642d1640 1 Processor -- start 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.529+0000 7fbd642d1640 1 -- start start 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.529+0000 7fbd642d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd5c1045f0 0x7fbd5c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd642d1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd5c106f50 0x7fbd5c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd642d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c1098b0 0x7fbd5c1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd642d1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c10f630 con 0x7fbd5c1098b0 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd642d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c10f4b0 con 0x7fbd5c1045f0 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd642d1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c10f7b0 con 0x7fbd5c106f50 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c1098b0 0x7fbd5c1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c1098b0 0x7fbd5c1a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54532/0 (socket says 172.21.15.27:54532) 2024-04-25T13:38:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 -- 172.21.15.27:0/3202708418 learned_addr learned my addr 172.21.15.27:0/3202708418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6206d640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd5c1045f0 0x7fbd5c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6186c640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd5c106f50 0x7fbd5c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 -- 172.21.15.27:0/3202708418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd5c106f50 msgr2=0x7fbd5c19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd5c106f50 0x7fbd5c19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 -- 172.21.15.27:0/3202708418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd5c1045f0 msgr2=0x7fbd5c19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd5c1045f0 0x7fbd5c19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 -- 172.21.15.27:0/3202708418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5c1a1d40 con 0x7fbd5c1098b0 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6186c640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd5c106f50 0x7fbd5c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd6286e640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c1098b0 0x7fbd5c1a16d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400e9d0 tx=0x7fbd5400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd537fe640 1 -- 172.21.15.27:0/3202708418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5400cd80 con 0x7fbd5c1098b0 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd537fe640 1 -- 172.21.15.27:0/3202708418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd54005020 con 0x7fbd5c1098b0 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd5c1a1fd0 con 0x7fbd5c1098b0 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd537fe640 1 -- 172.21.15.27:0/3202708418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd540116d0 con 0x7fbd5c1098b0 2024-04-25T13:38:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.530+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd5c1a2490 con 0x7fbd5c1098b0 2024-04-25T13:38:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.531+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd5c0ff710 con 0x7fbd5c1098b0 2024-04-25T13:38:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.533+0000 7fbd537fe640 1 -- 172.21.15.27:0/3202708418 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbd540051d0 con 0x7fbd5c1098b0 2024-04-25T13:38:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.534+0000 7fbd537fe640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbd480afaf0 0x7fbd480b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.534+0000 7fbd6206d640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbd480afaf0 0x7fbd480b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.534+0000 7fbd537fe640 1 -- 172.21.15.27:0/3202708418 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fbd540dec40 con 0x7fbd5c1098b0 2024-04-25T13:38:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.534+0000 7fbd6206d640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbd480afaf0 0x7fbd480b2000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0757f0 tx=0x7fbd5800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:38.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.538+0000 7fbd537fe640 1 -- 172.21.15.27:0/3202708418 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd54016030 con 0x7fbd5c1098b0 2024-04-25T13:38:38.733 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 podman[63583]: 2024-04-25 13:38:38.372382398 +0000 UTC m=+0.018659474 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:38:38.733 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 podman[63583]: 2024-04-25 13:38:38.489238815 +0000 UTC m=+0.135515870 container create 172f6a43af0244f31bca7830bde97da5bb22a93b0266befd86211bb434dcfe7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-25T13:38:38.733 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 podman[63583]: 2024-04-25 13:38:38.733745615 +0000 UTC m=+0.380022671 container init 172f6a43af0244f31bca7830bde97da5bb22a93b0266befd86211bb434dcfe7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T13:38:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.762+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd5c02db50 con 0x7fbd5c1098b0 2024-04-25T13:38:38.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.763+0000 7fbd537fe640 1 -- 172.21.15.27:0/3202708418 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7fbd540a7cc0 con 0x7fbd5c1098b0 2024-04-25T13:38:38.763 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:38.764 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T13:38:38.764 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 12 2024-04-25T13:38:38.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.765+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbd480afaf0 msgr2=0x7fbd480b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:38.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.765+0000 7fbd642d1640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbd480afaf0 0x7fbd480b2000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0757f0 tx=0x7fbd5800b390 comp rx=0 tx=0).stop 2024-04-25T13:38:38.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.765+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c1098b0 msgr2=0x7fbd5c1a16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:38.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.765+0000 7fbd642d1640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c1098b0 0x7fbd5c1a16d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400e9d0 tx=0x7fbd5400ee90 comp rx=0 tx=0).stop 2024-04-25T13:38:38.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.766+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 shutdown_connections 2024-04-25T13:38:38.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.766+0000 7fbd642d1640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbd480afaf0 0x7fbd480b2000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.766+0000 7fbd642d1640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd5c1098b0 0x7fbd5c1a16d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.766+0000 7fbd642d1640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd5c106f50 0x7fbd5c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.766+0000 7fbd642d1640 1 --2- 172.21.15.27:0/3202708418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd5c1045f0 0x7fbd5c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:38.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.766+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 >> 172.21.15.27:0/3202708418 conn(0x7fbd5c0fdbf0 msgr2=0x7fbd5c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:38.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.766+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 shutdown_connections 2024-04-25T13:38:38.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:38.766+0000 7fbd642d1640 1 -- 172.21.15.27:0/3202708418 wait complete. 2024-04-25T13:38:38.997 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 podman[63583]: 2024-04-25 13:38:38.736480627 +0000 UTC m=+0.382757686 container start 172f6a43af0244f31bca7830bde97da5bb22a93b0266befd86211bb434dcfe7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T13:38:38.997 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 ceph-mds[63602]: -- 172.21.15.45:0/1049787026 <== mon.2 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5631263d9a40 con 0x563126421a80 2024-04-25T13:38:38.997 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b[63598]: starting mds.b at 2024-04-25T13:38:38.997 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 bash[63583]: 172f6a43af0244f31bca7830bde97da5bb22a93b0266befd86211bb434dcfe7a 2024-04-25T13:38:38.997 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:38 smithi045 systemd[1]: Started Ceph mds.b for db01fc44-0307-11ef-bc93-c7b262605968. 2024-04-25T13:38:39.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:38 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3202708418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:39.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:38 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:38 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:38 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:38 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.153 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:38 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:38 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3202708418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:39.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:38 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:38 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:38 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:38 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:38 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:38 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3202708418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:39.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:38 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:38 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:38 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:38 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:39.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:38 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:40.203 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:39 smithi045 ceph-mon[32731]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:40.203 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:39 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] up:boot 2024-04-25T13:38:40.203 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:39 smithi045 ceph-mon[32731]: fsmap 12 up:standby 2024-04-25T13:38:40.203 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:39 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T13:38:40.204 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:39 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:40.209 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:39 smithi122 ceph-mon[32843]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:40.209 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:39 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] up:boot 2024-04-25T13:38:40.209 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:39 smithi122 ceph-mon[32843]: fsmap 12 up:standby 2024-04-25T13:38:40.209 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:39 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T13:38:40.209 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:39 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:39 smithi027 ceph-mon[29140]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:40.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:39 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] up:boot 2024-04-25T13:38:40.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:39 smithi027 ceph-mon[29140]: fsmap 12 up:standby 2024-04-25T13:38:40.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:39 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T13:38:40.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:39 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:40.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:40.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 -- 172.21.15.27:0/4161246466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 msgr2=0x7f8170105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:40.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 --2- 172.21.15.27:0/4161246466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 0x7f8170105150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8164009900 tx=0x7f816402a410 comp rx=0 tx=0).stop 2024-04-25T13:38:40.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 -- 172.21.15.27:0/4161246466 shutdown_connections 2024-04-25T13:38:40.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 --2- 172.21.15.27:0/4161246466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8170109a40 0x7f817010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:40.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 --2- 172.21.15.27:0/4161246466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 0x7f8170105150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:40.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 --2- 172.21.15.27:0/4161246466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8170103b00 0x7f8170104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:40.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 -- 172.21.15.27:0/4161246466 >> 172.21.15.27:0/4161246466 conn(0x7f81700fed10 msgr2=0x7f81701011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:40.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 -- 172.21.15.27:0/4161246466 shutdown_connections 2024-04-25T13:38:40.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.895+0000 7f8175d8f640 1 -- 172.21.15.27:0/4161246466 wait complete. 2024-04-25T13:38:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f8175d8f640 1 Processor -- start 2024-04-25T13:38:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f8175d8f640 1 -- start start 2024-04-25T13:38:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f8175d8f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8170103b00 0x7f817019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f8175d8f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 0x7f817019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f8175d8f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8170109a40 0x7f81701a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f8175d8f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f817010f680 con 0x7f8170104d40 2024-04-25T13:38:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f8175d8f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f817010f500 con 0x7f8170103b00 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f8175d8f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f817010f800 con 0x7f8170109a40 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f816effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 0x7f817019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f816effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 0x7f817019a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54552/0 (socket says 172.21.15.27:54552) 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f816effd640 1 -- 172.21.15.27:0/1049615515 learned_addr learned my addr 172.21.15.27:0/1049615515 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f816ffff640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8170109a40 0x7f81701a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.896+0000 7f816f7fe640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8170103b00 0x7f817019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816effd640 1 -- 172.21.15.27:0/1049615515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8170109a40 msgr2=0x7f81701a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816effd640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8170109a40 0x7f81701a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816effd640 1 -- 172.21.15.27:0/1049615515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8170103b00 msgr2=0x7f817019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816effd640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8170103b00 0x7f817019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816effd640 1 -- 172.21.15.27:0/1049615515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81701a1d60 con 0x7f8170104d40 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816ffff640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8170109a40 0x7f81701a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816effd640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 0x7f817019a7d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8164031440 tx=0x7f8164003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816cff9640 1 -- 172.21.15.27:0/1049615515 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816403f070 con 0x7f8170104d40 2024-04-25T13:38:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816cff9640 1 -- 172.21.15.27:0/1049615515 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8164031c50 con 0x7f8170104d40 2024-04-25T13:38:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f816cff9640 1 -- 172.21.15.27:0/1049615515 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8164035030 con 0x7f8170104d40 2024-04-25T13:38:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81701a1ff0 con 0x7f8170104d40 2024-04-25T13:38:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.897+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81701a2450 con 0x7f8170104d40 2024-04-25T13:38:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.898+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8170108850 con 0x7f8170104d40 2024-04-25T13:38:40.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.903+0000 7f816cff9640 1 -- 172.21.15.27:0/1049615515 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8164042070 con 0x7f8170104d40 2024-04-25T13:38:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.904+0000 7f816cff9640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f81580afb70 0x7f81580b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.904+0000 7f816f7fe640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f81580afb70 0x7f81580b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.904+0000 7f816cff9640 1 -- 172.21.15.27:0/1049615515 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f81640ff660 con 0x7f8170104d40 2024-04-25T13:38:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.905+0000 7f816f7fe640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f81580afb70 0x7f81580b2080 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8170104700 tx=0x7f8160005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:40.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:40.908+0000 7f816cff9640 1 -- 172.21.15.27:0/1049615515 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81640c86c0 con 0x7f8170104d40 2024-04-25T13:38:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.136+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f817002db50 con 0x7f8170104d40 2024-04-25T13:38:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.136+0000 7f816cff9640 1 -- 172.21.15.27:0/1049615515 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7f81640c7fc0 con 0x7f8170104d40 2024-04-25T13:38:41.137 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:41.137 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24517,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T13:38:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 13 2024-04-25T13:38:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f81580afb70 msgr2=0x7f81580b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f81580afb70 0x7f81580b2080 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8170104700 tx=0x7f8160005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 msgr2=0x7f817019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 0x7f817019a7d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8164031440 tx=0x7f8164003c10 comp rx=0 tx=0).stop 2024-04-25T13:38:41.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 shutdown_connections 2024-04-25T13:38:41.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f81580afb70 0x7f81580b2080 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8170109a40 0x7f81701a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8170104d40 0x7f817019a7d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 --2- 172.21.15.27:0/1049615515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8170103b00 0x7f817019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 >> 172.21.15.27:0/1049615515 conn(0x7f81700fed10 msgr2=0x7f8170105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:41.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 shutdown_connections 2024-04-25T13:38:41.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.139+0000 7f8175d8f640 1 -- 172.21.15.27:0/1049615515 wait complete. 2024-04-25T13:38:41.150 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 -- 172.21.15.27:0/3129238295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb4103c00 msgr2=0x7f3bb4104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 --2- 172.21.15.27:0/3129238295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb4103c00 0x7f3bb4104010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009900 tx=0x7f3ba802a410 comp rx=0 tx=0).stop 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 -- 172.21.15.27:0/3129238295 shutdown_connections 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 --2- 172.21.15.27:0/3129238295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bb4108900 0x7f3bb410ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 --2- 172.21.15.27:0/3129238295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb4103c00 0x7f3bb4104010 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 --2- 172.21.15.27:0/3129238295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb41029c0 0x7f3bb41036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 -- 172.21.15.27:0/3129238295 >> 172.21.15.27:0/3129238295 conn(0x7f3bb40fdbd0 msgr2=0x7f3bb4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 -- 172.21.15.27:0/3129238295 shutdown_connections 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.261+0000 7f3bba6af640 1 -- 172.21.15.27:0/3129238295 wait complete. 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bba6af640 1 Processor -- start 2024-04-25T13:38:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bba6af640 1 -- start start 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bba6af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb41029c0 0x7f3bb406a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bba6af640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4103c00 0x7f3bb419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bba6af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bb4108900 0x7f3bb41a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bba6af640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb410e4f0 con 0x7f3bb41029c0 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bba6af640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb410e370 con 0x7f3bb4103c00 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bba6af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb410e670 con 0x7f3bb4108900 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bb3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb41029c0 0x7f3bb406a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bb3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb41029c0 0x7f3bb406a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54584/0 (socket says 172.21.15.27:54584) 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bb3fff640 1 -- 172.21.15.27:0/206352020 learned_addr learned my addr 172.21.15.27:0/206352020 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.262+0000 7f3bb8c4c640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bb4108900 0x7f3bb41a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb37fe640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4103c00 0x7f3bb419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb3fff640 1 -- 172.21.15.27:0/206352020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bb4108900 msgr2=0x7f3bb41a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb3fff640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bb4108900 0x7f3bb41a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb3fff640 1 -- 172.21.15.27:0/206352020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4103c00 msgr2=0x7f3bb419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb3fff640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4103c00 0x7f3bb419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb3fff640 1 -- 172.21.15.27:0/206352020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb41a1d20 con 0x7f3bb41029c0 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb3fff640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb41029c0 0x7f3bb406a920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400d740 tx=0x7f3ba400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb17fa640 1 -- 172.21.15.27:0/206352020 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4017070 con 0x7f3bb41029c0 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb17fa640 1 -- 172.21.15.27:0/206352020 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba400de50 con 0x7f3bb41029c0 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb41a1fb0 con 0x7f3bb41029c0 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bb17fa640 1 -- 172.21.15.27:0/206352020 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4003c20 con 0x7f3bb41029c0 2024-04-25T13:38:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.263+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb41a2470 con 0x7f3bb41029c0 2024-04-25T13:38:41.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.264+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bb4107710 con 0x7f3bb41029c0 2024-04-25T13:38:41.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.266+0000 7f3bb17fa640 1 -- 172.21.15.27:0/206352020 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3ba4003dc0 con 0x7f3bb41029c0 2024-04-25T13:38:41.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.267+0000 7f3bb17fa640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3b9c0afaf0 0x7f3b9c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.267+0000 7f3bb37fe640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3b9c0afaf0 0x7f3b9c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.267+0000 7f3bb17fa640 1 -- 172.21.15.27:0/206352020 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f3ba40de580 con 0x7f3bb41029c0 2024-04-25T13:38:41.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.267+0000 7f3bb37fe640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3b9c0afaf0 0x7f3b9c0b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8000c00 tx=0x7f3ba8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:41.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.271+0000 7f3bb17fa640 1 -- 172.21.15.27:0/206352020 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ba40a7600 con 0x7f3bb41029c0 2024-04-25T13:38:41.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:41.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:41 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1049615515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.460+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3bb402db50 con 0x7f3bb41029c0 2024-04-25T13:38:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.461+0000 7f3bb17fa640 1 -- 172.21.15.27:0/206352020 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18410 (secure 0 0 0) 0x7f3ba40c9020 con 0x7f3bb41029c0 2024-04-25T13:38:41.462 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:41.462 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:37:54.399799+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3b9c0afaf0 msgr2=0x7f3b9c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3b9c0afaf0 0x7f3b9c0b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8000c00 tx=0x7f3ba8031400 comp rx=0 tx=0).stop 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb41029c0 msgr2=0x7f3bb406a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb41029c0 0x7f3bb406a920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400d740 tx=0x7f3ba400dc00 comp rx=0 tx=0).stop 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 shutdown_connections 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3b9c0afaf0 0x7f3b9c0b2000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3bb4108900 0x7f3bb41a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bb4103c00 0x7f3bb419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 --2- 172.21.15.27:0/206352020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bb41029c0 0x7f3bb406a920 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.463+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 >> 172.21.15.27:0/206352020 conn(0x7f3bb40fdbd0 msgr2=0x7f3bb41048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.464+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 shutdown_connections 2024-04-25T13:38:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.464+0000 7f3bba6af640 1 -- 172.21.15.27:0/206352020 wait complete. 2024-04-25T13:38:41.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:38:41.478 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:41 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1049615515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:41.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 -- 172.21.15.27:0/1279209285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380109a40 msgr2=0x7f338010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 --2- 172.21.15.27:0/1279209285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380109a40 0x7f338010be60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3378009ca0 tx=0x7f337802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 -- 172.21.15.27:0/1279209285 shutdown_connections 2024-04-25T13:38:41.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 --2- 172.21.15.27:0/1279209285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380109a40 0x7f338010be60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 --2- 172.21.15.27:0/1279209285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3380104d40 0x7f3380105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 --2- 172.21.15.27:0/1279209285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3380103b00 0x7f3380104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 -- 172.21.15.27:0/1279209285 >> 172.21.15.27:0/1279209285 conn(0x7f33800fed10 msgr2=0x7f33801011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 -- 172.21.15.27:0/1279209285 shutdown_connections 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.593+0000 7f3386d84640 1 -- 172.21.15.27:0/1279209285 wait complete. 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3386d84640 1 Processor -- start 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3386d84640 1 -- start start 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3386d84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380103b00 0x7f338019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3386d84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3380104d40 0x7f338019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3386d84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3380109a40 0x7f33801a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3386d84640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f338010f630 con 0x7f3380103b00 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3386d84640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f338010f4b0 con 0x7f3380104d40 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3386d84640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f338010f7b0 con 0x7f3380109a40 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3384b20640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380103b00 0x7f338019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3384b20640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380103b00 0x7f338019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54606/0 (socket says 172.21.15.27:54606) 2024-04-25T13:38:41.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3384b20640 1 -- 172.21.15.27:0/381414683 learned_addr learned my addr 172.21.15.27:0/381414683 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3385321640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3380109a40 0x7f33801a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f337ffff640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3380104d40 0x7f338019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3384b20640 1 -- 172.21.15.27:0/381414683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3380109a40 msgr2=0x7f33801a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3384b20640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3380109a40 0x7f33801a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3384b20640 1 -- 172.21.15.27:0/381414683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3380104d40 msgr2=0x7f338019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3384b20640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3380104d40 0x7f338019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.594+0000 7f3384b20640 1 -- 172.21.15.27:0/381414683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33801a1d50 con 0x7f3380103b00 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.595+0000 7f3384b20640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380103b00 0x7f338019a210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3370005d00 tx=0x7f337000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.595+0000 7f337dffb640 1 -- 172.21.15.27:0/381414683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3370017070 con 0x7f3380103b00 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.595+0000 7f337dffb640 1 -- 172.21.15.27:0/381414683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3370009c30 con 0x7f3380103b00 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.595+0000 7f337dffb640 1 -- 172.21.15.27:0/381414683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337000f620 con 0x7f3380103b00 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.595+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33801a1fe0 con 0x7f3380103b00 2024-04-25T13:38:41.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.595+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33801a23f0 con 0x7f3380103b00 2024-04-25T13:38:41.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.596+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f338006a190 con 0x7f3380103b00 2024-04-25T13:38:41.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.597+0000 7f337dffb640 1 -- 172.21.15.27:0/381414683 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f337001a070 con 0x7f3380103b00 2024-04-25T13:38:41.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.598+0000 7f337dffb640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f336c0afaf0 0x7f336c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.598+0000 7f337ffff640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f336c0afaf0 0x7f336c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.598+0000 7f337dffb640 1 -- 172.21.15.27:0/381414683 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f33700de140 con 0x7f3380103b00 2024-04-25T13:38:41.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.599+0000 7f337ffff640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f336c0afaf0 0x7f336c0b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f338019b2b0 tx=0x7f336800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:41.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.602+0000 7f337dffb640 1 -- 172.21.15.27:0/381414683 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f338006a190 con 0x7f3380103b00 2024-04-25T13:38:41.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.789+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f338002dc30 con 0x7f3380103b00 2024-04-25T13:38:41.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.790+0000 7f337dffb640 1 -- 172.21.15.27:0/381414683 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f3370009de0 con 0x7f3380103b00 2024-04-25T13:38:41.790 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:38:41.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f336c0afaf0 msgr2=0x7f336c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f336c0afaf0 0x7f336c0b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f338019b2b0 tx=0x7f336800b410 comp rx=0 tx=0).stop 2024-04-25T13:38:41.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380103b00 msgr2=0x7f338019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380103b00 0x7f338019a210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3370005d00 tx=0x7f337000dde0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 shutdown_connections 2024-04-25T13:38:41.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f336c0afaf0 0x7f336c0b2000 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3380109a40 0x7f33801a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3380104d40 0x7f338019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 --2- 172.21.15.27:0/381414683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3380103b00 0x7f338019a210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 >> 172.21.15.27:0/381414683 conn(0x7f33800fed10 msgr2=0x7f3380105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:41.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 shutdown_connections 2024-04-25T13:38:41.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.792+0000 7f3386d84640 1 -- 172.21.15.27:0/381414683 wait complete. 2024-04-25T13:38:41.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T13:38:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:41.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:41 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1049615515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:41.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 -- 172.21.15.27:0/3201345582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d601029c0 msgr2=0x7f1d601036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/3201345582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d601029c0 0x7f1d601036c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009900 tx=0x7f1d5002a410 comp rx=0 tx=0).stop 2024-04-25T13:38:41.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 -- 172.21.15.27:0/3201345582 shutdown_connections 2024-04-25T13:38:41.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/3201345582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d60108900 0x7f1d6010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/3201345582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d60103c00 0x7f1d60104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/3201345582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d601029c0 0x7f1d601036c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 -- 172.21.15.27:0/3201345582 >> 172.21.15.27:0/3201345582 conn(0x7f1d600fdbd0 msgr2=0x7f1d60100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:41.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 -- 172.21.15.27:0/3201345582 shutdown_connections 2024-04-25T13:38:41.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.917+0000 7f1d64ab6640 1 -- 172.21.15.27:0/3201345582 wait complete. 2024-04-25T13:38:41.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d64ab6640 1 Processor -- start 2024-04-25T13:38:41.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d64ab6640 1 -- start start 2024-04-25T13:38:41.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d64ab6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d601029c0 0x7f1d6019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d64ab6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d60103c00 0x7f1d6019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d64ab6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d60108900 0x7f1d601a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d64ab6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6010e4a0 con 0x7f1d60108900 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d64ab6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6010e320 con 0x7f1d601029c0 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d64ab6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6010e620 con 0x7f1d60103c00 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d5ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d60108900 0x7f1d601a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d5ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d60108900 0x7f1d601a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54628/0 (socket says 172.21.15.27:54628) 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d5ed9d640 1 -- 172.21.15.27:0/1455865491 learned_addr learned my addr 172.21.15.27:0/1455865491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d5e59c640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d601029c0 0x7f1d6019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.918+0000 7f1d5dd9b640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d60103c00 0x7f1d6019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d5ed9d640 1 -- 172.21.15.27:0/1455865491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d60103c00 msgr2=0x7f1d6019a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d5ed9d640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d60103c00 0x7f1d6019a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d5ed9d640 1 -- 172.21.15.27:0/1455865491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d601029c0 msgr2=0x7f1d6019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:41.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d5ed9d640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d601029c0 0x7f1d6019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d5ed9d640 1 -- 172.21.15.27:0/1455865491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d601a1d90 con 0x7f1d60108900 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d5ed9d640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d60108900 0x7f1d601a1720 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009fd0 tx=0x7f1d5800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d4f7fe640 1 -- 172.21.15.27:0/1455865491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58017070 con 0x7f1d60108900 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d4f7fe640 1 -- 172.21.15.27:0/1455865491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d58003e20 con 0x7f1d60108900 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d601a2080 con 0x7f1d60108900 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d4f7fe640 1 -- 172.21.15.27:0/1455865491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58010460 con 0x7f1d60108900 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.919+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d601a2580 con 0x7f1d60108900 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.920+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d601a27e0 con 0x7f1d60108900 2024-04-25T13:38:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.922+0000 7f1d4f7fe640 1 -- 172.21.15.27:0/1455865491 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1d58003970 con 0x7f1d60108900 2024-04-25T13:38:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.923+0000 7f1d4f7fe640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d480afaf0 0x7f1d480b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.923+0000 7f1d5e59c640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d480afaf0 0x7f1d480b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.923+0000 7f1d4f7fe640 1 -- 172.21.15.27:0/1455865491 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f1d580de470 con 0x7f1d60108900 2024-04-25T13:38:41.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.924+0000 7f1d5e59c640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d480afaf0 0x7f1d480b2000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1d50005ab0 tx=0x7f1d50031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:41.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:41.928+0000 7f1d4f7fe640 1 -- 172.21.15.27:0/1455865491 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1d580a6ea0 con 0x7f1d60108900 2024-04-25T13:38:42.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.118+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1d601a29b0 con 0x7f1d60108900 2024-04-25T13:38:42.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.118+0000 7f1d4f7fe640 1 -- 172.21.15.27:0/1455865491 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f1d580ab1a0 con 0x7f1d60108900 2024-04-25T13:38:42.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:42.119 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-04-25T13:38:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:38:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-04-25T13:38:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-04-25T13:38:42.120 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:38:42.120 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-04-25T13:38:42.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d480afaf0 msgr2=0x7f1d480b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:42.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d480afaf0 0x7f1d480b2000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1d50005ab0 tx=0x7f1d50031400 comp rx=0 tx=0).stop 2024-04-25T13:38:42.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d60108900 msgr2=0x7f1d601a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d60108900 0x7f1d601a1720 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009fd0 tx=0x7f1d5800b260 comp rx=0 tx=0).stop 2024-04-25T13:38:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 shutdown_connections 2024-04-25T13:38:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d480afaf0 0x7f1d480b2000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d60108900 0x7f1d601a1720 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d60103c00 0x7f1d6019a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 --2- 172.21.15.27:0/1455865491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d601029c0 0x7f1d6019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 >> 172.21.15.27:0/1455865491 conn(0x7f1d600fdbd0 msgr2=0x7f1d60109e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 shutdown_connections 2024-04-25T13:38:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.121+0000 7f1d64ab6640 1 -- 172.21.15.27:0/1455865491 wait complete. 2024-04-25T13:38:42.133 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T13:38:42.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T13:38:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 -- 172.21.15.27:0/1051684431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40106c60 msgr2=0x7fac401090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 --2- 172.21.15.27:0/1051684431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40106c60 0x7fac401090e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009900 tx=0x7fac3c02a410 comp rx=0 tx=0).stop 2024-04-25T13:38:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 -- 172.21.15.27:0/1051684431 shutdown_connections 2024-04-25T13:38:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 --2- 172.21.15.27:0/1051684431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac40109770 0x7fac4010bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 --2- 172.21.15.27:0/1051684431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40106c60 0x7fac401090e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 --2- 172.21.15.27:0/1051684431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac40104270 0x7fac40106690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 -- 172.21.15.27:0/1051684431 >> 172.21.15.27:0/1051684431 conn(0x7fac400fdbd0 msgr2=0x7fac40100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 -- 172.21.15.27:0/1051684431 shutdown_connections 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 -- 172.21.15.27:0/1051684431 wait complete. 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 Processor -- start 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.260+0000 7fac48061640 1 -- start start 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac48061640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac40104270 0x7fac40103ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac48061640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac40106c60 0x7fac40102180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac48061640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40109770 0x7fac401026c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac48061640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac4010f5e0 con 0x7fac40109770 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac48061640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fac4010f460 con 0x7fac40104270 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac48061640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fac4010f760 con 0x7fac40106c60 2024-04-25T13:38:42.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac465fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40109770 0x7fac401026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac455fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac40106c60 0x7fac40102180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac465fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40109770 0x7fac401026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54646/0 (socket says 172.21.15.27:54646) 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac465fe640 1 -- 172.21.15.27:0/2242011457 learned_addr learned my addr 172.21.15.27:0/2242011457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac45dfd640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac40104270 0x7fac40103ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac455fc640 1 -- 172.21.15.27:0/2242011457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac40104270 msgr2=0x7fac40103ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac455fc640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac40104270 0x7fac40103ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac455fc640 1 -- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40109770 msgr2=0x7fac401026c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac455fc640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40109770 0x7fac401026c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac455fc640 1 -- 172.21.15.27:0/2242011457 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac40102df0 con 0x7fac40106c60 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac465fe640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40109770 0x7fac401026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:38:42.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.261+0000 7fac455fc640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac40106c60 0x7fac40102180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fac3c006010 tx=0x7fac3c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.262+0000 7fac36ffd640 1 -- 172.21.15.27:0/2242011457 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c03f070 con 0x7fac40106c60 2024-04-25T13:38:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.262+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac401a4630 con 0x7fac40106c60 2024-04-25T13:38:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.262+0000 7fac36ffd640 1 -- 172.21.15.27:0/2242011457 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c003ca0 con 0x7fac40106c60 2024-04-25T13:38:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.262+0000 7fac36ffd640 1 -- 172.21.15.27:0/2242011457 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c035070 con 0x7fac40106c60 2024-04-25T13:38:42.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.262+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac401a4a90 con 0x7fac40106c60 2024-04-25T13:38:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.263+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac4006c2a0 con 0x7fac40106c60 2024-04-25T13:38:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.265+0000 7fac36ffd640 1 -- 172.21.15.27:0/2242011457 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fac3c0037f0 con 0x7fac40106c60 2024-04-25T13:38:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.265+0000 7fac36ffd640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac2c0afaf0 0x7fac2c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.265+0000 7fac45dfd640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac2c0afaf0 0x7fac2c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.265+0000 7fac36ffd640 1 -- 172.21.15.27:0/2242011457 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fac3c0feb60 con 0x7fac40106c60 2024-04-25T13:38:42.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.266+0000 7fac45dfd640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac2c0afaf0 0x7fac2c0b2000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fac4006a7f0 tx=0x7fac3000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:42.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.271+0000 7fac36ffd640 1 -- 172.21.15.27:0/2242011457 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac3c0c7590 con 0x7fac40106c60 2024-04-25T13:38:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:42 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/206352020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:42.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:42.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:42.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:42 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:42.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:42 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/381414683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:42.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:42 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1455865491' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T13:38:42.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:42.465+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fac401a50a0 con 0x7fac40106c60 2024-04-25T13:38:42.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:42 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/206352020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:42.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:42.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:42.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:42.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:42 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:42.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:42 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/381414683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:42.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:42 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1455865491' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T13:38:42.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:42 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/206352020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:42 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:38:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:42 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/381414683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:38:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:42 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1455865491' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T13:38:43.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.331+0000 7fac36ffd640 1 -- 172.21.15.27:0/2242011457 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7fac3c0cb890 con 0x7fac40106c60 2024-04-25T13:38:43.332 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-04-25T13:38:43.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.336+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac2c0afaf0 msgr2=0x7fac2c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:43.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.336+0000 7fac48061640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac2c0afaf0 0x7fac2c0b2000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fac4006a7f0 tx=0x7fac3000b390 comp rx=0 tx=0).stop 2024-04-25T13:38:43.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.336+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac40106c60 msgr2=0x7fac40102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:43.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.336+0000 7fac48061640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac40106c60 0x7fac40102180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fac3c006010 tx=0x7fac3c004270 comp rx=0 tx=0).stop 2024-04-25T13:38:43.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.337+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 shutdown_connections 2024-04-25T13:38:43.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.337+0000 7fac48061640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac2c0afaf0 0x7fac2c0b2000 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.337+0000 7fac48061640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac40109770 0x7fac401026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.337+0000 7fac48061640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac40106c60 0x7fac40102180 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.337+0000 7fac48061640 1 --2- 172.21.15.27:0/2242011457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac40104270 0x7fac40103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.337+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 >> 172.21.15.27:0/2242011457 conn(0x7fac400fdbd0 msgr2=0x7fac40104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:43.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.337+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 shutdown_connections 2024-04-25T13:38:43.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.337+0000 7fac48061640 1 -- 172.21.15.27:0/2242011457 wait complete. 2024-04-25T13:38:43.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T13:38:43.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 -- 172.21.15.27:0/3726640475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 msgr2=0x7ff5ec1033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:43.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3726640475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 0x7ff5ec1033a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009900 tx=0x7ff5dc02a410 comp rx=0 tx=0).stop 2024-04-25T13:38:43.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 -- 172.21.15.27:0/3726640475 shutdown_connections 2024-04-25T13:38:43.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3726640475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5ec104450 0x7ff5ec10ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3726640475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec103970 0x7ff5ec103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3726640475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 0x7ff5ec1033a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 -- 172.21.15.27:0/3726640475 >> 172.21.15.27:0/3726640475 conn(0x7ff5ec0fd330 msgr2=0x7ff5ec0ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:43.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 -- 172.21.15.27:0/3726640475 shutdown_connections 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.479+0000 7ff5f128c640 1 -- 172.21.15.27:0/3726640475 wait complete. 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5f128c640 1 Processor -- start 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5f128c640 1 -- start start 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5f128c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 0x7ff5ec102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5f128c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec103970 0x7ff5ec19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5f128c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5ec104450 0x7ff5ec1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5f128c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec10e500 con 0x7ff5ec102f90 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5f128c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec10e380 con 0x7ff5ec104450 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5f128c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec10e680 con 0x7ff5ec103970 2024-04-25T13:38:43.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 0x7ff5ec102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 0x7ff5ec102230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54666/0 (socket says 172.21.15.27:54666) 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5eaffd640 1 -- 172.21.15.27:0/3854744711 learned_addr learned my addr 172.21.15.27:0/3854744711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.480+0000 7ff5eb7fe640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5ec104450 0x7ff5ec1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5ea7fc640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec103970 0x7ff5ec19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5eaffd640 1 -- 172.21.15.27:0/3854744711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec103970 msgr2=0x7ff5ec19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5eaffd640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec103970 0x7ff5ec19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5eaffd640 1 -- 172.21.15.27:0/3854744711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5ec104450 msgr2=0x7ff5ec1a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5eaffd640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5ec104450 0x7ff5ec1a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5eaffd640 1 -- 172.21.15.27:0/3854744711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec1a1d80 con 0x7ff5ec102f90 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5eaffd640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 0x7ff5ec102230 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc006010 tx=0x7ff5dc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5d3fff640 1 -- 172.21.15.27:0/3854744711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc03f030 con 0x7ff5ec102f90 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5d3fff640 1 -- 172.21.15.27:0/3854744711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5dc031910 con 0x7ff5ec102f90 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ec1a2010 con 0x7ff5ec102f90 2024-04-25T13:38:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5d3fff640 1 -- 172.21.15.27:0/3854744711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc035070 con 0x7ff5ec102f90 2024-04-25T13:38:43.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.481+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5ec1a2500 con 0x7ff5ec102f90 2024-04-25T13:38:43.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.482+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5ec107930 con 0x7ff5ec102f90 2024-04-25T13:38:43.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.484+0000 7ff5d3fff640 1 -- 172.21.15.27:0/3854744711 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff5dc042030 con 0x7ff5ec102f90 2024-04-25T13:38:43.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.484+0000 7ff5d3fff640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff5d40afaf0 0x7ff5d40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:43.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.484+0000 7ff5ea7fc640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff5d40afaf0 0x7ff5d40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:43.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.484+0000 7ff5d3fff640 1 -- 172.21.15.27:0/3854744711 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7497+0+0 (secure 0 0 0) 0x7ff5dc0fec80 con 0x7ff5ec102f90 2024-04-25T13:38:43.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.485+0000 7ff5ea7fc640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff5d40afaf0 0x7ff5d40b2000 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec101bd0 tx=0x7ff5e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:43.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.489+0000 7ff5d3fff640 1 -- 172.21.15.27:0/3854744711 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5dc0c7b90 con 0x7ff5ec102f90 2024-04-25T13:38:43.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:43 smithi045 ceph-mon[32731]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:43.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:43 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2242011457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T13:38:43.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:43 smithi045 ceph-mon[32731]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T13:38:43.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:43 smithi027 ceph-mon[29140]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:43.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:43 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2242011457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T13:38:43.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:43 smithi027 ceph-mon[29140]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T13:38:43.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:43.679+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff5ec0637a0 con 0x7ff5ec102f90 2024-04-25T13:38:43.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:43 smithi122 ceph-mon[32843]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:43.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:43 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2242011457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T13:38:43.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:43 smithi122 ceph-mon[32843]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T13:38:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.329+0000 7ff5d3fff640 1 -- 172.21.15.27:0/3854744711 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v74) ==== 170+0+0 (secure 0 0 0) 0x7ff5dc0c7490 con 0x7ff5ec102f90 2024-04-25T13:38:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-04-25T13:38:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff5d40afaf0 msgr2=0x7ff5d40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff5d40afaf0 0x7ff5d40b2000 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec101bd0 tx=0x7ff5e0005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 msgr2=0x7ff5ec102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 0x7ff5ec102230 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc006010 tx=0x7ff5dc004270 comp rx=0 tx=0).stop 2024-04-25T13:38:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 shutdown_connections 2024-04-25T13:38:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff5d40afaf0 0x7ff5d40b2000 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff5ec104450 0x7ff5ec1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5ec103970 0x7ff5ec19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 --2- 172.21.15.27:0/3854744711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5ec102f90 0x7ff5ec102230 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.333+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 >> 172.21.15.27:0/3854744711 conn(0x7ff5ec0fd330 msgr2=0x7ff5ec104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.334+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 shutdown_connections 2024-04-25T13:38:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.334+0000 7ff5f128c640 1 -- 172.21.15.27:0/3854744711 wait complete. 2024-04-25T13:38:44.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T13:38:44.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 -- 172.21.15.27:0/3904264574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884103930 msgr2=0x7f4884103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:44.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 --2- 172.21.15.27:0/3904264574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884103930 0x7f4884103de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4870009900 tx=0x7f487002a410 comp rx=0 tx=0).stop 2024-04-25T13:38:44.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 -- 172.21.15.27:0/3904264574 shutdown_connections 2024-04-25T13:38:44.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 --2- 172.21.15.27:0/3904264574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4884104470 0x7f488410abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 --2- 172.21.15.27:0/3904264574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884103930 0x7f4884103de0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 --2- 172.21.15.27:0/3904264574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4884102f50 0x7f4884103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 -- 172.21.15.27:0/3904264574 >> 172.21.15.27:0/3904264574 conn(0x7f48840fd350 msgr2=0x7f48840ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 -- 172.21.15.27:0/3904264574 shutdown_connections 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.478+0000 7f488b0af640 1 -- 172.21.15.27:0/3904264574 wait complete. 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488b0af640 1 Processor -- start 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488b0af640 1 -- start start 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488b0af640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4884102f50 0x7f48841021e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488b0af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4884103930 0x7f488419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488b0af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884104470 0x7f48841a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488b0af640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f488410e540 con 0x7f4884104470 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488b0af640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f488410e3c0 con 0x7f4884102f50 2024-04-25T13:38:44.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488b0af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f488410e6c0 con 0x7f4884103930 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488964c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884104470 0x7f48841a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488964c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884104470 0x7f48841a1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55844/0 (socket says 172.21.15.27:55844) 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488964c640 1 -- 172.21.15.27:0/81270057 learned_addr learned my addr 172.21.15.27:0/81270057 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f4883fff640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4884103930 0x7f488419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f4888e4b640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4884102f50 0x7f48841021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f4883fff640 1 -- 172.21.15.27:0/81270057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4884102f50 msgr2=0x7f48841021e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f4883fff640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4884102f50 0x7f48841021e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f4883fff640 1 -- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884104470 msgr2=0x7f48841a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f4883fff640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884104470 0x7f48841a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f4883fff640 1 -- 172.21.15.27:0/81270057 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48841a1d00 con 0x7f4884103930 2024-04-25T13:38:44.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f488964c640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884104470 0x7f48841a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:38:44.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.479+0000 7f4883fff640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4884103930 0x7f488419f020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4870006010 tx=0x7f4870004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:44.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.480+0000 7f4881ffb640 1 -- 172.21.15.27:0/81270057 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487003f070 con 0x7f4884103930 2024-04-25T13:38:44.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.480+0000 7f4881ffb640 1 -- 172.21.15.27:0/81270057 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4870003ca0 con 0x7f4884103930 2024-04-25T13:38:44.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.480+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48841a1f90 con 0x7f4884103930 2024-04-25T13:38:44.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.480+0000 7f4881ffb640 1 -- 172.21.15.27:0/81270057 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870035070 con 0x7f4884103930 2024-04-25T13:38:44.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.480+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48841a2390 con 0x7f4884103930 2024-04-25T13:38:44.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.481+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48841078f0 con 0x7f4884103930 2024-04-25T13:38:44.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.482+0000 7f4881ffb640 1 -- 172.21.15.27:0/81270057 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f487003b4f0 con 0x7f4884103930 2024-04-25T13:38:44.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.483+0000 7f4881ffb640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48740afaf0 0x7f48740b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:44.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.483+0000 7f4888e4b640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48740afaf0 0x7f48740b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:44.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.483+0000 7f4881ffb640 1 -- 172.21.15.27:0/81270057 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7879+0+0 (secure 0 0 0) 0x7f48700ff560 con 0x7f4884103930 2024-04-25T13:38:44.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.484+0000 7f4888e4b640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48740afaf0 0x7f48740b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f488406b500 tx=0x7f4878005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:44.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.487+0000 7f4881ffb640 1 -- 172.21.15.27:0/81270057 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48700c7ca0 con 0x7f4884103930 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:44.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:44.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:44 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3854744711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T13:38:44.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:44.714+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f488402db50 con 0x7f4884103930 2024-04-25T13:38:44.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T13:38:44.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T13:38:44.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:44.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:44.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:44 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3854744711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:38:44.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:38:44.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:38:44.829 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:44 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3854744711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T13:38:45.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.360+0000 7f4881ffb640 1 -- 172.21.15.27:0/81270057 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f48700cbfa0 con 0x7f4884103930 2024-04-25T13:38:45.360 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.364+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48740afaf0 msgr2=0x7f48740b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.364+0000 7f488b0af640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48740afaf0 0x7f48740b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f488406b500 tx=0x7f4878005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.364+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4884103930 msgr2=0x7f488419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.364+0000 7f488b0af640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4884103930 0x7f488419f020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4870006010 tx=0x7f4870004270 comp rx=0 tx=0).stop 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.365+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 shutdown_connections 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.365+0000 7f488b0af640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f48740afaf0 0x7f48740b2000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.365+0000 7f488b0af640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4884104470 0x7f48841a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.365+0000 7f488b0af640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4884103930 0x7f488419f020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.365+0000 7f488b0af640 1 --2- 172.21.15.27:0/81270057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4884102f50 0x7f48841021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.365+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 >> 172.21.15.27:0/81270057 conn(0x7f48840fd350 msgr2=0x7f4884104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:45.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.365+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 shutdown_connections 2024-04-25T13:38:45.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.365+0000 7f488b0af640 1 -- 172.21.15.27:0/81270057 wait complete. 2024-04-25T13:38:45.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:38:45.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 -- 172.21.15.27:0/199591191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c104d10 msgr2=0x7f957c105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 --2- 172.21.15.27:0/199591191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c104d10 0x7f957c105120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9570009900 tx=0x7f957002a410 comp rx=0 tx=0).stop 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 -- 172.21.15.27:0/199591191 shutdown_connections 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 --2- 172.21.15.27:0/199591191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f957c109a10 0x7f957c10be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 --2- 172.21.15.27:0/199591191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c104d10 0x7f957c105120 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 --2- 172.21.15.27:0/199591191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c103ad0 0x7f957c1047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 -- 172.21.15.27:0/199591191 >> 172.21.15.27:0/199591191 conn(0x7f957c0fecc0 msgr2=0x7f957c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 -- 172.21.15.27:0/199591191 shutdown_connections 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 -- 172.21.15.27:0/199591191 wait complete. 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.505+0000 7f9582f34640 1 Processor -- start 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9582f34640 1 -- start start 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9582f34640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c103ad0 0x7f957c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9582f34640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f957c104d10 0x7f957c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9582f34640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c109a10 0x7f957c1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9582f34640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f957c10f600 con 0x7f957c103ad0 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9582f34640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f957c10f480 con 0x7f957c109a10 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9582f34640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f957c10f780 con 0x7f957c104d10 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c103ad0 0x7f957c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c103ad0 0x7f957c19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55866/0 (socket says 172.21.15.27:55866) 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 -- 172.21.15.27:0/773043075 learned_addr learned my addr 172.21.15.27:0/773043075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f95814d1640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c109a10 0x7f957c1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f957bfff640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f957c104d10 0x7f957c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 -- 172.21.15.27:0/773043075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f957c104d10 msgr2=0x7f957c19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f957c104d10 0x7f957c19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 -- 172.21.15.27:0/773043075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c109a10 msgr2=0x7f957c1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c109a10 0x7f957c1a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 -- 172.21.15.27:0/773043075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957c1a1d40 con 0x7f957c103ad0 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f957bfff640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f957c104d10 0x7f957c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f95814d1640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c109a10 0x7f957c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:45.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9580cd0640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c103ad0 0x7f957c19a200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f957400ea50 tx=0x7f957400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.506+0000 7f9579ffb640 1 -- 172.21.15.27:0/773043075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957400cdc0 con 0x7f957c103ad0 2024-04-25T13:38:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.507+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957c1a1fd0 con 0x7f957c103ad0 2024-04-25T13:38:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.507+0000 7f9579ffb640 1 -- 172.21.15.27:0/773043075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9574011920 con 0x7f957c103ad0 2024-04-25T13:38:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.507+0000 7f9579ffb640 1 -- 172.21.15.27:0/773043075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957401aae0 con 0x7f957c103ad0 2024-04-25T13:38:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.507+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f957c1a2490 con 0x7f957c103ad0 2024-04-25T13:38:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.508+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957c108820 con 0x7f957c103ad0 2024-04-25T13:38:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.509+0000 7f9579ffb640 1 -- 172.21.15.27:0/773043075 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f957401ac80 con 0x7f957c103ad0 2024-04-25T13:38:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.510+0000 7f9579ffb640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95640afaf0 0x7f95640b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.510+0000 7f957bfff640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95640afaf0 0x7f95640b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.510+0000 7f9579ffb640 1 -- 172.21.15.27:0/773043075 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f95740df320 con 0x7f957c103ad0 2024-04-25T13:38:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.511+0000 7f957bfff640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95640afaf0 0x7f95640b2000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9570006010 tx=0x7f9570000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.514+0000 7f9579ffb640 1 -- 172.21.15.27:0/773043075 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9574055ff0 con 0x7f957c103ad0 2024-04-25T13:38:45.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:45 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3854744711' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T13:38:45.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:45 smithi027 ceph-mon[29140]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T13:38:45.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:45 smithi027 ceph-mon[29140]: pgmap v281: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:45.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:45 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/81270057' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T13:38:45.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:45 smithi027 ceph-mon[29140]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T13:38:45.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:45 smithi027 ceph-mon[29140]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T13:38:45.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:45 smithi027 ceph-mon[29140]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T13:38:45.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:45 smithi027 ceph-mon[29140]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T13:38:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.703+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f957c0637a0 con 0x7f957c103ad0 2024-04-25T13:38:45.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.704+0000 7f9579ffb640 1 -- 172.21.15.27:0/773043075 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+22186 (secure 0 0 0) 0x7f9574016030 con 0x7f957c103ad0 2024-04-25T13:38:45.705 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:45.705 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:38:44.716185+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95640afaf0 msgr2=0x7f95640b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95640afaf0 0x7f95640b2000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9570006010 tx=0x7f9570000bc0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c103ad0 msgr2=0x7f957c19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c103ad0 0x7f957c19a200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f957400ea50 tx=0x7f957400ef10 comp rx=0 tx=0).stop 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 shutdown_connections 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95640afaf0 0x7f95640b2000 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c109a10 0x7f957c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f957c104d10 0x7f957c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 --2- 172.21.15.27:0/773043075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f957c103ad0 0x7f957c19a200 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 >> 172.21.15.27:0/773043075 conn(0x7f957c0fecc0 msgr2=0x7f957c1059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 shutdown_connections 2024-04-25T13:38:45.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.706+0000 7f9582f34640 1 -- 172.21.15.27:0/773043075 wait complete. 2024-04-25T13:38:45.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:38:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:45 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3854744711' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T13:38:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:45 smithi122 ceph-mon[32843]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T13:38:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:45 smithi122 ceph-mon[32843]: pgmap v281: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:45 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/81270057' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T13:38:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:45 smithi122 ceph-mon[32843]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T13:38:45.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:45 smithi122 ceph-mon[32843]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T13:38:45.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:45 smithi122 ceph-mon[32843]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T13:38:45.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:45 smithi122 ceph-mon[32843]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T13:38:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:45 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3854744711' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T13:38:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:45 smithi045 ceph-mon[32731]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T13:38:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:45 smithi045 ceph-mon[32731]: pgmap v281: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T13:38:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:45 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/81270057' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T13:38:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:45 smithi045 ceph-mon[32731]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T13:38:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:45 smithi045 ceph-mon[32731]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T13:38:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:45 smithi045 ceph-mon[32731]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T13:38:45.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:45 smithi045 ceph-mon[32731]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T13:38:45.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/2903701294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 msgr2=0x7f6bcc102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/2903701294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 0x7f6bcc102250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009900 tx=0x7f6bc002a410 comp rx=0 tx=0).stop 2024-04-25T13:38:45.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/2903701294 shutdown_connections 2024-04-25T13:38:45.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/2903701294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bcc109790 0x7f6bcc10bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/2903701294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bcc106c20 0x7f6bcc109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/2903701294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 0x7f6bcc102250 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/2903701294 >> 172.21.15.27:0/2903701294 conn(0x7f6bcc0fd370 msgr2=0x7f6bcc0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:45.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/2903701294 shutdown_connections 2024-04-25T13:38:45.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.830+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/2903701294 wait complete. 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bd2a3c640 1 Processor -- start 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bd2a3c640 1 -- start start 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bd2a3c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 0x7f6bcc19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bd2a3c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bcc106c20 0x7f6bcc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bd2a3c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bcc109790 0x7f6bcc1a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bd2a3c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc10f5f0 con 0x7f6bcc101e40 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bd2a3c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc10f470 con 0x7f6bcc109790 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bd2a3c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc10f770 con 0x7f6bcc106c20 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bcbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 0x7f6bcc19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bcbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 0x7f6bcc19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55874/0 (socket says 172.21.15.27:55874) 2024-04-25T13:38:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.831+0000 7f6bcbfff640 1 -- 172.21.15.27:0/643803276 learned_addr learned my addr 172.21.15.27:0/643803276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.832+0000 7f6bcb7fe640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bcc106c20 0x7f6bcc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.832+0000 7f6bd0fd9640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bcc109790 0x7f6bcc1a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.832+0000 7f6bcbfff640 1 -- 172.21.15.27:0/643803276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bcc106c20 msgr2=0x7f6bcc19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.832+0000 7f6bcbfff640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bcc106c20 0x7f6bcc19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.832+0000 7f6bcbfff640 1 -- 172.21.15.27:0/643803276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bcc109790 msgr2=0x7f6bcc1a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.832+0000 7f6bcbfff640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bcc109790 0x7f6bcc1a16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.832+0000 7f6bcbfff640 1 -- 172.21.15.27:0/643803276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc1a1d30 con 0x7f6bcc101e40 2024-04-25T13:38:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.832+0000 7f6bcbfff640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 0x7f6bcc19a1f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0006010 tx=0x7f6bc0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:45.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.834+0000 7f6bc97fa640 1 -- 172.21.15.27:0/643803276 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc003f030 con 0x7f6bcc101e40 2024-04-25T13:38:45.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.834+0000 7f6bc97fa640 1 -- 172.21.15.27:0/643803276 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bc0003aa0 con 0x7f6bcc101e40 2024-04-25T13:38:45.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.834+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bcc1a1fc0 con 0x7f6bcc101e40 2024-04-25T13:38:45.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.834+0000 7f6bc97fa640 1 -- 172.21.15.27:0/643803276 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0035070 con 0x7f6bcc101e40 2024-04-25T13:38:45.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.835+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bcc1a23d0 con 0x7f6bcc101e40 2024-04-25T13:38:45.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.836+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bcc0fee90 con 0x7f6bcc101e40 2024-04-25T13:38:45.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.841+0000 7f6bc97fa640 1 -- 172.21.15.27:0/643803276 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6bc0042030 con 0x7f6bcc101e40 2024-04-25T13:38:45.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.841+0000 7f6bc97fa640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bb40afaf0 0x7f6bb40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:45.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.841+0000 7f6bc97fa640 1 -- 172.21.15.27:0/643803276 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f6bc00ff610 con 0x7f6bcc101e40 2024-04-25T13:38:45.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.841+0000 7f6bcb7fe640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bb40afaf0 0x7f6bb40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:45.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.842+0000 7f6bcb7fe640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bb40afaf0 0x7f6bb40b2000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc005fd0 tx=0x7f6bbc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:45.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:45.845+0000 7f6bc97fa640 1 -- 172.21.15.27:0/643803276 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bc00c8340 con 0x7f6bcc101e40 2024-04-25T13:38:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.037+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bcc02dd20 con 0x7f6bcc101e40 2024-04-25T13:38:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.039+0000 7f6bc97fa640 1 -- 172.21.15.27:0/643803276 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+22186 (secure 0 0 0) 0x7f6bc00c7c40 con 0x7f6bcc101e40 2024-04-25T13:38:46.039 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:46.039 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:38:44.716185+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:38:46.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bb40afaf0 msgr2=0x7f6bb40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bb40afaf0 0x7f6bb40b2000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc005fd0 tx=0x7f6bbc005850 comp rx=0 tx=0).stop 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 msgr2=0x7f6bcc19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 0x7f6bcc19a1f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0006010 tx=0x7f6bc0004270 comp rx=0 tx=0).stop 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 shutdown_connections 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6bb40afaf0 0x7f6bb40b2000 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6bcc109790 0x7f6bcc1a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6bcc106c20 0x7f6bcc19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 --2- 172.21.15.27:0/643803276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bcc101e40 0x7f6bcc19a1f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 >> 172.21.15.27:0/643803276 conn(0x7f6bcc0fd370 msgr2=0x7f6bcc107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 shutdown_connections 2024-04-25T13:38:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.041+0000 7f6bd2a3c640 1 -- 172.21.15.27:0/643803276 wait complete. 2024-04-25T13:38:46.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T13:38:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 -- 172.21.15.27:0/2347238398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0103b00 msgr2=0x7ff6a0104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 --2- 172.21.15.27:0/2347238398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0103b00 0x7ff6a0104800 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009900 tx=0x7ff69c02a410 comp rx=0 tx=0).stop 2024-04-25T13:38:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 -- 172.21.15.27:0/2347238398 shutdown_connections 2024-04-25T13:38:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 --2- 172.21.15.27:0/2347238398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0109a40 0x7ff6a010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 --2- 172.21.15.27:0/2347238398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a0104d40 0x7ff6a0105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 --2- 172.21.15.27:0/2347238398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0103b00 0x7ff6a0104800 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 -- 172.21.15.27:0/2347238398 >> 172.21.15.27:0/2347238398 conn(0x7ff6a00fed10 msgr2=0x7ff6a01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 -- 172.21.15.27:0/2347238398 shutdown_connections 2024-04-25T13:38:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.168+0000 7ff6a7267640 1 -- 172.21.15.27:0/2347238398 wait complete. 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a7267640 1 Processor -- start 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a7267640 1 -- start start 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a7267640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0103b00 0x7ff6a019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a7267640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a0104d40 0x7ff6a019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a7267640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0109a40 0x7ff6a01a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a7267640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a010f650 con 0x7ff6a0109a40 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a7267640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a010f4d0 con 0x7ff6a0103b00 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a7267640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a010f7d0 con 0x7ff6a0104d40 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5804640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0109a40 0x7ff6a01a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5804640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0109a40 0x7ff6a01a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55896/0 (socket says 172.21.15.27:55896) 2024-04-25T13:38:46.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5804640 1 -- 172.21.15.27:0/3602801310 learned_addr learned my addr 172.21.15.27:0/3602801310 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a4802640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a0104d40 0x7ff6a019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5003640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0103b00 0x7ff6a019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5804640 1 -- 172.21.15.27:0/3602801310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a0104d40 msgr2=0x7ff6a019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5804640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a0104d40 0x7ff6a019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5804640 1 -- 172.21.15.27:0/3602801310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0103b00 msgr2=0x7ff6a019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5804640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0103b00 0x7ff6a019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.169+0000 7ff6a5804640 1 -- 172.21.15.27:0/3602801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a01a1dd0 con 0x7ff6a0109a40 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.170+0000 7ff6a5804640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0109a40 0x7ff6a01a1760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff69800eb10 tx=0x7ff69800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.170+0000 7ff6967fc640 1 -- 172.21.15.27:0/3602801310 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff698004b20 con 0x7ff6a0109a40 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.170+0000 7ff6967fc640 1 -- 172.21.15.27:0/3602801310 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff698004cc0 con 0x7ff6a0109a40 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.170+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a01a20c0 con 0x7ff6a0109a40 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.170+0000 7ff6967fc640 1 -- 172.21.15.27:0/3602801310 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff698010670 con 0x7ff6a0109a40 2024-04-25T13:38:46.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.170+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a01a25c0 con 0x7ff6a0109a40 2024-04-25T13:38:46.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.171+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a0108850 con 0x7ff6a0109a40 2024-04-25T13:38:46.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.172+0000 7ff6967fc640 1 -- 172.21.15.27:0/3602801310 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff698005000 con 0x7ff6a0109a40 2024-04-25T13:38:46.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.173+0000 7ff6967fc640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff68c0afaf0 0x7ff68c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:46.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.173+0000 7ff6a5003640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff68c0afaf0 0x7ff68c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:46.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.173+0000 7ff6967fc640 1 -- 172.21.15.27:0/3602801310 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7ff6980df300 con 0x7ff6a0109a40 2024-04-25T13:38:46.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.173+0000 7ff6a5003640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff68c0afaf0 0x7ff68c0b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff69c0096c0 tx=0x7ff69c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:46.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.177+0000 7ff6967fc640 1 -- 172.21.15.27:0/3602801310 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff698053ff0 con 0x7ff6a0109a40 2024-04-25T13:38:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:46.410+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff6a002db50 con 0x7ff6a0109a40 2024-04-25T13:38:46.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:46 smithi027 ceph-mon[29140]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T13:38:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:46 smithi027 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T13:38:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:46 smithi027 ceph-mon[29140]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T13:38:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:46 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/773043075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:46 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] up:active 2024-04-25T13:38:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:46 smithi027 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:46 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/643803276' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:46.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:46 smithi027 ceph-mon[29140]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T13:38:46.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:46 smithi122 ceph-mon[32843]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T13:38:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:46 smithi122 ceph-mon[32843]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T13:38:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:46 smithi122 ceph-mon[32843]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T13:38:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:46 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/773043075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:46 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] up:active 2024-04-25T13:38:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:46 smithi122 ceph-mon[32843]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:46 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/643803276' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:46.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:46 smithi122 ceph-mon[32843]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T13:38:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:46 smithi045 ceph-mon[32731]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T13:38:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:46 smithi045 ceph-mon[32731]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T13:38:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:46 smithi045 ceph-mon[32731]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T13:38:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:46 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/773043075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:46 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] up:active 2024-04-25T13:38:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:46 smithi045 ceph-mon[32731]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:46 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/643803276' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:46.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:46 smithi045 ceph-mon[32731]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T13:38:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.373+0000 7ff6967fc640 1 -- 172.21.15.27:0/3602801310 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7ff6980a8030 con 0x7ff6a0109a40 2024-04-25T13:38:47.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff68c0afaf0 msgr2=0x7ff68c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff68c0afaf0 0x7ff68c0b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff69c0096c0 tx=0x7ff69c005fd0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0109a40 msgr2=0x7ff6a01a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0109a40 0x7ff6a01a1760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff69800eb10 tx=0x7ff69800c2d0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 shutdown_connections 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff68c0afaf0 0x7ff68c0b2000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6a0109a40 0x7ff6a01a1760 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6a0104d40 0x7ff6a019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 --2- 172.21.15.27:0/3602801310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0103b00 0x7ff6a019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 >> 172.21.15.27:0/3602801310 conn(0x7ff6a00fed10 msgr2=0x7ff6a0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 shutdown_connections 2024-04-25T13:38:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.376+0000 7ff6a7267640 1 -- 172.21.15.27:0/3602801310 wait complete. 2024-04-25T13:38:47.391 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T13:38:47.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 -- 172.21.15.27:0/1275066007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 msgr2=0x7f40a0104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 --2- 172.21.15.27:0/1275066007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 0x7f40a0104040 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4090009900 tx=0x7f409002a410 comp rx=0 tx=0).stop 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 -- 172.21.15.27:0/1275066007 shutdown_connections 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 --2- 172.21.15.27:0/1275066007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40a0108930 0x7f40a010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 --2- 172.21.15.27:0/1275066007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 0x7f40a0104040 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 --2- 172.21.15.27:0/1275066007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40a01029f0 0x7f40a01036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 -- 172.21.15.27:0/1275066007 >> 172.21.15.27:0/1275066007 conn(0x7f40a00fdbc0 msgr2=0x7f40a0100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 -- 172.21.15.27:0/1275066007 shutdown_connections 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.513+0000 7f40a7f55640 1 -- 172.21.15.27:0/1275066007 wait complete. 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a7f55640 1 Processor -- start 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a7f55640 1 -- start start 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a7f55640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40a01029f0 0x7f40a019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:47.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a7f55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 0x7f40a019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a7f55640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40a0108930 0x7f40a01a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a7f55640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f40a010e510 con 0x7f40a0103c30 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a7f55640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f40a010e390 con 0x7f40a01029f0 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a7f55640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f40a010e690 con 0x7f40a0108930 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a54f0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 0x7f40a019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a54f0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 0x7f40a019a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55906/0 (socket says 172.21.15.27:55906) 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.514+0000 7f40a54f0640 1 -- 172.21.15.27:0/3184967939 learned_addr learned my addr 172.21.15.27:0/3184967939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a5cf1640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40a01029f0 0x7f40a019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a54f0640 1 -- 172.21.15.27:0/3184967939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40a0108930 msgr2=0x7f40a01a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a64f2640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40a0108930 0x7f40a01a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a54f0640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40a0108930 0x7f40a01a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a54f0640 1 -- 172.21.15.27:0/3184967939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40a01029f0 msgr2=0x7f40a019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a54f0640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40a01029f0 0x7f40a019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a54f0640 1 -- 172.21.15.27:0/3184967939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a01a1cf0 con 0x7f40a0103c30 2024-04-25T13:38:47.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a54f0640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 0x7f40a019a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4090006010 tx=0x7f40900043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:47.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f4096ffd640 1 -- 172.21.15.27:0/3184967939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409003f030 con 0x7f40a0103c30 2024-04-25T13:38:47.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f4096ffd640 1 -- 172.21.15.27:0/3184967939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4090003cb0 con 0x7f40a0103c30 2024-04-25T13:38:47.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a01a1f80 con 0x7f40a0103c30 2024-04-25T13:38:47.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f4096ffd640 1 -- 172.21.15.27:0/3184967939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090035070 con 0x7f40a0103c30 2024-04-25T13:38:47.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.515+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a01a23e0 con 0x7f40a0103c30 2024-04-25T13:38:47.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.516+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a01a2700 con 0x7f40a0103c30 2024-04-25T13:38:47.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.518+0000 7f4096ffd640 1 -- 172.21.15.27:0/3184967939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4090042030 con 0x7f40a0103c30 2024-04-25T13:38:47.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.518+0000 7f4096ffd640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f408c0afb70 0x7f408c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:47.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.519+0000 7f40a5cf1640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f408c0afb70 0x7f408c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:47.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.519+0000 7f4096ffd640 1 -- 172.21.15.27:0/3184967939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f40900ff880 con 0x7f40a0103c30 2024-04-25T13:38:47.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.519+0000 7f40a5cf1640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f408c0afb70 0x7f408c0b2080 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f409c0058c0 tx=0x7f409c005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:47.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.522+0000 7f4096ffd640 1 -- 172.21.15.27:0/3184967939 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f40900c85b0 con 0x7f40a0103c30 2024-04-25T13:38:47.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:47 smithi027 ceph-mon[29140]: pgmap v284: 97 pgs: 16 creating+peering, 32 unknown, 49 active+clean; 578 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-04-25T13:38:47.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:47 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3602801310' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T13:38:47.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:47 smithi027 ceph-mon[29140]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T13:38:47.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:47 smithi027 ceph-mon[29140]: Cluster is now healthy 2024-04-25T13:38:47.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:47.749+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f40a01a29b0 con 0x7f40a0103c30 2024-04-25T13:38:47.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:47 smithi122 ceph-mon[32843]: pgmap v284: 97 pgs: 16 creating+peering, 32 unknown, 49 active+clean; 578 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-04-25T13:38:47.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:47 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3602801310' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T13:38:47.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:47 smithi122 ceph-mon[32843]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T13:38:47.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:47 smithi122 ceph-mon[32843]: Cluster is now healthy 2024-04-25T13:38:47.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:47 smithi045 ceph-mon[32731]: pgmap v284: 97 pgs: 16 creating+peering, 32 unknown, 49 active+clean; 578 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-04-25T13:38:47.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:47 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3602801310' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T13:38:47.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:47 smithi045 ceph-mon[32731]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T13:38:47.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:47 smithi045 ceph-mon[32731]: Cluster is now healthy 2024-04-25T13:38:48.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.379+0000 7f4096ffd640 1 -- 172.21.15.27:0/3184967939 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7f40900c7eb0 con 0x7f40a0103c30 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.382+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f408c0afb70 msgr2=0x7f408c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.382+0000 7f40a7f55640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f408c0afb70 0x7f408c0b2080 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f409c0058c0 tx=0x7f409c005c80 comp rx=0 tx=0).stop 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.382+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 msgr2=0x7f40a019a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.382+0000 7f40a7f55640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 0x7f40a019a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4090006010 tx=0x7f40900043b0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.382+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 shutdown_connections 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.382+0000 7f40a7f55640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f408c0afb70 0x7f408c0b2080 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.383+0000 7f40a7f55640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40a0108930 0x7f40a01a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.383+0000 7f40a7f55640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40a0103c30 0x7f40a019a7a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.383+0000 7f40a7f55640 1 --2- 172.21.15.27:0/3184967939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f40a01029f0 0x7f40a019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.383+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 >> 172.21.15.27:0/3184967939 conn(0x7f40a00fdbc0 msgr2=0x7f40a0109e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.383+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 shutdown_connections 2024-04-25T13:38:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.383+0000 7f40a7f55640 1 -- 172.21.15.27:0/3184967939 wait complete. 2024-04-25T13:38:48.398 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T13:38:48.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 -- 172.21.15.27:0/70018044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 msgr2=0x7fab34109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/70018044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 0x7fab34109370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fab24009900 tx=0x7fab2402a410 comp rx=0 tx=0).stop 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 -- 172.21.15.27:0/70018044 shutdown_connections 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/70018044 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab341098b0 0x7fab3410bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/70018044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 0x7fab34109370 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/70018044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab341045f0 0x7fab34106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 -- 172.21.15.27:0/70018044 >> 172.21.15.27:0/70018044 conn(0x7fab340fdbf0 msgr2=0x7fab341000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 -- 172.21.15.27:0/70018044 shutdown_connections 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.519+0000 7fab3c0bf640 1 -- 172.21.15.27:0/70018044 wait complete. 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3c0bf640 1 Processor -- start 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3c0bf640 1 -- start start 2024-04-25T13:38:48.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3c0bf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab341045f0 0x7fab3419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3c0bf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 0x7fab3419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3c0bf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab341098b0 0x7fab341a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3c0bf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fab3410f610 con 0x7fab34106f50 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3c0bf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab3410f490 con 0x7fab341098b0 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3c0bf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fab3410f790 con 0x7fab341045f0 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3965a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 0x7fab3419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3965a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 0x7fab3419a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55934/0 (socket says 172.21.15.27:55934) 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab3965a640 1 -- 172.21.15.27:0/4167150567 learned_addr learned my addr 172.21.15.27:0/4167150567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.520+0000 7fab39e5b640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab341045f0 0x7fab3419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3a65c640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab341098b0 0x7fab341a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3965a640 1 -- 172.21.15.27:0/4167150567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab341045f0 msgr2=0x7fab3419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3965a640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab341045f0 0x7fab3419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3965a640 1 -- 172.21.15.27:0/4167150567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab341098b0 msgr2=0x7fab341a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3965a640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab341098b0 0x7fab341a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:48.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3965a640 1 -- 172.21.15.27:0/4167150567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab341a1cd0 con 0x7fab34106f50 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3a65c640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab341098b0 0x7fab341a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3965a640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 0x7fab3419a780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fab24006010 tx=0x7fab24031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab2affd640 1 -- 172.21.15.27:0/4167150567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2403f030 con 0x7fab34106f50 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab341a1f60 con 0x7fab34106f50 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab2affd640 1 -- 172.21.15.27:0/4167150567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab24003aa0 con 0x7fab34106f50 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab2affd640 1 -- 172.21.15.27:0/4167150567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24035070 con 0x7fab34106f50 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.521+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab341a2370 con 0x7fab34106f50 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.522+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab340ff710 con 0x7fab34106f50 2024-04-25T13:38:48.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.524+0000 7fab2affd640 1 -- 172.21.15.27:0/4167150567 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fab24042030 con 0x7fab34106f50 2024-04-25T13:38:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.525+0000 7fab2affd640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fab200afaf0 0x7fab200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.525+0000 7fab39e5b640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fab200afaf0 0x7fab200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.525+0000 7fab2affd640 1 -- 172.21.15.27:0/4167150567 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fab240ff740 con 0x7fab34106f50 2024-04-25T13:38:48.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.525+0000 7fab39e5b640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fab200afaf0 0x7fab200b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab340757f0 tx=0x7fab30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.529+0000 7fab2affd640 1 -- 172.21.15.27:0/4167150567 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab240c8470 con 0x7fab34106f50 2024-04-25T13:38:48.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:48 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3602801310' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T13:38:48.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:48 smithi027 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:48.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:48 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3184967939' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T13:38:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:48.755+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fab3402dbf0 con 0x7fab34106f50 2024-04-25T13:38:48.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:48 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3602801310' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T13:38:48.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:48 smithi122 ceph-mon[32843]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:48.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:48 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3184967939' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T13:38:48.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:48 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3602801310' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T13:38:48.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:48 smithi045 ceph-mon[32731]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:48.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:48 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3184967939' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T13:38:49.328 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:38:49 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b[63598]: 2024-04-25T13:38:49.005+0000 7fdcaef10700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T13:38:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.389+0000 7fab2affd640 1 -- 172.21.15.27:0/4167150567 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7fab240c7d70 con 0x7fab34106f50 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.391+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fab200afaf0 msgr2=0x7fab200b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.391+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fab200afaf0 0x7fab200b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab340757f0 tx=0x7fab30005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.391+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 msgr2=0x7fab3419a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.391+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 0x7fab3419a780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fab24006010 tx=0x7fab24031a50 comp rx=0 tx=0).stop 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.392+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 shutdown_connections 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.392+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fab200afaf0 0x7fab200b2000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.392+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab341098b0 0x7fab341a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.392+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab34106f50 0x7fab3419a780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.392+0000 7fab3c0bf640 1 --2- 172.21.15.27:0/4167150567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fab341045f0 0x7fab3419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.392+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 >> 172.21.15.27:0/4167150567 conn(0x7fab340fdbf0 msgr2=0x7fab341065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:49.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.392+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 shutdown_connections 2024-04-25T13:38:49.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.392+0000 7fab3c0bf640 1 -- 172.21.15.27:0/4167150567 wait complete. 2024-04-25T13:38:49.406 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T13:38:49.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T13:38:49.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 -- 172.21.15.27:0/486474299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 msgr2=0x7f242810be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 --2- 172.21.15.27:0/486474299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 0x7f242810be50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2420009ca0 tx=0x7f242002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 -- 172.21.15.27:0/486474299 shutdown_connections 2024-04-25T13:38:49.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 --2- 172.21.15.27:0/486474299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 0x7f242810be50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 --2- 172.21.15.27:0/486474299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2428104d30 0x7f2428105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 --2- 172.21.15.27:0/486474299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2428103af0 0x7f24281047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 -- 172.21.15.27:0/486474299 >> 172.21.15.27:0/486474299 conn(0x7f24280fece0 msgr2=0x7f2428101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 -- 172.21.15.27:0/486474299 shutdown_connections 2024-04-25T13:38:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 -- 172.21.15.27:0/486474299 wait complete. 2024-04-25T13:38:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.528+0000 7f242e013640 1 Processor -- start 2024-04-25T13:38:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f242e013640 1 -- start start 2024-04-25T13:38:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f242e013640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2428103af0 0x7f242819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f242e013640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2428104d30 0x7f242819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f242e013640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 0x7f24281a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f242e013640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f242810f600 con 0x7f2428109a30 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f242e013640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f242810f480 con 0x7f2428103af0 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f242e013640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f242810f780 con 0x7f2428104d30 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 0x7f24281a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 0x7f24281a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55942/0 (socket says 172.21.15.27:55942) 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 -- 172.21.15.27:0/3621549522 learned_addr learned my addr 172.21.15.27:0/3621549522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2426ffd640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2428104d30 0x7f242819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f24277fe640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2428103af0 0x7f242819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 -- 172.21.15.27:0/3621549522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2428104d30 msgr2=0x7f242819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2428104d30 0x7f242819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 -- 172.21.15.27:0/3621549522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2428103af0 msgr2=0x7f242819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2428103af0 0x7f242819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 -- 172.21.15.27:0/3621549522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24281a1c40 con 0x7f2428109a30 2024-04-25T13:38:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f24277fe640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2428103af0 0x7f242819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:49.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.529+0000 7f2427fff640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 0x7f24281a1680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f24200097b0 tx=0x7f2420003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:49.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.530+0000 7f2424ff9640 1 -- 172.21.15.27:0/3621549522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420004240 con 0x7f2428109a30 2024-04-25T13:38:49.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.530+0000 7f2424ff9640 1 -- 172.21.15.27:0/3621549522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f242002f380 con 0x7f2428109a30 2024-04-25T13:38:49.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.530+0000 7f2424ff9640 1 -- 172.21.15.27:0/3621549522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242002fc30 con 0x7f2428109a30 2024-04-25T13:38:49.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.530+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24281a1ed0 con 0x7f2428109a30 2024-04-25T13:38:49.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.530+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24281a22e0 con 0x7f2428109a30 2024-04-25T13:38:49.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.531+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2428108840 con 0x7f2428109a30 2024-04-25T13:38:49.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.532+0000 7f2424ff9640 1 -- 172.21.15.27:0/3621549522 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f24200043e0 con 0x7f2428109a30 2024-04-25T13:38:49.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.533+0000 7f2424ff9640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24100afb70 0x7f24100b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:49.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.533+0000 7f24277fe640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24100afb70 0x7f24100b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:49.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.533+0000 7f2424ff9640 1 -- 172.21.15.27:0/3621549522 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f24201003e0 con 0x7f2428109a30 2024-04-25T13:38:49.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.533+0000 7f24277fe640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24100afb70 0x7f24100b2080 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f24281046f0 tx=0x7f2418005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:49.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.536+0000 7f2424ff9640 1 -- 172.21.15.27:0/3621549522 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24200c9110 con 0x7f2428109a30 2024-04-25T13:38:49.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-mon[29140]: pgmap v285: 97 pgs: 16 creating+peering, 7 unknown, 74 active+clean; 578 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-25T13:38:49.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3184967939' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T13:38:49.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:49.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-25T13:38:49.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/4167150567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T13:38:49.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.750+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2428062fe0 con 0x7f24100afb70 2024-04-25T13:38:49.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.772+0000 7f2424ff9640 1 -- 172.21.15.27:0/3621549522 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2428062fe0 con 0x7f24100afb70 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24100afb70 msgr2=0x7f24100b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24100afb70 0x7f24100b2080 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f24281046f0 tx=0x7f2418005e70 comp rx=0 tx=0).stop 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 msgr2=0x7f24281a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 0x7f24281a1680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f24200097b0 tx=0x7f2420003c20 comp rx=0 tx=0).stop 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 shutdown_connections 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f24100afb70 0x7f24100b2080 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2428109a30 0x7f24281a1680 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2428104d30 0x7f242819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 --2- 172.21.15.27:0/3621549522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2428103af0 0x7f242819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 >> 172.21.15.27:0/3621549522 conn(0x7f24280fece0 msgr2=0x7f2428105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.774+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 shutdown_connections 2024-04-25T13:38:49.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.775+0000 7f242e013640 1 -- 172.21.15.27:0/3621549522 wait complete. 2024-04-25T13:38:49.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T13:38:49.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:49 smithi122 ceph-mon[32843]: pgmap v285: 97 pgs: 16 creating+peering, 7 unknown, 74 active+clean; 578 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-25T13:38:49.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:49 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3184967939' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T13:38:49.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:49 smithi122 ceph-mon[32843]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:49.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:49 smithi122 ceph-mon[32843]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-25T13:38:49.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:49 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/4167150567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T13:38:49.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:49 smithi045 ceph-mon[32731]: pgmap v285: 97 pgs: 16 creating+peering, 7 unknown, 74 active+clean; 578 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-25T13:38:49.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:49 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3184967939' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T13:38:49.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:49 smithi045 ceph-mon[32731]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T13:38:49.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:49 smithi045 ceph-mon[32731]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-25T13:38:49.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:49 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/4167150567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T13:38:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 -- 172.21.15.27:0/3454586425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4108930 msgr2=0x7f2ec410ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 --2- 172.21.15.27:0/3454586425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4108930 0x7f2ec410ad50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc009ca0 tx=0x7f2ebc02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 -- 172.21.15.27:0/3454586425 shutdown_connections 2024-04-25T13:38:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 --2- 172.21.15.27:0/3454586425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4108930 0x7f2ec410ad50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 --2- 172.21.15.27:0/3454586425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec4103c30 0x7f2ec4104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 --2- 172.21.15.27:0/3454586425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec41029f0 0x7f2ec41036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 -- 172.21.15.27:0/3454586425 >> 172.21.15.27:0/3454586425 conn(0x7f2ec40fdbc0 msgr2=0x7f2ec4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 -- 172.21.15.27:0/3454586425 shutdown_connections 2024-04-25T13:38:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 -- 172.21.15.27:0/3454586425 wait complete. 2024-04-25T13:38:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.902+0000 7f2ec993f640 1 Processor -- start 2024-04-25T13:38:49.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec993f640 1 -- start start 2024-04-25T13:38:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec993f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec41029f0 0x7f2ec40757f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec993f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4103c30 0x7f2ec419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec993f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec4108930 0x7f2ec41a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec993f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410e520 con 0x7f2ec4103c30 2024-04-25T13:38:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec993f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410e3a0 con 0x7f2ec4108930 2024-04-25T13:38:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec993f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410e6a0 con 0x7f2ec41029f0 2024-04-25T13:38:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4103c30 0x7f2ec419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4103c30 0x7f2ec419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55976/0 (socket says 172.21.15.27:55976) 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 -- 172.21.15.27:0/957556565 learned_addr learned my addr 172.21.15.27:0/957556565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec2ffd640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec41029f0 0x7f2ec40757f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec37fe640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec4108930 0x7f2ec41a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 -- 172.21.15.27:0/957556565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec41029f0 msgr2=0x7f2ec40757f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec41029f0 0x7f2ec40757f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 -- 172.21.15.27:0/957556565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec4108930 msgr2=0x7f2ec41a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec4108930 0x7f2ec41a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 -- 172.21.15.27:0/957556565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec41a1ce0 con 0x7f2ec4103c30 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec2ffd640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec41029f0 0x7f2ec40757f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec37fe640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec4108930 0x7f2ec41a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.903+0000 7f2ec27fc640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4103c30 0x7f2ec419efc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800d770 tx=0x7f2eb800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.904+0000 7f2ec893d640 1 -- 172.21.15.27:0/957556565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8017070 con 0x7f2ec4103c30 2024-04-25T13:38:49.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.904+0000 7f2ec893d640 1 -- 172.21.15.27:0/957556565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2eb80054d0 con 0x7f2ec4103c30 2024-04-25T13:38:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.904+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec41a1fd0 con 0x7f2ec4103c30 2024-04-25T13:38:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.904+0000 7f2ec893d640 1 -- 172.21.15.27:0/957556565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8004c30 con 0x7f2ec4103c30 2024-04-25T13:38:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.904+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec41a2440 con 0x7f2ec4103c30 2024-04-25T13:38:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.905+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec4107740 con 0x7f2ec4103c30 2024-04-25T13:38:49.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.906+0000 7f2ec893d640 1 -- 172.21.15.27:0/957556565 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2eb8004dd0 con 0x7f2ec4103c30 2024-04-25T13:38:49.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.907+0000 7f2ec893d640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2eac0afaf0 0x7f2eac0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:49.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.907+0000 7f2ec2ffd640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2eac0afaf0 0x7f2eac0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:49.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.907+0000 7f2ec893d640 1 -- 172.21.15.27:0/957556565 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f2eb80dea50 con 0x7f2ec4103c30 2024-04-25T13:38:49.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.907+0000 7f2ec2ffd640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2eac0afaf0 0x7f2eac0b2000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2ec41035f0 tx=0x7f2eb400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:49.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:49.910+0000 7f2ec893d640 1 -- 172.21.15.27:0/957556565 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2eb80a7780 con 0x7f2ec4103c30 2024-04-25T13:38:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.126+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2ec4062fe0 con 0x7f2eac0afaf0 2024-04-25T13:38:50.150 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:38:49.755+0000 7f0feb551700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T13:38:50.150 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:38:49.755+0000 7f0feb551700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T13:38:50.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:38:49.755+0000 7f0feb551700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T13:38:50.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:38:49.755+0000 7f0feb551700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T13:38:50.151 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:38:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:38:49.755+0000 7f0feb551700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T13:38:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.152+0000 7f2ec893d640 1 -- 172.21.15.27:0/957556565 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2ec4062fe0 con 0x7f2eac0afaf0 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2eac0afaf0 msgr2=0x7f2eac0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2eac0afaf0 0x7f2eac0b2000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2ec41035f0 tx=0x7f2eb400a560 comp rx=0 tx=0).stop 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4103c30 msgr2=0x7f2ec419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4103c30 0x7f2ec419efc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800d770 tx=0x7f2eb800dc30 comp rx=0 tx=0).stop 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 shutdown_connections 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2eac0afaf0 0x7f2eac0b2000 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ec4108930 0x7f2ec41a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ec4103c30 0x7f2ec419efc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 --2- 172.21.15.27:0/957556565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2ec41029f0 0x7f2ec40757f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.154+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 >> 172.21.15.27:0/957556565 conn(0x7f2ec40fdbc0 msgr2=0x7f2ec4104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.155+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 shutdown_connections 2024-04-25T13:38:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.155+0000 7f2ec993f640 1 -- 172.21.15.27:0/957556565 wait complete. 2024-04-25T13:38:50.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/1047793579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0108930 msgr2=0x7fbaa010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/1047793579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0108930 0x7fbaa010ad50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fba98009ca0 tx=0x7fba9802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/1047793579 shutdown_connections 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/1047793579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0108930 0x7fbaa010ad50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/1047793579 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaa0103c30 0x7fbaa0104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/1047793579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa01029f0 0x7fbaa01036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/1047793579 >> 172.21.15.27:0/1047793579 conn(0x7fbaa00fdbc0 msgr2=0x7fbaa0100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/1047793579 shutdown_connections 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/1047793579 wait complete. 2024-04-25T13:38:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 Processor -- start 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.282+0000 7fbaa7fe3640 1 -- start start 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa7fe3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa01029f0 0x7fbaa0075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa7fe3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0103c30 0x7fbaa01a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa7fe3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaa0108930 0x7fbaa01a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa7fe3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa010e520 con 0x7fbaa0103c30 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa7fe3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa010e3a0 con 0x7fbaa01029f0 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa7fe3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa010e6a0 con 0x7fbaa0108930 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0103c30 0x7fbaa01a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0103c30 0x7fbaa01a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56004/0 (socket says 172.21.15.27:56004) 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 -- 172.21.15.27:0/2436034826 learned_addr learned my addr 172.21.15.27:0/2436034826 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa5d7f640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa01029f0 0x7fbaa0075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa6580640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaa0108930 0x7fbaa01a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 -- 172.21.15.27:0/2436034826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaa0108930 msgr2=0x7fbaa01a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaa0108930 0x7fbaa01a38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 -- 172.21.15.27:0/2436034826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa01029f0 msgr2=0x7fbaa0075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa01029f0 0x7fbaa0075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 -- 172.21.15.27:0/2436034826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa01a3ee0 con 0x7fbaa0103c30 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa5d7f640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa01029f0 0x7fbaa0075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa6580640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaa0108930 0x7fbaa01a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa557e640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0103c30 0x7fbaa01a1350 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fba9000d770 tx=0x7fba9000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fba96ffd640 1 -- 172.21.15.27:0/2436034826 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90017070 con 0x7fbaa0103c30 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fba96ffd640 1 -- 172.21.15.27:0/2436034826 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba900054d0 con 0x7fbaa0103c30 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa01a41d0 con 0x7fbaa0103c30 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fba96ffd640 1 -- 172.21.15.27:0/2436034826 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90004c30 con 0x7fbaa0103c30 2024-04-25T13:38:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.283+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa01a4720 con 0x7fbaa0103c30 2024-04-25T13:38:50.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.285+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa0107740 con 0x7fbaa0103c30 2024-04-25T13:38:50.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.286+0000 7fba96ffd640 1 -- 172.21.15.27:0/2436034826 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fba90004e10 con 0x7fbaa0103c30 2024-04-25T13:38:50.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.286+0000 7fba96ffd640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba8c0afb70 0x7fba8c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:50.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.287+0000 7fbaa5d7f640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba8c0afb70 0x7fba8c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:50.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.287+0000 7fba96ffd640 1 -- 172.21.15.27:0/2436034826 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fba900de5c0 con 0x7fbaa0103c30 2024-04-25T13:38:50.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.287+0000 7fbaa5d7f640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba8c0afb70 0x7fba8c0b2080 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbaa01035f0 tx=0x7fba9c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.291+0000 7fba96ffd640 1 -- 172.21.15.27:0/2436034826 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba900a72f0 con 0x7fbaa0103c30 2024-04-25T13:38:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.410+0000 7fba96ffd640 1 -- 172.21.15.27:0/2436034826 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fba900a2e80 con 0x7fbaa0103c30 2024-04-25T13:38:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.519+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fbaa0063070 con 0x7fba8c0afb70 2024-04-25T13:38:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.524+0000 7fba96ffd640 1 -- 172.21.15.27:0/2436034826 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fbaa0063070 con 0x7fba8c0afb70 2024-04-25T13:38:50.524 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/qa/sv_0/4fb67611-959a-46a1-a18e-b8efb6a33d0c 2024-04-25T13:38:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba8c0afb70 msgr2=0x7fba8c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba8c0afb70 0x7fba8c0b2080 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbaa01035f0 tx=0x7fba9c00a560 comp rx=0 tx=0).stop 2024-04-25T13:38:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0103c30 msgr2=0x7fbaa01a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0103c30 0x7fbaa01a1350 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fba9000d770 tx=0x7fba9000dc30 comp rx=0 tx=0).stop 2024-04-25T13:38:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 shutdown_connections 2024-04-25T13:38:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba8c0afb70 0x7fba8c0b2080 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaa0108930 0x7fbaa01a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaa0103c30 0x7fbaa01a1350 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 --2- 172.21.15.27:0/2436034826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa01029f0 0x7fbaa0075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 >> 172.21.15.27:0/2436034826 conn(0x7fbaa00fdbc0 msgr2=0x7fbaa0104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 shutdown_connections 2024-04-25T13:38:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.526+0000 7fbaa7fe3640 1 -- 172.21.15.27:0/2436034826 wait complete. 2024-04-25T13:38:50.536 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:50 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/4167150567' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T13:38:50.536 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:50 smithi027 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-25T13:38:50.536 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:50 smithi027 ceph-mon[29140]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:50.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T13:38:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 -- 172.21.15.27:0/3251852830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8109a30 msgr2=0x7f91c810be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 --2- 172.21.15.27:0/3251852830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8109a30 0x7f91c810be50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f91c0009ca0 tx=0x7f91c002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 -- 172.21.15.27:0/3251852830 shutdown_connections 2024-04-25T13:38:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 --2- 172.21.15.27:0/3251852830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8109a30 0x7f91c810be50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 --2- 172.21.15.27:0/3251852830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c8104d30 0x7f91c8105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 --2- 172.21.15.27:0/3251852830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c8103af0 0x7f91c81047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 -- 172.21.15.27:0/3251852830 >> 172.21.15.27:0/3251852830 conn(0x7f91c80fece0 msgr2=0x7f91c8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 -- 172.21.15.27:0/3251852830 shutdown_connections 2024-04-25T13:38:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.653+0000 7f91d0470640 1 -- 172.21.15.27:0/3251852830 wait complete. 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91d0470640 1 Processor -- start 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91d0470640 1 -- start start 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91d0470640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c8103af0 0x7f91c819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91d0470640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8104d30 0x7f91c819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91d0470640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c8109a30 0x7f91c81a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91d0470640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f91c810f670 con 0x7f91c8104d30 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91d0470640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91c810f4f0 con 0x7f91c8109a30 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91d0470640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f91c810f7f0 con 0x7f91c8103af0 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cda0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8104d30 0x7f91c819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cda0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8104d30 0x7f91c819a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56018/0 (socket says 172.21.15.27:56018) 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cda0b640 1 -- 172.21.15.27:0/688395913 learned_addr learned my addr 172.21.15.27:0/688395913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91ce20c640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c8103af0 0x7f91c819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cea0d640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c8109a30 0x7f91c81a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cda0b640 1 -- 172.21.15.27:0/688395913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c8103af0 msgr2=0x7f91c819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cda0b640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c8103af0 0x7f91c819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cda0b640 1 -- 172.21.15.27:0/688395913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c8109a30 msgr2=0x7f91c81a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cda0b640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c8109a30 0x7f91c81a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cda0b640 1 -- 172.21.15.27:0/688395913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c81a1d20 con 0x7f91c8104d30 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.654+0000 7f91cea0d640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c8109a30 0x7f91c81a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.655+0000 7f91cda0b640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8104d30 0x7f91c819a720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f91b800d950 tx=0x7f91b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.655+0000 7f91bf7fe640 1 -- 172.21.15.27:0/688395913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b8017070 con 0x7f91c8104d30 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.655+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c81a1fb0 con 0x7f91c8104d30 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.655+0000 7f91bf7fe640 1 -- 172.21.15.27:0/688395913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91b8009c30 con 0x7f91c8104d30 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.655+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c81a24f0 con 0x7f91c8104d30 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.655+0000 7f91bf7fe640 1 -- 172.21.15.27:0/688395913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b800f620 con 0x7f91c8104d30 2024-04-25T13:38:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.656+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91c8108840 con 0x7f91c8104d30 2024-04-25T13:38:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.657+0000 7f91bf7fe640 1 -- 172.21.15.27:0/688395913 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f91b8010460 con 0x7f91c8104d30 2024-04-25T13:38:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.658+0000 7f91bf7fe640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f91b40afb80 0x7f91b40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.658+0000 7f91ce20c640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f91b40afb80 0x7f91b40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.658+0000 7f91bf7fe640 1 -- 172.21.15.27:0/688395913 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f91b80de840 con 0x7f91c8104d30 2024-04-25T13:38:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.659+0000 7f91ce20c640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f91b40afb80 0x7f91b40b2090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f91c4005fd0 tx=0x7f91c400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:50.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.662+0000 7f91bf7fe640 1 -- 172.21.15.27:0/688395913 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91b80a7470 con 0x7f91c8104d30 2024-04-25T13:38:50.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:50 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/4167150567' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T13:38:50.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:50 smithi122 ceph-mon[32843]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-25T13:38:50.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:50 smithi122 ceph-mon[32843]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:50.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:50 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/4167150567' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T13:38:50.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:50 smithi045 ceph-mon[32731]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-25T13:38:50.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:50 smithi045 ceph-mon[32731]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:50.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.877+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f91c8062fe0 con 0x7f91b40afb80 2024-04-25T13:38:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.905+0000 7f91bf7fe640 1 -- 172.21.15.27:0/688395913 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f91c8062fe0 con 0x7f91b40afb80 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f91b40afb80 msgr2=0x7f91b40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f91b40afb80 0x7f91b40b2090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f91c4005fd0 tx=0x7f91c400b390 comp rx=0 tx=0).stop 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8104d30 msgr2=0x7f91c819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8104d30 0x7f91c819a720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f91b800d950 tx=0x7f91b800de10 comp rx=0 tx=0).stop 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 shutdown_connections 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f91b40afb80 0x7f91b40b2090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c8109a30 0x7f91c81a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c8104d30 0x7f91c819a720 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 --2- 172.21.15.27:0/688395913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c8103af0 0x7f91c819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 >> 172.21.15.27:0/688395913 conn(0x7f91c80fece0 msgr2=0x7f91c8105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 shutdown_connections 2024-04-25T13:38:50.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:50.907+0000 7f91d0470640 1 -- 172.21.15.27:0/688395913 wait complete. 2024-04-25T13:38:50.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 -- 172.21.15.27:0/1647907131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 msgr2=0x7fc3281036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 --2- 172.21.15.27:0/1647907131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 0x7fc3281036f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc318009900 tx=0x7fc31802a410 comp rx=0 tx=0).stop 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 -- 172.21.15.27:0/1647907131 shutdown_connections 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 --2- 172.21.15.27:0/1647907131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc328108930 0x7fc32810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 --2- 172.21.15.27:0/1647907131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc328103c30 0x7fc328104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 --2- 172.21.15.27:0/1647907131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 0x7fc3281036f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 -- 172.21.15.27:0/1647907131 >> 172.21.15.27:0/1647907131 conn(0x7fc3280fdbc0 msgr2=0x7fc328100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 -- 172.21.15.27:0/1647907131 shutdown_connections 2024-04-25T13:38:51.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.035+0000 7fc32d00c640 1 -- 172.21.15.27:0/1647907131 wait complete. 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.036+0000 7fc32d00c640 1 Processor -- start 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.036+0000 7fc32d00c640 1 -- start start 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.036+0000 7fc32d00c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 0x7fc328199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.036+0000 7fc32d00c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc328103c30 0x7fc328199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc32d00c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc328108930 0x7fc3281a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc32d00c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc32810e4f0 con 0x7fc3281029f0 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc32d00c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc32810e370 con 0x7fc328103c30 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc32d00c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc32810e670 con 0x7fc328108930 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 0x7fc328199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 0x7fc328199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56032/0 (socket says 172.21.15.27:56032) 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 -- 172.21.15.27:0/2213728069 learned_addr learned my addr 172.21.15.27:0/2213728069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc31ffff640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc328103c30 0x7fc328199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc32759e640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc328108930 0x7fc3281a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 -- 172.21.15.27:0/2213728069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc328108930 msgr2=0x7fc3281a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc328108930 0x7fc3281a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 -- 172.21.15.27:0/2213728069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc328103c30 msgr2=0x7fc328199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc328103c30 0x7fc328199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 -- 172.21.15.27:0/2213728069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3281a0c60 con 0x7fc3281029f0 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc31ffff640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc328103c30 0x7fc328199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc326d9d640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 0x7fc328199120 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc318006010 tx=0x7fc318031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc324d99640 1 -- 172.21.15.27:0/2213728069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318046070 con 0x7fc3281029f0 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc328074dd0 con 0x7fc3281029f0 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc324d99640 1 -- 172.21.15.27:0/2213728069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc318003aa0 con 0x7fc3281029f0 2024-04-25T13:38:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc324d99640 1 -- 172.21.15.27:0/2213728069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318035050 con 0x7fc3281029f0 2024-04-25T13:38:51.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.037+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3280752c0 con 0x7fc3281029f0 2024-04-25T13:38:51.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.039+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc32806a160 con 0x7fc3281029f0 2024-04-25T13:38:51.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.040+0000 7fc324d99640 1 -- 172.21.15.27:0/2213728069 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fc318049070 con 0x7fc3281029f0 2024-04-25T13:38:51.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.040+0000 7fc324d99640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3140afb80 0x7fc3140b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.040+0000 7fc31ffff640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3140afb80 0x7fc3140b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.041+0000 7fc324d99640 1 -- 172.21.15.27:0/2213728069 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fc318100890 con 0x7fc3281029f0 2024-04-25T13:38:51.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.041+0000 7fc31ffff640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3140afb80 0x7fc3140b2090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc310005fd0 tx=0x7fc310005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.045+0000 7fc324d99640 1 -- 172.21.15.27:0/2213728069 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3180c94c0 con 0x7fc3281029f0 2024-04-25T13:38:51.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.260+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fc328107740 con 0x7fc3140afb80 2024-04-25T13:38:51.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.265+0000 7fc324d99640 1 -- 172.21.15.27:0/2213728069 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fc328107740 con 0x7fc3140afb80 2024-04-25T13:38:51.265 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/qa/sv_1/aa501c8f-0208-4329-abf5-aee6b1b25d7c 2024-04-25T13:38:51.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3140afb80 msgr2=0x7fc3140b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3140afb80 0x7fc3140b2090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc310005fd0 tx=0x7fc310005850 comp rx=0 tx=0).stop 2024-04-25T13:38:51.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 msgr2=0x7fc328199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 0x7fc328199120 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc318006010 tx=0x7fc318031a50 comp rx=0 tx=0).stop 2024-04-25T13:38:51.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 shutdown_connections 2024-04-25T13:38:51.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3140afb80 0x7fc3140b2090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc328108930 0x7fc3281a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc328103c30 0x7fc328199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 --2- 172.21.15.27:0/2213728069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3281029f0 0x7fc328199120 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 >> 172.21.15.27:0/2213728069 conn(0x7fc3280fdbc0 msgr2=0x7fc328104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 shutdown_connections 2024-04-25T13:38:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.267+0000 7fc32d00c640 1 -- 172.21.15.27:0/2213728069 wait complete. 2024-04-25T13:38:51.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:38:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 -- 172.21.15.27:0/2825633297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 msgr2=0x7f52fc10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 --2- 172.21.15.27:0/2825633297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 0x7f52fc10ad40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009ca0 tx=0x7f52ec02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 -- 172.21.15.27:0/2825633297 shutdown_connections 2024-04-25T13:38:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 --2- 172.21.15.27:0/2825633297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 0x7f52fc10ad40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 --2- 172.21.15.27:0/2825633297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc103c20 0x7f52fc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 --2- 172.21.15.27:0/2825633297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52fc1029e0 0x7f52fc1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 -- 172.21.15.27:0/2825633297 >> 172.21.15.27:0/2825633297 conn(0x7f52fc0fdbd0 msgr2=0x7f52fc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 -- 172.21.15.27:0/2825633297 shutdown_connections 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.390+0000 7f530178f640 1 -- 172.21.15.27:0/2825633297 wait complete. 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f530178f640 1 Processor -- start 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f530178f640 1 -- start start 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f530178f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52fc1029e0 0x7f52fc19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f530178f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc103c20 0x7f52fc19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f530178f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 0x7f52fc1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f530178f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc10f610 con 0x7f52fc108920 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f530178f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc10f490 con 0x7f52fc103c20 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f530178f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc10f790 con 0x7f52fc1029e0 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 0x7f52fc1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fb7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 0x7f52fc1a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56054/0 (socket says 172.21.15.27:56054) 2024-04-25T13:38:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fb7fe640 1 -- 172.21.15.27:0/2734795461 learned_addr learned my addr 172.21.15.27:0/2734795461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fa7fc640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc103c20 0x7f52fc19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52faffd640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52fc1029e0 0x7f52fc19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fb7fe640 1 -- 172.21.15.27:0/2734795461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52fc1029e0 msgr2=0x7f52fc19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fb7fe640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52fc1029e0 0x7f52fc19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fb7fe640 1 -- 172.21.15.27:0/2734795461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc103c20 msgr2=0x7f52fc19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fb7fe640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc103c20 0x7f52fc19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.391+0000 7f52fb7fe640 1 -- 172.21.15.27:0/2734795461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc1a1d90 con 0x7f52fc108920 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.392+0000 7f52fb7fe640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 0x7f52fc1a1720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f52ec005e30 tx=0x7f52ec02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.392+0000 7f52e3fff640 1 -- 172.21.15.27:0/2734795461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec003bf0 con 0x7f52fc108920 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.392+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc1a2020 con 0x7f52fc108920 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.392+0000 7f52e3fff640 1 -- 172.21.15.27:0/2734795461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52ec003d90 con 0x7f52fc108920 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.392+0000 7f52e3fff640 1 -- 172.21.15.27:0/2734795461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec02fb90 con 0x7f52fc108920 2024-04-25T13:38:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.392+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc1a25c0 con 0x7f52fc108920 2024-04-25T13:38:51.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.393+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc107730 con 0x7f52fc108920 2024-04-25T13:38:51.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.395+0000 7f52e3fff640 1 -- 172.21.15.27:0/2734795461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f52ec02fd30 con 0x7f52fc108920 2024-04-25T13:38:51.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.395+0000 7f52e3fff640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52e40afb80 0x7f52e40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.396+0000 7f52faffd640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52e40afb80 0x7f52e40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.396+0000 7f52e3fff640 1 -- 172.21.15.27:0/2734795461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f52ec100400 con 0x7f52fc108920 2024-04-25T13:38:51.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.396+0000 7f52faffd640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52e40afb80 0x7f52e40b2090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f52f4005fd0 tx=0x7f52f4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:51.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.399+0000 7f52e3fff640 1 -- 172.21.15.27:0/2734795461 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f52ec0c9140 con 0x7f52fc108920 2024-04-25T13:38:51.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.630+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52fc02db50 con 0x7f52fc108920 2024-04-25T13:38:51.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.631+0000 7f52e3fff640 1 -- 172.21.15.27:0/2734795461 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f52ec0098b0 con 0x7f52fc108920 2024-04-25T13:38:51.631 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:51.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:38:51.632 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:38:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52e40afb80 msgr2=0x7f52e40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52e40afb80 0x7f52e40b2090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f52f4005fd0 tx=0x7f52f4005850 comp rx=0 tx=0).stop 2024-04-25T13:38:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 msgr2=0x7f52fc1a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 0x7f52fc1a1720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f52ec005e30 tx=0x7f52ec02f380 comp rx=0 tx=0).stop 2024-04-25T13:38:51.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 shutdown_connections 2024-04-25T13:38:51.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52e40afb80 0x7f52e40b2090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52fc108920 0x7f52fc1a1720 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52fc103c20 0x7f52fc19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 --2- 172.21.15.27:0/2734795461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52fc1029e0 0x7f52fc19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 >> 172.21.15.27:0/2734795461 conn(0x7f52fc0fdbd0 msgr2=0x7f52fc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:51.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 shutdown_connections 2024-04-25T13:38:51.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.633+0000 7f530178f640 1 -- 172.21.15.27:0/2734795461 wait complete. 2024-04-25T13:38:51.643 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:51 smithi027 ceph-mon[29140]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.643 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:51 smithi027 ceph-mon[29140]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-25T13:38:51.644 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:51 smithi027 ceph-mon[29140]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T13:38:51.644 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:51 smithi027 ceph-mon[29140]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.644 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:51 smithi027 ceph-mon[29140]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:38:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 -- 172.21.15.27:0/1685869724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 msgr2=0x7f4a8810ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/1685869724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 0x7f4a8810ad20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009ca0 tx=0x7f4a8002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 -- 172.21.15.27:0/1685869724 shutdown_connections 2024-04-25T13:38:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/1685869724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 0x7f4a8810ad20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/1685869724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a88103c00 0x7f4a88104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/1685869724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a881029c0 0x7f4a881036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 -- 172.21.15.27:0/1685869724 >> 172.21.15.27:0/1685869724 conn(0x7f4a880fdbd0 msgr2=0x7f4a88100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 -- 172.21.15.27:0/1685869724 shutdown_connections 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 -- 172.21.15.27:0/1685869724 wait complete. 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 Processor -- start 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.757+0000 7f4a8f68c640 1 -- start start 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8f68c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a881029c0 0x7f4a8819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8f68c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a88103c00 0x7f4a8819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8f68c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 0x7f4a881a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8f68c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8810f5f0 con 0x7f4a88108900 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8f68c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8810f470 con 0x7f4a881029c0 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8f68c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8810f770 con 0x7f4a88103c00 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 0x7f4a881a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8d428640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a881029c0 0x7f4a8819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 0x7f4a881a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56064/0 (socket says 172.21.15.27:56064) 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 -- 172.21.15.27:0/3061921969 learned_addr learned my addr 172.21.15.27:0/3061921969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8cc27640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a88103c00 0x7f4a8819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 -- 172.21.15.27:0/3061921969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a88103c00 msgr2=0x7f4a8819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a88103c00 0x7f4a8819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 -- 172.21.15.27:0/3061921969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a881029c0 msgr2=0x7f4a8819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a881029c0 0x7f4a8819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 -- 172.21.15.27:0/3061921969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a881a1d00 con 0x7f4a88108900 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8cc27640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a88103c00 0x7f4a8819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.758+0000 7f4a8dc29640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 0x7f4a881a1690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a80004af0 tx=0x7f4a80031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.759+0000 7f4a7e7fc640 1 -- 172.21.15.27:0/3061921969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80003b70 con 0x7f4a88108900 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.759+0000 7f4a7e7fc640 1 -- 172.21.15.27:0/3061921969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a80003d10 con 0x7f4a88108900 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.759+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a881a1f90 con 0x7f4a88108900 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.759+0000 7f4a7e7fc640 1 -- 172.21.15.27:0/3061921969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8002fb20 con 0x7f4a88108900 2024-04-25T13:38:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.759+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a881a23f0 con 0x7f4a88108900 2024-04-25T13:38:51.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.760+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8806afd0 con 0x7f4a88108900 2024-04-25T13:38:51.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.761+0000 7f4a7e7fc640 1 -- 172.21.15.27:0/3061921969 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f4a80004580 con 0x7f4a88108900 2024-04-25T13:38:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.762+0000 7f4a7e7fc640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a740afb80 0x7f4a740b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.762+0000 7f4a8d428640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a740afb80 0x7f4a740b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.762+0000 7f4a7e7fc640 1 -- 172.21.15.27:0/3061921969 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f4a801002d0 con 0x7f4a88108900 2024-04-25T13:38:51.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.762+0000 7f4a8d428640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a740afb80 0x7f4a740b2090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a881035c0 tx=0x7f4a84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.765+0000 7f4a7e7fc640 1 -- 172.21.15.27:0/3061921969 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a8806afd0 con 0x7f4a88108900 2024-04-25T13:38:51.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:51 smithi122 ceph-mon[32843]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:51 smithi122 ceph-mon[32843]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-25T13:38:51.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:51 smithi122 ceph-mon[32843]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T13:38:51.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:51 smithi122 ceph-mon[32843]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:51 smithi122 ceph-mon[32843]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:51 smithi045 ceph-mon[32731]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:51 smithi045 ceph-mon[32731]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-25T13:38:51.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:51 smithi045 ceph-mon[32731]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T13:38:51.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:51 smithi045 ceph-mon[32731]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:51 smithi045 ceph-mon[32731]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.955+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a8802dc10 con 0x7f4a88108900 2024-04-25T13:38:51.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.957+0000 7f4a7e7fc640 1 -- 172.21.15.27:0/3061921969 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+22173 (secure 0 0 0) 0x7f4a800c9010 con 0x7f4a88108900 2024-04-25T13:38:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:51.958 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:38:46.332369+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a740afb80 msgr2=0x7f4a740b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a740afb80 0x7f4a740b2090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a881035c0 tx=0x7f4a84005f90 comp rx=0 tx=0).stop 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 msgr2=0x7f4a881a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 0x7f4a881a1690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a80004af0 tx=0x7f4a80031700 comp rx=0 tx=0).stop 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 shutdown_connections 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a740afb80 0x7f4a740b2090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a88108900 0x7f4a881a1690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a88103c00 0x7f4a8819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 --2- 172.21.15.27:0/3061921969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a881029c0 0x7f4a8819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 >> 172.21.15.27:0/3061921969 conn(0x7f4a880fdbd0 msgr2=0x7f4a881048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 shutdown_connections 2024-04-25T13:38:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:51.959+0000 7f4a8f68c640 1 -- 172.21.15.27:0/3061921969 wait complete. 2024-04-25T13:38:51.971 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T13:38:51.971 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T13:38:52.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 -- 172.21.15.27:0/3584870619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc1029e0 msgr2=0x7f41bc1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 --2- 172.21.15.27:0/3584870619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc1029e0 0x7f41bc1036e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009900 tx=0x7f41ac02a410 comp rx=0 tx=0).stop 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 -- 172.21.15.27:0/3584870619 shutdown_connections 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 --2- 172.21.15.27:0/3584870619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41bc108920 0x7f41bc10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 --2- 172.21.15.27:0/3584870619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41bc103c20 0x7f41bc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 --2- 172.21.15.27:0/3584870619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc1029e0 0x7f41bc1036e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 -- 172.21.15.27:0/3584870619 >> 172.21.15.27:0/3584870619 conn(0x7f41bc0fdbd0 msgr2=0x7f41bc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 -- 172.21.15.27:0/3584870619 shutdown_connections 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.082+0000 7f41c3934640 1 -- 172.21.15.27:0/3584870619 wait complete. 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c3934640 1 Processor -- start 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c3934640 1 -- start start 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c3934640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41bc1029e0 0x7f41bc199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c3934640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc103c20 0x7f41bc199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:52.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c3934640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41bc108920 0x7f41bc1a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c3934640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc10e540 con 0x7f41bc103c20 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c3934640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc10e3c0 con 0x7f41bc108920 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c3934640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc10e6c0 con 0x7f41bc1029e0 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c0ecf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc103c20 0x7f41bc199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c0ecf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc103c20 0x7f41bc199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56082/0 (socket says 172.21.15.27:56082) 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c0ecf640 1 -- 172.21.15.27:0/1161681243 learned_addr learned my addr 172.21.15.27:0/1161681243 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c16d0640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41bc1029e0 0x7f41bc199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c1ed1640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41bc108920 0x7f41bc1a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c0ecf640 1 -- 172.21.15.27:0/1161681243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41bc1029e0 msgr2=0x7f41bc199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c0ecf640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41bc1029e0 0x7f41bc199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c0ecf640 1 -- 172.21.15.27:0/1161681243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41bc108920 msgr2=0x7f41bc1a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c0ecf640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41bc108920 0x7f41bc1a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.083+0000 7f41c0ecf640 1 -- 172.21.15.27:0/1161681243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41bc1a0d40 con 0x7f41bc103c20 2024-04-25T13:38:52.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.084+0000 7f41c0ecf640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc103c20 0x7f41bc199740 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f41b800b4a0 tx=0x7f41b800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:52.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.084+0000 7f41b27fc640 1 -- 172.21.15.27:0/1161681243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8014070 con 0x7f41bc103c20 2024-04-25T13:38:52.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.084+0000 7f41b27fc640 1 -- 172.21.15.27:0/1161681243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41b8010ac0 con 0x7f41bc103c20 2024-04-25T13:38:52.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.084+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41bc1a0fd0 con 0x7f41bc103c20 2024-04-25T13:38:52.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.084+0000 7f41b27fc640 1 -- 172.21.15.27:0/1161681243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8017c30 con 0x7f41bc103c20 2024-04-25T13:38:52.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.084+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41bc1a1510 con 0x7f41bc103c20 2024-04-25T13:38:52.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.085+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41bc107730 con 0x7f41bc103c20 2024-04-25T13:38:52.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.086+0000 7f41b27fc640 1 -- 172.21.15.27:0/1161681243 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f41b8017070 con 0x7f41bc103c20 2024-04-25T13:38:52.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.087+0000 7f41b27fc640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41a80afb80 0x7f41a80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:52.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.087+0000 7f41c16d0640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41a80afb80 0x7f41a80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:52.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.087+0000 7f41b27fc640 1 -- 172.21.15.27:0/1161681243 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f41b80de130 con 0x7f41bc103c20 2024-04-25T13:38:52.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.087+0000 7f41c16d0640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41a80afb80 0x7f41a80b2090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f41ac005ab0 tx=0x7f41ac031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:52.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.091+0000 7f41b27fc640 1 -- 172.21.15.27:0/1161681243 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41b80a6820 con 0x7f41bc103c20 2024-04-25T13:38:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.273+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f41bc062fe0 con 0x7f41a80afb80 2024-04-25T13:38:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.281+0000 7f41b27fc640 1 -- 172.21.15.27:0/1161681243 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258875 (secure 0 0 0) 0x7f41bc062fe0 con 0x7f41a80afb80 2024-04-25T13:38:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:52.286 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T13:38:52.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41a80afb80 msgr2=0x7f41a80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41a80afb80 0x7f41a80b2090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f41ac005ab0 tx=0x7f41ac031400 comp rx=0 tx=0).stop 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc103c20 msgr2=0x7f41bc199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc103c20 0x7f41bc199740 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f41b800b4a0 tx=0x7f41b800b960 comp rx=0 tx=0).stop 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 shutdown_connections 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41a80afb80 0x7f41a80b2090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41bc108920 0x7f41bc1a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41bc103c20 0x7f41bc199740 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 --2- 172.21.15.27:0/1161681243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41bc1029e0 0x7f41bc199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 >> 172.21.15.27:0/1161681243 conn(0x7f41bc0fdbd0 msgr2=0x7f41bc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 shutdown_connections 2024-04-25T13:38:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.284+0000 7f41c3934640 1 -- 172.21.15.27:0/1161681243 wait complete. 2024-04-25T13:38:52.295 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-04-25T13:38:50.347607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":168,"num_write_kb":2164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375468,"kb_used_data":9572,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124566356,"statfs":{"total":1151940427776,"available":1151555948544,"internally_reserved":0,"allocated":9801728,"data_stored":6348620,"data_compressed":108783,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19138,"internal_metadata":374453566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.016282"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336724+0000","last_change":"2024-04-25T13:38:45.356552+0000","last_active":"2024-04-25T13:38:46.336724+0000","last_peered":"2024-04-25T13:38:46.336724+0000","last_clean":"2024-04-25T13:38:46.336724+0000","last_became_active":"2024-04-25T13:38:45.356122+0000","last_became_peered":"2024-04-25T13:38:45.356122+0000","last_unstale":"2024-04-25T13:38:46.336724+0000","last_undegraded":"2024-04-25T13:38:46.336724+0000","last_fullsized":"2024-04-25T13:38:46.336724+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:55.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.359641+0000","last_change":"2024-04-25T13:38:45.358577+0000","last_active":"2024-04-25T13:38:45.359641+0000","last_peered":"2024-04-25T13:38:45.359641+0000","last_clean":"2024-04-25T13:38:45.359641+0000","last_became_active":"2024-04-25T13:38:45.358122+0000","last_became_peered":"2024-04-25T13:38:45.358122+0000","last_unstale":"2024-04-25T13:38:45.359641+0000","last_undegraded":"2024-04-25T13:38:45.359641+0000","last_fullsized":"2024-04-25T13:38:45.359641+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:58.526640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.364385+0000","last_change":"2024-04-25T13:38:45.362539+0000","last_active":"2024-04-25T13:38:45.364385+0000","last_peered":"2024-04-25T13:38:45.364385+0000","last_clean":"2024-04-25T13:38:45.364385+0000","last_became_active":"2024-04-25T13:38:45.361983+0000","last_became_peered":"2024-04-25T13:38:45.361983+0000","last_unstale":"2024-04-25T13:38:45.364385+0000","last_undegraded":"2024-04-25T13:38:45.364385+0000","last_fullsized":"2024-04-25T13:38:45.364385+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:28.235030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.337191+0000","last_change":"2024-04-25T13:38:45.360535+0000","last_active":"2024-04-25T13:38:46.337191+0000","last_peered":"2024-04-25T13:38:46.337191+0000","last_clean":"2024-04-25T13:38:46.337191+0000","last_became_active":"2024-04-25T13:38:45.357722+0000","last_became_peered":"2024-04-25T13:38:45.357722+0000","last_unstale":"2024-04-25T13:38:46.337191+0000","last_undegraded":"2024-04-25T13:38:46.337191+0000","last_fullsized":"2024-04-25T13:38:46.337191+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:35.347434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369200+0000","last_change":"2024-04-25T13:38:45.353653+0000","last_active":"2024-04-25T13:38:46.369200+0000","last_peered":"2024-04-25T13:38:46.369200+0000","last_clean":"2024-04-25T13:38:46.369200+0000","last_became_active":"2024-04-25T13:38:45.353254+0000","last_became_peered":"2024-04-25T13:38:45.353254+0000","last_unstale":"2024-04-25T13:38:46.369200+0000","last_undegraded":"2024-04-25T13:38:46.369200+0000","last_fullsized":"2024-04-25T13:38:46.369200+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:57.275732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.360380+0000","last_change":"2024-04-25T13:38:45.358241+0000","last_active":"2024-04-25T13:38:45.360380+0000","last_peered":"2024-04-25T13:38:45.360380+0000","last_clean":"2024-04-25T13:38:45.360380+0000","last_became_active":"2024-04-25T13:38:45.357843+0000","last_became_peered":"2024-04-25T13:38:45.357843+0000","last_unstale":"2024-04-25T13:38:45.360380+0000","last_undegraded":"2024-04-25T13:38:45.360380+0000","last_fullsized":"2024-04-25T13:38:45.360380+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:41:37.908280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629858+0000","last_change":"2024-04-25T13:38:45.350710+0000","last_active":"2024-04-25T13:38:46.629858+0000","last_peered":"2024-04-25T13:38:46.629858+0000","last_clean":"2024-04-25T13:38:46.629858+0000","last_became_active":"2024-04-25T13:38:45.350171+0000","last_became_peered":"2024-04-25T13:38:45.350171+0000","last_unstale":"2024-04-25T13:38:46.629858+0000","last_undegraded":"2024-04-25T13:38:46.629858+0000","last_fullsized":"2024-04-25T13:38:46.629858+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:24.933631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.359118+0000","last_change":"2024-04-25T13:38:45.356221+0000","last_active":"2024-04-25T13:38:45.359118+0000","last_peered":"2024-04-25T13:38:45.359118+0000","last_clean":"2024-04-25T13:38:45.359118+0000","last_became_active":"2024-04-25T13:38:45.355823+0000","last_became_peered":"2024-04-25T13:38:45.355823+0000","last_unstale":"2024-04-25T13:38:45.359118+0000","last_undegraded":"2024-04-25T13:38:45.359118+0000","last_fullsized":"2024-04-25T13:38:45.359118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:41.469912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.361638+0000","last_change":"2024-04-25T13:38:45.360940+0000","last_active":"2024-04-25T13:38:45.361638+0000","last_peered":"2024-04-25T13:38:45.361638+0000","last_clean":"2024-04-25T13:38:45.361638+0000","last_became_active":"2024-04-25T13:38:45.360241+0000","last_became_peered":"2024-04-25T13:38:45.360241+0000","last_unstale":"2024-04-25T13:38:45.361638+0000","last_undegraded":"2024-04-25T13:38:45.361638+0000","last_fullsized":"2024-04-25T13:38:45.361638+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:24.369237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.360417+0000","last_change":"2024-04-25T13:38:45.357908+0000","last_active":"2024-04-25T13:38:45.360417+0000","last_peered":"2024-04-25T13:38:45.360417+0000","last_clean":"2024-04-25T13:38:45.360417+0000","last_became_active":"2024-04-25T13:38:45.357491+0000","last_became_peered":"2024-04-25T13:38:45.357491+0000","last_unstale":"2024-04-25T13:38:45.360417+0000","last_undegraded":"2024-04-25T13:38:45.360417+0000","last_fullsized":"2024-04-25T13:38:45.360417+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:07.028650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630975+0000","last_change":"2024-04-25T13:38:45.368137+0000","last_active":"2024-04-25T13:38:46.630975+0000","last_peered":"2024-04-25T13:38:46.630975+0000","last_clean":"2024-04-25T13:38:46.630975+0000","last_became_active":"2024-04-25T13:38:45.367355+0000","last_became_peered":"2024-04-25T13:38:45.367355+0000","last_unstale":"2024-04-25T13:38:46.630975+0000","last_undegraded":"2024-04-25T13:38:46.630975+0000","last_fullsized":"2024-04-25T13:38:46.630975+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:09.502056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.337213+0000","last_change":"2024-04-25T13:38:45.359778+0000","last_active":"2024-04-25T13:38:46.337213+0000","last_peered":"2024-04-25T13:38:46.337213+0000","last_clean":"2024-04-25T13:38:46.337213+0000","last_became_active":"2024-04-25T13:38:45.357189+0000","last_became_peered":"2024-04-25T13:38:45.357189+0000","last_unstale":"2024-04-25T13:38:46.337213+0000","last_undegraded":"2024-04-25T13:38:46.337213+0000","last_fullsized":"2024-04-25T13:38:46.337213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:42:23.469010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629552+0000","last_change":"2024-04-25T13:38:45.353972+0000","last_active":"2024-04-25T13:38:46.629552+0000","last_peered":"2024-04-25T13:38:46.629552+0000","last_clean":"2024-04-25T13:38:46.629552+0000","last_became_active":"2024-04-25T13:38:45.353458+0000","last_became_peered":"2024-04-25T13:38:45.353458+0000","last_unstale":"2024-04-25T13:38:46.629552+0000","last_undegraded":"2024-04-25T13:38:46.629552+0000","last_fullsized":"2024-04-25T13:38:46.629552+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:17.087850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.359785+0000","last_change":"2024-04-25T13:38:45.358876+0000","last_active":"2024-04-25T13:38:45.359785+0000","last_peered":"2024-04-25T13:38:45.359785+0000","last_clean":"2024-04-25T13:38:45.359785+0000","last_became_active":"2024-04-25T13:38:45.358435+0000","last_became_peered":"2024-04-25T13:38:45.358435+0000","last_unstale":"2024-04-25T13:38:45.359785+0000","last_undegraded":"2024-04-25T13:38:45.359785+0000","last_fullsized":"2024-04-25T13:38:45.359785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:05:34.712826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.374660+0000","last_change":"2024-04-25T13:38:45.373557+0000","last_active":"2024-04-25T13:38:45.374660+0000","last_peered":"2024-04-25T13:38:45.374660+0000","last_clean":"2024-04-25T13:38:45.374660+0000","last_became_active":"2024-04-25T13:38:45.373179+0000","last_became_peered":"2024-04-25T13:38:45.373179+0000","last_unstale":"2024-04-25T13:38:45.374660+0000","last_undegraded":"2024-04-25T13:38:45.374660+0000","last_fullsized":"2024-04-25T13:38:45.374660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:02.295945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.360969+0000","last_change":"2024-04-25T13:38:45.358282+0000","last_active":"2024-04-25T13:38:45.360969+0000","last_peered":"2024-04-25T13:38:45.360969+0000","last_clean":"2024-04-25T13:38:45.360969+0000","last_became_active":"2024-04-25T13:38:45.357923+0000","last_became_peered":"2024-04-25T13:38:45.357923+0000","last_unstale":"2024-04-25T13:38:45.360969+0000","last_undegraded":"2024-04-25T13:38:45.360969+0000","last_fullsized":"2024-04-25T13:38:45.360969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:30.165459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336832+0000","last_change":"2024-04-25T13:38:45.357865+0000","last_active":"2024-04-25T13:38:46.336832+0000","last_peered":"2024-04-25T13:38:46.336832+0000","last_clean":"2024-04-25T13:38:46.336832+0000","last_became_active":"2024-04-25T13:38:45.355399+0000","last_became_peered":"2024-04-25T13:38:45.355399+0000","last_unstale":"2024-04-25T13:38:46.336832+0000","last_undegraded":"2024-04-25T13:38:46.336832+0000","last_fullsized":"2024-04-25T13:38:46.336832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:33:41.382462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.367003+0000","last_change":"2024-04-25T13:38:45.350724+0000","last_active":"2024-04-25T13:38:46.367003+0000","last_peered":"2024-04-25T13:38:46.367003+0000","last_clean":"2024-04-25T13:38:46.367003+0000","last_became_active":"2024-04-25T13:38:45.350284+0000","last_became_peered":"2024-04-25T13:38:45.350284+0000","last_unstale":"2024-04-25T13:38:46.367003+0000","last_undegraded":"2024-04-25T13:38:46.367003+0000","last_fullsized":"2024-04-25T13:38:46.367003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:30.960843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.363951+0000","last_change":"2024-04-25T13:38:45.362966+0000","last_active":"2024-04-25T13:38:45.363951+0000","last_peered":"2024-04-25T13:38:45.363951+0000","last_clean":"2024-04-25T13:38:45.363951+0000","last_became_active":"2024-04-25T13:38:45.362585+0000","last_became_peered":"2024-04-25T13:38:45.362585+0000","last_unstale":"2024-04-25T13:38:45.363951+0000","last_undegraded":"2024-04-25T13:38:45.363951+0000","last_fullsized":"2024-04-25T13:38:45.363951+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:57:49.544996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'4","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.366887+0000","last_change":"2024-04-25T13:38:44.354887+0000","last_active":"2024-04-25T13:38:46.366887+0000","last_peered":"2024-04-25T13:38:46.366887+0000","last_clean":"2024-04-25T13:38:46.366887+0000","last_became_active":"2024-04-25T13:38:44.354510+0000","last_became_peered":"2024-04-25T13:38:44.354510+0000","last_unstale":"2024-04-25T13:38:46.366887+0000","last_undegraded":"2024-04-25T13:38:46.366887+0000","last_fullsized":"2024-04-25T13:38:46.366887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:46.457087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.368941+0000","last_change":"2024-04-25T13:38:45.364056+0000","last_active":"2024-04-25T13:38:45.368941+0000","last_peered":"2024-04-25T13:38:45.368941+0000","last_clean":"2024-04-25T13:38:45.368941+0000","last_became_active":"2024-04-25T13:38:45.363572+0000","last_became_peered":"2024-04-25T13:38:45.363572+0000","last_unstale":"2024-04-25T13:38:45.368941+0000","last_undegraded":"2024-04-25T13:38:45.368941+0000","last_fullsized":"2024-04-25T13:38:45.368941+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:40:12.317831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'1","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.379672+0000","last_change":"2024-04-25T13:38:44.362371+0000","last_active":"2024-04-25T13:38:45.379672+0000","last_peered":"2024-04-25T13:38:45.379672+0000","last_clean":"2024-04-25T13:38:45.379672+0000","last_became_active":"2024-04-25T13:38:44.361262+0000","last_became_peered":"2024-04-25T13:38:44.361262+0000","last_unstale":"2024-04-25T13:38:45.379672+0000","last_undegraded":"2024-04-25T13:38:45.379672+0000","last_fullsized":"2024-04-25T13:38:45.379672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:01.442189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.364407+0000","last_change":"2024-04-25T13:38:45.362599+0000","last_active":"2024-04-25T13:38:45.364407+0000","last_peered":"2024-04-25T13:38:45.364407+0000","last_clean":"2024-04-25T13:38:45.364407+0000","last_became_active":"2024-04-25T13:38:45.361893+0000","last_became_peered":"2024-04-25T13:38:45.361893+0000","last_unstale":"2024-04-25T13:38:45.364407+0000","last_undegraded":"2024-04-25T13:38:45.364407+0000","last_fullsized":"2024-04-25T13:38:45.364407+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:08:23.220163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369744+0000","last_change":"2024-04-25T13:38:44.353063+0000","last_active":"2024-04-25T13:38:46.369744+0000","last_peered":"2024-04-25T13:38:46.369744+0000","last_clean":"2024-04-25T13:38:46.369744+0000","last_became_active":"2024-04-25T13:38:44.352403+0000","last_became_peered":"2024-04-25T13:38:44.352403+0000","last_unstale":"2024-04-25T13:38:46.369744+0000","last_undegraded":"2024-04-25T13:38:46.369744+0000","last_fullsized":"2024-04-25T13:38:46.369744+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:09.670589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.361110+0000","last_change":"2024-04-25T13:38:45.359078+0000","last_active":"2024-04-25T13:38:45.361110+0000","last_peered":"2024-04-25T13:38:45.361110+0000","last_clean":"2024-04-25T13:38:45.361110+0000","last_became_active":"2024-04-25T13:38:45.354063+0000","last_became_peered":"2024-04-25T13:38:45.354063+0000","last_unstale":"2024-04-25T13:38:45.361110+0000","last_undegraded":"2024-04-25T13:38:45.361110+0000","last_fullsized":"2024-04-25T13:38:45.361110+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:39:48.770584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369617+0000","last_change":"2024-04-25T13:38:44.349132+0000","last_active":"2024-04-25T13:38:46.369617+0000","last_peered":"2024-04-25T13:38:46.369617+0000","last_clean":"2024-04-25T13:38:46.369617+0000","last_became_active":"2024-04-25T13:38:44.348762+0000","last_became_peered":"2024-04-25T13:38:44.348762+0000","last_unstale":"2024-04-25T13:38:46.369617+0000","last_undegraded":"2024-04-25T13:38:46.369617+0000","last_fullsized":"2024-04-25T13:38:46.369617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:12.990304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629915+0000","last_change":"2024-04-25T13:38:45.369116+0000","last_active":"2024-04-25T13:38:46.629915+0000","last_peered":"2024-04-25T13:38:46.629915+0000","last_clean":"2024-04-25T13:38:46.629915+0000","last_became_active":"2024-04-25T13:38:45.368380+0000","last_became_peered":"2024-04-25T13:38:45.368380+0000","last_unstale":"2024-04-25T13:38:46.629915+0000","last_undegraded":"2024-04-25T13:38:46.629915+0000","last_fullsized":"2024-04-25T13:38:46.629915+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:02.259401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629585+0000","last_change":"2024-04-25T13:38:44.351873+0000","last_active":"2024-04-25T13:38:46.629585+0000","last_peered":"2024-04-25T13:38:46.629585+0000","last_clean":"2024-04-25T13:38:46.629585+0000","last_became_active":"2024-04-25T13:38:44.351518+0000","last_became_peered":"2024-04-25T13:38:44.351518+0000","last_unstale":"2024-04-25T13:38:46.629585+0000","last_undegraded":"2024-04-25T13:38:46.629585+0000","last_fullsized":"2024-04-25T13:38:46.629585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:06.839053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369080+0000","last_change":"2024-04-25T13:38:45.351369+0000","last_active":"2024-04-25T13:38:46.369080+0000","last_peered":"2024-04-25T13:38:46.369080+0000","last_clean":"2024-04-25T13:38:46.369080+0000","last_became_active":"2024-04-25T13:38:45.351018+0000","last_became_peered":"2024-04-25T13:38:45.351018+0000","last_unstale":"2024-04-25T13:38:46.369080+0000","last_undegraded":"2024-04-25T13:38:46.369080+0000","last_fullsized":"2024-04-25T13:38:46.369080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'1","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.378121+0000","last_change":"2024-04-25T13:38:44.345986+0000","last_active":"2024-04-25T13:38:45.378121+0000","last_peered":"2024-04-25T13:38:45.378121+0000","last_clean":"2024-04-25T13:38:45.378121+0000","last_became_active":"2024-04-25T13:38:44.345600+0000","last_became_peered":"2024-04-25T13:38:44.345600+0000","last_unstale":"2024-04-25T13:38:45.378121+0000","last_undegraded":"2024-04-25T13:38:45.378121+0000","last_fullsized":"2024-04-25T13:38:45.378121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:52.749192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629861+0000","last_change":"2024-04-25T13:38:45.368641+0000","last_active":"2024-04-25T13:38:46.629861+0000","last_peered":"2024-04-25T13:38:46.629861+0000","last_clean":"2024-04-25T13:38:46.629861+0000","last_became_active":"2024-04-25T13:38:45.368209+0000","last_became_peered":"2024-04-25T13:38:45.368209+0000","last_unstale":"2024-04-25T13:38:46.629861+0000","last_undegraded":"2024-04-25T13:38:46.629861+0000","last_fullsized":"2024-04-25T13:38:46.629861+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:47.990808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629102+0000","last_change":"2024-04-25T13:38:44.339927+0000","last_active":"2024-04-25T13:38:46.629102+0000","last_peered":"2024-04-25T13:38:46.629102+0000","last_clean":"2024-04-25T13:38:46.629102+0000","last_became_active":"2024-04-25T13:38:44.339588+0000","last_became_peered":"2024-04-25T13:38:44.339588+0000","last_unstale":"2024-04-25T13:38:46.629102+0000","last_undegraded":"2024-04-25T13:38:46.629102+0000","last_fullsized":"2024-04-25T13:38:46.629102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:51.707750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.359143+0000","last_change":"2024-04-25T13:38:45.357323+0000","last_active":"2024-04-25T13:38:45.359143+0000","last_peered":"2024-04-25T13:38:45.359143+0000","last_clean":"2024-04-25T13:38:45.359143+0000","last_became_active":"2024-04-25T13:38:45.356870+0000","last_became_peered":"2024-04-25T13:38:45.356870+0000","last_unstale":"2024-04-25T13:38:45.359143+0000","last_undegraded":"2024-04-25T13:38:45.359143+0000","last_fullsized":"2024-04-25T13:38:45.359143+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:36.805466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629345+0000","last_change":"2024-04-25T13:38:44.351294+0000","last_active":"2024-04-25T13:38:46.629345+0000","last_peered":"2024-04-25T13:38:46.629345+0000","last_clean":"2024-04-25T13:38:46.629345+0000","last_became_active":"2024-04-25T13:38:44.350971+0000","last_became_peered":"2024-04-25T13:38:44.350971+0000","last_unstale":"2024-04-25T13:38:46.629345+0000","last_undegraded":"2024-04-25T13:38:46.629345+0000","last_fullsized":"2024-04-25T13:38:46.629345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:22.452217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.366788+0000","last_change":"2024-04-25T13:38:45.345943+0000","last_active":"2024-04-25T13:38:46.366788+0000","last_peered":"2024-04-25T13:38:46.366788+0000","last_clean":"2024-04-25T13:38:46.366788+0000","last_became_active":"2024-04-25T13:38:45.345528+0000","last_became_peered":"2024-04-25T13:38:45.345528+0000","last_unstale":"2024-04-25T13:38:46.366788+0000","last_undegraded":"2024-04-25T13:38:46.366788+0000","last_fullsized":"2024-04-25T13:38:46.366788+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:57.327303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.336243+0000","last_change":"2024-04-25T13:38:44.343032+0000","last_active":"2024-04-25T13:38:45.336243+0000","last_peered":"2024-04-25T13:38:45.336243+0000","last_clean":"2024-04-25T13:38:45.336243+0000","last_became_active":"2024-04-25T13:38:44.342613+0000","last_became_peered":"2024-04-25T13:38:44.342613+0000","last_unstale":"2024-04-25T13:38:45.336243+0000","last_undegraded":"2024-04-25T13:38:45.336243+0000","last_fullsized":"2024-04-25T13:38:45.336243+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:33.957150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.360688+0000","last_change":"2024-04-25T13:38:45.354534+0000","last_active":"2024-04-25T13:38:45.360688+0000","last_peered":"2024-04-25T13:38:45.360688+0000","last_clean":"2024-04-25T13:38:45.360688+0000","last_became_active":"2024-04-25T13:38:45.354147+0000","last_became_peered":"2024-04-25T13:38:45.354147+0000","last_unstale":"2024-04-25T13:38:45.360688+0000","last_undegraded":"2024-04-25T13:38:45.360688+0000","last_fullsized":"2024-04-25T13:38:45.360688+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:17.060436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629966+0000","last_change":"2024-04-25T13:38:44.356103+0000","last_active":"2024-04-25T13:38:46.629966+0000","last_peered":"2024-04-25T13:38:46.629966+0000","last_clean":"2024-04-25T13:38:46.629966+0000","last_became_active":"2024-04-25T13:38:44.355702+0000","last_became_peered":"2024-04-25T13:38:44.355702+0000","last_unstale":"2024-04-25T13:38:46.629966+0000","last_undegraded":"2024-04-25T13:38:46.629966+0000","last_fullsized":"2024-04-25T13:38:46.629966+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:47.572922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.374654+0000","last_change":"2024-04-25T13:38:45.373843+0000","last_active":"2024-04-25T13:38:45.374654+0000","last_peered":"2024-04-25T13:38:45.374654+0000","last_clean":"2024-04-25T13:38:45.374654+0000","last_became_active":"2024-04-25T13:38:45.373423+0000","last_became_peered":"2024-04-25T13:38:45.373423+0000","last_unstale":"2024-04-25T13:38:45.374654+0000","last_undegraded":"2024-04-25T13:38:45.374654+0000","last_fullsized":"2024-04-25T13:38:45.374654+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:53.820952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630028+0000","last_change":"2024-04-25T13:38:44.355474+0000","last_active":"2024-04-25T13:38:46.630028+0000","last_peered":"2024-04-25T13:38:46.630028+0000","last_clean":"2024-04-25T13:38:46.630028+0000","last_became_active":"2024-04-25T13:38:44.355113+0000","last_became_peered":"2024-04-25T13:38:44.355113+0000","last_unstale":"2024-04-25T13:38:46.630028+0000","last_undegraded":"2024-04-25T13:38:46.630028+0000","last_fullsized":"2024-04-25T13:38:46.630028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.242974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.370643+0000","last_change":"2024-04-25T13:38:45.369618+0000","last_active":"2024-04-25T13:38:45.370643+0000","last_peered":"2024-04-25T13:38:45.370643+0000","last_clean":"2024-04-25T13:38:45.370643+0000","last_became_active":"2024-04-25T13:38:45.369252+0000","last_became_peered":"2024-04-25T13:38:45.369252+0000","last_unstale":"2024-04-25T13:38:45.370643+0000","last_undegraded":"2024-04-25T13:38:45.370643+0000","last_fullsized":"2024-04-25T13:38:45.370643+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:45.290569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"75'6","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.379571+0000","last_change":"2024-04-25T13:38:44.349850+0000","last_active":"2024-04-25T13:38:45.379571+0000","last_peered":"2024-04-25T13:38:45.379571+0000","last_clean":"2024-04-25T13:38:45.379571+0000","last_became_active":"2024-04-25T13:38:44.349404+0000","last_became_peered":"2024-04-25T13:38:44.349404+0000","last_unstale":"2024-04-25T13:38:45.379571+0000","last_undegraded":"2024-04-25T13:38:45.379571+0000","last_fullsized":"2024-04-25T13:38:45.379571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:40.025684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369427+0000","last_change":"2024-04-25T13:38:45.371554+0000","last_active":"2024-04-25T13:38:46.369427+0000","last_peered":"2024-04-25T13:38:46.369427+0000","last_clean":"2024-04-25T13:38:46.369427+0000","last_became_active":"2024-04-25T13:38:45.371042+0000","last_became_peered":"2024-04-25T13:38:45.371042+0000","last_unstale":"2024-04-25T13:38:46.369427+0000","last_undegraded":"2024-04-25T13:38:46.369427+0000","last_fullsized":"2024-04-25T13:38:46.369427+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:59.240536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'2","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.377074+0000","last_change":"2024-04-25T13:38:44.345979+0000","last_active":"2024-04-25T13:38:45.377074+0000","last_peered":"2024-04-25T13:38:45.377074+0000","last_clean":"2024-04-25T13:38:45.377074+0000","last_became_active":"2024-04-25T13:38:44.345596+0000","last_became_peered":"2024-04-25T13:38:44.345596+0000","last_unstale":"2024-04-25T13:38:45.377074+0000","last_undegraded":"2024-04-25T13:38:45.377074+0000","last_fullsized":"2024-04-25T13:38:45.377074+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:22:51.539024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629285+0000","last_change":"2024-04-25T13:38:45.364794+0000","last_active":"2024-04-25T13:38:46.629285+0000","last_peered":"2024-04-25T13:38:46.629285+0000","last_clean":"2024-04-25T13:38:46.629285+0000","last_became_active":"2024-04-25T13:38:45.364381+0000","last_became_peered":"2024-04-25T13:38:45.364381+0000","last_unstale":"2024-04-25T13:38:46.629285+0000","last_undegraded":"2024-04-25T13:38:46.629285+0000","last_fullsized":"2024-04-25T13:38:46.629285+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:16.856980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629836+0000","last_change":"2024-04-25T13:38:44.351269+0000","last_active":"2024-04-25T13:38:46.629836+0000","last_peered":"2024-04-25T13:38:46.629836+0000","last_clean":"2024-04-25T13:38:46.629836+0000","last_became_active":"2024-04-25T13:38:44.350933+0000","last_became_peered":"2024-04-25T13:38:44.350933+0000","last_unstale":"2024-04-25T13:38:46.629836+0000","last_undegraded":"2024-04-25T13:38:46.629836+0000","last_fullsized":"2024-04-25T13:38:46.629836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:02.677849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629790+0000","last_change":"2024-04-25T13:38:45.362732+0000","last_active":"2024-04-25T13:38:46.629790+0000","last_peered":"2024-04-25T13:38:46.629790+0000","last_clean":"2024-04-25T13:38:46.629790+0000","last_became_active":"2024-04-25T13:38:45.361957+0000","last_became_peered":"2024-04-25T13:38:45.361957+0000","last_unstale":"2024-04-25T13:38:46.629790+0000","last_undegraded":"2024-04-25T13:38:46.629790+0000","last_fullsized":"2024-04-25T13:38:46.629790+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:50.011442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336724+0000","last_change":"2024-04-25T13:38:44.349378+0000","last_active":"2024-04-25T13:38:46.336724+0000","last_peered":"2024-04-25T13:38:46.336724+0000","last_clean":"2024-04-25T13:38:46.336724+0000","last_became_active":"2024-04-25T13:38:44.348887+0000","last_became_peered":"2024-04-25T13:38:44.348887+0000","last_unstale":"2024-04-25T13:38:46.336724+0000","last_undegraded":"2024-04-25T13:38:46.336724+0000","last_fullsized":"2024-04-25T13:38:46.336724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.747205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631305+0000","last_change":"2024-04-25T13:38:45.349941+0000","last_active":"2024-04-25T13:38:46.631305+0000","last_peered":"2024-04-25T13:38:46.631305+0000","last_clean":"2024-04-25T13:38:46.631305+0000","last_became_active":"2024-04-25T13:38:45.349590+0000","last_became_peered":"2024-04-25T13:38:45.349590+0000","last_unstale":"2024-04-25T13:38:46.631305+0000","last_undegraded":"2024-04-25T13:38:46.631305+0000","last_fullsized":"2024-04-25T13:38:46.631305+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:42.904406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'8","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.368996+0000","last_change":"2024-04-25T13:38:44.354082+0000","last_active":"2024-04-25T13:38:46.368996+0000","last_peered":"2024-04-25T13:38:46.368996+0000","last_clean":"2024-04-25T13:38:46.368996+0000","last_became_active":"2024-04-25T13:38:44.353763+0000","last_became_peered":"2024-04-25T13:38:44.353763+0000","last_unstale":"2024-04-25T13:38:46.368996+0000","last_undegraded":"2024-04-25T13:38:46.368996+0000","last_fullsized":"2024-04-25T13:38:46.368996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:19:12.838644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630054+0000","last_change":"2024-04-25T13:38:45.368589+0000","last_active":"2024-04-25T13:38:46.630054+0000","last_peered":"2024-04-25T13:38:46.630054+0000","last_clean":"2024-04-25T13:38:46.630054+0000","last_became_active":"2024-04-25T13:38:45.368079+0000","last_became_peered":"2024-04-25T13:38:45.368079+0000","last_unstale":"2024-04-25T13:38:46.630054+0000","last_undegraded":"2024-04-25T13:38:46.630054+0000","last_fullsized":"2024-04-25T13:38:46.630054+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:10.180264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369415+0000","last_change":"2024-04-25T13:38:44.345148+0000","last_active":"2024-04-25T13:38:46.369415+0000","last_peered":"2024-04-25T13:38:46.369415+0000","last_clean":"2024-04-25T13:38:46.369415+0000","last_became_active":"2024-04-25T13:38:44.344816+0000","last_became_peered":"2024-04-25T13:38:44.344816+0000","last_unstale":"2024-04-25T13:38:46.369415+0000","last_undegraded":"2024-04-25T13:38:46.369415+0000","last_fullsized":"2024-04-25T13:38:46.369415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:03:38.134169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336647+0000","last_change":"2024-04-25T13:38:45.354340+0000","last_active":"2024-04-25T13:38:46.336647+0000","last_peered":"2024-04-25T13:38:46.336647+0000","last_clean":"2024-04-25T13:38:46.336647+0000","last_became_active":"2024-04-25T13:38:45.353933+0000","last_became_peered":"2024-04-25T13:38:45.353933+0000","last_unstale":"2024-04-25T13:38:46.336647+0000","last_undegraded":"2024-04-25T13:38:46.336647+0000","last_fullsized":"2024-04-25T13:38:46.336647+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:07.619051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.366794+0000","last_change":"2024-04-25T13:38:44.351183+0000","last_active":"2024-04-25T13:38:46.366794+0000","last_peered":"2024-04-25T13:38:46.366794+0000","last_clean":"2024-04-25T13:38:46.366794+0000","last_became_active":"2024-04-25T13:38:44.350785+0000","last_became_peered":"2024-04-25T13:38:44.350785+0000","last_unstale":"2024-04-25T13:38:46.366794+0000","last_undegraded":"2024-04-25T13:38:46.366794+0000","last_fullsized":"2024-04-25T13:38:46.366794+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:37:57.465012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.365571+0000","last_change":"2024-04-25T13:38:45.362355+0000","last_active":"2024-04-25T13:38:45.365571+0000","last_peered":"2024-04-25T13:38:45.365571+0000","last_clean":"2024-04-25T13:38:45.365571+0000","last_became_active":"2024-04-25T13:38:45.361996+0000","last_became_peered":"2024-04-25T13:38:45.361996+0000","last_unstale":"2024-04-25T13:38:45.365571+0000","last_undegraded":"2024-04-25T13:38:45.365571+0000","last_fullsized":"2024-04-25T13:38:45.365571+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:43.334146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630424+0000","last_change":"2024-04-25T13:38:44.356079+0000","last_active":"2024-04-25T13:38:46.630424+0000","last_peered":"2024-04-25T13:38:46.630424+0000","last_clean":"2024-04-25T13:38:46.630424+0000","last_became_active":"2024-04-25T13:38:44.355660+0000","last_became_peered":"2024-04-25T13:38:44.355660+0000","last_unstale":"2024-04-25T13:38:46.630424+0000","last_undegraded":"2024-04-25T13:38:46.630424+0000","last_fullsized":"2024-04-25T13:38:46.630424+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:35.161337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.357428+0000","last_change":"2024-04-25T13:38:45.356578+0000","last_active":"2024-04-25T13:38:45.357428+0000","last_peered":"2024-04-25T13:38:45.357428+0000","last_clean":"2024-04-25T13:38:45.357428+0000","last_became_active":"2024-04-25T13:38:45.356232+0000","last_became_peered":"2024-04-25T13:38:45.356232+0000","last_unstale":"2024-04-25T13:38:45.357428+0000","last_undegraded":"2024-04-25T13:38:45.357428+0000","last_fullsized":"2024-04-25T13:38:45.357428+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.127923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630278+0000","last_change":"2024-04-25T13:38:45.361209+0000","last_active":"2024-04-25T13:38:46.630278+0000","last_peered":"2024-04-25T13:38:46.630278+0000","last_clean":"2024-04-25T13:38:46.630278+0000","last_became_active":"2024-04-25T13:38:45.360830+0000","last_became_peered":"2024-04-25T13:38:45.360830+0000","last_unstale":"2024-04-25T13:38:46.630278+0000","last_undegraded":"2024-04-25T13:38:46.630278+0000","last_fullsized":"2024-04-25T13:38:46.630278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:59.661146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.628983+0000","last_change":"2024-04-25T13:38:44.351113+0000","last_active":"2024-04-25T13:38:46.628983+0000","last_peered":"2024-04-25T13:38:46.628983+0000","last_clean":"2024-04-25T13:38:46.628983+0000","last_became_active":"2024-04-25T13:38:44.350730+0000","last_became_peered":"2024-04-25T13:38:44.350730+0000","last_unstale":"2024-04-25T13:38:46.628983+0000","last_undegraded":"2024-04-25T13:38:46.628983+0000","last_fullsized":"2024-04-25T13:38:46.628983+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:38:22.747930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.359142+0000","last_change":"2024-04-25T13:38:45.356731+0000","last_active":"2024-04-25T13:38:45.359142+0000","last_peered":"2024-04-25T13:38:45.359142+0000","last_clean":"2024-04-25T13:38:45.359142+0000","last_became_active":"2024-04-25T13:38:45.355531+0000","last_became_peered":"2024-04-25T13:38:45.355531+0000","last_unstale":"2024-04-25T13:38:45.359142+0000","last_undegraded":"2024-04-25T13:38:45.359142+0000","last_fullsized":"2024-04-25T13:38:45.359142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:07.308500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336877+0000","last_change":"2024-04-25T13:38:44.345941+0000","last_active":"2024-04-25T13:38:46.336877+0000","last_peered":"2024-04-25T13:38:46.336877+0000","last_clean":"2024-04-25T13:38:46.336877+0000","last_became_active":"2024-04-25T13:38:44.344634+0000","last_became_peered":"2024-04-25T13:38:44.344634+0000","last_unstale":"2024-04-25T13:38:46.336877+0000","last_undegraded":"2024-04-25T13:38:46.336877+0000","last_fullsized":"2024-04-25T13:38:46.336877+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:42.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":89,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631246+0000","last_change":"2024-04-25T13:37:27.903345+0000","last_active":"2024-04-25T13:38:46.631246+0000","last_peered":"2024-04-25T13:38:46.631246+0000","last_clean":"2024-04-25T13:38:46.631246+0000","last_became_active":"2024-04-25T13:37:27.897986+0000","last_became_peered":"2024-04-25T13:37:27.897986+0000","last_unstale":"2024-04-25T13:38:46.631246+0000","last_undegraded":"2024-04-25T13:38:46.631246+0000","last_fullsized":"2024-04-25T13:38:46.631246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_clean_scrub_stamp":"2024-04-25T13:33:56.569417+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:56.390458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.628832+0000","last_change":"2024-04-25T13:38:45.369390+0000","last_active":"2024-04-25T13:38:46.628832+0000","last_peered":"2024-04-25T13:38:46.628832+0000","last_clean":"2024-04-25T13:38:46.628832+0000","last_became_active":"2024-04-25T13:38:45.368938+0000","last_became_peered":"2024-04-25T13:38:45.368938+0000","last_unstale":"2024-04-25T13:38:46.628832+0000","last_undegraded":"2024-04-25T13:38:46.628832+0000","last_fullsized":"2024-04-25T13:38:46.628832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:11.942204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631434+0000","last_change":"2024-04-25T13:38:44.355585+0000","last_active":"2024-04-25T13:38:46.631434+0000","last_peered":"2024-04-25T13:38:46.631434+0000","last_clean":"2024-04-25T13:38:46.631434+0000","last_became_active":"2024-04-25T13:38:44.355131+0000","last_became_peered":"2024-04-25T13:38:44.355131+0000","last_unstale":"2024-04-25T13:38:46.631434+0000","last_undegraded":"2024-04-25T13:38:46.631434+0000","last_fullsized":"2024-04-25T13:38:46.631434+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:05.240247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369572+0000","last_change":"2024-04-25T13:38:44.355106+0000","last_active":"2024-04-25T13:38:46.369572+0000","last_peered":"2024-04-25T13:38:46.369572+0000","last_clean":"2024-04-25T13:38:46.369572+0000","last_became_active":"2024-04-25T13:38:44.354727+0000","last_became_peered":"2024-04-25T13:38:44.354727+0000","last_unstale":"2024-04-25T13:38:46.369572+0000","last_undegraded":"2024-04-25T13:38:46.369572+0000","last_fullsized":"2024-04-25T13:38:46.369572+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:29.606589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629161+0000","last_change":"2024-04-25T13:38:45.366187+0000","last_active":"2024-04-25T13:38:46.629161+0000","last_peered":"2024-04-25T13:38:46.629161+0000","last_clean":"2024-04-25T13:38:46.629161+0000","last_became_active":"2024-04-25T13:38:45.365744+0000","last_became_peered":"2024-04-25T13:38:45.365744+0000","last_unstale":"2024-04-25T13:38:46.629161+0000","last_undegraded":"2024-04-25T13:38:46.629161+0000","last_fullsized":"2024-04-25T13:38:46.629161+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:07.970735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.366921+0000","last_change":"2024-04-25T13:38:44.359250+0000","last_active":"2024-04-25T13:38:46.366921+0000","last_peered":"2024-04-25T13:38:46.366921+0000","last_clean":"2024-04-25T13:38:46.366921+0000","last_became_active":"2024-04-25T13:38:44.358874+0000","last_became_peered":"2024-04-25T13:38:44.358874+0000","last_unstale":"2024-04-25T13:38:46.366921+0000","last_undegraded":"2024-04-25T13:38:46.366921+0000","last_fullsized":"2024-04-25T13:38:46.366921+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:28.820465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.375429+0000","last_change":"2024-04-25T13:38:45.370080+0000","last_active":"2024-04-25T13:38:45.375429+0000","last_peered":"2024-04-25T13:38:45.375429+0000","last_clean":"2024-04-25T13:38:45.375429+0000","last_became_active":"2024-04-25T13:38:45.369668+0000","last_became_peered":"2024-04-25T13:38:45.369668+0000","last_unstale":"2024-04-25T13:38:45.375429+0000","last_undegraded":"2024-04-25T13:38:45.375429+0000","last_fullsized":"2024-04-25T13:38:45.375429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:39.455859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.337276+0000","last_change":"2024-04-25T13:38:44.358907+0000","last_active":"2024-04-25T13:38:46.337276+0000","last_peered":"2024-04-25T13:38:46.337276+0000","last_clean":"2024-04-25T13:38:46.337276+0000","last_became_active":"2024-04-25T13:38:44.358455+0000","last_became_peered":"2024-04-25T13:38:44.358455+0000","last_unstale":"2024-04-25T13:38:46.337276+0000","last_undegraded":"2024-04-25T13:38:46.337276+0000","last_fullsized":"2024-04-25T13:38:46.337276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:50:52.916433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.374574+0000","last_change":"2024-04-25T13:38:45.373050+0000","last_active":"2024-04-25T13:38:45.374574+0000","last_peered":"2024-04-25T13:38:45.374574+0000","last_clean":"2024-04-25T13:38:45.374574+0000","last_became_active":"2024-04-25T13:38:45.372621+0000","last_became_peered":"2024-04-25T13:38:45.372621+0000","last_unstale":"2024-04-25T13:38:45.374574+0000","last_undegraded":"2024-04-25T13:38:45.374574+0000","last_fullsized":"2024-04-25T13:38:45.374574+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:59:06.287745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.345284+0000","last_change":"2024-04-25T13:38:44.355768+0000","last_active":"2024-04-25T13:38:45.345284+0000","last_peered":"2024-04-25T13:38:45.345284+0000","last_clean":"2024-04-25T13:38:45.345284+0000","last_became_active":"2024-04-25T13:38:44.355400+0000","last_became_peered":"2024-04-25T13:38:44.355400+0000","last_unstale":"2024-04-25T13:38:45.345284+0000","last_undegraded":"2024-04-25T13:38:45.345284+0000","last_fullsized":"2024-04-25T13:38:45.345284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:38.533717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369168+0000","last_change":"2024-04-25T13:38:45.350582+0000","last_active":"2024-04-25T13:38:46.369168+0000","last_peered":"2024-04-25T13:38:46.369168+0000","last_clean":"2024-04-25T13:38:46.369168+0000","last_became_active":"2024-04-25T13:38:45.350220+0000","last_became_peered":"2024-04-25T13:38:45.350220+0000","last_unstale":"2024-04-25T13:38:46.369168+0000","last_undegraded":"2024-04-25T13:38:46.369168+0000","last_fullsized":"2024-04-25T13:38:46.369168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.463519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.350648+0000","last_change":"2024-04-25T13:38:44.360459+0000","last_active":"2024-04-25T13:38:45.350648+0000","last_peered":"2024-04-25T13:38:45.350648+0000","last_clean":"2024-04-25T13:38:45.350648+0000","last_became_active":"2024-04-25T13:38:44.359709+0000","last_became_peered":"2024-04-25T13:38:44.359709+0000","last_unstale":"2024-04-25T13:38:45.350648+0000","last_undegraded":"2024-04-25T13:38:45.350648+0000","last_fullsized":"2024-04-25T13:38:45.350648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:53.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336393+0000","last_change":"2024-04-25T13:38:45.357291+0000","last_active":"2024-04-25T13:38:46.336393+0000","last_peered":"2024-04-25T13:38:46.336393+0000","last_clean":"2024-04-25T13:38:46.336393+0000","last_became_active":"2024-04-25T13:38:45.356929+0000","last_became_peered":"2024-04-25T13:38:45.356929+0000","last_unstale":"2024-04-25T13:38:46.336393+0000","last_undegraded":"2024-04-25T13:38:46.336393+0000","last_fullsized":"2024-04-25T13:38:46.336393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:02:08.619208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.350914+0000","last_change":"2024-04-25T13:38:44.343885+0000","last_active":"2024-04-25T13:38:45.350914+0000","last_peered":"2024-04-25T13:38:45.350914+0000","last_clean":"2024-04-25T13:38:45.350914+0000","last_became_active":"2024-04-25T13:38:44.343541+0000","last_became_peered":"2024-04-25T13:38:44.343541+0000","last_unstale":"2024-04-25T13:38:45.350914+0000","last_undegraded":"2024-04-25T13:38:45.350914+0000","last_fullsized":"2024-04-25T13:38:45.350914+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:50:01.904593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.375976+0000","last_change":"2024-04-25T13:38:45.371508+0000","last_active":"2024-04-25T13:38:45.375976+0000","last_peered":"2024-04-25T13:38:45.375976+0000","last_clean":"2024-04-25T13:38:45.375976+0000","last_became_active":"2024-04-25T13:38:45.370912+0000","last_became_peered":"2024-04-25T13:38:45.370912+0000","last_unstale":"2024-04-25T13:38:45.375976+0000","last_undegraded":"2024-04-25T13:38:45.375976+0000","last_fullsized":"2024-04-25T13:38:45.375976+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:50.124430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369186+0000","last_change":"2024-04-25T13:38:44.352794+0000","last_active":"2024-04-25T13:38:46.369186+0000","last_peered":"2024-04-25T13:38:46.369186+0000","last_clean":"2024-04-25T13:38:46.369186+0000","last_became_active":"2024-04-25T13:38:44.352422+0000","last_became_peered":"2024-04-25T13:38:44.352422+0000","last_unstale":"2024-04-25T13:38:46.369186+0000","last_undegraded":"2024-04-25T13:38:46.369186+0000","last_fullsized":"2024-04-25T13:38:46.369186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:40.543783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630622+0000","last_change":"2024-04-25T13:38:45.363322+0000","last_active":"2024-04-25T13:38:46.630622+0000","last_peered":"2024-04-25T13:38:46.630622+0000","last_clean":"2024-04-25T13:38:46.630622+0000","last_became_active":"2024-04-25T13:38:45.362948+0000","last_became_peered":"2024-04-25T13:38:45.362948+0000","last_unstale":"2024-04-25T13:38:46.630622+0000","last_undegraded":"2024-04-25T13:38:46.630622+0000","last_fullsized":"2024-04-25T13:38:46.630622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:20.815774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.353923+0000","last_change":"2024-04-25T13:38:44.358588+0000","last_active":"2024-04-25T13:38:45.353923+0000","last_peered":"2024-04-25T13:38:45.353923+0000","last_clean":"2024-04-25T13:38:45.353923+0000","last_became_active":"2024-04-25T13:38:44.358269+0000","last_became_peered":"2024-04-25T13:38:44.358269+0000","last_unstale":"2024-04-25T13:38:45.353923+0000","last_undegraded":"2024-04-25T13:38:45.353923+0000","last_fullsized":"2024-04-25T13:38:45.353923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:35.405572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631000+0000","last_change":"2024-04-25T13:38:45.369272+0000","last_active":"2024-04-25T13:38:46.631000+0000","last_peered":"2024-04-25T13:38:46.631000+0000","last_clean":"2024-04-25T13:38:46.631000+0000","last_became_active":"2024-04-25T13:38:45.368774+0000","last_became_peered":"2024-04-25T13:38:45.368774+0000","last_unstale":"2024-04-25T13:38:46.631000+0000","last_undegraded":"2024-04-25T13:38:46.631000+0000","last_fullsized":"2024-04-25T13:38:46.631000+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:39.149557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629863+0000","last_change":"2024-04-25T13:38:44.349479+0000","last_active":"2024-04-25T13:38:46.629863+0000","last_peered":"2024-04-25T13:38:46.629863+0000","last_clean":"2024-04-25T13:38:46.629863+0000","last_became_active":"2024-04-25T13:38:44.349104+0000","last_became_peered":"2024-04-25T13:38:44.349104+0000","last_unstale":"2024-04-25T13:38:46.629863+0000","last_undegraded":"2024-04-25T13:38:46.629863+0000","last_fullsized":"2024-04-25T13:38:46.629863+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:39.257472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630655+0000","last_change":"2024-04-25T13:38:45.362120+0000","last_active":"2024-04-25T13:38:46.630655+0000","last_peered":"2024-04-25T13:38:46.630655+0000","last_clean":"2024-04-25T13:38:46.630655+0000","last_became_active":"2024-04-25T13:38:45.361402+0000","last_became_peered":"2024-04-25T13:38:45.361402+0000","last_unstale":"2024-04-25T13:38:46.630655+0000","last_undegraded":"2024-04-25T13:38:46.630655+0000","last_fullsized":"2024-04-25T13:38:46.630655+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:32.117236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629299+0000","last_change":"2024-04-25T13:38:44.349502+0000","last_active":"2024-04-25T13:38:46.629299+0000","last_peered":"2024-04-25T13:38:46.629299+0000","last_clean":"2024-04-25T13:38:46.629299+0000","last_became_active":"2024-04-25T13:38:44.349127+0000","last_became_peered":"2024-04-25T13:38:44.349127+0000","last_unstale":"2024-04-25T13:38:46.629299+0000","last_undegraded":"2024-04-25T13:38:46.629299+0000","last_fullsized":"2024-04-25T13:38:46.629299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:34:50.673352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631069+0000","last_change":"2024-04-25T13:38:45.364746+0000","last_active":"2024-04-25T13:38:46.631069+0000","last_peered":"2024-04-25T13:38:46.631069+0000","last_clean":"2024-04-25T13:38:46.631069+0000","last_became_active":"2024-04-25T13:38:45.364364+0000","last_became_peered":"2024-04-25T13:38:45.364364+0000","last_unstale":"2024-04-25T13:38:46.631069+0000","last_undegraded":"2024-04-25T13:38:46.631069+0000","last_fullsized":"2024-04-25T13:38:46.631069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:42.145637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336400+0000","last_change":"2024-04-25T13:38:45.356683+0000","last_active":"2024-04-25T13:38:46.336400+0000","last_peered":"2024-04-25T13:38:46.336400+0000","last_clean":"2024-04-25T13:38:46.336400+0000","last_became_active":"2024-04-25T13:38:45.356048+0000","last_became_peered":"2024-04-25T13:38:45.356048+0000","last_unstale":"2024-04-25T13:38:46.336400+0000","last_undegraded":"2024-04-25T13:38:46.336400+0000","last_fullsized":"2024-04-25T13:38:46.336400+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:49.578317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.359308+0000","last_change":"2024-04-25T13:38:45.358298+0000","last_active":"2024-04-25T13:38:45.359308+0000","last_peered":"2024-04-25T13:38:45.359308+0000","last_clean":"2024-04-25T13:38:45.359308+0000","last_became_active":"2024-04-25T13:38:45.357986+0000","last_became_peered":"2024-04-25T13:38:45.357986+0000","last_unstale":"2024-04-25T13:38:45.359308+0000","last_undegraded":"2024-04-25T13:38:45.359308+0000","last_fullsized":"2024-04-25T13:38:45.359308+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:39.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.363965+0000","last_change":"2024-04-25T13:38:45.363043+0000","last_active":"2024-04-25T13:38:45.363965+0000","last_peered":"2024-04-25T13:38:45.363965+0000","last_clean":"2024-04-25T13:38:45.363965+0000","last_became_active":"2024-04-25T13:38:45.362686+0000","last_became_peered":"2024-04-25T13:38:45.362686+0000","last_unstale":"2024-04-25T13:38:45.363965+0000","last_undegraded":"2024-04-25T13:38:45.363965+0000","last_fullsized":"2024-04-25T13:38:45.363965+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:13.306214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.374624+0000","last_change":"2024-04-25T13:38:45.373256+0000","last_active":"2024-04-25T13:38:45.374624+0000","last_peered":"2024-04-25T13:38:45.374624+0000","last_clean":"2024-04-25T13:38:45.374624+0000","last_became_active":"2024-04-25T13:38:45.372844+0000","last_became_peered":"2024-04-25T13:38:45.372844+0000","last_unstale":"2024-04-25T13:38:45.374624+0000","last_undegraded":"2024-04-25T13:38:45.374624+0000","last_fullsized":"2024-04-25T13:38:45.374624+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:56.716980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629218+0000","last_change":"2024-04-25T13:38:45.370007+0000","last_active":"2024-04-25T13:38:46.629218+0000","last_peered":"2024-04-25T13:38:46.629218+0000","last_clean":"2024-04-25T13:38:46.629218+0000","last_became_active":"2024-04-25T13:38:45.369606+0000","last_became_peered":"2024-04-25T13:38:45.369606+0000","last_unstale":"2024-04-25T13:38:46.629218+0000","last_undegraded":"2024-04-25T13:38:46.629218+0000","last_fullsized":"2024-04-25T13:38:46.629218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:45.514756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336579+0000","last_change":"2024-04-25T13:38:45.362075+0000","last_active":"2024-04-25T13:38:46.336579+0000","last_peered":"2024-04-25T13:38:46.336579+0000","last_clean":"2024-04-25T13:38:46.336579+0000","last_became_active":"2024-04-25T13:38:45.361720+0000","last_became_peered":"2024-04-25T13:38:45.361720+0000","last_unstale":"2024-04-25T13:38:46.336579+0000","last_undegraded":"2024-04-25T13:38:46.336579+0000","last_fullsized":"2024-04-25T13:38:46.336579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:36.169185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.356175+0000","last_change":"2024-04-25T13:38:45.352101+0000","last_active":"2024-04-25T13:38:45.356175+0000","last_peered":"2024-04-25T13:38:45.356175+0000","last_clean":"2024-04-25T13:38:45.356175+0000","last_became_active":"2024-04-25T13:38:45.351739+0000","last_became_peered":"2024-04-25T13:38:45.351739+0000","last_unstale":"2024-04-25T13:38:45.356175+0000","last_undegraded":"2024-04-25T13:38:45.356175+0000","last_fullsized":"2024-04-25T13:38:45.356175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:33.613263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.366505+0000","last_change":"2024-04-25T13:38:45.363855+0000","last_active":"2024-04-25T13:38:45.366505+0000","last_peered":"2024-04-25T13:38:45.366505+0000","last_clean":"2024-04-25T13:38:45.366505+0000","last_became_active":"2024-04-25T13:38:45.363377+0000","last_became_peered":"2024-04-25T13:38:45.363377+0000","last_unstale":"2024-04-25T13:38:45.366505+0000","last_undegraded":"2024-04-25T13:38:45.366505+0000","last_fullsized":"2024-04-25T13:38:45.366505+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:56.687548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629479+0000","last_change":"2024-04-25T13:38:45.359844+0000","last_active":"2024-04-25T13:38:46.629479+0000","last_peered":"2024-04-25T13:38:46.629479+0000","last_clean":"2024-04-25T13:38:46.629479+0000","last_became_active":"2024-04-25T13:38:45.359428+0000","last_became_peered":"2024-04-25T13:38:45.359428+0000","last_unstale":"2024-04-25T13:38:46.629479+0000","last_undegraded":"2024-04-25T13:38:46.629479+0000","last_fullsized":"2024-04-25T13:38:46.629479+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:01.370409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.375317+0000","last_change":"2024-04-25T13:38:45.370415+0000","last_active":"2024-04-25T13:38:45.375317+0000","last_peered":"2024-04-25T13:38:45.375317+0000","last_clean":"2024-04-25T13:38:45.375317+0000","last_became_active":"2024-04-25T13:38:45.369525+0000","last_became_peered":"2024-04-25T13:38:45.369525+0000","last_unstale":"2024-04-25T13:38:45.375317+0000","last_undegraded":"2024-04-25T13:38:45.375317+0000","last_fullsized":"2024-04-25T13:38:45.375317+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:36:10.757961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T13:38:45.366216+0000","last_change":"2024-04-25T13:38:45.365309+0000","last_active":"2024-04-25T13:38:45.366216+0000","last_peered":"2024-04-25T13:38:45.366216+0000","last_clean":"2024-04-25T13:38:45.366216+0000","last_became_active":"2024-04-25T13:38:45.364893+0000","last_became_peered":"2024-04-25T13:38:45.364893+0000","last_unstale":"2024-04-25T13:38:45.366216+0000","last_undegraded":"2024-04-25T13:38:45.366216+0000","last_fullsized":"2024-04-25T13:38:45.366216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:23.335066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629626+0000","last_change":"2024-04-25T13:38:45.365150+0000","last_active":"2024-04-25T13:38:46.629626+0000","last_peered":"2024-04-25T13:38:46.629626+0000","last_clean":"2024-04-25T13:38:46.629626+0000","last_became_active":"2024-04-25T13:38:45.364649+0000","last_became_peered":"2024-04-25T13:38:45.364649+0000","last_unstale":"2024-04-25T13:38:46.629626+0000","last_undegraded":"2024-04-25T13:38:46.629626+0000","last_fullsized":"2024-04-25T13:38:46.629626+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:25.035789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336381+0000","last_change":"2024-04-25T13:38:45.352108+0000","last_active":"2024-04-25T13:38:46.336381+0000","last_peered":"2024-04-25T13:38:46.336381+0000","last_clean":"2024-04-25T13:38:46.336381+0000","last_became_active":"2024-04-25T13:38:45.349887+0000","last_became_peered":"2024-04-25T13:38:45.349887+0000","last_unstale":"2024-04-25T13:38:46.336381+0000","last_undegraded":"2024-04-25T13:38:46.336381+0000","last_fullsized":"2024-04-25T13:38:46.336381+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:09.339685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285228,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":806912,"data_stored":386813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874258,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":831488,"data_stored":959534,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037785,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":811008,"data_stored":387395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234014,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":815104,"data_stored":387417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430243,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36184,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708968,"statfs":{"total":95995035648,"available":95957983232,"internally_reserved":0,"allocated":860160,"data_stored":977763,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593769,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":806912,"data_stored":386813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36132,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709020,"statfs":{"total":95995035648,"available":95958036480,"internally_reserved":0,"allocated":806912,"data_stored":379855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":782336,"data_stored":369166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":811008,"data_stored":379889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":802816,"data_stored":378347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":811008,"data_stored":386835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789997,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":856064,"data_stored":968793,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T13:38:52.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T13:38:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 -- 172.21.15.27:0/541331883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac14104d40 msgr2=0x7fac14107160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 --2- 172.21.15.27:0/541331883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac14104d40 0x7fac14107160 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fac04009900 tx=0x7fac0402a410 comp rx=0 tx=0).stop 2024-04-25T13:38:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 -- 172.21.15.27:0/541331883 shutdown_connections 2024-04-25T13:38:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 --2- 172.21.15.27:0/541331883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac141076a0 0x7fac14107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 --2- 172.21.15.27:0/541331883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac14104d40 0x7fac14107160 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 --2- 172.21.15.27:0/541331883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac141023e0 0x7fac14104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 -- 172.21.15.27:0/541331883 >> 172.21.15.27:0/541331883 conn(0x7fac140fdbf0 msgr2=0x7fac141000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:52.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 -- 172.21.15.27:0/541331883 shutdown_connections 2024-04-25T13:38:52.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.449+0000 7fac1bc51640 1 -- 172.21.15.27:0/541331883 wait complete. 2024-04-25T13:38:52.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac1bc51640 1 Processor -- start 2024-04-25T13:38:52.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac1bc51640 1 -- start start 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac1bc51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac141023e0 0x7fac141a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac1bc51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac14104d40 0x7fac141a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac1bc51640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac141076a0 0x7fac141a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac1bc51640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac14115c20 con 0x7fac141023e0 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac1bc51640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fac14115aa0 con 0x7fac14104d40 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac1bc51640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fac14115da0 con 0x7fac141076a0 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac199ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac141023e0 0x7fac141a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac199ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac141023e0 0x7fac141a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56092/0 (socket says 172.21.15.27:56092) 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.450+0000 7fac199ed640 1 -- 172.21.15.27:0/767614577 learned_addr learned my addr 172.21.15.27:0/767614577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac191ec640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac14104d40 0x7fac141a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac1a1ee640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac141076a0 0x7fac141a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac199ed640 1 -- 172.21.15.27:0/767614577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac141076a0 msgr2=0x7fac141a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac199ed640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac141076a0 0x7fac141a7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac199ed640 1 -- 172.21.15.27:0/767614577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac14104d40 msgr2=0x7fac141a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac199ed640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac14104d40 0x7fac141a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac199ed640 1 -- 172.21.15.27:0/767614577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac141a8290 con 0x7fac141023e0 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac191ec640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac14104d40 0x7fac141a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac199ed640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac141023e0 0x7fac141a0800 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fac1000c970 tx=0x7fac1000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac0affd640 1 -- 172.21.15.27:0/767614577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10014070 con 0x7fac141023e0 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac141a8580 con 0x7fac141023e0 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac0affd640 1 -- 172.21.15.27:0/767614577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac10004440 con 0x7fac141023e0 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac0affd640 1 -- 172.21.15.27:0/767614577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10003e50 con 0x7fac141023e0 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.451+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac141a8b20 con 0x7fac141023e0 2024-04-25T13:38:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.452+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac140ff710 con 0x7fac141023e0 2024-04-25T13:38:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.455+0000 7fac0affd640 1 -- 172.21.15.27:0/767614577 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fac10017070 con 0x7fac141023e0 2024-04-25T13:38:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.455+0000 7fac0affd640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac000afd20 0x7fac000b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.455+0000 7fac191ec640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac000afd20 0x7fac000b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.455+0000 7fac0affd640 1 -- 172.21.15.27:0/767614577 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fac100defe0 con 0x7fac141023e0 2024-04-25T13:38:52.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.456+0000 7fac191ec640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac000afd20 0x7fac000b2230 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac04000c00 tx=0x7fac04005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:52.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.459+0000 7fac0affd640 1 -- 172.21.15.27:0/767614577 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac1001e030 con 0x7fac141023e0 2024-04-25T13:38:52.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:52 smithi045 ceph-mon[32731]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:52.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2734795461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:52.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3061921969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.642+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fac14062fe0 con 0x7fac000afd20 2024-04-25T13:38:52.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.649+0000 7fac0affd640 1 -- 172.21.15.27:0/767614577 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258876 (secure 0 0 0) 0x7fac14062fe0 con 0x7fac000afd20 2024-04-25T13:38:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:52.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:52 smithi027 ceph-mon[29140]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:52.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2734795461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:52.652 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3061921969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:52.654 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T13:38:52.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac000afd20 msgr2=0x7fac000b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac000afd20 0x7fac000b2230 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac04000c00 tx=0x7fac04005d50 comp rx=0 tx=0).stop 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac141023e0 msgr2=0x7fac141a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac141023e0 0x7fac141a0800 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fac1000c970 tx=0x7fac1000ce30 comp rx=0 tx=0).stop 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 shutdown_connections 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac000afd20 0x7fac000b2230 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac141076a0 0x7fac141a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac14104d40 0x7fac141a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 --2- 172.21.15.27:0/767614577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac141023e0 0x7fac141a0800 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.652+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 >> 172.21.15.27:0/767614577 conn(0x7fac140fdbf0 msgr2=0x7fac14103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.653+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 shutdown_connections 2024-04-25T13:38:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:52.653+0000 7fac1bc51640 1 -- 172.21.15.27:0/767614577 wait complete. 2024-04-25T13:38:52.664 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-04-25T13:38:52.348071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":629476,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":88,"num_read_kb":91,"num_write":180,"num_write_kb":2203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":290,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375620,"kb_used_data":9752,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124566204,"statfs":{"total":1151940427776,"available":1151555792896,"internally_reserved":0,"allocated":9986048,"data_stored":6478173,"data_compressed":108783,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19138,"internal_metadata":374453566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39108,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":27,"num_write":57,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002060"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336724+0000","last_change":"2024-04-25T13:38:45.356552+0000","last_active":"2024-04-25T13:38:46.336724+0000","last_peered":"2024-04-25T13:38:46.336724+0000","last_clean":"2024-04-25T13:38:46.336724+0000","last_became_active":"2024-04-25T13:38:45.356122+0000","last_became_peered":"2024-04-25T13:38:45.356122+0000","last_unstale":"2024-04-25T13:38:46.336724+0000","last_undegraded":"2024-04-25T13:38:46.336724+0000","last_fullsized":"2024-04-25T13:38:46.336724+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:55.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629520+0000","last_change":"2024-04-25T13:38:45.358577+0000","last_active":"2024-04-25T13:38:46.629520+0000","last_peered":"2024-04-25T13:38:46.629520+0000","last_clean":"2024-04-25T13:38:46.629520+0000","last_became_active":"2024-04-25T13:38:45.358122+0000","last_became_peered":"2024-04-25T13:38:45.358122+0000","last_unstale":"2024-04-25T13:38:46.629520+0000","last_undegraded":"2024-04-25T13:38:46.629520+0000","last_fullsized":"2024-04-25T13:38:46.629520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:52.508459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629456+0000","last_change":"2024-04-25T13:38:45.362539+0000","last_active":"2024-04-25T13:38:46.629456+0000","last_peered":"2024-04-25T13:38:46.629456+0000","last_clean":"2024-04-25T13:38:46.629456+0000","last_became_active":"2024-04-25T13:38:45.361983+0000","last_became_peered":"2024-04-25T13:38:45.361983+0000","last_unstale":"2024-04-25T13:38:46.629456+0000","last_undegraded":"2024-04-25T13:38:46.629456+0000","last_fullsized":"2024-04-25T13:38:46.629456+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:13.267692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.337191+0000","last_change":"2024-04-25T13:38:45.360535+0000","last_active":"2024-04-25T13:38:46.337191+0000","last_peered":"2024-04-25T13:38:46.337191+0000","last_clean":"2024-04-25T13:38:46.337191+0000","last_became_active":"2024-04-25T13:38:45.357722+0000","last_became_peered":"2024-04-25T13:38:45.357722+0000","last_unstale":"2024-04-25T13:38:46.337191+0000","last_undegraded":"2024-04-25T13:38:46.337191+0000","last_fullsized":"2024-04-25T13:38:46.337191+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:35.347434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369200+0000","last_change":"2024-04-25T13:38:45.353653+0000","last_active":"2024-04-25T13:38:46.369200+0000","last_peered":"2024-04-25T13:38:46.369200+0000","last_clean":"2024-04-25T13:38:46.369200+0000","last_became_active":"2024-04-25T13:38:45.353254+0000","last_became_peered":"2024-04-25T13:38:45.353254+0000","last_unstale":"2024-04-25T13:38:46.369200+0000","last_undegraded":"2024-04-25T13:38:46.369200+0000","last_fullsized":"2024-04-25T13:38:46.369200+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:57.275732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630416+0000","last_change":"2024-04-25T13:38:45.358241+0000","last_active":"2024-04-25T13:38:46.630416+0000","last_peered":"2024-04-25T13:38:46.630416+0000","last_clean":"2024-04-25T13:38:46.630416+0000","last_became_active":"2024-04-25T13:38:45.357843+0000","last_became_peered":"2024-04-25T13:38:45.357843+0000","last_unstale":"2024-04-25T13:38:46.630416+0000","last_undegraded":"2024-04-25T13:38:46.630416+0000","last_fullsized":"2024-04-25T13:38:46.630416+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:24.560357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629858+0000","last_change":"2024-04-25T13:38:45.350710+0000","last_active":"2024-04-25T13:38:46.629858+0000","last_peered":"2024-04-25T13:38:46.629858+0000","last_clean":"2024-04-25T13:38:46.629858+0000","last_became_active":"2024-04-25T13:38:45.350171+0000","last_became_peered":"2024-04-25T13:38:45.350171+0000","last_unstale":"2024-04-25T13:38:46.629858+0000","last_undegraded":"2024-04-25T13:38:46.629858+0000","last_fullsized":"2024-04-25T13:38:46.629858+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:24.933631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.335483+0000","last_change":"2024-04-25T13:38:45.356221+0000","last_active":"2024-04-25T13:38:46.335483+0000","last_peered":"2024-04-25T13:38:46.335483+0000","last_clean":"2024-04-25T13:38:46.335483+0000","last_became_active":"2024-04-25T13:38:45.355823+0000","last_became_peered":"2024-04-25T13:38:45.355823+0000","last_unstale":"2024-04-25T13:38:46.335483+0000","last_undegraded":"2024-04-25T13:38:46.335483+0000","last_fullsized":"2024-04-25T13:38:46.335483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:25:02.040407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629840+0000","last_change":"2024-04-25T13:38:45.360940+0000","last_active":"2024-04-25T13:38:46.629840+0000","last_peered":"2024-04-25T13:38:46.629840+0000","last_clean":"2024-04-25T13:38:46.629840+0000","last_became_active":"2024-04-25T13:38:45.360241+0000","last_became_peered":"2024-04-25T13:38:45.360241+0000","last_unstale":"2024-04-25T13:38:46.629840+0000","last_undegraded":"2024-04-25T13:38:46.629840+0000","last_fullsized":"2024-04-25T13:38:46.629840+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:05.251236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631411+0000","last_change":"2024-04-25T13:38:45.357908+0000","last_active":"2024-04-25T13:38:46.631411+0000","last_peered":"2024-04-25T13:38:46.631411+0000","last_clean":"2024-04-25T13:38:46.631411+0000","last_became_active":"2024-04-25T13:38:45.357491+0000","last_became_peered":"2024-04-25T13:38:45.357491+0000","last_unstale":"2024-04-25T13:38:46.631411+0000","last_undegraded":"2024-04-25T13:38:46.631411+0000","last_fullsized":"2024-04-25T13:38:46.631411+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:30.931611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630975+0000","last_change":"2024-04-25T13:38:45.368137+0000","last_active":"2024-04-25T13:38:46.630975+0000","last_peered":"2024-04-25T13:38:46.630975+0000","last_clean":"2024-04-25T13:38:46.630975+0000","last_became_active":"2024-04-25T13:38:45.367355+0000","last_became_peered":"2024-04-25T13:38:45.367355+0000","last_unstale":"2024-04-25T13:38:46.630975+0000","last_undegraded":"2024-04-25T13:38:46.630975+0000","last_fullsized":"2024-04-25T13:38:46.630975+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:09.502056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.337213+0000","last_change":"2024-04-25T13:38:45.359778+0000","last_active":"2024-04-25T13:38:46.337213+0000","last_peered":"2024-04-25T13:38:46.337213+0000","last_clean":"2024-04-25T13:38:46.337213+0000","last_became_active":"2024-04-25T13:38:45.357189+0000","last_became_peered":"2024-04-25T13:38:45.357189+0000","last_unstale":"2024-04-25T13:38:46.337213+0000","last_undegraded":"2024-04-25T13:38:46.337213+0000","last_fullsized":"2024-04-25T13:38:46.337213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:42:23.469010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629552+0000","last_change":"2024-04-25T13:38:45.353972+0000","last_active":"2024-04-25T13:38:46.629552+0000","last_peered":"2024-04-25T13:38:46.629552+0000","last_clean":"2024-04-25T13:38:46.629552+0000","last_became_active":"2024-04-25T13:38:45.353458+0000","last_became_peered":"2024-04-25T13:38:45.353458+0000","last_unstale":"2024-04-25T13:38:46.629552+0000","last_undegraded":"2024-04-25T13:38:46.629552+0000","last_fullsized":"2024-04-25T13:38:46.629552+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:17.087850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629007+0000","last_change":"2024-04-25T13:38:45.358876+0000","last_active":"2024-04-25T13:38:46.629007+0000","last_peered":"2024-04-25T13:38:46.629007+0000","last_clean":"2024-04-25T13:38:46.629007+0000","last_became_active":"2024-04-25T13:38:45.358435+0000","last_became_peered":"2024-04-25T13:38:45.358435+0000","last_unstale":"2024-04-25T13:38:46.629007+0000","last_undegraded":"2024-04-25T13:38:46.629007+0000","last_fullsized":"2024-04-25T13:38:46.629007+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:18:47.287112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.628961+0000","last_change":"2024-04-25T13:38:45.373557+0000","last_active":"2024-04-25T13:38:46.628961+0000","last_peered":"2024-04-25T13:38:46.628961+0000","last_clean":"2024-04-25T13:38:46.628961+0000","last_became_active":"2024-04-25T13:38:45.373179+0000","last_became_peered":"2024-04-25T13:38:45.373179+0000","last_unstale":"2024-04-25T13:38:46.628961+0000","last_undegraded":"2024-04-25T13:38:46.628961+0000","last_fullsized":"2024-04-25T13:38:46.628961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:01.790266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629440+0000","last_change":"2024-04-25T13:38:45.358282+0000","last_active":"2024-04-25T13:38:46.629440+0000","last_peered":"2024-04-25T13:38:46.629440+0000","last_clean":"2024-04-25T13:38:46.629440+0000","last_became_active":"2024-04-25T13:38:45.357923+0000","last_became_peered":"2024-04-25T13:38:45.357923+0000","last_unstale":"2024-04-25T13:38:46.629440+0000","last_undegraded":"2024-04-25T13:38:46.629440+0000","last_fullsized":"2024-04-25T13:38:46.629440+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:47.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336832+0000","last_change":"2024-04-25T13:38:45.357865+0000","last_active":"2024-04-25T13:38:46.336832+0000","last_peered":"2024-04-25T13:38:46.336832+0000","last_clean":"2024-04-25T13:38:46.336832+0000","last_became_active":"2024-04-25T13:38:45.355399+0000","last_became_peered":"2024-04-25T13:38:45.355399+0000","last_unstale":"2024-04-25T13:38:46.336832+0000","last_undegraded":"2024-04-25T13:38:46.336832+0000","last_fullsized":"2024-04-25T13:38:46.336832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:33:41.382462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.367003+0000","last_change":"2024-04-25T13:38:45.350724+0000","last_active":"2024-04-25T13:38:46.367003+0000","last_peered":"2024-04-25T13:38:46.367003+0000","last_clean":"2024-04-25T13:38:46.367003+0000","last_became_active":"2024-04-25T13:38:45.350284+0000","last_became_peered":"2024-04-25T13:38:45.350284+0000","last_unstale":"2024-04-25T13:38:46.367003+0000","last_undegraded":"2024-04-25T13:38:46.367003+0000","last_fullsized":"2024-04-25T13:38:46.367003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:30.960843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630472+0000","last_change":"2024-04-25T13:38:45.362966+0000","last_active":"2024-04-25T13:38:46.630472+0000","last_peered":"2024-04-25T13:38:46.630472+0000","last_clean":"2024-04-25T13:38:46.630472+0000","last_became_active":"2024-04-25T13:38:45.362585+0000","last_became_peered":"2024-04-25T13:38:45.362585+0000","last_unstale":"2024-04-25T13:38:46.630472+0000","last_undegraded":"2024-04-25T13:38:46.630472+0000","last_fullsized":"2024-04-25T13:38:46.630472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:46:10.825028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'4","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.366887+0000","last_change":"2024-04-25T13:38:44.354887+0000","last_active":"2024-04-25T13:38:46.366887+0000","last_peered":"2024-04-25T13:38:46.366887+0000","last_clean":"2024-04-25T13:38:46.366887+0000","last_became_active":"2024-04-25T13:38:44.354510+0000","last_became_peered":"2024-04-25T13:38:44.354510+0000","last_unstale":"2024-04-25T13:38:46.366887+0000","last_undegraded":"2024-04-25T13:38:46.366887+0000","last_fullsized":"2024-04-25T13:38:46.366887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:46.457087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336013+0000","last_change":"2024-04-25T13:38:45.364056+0000","last_active":"2024-04-25T13:38:46.336013+0000","last_peered":"2024-04-25T13:38:46.336013+0000","last_clean":"2024-04-25T13:38:46.336013+0000","last_became_active":"2024-04-25T13:38:45.363572+0000","last_became_peered":"2024-04-25T13:38:45.363572+0000","last_unstale":"2024-04-25T13:38:46.336013+0000","last_undegraded":"2024-04-25T13:38:46.336013+0000","last_fullsized":"2024-04-25T13:38:46.336013+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:49:12.136637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336107+0000","last_change":"2024-04-25T13:38:44.362371+0000","last_active":"2024-04-25T13:38:46.336107+0000","last_peered":"2024-04-25T13:38:46.336107+0000","last_clean":"2024-04-25T13:38:46.336107+0000","last_became_active":"2024-04-25T13:38:44.361262+0000","last_became_peered":"2024-04-25T13:38:44.361262+0000","last_unstale":"2024-04-25T13:38:46.336107+0000","last_undegraded":"2024-04-25T13:38:46.336107+0000","last_fullsized":"2024-04-25T13:38:46.336107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:01.442189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629803+0000","last_change":"2024-04-25T13:38:45.362599+0000","last_active":"2024-04-25T13:38:46.629803+0000","last_peered":"2024-04-25T13:38:46.629803+0000","last_clean":"2024-04-25T13:38:46.629803+0000","last_became_active":"2024-04-25T13:38:45.361893+0000","last_became_peered":"2024-04-25T13:38:45.361893+0000","last_unstale":"2024-04-25T13:38:46.629803+0000","last_undegraded":"2024-04-25T13:38:46.629803+0000","last_fullsized":"2024-04-25T13:38:46.629803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:52.414826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369744+0000","last_change":"2024-04-25T13:38:44.353063+0000","last_active":"2024-04-25T13:38:46.369744+0000","last_peered":"2024-04-25T13:38:46.369744+0000","last_clean":"2024-04-25T13:38:46.369744+0000","last_became_active":"2024-04-25T13:38:44.352403+0000","last_became_peered":"2024-04-25T13:38:44.352403+0000","last_unstale":"2024-04-25T13:38:46.369744+0000","last_undegraded":"2024-04-25T13:38:46.369744+0000","last_fullsized":"2024-04-25T13:38:46.369744+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:09.670589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629589+0000","last_change":"2024-04-25T13:38:45.359078+0000","last_active":"2024-04-25T13:38:46.629589+0000","last_peered":"2024-04-25T13:38:46.629589+0000","last_clean":"2024-04-25T13:38:46.629589+0000","last_became_active":"2024-04-25T13:38:45.354063+0000","last_became_peered":"2024-04-25T13:38:45.354063+0000","last_unstale":"2024-04-25T13:38:46.629589+0000","last_undegraded":"2024-04-25T13:38:46.629589+0000","last_fullsized":"2024-04-25T13:38:46.629589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:16.694340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369617+0000","last_change":"2024-04-25T13:38:44.349132+0000","last_active":"2024-04-25T13:38:46.369617+0000","last_peered":"2024-04-25T13:38:46.369617+0000","last_clean":"2024-04-25T13:38:46.369617+0000","last_became_active":"2024-04-25T13:38:44.348762+0000","last_became_peered":"2024-04-25T13:38:44.348762+0000","last_unstale":"2024-04-25T13:38:46.369617+0000","last_undegraded":"2024-04-25T13:38:46.369617+0000","last_fullsized":"2024-04-25T13:38:46.369617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:12.990304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629915+0000","last_change":"2024-04-25T13:38:45.369116+0000","last_active":"2024-04-25T13:38:46.629915+0000","last_peered":"2024-04-25T13:38:46.629915+0000","last_clean":"2024-04-25T13:38:46.629915+0000","last_became_active":"2024-04-25T13:38:45.368380+0000","last_became_peered":"2024-04-25T13:38:45.368380+0000","last_unstale":"2024-04-25T13:38:46.629915+0000","last_undegraded":"2024-04-25T13:38:46.629915+0000","last_fullsized":"2024-04-25T13:38:46.629915+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:02.259401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629585+0000","last_change":"2024-04-25T13:38:44.351873+0000","last_active":"2024-04-25T13:38:46.629585+0000","last_peered":"2024-04-25T13:38:46.629585+0000","last_clean":"2024-04-25T13:38:46.629585+0000","last_became_active":"2024-04-25T13:38:44.351518+0000","last_became_peered":"2024-04-25T13:38:44.351518+0000","last_unstale":"2024-04-25T13:38:46.629585+0000","last_undegraded":"2024-04-25T13:38:46.629585+0000","last_fullsized":"2024-04-25T13:38:46.629585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:06.839053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369080+0000","last_change":"2024-04-25T13:38:45.351369+0000","last_active":"2024-04-25T13:38:46.369080+0000","last_peered":"2024-04-25T13:38:46.369080+0000","last_clean":"2024-04-25T13:38:46.369080+0000","last_became_active":"2024-04-25T13:38:45.351018+0000","last_became_peered":"2024-04-25T13:38:45.351018+0000","last_unstale":"2024-04-25T13:38:46.369080+0000","last_undegraded":"2024-04-25T13:38:46.369080+0000","last_fullsized":"2024-04-25T13:38:46.369080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630283+0000","last_change":"2024-04-25T13:38:44.345986+0000","last_active":"2024-04-25T13:38:46.630283+0000","last_peered":"2024-04-25T13:38:46.630283+0000","last_clean":"2024-04-25T13:38:46.630283+0000","last_became_active":"2024-04-25T13:38:44.345600+0000","last_became_peered":"2024-04-25T13:38:44.345600+0000","last_unstale":"2024-04-25T13:38:46.630283+0000","last_undegraded":"2024-04-25T13:38:46.630283+0000","last_fullsized":"2024-04-25T13:38:46.630283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:52.749192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629861+0000","last_change":"2024-04-25T13:38:45.368641+0000","last_active":"2024-04-25T13:38:46.629861+0000","last_peered":"2024-04-25T13:38:46.629861+0000","last_clean":"2024-04-25T13:38:46.629861+0000","last_became_active":"2024-04-25T13:38:45.368209+0000","last_became_peered":"2024-04-25T13:38:45.368209+0000","last_unstale":"2024-04-25T13:38:46.629861+0000","last_undegraded":"2024-04-25T13:38:46.629861+0000","last_fullsized":"2024-04-25T13:38:46.629861+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:47.990808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629102+0000","last_change":"2024-04-25T13:38:44.339927+0000","last_active":"2024-04-25T13:38:46.629102+0000","last_peered":"2024-04-25T13:38:46.629102+0000","last_clean":"2024-04-25T13:38:46.629102+0000","last_became_active":"2024-04-25T13:38:44.339588+0000","last_became_peered":"2024-04-25T13:38:44.339588+0000","last_unstale":"2024-04-25T13:38:46.629102+0000","last_undegraded":"2024-04-25T13:38:46.629102+0000","last_fullsized":"2024-04-25T13:38:46.629102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:51.707750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.335525+0000","last_change":"2024-04-25T13:38:45.357323+0000","last_active":"2024-04-25T13:38:46.335525+0000","last_peered":"2024-04-25T13:38:46.335525+0000","last_clean":"2024-04-25T13:38:46.335525+0000","last_became_active":"2024-04-25T13:38:45.356870+0000","last_became_peered":"2024-04-25T13:38:45.356870+0000","last_unstale":"2024-04-25T13:38:46.335525+0000","last_undegraded":"2024-04-25T13:38:46.335525+0000","last_fullsized":"2024-04-25T13:38:46.335525+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:36.838709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629345+0000","last_change":"2024-04-25T13:38:44.351294+0000","last_active":"2024-04-25T13:38:46.629345+0000","last_peered":"2024-04-25T13:38:46.629345+0000","last_clean":"2024-04-25T13:38:46.629345+0000","last_became_active":"2024-04-25T13:38:44.350971+0000","last_became_peered":"2024-04-25T13:38:44.350971+0000","last_unstale":"2024-04-25T13:38:46.629345+0000","last_undegraded":"2024-04-25T13:38:46.629345+0000","last_fullsized":"2024-04-25T13:38:46.629345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:22.452217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.366788+0000","last_change":"2024-04-25T13:38:45.345943+0000","last_active":"2024-04-25T13:38:46.366788+0000","last_peered":"2024-04-25T13:38:46.366788+0000","last_clean":"2024-04-25T13:38:46.366788+0000","last_became_active":"2024-04-25T13:38:45.345528+0000","last_became_peered":"2024-04-25T13:38:45.345528+0000","last_unstale":"2024-04-25T13:38:46.366788+0000","last_undegraded":"2024-04-25T13:38:46.366788+0000","last_fullsized":"2024-04-25T13:38:46.366788+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:57.327303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631131+0000","last_change":"2024-04-25T13:38:44.343032+0000","last_active":"2024-04-25T13:38:46.631131+0000","last_peered":"2024-04-25T13:38:46.631131+0000","last_clean":"2024-04-25T13:38:46.631131+0000","last_became_active":"2024-04-25T13:38:44.342613+0000","last_became_peered":"2024-04-25T13:38:44.342613+0000","last_unstale":"2024-04-25T13:38:46.631131+0000","last_undegraded":"2024-04-25T13:38:46.631131+0000","last_fullsized":"2024-04-25T13:38:46.631131+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:33.957150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629937+0000","last_change":"2024-04-25T13:38:45.354534+0000","last_active":"2024-04-25T13:38:46.629937+0000","last_peered":"2024-04-25T13:38:46.629937+0000","last_clean":"2024-04-25T13:38:46.629937+0000","last_became_active":"2024-04-25T13:38:45.354147+0000","last_became_peered":"2024-04-25T13:38:45.354147+0000","last_unstale":"2024-04-25T13:38:46.629937+0000","last_undegraded":"2024-04-25T13:38:46.629937+0000","last_fullsized":"2024-04-25T13:38:46.629937+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:31.537339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629966+0000","last_change":"2024-04-25T13:38:44.356103+0000","last_active":"2024-04-25T13:38:46.629966+0000","last_peered":"2024-04-25T13:38:46.629966+0000","last_clean":"2024-04-25T13:38:46.629966+0000","last_became_active":"2024-04-25T13:38:44.355702+0000","last_became_peered":"2024-04-25T13:38:44.355702+0000","last_unstale":"2024-04-25T13:38:46.629966+0000","last_undegraded":"2024-04-25T13:38:46.629966+0000","last_fullsized":"2024-04-25T13:38:46.629966+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:47.572922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629760+0000","last_change":"2024-04-25T13:38:45.373843+0000","last_active":"2024-04-25T13:38:46.629760+0000","last_peered":"2024-04-25T13:38:46.629760+0000","last_clean":"2024-04-25T13:38:46.629760+0000","last_became_active":"2024-04-25T13:38:45.373423+0000","last_became_peered":"2024-04-25T13:38:45.373423+0000","last_unstale":"2024-04-25T13:38:46.629760+0000","last_undegraded":"2024-04-25T13:38:46.629760+0000","last_fullsized":"2024-04-25T13:38:46.629760+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:43:21.490921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630028+0000","last_change":"2024-04-25T13:38:44.355474+0000","last_active":"2024-04-25T13:38:46.630028+0000","last_peered":"2024-04-25T13:38:46.630028+0000","last_clean":"2024-04-25T13:38:46.630028+0000","last_became_active":"2024-04-25T13:38:44.355113+0000","last_became_peered":"2024-04-25T13:38:44.355113+0000","last_unstale":"2024-04-25T13:38:46.630028+0000","last_undegraded":"2024-04-25T13:38:46.630028+0000","last_fullsized":"2024-04-25T13:38:46.630028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.242974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629631+0000","last_change":"2024-04-25T13:38:45.369618+0000","last_active":"2024-04-25T13:38:46.629631+0000","last_peered":"2024-04-25T13:38:46.629631+0000","last_clean":"2024-04-25T13:38:46.629631+0000","last_became_active":"2024-04-25T13:38:45.369252+0000","last_became_peered":"2024-04-25T13:38:45.369252+0000","last_unstale":"2024-04-25T13:38:46.629631+0000","last_undegraded":"2024-04-25T13:38:46.629631+0000","last_fullsized":"2024-04-25T13:38:46.629631+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:40.336573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"76'17","reported_seq":41,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:50.896749+0000","last_change":"2024-04-25T13:38:44.349850+0000","last_active":"2024-04-25T13:38:50.896749+0000","last_peered":"2024-04-25T13:38:50.896749+0000","last_clean":"2024-04-25T13:38:50.896749+0000","last_became_active":"2024-04-25T13:38:44.349404+0000","last_became_peered":"2024-04-25T13:38:44.349404+0000","last_unstale":"2024-04-25T13:38:50.896749+0000","last_undegraded":"2024-04-25T13:38:50.896749+0000","last_fullsized":"2024-04-25T13:38:50.896749+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:40.025684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":24,"num_write":17,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369427+0000","last_change":"2024-04-25T13:38:45.371554+0000","last_active":"2024-04-25T13:38:46.369427+0000","last_peered":"2024-04-25T13:38:46.369427+0000","last_clean":"2024-04-25T13:38:46.369427+0000","last_became_active":"2024-04-25T13:38:45.371042+0000","last_became_peered":"2024-04-25T13:38:45.371042+0000","last_unstale":"2024-04-25T13:38:46.369427+0000","last_undegraded":"2024-04-25T13:38:46.369427+0000","last_fullsized":"2024-04-25T13:38:46.369427+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:59.240536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'2","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630351+0000","last_change":"2024-04-25T13:38:44.345979+0000","last_active":"2024-04-25T13:38:46.630351+0000","last_peered":"2024-04-25T13:38:46.630351+0000","last_clean":"2024-04-25T13:38:46.630351+0000","last_became_active":"2024-04-25T13:38:44.345596+0000","last_became_peered":"2024-04-25T13:38:44.345596+0000","last_unstale":"2024-04-25T13:38:46.630351+0000","last_undegraded":"2024-04-25T13:38:46.630351+0000","last_fullsized":"2024-04-25T13:38:46.630351+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:22:51.539024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629285+0000","last_change":"2024-04-25T13:38:45.364794+0000","last_active":"2024-04-25T13:38:46.629285+0000","last_peered":"2024-04-25T13:38:46.629285+0000","last_clean":"2024-04-25T13:38:46.629285+0000","last_became_active":"2024-04-25T13:38:45.364381+0000","last_became_peered":"2024-04-25T13:38:45.364381+0000","last_unstale":"2024-04-25T13:38:46.629285+0000","last_undegraded":"2024-04-25T13:38:46.629285+0000","last_fullsized":"2024-04-25T13:38:46.629285+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:16.856980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629836+0000","last_change":"2024-04-25T13:38:44.351269+0000","last_active":"2024-04-25T13:38:46.629836+0000","last_peered":"2024-04-25T13:38:46.629836+0000","last_clean":"2024-04-25T13:38:46.629836+0000","last_became_active":"2024-04-25T13:38:44.350933+0000","last_became_peered":"2024-04-25T13:38:44.350933+0000","last_unstale":"2024-04-25T13:38:46.629836+0000","last_undegraded":"2024-04-25T13:38:46.629836+0000","last_fullsized":"2024-04-25T13:38:46.629836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:02.677849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629790+0000","last_change":"2024-04-25T13:38:45.362732+0000","last_active":"2024-04-25T13:38:46.629790+0000","last_peered":"2024-04-25T13:38:46.629790+0000","last_clean":"2024-04-25T13:38:46.629790+0000","last_became_active":"2024-04-25T13:38:45.361957+0000","last_became_peered":"2024-04-25T13:38:45.361957+0000","last_unstale":"2024-04-25T13:38:46.629790+0000","last_undegraded":"2024-04-25T13:38:46.629790+0000","last_fullsized":"2024-04-25T13:38:46.629790+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:50.011442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336724+0000","last_change":"2024-04-25T13:38:44.349378+0000","last_active":"2024-04-25T13:38:46.336724+0000","last_peered":"2024-04-25T13:38:46.336724+0000","last_clean":"2024-04-25T13:38:46.336724+0000","last_became_active":"2024-04-25T13:38:44.348887+0000","last_became_peered":"2024-04-25T13:38:44.348887+0000","last_unstale":"2024-04-25T13:38:46.336724+0000","last_undegraded":"2024-04-25T13:38:46.336724+0000","last_fullsized":"2024-04-25T13:38:46.336724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.747205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631305+0000","last_change":"2024-04-25T13:38:45.349941+0000","last_active":"2024-04-25T13:38:46.631305+0000","last_peered":"2024-04-25T13:38:46.631305+0000","last_clean":"2024-04-25T13:38:46.631305+0000","last_became_active":"2024-04-25T13:38:45.349590+0000","last_became_peered":"2024-04-25T13:38:45.349590+0000","last_unstale":"2024-04-25T13:38:46.631305+0000","last_undegraded":"2024-04-25T13:38:46.631305+0000","last_fullsized":"2024-04-25T13:38:46.631305+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:42.904406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'8","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.368996+0000","last_change":"2024-04-25T13:38:44.354082+0000","last_active":"2024-04-25T13:38:46.368996+0000","last_peered":"2024-04-25T13:38:46.368996+0000","last_clean":"2024-04-25T13:38:46.368996+0000","last_became_active":"2024-04-25T13:38:44.353763+0000","last_became_peered":"2024-04-25T13:38:44.353763+0000","last_unstale":"2024-04-25T13:38:46.368996+0000","last_undegraded":"2024-04-25T13:38:46.368996+0000","last_fullsized":"2024-04-25T13:38:46.368996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:19:12.838644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630054+0000","last_change":"2024-04-25T13:38:45.368589+0000","last_active":"2024-04-25T13:38:46.630054+0000","last_peered":"2024-04-25T13:38:46.630054+0000","last_clean":"2024-04-25T13:38:46.630054+0000","last_became_active":"2024-04-25T13:38:45.368079+0000","last_became_peered":"2024-04-25T13:38:45.368079+0000","last_unstale":"2024-04-25T13:38:46.630054+0000","last_undegraded":"2024-04-25T13:38:46.630054+0000","last_fullsized":"2024-04-25T13:38:46.630054+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:10.180264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369415+0000","last_change":"2024-04-25T13:38:44.345148+0000","last_active":"2024-04-25T13:38:46.369415+0000","last_peered":"2024-04-25T13:38:46.369415+0000","last_clean":"2024-04-25T13:38:46.369415+0000","last_became_active":"2024-04-25T13:38:44.344816+0000","last_became_peered":"2024-04-25T13:38:44.344816+0000","last_unstale":"2024-04-25T13:38:46.369415+0000","last_undegraded":"2024-04-25T13:38:46.369415+0000","last_fullsized":"2024-04-25T13:38:46.369415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:03:38.134169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336647+0000","last_change":"2024-04-25T13:38:45.354340+0000","last_active":"2024-04-25T13:38:46.336647+0000","last_peered":"2024-04-25T13:38:46.336647+0000","last_clean":"2024-04-25T13:38:46.336647+0000","last_became_active":"2024-04-25T13:38:45.353933+0000","last_became_peered":"2024-04-25T13:38:45.353933+0000","last_unstale":"2024-04-25T13:38:46.336647+0000","last_undegraded":"2024-04-25T13:38:46.336647+0000","last_fullsized":"2024-04-25T13:38:46.336647+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:07.619051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.366794+0000","last_change":"2024-04-25T13:38:44.351183+0000","last_active":"2024-04-25T13:38:46.366794+0000","last_peered":"2024-04-25T13:38:46.366794+0000","last_clean":"2024-04-25T13:38:46.366794+0000","last_became_active":"2024-04-25T13:38:44.350785+0000","last_became_peered":"2024-04-25T13:38:44.350785+0000","last_unstale":"2024-04-25T13:38:46.366794+0000","last_undegraded":"2024-04-25T13:38:46.366794+0000","last_fullsized":"2024-04-25T13:38:46.366794+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:37:57.465012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"76'1","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631464+0000","last_change":"2024-04-25T13:38:45.362355+0000","last_active":"2024-04-25T13:38:46.631464+0000","last_peered":"2024-04-25T13:38:46.631464+0000","last_clean":"2024-04-25T13:38:46.631464+0000","last_became_active":"2024-04-25T13:38:45.361996+0000","last_became_peered":"2024-04-25T13:38:45.361996+0000","last_unstale":"2024-04-25T13:38:46.631464+0000","last_undegraded":"2024-04-25T13:38:46.631464+0000","last_fullsized":"2024-04-25T13:38:46.631464+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:53:24.790450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630424+0000","last_change":"2024-04-25T13:38:44.356079+0000","last_active":"2024-04-25T13:38:46.630424+0000","last_peered":"2024-04-25T13:38:46.630424+0000","last_clean":"2024-04-25T13:38:46.630424+0000","last_became_active":"2024-04-25T13:38:44.355660+0000","last_became_peered":"2024-04-25T13:38:44.355660+0000","last_unstale":"2024-04-25T13:38:46.630424+0000","last_undegraded":"2024-04-25T13:38:46.630424+0000","last_fullsized":"2024-04-25T13:38:46.630424+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:35.161337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630844+0000","last_change":"2024-04-25T13:38:45.356578+0000","last_active":"2024-04-25T13:38:46.630844+0000","last_peered":"2024-04-25T13:38:46.630844+0000","last_clean":"2024-04-25T13:38:46.630844+0000","last_became_active":"2024-04-25T13:38:45.356232+0000","last_became_peered":"2024-04-25T13:38:45.356232+0000","last_unstale":"2024-04-25T13:38:46.630844+0000","last_undegraded":"2024-04-25T13:38:46.630844+0000","last_fullsized":"2024-04-25T13:38:46.630844+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.476890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630278+0000","last_change":"2024-04-25T13:38:45.361209+0000","last_active":"2024-04-25T13:38:46.630278+0000","last_peered":"2024-04-25T13:38:46.630278+0000","last_clean":"2024-04-25T13:38:46.630278+0000","last_became_active":"2024-04-25T13:38:45.360830+0000","last_became_peered":"2024-04-25T13:38:45.360830+0000","last_unstale":"2024-04-25T13:38:46.630278+0000","last_undegraded":"2024-04-25T13:38:46.630278+0000","last_fullsized":"2024-04-25T13:38:46.630278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:59.661146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.628983+0000","last_change":"2024-04-25T13:38:44.351113+0000","last_active":"2024-04-25T13:38:46.628983+0000","last_peered":"2024-04-25T13:38:46.628983+0000","last_clean":"2024-04-25T13:38:46.628983+0000","last_became_active":"2024-04-25T13:38:44.350730+0000","last_became_peered":"2024-04-25T13:38:44.350730+0000","last_unstale":"2024-04-25T13:38:46.628983+0000","last_undegraded":"2024-04-25T13:38:46.628983+0000","last_fullsized":"2024-04-25T13:38:46.628983+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:38:22.747930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.335904+0000","last_change":"2024-04-25T13:38:45.356731+0000","last_active":"2024-04-25T13:38:46.335904+0000","last_peered":"2024-04-25T13:38:46.335904+0000","last_clean":"2024-04-25T13:38:46.335904+0000","last_became_active":"2024-04-25T13:38:45.355531+0000","last_became_peered":"2024-04-25T13:38:45.355531+0000","last_unstale":"2024-04-25T13:38:46.335904+0000","last_undegraded":"2024-04-25T13:38:46.335904+0000","last_fullsized":"2024-04-25T13:38:46.335904+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:15.482469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336877+0000","last_change":"2024-04-25T13:38:44.345941+0000","last_active":"2024-04-25T13:38:46.336877+0000","last_peered":"2024-04-25T13:38:46.336877+0000","last_clean":"2024-04-25T13:38:46.336877+0000","last_became_active":"2024-04-25T13:38:44.344634+0000","last_became_peered":"2024-04-25T13:38:44.344634+0000","last_unstale":"2024-04-25T13:38:46.336877+0000","last_undegraded":"2024-04-25T13:38:46.336877+0000","last_fullsized":"2024-04-25T13:38:46.336877+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:42.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":89,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631246+0000","last_change":"2024-04-25T13:37:27.903345+0000","last_active":"2024-04-25T13:38:46.631246+0000","last_peered":"2024-04-25T13:38:46.631246+0000","last_clean":"2024-04-25T13:38:46.631246+0000","last_became_active":"2024-04-25T13:37:27.897986+0000","last_became_peered":"2024-04-25T13:37:27.897986+0000","last_unstale":"2024-04-25T13:38:46.631246+0000","last_undegraded":"2024-04-25T13:38:46.631246+0000","last_fullsized":"2024-04-25T13:38:46.631246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_clean_scrub_stamp":"2024-04-25T13:33:56.569417+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:56.390458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.628832+0000","last_change":"2024-04-25T13:38:45.369390+0000","last_active":"2024-04-25T13:38:46.628832+0000","last_peered":"2024-04-25T13:38:46.628832+0000","last_clean":"2024-04-25T13:38:46.628832+0000","last_became_active":"2024-04-25T13:38:45.368938+0000","last_became_peered":"2024-04-25T13:38:45.368938+0000","last_unstale":"2024-04-25T13:38:46.628832+0000","last_undegraded":"2024-04-25T13:38:46.628832+0000","last_fullsized":"2024-04-25T13:38:46.628832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:11.942204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631434+0000","last_change":"2024-04-25T13:38:44.355585+0000","last_active":"2024-04-25T13:38:46.631434+0000","last_peered":"2024-04-25T13:38:46.631434+0000","last_clean":"2024-04-25T13:38:46.631434+0000","last_became_active":"2024-04-25T13:38:44.355131+0000","last_became_peered":"2024-04-25T13:38:44.355131+0000","last_unstale":"2024-04-25T13:38:46.631434+0000","last_undegraded":"2024-04-25T13:38:46.631434+0000","last_fullsized":"2024-04-25T13:38:46.631434+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:05.240247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369572+0000","last_change":"2024-04-25T13:38:44.355106+0000","last_active":"2024-04-25T13:38:46.369572+0000","last_peered":"2024-04-25T13:38:46.369572+0000","last_clean":"2024-04-25T13:38:46.369572+0000","last_became_active":"2024-04-25T13:38:44.354727+0000","last_became_peered":"2024-04-25T13:38:44.354727+0000","last_unstale":"2024-04-25T13:38:46.369572+0000","last_undegraded":"2024-04-25T13:38:46.369572+0000","last_fullsized":"2024-04-25T13:38:46.369572+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:29.606589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629161+0000","last_change":"2024-04-25T13:38:45.366187+0000","last_active":"2024-04-25T13:38:46.629161+0000","last_peered":"2024-04-25T13:38:46.629161+0000","last_clean":"2024-04-25T13:38:46.629161+0000","last_became_active":"2024-04-25T13:38:45.365744+0000","last_became_peered":"2024-04-25T13:38:45.365744+0000","last_unstale":"2024-04-25T13:38:46.629161+0000","last_undegraded":"2024-04-25T13:38:46.629161+0000","last_fullsized":"2024-04-25T13:38:46.629161+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:07.970735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.366921+0000","last_change":"2024-04-25T13:38:44.359250+0000","last_active":"2024-04-25T13:38:46.366921+0000","last_peered":"2024-04-25T13:38:46.366921+0000","last_clean":"2024-04-25T13:38:46.366921+0000","last_became_active":"2024-04-25T13:38:44.358874+0000","last_became_peered":"2024-04-25T13:38:44.358874+0000","last_unstale":"2024-04-25T13:38:46.366921+0000","last_undegraded":"2024-04-25T13:38:46.366921+0000","last_fullsized":"2024-04-25T13:38:46.366921+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:28.820465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336207+0000","last_change":"2024-04-25T13:38:45.370080+0000","last_active":"2024-04-25T13:38:46.336207+0000","last_peered":"2024-04-25T13:38:46.336207+0000","last_clean":"2024-04-25T13:38:46.336207+0000","last_became_active":"2024-04-25T13:38:45.369668+0000","last_became_peered":"2024-04-25T13:38:45.369668+0000","last_unstale":"2024-04-25T13:38:46.336207+0000","last_undegraded":"2024-04-25T13:38:46.336207+0000","last_fullsized":"2024-04-25T13:38:46.336207+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:03.197468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.337276+0000","last_change":"2024-04-25T13:38:44.358907+0000","last_active":"2024-04-25T13:38:46.337276+0000","last_peered":"2024-04-25T13:38:46.337276+0000","last_clean":"2024-04-25T13:38:46.337276+0000","last_became_active":"2024-04-25T13:38:44.358455+0000","last_became_peered":"2024-04-25T13:38:44.358455+0000","last_unstale":"2024-04-25T13:38:46.337276+0000","last_undegraded":"2024-04-25T13:38:46.337276+0000","last_fullsized":"2024-04-25T13:38:46.337276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:50:52.916433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629605+0000","last_change":"2024-04-25T13:38:45.373050+0000","last_active":"2024-04-25T13:38:46.629605+0000","last_peered":"2024-04-25T13:38:46.629605+0000","last_clean":"2024-04-25T13:38:46.629605+0000","last_became_active":"2024-04-25T13:38:45.372621+0000","last_became_peered":"2024-04-25T13:38:45.372621+0000","last_unstale":"2024-04-25T13:38:46.629605+0000","last_undegraded":"2024-04-25T13:38:46.629605+0000","last_fullsized":"2024-04-25T13:38:46.629605+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:29:43.259296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629295+0000","last_change":"2024-04-25T13:38:44.355768+0000","last_active":"2024-04-25T13:38:46.629295+0000","last_peered":"2024-04-25T13:38:46.629295+0000","last_clean":"2024-04-25T13:38:46.629295+0000","last_became_active":"2024-04-25T13:38:44.355400+0000","last_became_peered":"2024-04-25T13:38:44.355400+0000","last_unstale":"2024-04-25T13:38:46.629295+0000","last_undegraded":"2024-04-25T13:38:46.629295+0000","last_fullsized":"2024-04-25T13:38:46.629295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:38.533717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369168+0000","last_change":"2024-04-25T13:38:45.350582+0000","last_active":"2024-04-25T13:38:46.369168+0000","last_peered":"2024-04-25T13:38:46.369168+0000","last_clean":"2024-04-25T13:38:46.369168+0000","last_became_active":"2024-04-25T13:38:45.350220+0000","last_became_peered":"2024-04-25T13:38:45.350220+0000","last_unstale":"2024-04-25T13:38:46.369168+0000","last_undegraded":"2024-04-25T13:38:46.369168+0000","last_fullsized":"2024-04-25T13:38:46.369168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.463519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.335542+0000","last_change":"2024-04-25T13:38:44.360459+0000","last_active":"2024-04-25T13:38:46.335542+0000","last_peered":"2024-04-25T13:38:46.335542+0000","last_clean":"2024-04-25T13:38:46.335542+0000","last_became_active":"2024-04-25T13:38:44.359709+0000","last_became_peered":"2024-04-25T13:38:44.359709+0000","last_unstale":"2024-04-25T13:38:46.335542+0000","last_undegraded":"2024-04-25T13:38:46.335542+0000","last_fullsized":"2024-04-25T13:38:46.335542+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:53.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336393+0000","last_change":"2024-04-25T13:38:45.357291+0000","last_active":"2024-04-25T13:38:46.336393+0000","last_peered":"2024-04-25T13:38:46.336393+0000","last_clean":"2024-04-25T13:38:46.336393+0000","last_became_active":"2024-04-25T13:38:45.356929+0000","last_became_peered":"2024-04-25T13:38:45.356929+0000","last_unstale":"2024-04-25T13:38:46.336393+0000","last_undegraded":"2024-04-25T13:38:46.336393+0000","last_fullsized":"2024-04-25T13:38:46.336393+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:02:08.619208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629952+0000","last_change":"2024-04-25T13:38:44.343885+0000","last_active":"2024-04-25T13:38:46.629952+0000","last_peered":"2024-04-25T13:38:46.629952+0000","last_clean":"2024-04-25T13:38:46.629952+0000","last_became_active":"2024-04-25T13:38:44.343541+0000","last_became_peered":"2024-04-25T13:38:44.343541+0000","last_unstale":"2024-04-25T13:38:46.629952+0000","last_undegraded":"2024-04-25T13:38:46.629952+0000","last_fullsized":"2024-04-25T13:38:46.629952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:50:01.904593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.335881+0000","last_change":"2024-04-25T13:38:45.371508+0000","last_active":"2024-04-25T13:38:46.335881+0000","last_peered":"2024-04-25T13:38:46.335881+0000","last_clean":"2024-04-25T13:38:46.335881+0000","last_became_active":"2024-04-25T13:38:45.370912+0000","last_became_peered":"2024-04-25T13:38:45.370912+0000","last_unstale":"2024-04-25T13:38:46.335881+0000","last_undegraded":"2024-04-25T13:38:46.335881+0000","last_fullsized":"2024-04-25T13:38:46.335881+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:07.075511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.369186+0000","last_change":"2024-04-25T13:38:44.352794+0000","last_active":"2024-04-25T13:38:46.369186+0000","last_peered":"2024-04-25T13:38:46.369186+0000","last_clean":"2024-04-25T13:38:46.369186+0000","last_became_active":"2024-04-25T13:38:44.352422+0000","last_became_peered":"2024-04-25T13:38:44.352422+0000","last_unstale":"2024-04-25T13:38:46.369186+0000","last_undegraded":"2024-04-25T13:38:46.369186+0000","last_fullsized":"2024-04-25T13:38:46.369186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:40.543783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630622+0000","last_change":"2024-04-25T13:38:45.363322+0000","last_active":"2024-04-25T13:38:46.630622+0000","last_peered":"2024-04-25T13:38:46.630622+0000","last_clean":"2024-04-25T13:38:46.630622+0000","last_became_active":"2024-04-25T13:38:45.362948+0000","last_became_peered":"2024-04-25T13:38:45.362948+0000","last_unstale":"2024-04-25T13:38:46.630622+0000","last_undegraded":"2024-04-25T13:38:46.630622+0000","last_fullsized":"2024-04-25T13:38:46.630622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:20.815774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630524+0000","last_change":"2024-04-25T13:38:44.358588+0000","last_active":"2024-04-25T13:38:46.630524+0000","last_peered":"2024-04-25T13:38:46.630524+0000","last_clean":"2024-04-25T13:38:46.630524+0000","last_became_active":"2024-04-25T13:38:44.358269+0000","last_became_peered":"2024-04-25T13:38:44.358269+0000","last_unstale":"2024-04-25T13:38:46.630524+0000","last_undegraded":"2024-04-25T13:38:46.630524+0000","last_fullsized":"2024-04-25T13:38:46.630524+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:35.405572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631000+0000","last_change":"2024-04-25T13:38:45.369272+0000","last_active":"2024-04-25T13:38:46.631000+0000","last_peered":"2024-04-25T13:38:46.631000+0000","last_clean":"2024-04-25T13:38:46.631000+0000","last_became_active":"2024-04-25T13:38:45.368774+0000","last_became_peered":"2024-04-25T13:38:45.368774+0000","last_unstale":"2024-04-25T13:38:46.631000+0000","last_undegraded":"2024-04-25T13:38:46.631000+0000","last_fullsized":"2024-04-25T13:38:46.631000+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:39.149557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.682206+0000","last_change":"2024-04-25T13:38:44.349479+0000","last_active":"2024-04-25T13:38:46.682206+0000","last_peered":"2024-04-25T13:38:46.682206+0000","last_clean":"2024-04-25T13:38:46.682206+0000","last_became_active":"2024-04-25T13:38:44.349104+0000","last_became_peered":"2024-04-25T13:38:44.349104+0000","last_unstale":"2024-04-25T13:38:46.682206+0000","last_undegraded":"2024-04-25T13:38:46.682206+0000","last_fullsized":"2024-04-25T13:38:46.682206+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:39.257472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630655+0000","last_change":"2024-04-25T13:38:45.362120+0000","last_active":"2024-04-25T13:38:46.630655+0000","last_peered":"2024-04-25T13:38:46.630655+0000","last_clean":"2024-04-25T13:38:46.630655+0000","last_became_active":"2024-04-25T13:38:45.361402+0000","last_became_peered":"2024-04-25T13:38:45.361402+0000","last_unstale":"2024-04-25T13:38:46.630655+0000","last_undegraded":"2024-04-25T13:38:46.630655+0000","last_fullsized":"2024-04-25T13:38:46.630655+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:32.117236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629299+0000","last_change":"2024-04-25T13:38:44.349502+0000","last_active":"2024-04-25T13:38:46.629299+0000","last_peered":"2024-04-25T13:38:46.629299+0000","last_clean":"2024-04-25T13:38:46.629299+0000","last_became_active":"2024-04-25T13:38:44.349127+0000","last_became_peered":"2024-04-25T13:38:44.349127+0000","last_unstale":"2024-04-25T13:38:46.629299+0000","last_undegraded":"2024-04-25T13:38:46.629299+0000","last_fullsized":"2024-04-25T13:38:46.629299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:34:50.673352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.631069+0000","last_change":"2024-04-25T13:38:45.364746+0000","last_active":"2024-04-25T13:38:46.631069+0000","last_peered":"2024-04-25T13:38:46.631069+0000","last_clean":"2024-04-25T13:38:46.631069+0000","last_became_active":"2024-04-25T13:38:45.364364+0000","last_became_peered":"2024-04-25T13:38:45.364364+0000","last_unstale":"2024-04-25T13:38:46.631069+0000","last_undegraded":"2024-04-25T13:38:46.631069+0000","last_fullsized":"2024-04-25T13:38:46.631069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:42.145637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336400+0000","last_change":"2024-04-25T13:38:45.356683+0000","last_active":"2024-04-25T13:38:46.336400+0000","last_peered":"2024-04-25T13:38:46.336400+0000","last_clean":"2024-04-25T13:38:46.336400+0000","last_became_active":"2024-04-25T13:38:45.356048+0000","last_became_peered":"2024-04-25T13:38:45.356048+0000","last_unstale":"2024-04-25T13:38:46.336400+0000","last_undegraded":"2024-04-25T13:38:46.336400+0000","last_fullsized":"2024-04-25T13:38:46.336400+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:49.578317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629785+0000","last_change":"2024-04-25T13:38:45.358298+0000","last_active":"2024-04-25T13:38:46.629785+0000","last_peered":"2024-04-25T13:38:46.629785+0000","last_clean":"2024-04-25T13:38:46.629785+0000","last_became_active":"2024-04-25T13:38:45.357986+0000","last_became_peered":"2024-04-25T13:38:45.357986+0000","last_unstale":"2024-04-25T13:38:46.629785+0000","last_undegraded":"2024-04-25T13:38:46.629785+0000","last_fullsized":"2024-04-25T13:38:46.629785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:33.093156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629404+0000","last_change":"2024-04-25T13:38:45.363043+0000","last_active":"2024-04-25T13:38:46.629404+0000","last_peered":"2024-04-25T13:38:46.629404+0000","last_clean":"2024-04-25T13:38:46.629404+0000","last_became_active":"2024-04-25T13:38:45.362686+0000","last_became_peered":"2024-04-25T13:38:45.362686+0000","last_unstale":"2024-04-25T13:38:46.629404+0000","last_undegraded":"2024-04-25T13:38:46.629404+0000","last_fullsized":"2024-04-25T13:38:46.629404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:56.602134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629260+0000","last_change":"2024-04-25T13:38:45.373256+0000","last_active":"2024-04-25T13:38:46.629260+0000","last_peered":"2024-04-25T13:38:46.629260+0000","last_clean":"2024-04-25T13:38:46.629260+0000","last_became_active":"2024-04-25T13:38:45.372844+0000","last_became_peered":"2024-04-25T13:38:45.372844+0000","last_unstale":"2024-04-25T13:38:46.629260+0000","last_undegraded":"2024-04-25T13:38:46.629260+0000","last_fullsized":"2024-04-25T13:38:46.629260+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:39:34.590536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629218+0000","last_change":"2024-04-25T13:38:45.370007+0000","last_active":"2024-04-25T13:38:46.629218+0000","last_peered":"2024-04-25T13:38:46.629218+0000","last_clean":"2024-04-25T13:38:46.629218+0000","last_became_active":"2024-04-25T13:38:45.369606+0000","last_became_peered":"2024-04-25T13:38:45.369606+0000","last_unstale":"2024-04-25T13:38:46.629218+0000","last_undegraded":"2024-04-25T13:38:46.629218+0000","last_fullsized":"2024-04-25T13:38:46.629218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:45.514756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336579+0000","last_change":"2024-04-25T13:38:45.362075+0000","last_active":"2024-04-25T13:38:46.336579+0000","last_peered":"2024-04-25T13:38:46.336579+0000","last_clean":"2024-04-25T13:38:46.336579+0000","last_became_active":"2024-04-25T13:38:45.361720+0000","last_became_peered":"2024-04-25T13:38:45.361720+0000","last_unstale":"2024-04-25T13:38:46.336579+0000","last_undegraded":"2024-04-25T13:38:46.336579+0000","last_fullsized":"2024-04-25T13:38:46.336579+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:36.169185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630419+0000","last_change":"2024-04-25T13:38:45.352101+0000","last_active":"2024-04-25T13:38:46.630419+0000","last_peered":"2024-04-25T13:38:46.630419+0000","last_clean":"2024-04-25T13:38:46.630419+0000","last_became_active":"2024-04-25T13:38:45.351739+0000","last_became_peered":"2024-04-25T13:38:45.351739+0000","last_unstale":"2024-04-25T13:38:46.630419+0000","last_undegraded":"2024-04-25T13:38:46.630419+0000","last_fullsized":"2024-04-25T13:38:46.630419+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:05.612527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336049+0000","last_change":"2024-04-25T13:38:45.363855+0000","last_active":"2024-04-25T13:38:46.336049+0000","last_peered":"2024-04-25T13:38:46.336049+0000","last_clean":"2024-04-25T13:38:46.336049+0000","last_became_active":"2024-04-25T13:38:45.363377+0000","last_became_peered":"2024-04-25T13:38:45.363377+0000","last_unstale":"2024-04-25T13:38:46.336049+0000","last_undegraded":"2024-04-25T13:38:46.336049+0000","last_fullsized":"2024-04-25T13:38:46.336049+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:26.828215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629479+0000","last_change":"2024-04-25T13:38:45.359844+0000","last_active":"2024-04-25T13:38:46.629479+0000","last_peered":"2024-04-25T13:38:46.629479+0000","last_clean":"2024-04-25T13:38:46.629479+0000","last_became_active":"2024-04-25T13:38:45.359428+0000","last_became_peered":"2024-04-25T13:38:45.359428+0000","last_unstale":"2024-04-25T13:38:46.629479+0000","last_undegraded":"2024-04-25T13:38:46.629479+0000","last_fullsized":"2024-04-25T13:38:46.629479+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:01.370409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.335513+0000","last_change":"2024-04-25T13:38:45.370415+0000","last_active":"2024-04-25T13:38:46.335513+0000","last_peered":"2024-04-25T13:38:46.335513+0000","last_clean":"2024-04-25T13:38:46.335513+0000","last_became_active":"2024-04-25T13:38:45.369525+0000","last_became_peered":"2024-04-25T13:38:45.369525+0000","last_unstale":"2024-04-25T13:38:46.335513+0000","last_undegraded":"2024-04-25T13:38:46.335513+0000","last_fullsized":"2024-04-25T13:38:46.335513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:39.532380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.630452+0000","last_change":"2024-04-25T13:38:45.365309+0000","last_active":"2024-04-25T13:38:46.630452+0000","last_peered":"2024-04-25T13:38:46.630452+0000","last_clean":"2024-04-25T13:38:46.630452+0000","last_became_active":"2024-04-25T13:38:45.364893+0000","last_became_peered":"2024-04-25T13:38:45.364893+0000","last_unstale":"2024-04-25T13:38:46.630452+0000","last_undegraded":"2024-04-25T13:38:46.630452+0000","last_fullsized":"2024-04-25T13:38:46.630452+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:04:54.454443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.629626+0000","last_change":"2024-04-25T13:38:45.365150+0000","last_active":"2024-04-25T13:38:46.629626+0000","last_peered":"2024-04-25T13:38:46.629626+0000","last_clean":"2024-04-25T13:38:46.629626+0000","last_became_active":"2024-04-25T13:38:45.364649+0000","last_became_peered":"2024-04-25T13:38:45.364649+0000","last_unstale":"2024-04-25T13:38:46.629626+0000","last_undegraded":"2024-04-25T13:38:46.629626+0000","last_fullsized":"2024-04-25T13:38:46.629626+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:25.035789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T13:38:46.336381+0000","last_change":"2024-04-25T13:38:45.352108+0000","last_active":"2024-04-25T13:38:46.336381+0000","last_peered":"2024-04-25T13:38:46.336381+0000","last_clean":"2024-04-25T13:38:46.336381+0000","last_became_active":"2024-04-25T13:38:45.349887+0000","last_became_peered":"2024-04-25T13:38:45.349887+0000","last_unstale":"2024-04-25T13:38:46.336381+0000","last_undegraded":"2024-04-25T13:38:46.336381+0000","last_fullsized":"2024-04-25T13:38:46.336381+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:09.339685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285228,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":38991,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":27,"num_write":56,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":99675,"data_compressed":3684,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":806912,"data_stored":386813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874259,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":998387,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037785,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":811008,"data_stored":387395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234014,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":815104,"data_stored":387417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430243,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36184,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708968,"statfs":{"total":95995035648,"available":95957983232,"internally_reserved":0,"allocated":860160,"data_stored":977763,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593769,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":806912,"data_stored":386813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36152,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709000,"statfs":{"total":95995035648,"available":95958016000,"internally_reserved":0,"allocated":835584,"data_stored":407708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":811008,"data_stored":386903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":407742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":815104,"data_stored":386917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":811008,"data_stored":386835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789998,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":977480,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":38193,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":20929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":38387,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T13:38:52.667 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T13:38:52.668 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T13:38:52.668 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T13:38:52.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:52 smithi122 ceph-mon[32843]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:38:52.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2734795461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:38:52.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3061921969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:38:52.948 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:38:53.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:53 smithi027 ceph-mon[29140]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:38:53.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:53 smithi027 ceph-mon[29140]: pgmap v287: 97 pgs: 97 active+clean; 615 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T13:38:53.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:53 smithi027 ceph-mon[29140]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:38:53.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:53 smithi122 ceph-mon[32843]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:38:53.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:53 smithi122 ceph-mon[32843]: pgmap v287: 97 pgs: 97 active+clean; 615 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T13:38:53.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:53 smithi122 ceph-mon[32843]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:38:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:53 smithi045 ceph-mon[32731]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:38:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:53 smithi045 ceph-mon[32731]: pgmap v287: 97 pgs: 97 active+clean; 615 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T13:38:53.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:53 smithi045 ceph-mon[32731]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:38:54.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 -- 172.21.15.27:0/867837492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 msgr2=0x7fbccc10c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:54.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/867837492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 0x7fbccc10c580 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc009990 tx=0x7fbcbc009ca0 comp rx=0 tx=0).stop 2024-04-25T13:38:54.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 -- 172.21.15.27:0/867837492 shutdown_connections 2024-04-25T13:38:54.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/867837492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 0x7fbccc10c580 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:54.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/867837492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbccc104200 0x7fbccc104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:54.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/867837492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbccc105330 0x7fbccc105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:54.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 -- 172.21.15.27:0/867837492 >> 172.21.15.27:0/867837492 conn(0x7fbccc0ff840 msgr2=0x7fbccc101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:54.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 -- 172.21.15.27:0/867837492 shutdown_connections 2024-04-25T13:38:54.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.956+0000 7fbcd46e5700 1 -- 172.21.15.27:0/867837492 wait complete. 2024-04-25T13:38:54.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd46e5700 1 Processor -- start 2024-04-25T13:38:54.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd46e5700 1 -- start start 2024-04-25T13:38:54.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd46e5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbccc104200 0x7fbccc19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:54.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd46e5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbccc105330 0x7fbccc19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:54.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd46e5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 0x7fbccc1a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:54.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd46e5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc10fce0 con 0x7fbccc105cd0 2024-04-25T13:38:54.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd46e5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc10fb60 con 0x7fbccc105330 2024-04-25T13:38:54.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd46e5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc10fe60 con 0x7fbccc104200 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd2c82700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 0x7fbccc1a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd2c82700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 0x7fbccc1a1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56592/0 (socket says 172.21.15.27:56592) 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.957+0000 7fbcd2c82700 1 -- 172.21.15.27:0/1294456489 learned_addr learned my addr 172.21.15.27:0/1294456489 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd2481700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbccc104200 0x7fbccc19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd1c80700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbccc105330 0x7fbccc19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd2c82700 1 -- 172.21.15.27:0/1294456489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbccc104200 msgr2=0x7fbccc19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd2c82700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbccc104200 0x7fbccc19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd2c82700 1 -- 172.21.15.27:0/1294456489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbccc105330 msgr2=0x7fbccc19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:54.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd2c82700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbccc105330 0x7fbccc19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd2c82700 1 -- 172.21.15.27:0/1294456489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbccc1a22f0 con 0x7fbccc105cd0 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd1c80700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbccc105330 0x7fbccc19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd2c82700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 0x7fbccc1a1c70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0092e0 tx=0x7fbcbc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbccb7fe700 1 -- 172.21.15.27:0/1294456489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc027070 con 0x7fbccc105cd0 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbccc1a2580 con 0x7fbccc105cd0 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbccb7fe700 1 -- 172.21.15.27:0/1294456489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcbc0040e0 con 0x7fbccc105cd0 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbccb7fe700 1 -- 172.21.15.27:0/1294456489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc022e30 con 0x7fbccc105cd0 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.958+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbccc1a2a50 con 0x7fbccc105cd0 2024-04-25T13:38:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.959+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbccc051190 con 0x7fbccc105cd0 2024-04-25T13:38:54.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.961+0000 7fbccb7fe700 1 -- 172.21.15.27:0/1294456489 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fbcbc02a070 con 0x7fbccc105cd0 2024-04-25T13:38:54.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.961+0000 7fbccb7fe700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcb40afc10 0x7fbcb40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:54.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.961+0000 7fbcd2481700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcb40afc10 0x7fbcb40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:54.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.961+0000 7fbccb7fe700 1 -- 172.21.15.27:0/1294456489 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fbcbc0dfb20 con 0x7fbccc105cd0 2024-04-25T13:38:54.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.962+0000 7fbcd2481700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcb40afc10 0x7fbcb40b2120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbccc105230 tx=0x7fbcc400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:54.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:54.966+0000 7fbccb7fe700 1 -- 172.21.15.27:0/1294456489 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbcbc013070 con 0x7fbccc105cd0 2024-04-25T13:38:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.262+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbccc02e510 con 0x7fbccc105cd0 2024-04-25T13:38:55.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.265+0000 7fbccb7fe700 1 -- 172.21.15.27:0/1294456489 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fbcbc0a87e0 con 0x7fbccc105cd0 2024-04-25T13:38:55.265 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-04-25T13:38:55.265 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDvXCpmh9izDxAALDBUGP8qY6qraTmgzTjSTA== 2024-04-25T13:38:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcb40afc10 msgr2=0x7fbcb40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcb40afc10 0x7fbcb40b2120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbccc105230 tx=0x7fbcc400d040 comp rx=0 tx=0).stop 2024-04-25T13:38:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 msgr2=0x7fbccc1a1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 0x7fbccc1a1c70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0092e0 tx=0x7fbcbc0047b0 comp rx=0 tx=0).stop 2024-04-25T13:38:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 shutdown_connections 2024-04-25T13:38:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcb40afc10 0x7fbcb40b2120 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:55.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbccc104200 0x7fbccc19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbccc105cd0 0x7fbccc1a1c70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 --2- 172.21.15.27:0/1294456489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbccc105330 0x7fbccc19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 >> 172.21.15.27:0/1294456489 conn(0x7fbccc0ff840 msgr2=0x7fbccc108560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 shutdown_connections 2024-04-25T13:38:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:55.267+0000 7fbcd46e5700 1 -- 172.21.15.27:0/1294456489 wait complete. 2024-04-25T13:38:55.599 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:55 smithi027 ceph-mon[29140]: pgmap v288: 97 pgs: 97 active+clean; 615 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2024-04-25T13:38:55.599 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:55 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1294456489' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T13:38:55.599 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:55 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1294456489' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T13:38:55.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:55 smithi122 ceph-mon[32843]: pgmap v288: 97 pgs: 97 active+clean; 615 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2024-04-25T13:38:55.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:55 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1294456489' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T13:38:55.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:55 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1294456489' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T13:38:55.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:55 smithi045 ceph-mon[32731]: pgmap v288: 97 pgs: 97 active+clean; 615 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2024-04-25T13:38:55.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:55 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1294456489' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T13:38:55.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:55 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1294456489' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T13:38:56.161 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:38:56.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T13:38:56.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T13:38:56.202 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T13:38:56.202 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T13:38:56.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T13:38:56.490 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:38:57.756 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:57 smithi027 ceph-mon[29140]: pgmap v289: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.3 KiB/s wr, 8 op/s 2024-04-25T13:38:57.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:57 smithi122 ceph-mon[32843]: pgmap v289: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.3 KiB/s wr, 8 op/s 2024-04-25T13:38:57.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:57 smithi045 ceph-mon[32731]: pgmap v289: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.3 KiB/s wr, 8 op/s 2024-04-25T13:38:58.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 -- 172.21.15.27:0/1990136876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 msgr2=0x7f10881048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1990136876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 0x7f10881048b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f10780099c0 tx=0x7f1078009cd0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 -- 172.21.15.27:0/1990136876 shutdown_connections 2024-04-25T13:38:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1990136876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1088109720 0x7f1088111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1990136876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 0x7f10881048b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1990136876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1088108dd0 0x7f10881091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 -- 172.21.15.27:0/1990136876 >> 172.21.15.27:0/1990136876 conn(0x7f10880ffd70 msgr2=0x7f10881021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 -- 172.21.15.27:0/1990136876 shutdown_connections 2024-04-25T13:38:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.338+0000 7f108d4c4700 1 -- 172.21.15.27:0/1990136876 wait complete. 2024-04-25T13:38:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.339+0000 7f108d4c4700 1 Processor -- start 2024-04-25T13:38:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.339+0000 7f108d4c4700 1 -- start start 2024-04-25T13:38:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.339+0000 7f108d4c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 0x7f108819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.339+0000 7f108d4c4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1088108dd0 0x7f108819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.339+0000 7f108d4c4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1088109720 0x7f108819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.339+0000 7f108d4c4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1088114440 con 0x7f10881044a0 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.339+0000 7f108d4c4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f10881142c0 con 0x7f1088109720 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.339+0000 7f108d4c4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f10881145c0 con 0x7f1088108dd0 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 0x7f108819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 0x7f108819ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56612/0 (socket says 172.21.15.27:56612) 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 -- 172.21.15.27:0/1817079699 learned_addr learned my addr 172.21.15.27:0/1817079699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f10877fe700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1088109720 0x7f108819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f10867fc700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1088108dd0 0x7f108819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 -- 172.21.15.27:0/1817079699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1088108dd0 msgr2=0x7f108819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1088108dd0 0x7f108819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 -- 172.21.15.27:0/1817079699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1088109720 msgr2=0x7f108819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1088109720 0x7f108819f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 -- 172.21.15.27:0/1817079699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108819fc20 con 0x7f10881044a0 2024-04-25T13:38:58.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f1086ffd700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 0x7f108819ad50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f10780098e0 tx=0x7f10780057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:58.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f106ffff700 1 -- 172.21.15.27:0/1817079699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078027070 con 0x7f10881044a0 2024-04-25T13:38:58.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108819feb0 con 0x7f10881044a0 2024-04-25T13:38:58.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f106ffff700 1 -- 172.21.15.27:0/1817079699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1078003f20 con 0x7f10881044a0 2024-04-25T13:38:58.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.340+0000 7f106ffff700 1 -- 172.21.15.27:0/1817079699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078022d70 con 0x7f10881044a0 2024-04-25T13:38:58.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.341+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10881a0390 con 0x7f10881044a0 2024-04-25T13:38:58.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.342+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1088051190 con 0x7f10881044a0 2024-04-25T13:38:58.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.343+0000 7f106ffff700 1 -- 172.21.15.27:0/1817079699 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f107802a070 con 0x7f10881044a0 2024-04-25T13:38:58.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.344+0000 7f106ffff700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10700afc10 0x7f10700b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:38:58.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.344+0000 7f10867fc700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10700afc10 0x7f10700b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:38:58.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.344+0000 7f106ffff700 1 -- 172.21.15.27:0/1817079699 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f10780e0340 con 0x7f10881044a0 2024-04-25T13:38:58.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.344+0000 7f10867fc700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10700afc10 0x7f10700b2120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f10881073c0 tx=0x7f107c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:38:58.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.347+0000 7f106ffff700 1 -- 172.21.15.27:0/1817079699 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1078031080 con 0x7f10881044a0 2024-04-25T13:38:58.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.618+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f108802e510 con 0x7f10881044a0 2024-04-25T13:38:58.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.624+0000 7f106ffff700 1 -- 172.21.15.27:0/1817079699 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+284480 (secure 0 0 0) 0x7f1078013070 con 0x7f10881044a0 2024-04-25T13:38:58.627 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:38:58.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10700afc10 msgr2=0x7f10700b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:58.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10700afc10 0x7f10700b2120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f10881073c0 tx=0x7f107c009190 comp rx=0 tx=0).stop 2024-04-25T13:38:58.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 msgr2=0x7f108819ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 0x7f108819ad50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f10780098e0 tx=0x7f10780057c0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 shutdown_connections 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10700afc10 0x7f10700b2120 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1088109720 0x7f108819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10881044a0 0x7f108819ad50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 --2- 172.21.15.27:0/1817079699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1088108dd0 0x7f108819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 >> 172.21.15.27:0/1817079699 conn(0x7f10880ffd70 msgr2=0x7f1088102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 shutdown_connections 2024-04-25T13:38:58.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:38:58.627+0000 7f108d4c4700 1 -- 172.21.15.27:0/1817079699 wait complete. 2024-04-25T13:38:59.506 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":1716439151},{"type":"v1","addr":"172.21.15.27:6801","nonce":1716439151}]},"active_addr":"172.21.15.27:6801/1716439151","active_change":"2024-04-25T13:31:30.235662+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.27:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":4276929155}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1102216915}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3865533517}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3278365795}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1188946823}]}]} 2024-04-25T13:38:59.513 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T13:38:59.513 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T13:38:59.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T13:38:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:38:59.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:59 smithi122 ceph-mon[32843]: pgmap v290: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-25T13:38:59.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:38:59 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1817079699' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T13:38:59.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:59 smithi045 ceph-mon[32731]: pgmap v290: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-25T13:38:59.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:38:59 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1817079699' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T13:38:59.885 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:59 smithi027 ceph-mon[29140]: pgmap v290: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2024-04-25T13:38:59.886 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:38:59 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1817079699' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T13:39:01.471 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:01 smithi027 ceph-mon[29140]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-25T13:39:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 -- 172.21.15.27:0/2454052996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 msgr2=0x7f29a41057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/2454052996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 0x7f29a41057d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f299c009990 tx=0x7f299c009ca0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 -- 172.21.15.27:0/2454052996 shutdown_connections 2024-04-25T13:39:01.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/2454052996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29a410e7f0 0x7f29a4110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/2454052996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 0x7f29a41057d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/2454052996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a4105d10 0x7f29a410e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 -- 172.21.15.27:0/2454052996 >> 172.21.15.27:0/2454052996 conn(0x7f29a4078cd0 msgr2=0x7f29a4079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 -- 172.21.15.27:0/2454052996 shutdown_connections 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 -- 172.21.15.27:0/2454052996 wait complete. 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 Processor -- start 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.585+0000 7f29ac7a9700 1 -- start start 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29ac7a9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 0x7f29a41a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29ac7a9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a4105d10 0x7f29a41a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29ac7a9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29a410e7f0 0x7f29a419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29ac7a9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f29a4114340 con 0x7f29a41053c0 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29ac7a9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41141c0 con 0x7f29a410e7f0 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29ac7a9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41144c0 con 0x7f29a4105d10 2024-04-25T13:39:01.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 0x7f29a41a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 0x7f29a41a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56628/0 (socket says 172.21.15.27:56628) 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 -- 172.21.15.27:0/890364008 learned_addr learned my addr 172.21.15.27:0/890364008 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aad46700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29a410e7f0 0x7f29a419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29a9d44700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a4105d10 0x7f29a41a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 -- 172.21.15.27:0/890364008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a4105d10 msgr2=0x7f29a41a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a4105d10 0x7f29a41a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 -- 172.21.15.27:0/890364008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29a410e7f0 msgr2=0x7f29a419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29a410e7f0 0x7f29a419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 -- 172.21.15.27:0/890364008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a419b0e0 con 0x7f29a41053c0 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29a9d44700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a4105d10 0x7f29a41a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:39:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.586+0000 7f29aa545700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 0x7f29a41a0880 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f299c000c00 tx=0x7f299c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.587+0000 7f29a37fe700 1 -- 172.21.15.27:0/890364008 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c027070 con 0x7f29a41053c0 2024-04-25T13:39:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.587+0000 7f29a37fe700 1 -- 172.21.15.27:0/890364008 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f299c0040e0 con 0x7f29a41053c0 2024-04-25T13:39:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.587+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a419b370 con 0x7f29a41053c0 2024-04-25T13:39:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.587+0000 7f29a37fe700 1 -- 172.21.15.27:0/890364008 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299c00f460 con 0x7f29a41053c0 2024-04-25T13:39:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.587+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a419b850 con 0x7f29a41053c0 2024-04-25T13:39:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.588+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a4051190 con 0x7f29a41053c0 2024-04-25T13:39:01.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.589+0000 7f29a37fe700 1 -- 172.21.15.27:0/890364008 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f299c02a070 con 0x7f29a41053c0 2024-04-25T13:39:01.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.590+0000 7f29a37fe700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f298c0afc10 0x7f298c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:01.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.590+0000 7f29a9d44700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f298c0afc10 0x7f298c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:01.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.590+0000 7f29a37fe700 1 -- 172.21.15.27:0/890364008 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f299c0e0030 con 0x7f29a41053c0 2024-04-25T13:39:01.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.591+0000 7f29a9d44700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f298c0afc10 0x7f298c0b2120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2998005fd0 tx=0x7f2998005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:01.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.594+0000 7f29a37fe700 1 -- 172.21.15.27:0/890364008 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f299c031080 con 0x7f29a41053c0 2024-04-25T13:39:01.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.800+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29a41a7710 con 0x7f29a41053c0 2024-04-25T13:39:01.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.802+0000 7f29a37fe700 1 -- 172.21.15.27:0/890364008 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+22173 (secure 0 0 0) 0x7f299c013070 con 0x7f29a41053c0 2024-04-25T13:39:01.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:39:01.803 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:38:46.332369+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:39:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f298c0afc10 msgr2=0x7f298c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f298c0afc10 0x7f298c0b2120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2998005fd0 tx=0x7f2998005e30 comp rx=0 tx=0).stop 2024-04-25T13:39:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 msgr2=0x7f29a41a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 0x7f29a41a0880 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f299c000c00 tx=0x7f299c0047b0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 shutdown_connections 2024-04-25T13:39:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f298c0afc10 0x7f298c0b2120 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29a410e7f0 0x7f29a419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a41053c0 0x7f29a41a0880 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 --2- 172.21.15.27:0/890364008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a4105d10 0x7f29a41a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 >> 172.21.15.27:0/890364008 conn(0x7f29a4078cd0 msgr2=0x7f29a40ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 shutdown_connections 2024-04-25T13:39:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:01.804+0000 7f29ac7a9700 1 -- 172.21.15.27:0/890364008 wait complete. 2024-04-25T13:39:01.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:01 smithi122 ceph-mon[32843]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-25T13:39:01.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:01 smithi045 ceph-mon[32731]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-25T13:39:02.510 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T13:39:02.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T13:39:02.766 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:02.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:02 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/890364008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:39:02.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:02 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/890364008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:39:02.888 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:02 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/890364008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:39:03.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:03 smithi027 ceph-mon[29140]: pgmap v292: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-04-25T13:39:03.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:03 smithi122 ceph-mon[32843]: pgmap v292: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-04-25T13:39:03.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:03 smithi045 ceph-mon[32731]: pgmap v292: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-04-25T13:39:04.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 -- 172.21.15.27:0/3874202662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae98102640 msgr2=0x7fae98102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:04.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3874202662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae98102640 0x7fae98102a50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fae84009990 tx=0x7fae84009ca0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 -- 172.21.15.27:0/3874202662 shutdown_connections 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3874202662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae981091b0 0x7fae981095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3874202662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae98102f90 0x7fae98111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3874202662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae98102640 0x7fae98102a50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 -- 172.21.15.27:0/3874202662 >> 172.21.15.27:0/3874202662 conn(0x7fae980fe370 msgr2=0x7fae981007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 -- 172.21.15.27:0/3874202662 shutdown_connections 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.649+0000 7fae9ebae700 1 -- 172.21.15.27:0/3874202662 wait complete. 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9ebae700 1 Processor -- start 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9ebae700 1 -- start start 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9ebae700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae98102640 0x7fae9819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9ebae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae98102f90 0x7fae9819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9ebae700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae981091b0 0x7fae9819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9ebae700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae98114450 con 0x7fae981091b0 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9ebae700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae981142d0 con 0x7fae98102640 2024-04-25T13:39:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9ebae700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fae981145d0 con 0x7fae98102f90 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9d14b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae981091b0 0x7fae9819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9d14b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae981091b0 0x7fae9819f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59266/0 (socket says 172.21.15.27:59266) 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9d14b700 1 -- 172.21.15.27:0/3488636233 learned_addr learned my addr 172.21.15.27:0/3488636233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae9c94a700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae98102640 0x7fae9819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.650+0000 7fae97fff700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae98102f90 0x7fae9819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9d14b700 1 -- 172.21.15.27:0/3488636233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae98102f90 msgr2=0x7fae9819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9d14b700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae98102f90 0x7fae9819b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9d14b700 1 -- 172.21.15.27:0/3488636233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae98102640 msgr2=0x7fae9819ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9d14b700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae98102640 0x7fae9819ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9d14b700 1 -- 172.21.15.27:0/3488636233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9819fc70 con 0x7fae981091b0 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae97fff700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae98102f90 0x7fae9819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9c94a700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae98102640 0x7fae9819ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:39:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9d14b700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae981091b0 0x7fae9819f5d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fae9000ada0 tx=0x7fae9000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae95ffb700 1 -- 172.21.15.27:0/3488636233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90017020 con 0x7fae981091b0 2024-04-25T13:39:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9819ff60 con 0x7fae981091b0 2024-04-25T13:39:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae95ffb700 1 -- 172.21.15.27:0/3488636233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae9000ce30 con 0x7fae981091b0 2024-04-25T13:39:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae95ffb700 1 -- 172.21.15.27:0/3488636233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae900054a0 con 0x7fae981091b0 2024-04-25T13:39:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.651+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae981a7880 con 0x7fae981091b0 2024-04-25T13:39:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.652+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae9810c750 con 0x7fae981091b0 2024-04-25T13:39:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.653+0000 7fae95ffb700 1 -- 172.21.15.27:0/3488636233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fae9000c980 con 0x7fae981091b0 2024-04-25T13:39:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.654+0000 7fae95ffb700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae880afc10 0x7fae880b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.654+0000 7fae9c94a700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae880afc10 0x7fae880b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.654+0000 7fae95ffb700 1 -- 172.21.15.27:0/3488636233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fae900de450 con 0x7fae981091b0 2024-04-25T13:39:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.655+0000 7fae9c94a700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae880afc10 0x7fae880b2120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fae9806c940 tx=0x7fae8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:04.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.659+0000 7fae95ffb700 1 -- 172.21.15.27:0/3488636233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae90021020 con 0x7fae981091b0 2024-04-25T13:39:04.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.870+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae9819c100 con 0x7fae981091b0 2024-04-25T13:39:04.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.872+0000 7fae95ffb700 1 -- 172.21.15.27:0/3488636233 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+22173 (secure 0 0 0) 0x7fae900a7110 con 0x7fae981091b0 2024-04-25T13:39:04.872 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:39:04.872 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:38:46.332369+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.874+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae880afc10 msgr2=0x7fae880b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.874+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae880afc10 0x7fae880b2120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fae9806c940 tx=0x7fae8c009260 comp rx=0 tx=0).stop 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.874+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae981091b0 msgr2=0x7fae9819f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.874+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae981091b0 0x7fae9819f5d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fae9000ada0 tx=0x7fae9000c330 comp rx=0 tx=0).stop 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.875+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 shutdown_connections 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.875+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae880afc10 0x7fae880b2120 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.875+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae981091b0 0x7fae9819f5d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.875+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae98102640 0x7fae9819ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.875+0000 7fae9ebae700 1 --2- 172.21.15.27:0/3488636233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae98102f90 0x7fae9819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:04.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.875+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 >> 172.21.15.27:0/3488636233 conn(0x7fae980fe370 msgr2=0x7fae980fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:04.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.875+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 shutdown_connections 2024-04-25T13:39:04.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:04.875+0000 7fae9ebae700 1 -- 172.21.15.27:0/3488636233 wait complete. 2024-04-25T13:39:05.640 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:05 smithi027 ceph-mon[29140]: pgmap v293: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2024-04-25T13:39:05.640 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:05 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3488636233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:39:05.672 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T13:39:05.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T13:39:05.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T13:39:05.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T13:39:05.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T13:39:05.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T13:39:05.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T13:39:05.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T13:39:05.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T13:39:05.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T13:39:05.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T13:39:05.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T13:39:05.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:05 smithi122 ceph-mon[32843]: pgmap v293: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2024-04-25T13:39:05.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:05 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3488636233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:39:05.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:05 smithi045 ceph-mon[32731]: pgmap v293: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2024-04-25T13:39:05.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:05 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3488636233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:39:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:07.112 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:07.122 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:07.125 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:07.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:07 smithi027 ceph-mon[29140]: pgmap v294: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-25T13:39:07.662 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:07.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:07 smithi122 ceph-mon[32843]: pgmap v294: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-25T13:39:07.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:07 smithi045 ceph-mon[32731]: pgmap v294: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-25T13:39:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.152+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3402576097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c108890 msgr2=0x7ff19c108ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.152+0000 7ff1a2cd6700 1 --2- 172.21.15.27:0/3402576097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c108890 0x7ff19c108ca0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff190009990 tx=0x7ff190009ca0 comp rx=0 tx=0).stop 2024-04-25T13:39:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.152+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3402576097 shutdown_connections 2024-04-25T13:39:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.152+0000 7ff1a2cd6700 1 --2- 172.21.15.27:0/3402576097 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff19c1031b0 0x7ff19c10f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.152+0000 7ff1a2cd6700 1 --2- 172.21.15.27:0/3402576097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c108890 0x7ff19c108ca0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.152+0000 7ff1a2cd6700 1 --2- 172.21.15.27:0/3402576097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff19c102860 0x7ff19c102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.152+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3402576097 >> 172.21.15.27:0/3402576097 conn(0x7ff19c0fe3f0 msgr2=0x7ff19c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:09.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.153+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3402576097 shutdown_connections 2024-04-25T13:39:09.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.153+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3402576097 wait complete. 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.153+0000 7ff1a2cd6700 1 Processor -- start 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.153+0000 7ff1a2cd6700 1 -- start start 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a2cd6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff19c102860 0x7ff19c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a2cd6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c1031b0 0x7ff19c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a2cd6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff19c108890 0x7ff19c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a2cd6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c112180 con 0x7ff19c1031b0 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a2cd6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c112000 con 0x7ff19c108890 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a2cd6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c112300 con 0x7ff19c102860 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c1031b0 0x7ff19c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c1031b0 0x7ff19c19ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59278/0 (socket says 172.21.15.27:59278) 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 -- 172.21.15.27:0/3055722174 learned_addr learned my addr 172.21.15.27:0/3055722174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a1273700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff19c108890 0x7ff19c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a0a72700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff19c102860 0x7ff19c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 -- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff19c102860 msgr2=0x7ff19c19e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff19c102860 0x7ff19c19e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 -- 172.21.15.27:0/3055722174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff19c108890 msgr2=0x7ff19c1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff19c108890 0x7ff19c1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 -- 172.21.15.27:0/3055722174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c199090 con 0x7ff19c1031b0 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a0a72700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff19c102860 0x7ff19c19e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff1a1273700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff19c108890 0x7ff19c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.154+0000 7ff19bfff700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c1031b0 0x7ff19c19ed00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff18800ee00 tx=0x7ff18800a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.155+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18800ae40 con 0x7ff19c1031b0 2024-04-25T13:39:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.155+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff18800ce30 con 0x7ff19c1031b0 2024-04-25T13:39:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.155+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19c199380 con 0x7ff19c1031b0 2024-04-25T13:39:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.155+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18800b820 con 0x7ff19c1031b0 2024-04-25T13:39:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.155+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19c1a5650 con 0x7ff19c1031b0 2024-04-25T13:39:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.156+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff19c063da0 con 0x7ff19c1031b0 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.157+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7ff18800b9c0 con 0x7ff19c1031b0 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.158+0000 7ff199ffb700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff18c0afc90 0x7ff18c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.158+0000 7ff1a0a72700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff18c0afc90 0x7ff18c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.158+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7ff1880e02c0 con 0x7ff19c1031b0 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.158+0000 7ff199ffb700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] conn(0x7ff18c0b94c0 0x7ff18c0bb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.160+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 --> [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff188019ec0 con 0x7ff18c0b94c0 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.160+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7ff1880e07a0 con 0x7ff19c1031b0 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.160+0000 7ff1a0a72700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff18c0afc90 0x7ff18c0b21a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff190000c00 tx=0x7ff190005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.160+0000 7ff1a1273700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] conn(0x7ff18c0b94c0 0x7ff18c0bb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:09.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.160+0000 7ff1a1273700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] conn(0x7ff18c0b94c0 0x7ff18c0bb910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:09.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.160+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 <== osd.2 v2:172.21.15.122:6800/1296998909 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff188019ec0 con 0x7ff18c0b94c0 2024-04-25T13:39:09.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.180+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 --> [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff19c0641a0 con 0x7ff18c0b94c0 2024-04-25T13:39:09.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff199ffb700 1 -- 172.21.15.27:0/3055722174 <== osd.2 v2:172.21.15.122:6800/1296998909 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff19c0641a0 con 0x7ff18c0b94c0 2024-04-25T13:39:09.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] conn(0x7ff18c0b94c0 msgr2=0x7ff18c0bb910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:09.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] conn(0x7ff18c0b94c0 0x7ff18c0bb910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff18c0afc90 msgr2=0x7ff18c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff18c0afc90 0x7ff18c0b21a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff190000c00 tx=0x7ff190005fb0 comp rx=0 tx=0).stop 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c1031b0 msgr2=0x7ff19c19ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 --2- 172.21.15.27:0/3055722174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19c1031b0 0x7ff19c19ed00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff18800ee00 tx=0x7ff18800a630 comp rx=0 tx=0).stop 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a1273700 1 -- 172.21.15.27:0/3055722174 reap_dead start 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 shutdown_connections 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 >> 172.21.15.27:0/3055722174 conn(0x7ff19c0fe3f0 msgr2=0x7ff19c100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 shutdown_connections 2024-04-25T13:39:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:09.181+0000 7ff1a2cd6700 1 -- 172.21.15.27:0/3055722174 wait complete. 2024-04-25T13:39:09.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:09 smithi027 ceph-mon[29140]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-25T13:39:09.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:09 smithi122 ceph-mon[32843]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-25T13:39:09.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:09 smithi045 ceph-mon[32731]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-25T13:39:10.537 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:10.538 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:10.538 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:10.539 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:10.541 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:10.541 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:10.542 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:11.164 INFO:teuthology.orchestra.run.smithi027.stdout:77309411392 2024-04-25T13:39:11.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T13:39:11.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:11 smithi027 ceph-mon[29140]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-25T13:39:11.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:11 smithi122 ceph-mon[32843]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-25T13:39:11.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:11 smithi045 ceph-mon[32731]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-25T13:39:13.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 -- 172.21.15.27:0/1975190280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838103360 msgr2=0x7fd838103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 --2- 172.21.15.27:0/1975190280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838103360 0x7fd838103770 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd8340099c0 tx=0x7fd834009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 -- 172.21.15.27:0/1975190280 shutdown_connections 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 --2- 172.21.15.27:0/1975190280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8381085e0 0x7fd83810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 --2- 172.21.15.27:0/1975190280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838103360 0x7fd838103770 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 --2- 172.21.15.27:0/1975190280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd838107c90 0x7fd8381080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 -- 172.21.15.27:0/1975190280 >> 172.21.15.27:0/1975190280 conn(0x7fd8380fecb0 msgr2=0x7fd8381010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 -- 172.21.15.27:0/1975190280 shutdown_connections 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.421+0000 7fd840288700 1 -- 172.21.15.27:0/1975190280 wait complete. 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd840288700 1 Processor -- start 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd840288700 1 -- start start 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd840288700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd838103360 0x7fd83819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd840288700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd838107c90 0x7fd83819bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd840288700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8381085e0 0x7fd83819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd840288700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd838113210 con 0x7fd8381085e0 2024-04-25T13:39:13.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd840288700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd838113090 con 0x7fd838103360 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd840288700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd838113390 con 0x7fd838107c90 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd83e825700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8381085e0 0x7fd83819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd83e825700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8381085e0 0x7fd83819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59294/0 (socket says 172.21.15.27:59294) 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd83e825700 1 -- 172.21.15.27:0/1345743480 learned_addr learned my addr 172.21.15.27:0/1345743480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd83e024700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd838103360 0x7fd83819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.422+0000 7fd83d823700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd838107c90 0x7fd83819bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd83e825700 1 -- 172.21.15.27:0/1345743480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd838107c90 msgr2=0x7fd83819bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd83e825700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd838107c90 0x7fd83819bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd83e825700 1 -- 172.21.15.27:0/1345743480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd838103360 msgr2=0x7fd83819ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd83e825700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd838103360 0x7fd83819ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd83e825700 1 -- 172.21.15.27:0/1345743480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8381a6db0 con 0x7fd8381085e0 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd83e825700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8381085e0 0x7fd83819c510 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd830007ee0 tx=0x7fd83000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83000dbd0 con 0x7fd8381085e0 2024-04-25T13:39:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd83000dd70 con 0x7fd8381085e0 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8381a70a0 con 0x7fd8381085e0 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830005990 con 0x7fd8381085e0 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.423+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8381a7590 con 0x7fd8381085e0 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.424+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd838063da0 con 0x7fd8381085e0 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.425+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd83001a020 con 0x7fd8381085e0 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.426+0000 7fd82f7fe700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd8240afc10 0x7fd8240b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.426+0000 7fd83e024700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd8240afc10 0x7fd8240b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.426+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fd8300dedc0 con 0x7fd8381085e0 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.426+0000 7fd82f7fe700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] conn(0x7fd8240b9440 0x7fd8240bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.426+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 --> [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd83001fe50 con 0x7fd8240b9440 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.426+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fd8300df2a0 con 0x7fd8381085e0 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.426+0000 7fd83d823700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] conn(0x7fd8240b9440 0x7fd8240bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.427+0000 7fd83e024700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd8240afc10 0x7fd8240b2120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd834009990 tx=0x7fd834000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.427+0000 7fd83d823700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] conn(0x7fd8240b9440 0x7fd8240bb890 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.428+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 <== osd.9 v2:172.21.15.27:6826/3822838727 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd83001fe50 con 0x7fd8240b9440 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.444+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 --> [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd8380641a0 con 0x7fd8240b9440 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.444+0000 7fd82f7fe700 1 -- 172.21.15.27:0/1345743480 <== osd.9 v2:172.21.15.27:6826/3822838727 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd8380641a0 con 0x7fd8240b9440 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] conn(0x7fd8240b9440 msgr2=0x7fd8240bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] conn(0x7fd8240b9440 0x7fd8240bb890 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd8240afc10 msgr2=0x7fd8240b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd8240afc10 0x7fd8240b2120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd834009990 tx=0x7fd834000bc0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8381085e0 msgr2=0x7fd83819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 --2- 172.21.15.27:0/1345743480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8381085e0 0x7fd83819c510 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd830007ee0 tx=0x7fd83000d3b0 comp rx=0 tx=0).stop 2024-04-25T13:39:13.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd83e825700 1 -- 172.21.15.27:0/1345743480 reap_dead start 2024-04-25T13:39:13.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 shutdown_connections 2024-04-25T13:39:13.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 >> 172.21.15.27:0/1345743480 conn(0x7fd8380fecb0 msgr2=0x7fd838106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:13.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 shutdown_connections 2024-04-25T13:39:13.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:13.445+0000 7fd840288700 1 -- 172.21.15.27:0/1345743480 wait complete. 2024-04-25T13:39:13.766 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:13 smithi027 ceph-mon[29140]: pgmap v297: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-25T13:39:13.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:13 smithi122 ceph-mon[32843]: pgmap v297: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-25T13:39:13.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:13 smithi045 ceph-mon[32731]: pgmap v297: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-25T13:39:14.112 INFO:teuthology.orchestra.run.smithi027.stdout:257698037790 2024-04-25T13:39:14.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T13:39:15.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.372+0000 7f4666418700 1 -- 172.21.15.27:0/2499938443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4660107c90 msgr2=0x7f46601080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:15.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.372+0000 7f4666418700 1 --2- 172.21.15.27:0/2499938443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4660107c90 0x7f46601080a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f46540099c0 tx=0x7f4654009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:15.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.372+0000 7f4666418700 1 -- 172.21.15.27:0/2499938443 shutdown_connections 2024-04-25T13:39:15.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.372+0000 7f4666418700 1 --2- 172.21.15.27:0/2499938443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46601085e0 0x7f466010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:15.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.372+0000 7f4666418700 1 --2- 172.21.15.27:0/2499938443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4660103360 0x7f4660103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:15.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.372+0000 7f4666418700 1 --2- 172.21.15.27:0/2499938443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4660107c90 0x7f46601080a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:15.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.372+0000 7f4666418700 1 -- 172.21.15.27:0/2499938443 >> 172.21.15.27:0/2499938443 conn(0x7f46600fec90 msgr2=0x7f46601010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:15.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.373+0000 7f4666418700 1 -- 172.21.15.27:0/2499938443 shutdown_connections 2024-04-25T13:39:15.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.373+0000 7f4666418700 1 -- 172.21.15.27:0/2499938443 wait complete. 2024-04-25T13:39:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.373+0000 7f4666418700 1 Processor -- start 2024-04-25T13:39:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f4666418700 1 -- start start 2024-04-25T13:39:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f4666418700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4660103360 0x7f466019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f4666418700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4660107c90 0x7f466019fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f4666418700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46601085e0 0x7f4660199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f4666418700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4660113240 con 0x7f46601085e0 2024-04-25T13:39:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f4666418700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46601130c0 con 0x7f4660107c90 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f4666418700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f46601133c0 con 0x7f4660103360 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f46649b5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46601085e0 0x7f4660199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f46649b5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46601085e0 0x7f4660199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33866/0 (socket says 172.21.15.27:33866) 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f46649b5700 1 -- 172.21.15.27:0/2961607937 learned_addr learned my addr 172.21.15.27:0/2961607937 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.374+0000 7f465ffff700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4660103360 0x7f466019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f465f7fe700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4660107c90 0x7f466019fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f46649b5700 1 -- 172.21.15.27:0/2961607937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4660103360 msgr2=0x7f466019f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f46649b5700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4660103360 0x7f466019f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f46649b5700 1 -- 172.21.15.27:0/2961607937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4660107c90 msgr2=0x7f466019fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f46649b5700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4660107c90 0x7f466019fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f46649b5700 1 -- 172.21.15.27:0/2961607937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466019a150 con 0x7f46601085e0 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f46649b5700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46601085e0 0x7f4660199aa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f465800ef30 tx=0x7f465800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46580092e0 con 0x7f46601085e0 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466019a3e0 con 0x7f46601085e0 2024-04-25T13:39:15.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4658004bb0 con 0x7f46601085e0 2024-04-25T13:39:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4658004280 con 0x7f46601085e0 2024-04-25T13:39:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.375+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f466006b4e0 con 0x7f46601085e0 2024-04-25T13:39:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.376+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f466006b9a0 con 0x7f46601085e0 2024-04-25T13:39:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.378+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f465801b030 con 0x7f46601085e0 2024-04-25T13:39:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.378+0000 7f465d7fa700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46480afc10 0x7f46480b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.379+0000 7f465ffff700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46480afc10 0x7f46480b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.379+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f46580df050 con 0x7f46601085e0 2024-04-25T13:39:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.379+0000 7f465d7fa700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] conn(0x7f46480b9440 0x7f46480bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.379+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 --> [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46480bbf60 con 0x7f46480b9440 2024-04-25T13:39:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.379+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f46580df530 con 0x7f46601085e0 2024-04-25T13:39:15.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.379+0000 7f465ffff700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46480afc10 0x7f46480b2120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f466019abe0 tx=0x7f4650010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.379+0000 7f465f7fe700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] conn(0x7f46480b9440 0x7f46480bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.380+0000 7f465f7fe700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] conn(0x7f46480b9440 0x7f46480bb890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.381+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 <== osd.7 v2:172.21.15.45:6816/1966399238 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f46480bbf60 con 0x7f46480b9440 2024-04-25T13:39:15.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.397+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 --> [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f466006be20 con 0x7f46480b9440 2024-04-25T13:39:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f465d7fa700 1 -- 172.21.15.27:0/2961607937 <== osd.7 v2:172.21.15.45:6816/1966399238 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f466006be20 con 0x7f46480b9440 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 >> [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] conn(0x7f46480b9440 msgr2=0x7f46480bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] conn(0x7f46480b9440 0x7f46480bb890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46480afc10 msgr2=0x7f46480b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46480afc10 0x7f46480b2120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f466019abe0 tx=0x7f4650010040 comp rx=0 tx=0).stop 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46601085e0 msgr2=0x7f4660199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 --2- 172.21.15.27:0/2961607937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46601085e0 0x7f4660199aa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f465800ef30 tx=0x7f465800c3a0 comp rx=0 tx=0).stop 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f46649b5700 1 -- 172.21.15.27:0/2961607937 reap_dead start 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 shutdown_connections 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 >> 172.21.15.27:0/2961607937 conn(0x7f46600fec90 msgr2=0x7f46601010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 shutdown_connections 2024-04-25T13:39:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:15.398+0000 7f4666418700 1 -- 172.21.15.27:0/2961607937 wait complete. 2024-04-25T13:39:15.610 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:15 smithi027 ceph-mon[29140]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:15.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:15 smithi122 ceph-mon[32843]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:15.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:15 smithi045 ceph-mon[32731]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:17.740 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:17 smithi027 ceph-mon[29140]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:39:17.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:17 smithi122 ceph-mon[32843]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:39:17.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:17 smithi045 ceph-mon[32731]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:39:19.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:19 smithi122 ceph-mon[32843]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:19.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:19 smithi045 ceph-mon[32731]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:19.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:19 smithi027 ceph-mon[29140]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:21.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:21 smithi027 ceph-mon[29140]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:21.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:21 smithi122 ceph-mon[32843]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:21.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:21 smithi045 ceph-mon[32731]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:39:23.714 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:23 smithi027 ceph-mon[29140]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:39:23.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:23 smithi122 ceph-mon[32843]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:39:23.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:23 smithi045 ceph-mon[32731]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:39:24.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 -- 172.21.15.27:0/1793057343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 msgr2=0x7f7540103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:24.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 --2- 172.21.15.27:0/1793057343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 0x7f7540103770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f75300099c0 tx=0x7f7530009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:24.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 -- 172.21.15.27:0/1793057343 shutdown_connections 2024-04-25T13:39:24.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 --2- 172.21.15.27:0/1793057343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75401085e0 0x7f754010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:24.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 --2- 172.21.15.27:0/1793057343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 0x7f7540103770 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:24.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 --2- 172.21.15.27:0/1793057343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7540107c90 0x7f75401080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:24.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 -- 172.21.15.27:0/1793057343 >> 172.21.15.27:0/1793057343 conn(0x7f75400fec90 msgr2=0x7f75401010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:24.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 -- 172.21.15.27:0/1793057343 shutdown_connections 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.915+0000 7f7544c92700 1 -- 172.21.15.27:0/1793057343 wait complete. 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f7544c92700 1 Processor -- start 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f7544c92700 1 -- start start 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f7544c92700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 0x7f754019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f7544c92700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7540107c90 0x7f754019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f7544c92700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75401085e0 0x7f75401a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f7544c92700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f75401132b0 con 0x7f7540103360 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f7544c92700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7540113130 con 0x7f7540107c90 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f7544c92700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7540113430 con 0x7f75401085e0 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.916+0000 7f753e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 0x7f754019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 0x7f754019cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52922/0 (socket says 172.21.15.27:52922) 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753ed9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75401085e0 0x7f75401a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753e59c700 1 -- 172.21.15.27:0/3995338703 learned_addr learned my addr 172.21.15.27:0/3995338703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753dd9b700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7540107c90 0x7f754019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753e59c700 1 -- 172.21.15.27:0/3995338703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75401085e0 msgr2=0x7f75401a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753e59c700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75401085e0 0x7f75401a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753e59c700 1 -- 172.21.15.27:0/3995338703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7540107c90 msgr2=0x7f754019d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753e59c700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7540107c90 0x7f754019d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753e59c700 1 -- 172.21.15.27:0/3995338703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75401a1d80 con 0x7f7540103360 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753ed9d700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f75401085e0 0x7f75401a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.917+0000 7f753e59c700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 0x7f754019cf50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7530009990 tx=0x7f75300047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.918+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7530026070 con 0x7f7540103360 2024-04-25T13:39:24.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.918+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7530003f20 con 0x7f7540103360 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.918+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75401a2010 con 0x7f7540103360 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.918+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75401a9990 con 0x7f7540103360 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.919+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753000fc60 con 0x7f7540103360 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.919+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7520001060 con 0x7f7540103360 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.921+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f7530029070 con 0x7f7540103360 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.921+0000 7f752f7fe700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75280afcc0 0x7f75280b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.921+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f75300e0010 con 0x7f7540103360 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.922+0000 7f753dd9b700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75280afcc0 0x7f75280b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.922+0000 7f752f7fe700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] conn(0x7f75280b94f0 0x7f75280bb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.922+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 --> [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75300043d0 con 0x7f75280b94f0 2024-04-25T13:39:24.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.922+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f75300e04f0 con 0x7f7540103360 2024-04-25T13:39:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.922+0000 7f753ed9d700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] conn(0x7f75280b94f0 0x7f75280bb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.922+0000 7f753dd9b700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75280afcc0 0x7f75280b21d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f754006bdd0 tx=0x7f753400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:24.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.923+0000 7f753ed9d700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] conn(0x7f75280b94f0 0x7f75280bb940 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:24.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.924+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 <== osd.4 v2:172.21.15.45:6808/2006568001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75300043d0 con 0x7f75280b94f0 2024-04-25T13:39:24.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.941+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 --> [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7520002e90 con 0x7f75280b94f0 2024-04-25T13:39:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.941+0000 7f752f7fe700 1 -- 172.21.15.27:0/3995338703 <== osd.4 v2:172.21.15.45:6808/2006568001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7520002e90 con 0x7f75280b94f0 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.941+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 >> [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] conn(0x7f75280b94f0 msgr2=0x7f75280bb940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.941+0000 7f7544c92700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] conn(0x7f75280b94f0 0x7f75280bb940 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.941+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75280afcc0 msgr2=0x7f75280b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.941+0000 7f7544c92700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75280afcc0 0x7f75280b21d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f754006bdd0 tx=0x7f753400a300 comp rx=0 tx=0).stop 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.942+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 msgr2=0x7f754019cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.942+0000 7f7544c92700 1 --2- 172.21.15.27:0/3995338703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7540103360 0x7f754019cf50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7530009990 tx=0x7f75300047b0 comp rx=0 tx=0).stop 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.942+0000 7f753ed9d700 1 -- 172.21.15.27:0/3995338703 reap_dead start 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.942+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 shutdown_connections 2024-04-25T13:39:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.942+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 >> 172.21.15.27:0/3995338703 conn(0x7f75400fec90 msgr2=0x7f75401062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.942+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 shutdown_connections 2024-04-25T13:39:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:24.942+0000 7f7544c92700 1 -- 172.21.15.27:0/3995338703 wait complete. 2024-04-25T13:39:25.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:25 smithi122 ceph-mon[32843]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:25.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:25 smithi045 ceph-mon[32731]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:25.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:25 smithi027 ceph-mon[29140]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:26.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 -- 172.21.15.27:0/340745900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 msgr2=0x7f53c8103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:26.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 --2- 172.21.15.27:0/340745900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 0x7f53c8103830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f53c40099c0 tx=0x7f53c4009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:26.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 -- 172.21.15.27:0/340745900 shutdown_connections 2024-04-25T13:39:26.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 --2- 172.21.15.27:0/340745900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c8108650 0x7f53c810ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:26.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 --2- 172.21.15.27:0/340745900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 0x7f53c8103830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:26.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 --2- 172.21.15.27:0/340745900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53c8107d00 0x7f53c8108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:26.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 -- 172.21.15.27:0/340745900 >> 172.21.15.27:0/340745900 conn(0x7f53c8078cd0 msgr2=0x7f53c8079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:26.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 -- 172.21.15.27:0/340745900 shutdown_connections 2024-04-25T13:39:26.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.264+0000 7f53cfab3700 1 -- 172.21.15.27:0/340745900 wait complete. 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cfab3700 1 Processor -- start 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cfab3700 1 -- start start 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cfab3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 0x7f53c819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cfab3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53c8107d00 0x7f53c819c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cfab3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c8108650 0x7f53c819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cfab3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8113280 con 0x7f53c8103420 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cfab3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8113100 con 0x7f53c8107d00 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cfab3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8113400 con 0x7f53c8108650 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cd84f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 0x7f53c819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cd84f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 0x7f53c819bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52942/0 (socket says 172.21.15.27:52942) 2024-04-25T13:39:26.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cd04e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53c8107d00 0x7f53c819c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53cd84f700 1 -- 172.21.15.27:0/4246996945 learned_addr learned my addr 172.21.15.27:0/4246996945 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.265+0000 7f53ce050700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c8108650 0x7f53c819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53cd84f700 1 -- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c8108650 msgr2=0x7f53c819c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53cd84f700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c8108650 0x7f53c819c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53cd84f700 1 -- 172.21.15.27:0/4246996945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53c8107d00 msgr2=0x7f53c819c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53cd84f700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53c8107d00 0x7f53c819c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53cd84f700 1 -- 172.21.15.27:0/4246996945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c81a6f70 con 0x7f53c8103420 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53ce050700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53c8108650 0x7f53c819c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53cd84f700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 0x7f53c819bc20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f53c400b580 tx=0x7f53c40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4027070 con 0x7f53c8103420 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c81a7200 con 0x7f53c8103420 2024-04-25T13:39:26.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53c4003f20 con 0x7f53c8103420 2024-04-25T13:39:26.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4022d70 con 0x7f53c8103420 2024-04-25T13:39:26.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.266+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c81a7650 con 0x7f53c8103420 2024-04-25T13:39:26.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.267+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f53c8063da0 con 0x7f53c8103420 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.269+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f53c4022500 con 0x7f53c8103420 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.269+0000 7f53beffd700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53b40afc10 0x7f53b40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.269+0000 7f53cd04e700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53b40afc10 0x7f53b40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.269+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f53c40dfe30 con 0x7f53c8103420 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.270+0000 7f53beffd700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] conn(0x7f53b40b9440 0x7f53b40bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.270+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 --> [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53c40043d0 con 0x7f53b40b9440 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.270+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f53c40e0310 con 0x7f53c8103420 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.270+0000 7f53cd04e700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53b40afc10 0x7f53b40b2120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f53c80792b0 tx=0x7f53b800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:26.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.270+0000 7f53ce050700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] conn(0x7f53b40b9440 0x7f53b40bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:26.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.271+0000 7f53ce050700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] conn(0x7f53b40b9440 0x7f53b40bb890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:26.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.271+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 <== osd.11 v2:172.21.15.122:6824/3308484009 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53c40043d0 con 0x7f53b40b9440 2024-04-25T13:39:26.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.288+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 --> [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f53c80641a0 con 0x7f53b40b9440 2024-04-25T13:39:26.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53beffd700 1 -- 172.21.15.27:0/4246996945 <== osd.11 v2:172.21.15.122:6824/3308484009 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f53c80641a0 con 0x7f53b40b9440 2024-04-25T13:39:26.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] conn(0x7f53b40b9440 msgr2=0x7f53b40bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:26.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] conn(0x7f53b40b9440 0x7f53b40bb890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:26.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53b40afc10 msgr2=0x7f53b40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53b40afc10 0x7f53b40b2120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f53c80792b0 tx=0x7f53b800b410 comp rx=0 tx=0).stop 2024-04-25T13:39:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 msgr2=0x7f53c819bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 --2- 172.21.15.27:0/4246996945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53c8103420 0x7f53c819bc20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f53c400b580 tx=0x7f53c40047b0 comp rx=0 tx=0).stop 2024-04-25T13:39:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53ce050700 1 -- 172.21.15.27:0/4246996945 reap_dead start 2024-04-25T13:39:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 shutdown_connections 2024-04-25T13:39:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 >> 172.21.15.27:0/4246996945 conn(0x7f53c8078cd0 msgr2=0x7f53c8100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 shutdown_connections 2024-04-25T13:39:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:26.289+0000 7f53cfab3700 1 -- 172.21.15.27:0/4246996945 wait complete. 2024-04-25T13:39:26.940 INFO:teuthology.orchestra.run.smithi027.stdout:206158430249 2024-04-25T13:39:26.941 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T13:39:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.155+0000 7ff86ce9d700 1 -- 172.21.15.27:0/1327557650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 msgr2=0x7ff86810e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.155+0000 7ff86ce9d700 1 --2- 172.21.15.27:0/1327557650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 0x7ff86810e230 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009990 tx=0x7ff85c009ca0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.155+0000 7ff86ce9d700 1 -- 172.21.15.27:0/1327557650 shutdown_connections 2024-04-25T13:39:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.155+0000 7ff86ce9d700 1 --2- 172.21.15.27:0/1327557650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff86810e7c0 0x7ff868110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.155+0000 7ff86ce9d700 1 --2- 172.21.15.27:0/1327557650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff868105390 0x7ff8681057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.155+0000 7ff86ce9d700 1 --2- 172.21.15.27:0/1327557650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 0x7ff86810e230 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.155+0000 7ff86ce9d700 1 -- 172.21.15.27:0/1327557650 >> 172.21.15.27:0/1327557650 conn(0x7ff8680fec90 msgr2=0x7ff8681010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.156+0000 7ff86ce9d700 1 -- 172.21.15.27:0/1327557650 shutdown_connections 2024-04-25T13:39:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.156+0000 7ff86ce9d700 1 -- 172.21.15.27:0/1327557650 wait complete. 2024-04-25T13:39:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86ce9d700 1 Processor -- start 2024-04-25T13:39:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86ce9d700 1 -- start start 2024-04-25T13:39:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86ce9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff868105390 0x7ff86810e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86ce9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 0x7ff868108210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86ce9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86810e7c0 0x7ff868108790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86ce9d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff86806d810 con 0x7ff868105ce0 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86ce9d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff86806d690 con 0x7ff86810e7c0 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86ce9d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff86806d990 con 0x7ff868105390 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff865d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 0x7ff868108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff865d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 0x7ff868108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52958/0 (socket says 172.21.15.27:52958) 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff865d9b700 1 -- 172.21.15.27:0/2216321815 learned_addr learned my addr 172.21.15.27:0/2216321815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff86659c700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff868105390 0x7ff86810e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.157+0000 7ff866d9d700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86810e7c0 0x7ff868108790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff865d9b700 1 -- 172.21.15.27:0/2216321815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff868105390 msgr2=0x7ff86810e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff865d9b700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff868105390 0x7ff86810e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff865d9b700 1 -- 172.21.15.27:0/2216321815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86810e7c0 msgr2=0x7ff868108790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff865d9b700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff86810e7c0 0x7ff868108790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff865d9b700 1 -- 172.21.15.27:0/2216321815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff868108fc0 con 0x7ff868105ce0 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff865d9b700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 0x7ff868108210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff85c005910 tx=0x7ff85c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c027070 con 0x7ff868105ce0 2024-04-25T13:39:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c004020 con 0x7ff868105ce0 2024-04-25T13:39:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8681af9e0 con 0x7ff868105ce0 2024-04-25T13:39:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c00f460 con 0x7ff868105ce0 2024-04-25T13:39:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.158+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8681afee0 con 0x7ff868105ce0 2024-04-25T13:39:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.159+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff868063da0 con 0x7ff868105ce0 2024-04-25T13:39:27.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.161+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7ff85c02a070 con 0x7ff868105ce0 2024-04-25T13:39:27.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.162+0000 7ff8577fe700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8500afc90 0x7ff8500b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.162+0000 7ff86659c700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8500afc90 0x7ff8500b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.162+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7ff85c0e0890 con 0x7ff868105ce0 2024-04-25T13:39:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.162+0000 7ff8577fe700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] conn(0x7ff8500b94c0 0x7ff8500bb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.162+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 --> [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff85c02de50 con 0x7ff8500b94c0 2024-04-25T13:39:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.162+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7ff85c0e0d70 con 0x7ff868105ce0 2024-04-25T13:39:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.162+0000 7ff86659c700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8500afc90 0x7ff8500b21a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff858005950 tx=0x7ff85800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.162+0000 7ff866d9d700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] conn(0x7ff8500b94c0 0x7ff8500bb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.163+0000 7ff866d9d700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] conn(0x7ff8500b94c0 0x7ff8500bb910 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:27.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.164+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 <== osd.5 v2:172.21.15.122:6808/3058238339 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff85c02de50 con 0x7ff8500b94c0 2024-04-25T13:39:27.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.183+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 --> [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff868064220 con 0x7ff8500b94c0 2024-04-25T13:39:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff8577fe700 1 -- 172.21.15.27:0/2216321815 <== osd.5 v2:172.21.15.122:6808/3058238339 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff868064220 con 0x7ff8500b94c0 2024-04-25T13:39:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 >> [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] conn(0x7ff8500b94c0 msgr2=0x7ff8500bb910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] conn(0x7ff8500b94c0 0x7ff8500bb910 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8500afc90 msgr2=0x7ff8500b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8500afc90 0x7ff8500b21a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff858005950 tx=0x7ff85800a300 comp rx=0 tx=0).stop 2024-04-25T13:39:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 msgr2=0x7ff868108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 --2- 172.21.15.27:0/2216321815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff868105ce0 0x7ff868108210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff85c005910 tx=0x7ff85c0047b0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff866d9d700 1 -- 172.21.15.27:0/2216321815 reap_dead start 2024-04-25T13:39:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 shutdown_connections 2024-04-25T13:39:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 >> 172.21.15.27:0/2216321815 conn(0x7ff8680fec90 msgr2=0x7ff8680ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 shutdown_connections 2024-04-25T13:39:27.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.184+0000 7ff86ce9d700 1 -- 172.21.15.27:0/2216321815 wait complete. 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 -- 172.21.15.27:0/468164116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 msgr2=0x7f83181080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 --2- 172.21.15.27:0/468164116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 0x7f83181080a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f83000099c0 tx=0x7f8300009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 -- 172.21.15.27:0/468164116 shutdown_connections 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 --2- 172.21.15.27:0/468164116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83181085e0 0x7f831810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 --2- 172.21.15.27:0/468164116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8318103360 0x7f8318103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 --2- 172.21.15.27:0/468164116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 0x7f83181080a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 -- 172.21.15.27:0/468164116 >> 172.21.15.27:0/468164116 conn(0x7f83180fecb0 msgr2=0x7f83181010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 -- 172.21.15.27:0/468164116 shutdown_connections 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.627+0000 7f831c84f700 1 -- 172.21.15.27:0/468164116 wait complete. 2024-04-25T13:39:27.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831c84f700 1 Processor -- start 2024-04-25T13:39:27.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831c84f700 1 -- start start 2024-04-25T13:39:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831c84f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8318103360 0x7f831819f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831c84f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 0x7f831819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831c84f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83181085e0 0x7f8318199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831c84f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f83181131f0 con 0x7f8318107c90 2024-04-25T13:39:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831c84f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8318113070 con 0x7f8318103360 2024-04-25T13:39:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831c84f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8318113370 con 0x7f83181085e0 2024-04-25T13:39:27.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f8315d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 0x7f831819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f8315d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 0x7f831819fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52972/0 (socket says 172.21.15.27:52972) 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f8315d9b700 1 -- 172.21.15.27:0/4053297492 learned_addr learned my addr 172.21.15.27:0/4053297492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f831659c700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8318103360 0x7f831819f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.628+0000 7f8316d9d700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83181085e0 0x7f8318199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f8315d9b700 1 -- 172.21.15.27:0/4053297492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83181085e0 msgr2=0x7f8318199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f8315d9b700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83181085e0 0x7f8318199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f8315d9b700 1 -- 172.21.15.27:0/4053297492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8318103360 msgr2=0x7f831819f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f8315d9b700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8318103360 0x7f831819f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f8315d9b700 1 -- 172.21.15.27:0/4053297492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831819a1b0 con 0x7f8318107c90 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f8315d9b700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 0x7f831819fdf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8300006010 tx=0x7f83000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300027070 con 0x7f8318107c90 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831819a440 con 0x7f8318107c90 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8300003f20 con 0x7f8318107c90 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300022c60 con 0x7f8318107c90 2024-04-25T13:39:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.629+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83181a6740 con 0x7f8318107c90 2024-04-25T13:39:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.630+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8318063d60 con 0x7f8318107c90 2024-04-25T13:39:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.632+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f830002a070 con 0x7f8318107c90 2024-04-25T13:39:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.632+0000 7f830f7fe700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83040afc10 0x7f83040b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.632+0000 7f831659c700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83040afc10 0x7f83040b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.632+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f8300022e00 con 0x7f8318107c90 2024-04-25T13:39:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.633+0000 7f830f7fe700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] conn(0x7f83040b9440 0x7f83040bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.633+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 --> [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8300001d30 con 0x7f83040b9440 2024-04-25T13:39:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.633+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f83000e09c0 con 0x7f8318107c90 2024-04-25T13:39:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.633+0000 7f8316d9d700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] conn(0x7f83040b9440 0x7f83040bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.633+0000 7f831659c700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83040afc10 0x7f83040b2120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8310009550 tx=0x7f83100091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.633+0000 7f8316d9d700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] conn(0x7f83040b9440 0x7f83040bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.634+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 <== osd.0 v2:172.21.15.27:6802/342047280 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8300001d30 con 0x7f83040b9440 2024-04-25T13:39:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.650+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 --> [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8318064160 con 0x7f83040b9440 2024-04-25T13:39:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f830f7fe700 1 -- 172.21.15.27:0/4053297492 <== osd.0 v2:172.21.15.27:6802/342047280 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8318064160 con 0x7f83040b9440 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] conn(0x7f83040b9440 msgr2=0x7f83040bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] conn(0x7f83040b9440 0x7f83040bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83040afc10 msgr2=0x7f83040b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83040afc10 0x7f83040b2120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8310009550 tx=0x7f83100091b0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 msgr2=0x7f831819fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 --2- 172.21.15.27:0/4053297492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8318107c90 0x7f831819fdf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8300006010 tx=0x7f83000047b0 comp rx=0 tx=0).stop 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f8316d9d700 1 -- 172.21.15.27:0/4053297492 reap_dead start 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 shutdown_connections 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 >> 172.21.15.27:0/4053297492 conn(0x7f83180fecb0 msgr2=0x7f8318107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 shutdown_connections 2024-04-25T13:39:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:27.651+0000 7f831c84f700 1 -- 172.21.15.27:0/4053297492 wait complete. 2024-04-25T13:39:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:27 smithi122 ceph-mon[32843]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:27.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:27 smithi045 ceph-mon[32731]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:27.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:27 smithi027 ceph-mon[29140]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 -- 172.21.15.27:0/342877580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 msgr2=0x7f8f08111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 --2- 172.21.15.27:0/342877580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 0x7f8f08111080 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80099c0 tx=0x7f8ef8009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 -- 172.21.15.27:0/342877580 shutdown_connections 2024-04-25T13:39:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 --2- 172.21.15.27:0/342877580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f081044a0 0x7f8f081048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 --2- 172.21.15.27:0/342877580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 0x7f8f08111080 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 --2- 172.21.15.27:0/342877580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f08108dd0 0x7f8f081091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 -- 172.21.15.27:0/342877580 >> 172.21.15.27:0/342877580 conn(0x7f8f080ffd70 msgr2=0x7f8f081021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 -- 172.21.15.27:0/342877580 shutdown_connections 2024-04-25T13:39:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.060+0000 7f8f0f97a700 1 -- 172.21.15.27:0/342877580 wait complete. 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0f97a700 1 Processor -- start 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0f97a700 1 -- start start 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0f97a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f081044a0 0x7f8f08076aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0f97a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f08108dd0 0x7f8f08076fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0f97a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 0x7f8f08077520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0f97a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8f08114350 con 0x7f8f08109720 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0f97a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081141d0 con 0x7f8f081044a0 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0f97a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081144d0 con 0x7f8f08108dd0 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0df17700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 0x7f8f08077520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0df17700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 0x7f8f08077520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52998/0 (socket says 172.21.15.27:52998) 2024-04-25T13:39:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0df17700 1 -- 172.21.15.27:0/1588059170 learned_addr learned my addr 172.21.15.27:0/1588059170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.061+0000 7f8f0d716700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f081044a0 0x7f8f08076aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0cf15700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f08108dd0 0x7f8f08076fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0df17700 1 -- 172.21.15.27:0/1588059170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f08108dd0 msgr2=0x7f8f08076fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0df17700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f08108dd0 0x7f8f08076fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0df17700 1 -- 172.21.15.27:0/1588059170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f081044a0 msgr2=0x7f8f08076aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0df17700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f081044a0 0x7f8f08076aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0df17700 1 -- 172.21.15.27:0/1588059170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f081a8fe0 con 0x7f8f08109720 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0df17700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 0x7f8f08077520 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8005910 tx=0x7f8ef8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8027070 con 0x7f8f08109720 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef8003f20 con 0x7f8f08109720 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f081a9270 con 0x7f8f08109720 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8022ba0 con 0x7f8f08109720 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.062+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f081a96c0 con 0x7f8f08109720 2024-04-25T13:39:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.063+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8f08063da0 con 0x7f8f08109720 2024-04-25T13:39:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.065+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f8ef802a070 con 0x7f8f08109720 2024-04-25T13:39:28.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.066+0000 7f8f067fc700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ef00afc10 0x7f8ef00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:28.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.066+0000 7f8f0d716700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ef00afc10 0x7f8ef00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:28.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.066+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f8ef80e0120 con 0x7f8f08109720 2024-04-25T13:39:28.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.066+0000 7f8f067fc700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] conn(0x7f8ef00b9440 0x7f8ef00bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:28.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.066+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 --> [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ef8032ea0 con 0x7f8ef00b9440 2024-04-25T13:39:28.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.066+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f8ef80e0600 con 0x7f8f08109720 2024-04-25T13:39:28.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.066+0000 7f8f0d716700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ef00afc10 0x7f8ef00b2120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f000060b0 tx=0x7f8f0000c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:28.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.066+0000 7f8f0cf15700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] conn(0x7f8ef00b9440 0x7f8ef00bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:28.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.067+0000 7f8f0cf15700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] conn(0x7f8ef00b9440 0x7f8ef00bb890 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:28.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.068+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 <== osd.10 v2:172.21.15.45:6824/374944878 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ef8032ea0 con 0x7f8ef00b9440 2024-04-25T13:39:28.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.084+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 --> [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8f080641a0 con 0x7f8ef00b9440 2024-04-25T13:39:28.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.084+0000 7f8f067fc700 1 -- 172.21.15.27:0/1588059170 <== osd.10 v2:172.21.15.45:6824/374944878 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8f080641a0 con 0x7f8ef00b9440 2024-04-25T13:39:28.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 >> [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] conn(0x7f8ef00b9440 msgr2=0x7f8ef00bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] conn(0x7f8ef00b9440 0x7f8ef00bb890 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ef00afc10 msgr2=0x7f8ef00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ef00afc10 0x7f8ef00b2120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f000060b0 tx=0x7f8f0000c040 comp rx=0 tx=0).stop 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 msgr2=0x7f8f08077520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 --2- 172.21.15.27:0/1588059170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f08109720 0x7f8f08077520 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8005910 tx=0x7f8ef8009580 comp rx=0 tx=0).stop 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0df17700 1 -- 172.21.15.27:0/1588059170 reap_dead start 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 shutdown_connections 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 >> 172.21.15.27:0/1588059170 conn(0x7f8f080ffd70 msgr2=0x7f8f08108540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 shutdown_connections 2024-04-25T13:39:28.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.085+0000 7f8f0f97a700 1 -- 172.21.15.27:0/1588059170 wait complete. 2024-04-25T13:39:28.860 INFO:teuthology.orchestra.run.smithi027.stdout:158913790006 2024-04-25T13:39:28.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T13:39:29.465 INFO:teuthology.orchestra.run.smithi027.stdout:304942678038 2024-04-25T13:39:29.466 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T13:39:29.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 -- 172.21.15.27:0/4145645621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e01085e0 msgr2=0x7f45e010ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 --2- 172.21.15.27:0/4145645621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e01085e0 0x7f45e010ff40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f45d00099c0 tx=0x7f45d0009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 -- 172.21.15.27:0/4145645621 shutdown_connections 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 --2- 172.21.15.27:0/4145645621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45e0103360 0x7f45e0103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 --2- 172.21.15.27:0/4145645621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e01085e0 0x7f45e010ff40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 --2- 172.21.15.27:0/4145645621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45e0107c90 0x7f45e01080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 -- 172.21.15.27:0/4145645621 >> 172.21.15.27:0/4145645621 conn(0x7f45e00fec90 msgr2=0x7f45e01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 -- 172.21.15.27:0/4145645621 shutdown_connections 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.261+0000 7f45e574f700 1 -- 172.21.15.27:0/4145645621 wait complete. 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.262+0000 7f45e574f700 1 Processor -- start 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.262+0000 7f45e574f700 1 -- start start 2024-04-25T13:39:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.262+0000 7f45e574f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e0103360 0x7f45e019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.262+0000 7f45e574f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45e0107c90 0x7f45e019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.262+0000 7f45e574f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45e01085e0 0x7f45e01a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.262+0000 7f45e574f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f45e0113210 con 0x7f45e0103360 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e0103360 0x7f45e019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45e574f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f45e0113090 con 0x7f45e01085e0 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45e574f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f45e0113390 con 0x7f45e0107c90 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e0103360 0x7f45e019ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53022/0 (socket says 172.21.15.27:53022) 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 -- 172.21.15.27:0/1922679997 learned_addr learned my addr 172.21.15.27:0/1922679997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45de7fc700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45e0107c90 0x7f45e019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45df7fe700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45e01085e0 0x7f45e01a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 -- 172.21.15.27:0/1922679997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45e0107c90 msgr2=0x7f45e019d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45e0107c90 0x7f45e019d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 -- 172.21.15.27:0/1922679997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45e01085e0 msgr2=0x7f45e01a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45e01085e0 0x7f45e01a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 -- 172.21.15.27:0/1922679997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e01a1c70 con 0x7f45e0103360 2024-04-25T13:39:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45df7fe700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45e01085e0 0x7f45e01a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:39:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45deffd700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e0103360 0x7f45e019ce50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f45d80060b0 tx=0x7f45d800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d801a070 con 0x7f45e0103360 2024-04-25T13:39:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e01a1f60 con 0x7f45e0103360 2024-04-25T13:39:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.263+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e01a9930 con 0x7f45e0103360 2024-04-25T13:39:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.264+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45d800ddd0 con 0x7f45e0103360 2024-04-25T13:39:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.264+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8024060 con 0x7f45e0103360 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.264+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f45c0001030 con 0x7f45e0103360 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.266+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f45d801d070 con 0x7f45e0103360 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.267+0000 7f45d7fff700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45cc0afc10 0x7f45cc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.267+0000 7f45de7fc700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45cc0afc10 0x7f45cc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.267+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f45d80debc0 con 0x7f45e0103360 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.267+0000 7f45d7fff700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] conn(0x7f45cc0b9440 0x7f45cc0bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.267+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 --> [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45d8022e70 con 0x7f45cc0b9440 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.267+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f45d80df0a0 con 0x7f45e0103360 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.267+0000 7f45df7fe700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] conn(0x7f45cc0b9440 0x7f45cc0bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.267+0000 7f45de7fc700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45cc0afc10 0x7f45cc0b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f45e006c490 tx=0x7f45c800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.268+0000 7f45df7fe700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] conn(0x7f45cc0b9440 0x7f45cc0bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.268+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 <== osd.6 v2:172.21.15.27:6818/100381961 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45d8022e70 con 0x7f45cc0b9440 2024-04-25T13:39:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.286+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 --> [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f45c0002e60 con 0x7f45cc0b9440 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.286+0000 7f45d7fff700 1 -- 172.21.15.27:0/1922679997 <== osd.6 v2:172.21.15.27:6818/100381961 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f45c0002e60 con 0x7f45cc0b9440 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] conn(0x7f45cc0b9440 msgr2=0x7f45cc0bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] conn(0x7f45cc0b9440 0x7f45cc0bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45cc0afc10 msgr2=0x7f45cc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45cc0afc10 0x7f45cc0b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f45e006c490 tx=0x7f45c800a380 comp rx=0 tx=0).stop 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e0103360 msgr2=0x7f45e019ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 --2- 172.21.15.27:0/1922679997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e0103360 0x7f45e019ce50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f45d80060b0 tx=0x7f45d800d3b0 comp rx=0 tx=0).stop 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45df7fe700 1 -- 172.21.15.27:0/1922679997 reap_dead start 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 shutdown_connections 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 >> 172.21.15.27:0/1922679997 conn(0x7f45e00fec90 msgr2=0x7f45e010eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 shutdown_connections 2024-04-25T13:39:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:28.287+0000 7f45e574f700 1 -- 172.21.15.27:0/1922679997 wait complete. 2024-04-25T13:39:29.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:29 smithi122 ceph-mon[32843]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-25T13:39:29.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:29 smithi045 ceph-mon[32731]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-25T13:39:29.885 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:29 smithi027 ceph-mon[29140]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-25T13:39:30.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.613+0000 7fec7c076700 1 -- 172.21.15.27:0/2163575539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74107cc0 msgr2=0x7fec741080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:30.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.613+0000 7fec7c076700 1 --2- 172.21.15.27:0/2163575539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74107cc0 0x7fec741080d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fec700099c0 tx=0x7fec70009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:30.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.613+0000 7fec7c076700 1 -- 172.21.15.27:0/2163575539 shutdown_connections 2024-04-25T13:39:30.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.613+0000 7fec7c076700 1 --2- 172.21.15.27:0/2163575539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec74108610 0x7fec7410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:30.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.613+0000 7fec7c076700 1 --2- 172.21.15.27:0/2163575539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec74103390 0x7fec741037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:30.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.613+0000 7fec7c076700 1 --2- 172.21.15.27:0/2163575539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74107cc0 0x7fec741080d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:30.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.613+0000 7fec7c076700 1 -- 172.21.15.27:0/2163575539 >> 172.21.15.27:0/2163575539 conn(0x7fec740fec90 msgr2=0x7fec741010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:30.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.614+0000 7fec7c076700 1 -- 172.21.15.27:0/2163575539 shutdown_connections 2024-04-25T13:39:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.614+0000 7fec7c076700 1 -- 172.21.15.27:0/2163575539 wait complete. 2024-04-25T13:39:30.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.614+0000 7fec7c076700 1 Processor -- start 2024-04-25T13:39:30.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.614+0000 7fec7c076700 1 -- start start 2024-04-25T13:39:30.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7c076700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec74103390 0x7fec7419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:30.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7c076700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec74107cc0 0x7fec7419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7c076700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74108610 0x7fec741999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7c076700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fec74113220 con 0x7fec74108610 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7c076700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fec741130a0 con 0x7fec74107cc0 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7c076700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fec741133a0 con 0x7fec74103390 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74108610 0x7fec741999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74108610 0x7fec741999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53030/0 (socket says 172.21.15.27:53030) 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 -- 172.21.15.27:0/4139766806 learned_addr learned my addr 172.21.15.27:0/4139766806 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec79e12700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec74103390 0x7fec7419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec79611700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec74107cc0 0x7fec7419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 -- 172.21.15.27:0/4139766806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec74103390 msgr2=0x7fec7419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec74103390 0x7fec7419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 -- 172.21.15.27:0/4139766806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec74107cc0 msgr2=0x7fec7419fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec74107cc0 0x7fec7419fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 -- 172.21.15.27:0/4139766806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7419a090 con 0x7fec74108610 2024-04-25T13:39:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec7a613700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74108610 0x7fec741999e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec6c007ee0 tx=0x7fec6c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.615+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00dbd0 con 0x7fec74108610 2024-04-25T13:39:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.616+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec6c00dd70 con 0x7fec74108610 2024-04-25T13:39:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.616+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c005990 con 0x7fec74108610 2024-04-25T13:39:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.616+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec7419a380 con 0x7fec74108610 2024-04-25T13:39:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.616+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec7406b4e0 con 0x7fec74108610 2024-04-25T13:39:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.616+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec7419a840 con 0x7fec74108610 2024-04-25T13:39:30.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.618+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fec6c01a020 con 0x7fec74108610 2024-04-25T13:39:30.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.619+0000 7fec6affd700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fec600afc90 0x7fec600b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:30.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.619+0000 7fec79e12700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fec600afc90 0x7fec600b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:30.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.619+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8190+0+0 (secure 0 0 0) 0x7fec6c0deec0 con 0x7fec74108610 2024-04-25T13:39:30.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.619+0000 7fec6affd700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] conn(0x7fec600b97a0 0x7fec600bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:30.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.619+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 --> [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec600bc360 con 0x7fec600b97a0 2024-04-25T13:39:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.619+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7fec6c0df310 con 0x7fec74108610 2024-04-25T13:39:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.619+0000 7fec79e12700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fec600afc90 0x7fec600b21a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec64005950 tx=0x7fec6400e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.620+0000 7fec79611700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] conn(0x7fec600b97a0 0x7fec600bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:30.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.620+0000 7fec79611700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] conn(0x7fec600b97a0 0x7fec600bbc90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:30.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.621+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 <== osd.8 v2:172.21.15.122:6816/1308119675 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fec600bc360 con 0x7fec600b97a0 2024-04-25T13:39:30.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.637+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 --> [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec7406bcb0 con 0x7fec600b97a0 2024-04-25T13:39:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec6affd700 1 -- 172.21.15.27:0/4139766806 <== osd.8 v2:172.21.15.122:6816/1308119675 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fec7406bcb0 con 0x7fec600b97a0 2024-04-25T13:39:30.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 >> [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] conn(0x7fec600b97a0 msgr2=0x7fec600bbc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] conn(0x7fec600b97a0 0x7fec600bbc90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fec600afc90 msgr2=0x7fec600b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fec600afc90 0x7fec600b21a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec64005950 tx=0x7fec6400e040 comp rx=0 tx=0).stop 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74108610 msgr2=0x7fec741999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 --2- 172.21.15.27:0/4139766806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec74108610 0x7fec741999e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec6c007ee0 tx=0x7fec6c00d3b0 comp rx=0 tx=0).stop 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7a613700 1 -- 172.21.15.27:0/4139766806 reap_dead start 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 shutdown_connections 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 >> 172.21.15.27:0/4139766806 conn(0x7fec740fec90 msgr2=0x7fec74109570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 shutdown_connections 2024-04-25T13:39:30.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:30.638+0000 7fec7c076700 1 -- 172.21.15.27:0/4139766806 wait complete. 2024-04-25T13:39:30.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T13:39:30.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-25T13:39:30.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T13:39:30.732 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-25T13:39:30.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-25T13:39:30.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-25T13:39:30.733 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-25T13:39:30.757 INFO:teuthology.orchestra.run.smithi027.stdout:137438953531 2024-04-25T13:39:30.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T13:39:30.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T13:39:30.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-25T13:39:30.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T13:39:30.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-25T13:39:30.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-25T13:39:30.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-25T13:39:30.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-25T13:39:30.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T13:39:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-04-25T13:39:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T13:39:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-25T13:39:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-25T13:39:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-25T13:39:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-25T13:39:31.412 INFO:teuthology.orchestra.run.smithi027.stdout:231928234023 2024-04-25T13:39:31.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T13:39:31.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-25T13:39:31.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T13:39:31.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-25T13:39:31.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-04-25T13:39:31.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-25T13:39:31.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-25T13:39:31.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-25T13:39:31.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-25T13:39:31.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:31 smithi027 ceph-mon[29140]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T13:39:31.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-25T13:39:31.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T13:39:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-25T13:39:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-04-25T13:39:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-25T13:39:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-25T13:39:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-25T13:39:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-25T13:39:31.809 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:31 smithi122 ceph-mon[32843]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-25T13:39:31.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:31 smithi045 ceph-mon[32731]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T13:39:32.116 INFO:teuthology.orchestra.run.smithi027.stdout:34359738448 2024-04-25T13:39:32.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.550+0000 7f6f3c231700 1 -- 172.21.15.27:0/119624736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34103390 msgr2=0x7f6f341037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.550+0000 7f6f3c231700 1 --2- 172.21.15.27:0/119624736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34103390 0x7f6f341037a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6f300099c0 tx=0x7f6f30009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.551+0000 7f6f3c231700 1 -- 172.21.15.27:0/119624736 shutdown_connections 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.551+0000 7f6f3c231700 1 --2- 172.21.15.27:0/119624736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f34108610 0x7f6f3410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.551+0000 7f6f3c231700 1 --2- 172.21.15.27:0/119624736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34103390 0x7f6f341037a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.551+0000 7f6f3c231700 1 --2- 172.21.15.27:0/119624736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f34107cc0 0x7f6f341080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.551+0000 7f6f3c231700 1 -- 172.21.15.27:0/119624736 >> 172.21.15.27:0/119624736 conn(0x7f6f340fec90 msgr2=0x7f6f341010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.551+0000 7f6f3c231700 1 -- 172.21.15.27:0/119624736 shutdown_connections 2024-04-25T13:39:32.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.551+0000 7f6f3c231700 1 -- 172.21.15.27:0/119624736 wait complete. 2024-04-25T13:39:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3c231700 1 Processor -- start 2024-04-25T13:39:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3c231700 1 -- start start 2024-04-25T13:39:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3c231700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f34103390 0x7f6f3419bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3c231700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34107cc0 0x7f6f3419c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:32.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3c231700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f34108610 0x7f6f3419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3c231700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6f34113240 con 0x7f6f34107cc0 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3c231700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6f341130c0 con 0x7f6f34103390 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3c231700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6f341133c0 con 0x7f6f34108610 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f397cc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34107cc0 0x7f6f3419c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f397cc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34107cc0 0x7f6f3419c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53052/0 (socket says 172.21.15.27:53052) 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f397cc700 1 -- 172.21.15.27:0/3859474133 learned_addr learned my addr 172.21.15.27:0/3859474133 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f39fcd700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f34103390 0x7f6f3419bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.552+0000 7f6f3a7ce700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f34108610 0x7f6f3419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f397cc700 1 -- 172.21.15.27:0/3859474133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f34108610 msgr2=0x7f6f3419c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f397cc700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f34108610 0x7f6f3419c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f397cc700 1 -- 172.21.15.27:0/3859474133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f34103390 msgr2=0x7f6f3419bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f397cc700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f34103390 0x7f6f3419bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f397cc700 1 -- 172.21.15.27:0/3859474133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f341a6e50 con 0x7f6f34107cc0 2024-04-25T13:39:32.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f39fcd700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f34103390 0x7f6f3419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:32.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f397cc700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34107cc0 0x7f6f3419c020 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400d950 tx=0x7f6f2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:32.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24016070 con 0x7f6f34107cc0 2024-04-25T13:39:32.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f341a7140 con 0x7f6f34107cc0 2024-04-25T13:39:32.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f24004830 con 0x7f6f34107cc0 2024-04-25T13:39:32.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24005520 con 0x7f6f34107cc0 2024-04-25T13:39:32.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.553+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f341a76d0 con 0x7f6f34107cc0 2024-04-25T13:39:32.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.554+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6f34063da0 con 0x7f6f34107cc0 2024-04-25T13:39:32.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.556+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6f24004220 con 0x7f6f34107cc0 2024-04-25T13:39:32.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.557+0000 7f6f2affd700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6f200afc90 0x7f6f200b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:32.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.557+0000 7f6f39fcd700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6f200afc90 0x7f6f200b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:32.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.557+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6f240df090 con 0x7f6f34107cc0 2024-04-25T13:39:32.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.557+0000 7f6f2affd700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] conn(0x7f6f200b97a0 0x7f6f200bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:32.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.557+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 --> [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f24011ec0 con 0x7f6f200b97a0 2024-04-25T13:39:32.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.557+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f6f240df4e0 con 0x7f6f34107cc0 2024-04-25T13:39:32.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.557+0000 7f6f3a7ce700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] conn(0x7f6f200b97a0 0x7f6f200bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:32.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.557+0000 7f6f39fcd700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6f200afc90 0x7f6f200b21a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6f30000c00 tx=0x7f6f3001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:32.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.558+0000 7f6f3a7ce700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] conn(0x7f6f200b97a0 0x7f6f200bbc90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:32.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.558+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 <== osd.3 v2:172.21.15.27:6810/1699526500 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f24011ec0 con 0x7f6f200b97a0 2024-04-25T13:39:32.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.576+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 --> [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6f340641d0 con 0x7f6f200b97a0 2024-04-25T13:39:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f2affd700 1 -- 172.21.15.27:0/3859474133 <== osd.3 v2:172.21.15.27:6810/1699526500 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6f340641d0 con 0x7f6f200b97a0 2024-04-25T13:39:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] conn(0x7f6f200b97a0 msgr2=0x7f6f200bbc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] conn(0x7f6f200b97a0 0x7f6f200bbc90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6f200afc90 msgr2=0x7f6f200b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6f200afc90 0x7f6f200b21a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6f30000c00 tx=0x7f6f3001a040 comp rx=0 tx=0).stop 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34107cc0 msgr2=0x7f6f3419c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 --2- 172.21.15.27:0/3859474133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f34107cc0 0x7f6f3419c020 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400d950 tx=0x7f6f2400dc60 comp rx=0 tx=0).stop 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3a7ce700 1 -- 172.21.15.27:0/3859474133 reap_dead start 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 shutdown_connections 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 >> 172.21.15.27:0/3859474133 conn(0x7f6f340fec90 msgr2=0x7f6f34101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 shutdown_connections 2024-04-25T13:39:32.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:32.577+0000 7f6f3c231700 1 -- 172.21.15.27:0/3859474133 wait complete. 2024-04-25T13:39:32.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:32 smithi122 ceph-mon[32843]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T13:39:32.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:32 smithi045 ceph-mon[32731]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T13:39:32.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:32 smithi027 ceph-mon[29140]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T13:39:33.284 INFO:teuthology.orchestra.run.smithi027.stdout:111669149762 2024-04-25T13:39:33.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T13:39:33.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:33 smithi122 ceph-mon[32843]: pgmap v309: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:33.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:33 smithi045 ceph-mon[32731]: pgmap v309: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:33.865 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:33 smithi027 ceph-mon[29140]: pgmap v309: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:33.893 INFO:teuthology.orchestra.run.smithi027.stdout:184683593777 2024-04-25T13:39:33.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.230+0000 7f23535cc700 1 -- 172.21.15.27:0/1452366237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c107c90 msgr2=0x7f234c1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.230+0000 7f23535cc700 1 --2- 172.21.15.27:0/1452366237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c107c90 0x7f234c1080a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f233c0099c0 tx=0x7f233c009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.230+0000 7f23535cc700 1 -- 172.21.15.27:0/1452366237 shutdown_connections 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.230+0000 7f23535cc700 1 --2- 172.21.15.27:0/1452366237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f234c103360 0x7f234c103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.230+0000 7f23535cc700 1 --2- 172.21.15.27:0/1452366237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f234c1085e0 0x7f234c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.230+0000 7f23535cc700 1 --2- 172.21.15.27:0/1452366237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c107c90 0x7f234c1080a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.230+0000 7f23535cc700 1 -- 172.21.15.27:0/1452366237 >> 172.21.15.27:0/1452366237 conn(0x7f234c0fec90 msgr2=0x7f234c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.231+0000 7f23535cc700 1 -- 172.21.15.27:0/1452366237 shutdown_connections 2024-04-25T13:39:34.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.231+0000 7f23535cc700 1 -- 172.21.15.27:0/1452366237 wait complete. 2024-04-25T13:39:34.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.231+0000 7f23535cc700 1 Processor -- start 2024-04-25T13:39:34.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.231+0000 7f23535cc700 1 -- start start 2024-04-25T13:39:34.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f23535cc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f234c103360 0x7f234c19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:34.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f23535cc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f234c107c90 0x7f234c19c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f23535cc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c1085e0 0x7f234c19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f23535cc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f234c1131f0 con 0x7f234c1085e0 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f23535cc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f234c113070 con 0x7f234c103360 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f23535cc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f234c113370 con 0x7f234c107c90 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c1085e0 0x7f234c19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c1085e0 0x7f234c19c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42408/0 (socket says 172.21.15.27:42408) 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 -- 172.21.15.27:0/2452242609 learned_addr learned my addr 172.21.15.27:0/2452242609 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351368700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f234c103360 0x7f234c19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2350b67700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f234c107c90 0x7f234c19c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 -- 172.21.15.27:0/2452242609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f234c107c90 msgr2=0x7f234c19c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f234c107c90 0x7f234c19c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 -- 172.21.15.27:0/2452242609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f234c103360 msgr2=0x7f234c19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f234c103360 0x7f234c19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 -- 172.21.15.27:0/2452242609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234c1a6d90 con 0x7f234c1085e0 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.232+0000 7f2351b69700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c1085e0 0x7f234c19c570 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2344007ee0 tx=0x7f234400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.233+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344004bb0 con 0x7f234c1085e0 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.233+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2344004d50 con 0x7f234c1085e0 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.233+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f234c1a7080 con 0x7f234c1085e0 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.233+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344005920 con 0x7f234c1085e0 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.233+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f234c1a7590 con 0x7f234c1085e0 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.234+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f234c063d60 con 0x7f234c1085e0 2024-04-25T13:39:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.235+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f234401a020 con 0x7f234c1085e0 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.236+0000 7f23427fc700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23380afc90 0x7f23380b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.236+0000 7f2351368700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23380afc90 0x7f23380b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.236+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f23440dee30 con 0x7f234c1085e0 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.237+0000 7f23427fc700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] conn(0x7f23380b97a0 0x7f23380bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.237+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 --> [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2344019e70 con 0x7f23380b97a0 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.237+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f23440df1e0 con 0x7f234c1085e0 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.237+0000 7f2351368700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23380afc90 0x7f23380b21a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f234800ac50 tx=0x7f234800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.237+0000 7f2350b67700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] conn(0x7f23380b97a0 0x7f23380bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.237+0000 7f2350b67700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] conn(0x7f23380b97a0 0x7f23380bbc90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.238+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 <== osd.1 v2:172.21.15.45:6800/4202033932 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2344019e70 con 0x7f23380b97a0 2024-04-25T13:39:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.254+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 --> [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f234c064160 con 0x7f23380b97a0 2024-04-25T13:39:34.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.255+0000 7f23427fc700 1 -- 172.21.15.27:0/2452242609 <== osd.1 v2:172.21.15.45:6800/4202033932 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f234c064160 con 0x7f23380b97a0 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.255+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 >> [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] conn(0x7f23380b97a0 msgr2=0x7f23380bbc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.255+0000 7f23535cc700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] conn(0x7f23380b97a0 0x7f23380bbc90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.255+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23380afc90 msgr2=0x7f23380b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.255+0000 7f23535cc700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23380afc90 0x7f23380b21a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f234800ac50 tx=0x7f234800a380 comp rx=0 tx=0).stop 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.255+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c1085e0 msgr2=0x7f234c19c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.255+0000 7f23535cc700 1 --2- 172.21.15.27:0/2452242609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234c1085e0 0x7f234c19c570 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2344007ee0 tx=0x7f234400d3b0 comp rx=0 tx=0).stop 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.255+0000 7f2351b69700 1 -- 172.21.15.27:0/2452242609 reap_dead start 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.256+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 shutdown_connections 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.256+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 >> 172.21.15.27:0/2452242609 conn(0x7f234c0fec90 msgr2=0x7f234c109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.256+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 shutdown_connections 2024-04-25T13:39:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:34.256+0000 7f23535cc700 1 -- 172.21.15.27:0/2452242609 wait complete. 2024-04-25T13:39:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:34.695 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:34.705 INFO:teuthology.orchestra.run.smithi027.stdout:279172874267 2024-04-25T13:39:34.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T13:39:35.438 INFO:teuthology.orchestra.run.smithi027.stdout:55834574924 2024-04-25T13:39:35.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T13:39:35.762 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:35 smithi027 ceph-mon[29140]: pgmap v310: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T13:39:35.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:35 smithi122 ceph-mon[32843]: pgmap v310: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T13:39:35.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:35 smithi045 ceph-mon[32731]: pgmap v310: 97 pgs: 1 peering, 96 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T13:39:36.109 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:36.113 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:36.113 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:37.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:37 smithi122 ceph-mon[32843]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T13:39:37.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:37 smithi045 ceph-mon[32731]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T13:39:37.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:37 smithi027 ceph-mon[29140]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T13:39:38.666 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:38.666 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:38.666 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:39.739 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:39 smithi027 ceph-mon[29140]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T13:39:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:39 smithi122 ceph-mon[32843]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T13:39:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:39 smithi045 ceph-mon[32731]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T13:39:41.625 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:41 smithi027 ceph-mon[29140]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 829 B/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2024-04-25T13:39:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:41 smithi122 ceph-mon[32843]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 829 B/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2024-04-25T13:39:41.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:41 smithi045 ceph-mon[32731]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 829 B/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2024-04-25T13:39:41.952 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:41.953 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:43.668 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:43 smithi027 ceph-mon[29140]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-04-25T13:39:43.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:43 smithi122 ceph-mon[32843]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-04-25T13:39:43.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:43 smithi045 ceph-mon[32731]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-04-25T13:39:44.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:44 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:39:44.795 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:44 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:39:44.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:44 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:39:45.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:45 smithi027 ceph-mon[29140]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T13:39:45.763 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:45 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1292227237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T13:39:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:45 smithi122 ceph-mon[32843]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T13:39:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:45 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1292227237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T13:39:45.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:45 smithi045 ceph-mon[32731]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T13:39:45.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:45 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1292227237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T13:39:46.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:46 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:46.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:46 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:46.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:46 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:46.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:46 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:46.845 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:46 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/793536835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T13:39:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.455+0000 7f2a4b513700 1 -- 172.21.15.27:0/1112604355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 msgr2=0x7f2a44105740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.455+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1112604355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 0x7f2a44105740 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2a34009990 tx=0x7f2a34009ca0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.455+0000 7f2a4b513700 1 -- 172.21.15.27:0/1112604355 shutdown_connections 2024-04-25T13:39:46.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.455+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1112604355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a4410e7a0 0x7f2a44110c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.455+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1112604355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 0x7f2a44105740 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.455+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1112604355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a44105c80 0x7f2a4410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.455+0000 7f2a4b513700 1 -- 172.21.15.27:0/1112604355 >> 172.21.15.27:0/1112604355 conn(0x7f2a440fec90 msgr2=0x7f2a441010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.455+0000 7f2a4b513700 1 -- 172.21.15.27:0/1112604355 shutdown_connections 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 -- 172.21.15.27:0/1112604355 wait complete. 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 Processor -- start 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 -- start start 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 0x7f2a441a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a44105c80 0x7f2a441a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a4410e7a0 0x7f2a4419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2a441142f0 con 0x7f2a44105330 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a44114170 con 0x7f2a44105c80 2024-04-25T13:39:46.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.456+0000 7f2a4b513700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2a44114470 con 0x7f2a4410e7a0 2024-04-25T13:39:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 0x7f2a441a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 0x7f2a441a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47156/0 (socket says 172.21.15.27:47156) 2024-04-25T13:39:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 -- 172.21.15.27:0/1292227237 learned_addr learned my addr 172.21.15.27:0/1292227237 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a48aae700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a44105c80 0x7f2a441a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a49ab0700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a4410e7a0 0x7f2a4419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 -- 172.21.15.27:0/1292227237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a4410e7a0 msgr2=0x7f2a4419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a4410e7a0 0x7f2a4419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 -- 172.21.15.27:0/1292227237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a44105c80 msgr2=0x7f2a441a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a44105c80 0x7f2a441a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 -- 172.21.15.27:0/1292227237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4419b140 con 0x7f2a44105330 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a49ab0700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a4410e7a0 0x7f2a4419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a48aae700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a44105c80 0x7f2a441a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a492af700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 0x7f2a441a0870 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2a34000c00 tx=0x7f2a34009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a3a7fc700 1 -- 172.21.15.27:0/1292227237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a34027070 con 0x7f2a44105330 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4419b3d0 con 0x7f2a44105330 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a3a7fc700 1 -- 172.21.15.27:0/1292227237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a34022470 con 0x7f2a44105330 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a3a7fc700 1 -- 172.21.15.27:0/1292227237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a34017650 con 0x7f2a44105330 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.457+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a4419b8b0 con 0x7f2a44105330 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.459+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a44051190 con 0x7f2a44105330 2024-04-25T13:39:46.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.460+0000 7f2a3a7fc700 1 -- 172.21.15.27:0/1292227237 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2a3402a070 con 0x7f2a44105330 2024-04-25T13:39:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.461+0000 7f2a3a7fc700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a300afc10 0x7f2a300b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.461+0000 7f2a48aae700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a300afc10 0x7f2a300b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.462+0000 7f2a3a7fc700 1 -- 172.21.15.27:0/1292227237 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2a340e0080 con 0x7f2a44105330 2024-04-25T13:39:46.966 INFO:teuthology.orchestra.run.smithi027.stdout:55834574926 2024-04-25T13:39:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.462+0000 7f2a48aae700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a300afc10 0x7f2a300b2120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2a44102140 tx=0x7f2a40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:46.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.465+0000 7f2a3a7fc700 1 -- 172.21.15.27:0/1292227237 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a34013070 con 0x7f2a44105330 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.687+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f2a4402e510 con 0x7f2a44105330 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.687+0000 7f2a3a7fc700 1 -- 172.21.15.27:0/1292227237 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2a340a8bc0 con 0x7f2a44105330 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.690+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a300afc10 msgr2=0x7f2a300b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.690+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a300afc10 0x7f2a300b2120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2a44102140 tx=0x7f2a40009260 comp rx=0 tx=0).stop 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.690+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 msgr2=0x7f2a441a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.690+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 0x7f2a441a0870 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2a34000c00 tx=0x7f2a34009580 comp rx=0 tx=0).stop 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.691+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 shutdown_connections 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.691+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a300afc10 0x7f2a300b2120 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.691+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a4410e7a0 0x7f2a4419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.691+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a44105330 0x7f2a441a0870 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.691+0000 7f2a4b513700 1 --2- 172.21.15.27:0/1292227237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a44105c80 0x7f2a441a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.691+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 >> 172.21.15.27:0/1292227237 conn(0x7f2a440fec90 msgr2=0x7f2a440ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.691+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 shutdown_connections 2024-04-25T13:39:46.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:44.691+0000 7f2a4b513700 1 -- 172.21.15.27:0/1292227237 wait complete. 2024-04-25T13:39:47.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:46 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:47.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:46 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:47.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:46 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:47.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:46 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:47.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:46 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/793536835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T13:39:47.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:46 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:47.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:46 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:47.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:46 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:47.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:46 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:39:47.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:46 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/793536835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T13:39:47.616 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574924 got 55834574926 for osd.1 2024-04-25T13:39:47.616 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:47.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:47 smithi027 ceph-mon[29140]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T13:39:48.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:47 smithi122 ceph-mon[32843]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T13:39:48.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:47 smithi045 ceph-mon[32731]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T13:39:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 -- 172.21.15.27:0/2194893898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810103360 msgr2=0x7f3810103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 --2- 172.21.15.27:0/2194893898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810103360 0x7f3810103770 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f380c0099c0 tx=0x7f380c009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 -- 172.21.15.27:0/2194893898 shutdown_connections 2024-04-25T13:39:49.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 --2- 172.21.15.27:0/2194893898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38101085e0 0x7f381010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 --2- 172.21.15.27:0/2194893898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810103360 0x7f3810103770 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 --2- 172.21.15.27:0/2194893898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3810107c90 0x7f38101080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 -- 172.21.15.27:0/2194893898 >> 172.21.15.27:0/2194893898 conn(0x7f38100fecb0 msgr2=0x7f38101010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:49.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 -- 172.21.15.27:0/2194893898 shutdown_connections 2024-04-25T13:39:49.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.268+0000 7f3818286700 1 -- 172.21.15.27:0/2194893898 wait complete. 2024-04-25T13:39:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.269+0000 7f3818286700 1 Processor -- start 2024-04-25T13:39:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.269+0000 7f3818286700 1 -- start start 2024-04-25T13:39:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.269+0000 7f3818286700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3810103360 0x7f38101a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.269+0000 7f3818286700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810107c90 0x7f38101a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:49.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.269+0000 7f3818286700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38101085e0 0x7f381019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.269+0000 7f3818286700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3810113210 con 0x7f3810107c90 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.269+0000 7f3818286700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3810113090 con 0x7f38101085e0 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.269+0000 7f3818286700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3810113390 con 0x7f3810103360 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3810103360 0x7f38101a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3810103360 0x7f38101a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:45856/0 (socket says 172.21.15.27:45856) 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816823700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38101085e0 0x7f381019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 -- 172.21.15.27:0/915148393 learned_addr learned my addr 172.21.15.27:0/915148393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3815821700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810107c90 0x7f38101a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 -- 172.21.15.27:0/915148393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38101085e0 msgr2=0x7f381019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38101085e0 0x7f381019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 -- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810107c90 msgr2=0x7f38101a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810107c90 0x7f38101a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 -- 172.21.15.27:0/915148393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381019b110 con 0x7f3810103360 2024-04-25T13:39:49.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3815821700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810107c90 0x7f38101a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.270+0000 7f3816022700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3810103360 0x7f38101a0840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f380c006010 tx=0x7f380c0055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.271+0000 7f38077fe700 1 -- 172.21.15.27:0/915148393 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c026070 con 0x7f3810103360 2024-04-25T13:39:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.271+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381019b3a0 con 0x7f3810103360 2024-04-25T13:39:49.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.271+0000 7f38077fe700 1 -- 172.21.15.27:0/915148393 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f380c00f460 con 0x7f3810103360 2024-04-25T13:39:49.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.271+0000 7f38077fe700 1 -- 172.21.15.27:0/915148393 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c009e60 con 0x7f3810103360 2024-04-25T13:39:49.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.271+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38101a76d0 con 0x7f3810103360 2024-04-25T13:39:49.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.272+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3810104810 con 0x7f3810103360 2024-04-25T13:39:49.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.273+0000 7f38077fe700 1 -- 172.21.15.27:0/915148393 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f380c029070 con 0x7f3810103360 2024-04-25T13:39:49.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.274+0000 7f38077fe700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37fc0afc90 0x7f37fc0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:49.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.274+0000 7f3815821700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37fc0afc90 0x7f37fc0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:49.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.274+0000 7f38077fe700 1 -- 172.21.15.27:0/915148393 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f380c0df950 con 0x7f3810103360 2024-04-25T13:39:49.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.275+0000 7f3815821700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37fc0afc90 0x7f37fc0b21a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f38101064b0 tx=0x7f3800009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:49.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.278+0000 7f38077fe700 1 -- 172.21.15.27:0/915148393 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f380c030080 con 0x7f3810103360 2024-04-25T13:39:49.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.492+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f381019bf90 con 0x7f3810103360 2024-04-25T13:39:49.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.493+0000 7f38077fe700 1 -- 172.21.15.27:0/915148393 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f380c013070 con 0x7f3810103360 2024-04-25T13:39:49.493 INFO:teuthology.orchestra.run.smithi027.stdout:257698037797 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37fc0afc90 msgr2=0x7f37fc0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37fc0afc90 0x7f37fc0b21a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f38101064b0 tx=0x7f3800009260 comp rx=0 tx=0).stop 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3810103360 msgr2=0x7f38101a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3810103360 0x7f38101a0840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f380c006010 tx=0x7f380c0055f0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 shutdown_connections 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37fc0afc90 0x7f37fc0b21a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3810103360 0x7f38101a0840 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38101085e0 0x7f381019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 --2- 172.21.15.27:0/915148393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3810107c90 0x7f38101a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:49.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 >> 172.21.15.27:0/915148393 conn(0x7f38100fecb0 msgr2=0x7f3810075cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:49.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 shutdown_connections 2024-04-25T13:39:49.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:49.495+0000 7f3818286700 1 -- 172.21.15.27:0/915148393 wait complete. 2024-04-25T13:39:49.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:49 smithi027 ceph-mon[29140]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:49.847 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:49 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/915148393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T13:39:50.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:49 smithi122 ceph-mon[32843]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:50.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:49 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/915148393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T13:39:50.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:49 smithi045 ceph-mon[32731]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:50.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:49 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/915148393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T13:39:50.438 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037790 got 257698037797 for osd.9 2024-04-25T13:39:50.438 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 -- 172.21.15.27:0/1854330086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103360 msgr2=0x7f0b20103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 --2- 172.21.15.27:0/1854330086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103360 0x7f0b20103770 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0b100099c0 tx=0x7f0b10009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 -- 172.21.15.27:0/1854330086 shutdown_connections 2024-04-25T13:39:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 --2- 172.21.15.27:0/1854330086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b201085e0 0x7f0b2010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 --2- 172.21.15.27:0/1854330086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b20103360 0x7f0b20103770 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 --2- 172.21.15.27:0/1854330086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b20107c90 0x7f0b201080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 -- 172.21.15.27:0/1854330086 >> 172.21.15.27:0/1854330086 conn(0x7f0b200fec90 msgr2=0x7f0b201010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:50.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 -- 172.21.15.27:0/1854330086 shutdown_connections 2024-04-25T13:39:50.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 -- 172.21.15.27:0/1854330086 wait complete. 2024-04-25T13:39:50.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.635+0000 7f0b28432700 1 Processor -- start 2024-04-25T13:39:50.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b28432700 1 -- start start 2024-04-25T13:39:50.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b28432700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b20103360 0x7f0b2019ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b28432700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b20107c90 0x7f0b2019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b28432700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b201085e0 0x7f0b2019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b28432700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0b20113210 con 0x7f0b201085e0 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b28432700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0b20113090 con 0x7f0b20103360 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b28432700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0b20113390 con 0x7f0b20107c90 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b269cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b201085e0 0x7f0b2019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b269cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b201085e0 0x7f0b2019f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47212/0 (socket says 172.21.15.27:47212) 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b269cf700 1 -- 172.21.15.27:0/1142290481 learned_addr learned my addr 172.21.15.27:0/1142290481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b259cd700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b20107c90 0x7f0b2019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b261ce700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b20103360 0x7f0b2019ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b269cf700 1 -- 172.21.15.27:0/1142290481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b20107c90 msgr2=0x7f0b2019b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b269cf700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b20107c90 0x7f0b2019b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b269cf700 1 -- 172.21.15.27:0/1142290481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b20103360 msgr2=0x7f0b2019ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b269cf700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b20103360 0x7f0b2019ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.636+0000 7f0b269cf700 1 -- 172.21.15.27:0/1142290481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2019fb50 con 0x7f0b201085e0 2024-04-25T13:39:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.637+0000 7f0b269cf700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b201085e0 0x7f0b2019f560 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0b18007d70 tx=0x7f0b1800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.637+0000 7f0b177fe700 1 -- 172.21.15.27:0/1142290481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18016020 con 0x7f0b201085e0 2024-04-25T13:39:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.637+0000 7f0b177fe700 1 -- 172.21.15.27:0/1142290481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b1800ad40 con 0x7f0b201085e0 2024-04-25T13:39:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.637+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b2019fe40 con 0x7f0b201085e0 2024-04-25T13:39:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.637+0000 7f0b177fe700 1 -- 172.21.15.27:0/1142290481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b180115e0 con 0x7f0b201085e0 2024-04-25T13:39:50.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.637+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b201a0380 con 0x7f0b201085e0 2024-04-25T13:39:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.638+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b20051190 con 0x7f0b201085e0 2024-04-25T13:39:50.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.640+0000 7f0b177fe700 1 -- 172.21.15.27:0/1142290481 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f0b18019020 con 0x7f0b201085e0 2024-04-25T13:39:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.641+0000 7f0b177fe700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0b0c0afc90 0x7f0b0c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.641+0000 7f0b261ce700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0b0c0afc90 0x7f0b0c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.641+0000 7f0b177fe700 1 -- 172.21.15.27:0/1142290481 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0b180df0e0 con 0x7f0b201085e0 2024-04-25T13:39:50.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.641+0000 7f0b261ce700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0b0c0afc90 0x7f0b0c0b21a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b10003800 tx=0x7f0b1001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:50.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.645+0000 7f0b177fe700 1 -- 172.21.15.27:0/1142290481 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b18020020 con 0x7f0b201085e0 2024-04-25T13:39:50.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.854+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f0b2019c090 con 0x7f0b201085e0 2024-04-25T13:39:50.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.855+0000 7f0b177fe700 1 -- 172.21.15.27:0/1142290481 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0b180a7ca0 con 0x7f0b201085e0 2024-04-25T13:39:50.855 INFO:teuthology.orchestra.run.smithi027.stdout:77309411400 2024-04-25T13:39:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0b0c0afc90 msgr2=0x7f0b0c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0b0c0afc90 0x7f0b0c0b21a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b10003800 tx=0x7f0b1001a040 comp rx=0 tx=0).stop 2024-04-25T13:39:50.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b201085e0 msgr2=0x7f0b2019f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b201085e0 0x7f0b2019f560 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0b18007d70 tx=0x7f0b1800a340 comp rx=0 tx=0).stop 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 shutdown_connections 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0b0c0afc90 0x7f0b0c0b21a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b201085e0 0x7f0b2019f560 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b20103360 0x7f0b2019ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 --2- 172.21.15.27:0/1142290481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b20107c90 0x7f0b2019b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 >> 172.21.15.27:0/1142290481 conn(0x7f0b200fec90 msgr2=0x7f0b20106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 shutdown_connections 2024-04-25T13:39:50.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:50.857+0000 7f0b28432700 1 -- 172.21.15.27:0/1142290481 wait complete. 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 -- 172.21.15.27:0/2203230344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c1091a0 msgr2=0x7fda3c1095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 --2- 172.21.15.27:0/2203230344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c1091a0 0x7fda3c1095b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009990 tx=0x7fda2c009ca0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 -- 172.21.15.27:0/2203230344 shutdown_connections 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 --2- 172.21.15.27:0/2203230344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c102f80 0x7fda3c111010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 --2- 172.21.15.27:0/2203230344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c1091a0 0x7fda3c1095b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 --2- 172.21.15.27:0/2203230344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda3c102630 0x7fda3c102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 -- 172.21.15.27:0/2203230344 >> 172.21.15.27:0/2203230344 conn(0x7fda3c0fe3f0 msgr2=0x7fda3c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 -- 172.21.15.27:0/2203230344 shutdown_connections 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.012+0000 7fda4273d700 1 -- 172.21.15.27:0/2203230344 wait complete. 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.013+0000 7fda4273d700 1 Processor -- start 2024-04-25T13:39:50.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.013+0000 7fda4273d700 1 -- start start 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.013+0000 7fda4273d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda3c102630 0x7fda3c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.013+0000 7fda4273d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c102f80 0x7fda3c19b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.013+0000 7fda4273d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c1091a0 0x7fda3c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.013+0000 7fda4273d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c114440 con 0x7fda3c102f80 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda4273d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1142c0 con 0x7fda3c1091a0 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda4273d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1145c0 con 0x7fda3c102630 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda3c102630 0x7fda3c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda3c102630 0x7fda3c19ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:45834/0 (socket says 172.21.15.27:45834) 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c102f80 0x7fda3c19b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 -- 172.21.15.27:0/793536835 learned_addr learned my addr 172.21.15.27:0/793536835 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda40cda700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c1091a0 0x7fda3c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 -- 172.21.15.27:0/793536835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c1091a0 msgr2=0x7fda3c19f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c1091a0 0x7fda3c19f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 -- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c102f80 msgr2=0x7fda3c19b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c102f80 0x7fda3c19b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 -- 172.21.15.27:0/793536835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c19fce0 con 0x7fda3c102630 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3b7fe700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c102f80 0x7fda3c19b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda40cda700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c1091a0 0x7fda3c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda3bfff700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda3c102630 0x7fda3c19ae10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fda2c006010 tx=0x7fda2c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda397fa700 1 -- 172.21.15.27:0/793536835 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c027070 con 0x7fda3c102630 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3c19ff70 con 0x7fda3c102630 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda397fa700 1 -- 172.21.15.27:0/793536835 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda2c022470 con 0x7fda3c102630 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.014+0000 7fda397fa700 1 -- 172.21.15.27:0/793536835 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c017650 con 0x7fda3c102630 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.015+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda3c1a78f0 con 0x7fda3c102630 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.015+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda3c10c740 con 0x7fda3c102630 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.017+0000 7fda397fa700 1 -- 172.21.15.27:0/793536835 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fda2c02a070 con 0x7fda3c102630 2024-04-25T13:39:50.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.018+0000 7fda397fa700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fda240afc90 0x7fda240b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:50.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.018+0000 7fda3b7fe700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fda240afc90 0x7fda240b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:50.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.018+0000 7fda397fa700 1 -- 172.21.15.27:0/793536835 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fda2c0dfe50 con 0x7fda3c102630 2024-04-25T13:39:50.974 INFO:teuthology.orchestra.run.smithi027.stdout:304942678042 2024-04-25T13:39:50.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.018+0000 7fda3b7fe700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fda240afc90 0x7fda240b21a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda3c101ee0 tx=0x7fda30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:50.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.021+0000 7fda397fa700 1 -- 172.21.15.27:0/793536835 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda2c013070 con 0x7fda3c102630 2024-04-25T13:39:50.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.232+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fda3c051190 con 0x7fda3c102630 2024-04-25T13:39:50.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.232+0000 7fda397fa700 1 -- 172.21.15.27:0/793536835 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fda2c0a8930 con 0x7fda3c102630 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.234+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fda240afc90 msgr2=0x7fda240b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fda240afc90 0x7fda240b21a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda3c101ee0 tx=0x7fda30009260 comp rx=0 tx=0).stop 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda3c102630 msgr2=0x7fda3c19ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda3c102630 0x7fda3c19ae10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fda2c006010 tx=0x7fda2c0047b0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 shutdown_connections 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fda240afc90 0x7fda240b21a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda3c102630 0x7fda3c19ae10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda3c1091a0 0x7fda3c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 --2- 172.21.15.27:0/793536835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c102f80 0x7fda3c19b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 >> 172.21.15.27:0/793536835 conn(0x7fda3c0fe3f0 msgr2=0x7fda3c10a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 shutdown_connections 2024-04-25T13:39:50.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:46.235+0000 7fda4273d700 1 -- 172.21.15.27:0/793536835 wait complete. 2024-04-25T13:39:51.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:51 smithi027 ceph-mon[29140]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:51.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:51 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1142290481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T13:39:52.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:51 smithi122 ceph-mon[32843]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:52.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:51 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1142290481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T13:39:52.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:51 smithi045 ceph-mon[32731]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:52.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:51 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1142290481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T13:39:52.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 -- 172.21.15.27:0/3135423997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8103390 msgr2=0x7f07c81037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 --2- 172.21.15.27:0/3135423997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8103390 0x7f07c81037a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f07c40099c0 tx=0x7f07c4009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 -- 172.21.15.27:0/3135423997 shutdown_connections 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 --2- 172.21.15.27:0/3135423997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c8108610 0x7f07c810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 --2- 172.21.15.27:0/3135423997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8103390 0x7f07c81037a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 --2- 172.21.15.27:0/3135423997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07c8107cc0 0x7f07c81080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 -- 172.21.15.27:0/3135423997 >> 172.21.15.27:0/3135423997 conn(0x7f07c80fec90 msgr2=0x7f07c81010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 -- 172.21.15.27:0/3135423997 shutdown_connections 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 -- 172.21.15.27:0/3135423997 wait complete. 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 Processor -- start 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.980+0000 7f07d0186700 1 -- start start 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07d0186700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c8103390 0x7f07c81a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:52.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07d0186700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8107cc0 0x7f07c81a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07d0186700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07c8108610 0x7f07c819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07d0186700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f07c8113240 con 0x7f07c8107cc0 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07d0186700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f07c81130c0 con 0x7f07c8103390 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07d0186700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f07c81133c0 con 0x7f07c8108610 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cd721700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8107cc0 0x7f07c81a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cd721700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8107cc0 0x7f07c81a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47226/0 (socket says 172.21.15.27:47226) 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cd721700 1 -- 172.21.15.27:0/438962694 learned_addr learned my addr 172.21.15.27:0/438962694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cdf22700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c8103390 0x7f07c81a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07ce723700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07c8108610 0x7f07c819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cd721700 1 -- 172.21.15.27:0/438962694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07c8108610 msgr2=0x7f07c819ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cd721700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07c8108610 0x7f07c819ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cd721700 1 -- 172.21.15.27:0/438962694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c8103390 msgr2=0x7f07c81a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:52.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cd721700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c8103390 0x7f07c81a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.981+0000 7f07cd721700 1 -- 172.21.15.27:0/438962694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c819b260 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.982+0000 7f07cd721700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8107cc0 0x7f07c81a0ee0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c8f0 tx=0x7f07b800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.982+0000 7f07beffd700 1 -- 172.21.15.27:0/438962694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8007940 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.982+0000 7f07beffd700 1 -- 172.21.15.27:0/438962694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b800f450 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.982+0000 7f07beffd700 1 -- 172.21.15.27:0/438962694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8018660 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.982+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c819b550 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.982+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c81a7750 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.983+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07c8104840 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.984+0000 7f07beffd700 1 -- 172.21.15.27:0/438962694 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f07b800f890 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.985+0000 7f07beffd700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07b40afc90 0x7f07b40b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.985+0000 7f07cdf22700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07b40afc90 0x7f07b40b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.985+0000 7f07beffd700 1 -- 172.21.15.27:0/438962694 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f07b80dec80 con 0x7f07c8107cc0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.986+0000 7f07cdf22700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07b40afc90 0x7f07b40b21a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009990 tx=0x7f07c4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:52.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:51.989+0000 7f07beffd700 1 -- 172.21.15.27:0/438962694 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07b8014030 con 0x7f07c8107cc0 2024-04-25T13:39:52.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.198+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f07c802e510 con 0x7f07c8107cc0 2024-04-25T13:39:52.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.199+0000 7f07beffd700 1 -- 172.21.15.27:0/438962694 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f07b80a7840 con 0x7f07c8107cc0 2024-04-25T13:39:52.199 INFO:teuthology.orchestra.run.smithi027.stdout:231928234027 2024-04-25T13:39:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07b40afc90 msgr2=0x7f07b40b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07b40afc90 0x7f07b40b21a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009990 tx=0x7f07c4000bc0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8107cc0 msgr2=0x7f07c81a0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8107cc0 0x7f07c81a0ee0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c8f0 tx=0x7f07b800cc00 comp rx=0 tx=0).stop 2024-04-25T13:39:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 shutdown_connections 2024-04-25T13:39:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07b40afc90 0x7f07b40b21a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07c8108610 0x7f07c819ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07c8103390 0x7f07c81a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 --2- 172.21.15.27:0/438962694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8107cc0 0x7f07c81a0ee0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 >> 172.21.15.27:0/438962694 conn(0x7f07c80fec90 msgr2=0x7f07c8106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:52.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 shutdown_connections 2024-04-25T13:39:52.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.201+0000 7f07d0186700 1 -- 172.21.15.27:0/438962694 wait complete. 2024-04-25T13:39:52.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678038 got 304942678042 for osd.11 2024-04-25T13:39:52.717 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:52.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 -- 172.21.15.27:0/720698610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 msgr2=0x7f5954111090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:52.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 --2- 172.21.15.27:0/720698610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 0x7f5954111090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f594400b0d0 tx=0x7f594400b3e0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 -- 172.21.15.27:0/720698610 shutdown_connections 2024-04-25T13:39:52.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 --2- 172.21.15.27:0/720698610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 0x7f5954111090 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 --2- 172.21.15.27:0/720698610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59541044b0 0x7f59541048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 --2- 172.21.15.27:0/720698610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5954108de0 0x7f59541091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 -- 172.21.15.27:0/720698610 >> 172.21.15.27:0/720698610 conn(0x7f59540ffd40 msgr2=0x7f5954102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:52.908 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/438962694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 -- 172.21.15.27:0/720698610 shutdown_connections 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.907+0000 7f595c133700 1 -- 172.21.15.27:0/720698610 wait complete. 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.908+0000 7f595c133700 1 Processor -- start 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.908+0000 7f595c133700 1 -- start start 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595c133700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59541044b0 0x7f5954075990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595c133700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5954108de0 0x7f5954075ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595c133700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 0x7f5954076410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595c133700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5954114360 con 0x7f5954109730 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595c133700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f59541141e0 con 0x7f5954108de0 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595c133700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f59541144e0 con 0x7f59541044b0 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f59596ce700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5954108de0 0x7f5954075ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595a6d0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 0x7f5954076410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f5959ecf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59541044b0 0x7f5954075990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f59596ce700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5954108de0 0x7f5954075ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:36444/0 (socket says 172.21.15.27:36444) 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f59596ce700 1 -- 172.21.15.27:0/1318801281 learned_addr learned my addr 172.21.15.27:0/1318801281 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f5959ecf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59541044b0 0x7f5954075990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:45900/0 (socket says 172.21.15.27:45900) 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595a6d0700 1 -- 172.21.15.27:0/1318801281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59541044b0 msgr2=0x7f5954075990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595a6d0700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59541044b0 0x7f5954075990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595a6d0700 1 -- 172.21.15.27:0/1318801281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5954108de0 msgr2=0x7f5954075ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595a6d0700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5954108de0 0x7f5954075ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595a6d0700 1 -- 172.21.15.27:0/1318801281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59541a9090 con 0x7f5954109730 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f59596ce700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5954108de0 0x7f5954075ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.909+0000 7f595a6d0700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 0x7f5954076410 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5944009fd0 tx=0x7f5944007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.910+0000 7f5952ffd700 1 -- 172.21.15.27:0/1318801281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5944021020 con 0x7f5954109730 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.910+0000 7f595c133700 1 -- 172.21.15.27:0/1318801281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59541a9320 con 0x7f5954109730 2024-04-25T13:39:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.910+0000 7f5952ffd700 1 -- 172.21.15.27:0/1318801281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5944007e40 con 0x7f5954109730 2024-04-25T13:39:52.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.910+0000 7f5952ffd700 1 -- 172.21.15.27:0/1318801281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5944005050 con 0x7f5954109730 2024-04-25T13:39:52.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.910+0000 7f595c133700 1 -- 172.21.15.27:0/1318801281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59541a97d0 con 0x7f5954109730 2024-04-25T13:39:52.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.911+0000 7f5950ff9700 1 -- 172.21.15.27:0/1318801281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5954051190 con 0x7f5954109730 2024-04-25T13:39:52.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.912+0000 7f5952ffd700 1 -- 172.21.15.27:0/1318801281 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5944024020 con 0x7f5954109730 2024-04-25T13:39:52.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.913+0000 7f5952ffd700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f593c0afc10 0x7f593c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:52.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.913+0000 7f5959ecf700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f593c0afc10 0x7f593c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:52.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.913+0000 7f5952ffd700 1 -- 172.21.15.27:0/1318801281 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f594400f020 con 0x7f5954109730 2024-04-25T13:39:52.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.914+0000 7f5959ecf700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f593c0afc10 0x7f593c0b2120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f594c005950 tx=0x7f594c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:52.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:52.917+0000 7f5952ffd700 1 -- 172.21.15.27:0/1318801281 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59440a8d70 con 0x7f5954109730 2024-04-25T13:39:53.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/438962694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T13:39:53.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/438962694' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T13:39:53.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.129+0000 7f5950ff9700 1 -- 172.21.15.27:0/1318801281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f59540642a0 con 0x7f5954109730 2024-04-25T13:39:53.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.129+0000 7f5952ffd700 1 -- 172.21.15.27:0/1318801281 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f59440a8670 con 0x7f5954109730 2024-04-25T13:39:53.129 INFO:teuthology.orchestra.run.smithi027.stdout:137438953536 2024-04-25T13:39:53.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 -- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f593c0afc10 msgr2=0x7f593c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:53.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f593c0afc10 0x7f593c0b2120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f594c005950 tx=0x7f594c00b410 comp rx=0 tx=0).stop 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 -- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 msgr2=0x7f5954076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 0x7f5954076410 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5944009fd0 tx=0x7f5944007a00 comp rx=0 tx=0).stop 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 -- 172.21.15.27:0/1318801281 shutdown_connections 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f593c0afc10 0x7f593c0b2120 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59541044b0 0x7f5954075990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5954109730 0x7f5954076410 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 --2- 172.21.15.27:0/1318801281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5954108de0 0x7f5954075ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 -- 172.21.15.27:0/1318801281 >> 172.21.15.27:0/1318801281 conn(0x7f59540ffd40 msgr2=0x7f5954106e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 -- 172.21.15.27:0/1318801281 shutdown_connections 2024-04-25T13:39:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:53.131+0000 7f5950ff9700 1 -- 172.21.15.27:0/1318801281 wait complete. 2024-04-25T13:39:53.787 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:53 smithi027 ceph-mon[29140]: pgmap v319: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:53.787 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:53 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1318801281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T13:39:53.818 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411392 got 77309411400 for osd.2 2024-04-25T13:39:53.818 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:54.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:53 smithi122 ceph-mon[32843]: pgmap v319: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:54.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:53 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1318801281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T13:39:54.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:53 smithi045 ceph-mon[32731]: pgmap v319: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:54.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:53 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1318801281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T13:39:55.769 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953531 got 137438953536 for osd.4 2024-04-25T13:39:55.770 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:55.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:55 smithi027 ceph-mon[29140]: pgmap v320: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:56.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:55 smithi122 ceph-mon[32843]: pgmap v320: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:56.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:55 smithi045 ceph-mon[32731]: pgmap v320: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:39:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 -- 172.21.15.27:0/3586662431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c1085d0 msgr2=0x7f2b8c10ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 --2- 172.21.15.27:0/3586662431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c1085d0 0x7f2b8c10ff30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0099c0 tx=0x7f2b7c009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 -- 172.21.15.27:0/3586662431 shutdown_connections 2024-04-25T13:39:56.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 --2- 172.21.15.27:0/3586662431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c1085d0 0x7f2b8c10ff30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 --2- 172.21.15.27:0/3586662431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b8c103350 0x7f2b8c103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 --2- 172.21.15.27:0/3586662431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b8c107c80 0x7f2b8c108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 -- 172.21.15.27:0/3586662431 >> 172.21.15.27:0/3586662431 conn(0x7f2b8c0fec30 msgr2=0x7f2b8c101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:56.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 -- 172.21.15.27:0/3586662431 shutdown_connections 2024-04-25T13:39:56.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.307+0000 7f2b92d91700 1 -- 172.21.15.27:0/3586662431 wait complete. 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b92d91700 1 Processor -- start 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b92d91700 1 -- start start 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b92d91700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b8c103350 0x7f2b8c19bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b92d91700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c107c80 0x7f2b8c19c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b92d91700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b8c1085d0 0x7f2b8c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b92d91700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c113200 con 0x7f2b8c107c80 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b92d91700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c113080 con 0x7f2b8c103350 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b92d91700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c113380 con 0x7f2b8c1085d0 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b8bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c107c80 0x7f2b8c19c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.308+0000 7f2b8bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c107c80 0x7f2b8c19c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41240/0 (socket says 172.21.15.27:41240) 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b8bfff700 1 -- 172.21.15.27:0/2219307086 learned_addr learned my addr 172.21.15.27:0/2219307086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b90b2d700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b8c103350 0x7f2b8c19bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:56.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b9132e700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b8c1085d0 0x7f2b8c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b8bfff700 1 -- 172.21.15.27:0/2219307086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b8c1085d0 msgr2=0x7f2b8c19c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b8bfff700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b8c1085d0 0x7f2b8c19c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b8bfff700 1 -- 172.21.15.27:0/2219307086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b8c103350 msgr2=0x7f2b8c19bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b8bfff700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b8c103350 0x7f2b8c19bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b8bfff700 1 -- 172.21.15.27:0/2219307086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8c1a6e70 con 0x7f2b8c107c80 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b8bfff700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c107c80 0x7f2b8c19c040 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e970 tx=0x7f2b8000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b89ffb700 1 -- 172.21.15.27:0/2219307086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8000cbd0 con 0x7f2b8c107c80 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b89ffb700 1 -- 172.21.15.27:0/2219307086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b8000cd70 con 0x7f2b8c107c80 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8c199950 con 0x7f2b8c107c80 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.309+0000 7f2b89ffb700 1 -- 172.21.15.27:0/2219307086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80005680 con 0x7f2b8c107c80 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.310+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8c199e90 con 0x7f2b8c107c80 2024-04-25T13:39:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.311+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8c051190 con 0x7f2b8c107c80 2024-04-25T13:39:56.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.312+0000 7f2b89ffb700 1 -- 172.21.15.27:0/2219307086 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2b80004750 con 0x7f2b8c107c80 2024-04-25T13:39:56.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.313+0000 7f2b89ffb700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b740afc10 0x7f2b740b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:56.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.313+0000 7f2b90b2d700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b740afc10 0x7f2b740b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:56.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.313+0000 7f2b89ffb700 1 -- 172.21.15.27:0/2219307086 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2b800df140 con 0x7f2b8c107c80 2024-04-25T13:39:56.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.313+0000 7f2b90b2d700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b740afc10 0x7f2b740b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2b84009550 tx=0x7f2b840091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.317+0000 7f2b89ffb700 1 -- 172.21.15.27:0/2219307086 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b80053ff0 con 0x7f2b8c107c80 2024-04-25T13:39:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 -- 172.21.15.27:0/3974965596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8108e00 msgr2=0x7fa4e8109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/3974965596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8108e00 0x7fa4e8109210 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0099c0 tx=0x7fa4dc009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 -- 172.21.15.27:0/3974965596 shutdown_connections 2024-04-25T13:39:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/3974965596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e8109750 0x7fa4e81110b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/3974965596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e8104520 0x7fa4e8104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/3974965596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8108e00 0x7fa4e8109210 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 -- 172.21.15.27:0/3974965596 >> 172.21.15.27:0/3974965596 conn(0x7fa4e8079dd0 msgr2=0x7fa4e807a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 -- 172.21.15.27:0/3974965596 shutdown_connections 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.976+0000 7fa4ee04e700 1 -- 172.21.15.27:0/3974965596 wait complete. 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.977+0000 7fa4ee04e700 1 Processor -- start 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.977+0000 7fa4ee04e700 1 -- start start 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.977+0000 7fa4ee04e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e8104520 0x7fa4e8075990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.977+0000 7fa4ee04e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e8108e00 0x7fa4e8075ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.977+0000 7fa4ee04e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8109750 0x7fa4e8076410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.977+0000 7fa4ee04e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e81143b0 con 0x7fa4e8109750 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.977+0000 7fa4ee04e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e8114230 con 0x7fa4e8108e00 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.977+0000 7fa4ee04e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e8114530 con 0x7fa4e8104520 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8109750 0x7fa4e8076410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8109750 0x7fa4e8076410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41216/0 (socket says 172.21.15.27:41216) 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 -- 172.21.15.27:0/1857704203 learned_addr learned my addr 172.21.15.27:0/1857704203 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e6ffd700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e8108e00 0x7fa4e8075ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e77fe700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e8104520 0x7fa4e8075990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 -- 172.21.15.27:0/1857704203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e8104520 msgr2=0x7fa4e8075990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e8104520 0x7fa4e8075990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 -- 172.21.15.27:0/1857704203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e8108e00 msgr2=0x7fa4e8075ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e8108e00 0x7fa4e8075ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 -- 172.21.15.27:0/1857704203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e81a90c0 con 0x7fa4e8109750 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e7fff700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8109750 0x7fa4e8076410 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0007ee0 tx=0x7fa4e000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e4ff9700 1 -- 172.21.15.27:0/1857704203 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e000dbd0 con 0x7fa4e8109750 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e81a93b0 con 0x7fa4e8109750 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e4ff9700 1 -- 172.21.15.27:0/1857704203 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4e000dd70 con 0x7fa4e8109750 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4e4ff9700 1 -- 172.21.15.27:0/1857704203 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0005990 con 0x7fa4e8109750 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.978+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e81a98c0 con 0x7fa4e8109750 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.980+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e8105980 con 0x7fa4e8109750 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.981+0000 7fa4e4ff9700 1 -- 172.21.15.27:0/1857704203 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa4e001a020 con 0x7fa4e8109750 2024-04-25T13:39:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.981+0000 7fa4e4ff9700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4d00afc10 0x7fa4d00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.981+0000 7fa4e77fe700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4d00afc10 0x7fa4d00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.982+0000 7fa4e4ff9700 1 -- 172.21.15.27:0/1857704203 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa4e00de540 con 0x7fa4e8109750 2024-04-25T13:39:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.982+0000 7fa4e77fe700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4d00afc10 0x7fa4d00b2120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005fd0 tx=0x7fa4d8005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:55.986+0000 7fa4e4ff9700 1 -- 172.21.15.27:0/1857704203 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa4e0010070 con 0x7fa4e8109750 2024-04-25T13:39:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.203+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa4e8076ec0 con 0x7fa4e8109750 2024-04-25T13:39:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.203+0000 7fa4e4ff9700 1 -- 172.21.15.27:0/1857704203 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa4e00a6ff0 con 0x7fa4e8109750 2024-04-25T13:39:56.496 INFO:teuthology.orchestra.run.smithi027.stdout:184683593782 2024-04-25T13:39:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.205+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4d00afc10 msgr2=0x7fa4d00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.205+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4d00afc10 0x7fa4d00b2120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005fd0 tx=0x7fa4d8005f60 comp rx=0 tx=0).stop 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.205+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8109750 msgr2=0x7fa4e8076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.205+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8109750 0x7fa4e8076410 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0007ee0 tx=0x7fa4e000d3b0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.206+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 shutdown_connections 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.206+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa4d00afc10 0x7fa4d00b2120 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.206+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4e8104520 0x7fa4e8075990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.206+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e8109750 0x7fa4e8076410 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.206+0000 7fa4ee04e700 1 --2- 172.21.15.27:0/1857704203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e8108e00 0x7fa4e8075ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.206+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 >> 172.21.15.27:0/1857704203 conn(0x7fa4e8079dd0 msgr2=0x7fa4e8101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.206+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 shutdown_connections 2024-04-25T13:39:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.206+0000 7fa4ee04e700 1 -- 172.21.15.27:0/1857704203 wait complete. 2024-04-25T13:39:56.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.534+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f2b8c02e510 con 0x7f2b8c107c80 2024-04-25T13:39:56.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.534+0000 7f2b89ffb700 1 -- 172.21.15.27:0/2219307086 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2b800a7bf0 con 0x7f2b8c107c80 2024-04-25T13:39:56.535 INFO:teuthology.orchestra.run.smithi027.stdout:158913790011 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.536+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b740afc10 msgr2=0x7f2b740b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b740afc10 0x7f2b740b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2b84009550 tx=0x7f2b840091b0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c107c80 msgr2=0x7f2b8c19c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c107c80 0x7f2b8c19c040 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e970 tx=0x7f2b8000ec80 comp rx=0 tx=0).stop 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 shutdown_connections 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b740afc10 0x7f2b740b2120 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b8c1085d0 0x7f2b8c19c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b8c103350 0x7f2b8c19bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 --2- 172.21.15.27:0/2219307086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b8c107c80 0x7f2b8c19c040 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 >> 172.21.15.27:0/2219307086 conn(0x7f2b8c0fec30 msgr2=0x7f2b8c1073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 shutdown_connections 2024-04-25T13:39:56.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.537+0000 7f2b92d91700 1 -- 172.21.15.27:0/2219307086 wait complete. 2024-04-25T13:39:56.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:56 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1857704203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T13:39:56.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:56 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2219307086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T13:39:57.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:56 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1857704203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T13:39:57.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:56 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2219307086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T13:39:57.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:56 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1857704203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T13:39:57.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:56 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2219307086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T13:39:57.118 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:39:57.418 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593777 got 184683593782 for osd.6 2024-04-25T13:39:57.419 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:57.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234023 got 231928234027 for osd.8 2024-04-25T13:39:57.838 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:57.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:57 smithi027 ceph-mon[29140]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:57.925 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:57 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2924606240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.038+0000 7f48a8e18700 1 -- 172.21.15.27:0/479312299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a41044e0 msgr2=0x7f48a41048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.038+0000 7f48a8e18700 1 --2- 172.21.15.27:0/479312299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a41044e0 0x7f48a41048f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f48980099c0 tx=0x7f4898009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.038+0000 7f48a8e18700 1 -- 172.21.15.27:0/479312299 shutdown_connections 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.038+0000 7f48a8e18700 1 --2- 172.21.15.27:0/479312299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a4109760 0x7f48a41110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.038+0000 7f48a8e18700 1 --2- 172.21.15.27:0/479312299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a41044e0 0x7f48a41048f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.038+0000 7f48a8e18700 1 --2- 172.21.15.27:0/479312299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a4108e10 0x7f48a4109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.038+0000 7f48a8e18700 1 -- 172.21.15.27:0/479312299 >> 172.21.15.27:0/479312299 conn(0x7f48a407ae40 msgr2=0x7f48a407b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.038+0000 7f48a8e18700 1 -- 172.21.15.27:0/479312299 shutdown_connections 2024-04-25T13:39:58.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.039+0000 7f48a8e18700 1 -- 172.21.15.27:0/479312299 wait complete. 2024-04-25T13:39:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.039+0000 7f48a8e18700 1 Processor -- start 2024-04-25T13:39:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.039+0000 7f48a8e18700 1 -- start start 2024-04-25T13:39:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a8e18700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a41044e0 0x7f48a4075990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a8e18700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a4108e10 0x7f48a4075ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:58.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a8e18700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4109760 0x7f48a4076410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a8e18700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f48a4114390 con 0x7f48a4109760 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a8e18700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f48a4114210 con 0x7f48a41044e0 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a8e18700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f48a4114510 con 0x7f48a4108e10 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4109760 0x7f48a4076410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4109760 0x7f48a4076410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41268/0 (socket says 172.21.15.27:41268) 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 -- 172.21.15.27:0/778101576 learned_addr learned my addr 172.21.15.27:0/778101576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a259c700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a41044e0 0x7f48a4075990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a1d9b700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a4108e10 0x7f48a4075ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 -- 172.21.15.27:0/778101576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a4108e10 msgr2=0x7f48a4075ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a4108e10 0x7f48a4075ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 -- 172.21.15.27:0/778101576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a41044e0 msgr2=0x7f48a4075990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a41044e0 0x7f48a4075990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 -- 172.21.15.27:0/778101576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a41a9080 con 0x7f48a4109760 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.040+0000 7f48a2d9d700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4109760 0x7f48a4076410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f489c007ee0 tx=0x7f489c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:58.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.041+0000 7f48937fe700 1 -- 172.21.15.27:0/778101576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c004bb0 con 0x7f48a4109760 2024-04-25T13:39:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.041+0000 7f48937fe700 1 -- 172.21.15.27:0/778101576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c004d50 con 0x7f48a4109760 2024-04-25T13:39:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.041+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a41a9370 con 0x7f48a4109760 2024-04-25T13:39:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.041+0000 7f48937fe700 1 -- 172.21.15.27:0/778101576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c005920 con 0x7f48a4109760 2024-04-25T13:39:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.041+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a41a9860 con 0x7f48a4109760 2024-04-25T13:39:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.042+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a4051190 con 0x7f48a4109760 2024-04-25T13:39:58.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.043+0000 7f48937fe700 1 -- 172.21.15.27:0/778101576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f489c01a020 con 0x7f48a4109760 2024-04-25T13:39:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.044+0000 7f48937fe700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f488c0afc10 0x7f488c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.044+0000 7f48a259c700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f488c0afc10 0x7f488c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.044+0000 7f48937fe700 1 -- 172.21.15.27:0/778101576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f489c0dea30 con 0x7f48a4109760 2024-04-25T13:39:58.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.045+0000 7f48a259c700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f488c0afc10 0x7f488c0b2120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4898009990 tx=0x7f48980036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:58.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.048+0000 7f48937fe700 1 -- 172.21.15.27:0/778101576 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f489c010070 con 0x7f48a4109760 2024-04-25T13:39:58.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:57 smithi122 ceph-mon[32843]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:58.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:57 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2924606240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T13:39:58.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:57 smithi045 ceph-mon[32731]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:39:58.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:57 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2924606240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T13:39:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.258+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f48a402e6d0 con 0x7f48a4109760 2024-04-25T13:39:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.259+0000 7f48937fe700 1 -- 172.21.15.27:0/778101576 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f489c0a74e0 con 0x7f48a4109760 2024-04-25T13:39:58.259 INFO:teuthology.orchestra.run.smithi027.stdout:111669149767 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f488c0afc10 msgr2=0x7f488c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f488c0afc10 0x7f488c0b2120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4898009990 tx=0x7f48980036e0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4109760 msgr2=0x7f48a4076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4109760 0x7f48a4076410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f489c007ee0 tx=0x7f489c00d3b0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 shutdown_connections 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f488c0afc10 0x7f488c0b2120 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4109760 0x7f48a4076410 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48a41044e0 0x7f48a4075990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 --2- 172.21.15.27:0/778101576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a4108e10 0x7f48a4075ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 >> 172.21.15.27:0/778101576 conn(0x7f48a407ae40 msgr2=0x7f48a4101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 shutdown_connections 2024-04-25T13:39:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:58.261+0000 7f48a8e18700 1 -- 172.21.15.27:0/778101576 wait complete. 2024-04-25T13:39:58.466 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790006 got 158913790011 for osd.5 2024-04-25T13:39:58.466 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:58.934 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:58 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/778101576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T13:39:59.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.506+0000 7f97d912b700 1 -- 172.21.15.27:0/2933940120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4103390 msgr2=0x7f97d41037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:59.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.506+0000 7f97d912b700 1 --2- 172.21.15.27:0/2933940120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4103390 0x7f97d41037a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f97c40099c0 tx=0x7f97c4009cd0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.507+0000 7f97d912b700 1 -- 172.21.15.27:0/2933940120 shutdown_connections 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.507+0000 7f97d912b700 1 --2- 172.21.15.27:0/2933940120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4108610 0x7f97d410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.507+0000 7f97d912b700 1 --2- 172.21.15.27:0/2933940120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4103390 0x7f97d41037a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.507+0000 7f97d912b700 1 --2- 172.21.15.27:0/2933940120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97d4107cc0 0x7f97d41080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.507+0000 7f97d912b700 1 -- 172.21.15.27:0/2933940120 >> 172.21.15.27:0/2933940120 conn(0x7f97d40fec90 msgr2=0x7f97d41010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.507+0000 7f97d912b700 1 -- 172.21.15.27:0/2933940120 shutdown_connections 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.507+0000 7f97d912b700 1 -- 172.21.15.27:0/2933940120 wait complete. 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.507+0000 7f97d912b700 1 Processor -- start 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d912b700 1 -- start start 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d912b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4103390 0x7f97d419bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d912b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4107cc0 0x7f97d419c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d912b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97d4108610 0x7f97d419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d912b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97d4113240 con 0x7f97d4107cc0 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d912b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97d41130c0 con 0x7f97d4103390 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d912b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f97d41133c0 con 0x7f97d4108610 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4107cc0 0x7f97d419c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4107cc0 0x7f97d419c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41250/0 (socket says 172.21.15.27:41250) 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d259c700 1 -- 172.21.15.27:0/2924606240 learned_addr learned my addr 172.21.15.27:0/2924606240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d359e700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97d4108610 0x7f97d419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d2d9d700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4103390 0x7f97d419bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d259c700 1 -- 172.21.15.27:0/2924606240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97d4108610 msgr2=0x7f97d419c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.508+0000 7f97d259c700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97d4108610 0x7f97d419c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97d259c700 1 -- 172.21.15.27:0/2924606240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4103390 msgr2=0x7f97d419bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97d259c700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4103390 0x7f97d419bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97d259c700 1 -- 172.21.15.27:0/2924606240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d41a6e50 con 0x7f97d4107cc0 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97d359e700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97d4108610 0x7f97d419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97d2d9d700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4103390 0x7f97d419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:39:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97d259c700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4107cc0 0x7f97d419c020 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f97c800e970 tx=0x7f97c800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97bbfff700 1 -- 172.21.15.27:0/2924606240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8004a30 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d41a7140 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97bbfff700 1 -- 172.21.15.27:0/2924606240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97c800cc90 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97bbfff700 1 -- 172.21.15.27:0/2924606240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8005610 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.509+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d41a76d0 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.510+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d4051190 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.512+0000 7f97bbfff700 1 -- 172.21.15.27:0/2924606240 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f97c800c7e0 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.512+0000 7f97bbfff700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97bc0afc10 0x7f97bc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.513+0000 7f97d2d9d700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97bc0afc10 0x7f97bc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.513+0000 7f97bbfff700 1 -- 172.21.15.27:0/2924606240 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f97c80df100 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.513+0000 7f97d2d9d700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97bc0afc10 0x7f97bc0b2120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f97c4005b40 tx=0x7f97c401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.516+0000 7f97bbfff700 1 -- 172.21.15.27:0/2924606240 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97c8053ff0 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.728+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f97d406baf0 con 0x7f97d4107cc0 2024-04-25T13:39:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.729+0000 7f97bbfff700 1 -- 172.21.15.27:0/2924606240 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f97c80a7bb0 con 0x7f97d4107cc0 2024-04-25T13:39:59.030 INFO:teuthology.orchestra.run.smithi027.stdout:279172874273 2024-04-25T13:39:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97bc0afc10 msgr2=0x7f97bc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97bc0afc10 0x7f97bc0b2120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f97c4005b40 tx=0x7f97c401a040 comp rx=0 tx=0).stop 2024-04-25T13:39:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4107cc0 msgr2=0x7f97d419c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:39:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4107cc0 0x7f97d419c020 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f97c800e970 tx=0x7f97c800ec80 comp rx=0 tx=0).stop 2024-04-25T13:39:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 shutdown_connections 2024-04-25T13:39:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97bc0afc10 0x7f97bc0b2120 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97d4108610 0x7f97d419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4103390 0x7f97d419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 --2- 172.21.15.27:0/2924606240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d4107cc0 0x7f97d419c020 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:39:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 >> 172.21.15.27:0/2924606240 conn(0x7f97d40fec90 msgr2=0x7f97d4101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:39:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 shutdown_connections 2024-04-25T13:39:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:39:56.731+0000 7f97d912b700 1 -- 172.21.15.27:0/2924606240 wait complete. 2024-04-25T13:39:59.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:58 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/778101576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T13:39:59.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:58 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/778101576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T13:39:59.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874267 got 279172874273 for osd.10 2024-04-25T13:39:59.665 DEBUG:teuthology.parallel:result is None 2024-04-25T13:39:59.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:39:59 smithi027 ceph-mon[29140]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:00.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:39:59 smithi122 ceph-mon[32843]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:00.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:39:59 smithi045 ceph-mon[32731]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:00.222 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149762 got 111669149767 for osd.3 2024-04-25T13:40:00.222 DEBUG:teuthology.parallel:result is None 2024-04-25T13:40:00.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:00 smithi027 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T13:40:01.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:00 smithi122 ceph-mon[32843]: overall HEALTH_OK 2024-04-25T13:40:01.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:00 smithi045 ceph-mon[32731]: overall HEALTH_OK 2024-04-25T13:40:01.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:01 smithi027 ceph-mon[29140]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:01.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:01 smithi027 ceph-mon[29140]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 -- 172.21.15.27:0/1230118227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 msgr2=0x7fb9fc10ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 --2- 172.21.15.27:0/1230118227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 0x7fb9fc10ff40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400b0d0 tx=0x7fb9f400b3e0 comp rx=0 tx=0).stop 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 -- 172.21.15.27:0/1230118227 shutdown_connections 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 --2- 172.21.15.27:0/1230118227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 0x7fb9fc10ff40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 --2- 172.21.15.27:0/1230118227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc103360 0x7fb9fc103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 --2- 172.21.15.27:0/1230118227 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9fc107c90 0x7fb9fc1080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 -- 172.21.15.27:0/1230118227 >> 172.21.15.27:0/1230118227 conn(0x7fb9fc0fec90 msgr2=0x7fb9fc1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 -- 172.21.15.27:0/1230118227 shutdown_connections 2024-04-25T13:40:01.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.989+0000 7fba0454a700 1 -- 172.21.15.27:0/1230118227 wait complete. 2024-04-25T13:40:01.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba0454a700 1 Processor -- start 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba0454a700 1 -- start start 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba0454a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9fc103360 0x7fb9fc19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba0454a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc107c90 0x7fb9fc19c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba0454a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 0x7fb9fc19c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba0454a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc113210 con 0x7fb9fc1085e0 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba0454a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc113090 con 0x7fb9fc107c90 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba0454a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb9fc113390 con 0x7fb9fc103360 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba022e6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9fc103360 0x7fb9fc19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba022e6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9fc103360 0x7fb9fc19baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50582/0 (socket says 172.21.15.27:50582) 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba022e6700 1 -- 172.21.15.27:0/3686555312 learned_addr learned my addr 172.21.15.27:0/3686555312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba01ae5700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc107c90 0x7fb9fc19c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.990+0000 7fba02ae7700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 0x7fb9fc19c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:01.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba022e6700 1 -- 172.21.15.27:0/3686555312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc107c90 msgr2=0x7fb9fc19c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:01.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba022e6700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc107c90 0x7fb9fc19c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:01.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba022e6700 1 -- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 msgr2=0x7fb9fc19c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:01.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba022e6700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 0x7fb9fc19c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:01.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba022e6700 1 -- 172.21.15.27:0/3686555312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9fc1a6db0 con 0x7fb9fc103360 2024-04-25T13:40:01.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba02ae7700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 0x7fb9fc19c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:40:01.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba022e6700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9fc103360 0x7fb9fc19baf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800c920 tx=0x7fb9f800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:01.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fb9f37fe700 1 -- 172.21.15.27:0/3686555312 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8004a30 con 0x7fb9fc103360 2024-04-25T13:40:01.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fb9f37fe700 1 -- 172.21.15.27:0/3686555312 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f80054d0 con 0x7fb9fc103360 2024-04-25T13:40:01.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9fc1a70a0 con 0x7fb9fc103360 2024-04-25T13:40:01.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fb9f37fe700 1 -- 172.21.15.27:0/3686555312 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8007c70 con 0x7fb9fc103360 2024-04-25T13:40:01.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.991+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9fc1a7590 con 0x7fb9fc103360 2024-04-25T13:40:01.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.992+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9fc051190 con 0x7fb9fc103360 2024-04-25T13:40:01.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.994+0000 7fb9f37fe700 1 -- 172.21.15.27:0/3686555312 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb9f8004c60 con 0x7fb9fc103360 2024-04-25T13:40:01.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.995+0000 7fb9f37fe700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9e80afb80 0x7fb9e80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:01.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.995+0000 7fba01ae5700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9e80afb80 0x7fb9e80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:01.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.995+0000 7fb9f37fe700 1 -- 172.21.15.27:0/3686555312 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb9f80deb10 con 0x7fb9fc103360 2024-04-25T13:40:01.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.995+0000 7fba01ae5700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9e80afb80 0x7fb9e80b2090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc102230 tx=0x7fb9ec00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:01.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:01.998+0000 7fb9f37fe700 1 -- 172.21.15.27:0/3686555312 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9f8014030 con 0x7fb9fc103360 2024-04-25T13:40:02.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:01 smithi122 ceph-mon[32843]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:02.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:01 smithi122 ceph-mon[32843]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T13:40:02.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:01 smithi045 ceph-mon[32731]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:02.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:01 smithi045 ceph-mon[32731]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T13:40:02.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.214+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fb9fc02e510 con 0x7fb9fc103360 2024-04-25T13:40:02.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.215+0000 7fb9f37fe700 1 -- 172.21.15.27:0/3686555312 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb9f80a76c0 con 0x7fb9fc103360 2024-04-25T13:40:02.215 INFO:teuthology.orchestra.run.smithi027.stdout:206158430258 2024-04-25T13:40:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9e80afb80 msgr2=0x7fb9e80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9e80afb80 0x7fb9e80b2090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc102230 tx=0x7fb9ec00a380 comp rx=0 tx=0).stop 2024-04-25T13:40:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9fc103360 msgr2=0x7fb9fc19baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9fc103360 0x7fb9fc19baf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800c920 tx=0x7fb9f800cc30 comp rx=0 tx=0).stop 2024-04-25T13:40:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 shutdown_connections 2024-04-25T13:40:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9e80afb80 0x7fb9e80b2090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9fc103360 0x7fb9fc19baf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9fc1085e0 0x7fb9fc19c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 --2- 172.21.15.27:0/3686555312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9fc107c90 0x7fb9fc19c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 >> 172.21.15.27:0/3686555312 conn(0x7fb9fc0fec90 msgr2=0x7fb9fc107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 shutdown_connections 2024-04-25T13:40:02.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.217+0000 7fba0454a700 1 -- 172.21.15.27:0/3686555312 wait complete. 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.688+0000 7fa5424d3700 1 -- 172.21.15.27:0/1282866407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c103360 msgr2=0x7fa53c103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.688+0000 7fa5424d3700 1 --2- 172.21.15.27:0/1282866407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c103360 0x7fa53c103770 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0099c0 tx=0x7fa52c009cd0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.689+0000 7fa5424d3700 1 -- 172.21.15.27:0/1282866407 shutdown_connections 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.689+0000 7fa5424d3700 1 --2- 172.21.15.27:0/1282866407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa53c1085e0 0x7fa53c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.689+0000 7fa5424d3700 1 --2- 172.21.15.27:0/1282866407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c103360 0x7fa53c103770 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.689+0000 7fa5424d3700 1 --2- 172.21.15.27:0/1282866407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa53c107c90 0x7fa53c1080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.689+0000 7fa5424d3700 1 -- 172.21.15.27:0/1282866407 >> 172.21.15.27:0/1282866407 conn(0x7fa53c0fecb0 msgr2=0x7fa53c1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.689+0000 7fa5424d3700 1 -- 172.21.15.27:0/1282866407 shutdown_connections 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.689+0000 7fa5424d3700 1 -- 172.21.15.27:0/1282866407 wait complete. 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa5424d3700 1 Processor -- start 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa5424d3700 1 -- start start 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa5424d3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa53c103360 0x7fa53c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa5424d3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c107c90 0x7fa53c1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa5424d3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa53c1085e0 0x7fa53c19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa5424d3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c114360 con 0x7fa53c107c90 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa5424d3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c1141e0 con 0x7fa53c1085e0 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa5424d3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c1144e0 con 0x7fa53c103360 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa53b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c107c90 0x7fa53c1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa53b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c107c90 0x7fa53c1a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41288/0 (socket says 172.21.15.27:41288) 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa53b7fe700 1 -- 172.21.15.27:0/3280503888 learned_addr learned my addr 172.21.15.27:0/3280503888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa53bfff700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa53c103360 0x7fa53c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa540a70700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa53c1085e0 0x7fa53c19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.690+0000 7fa53b7fe700 1 -- 172.21.15.27:0/3280503888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa53c103360 msgr2=0x7fa53c1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa53b7fe700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa53c103360 0x7fa53c1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa53b7fe700 1 -- 172.21.15.27:0/3280503888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa53c1085e0 msgr2=0x7fa53c19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa53b7fe700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa53c1085e0 0x7fa53c19aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa53b7fe700 1 -- 172.21.15.27:0/3280503888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53c19b170 con 0x7fa53c107c90 2024-04-25T13:40:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa540a70700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa53c1085e0 0x7fa53c19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa53b7fe700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c107c90 0x7fa53c1a0de0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa53000d980 tx=0x7fa53000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa5397fa700 1 -- 172.21.15.27:0/3280503888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530017070 con 0x7fa53c107c90 2024-04-25T13:40:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa5397fa700 1 -- 172.21.15.27:0/3280503888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa530009de0 con 0x7fa53c107c90 2024-04-25T13:40:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53c19b460 con 0x7fa53c107c90 2024-04-25T13:40:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa5397fa700 1 -- 172.21.15.27:0/3280503888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53000f3c0 con 0x7fa53c107c90 2024-04-25T13:40:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.691+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa53c1a7730 con 0x7fa53c107c90 2024-04-25T13:40:02.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.692+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa53c051190 con 0x7fa53c107c90 2024-04-25T13:40:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.694+0000 7fa5397fa700 1 -- 172.21.15.27:0/3280503888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa530009930 con 0x7fa53c107c90 2024-04-25T13:40:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.695+0000 7fa5397fa700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5240afc00 0x7fa5240b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.695+0000 7fa53bfff700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5240afc00 0x7fa5240b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.695+0000 7fa5397fa700 1 -- 172.21.15.27:0/3280503888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5300de6a0 con 0x7fa53c107c90 2024-04-25T13:40:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.696+0000 7fa53bfff700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5240afc00 0x7fa5240b2110 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa53c19bc00 tx=0x7fa52c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:02.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.700+0000 7fa5397fa700 1 -- 172.21.15.27:0/3280503888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5300a72b0 con 0x7fa53c107c90 2024-04-25T13:40:02.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.908+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa53c02e510 con 0x7fa53c107c90 2024-04-25T13:40:02.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.909+0000 7fa5397fa700 1 -- 172.21.15.27:0/3280503888 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa5300a6bb0 con 0x7fa53c107c90 2024-04-25T13:40:02.909 INFO:teuthology.orchestra.run.smithi027.stdout:34359738455 2024-04-25T13:40:02.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5240afc00 msgr2=0x7fa5240b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:02.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5240afc00 0x7fa5240b2110 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa53c19bc00 tx=0x7fa52c01a040 comp rx=0 tx=0).stop 2024-04-25T13:40:02.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c107c90 msgr2=0x7fa53c1a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:02.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c107c90 0x7fa53c1a0de0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa53000d980 tx=0x7fa53000dc90 comp rx=0 tx=0).stop 2024-04-25T13:40:02.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 shutdown_connections 2024-04-25T13:40:02.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5240afc00 0x7fa5240b2110 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa53c103360 0x7fa53c1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa53c1085e0 0x7fa53c19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 --2- 172.21.15.27:0/3280503888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa53c107c90 0x7fa53c1a0de0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:02.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 >> 172.21.15.27:0/3280503888 conn(0x7fa53c0fecb0 msgr2=0x7fa53c107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:02.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 shutdown_connections 2024-04-25T13:40:02.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:02.911+0000 7fa5424d3700 1 -- 172.21.15.27:0/3280503888 wait complete. 2024-04-25T13:40:03.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:02 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3686555312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T13:40:03.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:02 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3686555312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T13:40:03.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:02 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3686555312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T13:40:03.654 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738448 got 34359738455 for osd.0 2024-04-25T13:40:03.654 DEBUG:teuthology.parallel:result is None 2024-04-25T13:40:03.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:03 smithi027 ceph-mon[29140]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:03.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:03 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3280503888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T13:40:04.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:03 smithi122 ceph-mon[32843]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:04.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:03 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3280503888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T13:40:04.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:03 smithi045 ceph-mon[32731]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:04.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:03 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3280503888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T13:40:04.377 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430249 got 206158430258 for osd.7 2024-04-25T13:40:04.377 DEBUG:teuthology.parallel:result is None 2024-04-25T13:40:04.378 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T13:40:04.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T13:40:04.665 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:06.028 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:05 smithi027 ceph-mon[29140]: pgmap v325: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:40:06.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:05 smithi122 ceph-mon[32843]: pgmap v325: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:40:06.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:05 smithi045 ceph-mon[32731]: pgmap v325: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-04-25T13:40:06.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 -- 172.21.15.27:0/3849923852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444103390 msgr2=0x7ff4441037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:06.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 --2- 172.21.15.27:0/3849923852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444103390 0x7ff4441037a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff4340099c0 tx=0x7ff434009cd0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 -- 172.21.15.27:0/3849923852 shutdown_connections 2024-04-25T13:40:06.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 --2- 172.21.15.27:0/3849923852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff444108610 0x7ff44410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 --2- 172.21.15.27:0/3849923852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444103390 0x7ff4441037a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 --2- 172.21.15.27:0/3849923852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444107cc0 0x7ff4441080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 -- 172.21.15.27:0/3849923852 >> 172.21.15.27:0/3849923852 conn(0x7ff4440fec90 msgr2=0x7ff4441010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:06.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 -- 172.21.15.27:0/3849923852 shutdown_connections 2024-04-25T13:40:06.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.446+0000 7ff44916f700 1 -- 172.21.15.27:0/3849923852 wait complete. 2024-04-25T13:40:06.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.447+0000 7ff44916f700 1 Processor -- start 2024-04-25T13:40:06.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.447+0000 7ff44916f700 1 -- start start 2024-04-25T13:40:06.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44916f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444103390 0x7ff44419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:06.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44916f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444107cc0 0x7ff44419c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:06.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44916f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff444108610 0x7ff44419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:06.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44916f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff444113330 con 0x7ff444107cc0 2024-04-25T13:40:06.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44916f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff4441131b0 con 0x7ff444103390 2024-04-25T13:40:06.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44916f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff4441134b0 con 0x7ff444108610 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444107cc0 0x7ff44419c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444107cc0 0x7ff44419c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39314/0 (socket says 172.21.15.27:39314) 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44259c700 1 -- 172.21.15.27:0/703442982 learned_addr learned my addr 172.21.15.27:0/703442982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff44359e700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff444108610 0x7ff44419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.448+0000 7ff442d9d700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444103390 0x7ff44419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff44259c700 1 -- 172.21.15.27:0/703442982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff444108610 msgr2=0x7ff44419c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff44259c700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff444108610 0x7ff44419c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff44259c700 1 -- 172.21.15.27:0/703442982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444103390 msgr2=0x7ff44419bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff44259c700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444103390 0x7ff44419bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff44259c700 1 -- 172.21.15.27:0/703442982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4441a6f20 con 0x7ff444107cc0 2024-04-25T13:40:06.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff44259c700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444107cc0 0x7ff44419c0f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff43800b500 tx=0x7ff43800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:06.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff42bfff700 1 -- 172.21.15.27:0/703442982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438015020 con 0x7ff444107cc0 2024-04-25T13:40:06.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4441a7210 con 0x7ff444107cc0 2024-04-25T13:40:06.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff42bfff700 1 -- 172.21.15.27:0/703442982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff438011ce0 con 0x7ff444107cc0 2024-04-25T13:40:06.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff42bfff700 1 -- 172.21.15.27:0/703442982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438010d40 con 0x7ff444107cc0 2024-04-25T13:40:06.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.449+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4441a77a0 con 0x7ff444107cc0 2024-04-25T13:40:06.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.451+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff430005330 con 0x7ff444107cc0 2024-04-25T13:40:06.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.452+0000 7ff42bfff700 1 -- 172.21.15.27:0/703442982 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff438018020 con 0x7ff444107cc0 2024-04-25T13:40:06.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.452+0000 7ff42bfff700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff42c0afb80 0x7ff42c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:06.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.453+0000 7ff442d9d700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff42c0afb80 0x7ff42c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:06.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.453+0000 7ff42bfff700 1 -- 172.21.15.27:0/703442982 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff4380de200 con 0x7ff444107cc0 2024-04-25T13:40:06.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.453+0000 7ff442d9d700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff42c0afb80 0x7ff42c0b2090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff434003800 tx=0x7ff43401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:06.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.458+0000 7ff42bfff700 1 -- 172.21.15.27:0/703442982 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff43801f020 con 0x7ff444107cc0 2024-04-25T13:40:06.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.683+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff430000bc0 con 0x7ff42c0afb80 2024-04-25T13:40:06.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.691+0000 7ff42bfff700 1 -- 172.21.15.27:0/703442982 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259059 (secure 0 0 0) 0x7ff430000bc0 con 0x7ff42c0afb80 2024-04-25T13:40:06.695 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff42c0afb80 msgr2=0x7ff42c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff42c0afb80 0x7ff42c0b2090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff434003800 tx=0x7ff43401a040 comp rx=0 tx=0).stop 2024-04-25T13:40:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444107cc0 msgr2=0x7ff44419c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444107cc0 0x7ff44419c0f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff43800b500 tx=0x7ff43800b810 comp rx=0 tx=0).stop 2024-04-25T13:40:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 shutdown_connections 2024-04-25T13:40:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff42c0afb80 0x7ff42c0b2090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff444108610 0x7ff44419c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff444103390 0x7ff44419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 --2- 172.21.15.27:0/703442982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff444107cc0 0x7ff44419c0f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 >> 172.21.15.27:0/703442982 conn(0x7ff4440fec90 msgr2=0x7ff444109570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 shutdown_connections 2024-04-25T13:40:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:06.694+0000 7ff44916f700 1 -- 172.21.15.27:0/703442982 wait complete. 2024-04-25T13:40:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T13:40:07.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:06 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:06 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:06 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:40:07.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:06 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:40:07.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:06 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:06 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:06 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:06 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:40:07.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:06 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:40:07.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:06 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:06 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:06 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:06 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:40:07.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:06 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:40:07.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:06 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:40:07.492 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-25T13:40:06.369172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652695,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":246,"num_read_kb":211,"num_write":205,"num_write_kb":2238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951957,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":662,"ondisk_log_size":662,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375956,"kb_used_data":10164,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124565868,"statfs":{"total":1151940427776,"available":1151555448832,"internally_reserved":0,"allocated":10407936,"data_stored":6804213,"data_compressed":114309,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19138,"internal_metadata":374453566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":16,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003098"},"pg_stats":[{"pgid":"3.3f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368465+0000","last_change":"2024-04-25T13:38:45.356552+0000","last_active":"2024-04-25T13:40:00.368465+0000","last_peered":"2024-04-25T13:40:00.368465+0000","last_clean":"2024-04-25T13:40:00.368465+0000","last_became_active":"2024-04-25T13:38:45.356122+0000","last_became_peered":"2024-04-25T13:38:45.356122+0000","last_unstale":"2024-04-25T13:40:00.368465+0000","last_undegraded":"2024-04-25T13:40:00.368465+0000","last_fullsized":"2024-04-25T13:40:00.368465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:55.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354236+0000","last_change":"2024-04-25T13:38:45.358577+0000","last_active":"2024-04-25T13:40:00.354236+0000","last_peered":"2024-04-25T13:40:00.354236+0000","last_clean":"2024-04-25T13:40:00.354236+0000","last_became_active":"2024-04-25T13:38:45.358122+0000","last_became_peered":"2024-04-25T13:38:45.358122+0000","last_unstale":"2024-04-25T13:40:00.354236+0000","last_undegraded":"2024-04-25T13:40:00.354236+0000","last_fullsized":"2024-04-25T13:40:00.354236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:52.508459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368109+0000","last_change":"2024-04-25T13:38:45.362539+0000","last_active":"2024-04-25T13:40:00.368109+0000","last_peered":"2024-04-25T13:40:00.368109+0000","last_clean":"2024-04-25T13:40:00.368109+0000","last_became_active":"2024-04-25T13:38:45.361983+0000","last_became_peered":"2024-04-25T13:38:45.361983+0000","last_unstale":"2024-04-25T13:40:00.368109+0000","last_undegraded":"2024-04-25T13:40:00.368109+0000","last_fullsized":"2024-04-25T13:40:00.368109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:13.267692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.336727+0000","last_change":"2024-04-25T13:38:45.360535+0000","last_active":"2024-04-25T13:40:00.336727+0000","last_peered":"2024-04-25T13:40:00.336727+0000","last_clean":"2024-04-25T13:40:00.336727+0000","last_became_active":"2024-04-25T13:38:45.357722+0000","last_became_peered":"2024-04-25T13:38:45.357722+0000","last_unstale":"2024-04-25T13:40:00.336727+0000","last_undegraded":"2024-04-25T13:40:00.336727+0000","last_fullsized":"2024-04-25T13:40:00.336727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:35.347434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365993+0000","last_change":"2024-04-25T13:38:45.353653+0000","last_active":"2024-04-25T13:40:00.365993+0000","last_peered":"2024-04-25T13:40:00.365993+0000","last_clean":"2024-04-25T13:40:00.365993+0000","last_became_active":"2024-04-25T13:38:45.353254+0000","last_became_peered":"2024-04-25T13:38:45.353254+0000","last_unstale":"2024-04-25T13:40:00.365993+0000","last_undegraded":"2024-04-25T13:40:00.365993+0000","last_fullsized":"2024-04-25T13:40:00.365993+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:57.275732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356027+0000","last_change":"2024-04-25T13:38:45.358241+0000","last_active":"2024-04-25T13:40:00.356027+0000","last_peered":"2024-04-25T13:40:00.356027+0000","last_clean":"2024-04-25T13:40:00.356027+0000","last_became_active":"2024-04-25T13:38:45.357843+0000","last_became_peered":"2024-04-25T13:38:45.357843+0000","last_unstale":"2024-04-25T13:40:00.356027+0000","last_undegraded":"2024-04-25T13:40:00.356027+0000","last_fullsized":"2024-04-25T13:40:00.356027+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:24.560357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375607+0000","last_change":"2024-04-25T13:38:45.350710+0000","last_active":"2024-04-25T13:40:00.375607+0000","last_peered":"2024-04-25T13:40:00.375607+0000","last_clean":"2024-04-25T13:40:00.375607+0000","last_became_active":"2024-04-25T13:38:45.350171+0000","last_became_peered":"2024-04-25T13:38:45.350171+0000","last_unstale":"2024-04-25T13:40:00.375607+0000","last_undegraded":"2024-04-25T13:40:00.375607+0000","last_fullsized":"2024-04-25T13:40:00.375607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:24.933631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363691+0000","last_change":"2024-04-25T13:38:45.356221+0000","last_active":"2024-04-25T13:40:00.363691+0000","last_peered":"2024-04-25T13:40:00.363691+0000","last_clean":"2024-04-25T13:40:00.363691+0000","last_became_active":"2024-04-25T13:38:45.355823+0000","last_became_peered":"2024-04-25T13:38:45.355823+0000","last_unstale":"2024-04-25T13:40:00.363691+0000","last_undegraded":"2024-04-25T13:40:00.363691+0000","last_fullsized":"2024-04-25T13:40:00.363691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:25:02.040407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366879+0000","last_change":"2024-04-25T13:38:45.360940+0000","last_active":"2024-04-25T13:40:00.366879+0000","last_peered":"2024-04-25T13:40:00.366879+0000","last_clean":"2024-04-25T13:40:00.366879+0000","last_became_active":"2024-04-25T13:38:45.360241+0000","last_became_peered":"2024-04-25T13:38:45.360241+0000","last_unstale":"2024-04-25T13:40:00.366879+0000","last_undegraded":"2024-04-25T13:40:00.366879+0000","last_fullsized":"2024-04-25T13:40:00.366879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:05.251236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"78'16","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362862+0000","last_change":"2024-04-25T13:38:45.357908+0000","last_active":"2024-04-25T13:40:00.362862+0000","last_peered":"2024-04-25T13:40:00.362862+0000","last_clean":"2024-04-25T13:40:00.362862+0000","last_became_active":"2024-04-25T13:38:45.357491+0000","last_became_peered":"2024-04-25T13:38:45.357491+0000","last_unstale":"2024-04-25T13:40:00.362862+0000","last_undegraded":"2024-04-25T13:40:00.362862+0000","last_fullsized":"2024-04-25T13:40:00.362862+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:30.931611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374765+0000","last_change":"2024-04-25T13:38:45.368137+0000","last_active":"2024-04-25T13:40:00.374765+0000","last_peered":"2024-04-25T13:40:00.374765+0000","last_clean":"2024-04-25T13:40:00.374765+0000","last_became_active":"2024-04-25T13:38:45.367355+0000","last_became_peered":"2024-04-25T13:38:45.367355+0000","last_unstale":"2024-04-25T13:40:00.374765+0000","last_undegraded":"2024-04-25T13:40:00.374765+0000","last_fullsized":"2024-04-25T13:40:00.374765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:09.502056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"78'7","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349152+0000","last_change":"2024-04-25T13:39:32.041604+0000","last_active":"2024-04-25T13:40:00.349152+0000","last_peered":"2024-04-25T13:40:00.349152+0000","last_clean":"2024-04-25T13:40:00.349152+0000","last_became_active":"2024-04-25T13:39:32.041313+0000","last_became_peered":"2024-04-25T13:39:32.041313+0000","last_unstale":"2024-04-25T13:40:00.349152+0000","last_undegraded":"2024-04-25T13:40:00.349152+0000","last_fullsized":"2024-04-25T13:40:00.349152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:32.390348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357001+0000","last_change":"2024-04-25T13:38:45.353972+0000","last_active":"2024-04-25T13:40:00.357001+0000","last_peered":"2024-04-25T13:40:00.357001+0000","last_clean":"2024-04-25T13:40:00.357001+0000","last_became_active":"2024-04-25T13:38:45.353458+0000","last_became_peered":"2024-04-25T13:38:45.353458+0000","last_unstale":"2024-04-25T13:40:00.357001+0000","last_undegraded":"2024-04-25T13:40:00.357001+0000","last_fullsized":"2024-04-25T13:40:00.357001+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:17.087850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.347216+0000","last_change":"2024-04-25T13:38:45.358876+0000","last_active":"2024-04-25T13:40:00.347216+0000","last_peered":"2024-04-25T13:40:00.347216+0000","last_clean":"2024-04-25T13:40:00.347216+0000","last_became_active":"2024-04-25T13:38:45.358435+0000","last_became_peered":"2024-04-25T13:38:45.358435+0000","last_unstale":"2024-04-25T13:40:00.347216+0000","last_undegraded":"2024-04-25T13:40:00.347216+0000","last_fullsized":"2024-04-25T13:40:00.347216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:18:47.287112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362043+0000","last_change":"2024-04-25T13:38:45.373557+0000","last_active":"2024-04-25T13:40:00.362043+0000","last_peered":"2024-04-25T13:40:00.362043+0000","last_clean":"2024-04-25T13:40:00.362043+0000","last_became_active":"2024-04-25T13:38:45.373179+0000","last_became_peered":"2024-04-25T13:38:45.373179+0000","last_unstale":"2024-04-25T13:40:00.362043+0000","last_undegraded":"2024-04-25T13:40:00.362043+0000","last_fullsized":"2024-04-25T13:40:00.362043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:01.790266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374896+0000","last_change":"2024-04-25T13:38:45.358282+0000","last_active":"2024-04-25T13:40:00.374896+0000","last_peered":"2024-04-25T13:40:00.374896+0000","last_clean":"2024-04-25T13:40:00.374896+0000","last_became_active":"2024-04-25T13:38:45.357923+0000","last_became_peered":"2024-04-25T13:38:45.357923+0000","last_unstale":"2024-04-25T13:40:00.374896+0000","last_undegraded":"2024-04-25T13:40:00.374896+0000","last_fullsized":"2024-04-25T13:40:00.374896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:47.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368746+0000","last_change":"2024-04-25T13:38:45.357865+0000","last_active":"2024-04-25T13:40:00.368746+0000","last_peered":"2024-04-25T13:40:00.368746+0000","last_clean":"2024-04-25T13:40:00.368746+0000","last_became_active":"2024-04-25T13:38:45.355399+0000","last_became_peered":"2024-04-25T13:38:45.355399+0000","last_unstale":"2024-04-25T13:40:00.368746+0000","last_undegraded":"2024-04-25T13:40:00.368746+0000","last_fullsized":"2024-04-25T13:40:00.368746+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:33:41.382462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349318+0000","last_change":"2024-04-25T13:38:45.350724+0000","last_active":"2024-04-25T13:40:00.349318+0000","last_peered":"2024-04-25T13:40:00.349318+0000","last_clean":"2024-04-25T13:40:00.349318+0000","last_became_active":"2024-04-25T13:38:45.350284+0000","last_became_peered":"2024-04-25T13:38:45.350284+0000","last_unstale":"2024-04-25T13:40:00.349318+0000","last_undegraded":"2024-04-25T13:40:00.349318+0000","last_fullsized":"2024-04-25T13:40:00.349318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:30.960843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351696+0000","last_change":"2024-04-25T13:38:45.362966+0000","last_active":"2024-04-25T13:40:00.351696+0000","last_peered":"2024-04-25T13:40:00.351696+0000","last_clean":"2024-04-25T13:40:00.351696+0000","last_became_active":"2024-04-25T13:38:45.362585+0000","last_became_peered":"2024-04-25T13:38:45.362585+0000","last_unstale":"2024-04-25T13:40:00.351696+0000","last_undegraded":"2024-04-25T13:40:00.351696+0000","last_fullsized":"2024-04-25T13:40:00.351696+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:46:10.825028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'4","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.488020+0000","last_change":"2024-04-25T13:38:44.354887+0000","last_active":"2024-04-25T13:39:31.488020+0000","last_peered":"2024-04-25T13:39:31.488020+0000","last_clean":"2024-04-25T13:39:31.488020+0000","last_became_active":"2024-04-25T13:38:44.354510+0000","last_became_peered":"2024-04-25T13:38:44.354510+0000","last_unstale":"2024-04-25T13:39:31.488020+0000","last_undegraded":"2024-04-25T13:39:31.488020+0000","last_fullsized":"2024-04-25T13:39:31.488020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:46.457087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.361295+0000","last_change":"2024-04-25T13:38:45.364056+0000","last_active":"2024-04-25T13:40:00.361295+0000","last_peered":"2024-04-25T13:40:00.361295+0000","last_clean":"2024-04-25T13:40:00.361295+0000","last_became_active":"2024-04-25T13:38:45.363572+0000","last_became_peered":"2024-04-25T13:38:45.363572+0000","last_unstale":"2024-04-25T13:40:00.361295+0000","last_undegraded":"2024-04-25T13:40:00.361295+0000","last_fullsized":"2024-04-25T13:40:00.361295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:49:12.136637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.490264+0000","last_change":"2024-04-25T13:38:44.362371+0000","last_active":"2024-04-25T13:39:31.490264+0000","last_peered":"2024-04-25T13:39:31.490264+0000","last_clean":"2024-04-25T13:39:31.490264+0000","last_became_active":"2024-04-25T13:38:44.361262+0000","last_became_peered":"2024-04-25T13:38:44.361262+0000","last_unstale":"2024-04-25T13:39:31.490264+0000","last_undegraded":"2024-04-25T13:39:31.490264+0000","last_fullsized":"2024-04-25T13:39:31.490264+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:01.442189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367693+0000","last_change":"2024-04-25T13:38:45.362599+0000","last_active":"2024-04-25T13:40:00.367693+0000","last_peered":"2024-04-25T13:40:00.367693+0000","last_clean":"2024-04-25T13:40:00.367693+0000","last_became_active":"2024-04-25T13:38:45.361893+0000","last_became_peered":"2024-04-25T13:38:45.361893+0000","last_unstale":"2024-04-25T13:40:00.367693+0000","last_undegraded":"2024-04-25T13:40:00.367693+0000","last_fullsized":"2024-04-25T13:40:00.367693+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:52.414826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491185+0000","last_change":"2024-04-25T13:38:44.353063+0000","last_active":"2024-04-25T13:39:31.491185+0000","last_peered":"2024-04-25T13:39:31.491185+0000","last_clean":"2024-04-25T13:39:31.491185+0000","last_became_active":"2024-04-25T13:38:44.352403+0000","last_became_peered":"2024-04-25T13:38:44.352403+0000","last_unstale":"2024-04-25T13:39:31.491185+0000","last_undegraded":"2024-04-25T13:39:31.491185+0000","last_fullsized":"2024-04-25T13:39:31.491185+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:09.670589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369475+0000","last_change":"2024-04-25T13:38:45.359078+0000","last_active":"2024-04-25T13:40:00.369475+0000","last_peered":"2024-04-25T13:40:00.369475+0000","last_clean":"2024-04-25T13:40:00.369475+0000","last_became_active":"2024-04-25T13:38:45.354063+0000","last_became_peered":"2024-04-25T13:38:45.354063+0000","last_unstale":"2024-04-25T13:40:00.369475+0000","last_undegraded":"2024-04-25T13:40:00.369475+0000","last_fullsized":"2024-04-25T13:40:00.369475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:16.694340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492390+0000","last_change":"2024-04-25T13:38:44.349132+0000","last_active":"2024-04-25T13:39:31.492390+0000","last_peered":"2024-04-25T13:39:31.492390+0000","last_clean":"2024-04-25T13:39:31.492390+0000","last_became_active":"2024-04-25T13:38:44.348762+0000","last_became_peered":"2024-04-25T13:38:44.348762+0000","last_unstale":"2024-04-25T13:39:31.492390+0000","last_undegraded":"2024-04-25T13:39:31.492390+0000","last_fullsized":"2024-04-25T13:39:31.492390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:12.990304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368999+0000","last_change":"2024-04-25T13:38:45.369116+0000","last_active":"2024-04-25T13:40:00.368999+0000","last_peered":"2024-04-25T13:40:00.368999+0000","last_clean":"2024-04-25T13:40:00.368999+0000","last_became_active":"2024-04-25T13:38:45.368380+0000","last_became_peered":"2024-04-25T13:38:45.368380+0000","last_unstale":"2024-04-25T13:40:00.368999+0000","last_undegraded":"2024-04-25T13:40:00.368999+0000","last_fullsized":"2024-04-25T13:40:00.368999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:02.259401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485437+0000","last_change":"2024-04-25T13:38:44.351873+0000","last_active":"2024-04-25T13:39:31.485437+0000","last_peered":"2024-04-25T13:39:31.485437+0000","last_clean":"2024-04-25T13:39:31.485437+0000","last_became_active":"2024-04-25T13:38:44.351518+0000","last_became_peered":"2024-04-25T13:38:44.351518+0000","last_unstale":"2024-04-25T13:39:31.485437+0000","last_undegraded":"2024-04-25T13:39:31.485437+0000","last_fullsized":"2024-04-25T13:39:31.485437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:06.839053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342392+0000","last_change":"2024-04-25T13:38:45.351369+0000","last_active":"2024-04-25T13:40:00.342392+0000","last_peered":"2024-04-25T13:40:00.342392+0000","last_clean":"2024-04-25T13:40:00.342392+0000","last_became_active":"2024-04-25T13:38:45.351018+0000","last_became_peered":"2024-04-25T13:38:45.351018+0000","last_unstale":"2024-04-25T13:40:00.342392+0000","last_undegraded":"2024-04-25T13:40:00.342392+0000","last_fullsized":"2024-04-25T13:40:00.342392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040292+0000","last_change":"2024-04-25T13:38:44.345986+0000","last_active":"2024-04-25T13:39:32.040292+0000","last_peered":"2024-04-25T13:39:32.040292+0000","last_clean":"2024-04-25T13:39:32.040292+0000","last_became_active":"2024-04-25T13:38:44.345600+0000","last_became_peered":"2024-04-25T13:38:44.345600+0000","last_unstale":"2024-04-25T13:39:32.040292+0000","last_undegraded":"2024-04-25T13:39:32.040292+0000","last_fullsized":"2024-04-25T13:39:32.040292+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:52.749192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365960+0000","last_change":"2024-04-25T13:38:45.368641+0000","last_active":"2024-04-25T13:40:00.365960+0000","last_peered":"2024-04-25T13:40:00.365960+0000","last_clean":"2024-04-25T13:40:00.365960+0000","last_became_active":"2024-04-25T13:38:45.368209+0000","last_became_peered":"2024-04-25T13:38:45.368209+0000","last_unstale":"2024-04-25T13:40:00.365960+0000","last_undegraded":"2024-04-25T13:40:00.365960+0000","last_fullsized":"2024-04-25T13:40:00.365960+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:47.990808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484783+0000","last_change":"2024-04-25T13:38:44.339927+0000","last_active":"2024-04-25T13:39:31.484783+0000","last_peered":"2024-04-25T13:39:31.484783+0000","last_clean":"2024-04-25T13:39:31.484783+0000","last_became_active":"2024-04-25T13:38:44.339588+0000","last_became_peered":"2024-04-25T13:38:44.339588+0000","last_unstale":"2024-04-25T13:39:31.484783+0000","last_undegraded":"2024-04-25T13:39:31.484783+0000","last_fullsized":"2024-04-25T13:39:31.484783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:51.707750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365330+0000","last_change":"2024-04-25T13:38:45.357323+0000","last_active":"2024-04-25T13:40:00.365330+0000","last_peered":"2024-04-25T13:40:00.365330+0000","last_clean":"2024-04-25T13:40:00.365330+0000","last_became_active":"2024-04-25T13:38:45.356870+0000","last_became_peered":"2024-04-25T13:38:45.356870+0000","last_unstale":"2024-04-25T13:40:00.365330+0000","last_undegraded":"2024-04-25T13:40:00.365330+0000","last_fullsized":"2024-04-25T13:40:00.365330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:36.838709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485179+0000","last_change":"2024-04-25T13:38:44.351294+0000","last_active":"2024-04-25T13:39:31.485179+0000","last_peered":"2024-04-25T13:39:31.485179+0000","last_clean":"2024-04-25T13:39:31.485179+0000","last_became_active":"2024-04-25T13:38:44.350971+0000","last_became_peered":"2024-04-25T13:38:44.350971+0000","last_unstale":"2024-04-25T13:39:31.485179+0000","last_undegraded":"2024-04-25T13:39:31.485179+0000","last_fullsized":"2024-04-25T13:39:31.485179+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:22.452217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349813+0000","last_change":"2024-04-25T13:38:45.345943+0000","last_active":"2024-04-25T13:40:00.349813+0000","last_peered":"2024-04-25T13:40:00.349813+0000","last_clean":"2024-04-25T13:40:00.349813+0000","last_became_active":"2024-04-25T13:38:45.345528+0000","last_became_peered":"2024-04-25T13:38:45.345528+0000","last_unstale":"2024-04-25T13:40:00.349813+0000","last_undegraded":"2024-04-25T13:40:00.349813+0000","last_fullsized":"2024-04-25T13:40:00.349813+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:57.327303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040662+0000","last_change":"2024-04-25T13:38:44.343032+0000","last_active":"2024-04-25T13:39:32.040662+0000","last_peered":"2024-04-25T13:39:32.040662+0000","last_clean":"2024-04-25T13:39:32.040662+0000","last_became_active":"2024-04-25T13:38:44.342613+0000","last_became_peered":"2024-04-25T13:38:44.342613+0000","last_unstale":"2024-04-25T13:39:32.040662+0000","last_undegraded":"2024-04-25T13:39:32.040662+0000","last_fullsized":"2024-04-25T13:39:32.040662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:33.957150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369787+0000","last_change":"2024-04-25T13:38:45.354534+0000","last_active":"2024-04-25T13:40:00.369787+0000","last_peered":"2024-04-25T13:40:00.369787+0000","last_clean":"2024-04-25T13:40:00.369787+0000","last_became_active":"2024-04-25T13:38:45.354147+0000","last_became_peered":"2024-04-25T13:38:45.354147+0000","last_unstale":"2024-04-25T13:40:00.369787+0000","last_undegraded":"2024-04-25T13:40:00.369787+0000","last_fullsized":"2024-04-25T13:40:00.369787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:31.537339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487809+0000","last_change":"2024-04-25T13:38:44.356103+0000","last_active":"2024-04-25T13:39:31.487809+0000","last_peered":"2024-04-25T13:39:31.487809+0000","last_clean":"2024-04-25T13:39:31.487809+0000","last_became_active":"2024-04-25T13:38:44.355702+0000","last_became_peered":"2024-04-25T13:38:44.355702+0000","last_unstale":"2024-04-25T13:39:31.487809+0000","last_undegraded":"2024-04-25T13:39:31.487809+0000","last_fullsized":"2024-04-25T13:39:31.487809+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:47.572922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"78'9","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.344967+0000","last_change":"2024-04-25T13:39:31.494580+0000","last_active":"2024-04-25T13:40:00.344967+0000","last_peered":"2024-04-25T13:40:00.344967+0000","last_clean":"2024-04-25T13:40:00.344967+0000","last_became_active":"2024-04-25T13:39:31.494102+0000","last_became_peered":"2024-04-25T13:39:31.494102+0000","last_unstale":"2024-04-25T13:40:00.344967+0000","last_undegraded":"2024-04-25T13:40:00.344967+0000","last_fullsized":"2024-04-25T13:40:00.344967+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.056136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485841+0000","last_change":"2024-04-25T13:38:44.355474+0000","last_active":"2024-04-25T13:39:31.485841+0000","last_peered":"2024-04-25T13:39:31.485841+0000","last_clean":"2024-04-25T13:39:31.485841+0000","last_became_active":"2024-04-25T13:38:44.355113+0000","last_became_peered":"2024-04-25T13:38:44.355113+0000","last_unstale":"2024-04-25T13:39:31.485841+0000","last_undegraded":"2024-04-25T13:39:31.485841+0000","last_fullsized":"2024-04-25T13:39:31.485841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.242974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.370871+0000","last_change":"2024-04-25T13:38:45.369618+0000","last_active":"2024-04-25T13:40:00.370871+0000","last_peered":"2024-04-25T13:40:00.370871+0000","last_clean":"2024-04-25T13:40:00.370871+0000","last_became_active":"2024-04-25T13:38:45.369252+0000","last_became_peered":"2024-04-25T13:38:45.369252+0000","last_unstale":"2024-04-25T13:40:00.370871+0000","last_undegraded":"2024-04-25T13:40:00.370871+0000","last_fullsized":"2024-04-25T13:40:00.370871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:40.336573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"78'29","reported_seq":212,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:05.532787+0000","last_change":"2024-04-25T13:38:44.349850+0000","last_active":"2024-04-25T13:40:05.532787+0000","last_peered":"2024-04-25T13:40:05.532787+0000","last_clean":"2024-04-25T13:40:05.532787+0000","last_became_active":"2024-04-25T13:38:44.349404+0000","last_became_peered":"2024-04-25T13:38:44.349404+0000","last_unstale":"2024-04-25T13:40:05.532787+0000","last_undegraded":"2024-04-25T13:40:05.532787+0000","last_fullsized":"2024-04-25T13:40:05.532787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:40.025684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":141,"num_write":29,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.348329+0000","last_change":"2024-04-25T13:38:45.371554+0000","last_active":"2024-04-25T13:40:00.348329+0000","last_peered":"2024-04-25T13:40:00.348329+0000","last_clean":"2024-04-25T13:40:00.348329+0000","last_became_active":"2024-04-25T13:38:45.371042+0000","last_became_peered":"2024-04-25T13:38:45.371042+0000","last_unstale":"2024-04-25T13:40:00.348329+0000","last_undegraded":"2024-04-25T13:40:00.348329+0000","last_fullsized":"2024-04-25T13:40:00.348329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:59.240536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494618+0000","last_change":"2024-04-25T13:38:44.345979+0000","last_active":"2024-04-25T13:39:31.494618+0000","last_peered":"2024-04-25T13:39:31.494618+0000","last_clean":"2024-04-25T13:39:31.494618+0000","last_became_active":"2024-04-25T13:38:44.345596+0000","last_became_peered":"2024-04-25T13:38:44.345596+0000","last_unstale":"2024-04-25T13:39:31.494618+0000","last_undegraded":"2024-04-25T13:39:31.494618+0000","last_fullsized":"2024-04-25T13:39:31.494618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:22:51.539024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367445+0000","last_change":"2024-04-25T13:38:45.364794+0000","last_active":"2024-04-25T13:40:00.367445+0000","last_peered":"2024-04-25T13:40:00.367445+0000","last_clean":"2024-04-25T13:40:00.367445+0000","last_became_active":"2024-04-25T13:38:45.364381+0000","last_became_peered":"2024-04-25T13:38:45.364381+0000","last_unstale":"2024-04-25T13:40:00.367445+0000","last_undegraded":"2024-04-25T13:40:00.367445+0000","last_fullsized":"2024-04-25T13:40:00.367445+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:16.856980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485286+0000","last_change":"2024-04-25T13:38:44.351269+0000","last_active":"2024-04-25T13:39:31.485286+0000","last_peered":"2024-04-25T13:39:31.485286+0000","last_clean":"2024-04-25T13:39:31.485286+0000","last_became_active":"2024-04-25T13:38:44.350933+0000","last_became_peered":"2024-04-25T13:38:44.350933+0000","last_unstale":"2024-04-25T13:39:31.485286+0000","last_undegraded":"2024-04-25T13:39:31.485286+0000","last_fullsized":"2024-04-25T13:39:31.485286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:02.677849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.372710+0000","last_change":"2024-04-25T13:38:45.362732+0000","last_active":"2024-04-25T13:40:00.372710+0000","last_peered":"2024-04-25T13:40:00.372710+0000","last_clean":"2024-04-25T13:40:00.372710+0000","last_became_active":"2024-04-25T13:38:45.361957+0000","last_became_peered":"2024-04-25T13:38:45.361957+0000","last_unstale":"2024-04-25T13:40:00.372710+0000","last_undegraded":"2024-04-25T13:40:00.372710+0000","last_fullsized":"2024-04-25T13:40:00.372710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:50.011442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.776862+0000","last_change":"2024-04-25T13:38:44.349378+0000","last_active":"2024-04-25T13:39:31.776862+0000","last_peered":"2024-04-25T13:39:31.776862+0000","last_clean":"2024-04-25T13:39:31.776862+0000","last_became_active":"2024-04-25T13:38:44.348887+0000","last_became_peered":"2024-04-25T13:38:44.348887+0000","last_unstale":"2024-04-25T13:39:31.776862+0000","last_undegraded":"2024-04-25T13:39:31.776862+0000","last_fullsized":"2024-04-25T13:39:31.776862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.747205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375453+0000","last_change":"2024-04-25T13:38:45.349941+0000","last_active":"2024-04-25T13:40:00.375453+0000","last_peered":"2024-04-25T13:40:00.375453+0000","last_clean":"2024-04-25T13:40:00.375453+0000","last_became_active":"2024-04-25T13:38:45.349590+0000","last_became_peered":"2024-04-25T13:38:45.349590+0000","last_unstale":"2024-04-25T13:40:00.375453+0000","last_undegraded":"2024-04-25T13:40:00.375453+0000","last_fullsized":"2024-04-25T13:40:00.375453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:42.904406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.791483+0000","last_change":"2024-04-25T13:39:31.790997+0000","last_active":"2024-04-25T13:39:31.791483+0000","last_peered":"2024-04-25T13:39:31.791483+0000","last_clean":"2024-04-25T13:39:31.791483+0000","last_became_active":"2024-04-25T13:39:31.778451+0000","last_became_peered":"2024-04-25T13:39:31.778451+0000","last_unstale":"2024-04-25T13:39:31.791483+0000","last_undegraded":"2024-04-25T13:39:31.791483+0000","last_fullsized":"2024-04-25T13:39:31.791483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:17.735970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368028+0000","last_change":"2024-04-25T13:38:45.368589+0000","last_active":"2024-04-25T13:40:00.368028+0000","last_peered":"2024-04-25T13:40:00.368028+0000","last_clean":"2024-04-25T13:40:00.368028+0000","last_became_active":"2024-04-25T13:38:45.368079+0000","last_became_peered":"2024-04-25T13:38:45.368079+0000","last_unstale":"2024-04-25T13:40:00.368028+0000","last_undegraded":"2024-04-25T13:40:00.368028+0000","last_fullsized":"2024-04-25T13:40:00.368028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:10.180264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491662+0000","last_change":"2024-04-25T13:38:44.345148+0000","last_active":"2024-04-25T13:39:31.491662+0000","last_peered":"2024-04-25T13:39:31.491662+0000","last_clean":"2024-04-25T13:39:31.491662+0000","last_became_active":"2024-04-25T13:38:44.344816+0000","last_became_peered":"2024-04-25T13:38:44.344816+0000","last_unstale":"2024-04-25T13:39:31.491662+0000","last_undegraded":"2024-04-25T13:39:31.491662+0000","last_fullsized":"2024-04-25T13:39:31.491662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:03:38.134169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351253+0000","last_change":"2024-04-25T13:38:45.354340+0000","last_active":"2024-04-25T13:40:00.351253+0000","last_peered":"2024-04-25T13:40:00.351253+0000","last_clean":"2024-04-25T13:40:00.351253+0000","last_became_active":"2024-04-25T13:38:45.353933+0000","last_became_peered":"2024-04-25T13:38:45.353933+0000","last_unstale":"2024-04-25T13:40:00.351253+0000","last_undegraded":"2024-04-25T13:40:00.351253+0000","last_fullsized":"2024-04-25T13:40:00.351253+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:07.619051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487938+0000","last_change":"2024-04-25T13:38:44.351183+0000","last_active":"2024-04-25T13:39:31.487938+0000","last_peered":"2024-04-25T13:39:31.487938+0000","last_clean":"2024-04-25T13:39:31.487938+0000","last_became_active":"2024-04-25T13:38:44.350785+0000","last_became_peered":"2024-04-25T13:38:44.350785+0000","last_unstale":"2024-04-25T13:39:31.487938+0000","last_undegraded":"2024-04-25T13:39:31.487938+0000","last_fullsized":"2024-04-25T13:39:31.487938+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:37:57.465012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360705+0000","last_change":"2024-04-25T13:38:45.362355+0000","last_active":"2024-04-25T13:40:00.360705+0000","last_peered":"2024-04-25T13:40:00.360705+0000","last_clean":"2024-04-25T13:40:00.360705+0000","last_became_active":"2024-04-25T13:38:45.361996+0000","last_became_peered":"2024-04-25T13:38:45.361996+0000","last_unstale":"2024-04-25T13:40:00.360705+0000","last_undegraded":"2024-04-25T13:40:00.360705+0000","last_fullsized":"2024-04-25T13:40:00.360705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:53:24.790450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487406+0000","last_change":"2024-04-25T13:38:44.356079+0000","last_active":"2024-04-25T13:39:31.487406+0000","last_peered":"2024-04-25T13:39:31.487406+0000","last_clean":"2024-04-25T13:39:31.487406+0000","last_became_active":"2024-04-25T13:38:44.355660+0000","last_became_peered":"2024-04-25T13:38:44.355660+0000","last_unstale":"2024-04-25T13:39:31.487406+0000","last_undegraded":"2024-04-25T13:39:31.487406+0000","last_fullsized":"2024-04-25T13:39:31.487406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:35.161337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.353301+0000","last_change":"2024-04-25T13:38:45.356578+0000","last_active":"2024-04-25T13:40:00.353301+0000","last_peered":"2024-04-25T13:40:00.353301+0000","last_clean":"2024-04-25T13:40:00.353301+0000","last_became_active":"2024-04-25T13:38:45.356232+0000","last_became_peered":"2024-04-25T13:38:45.356232+0000","last_unstale":"2024-04-25T13:40:00.353301+0000","last_undegraded":"2024-04-25T13:40:00.353301+0000","last_fullsized":"2024-04-25T13:40:00.353301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.476890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"78'8","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360561+0000","last_change":"2024-04-25T13:39:31.778577+0000","last_active":"2024-04-25T13:40:00.360561+0000","last_peered":"2024-04-25T13:40:00.360561+0000","last_clean":"2024-04-25T13:40:00.360561+0000","last_became_active":"2024-04-25T13:39:31.778178+0000","last_became_peered":"2024-04-25T13:39:31.778178+0000","last_unstale":"2024-04-25T13:40:00.360561+0000","last_undegraded":"2024-04-25T13:40:00.360561+0000","last_fullsized":"2024-04-25T13:40:00.360561+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.383670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484649+0000","last_change":"2024-04-25T13:38:44.351113+0000","last_active":"2024-04-25T13:39:31.484649+0000","last_peered":"2024-04-25T13:39:31.484649+0000","last_clean":"2024-04-25T13:39:31.484649+0000","last_became_active":"2024-04-25T13:38:44.350730+0000","last_became_peered":"2024-04-25T13:38:44.350730+0000","last_unstale":"2024-04-25T13:39:31.484649+0000","last_undegraded":"2024-04-25T13:39:31.484649+0000","last_fullsized":"2024-04-25T13:39:31.484649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:38:22.747930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362720+0000","last_change":"2024-04-25T13:38:45.356731+0000","last_active":"2024-04-25T13:40:00.362720+0000","last_peered":"2024-04-25T13:40:00.362720+0000","last_clean":"2024-04-25T13:40:00.362720+0000","last_became_active":"2024-04-25T13:38:45.355531+0000","last_became_peered":"2024-04-25T13:38:45.355531+0000","last_unstale":"2024-04-25T13:40:00.362720+0000","last_undegraded":"2024-04-25T13:40:00.362720+0000","last_fullsized":"2024-04-25T13:40:00.362720+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:15.482469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777207+0000","last_change":"2024-04-25T13:38:44.345941+0000","last_active":"2024-04-25T13:39:31.777207+0000","last_peered":"2024-04-25T13:39:31.777207+0000","last_clean":"2024-04-25T13:39:31.777207+0000","last_became_active":"2024-04-25T13:38:44.344634+0000","last_became_peered":"2024-04-25T13:38:44.344634+0000","last_unstale":"2024-04-25T13:39:31.777207+0000","last_undegraded":"2024-04-25T13:39:31.777207+0000","last_fullsized":"2024-04-25T13:39:31.777207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:42.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":93,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487278+0000","last_change":"2024-04-25T13:37:27.903345+0000","last_active":"2024-04-25T13:39:31.487278+0000","last_peered":"2024-04-25T13:39:31.487278+0000","last_clean":"2024-04-25T13:39:31.487278+0000","last_became_active":"2024-04-25T13:37:27.897986+0000","last_became_peered":"2024-04-25T13:37:27.897986+0000","last_unstale":"2024-04-25T13:39:31.487278+0000","last_undegraded":"2024-04-25T13:39:31.487278+0000","last_fullsized":"2024-04-25T13:39:31.487278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_clean_scrub_stamp":"2024-04-25T13:33:56.569417+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:56.390458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360997+0000","last_change":"2024-04-25T13:38:45.369390+0000","last_active":"2024-04-25T13:40:00.360997+0000","last_peered":"2024-04-25T13:40:00.360997+0000","last_clean":"2024-04-25T13:40:00.360997+0000","last_became_active":"2024-04-25T13:38:45.368938+0000","last_became_peered":"2024-04-25T13:38:45.368938+0000","last_unstale":"2024-04-25T13:40:00.360997+0000","last_undegraded":"2024-04-25T13:40:00.360997+0000","last_fullsized":"2024-04-25T13:40:00.360997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:11.942204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487420+0000","last_change":"2024-04-25T13:38:44.355585+0000","last_active":"2024-04-25T13:39:31.487420+0000","last_peered":"2024-04-25T13:39:31.487420+0000","last_clean":"2024-04-25T13:39:31.487420+0000","last_became_active":"2024-04-25T13:38:44.355131+0000","last_became_peered":"2024-04-25T13:38:44.355131+0000","last_unstale":"2024-04-25T13:39:31.487420+0000","last_undegraded":"2024-04-25T13:39:31.487420+0000","last_fullsized":"2024-04-25T13:39:31.487420+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:05.240247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492418+0000","last_change":"2024-04-25T13:39:31.491742+0000","last_active":"2024-04-25T13:39:31.492418+0000","last_peered":"2024-04-25T13:39:31.492418+0000","last_clean":"2024-04-25T13:39:31.492418+0000","last_became_active":"2024-04-25T13:39:31.491266+0000","last_became_peered":"2024-04-25T13:39:31.491266+0000","last_unstale":"2024-04-25T13:39:31.492418+0000","last_undegraded":"2024-04-25T13:39:31.492418+0000","last_fullsized":"2024-04-25T13:39:31.492418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:21.932333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.355121+0000","last_change":"2024-04-25T13:38:45.366187+0000","last_active":"2024-04-25T13:40:00.355121+0000","last_peered":"2024-04-25T13:40:00.355121+0000","last_clean":"2024-04-25T13:40:00.355121+0000","last_became_active":"2024-04-25T13:38:45.365744+0000","last_became_peered":"2024-04-25T13:38:45.365744+0000","last_unstale":"2024-04-25T13:40:00.355121+0000","last_undegraded":"2024-04-25T13:40:00.355121+0000","last_fullsized":"2024-04-25T13:40:00.355121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:07.970735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"78'7","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487865+0000","last_change":"2024-04-25T13:38:44.359250+0000","last_active":"2024-04-25T13:39:31.487865+0000","last_peered":"2024-04-25T13:39:31.487865+0000","last_clean":"2024-04-25T13:39:31.487865+0000","last_became_active":"2024-04-25T13:38:44.358874+0000","last_became_peered":"2024-04-25T13:38:44.358874+0000","last_unstale":"2024-04-25T13:39:31.487865+0000","last_undegraded":"2024-04-25T13:39:31.487865+0000","last_fullsized":"2024-04-25T13:39:31.487865+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:28.820465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366411+0000","last_change":"2024-04-25T13:38:45.370080+0000","last_active":"2024-04-25T13:40:00.366411+0000","last_peered":"2024-04-25T13:40:00.366411+0000","last_clean":"2024-04-25T13:40:00.366411+0000","last_became_active":"2024-04-25T13:38:45.369668+0000","last_became_peered":"2024-04-25T13:38:45.369668+0000","last_unstale":"2024-04-25T13:40:00.366411+0000","last_undegraded":"2024-04-25T13:40:00.366411+0000","last_fullsized":"2024-04-25T13:40:00.366411+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:03.197468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777370+0000","last_change":"2024-04-25T13:38:44.358907+0000","last_active":"2024-04-25T13:39:31.777370+0000","last_peered":"2024-04-25T13:39:31.777370+0000","last_clean":"2024-04-25T13:39:31.777370+0000","last_became_active":"2024-04-25T13:38:44.358455+0000","last_became_peered":"2024-04-25T13:38:44.358455+0000","last_unstale":"2024-04-25T13:39:31.777370+0000","last_undegraded":"2024-04-25T13:39:31.777370+0000","last_fullsized":"2024-04-25T13:39:31.777370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:50:52.916433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365642+0000","last_change":"2024-04-25T13:38:45.373050+0000","last_active":"2024-04-25T13:40:00.365642+0000","last_peered":"2024-04-25T13:40:00.365642+0000","last_clean":"2024-04-25T13:40:00.365642+0000","last_became_active":"2024-04-25T13:38:45.372621+0000","last_became_peered":"2024-04-25T13:38:45.372621+0000","last_unstale":"2024-04-25T13:40:00.365642+0000","last_undegraded":"2024-04-25T13:40:00.365642+0000","last_fullsized":"2024-04-25T13:40:00.365642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:29:43.259296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485534+0000","last_change":"2024-04-25T13:38:44.355768+0000","last_active":"2024-04-25T13:39:31.485534+0000","last_peered":"2024-04-25T13:39:31.485534+0000","last_clean":"2024-04-25T13:39:31.485534+0000","last_became_active":"2024-04-25T13:38:44.355400+0000","last_became_peered":"2024-04-25T13:38:44.355400+0000","last_unstale":"2024-04-25T13:39:31.485534+0000","last_undegraded":"2024-04-25T13:39:31.485534+0000","last_fullsized":"2024-04-25T13:39:31.485534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:38.533717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342472+0000","last_change":"2024-04-25T13:38:45.350582+0000","last_active":"2024-04-25T13:40:00.342472+0000","last_peered":"2024-04-25T13:40:00.342472+0000","last_clean":"2024-04-25T13:40:00.342472+0000","last_became_active":"2024-04-25T13:38:45.350220+0000","last_became_peered":"2024-04-25T13:38:45.350220+0000","last_unstale":"2024-04-25T13:40:00.342472+0000","last_undegraded":"2024-04-25T13:40:00.342472+0000","last_fullsized":"2024-04-25T13:40:00.342472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.463519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.489729+0000","last_change":"2024-04-25T13:38:44.360459+0000","last_active":"2024-04-25T13:39:31.489729+0000","last_peered":"2024-04-25T13:39:31.489729+0000","last_clean":"2024-04-25T13:39:31.489729+0000","last_became_active":"2024-04-25T13:38:44.359709+0000","last_became_peered":"2024-04-25T13:38:44.359709+0000","last_unstale":"2024-04-25T13:39:31.489729+0000","last_undegraded":"2024-04-25T13:39:31.489729+0000","last_fullsized":"2024-04-25T13:39:31.489729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:53.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356628+0000","last_change":"2024-04-25T13:38:45.357291+0000","last_active":"2024-04-25T13:40:00.356628+0000","last_peered":"2024-04-25T13:40:00.356628+0000","last_clean":"2024-04-25T13:40:00.356628+0000","last_became_active":"2024-04-25T13:38:45.356929+0000","last_became_peered":"2024-04-25T13:38:45.356929+0000","last_unstale":"2024-04-25T13:40:00.356628+0000","last_undegraded":"2024-04-25T13:40:00.356628+0000","last_fullsized":"2024-04-25T13:40:00.356628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:02:08.619208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.493732+0000","last_change":"2024-04-25T13:38:44.343885+0000","last_active":"2024-04-25T13:39:31.493732+0000","last_peered":"2024-04-25T13:39:31.493732+0000","last_clean":"2024-04-25T13:39:31.493732+0000","last_became_active":"2024-04-25T13:38:44.343541+0000","last_became_peered":"2024-04-25T13:38:44.343541+0000","last_unstale":"2024-04-25T13:39:31.493732+0000","last_undegraded":"2024-04-25T13:39:31.493732+0000","last_fullsized":"2024-04-25T13:39:31.493732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:50:01.904593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365216+0000","last_change":"2024-04-25T13:38:45.371508+0000","last_active":"2024-04-25T13:40:00.365216+0000","last_peered":"2024-04-25T13:40:00.365216+0000","last_clean":"2024-04-25T13:40:00.365216+0000","last_became_active":"2024-04-25T13:38:45.370912+0000","last_became_peered":"2024-04-25T13:38:45.370912+0000","last_unstale":"2024-04-25T13:40:00.365216+0000","last_undegraded":"2024-04-25T13:40:00.365216+0000","last_fullsized":"2024-04-25T13:40:00.365216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:07.075511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'2","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.498142+0000","last_change":"2024-04-25T13:39:31.498142+0000","last_active":"2024-04-25T13:39:31.498142+0000","last_peered":"2024-04-25T13:39:31.498142+0000","last_clean":"2024-04-25T13:39:31.498142+0000","last_became_active":"2024-04-25T13:39:31.494646+0000","last_became_peered":"2024-04-25T13:39:31.494646+0000","last_unstale":"2024-04-25T13:39:31.498142+0000","last_undegraded":"2024-04-25T13:39:31.498142+0000","last_fullsized":"2024-04-25T13:39:31.498142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:46.084354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375168+0000","last_change":"2024-04-25T13:38:45.363322+0000","last_active":"2024-04-25T13:40:00.375168+0000","last_peered":"2024-04-25T13:40:00.375168+0000","last_clean":"2024-04-25T13:40:00.375168+0000","last_became_active":"2024-04-25T13:38:45.362948+0000","last_became_peered":"2024-04-25T13:38:45.362948+0000","last_unstale":"2024-04-25T13:40:00.375168+0000","last_undegraded":"2024-04-25T13:40:00.375168+0000","last_fullsized":"2024-04-25T13:40:00.375168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:20.815774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494630+0000","last_change":"2024-04-25T13:38:44.358588+0000","last_active":"2024-04-25T13:39:31.494630+0000","last_peered":"2024-04-25T13:39:31.494630+0000","last_clean":"2024-04-25T13:39:31.494630+0000","last_became_active":"2024-04-25T13:38:44.358269+0000","last_became_peered":"2024-04-25T13:38:44.358269+0000","last_unstale":"2024-04-25T13:39:31.494630+0000","last_undegraded":"2024-04-25T13:39:31.494630+0000","last_fullsized":"2024-04-25T13:39:31.494630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:35.405572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375021+0000","last_change":"2024-04-25T13:38:45.369272+0000","last_active":"2024-04-25T13:40:00.375021+0000","last_peered":"2024-04-25T13:40:00.375021+0000","last_clean":"2024-04-25T13:40:00.375021+0000","last_became_active":"2024-04-25T13:38:45.368774+0000","last_became_peered":"2024-04-25T13:38:45.368774+0000","last_unstale":"2024-04-25T13:40:00.375021+0000","last_undegraded":"2024-04-25T13:40:00.375021+0000","last_fullsized":"2024-04-25T13:40:00.375021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:39.149557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485416+0000","last_change":"2024-04-25T13:38:44.349479+0000","last_active":"2024-04-25T13:39:31.485416+0000","last_peered":"2024-04-25T13:39:31.485416+0000","last_clean":"2024-04-25T13:39:31.485416+0000","last_became_active":"2024-04-25T13:38:44.349104+0000","last_became_peered":"2024-04-25T13:38:44.349104+0000","last_unstale":"2024-04-25T13:39:31.485416+0000","last_undegraded":"2024-04-25T13:39:31.485416+0000","last_fullsized":"2024-04-25T13:39:31.485416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:39.257472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"78'7","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375501+0000","last_change":"2024-04-25T13:39:31.495349+0000","last_active":"2024-04-25T13:40:00.375501+0000","last_peered":"2024-04-25T13:40:00.375501+0000","last_clean":"2024-04-25T13:40:00.375501+0000","last_became_active":"2024-04-25T13:39:31.490741+0000","last_became_peered":"2024-04-25T13:39:31.490741+0000","last_unstale":"2024-04-25T13:40:00.375501+0000","last_undegraded":"2024-04-25T13:40:00.375501+0000","last_fullsized":"2024-04-25T13:40:00.375501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.474725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484923+0000","last_change":"2024-04-25T13:38:44.349502+0000","last_active":"2024-04-25T13:39:31.484923+0000","last_peered":"2024-04-25T13:39:31.484923+0000","last_clean":"2024-04-25T13:39:31.484923+0000","last_became_active":"2024-04-25T13:38:44.349127+0000","last_became_peered":"2024-04-25T13:38:44.349127+0000","last_unstale":"2024-04-25T13:39:31.484923+0000","last_undegraded":"2024-04-25T13:39:31.484923+0000","last_fullsized":"2024-04-25T13:39:31.484923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:34:50.673352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.371319+0000","last_change":"2024-04-25T13:38:45.364746+0000","last_active":"2024-04-25T13:40:00.371319+0000","last_peered":"2024-04-25T13:40:00.371319+0000","last_clean":"2024-04-25T13:40:00.371319+0000","last_became_active":"2024-04-25T13:38:45.364364+0000","last_became_peered":"2024-04-25T13:38:45.364364+0000","last_unstale":"2024-04-25T13:40:00.371319+0000","last_undegraded":"2024-04-25T13:40:00.371319+0000","last_fullsized":"2024-04-25T13:40:00.371319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:42.145637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357194+0000","last_change":"2024-04-25T13:38:45.356683+0000","last_active":"2024-04-25T13:40:00.357194+0000","last_peered":"2024-04-25T13:40:00.357194+0000","last_clean":"2024-04-25T13:40:00.357194+0000","last_became_active":"2024-04-25T13:38:45.356048+0000","last_became_peered":"2024-04-25T13:38:45.356048+0000","last_unstale":"2024-04-25T13:40:00.357194+0000","last_undegraded":"2024-04-25T13:40:00.357194+0000","last_fullsized":"2024-04-25T13:40:00.357194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:49.578317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"78'12","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362649+0000","last_change":"2024-04-25T13:38:45.358298+0000","last_active":"2024-04-25T13:40:00.362649+0000","last_peered":"2024-04-25T13:40:00.362649+0000","last_clean":"2024-04-25T13:40:00.362649+0000","last_became_active":"2024-04-25T13:38:45.357986+0000","last_became_peered":"2024-04-25T13:38:45.357986+0000","last_unstale":"2024-04-25T13:40:00.362649+0000","last_undegraded":"2024-04-25T13:40:00.362649+0000","last_fullsized":"2024-04-25T13:40:00.362649+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:33.093156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.355496+0000","last_change":"2024-04-25T13:38:45.363043+0000","last_active":"2024-04-25T13:40:00.355496+0000","last_peered":"2024-04-25T13:40:00.355496+0000","last_clean":"2024-04-25T13:40:00.355496+0000","last_became_active":"2024-04-25T13:38:45.362686+0000","last_became_peered":"2024-04-25T13:38:45.362686+0000","last_unstale":"2024-04-25T13:40:00.355496+0000","last_undegraded":"2024-04-25T13:40:00.355496+0000","last_fullsized":"2024-04-25T13:40:00.355496+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:56.602134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362067+0000","last_change":"2024-04-25T13:38:45.373256+0000","last_active":"2024-04-25T13:40:00.362067+0000","last_peered":"2024-04-25T13:40:00.362067+0000","last_clean":"2024-04-25T13:40:00.362067+0000","last_became_active":"2024-04-25T13:38:45.372844+0000","last_became_peered":"2024-04-25T13:38:45.372844+0000","last_unstale":"2024-04-25T13:40:00.362067+0000","last_undegraded":"2024-04-25T13:40:00.362067+0000","last_fullsized":"2024-04-25T13:40:00.362067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:39:34.590536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.364083+0000","last_change":"2024-04-25T13:38:45.370007+0000","last_active":"2024-04-25T13:40:00.364083+0000","last_peered":"2024-04-25T13:40:00.364083+0000","last_clean":"2024-04-25T13:40:00.364083+0000","last_became_active":"2024-04-25T13:38:45.369606+0000","last_became_peered":"2024-04-25T13:38:45.369606+0000","last_unstale":"2024-04-25T13:40:00.364083+0000","last_undegraded":"2024-04-25T13:40:00.364083+0000","last_fullsized":"2024-04-25T13:40:00.364083+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:45.514756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.350193+0000","last_change":"2024-04-25T13:38:45.362075+0000","last_active":"2024-04-25T13:40:00.350193+0000","last_peered":"2024-04-25T13:40:00.350193+0000","last_clean":"2024-04-25T13:40:00.350193+0000","last_became_active":"2024-04-25T13:38:45.361720+0000","last_became_peered":"2024-04-25T13:38:45.361720+0000","last_unstale":"2024-04-25T13:40:00.350193+0000","last_undegraded":"2024-04-25T13:40:00.350193+0000","last_fullsized":"2024-04-25T13:40:00.350193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:36.169185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354853+0000","last_change":"2024-04-25T13:38:45.352101+0000","last_active":"2024-04-25T13:40:00.354853+0000","last_peered":"2024-04-25T13:40:00.354853+0000","last_clean":"2024-04-25T13:40:00.354853+0000","last_became_active":"2024-04-25T13:38:45.351739+0000","last_became_peered":"2024-04-25T13:38:45.351739+0000","last_unstale":"2024-04-25T13:40:00.354853+0000","last_undegraded":"2024-04-25T13:40:00.354853+0000","last_fullsized":"2024-04-25T13:40:00.354853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:05.612527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362759+0000","last_change":"2024-04-25T13:38:45.363855+0000","last_active":"2024-04-25T13:40:00.362759+0000","last_peered":"2024-04-25T13:40:00.362759+0000","last_clean":"2024-04-25T13:40:00.362759+0000","last_became_active":"2024-04-25T13:38:45.363377+0000","last_became_peered":"2024-04-25T13:38:45.363377+0000","last_unstale":"2024-04-25T13:40:00.362759+0000","last_undegraded":"2024-04-25T13:40:00.362759+0000","last_fullsized":"2024-04-25T13:40:00.362759+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:26.828215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365565+0000","last_change":"2024-04-25T13:38:45.359844+0000","last_active":"2024-04-25T13:40:00.365565+0000","last_peered":"2024-04-25T13:40:00.365565+0000","last_clean":"2024-04-25T13:40:00.365565+0000","last_became_active":"2024-04-25T13:38:45.359428+0000","last_became_peered":"2024-04-25T13:38:45.359428+0000","last_unstale":"2024-04-25T13:40:00.365565+0000","last_undegraded":"2024-04-25T13:40:00.365565+0000","last_fullsized":"2024-04-25T13:40:00.365565+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:01.370409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.359173+0000","last_change":"2024-04-25T13:38:45.370415+0000","last_active":"2024-04-25T13:40:00.359173+0000","last_peered":"2024-04-25T13:40:00.359173+0000","last_clean":"2024-04-25T13:40:00.359173+0000","last_became_active":"2024-04-25T13:38:45.369525+0000","last_became_peered":"2024-04-25T13:38:45.369525+0000","last_unstale":"2024-04-25T13:40:00.359173+0000","last_undegraded":"2024-04-25T13:40:00.359173+0000","last_fullsized":"2024-04-25T13:40:00.359173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:39.532380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.376082+0000","last_change":"2024-04-25T13:38:45.365309+0000","last_active":"2024-04-25T13:40:00.376082+0000","last_peered":"2024-04-25T13:40:00.376082+0000","last_clean":"2024-04-25T13:40:00.376082+0000","last_became_active":"2024-04-25T13:38:45.364893+0000","last_became_peered":"2024-04-25T13:38:45.364893+0000","last_unstale":"2024-04-25T13:40:00.376082+0000","last_undegraded":"2024-04-25T13:40:00.376082+0000","last_fullsized":"2024-04-25T13:40:00.376082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:04:54.454443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368064+0000","last_change":"2024-04-25T13:38:45.365150+0000","last_active":"2024-04-25T13:40:00.368064+0000","last_peered":"2024-04-25T13:40:00.368064+0000","last_clean":"2024-04-25T13:40:00.368064+0000","last_became_active":"2024-04-25T13:38:45.364649+0000","last_became_peered":"2024-04-25T13:38:45.364649+0000","last_unstale":"2024-04-25T13:40:00.368064+0000","last_undegraded":"2024-04-25T13:40:00.368064+0000","last_fullsized":"2024-04-25T13:40:00.368064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:25.035789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354459+0000","last_change":"2024-04-25T13:38:45.352108+0000","last_active":"2024-04-25T13:40:00.354459+0000","last_peered":"2024-04-25T13:40:00.354459+0000","last_clean":"2024-04-25T13:40:00.354459+0000","last_became_active":"2024-04-25T13:38:45.349887+0000","last_became_peered":"2024-04-25T13:38:45.349887+0000","last_unstale":"2024-04-25T13:40:00.354459+0000","last_undegraded":"2024-04-25T13:40:00.354459+0000","last_fullsized":"2024-04-25T13:40:00.354459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:09.339685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285228,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62093,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":170,"num_read_kb":147,"num_write":80,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":186279,"data_compressed":5526,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678046,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":403844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874275,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":1055818,"data_compressed":38103,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037801,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234030,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430259,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":888832,"data_stored":994911,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593785,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":465139,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":403934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":465173,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":843776,"data_stored":404065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790013,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":994511,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61295,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61329,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61489,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T13:40:07.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T13:40:07.751 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:08.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:07 smithi027 ceph-mon[29140]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:08.129 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:07 smithi027 ceph-mon[29140]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:08.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:07 smithi122 ceph-mon[32843]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:08.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:07 smithi122 ceph-mon[32843]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:07 smithi045 ceph-mon[32731]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:07 smithi045 ceph-mon[32731]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 -- 172.21.15.27:0/1280159966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39401085e0 msgr2=0x7f394010ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 --2- 172.21.15.27:0/1280159966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39401085e0 0x7f394010ff40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f393800b0d0 tx=0x7f393800b3e0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 -- 172.21.15.27:0/1280159966 shutdown_connections 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 --2- 172.21.15.27:0/1280159966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39401085e0 0x7f394010ff40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 --2- 172.21.15.27:0/1280159966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3940103360 0x7f3940103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 --2- 172.21.15.27:0/1280159966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3940107c90 0x7f39401080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 -- 172.21.15.27:0/1280159966 >> 172.21.15.27:0/1280159966 conn(0x7f39400fecb0 msgr2=0x7f39401010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 -- 172.21.15.27:0/1280159966 shutdown_connections 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.493+0000 7f3947d21700 1 -- 172.21.15.27:0/1280159966 wait complete. 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3947d21700 1 Processor -- start 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3947d21700 1 -- start start 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3947d21700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3940103360 0x7f39401a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3947d21700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3940107c90 0x7f39401a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3947d21700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39401085e0 0x7f394019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3947d21700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3940114360 con 0x7f3940103360 2024-04-25T13:40:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3947d21700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39401141e0 con 0x7f3940107c90 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3947d21700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f39401144e0 con 0x7f39401085e0 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3945abd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3940103360 0x7f39401a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3945abd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3940103360 0x7f39401a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39334/0 (socket says 172.21.15.27:39334) 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.494+0000 7f3945abd700 1 -- 172.21.15.27:0/652398693 learned_addr learned my addr 172.21.15.27:0/652398693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f39462be700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39401085e0 0x7f394019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f39452bc700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3940107c90 0x7f39401a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3945abd700 1 -- 172.21.15.27:0/652398693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39401085e0 msgr2=0x7f394019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3945abd700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39401085e0 0x7f394019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3945abd700 1 -- 172.21.15.27:0/652398693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3940107c90 msgr2=0x7f39401a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3945abd700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3940107c90 0x7f39401a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3945abd700 1 -- 172.21.15.27:0/652398693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f394019b0a0 con 0x7f3940103360 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f39452bc700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3940107c90 0x7f39401a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3945abd700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3940103360 0x7f39401a0840 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f393c00c9b0 tx=0x7f393c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3932ffd700 1 -- 172.21.15.27:0/652398693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c007b00 con 0x7f3940103360 2024-04-25T13:40:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f394019b390 con 0x7f3940103360 2024-04-25T13:40:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3932ffd700 1 -- 172.21.15.27:0/652398693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f393c007ca0 con 0x7f3940103360 2024-04-25T13:40:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3932ffd700 1 -- 172.21.15.27:0/652398693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c018660 con 0x7f3940103360 2024-04-25T13:40:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.495+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39401a76d0 con 0x7f3940103360 2024-04-25T13:40:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.496+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3940051190 con 0x7f3940103360 2024-04-25T13:40:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.498+0000 7f3932ffd700 1 -- 172.21.15.27:0/652398693 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f393c00f450 con 0x7f3940103360 2024-04-25T13:40:09.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.499+0000 7f3932ffd700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f392c0afb80 0x7f392c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:09.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.499+0000 7f39452bc700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f392c0afb80 0x7f392c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:09.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.499+0000 7f3932ffd700 1 -- 172.21.15.27:0/652398693 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f393c0dee90 con 0x7f3940103360 2024-04-25T13:40:09.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.500+0000 7f39452bc700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f392c0afb80 0x7f392c0b2090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3934005e10 tx=0x7f3934005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.503+0000 7f3932ffd700 1 -- 172.21.15.27:0/652398693 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f393c014030 con 0x7f3940103360 2024-04-25T13:40:09.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.705+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f394019bf90 con 0x7f392c0afb80 2024-04-25T13:40:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.712+0000 7f3932ffd700 1 -- 172.21.15.27:0/652398693 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259019 (secure 0 0 0) 0x7f394019bf90 con 0x7f392c0afb80 2024-04-25T13:40:09.715 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f392c0afb80 msgr2=0x7f392c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f392c0afb80 0x7f392c0b2090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3934005e10 tx=0x7f3934005d60 comp rx=0 tx=0).stop 2024-04-25T13:40:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3940103360 msgr2=0x7f39401a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3940103360 0x7f39401a0840 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f393c00c9b0 tx=0x7f393c00ccc0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 shutdown_connections 2024-04-25T13:40:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f392c0afb80 0x7f392c0b2090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39401085e0 0x7f394019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3940103360 0x7f39401a0840 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 --2- 172.21.15.27:0/652398693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3940107c90 0x7f39401a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 >> 172.21.15.27:0/652398693 conn(0x7f39400fecb0 msgr2=0x7f39401073d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 shutdown_connections 2024-04-25T13:40:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:09.716+0000 7f3947d21700 1 -- 172.21.15.27:0/652398693 wait complete. 2024-04-25T13:40:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T13:40:10.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:09 smithi122 ceph-mon[32843]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:10.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:09 smithi045 ceph-mon[32731]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:10.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:09 smithi027 ceph-mon[29140]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:10.558 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-04-25T13:40:08.370091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652695,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":246,"num_read_kb":211,"num_write":205,"num_write_kb":2238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951957,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":662,"ondisk_log_size":662,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375956,"kb_used_data":10164,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124565868,"statfs":{"total":1151940427776,"available":1151555448832,"internally_reserved":0,"allocated":10407936,"data_stored":6804213,"data_compressed":114309,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19138,"internal_metadata":374453566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003491"},"pg_stats":[{"pgid":"3.3f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368465+0000","last_change":"2024-04-25T13:38:45.356552+0000","last_active":"2024-04-25T13:40:00.368465+0000","last_peered":"2024-04-25T13:40:00.368465+0000","last_clean":"2024-04-25T13:40:00.368465+0000","last_became_active":"2024-04-25T13:38:45.356122+0000","last_became_peered":"2024-04-25T13:38:45.356122+0000","last_unstale":"2024-04-25T13:40:00.368465+0000","last_undegraded":"2024-04-25T13:40:00.368465+0000","last_fullsized":"2024-04-25T13:40:00.368465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:55.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354236+0000","last_change":"2024-04-25T13:38:45.358577+0000","last_active":"2024-04-25T13:40:00.354236+0000","last_peered":"2024-04-25T13:40:00.354236+0000","last_clean":"2024-04-25T13:40:00.354236+0000","last_became_active":"2024-04-25T13:38:45.358122+0000","last_became_peered":"2024-04-25T13:38:45.358122+0000","last_unstale":"2024-04-25T13:40:00.354236+0000","last_undegraded":"2024-04-25T13:40:00.354236+0000","last_fullsized":"2024-04-25T13:40:00.354236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:52.508459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368109+0000","last_change":"2024-04-25T13:38:45.362539+0000","last_active":"2024-04-25T13:40:00.368109+0000","last_peered":"2024-04-25T13:40:00.368109+0000","last_clean":"2024-04-25T13:40:00.368109+0000","last_became_active":"2024-04-25T13:38:45.361983+0000","last_became_peered":"2024-04-25T13:38:45.361983+0000","last_unstale":"2024-04-25T13:40:00.368109+0000","last_undegraded":"2024-04-25T13:40:00.368109+0000","last_fullsized":"2024-04-25T13:40:00.368109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:13.267692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.336727+0000","last_change":"2024-04-25T13:38:45.360535+0000","last_active":"2024-04-25T13:40:00.336727+0000","last_peered":"2024-04-25T13:40:00.336727+0000","last_clean":"2024-04-25T13:40:00.336727+0000","last_became_active":"2024-04-25T13:38:45.357722+0000","last_became_peered":"2024-04-25T13:38:45.357722+0000","last_unstale":"2024-04-25T13:40:00.336727+0000","last_undegraded":"2024-04-25T13:40:00.336727+0000","last_fullsized":"2024-04-25T13:40:00.336727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:35.347434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365993+0000","last_change":"2024-04-25T13:38:45.353653+0000","last_active":"2024-04-25T13:40:00.365993+0000","last_peered":"2024-04-25T13:40:00.365993+0000","last_clean":"2024-04-25T13:40:00.365993+0000","last_became_active":"2024-04-25T13:38:45.353254+0000","last_became_peered":"2024-04-25T13:38:45.353254+0000","last_unstale":"2024-04-25T13:40:00.365993+0000","last_undegraded":"2024-04-25T13:40:00.365993+0000","last_fullsized":"2024-04-25T13:40:00.365993+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:57.275732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356027+0000","last_change":"2024-04-25T13:38:45.358241+0000","last_active":"2024-04-25T13:40:00.356027+0000","last_peered":"2024-04-25T13:40:00.356027+0000","last_clean":"2024-04-25T13:40:00.356027+0000","last_became_active":"2024-04-25T13:38:45.357843+0000","last_became_peered":"2024-04-25T13:38:45.357843+0000","last_unstale":"2024-04-25T13:40:00.356027+0000","last_undegraded":"2024-04-25T13:40:00.356027+0000","last_fullsized":"2024-04-25T13:40:00.356027+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:24.560357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375607+0000","last_change":"2024-04-25T13:38:45.350710+0000","last_active":"2024-04-25T13:40:00.375607+0000","last_peered":"2024-04-25T13:40:00.375607+0000","last_clean":"2024-04-25T13:40:00.375607+0000","last_became_active":"2024-04-25T13:38:45.350171+0000","last_became_peered":"2024-04-25T13:38:45.350171+0000","last_unstale":"2024-04-25T13:40:00.375607+0000","last_undegraded":"2024-04-25T13:40:00.375607+0000","last_fullsized":"2024-04-25T13:40:00.375607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:24.933631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363691+0000","last_change":"2024-04-25T13:38:45.356221+0000","last_active":"2024-04-25T13:40:00.363691+0000","last_peered":"2024-04-25T13:40:00.363691+0000","last_clean":"2024-04-25T13:40:00.363691+0000","last_became_active":"2024-04-25T13:38:45.355823+0000","last_became_peered":"2024-04-25T13:38:45.355823+0000","last_unstale":"2024-04-25T13:40:00.363691+0000","last_undegraded":"2024-04-25T13:40:00.363691+0000","last_fullsized":"2024-04-25T13:40:00.363691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:25:02.040407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366879+0000","last_change":"2024-04-25T13:38:45.360940+0000","last_active":"2024-04-25T13:40:00.366879+0000","last_peered":"2024-04-25T13:40:00.366879+0000","last_clean":"2024-04-25T13:40:00.366879+0000","last_became_active":"2024-04-25T13:38:45.360241+0000","last_became_peered":"2024-04-25T13:38:45.360241+0000","last_unstale":"2024-04-25T13:40:00.366879+0000","last_undegraded":"2024-04-25T13:40:00.366879+0000","last_fullsized":"2024-04-25T13:40:00.366879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:05.251236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"78'16","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362862+0000","last_change":"2024-04-25T13:38:45.357908+0000","last_active":"2024-04-25T13:40:00.362862+0000","last_peered":"2024-04-25T13:40:00.362862+0000","last_clean":"2024-04-25T13:40:00.362862+0000","last_became_active":"2024-04-25T13:38:45.357491+0000","last_became_peered":"2024-04-25T13:38:45.357491+0000","last_unstale":"2024-04-25T13:40:00.362862+0000","last_undegraded":"2024-04-25T13:40:00.362862+0000","last_fullsized":"2024-04-25T13:40:00.362862+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:30.931611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374765+0000","last_change":"2024-04-25T13:38:45.368137+0000","last_active":"2024-04-25T13:40:00.374765+0000","last_peered":"2024-04-25T13:40:00.374765+0000","last_clean":"2024-04-25T13:40:00.374765+0000","last_became_active":"2024-04-25T13:38:45.367355+0000","last_became_peered":"2024-04-25T13:38:45.367355+0000","last_unstale":"2024-04-25T13:40:00.374765+0000","last_undegraded":"2024-04-25T13:40:00.374765+0000","last_fullsized":"2024-04-25T13:40:00.374765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:09.502056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"78'7","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349152+0000","last_change":"2024-04-25T13:39:32.041604+0000","last_active":"2024-04-25T13:40:00.349152+0000","last_peered":"2024-04-25T13:40:00.349152+0000","last_clean":"2024-04-25T13:40:00.349152+0000","last_became_active":"2024-04-25T13:39:32.041313+0000","last_became_peered":"2024-04-25T13:39:32.041313+0000","last_unstale":"2024-04-25T13:40:00.349152+0000","last_undegraded":"2024-04-25T13:40:00.349152+0000","last_fullsized":"2024-04-25T13:40:00.349152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:32.390348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357001+0000","last_change":"2024-04-25T13:38:45.353972+0000","last_active":"2024-04-25T13:40:00.357001+0000","last_peered":"2024-04-25T13:40:00.357001+0000","last_clean":"2024-04-25T13:40:00.357001+0000","last_became_active":"2024-04-25T13:38:45.353458+0000","last_became_peered":"2024-04-25T13:38:45.353458+0000","last_unstale":"2024-04-25T13:40:00.357001+0000","last_undegraded":"2024-04-25T13:40:00.357001+0000","last_fullsized":"2024-04-25T13:40:00.357001+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:17.087850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.347216+0000","last_change":"2024-04-25T13:38:45.358876+0000","last_active":"2024-04-25T13:40:00.347216+0000","last_peered":"2024-04-25T13:40:00.347216+0000","last_clean":"2024-04-25T13:40:00.347216+0000","last_became_active":"2024-04-25T13:38:45.358435+0000","last_became_peered":"2024-04-25T13:38:45.358435+0000","last_unstale":"2024-04-25T13:40:00.347216+0000","last_undegraded":"2024-04-25T13:40:00.347216+0000","last_fullsized":"2024-04-25T13:40:00.347216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:18:47.287112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362043+0000","last_change":"2024-04-25T13:38:45.373557+0000","last_active":"2024-04-25T13:40:00.362043+0000","last_peered":"2024-04-25T13:40:00.362043+0000","last_clean":"2024-04-25T13:40:00.362043+0000","last_became_active":"2024-04-25T13:38:45.373179+0000","last_became_peered":"2024-04-25T13:38:45.373179+0000","last_unstale":"2024-04-25T13:40:00.362043+0000","last_undegraded":"2024-04-25T13:40:00.362043+0000","last_fullsized":"2024-04-25T13:40:00.362043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:01.790266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374896+0000","last_change":"2024-04-25T13:38:45.358282+0000","last_active":"2024-04-25T13:40:00.374896+0000","last_peered":"2024-04-25T13:40:00.374896+0000","last_clean":"2024-04-25T13:40:00.374896+0000","last_became_active":"2024-04-25T13:38:45.357923+0000","last_became_peered":"2024-04-25T13:38:45.357923+0000","last_unstale":"2024-04-25T13:40:00.374896+0000","last_undegraded":"2024-04-25T13:40:00.374896+0000","last_fullsized":"2024-04-25T13:40:00.374896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:47.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368746+0000","last_change":"2024-04-25T13:38:45.357865+0000","last_active":"2024-04-25T13:40:00.368746+0000","last_peered":"2024-04-25T13:40:00.368746+0000","last_clean":"2024-04-25T13:40:00.368746+0000","last_became_active":"2024-04-25T13:38:45.355399+0000","last_became_peered":"2024-04-25T13:38:45.355399+0000","last_unstale":"2024-04-25T13:40:00.368746+0000","last_undegraded":"2024-04-25T13:40:00.368746+0000","last_fullsized":"2024-04-25T13:40:00.368746+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:33:41.382462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349318+0000","last_change":"2024-04-25T13:38:45.350724+0000","last_active":"2024-04-25T13:40:00.349318+0000","last_peered":"2024-04-25T13:40:00.349318+0000","last_clean":"2024-04-25T13:40:00.349318+0000","last_became_active":"2024-04-25T13:38:45.350284+0000","last_became_peered":"2024-04-25T13:38:45.350284+0000","last_unstale":"2024-04-25T13:40:00.349318+0000","last_undegraded":"2024-04-25T13:40:00.349318+0000","last_fullsized":"2024-04-25T13:40:00.349318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:30.960843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351696+0000","last_change":"2024-04-25T13:38:45.362966+0000","last_active":"2024-04-25T13:40:00.351696+0000","last_peered":"2024-04-25T13:40:00.351696+0000","last_clean":"2024-04-25T13:40:00.351696+0000","last_became_active":"2024-04-25T13:38:45.362585+0000","last_became_peered":"2024-04-25T13:38:45.362585+0000","last_unstale":"2024-04-25T13:40:00.351696+0000","last_undegraded":"2024-04-25T13:40:00.351696+0000","last_fullsized":"2024-04-25T13:40:00.351696+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:46:10.825028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'4","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.488020+0000","last_change":"2024-04-25T13:38:44.354887+0000","last_active":"2024-04-25T13:39:31.488020+0000","last_peered":"2024-04-25T13:39:31.488020+0000","last_clean":"2024-04-25T13:39:31.488020+0000","last_became_active":"2024-04-25T13:38:44.354510+0000","last_became_peered":"2024-04-25T13:38:44.354510+0000","last_unstale":"2024-04-25T13:39:31.488020+0000","last_undegraded":"2024-04-25T13:39:31.488020+0000","last_fullsized":"2024-04-25T13:39:31.488020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:46.457087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.361295+0000","last_change":"2024-04-25T13:38:45.364056+0000","last_active":"2024-04-25T13:40:00.361295+0000","last_peered":"2024-04-25T13:40:00.361295+0000","last_clean":"2024-04-25T13:40:00.361295+0000","last_became_active":"2024-04-25T13:38:45.363572+0000","last_became_peered":"2024-04-25T13:38:45.363572+0000","last_unstale":"2024-04-25T13:40:00.361295+0000","last_undegraded":"2024-04-25T13:40:00.361295+0000","last_fullsized":"2024-04-25T13:40:00.361295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:49:12.136637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.490264+0000","last_change":"2024-04-25T13:38:44.362371+0000","last_active":"2024-04-25T13:39:31.490264+0000","last_peered":"2024-04-25T13:39:31.490264+0000","last_clean":"2024-04-25T13:39:31.490264+0000","last_became_active":"2024-04-25T13:38:44.361262+0000","last_became_peered":"2024-04-25T13:38:44.361262+0000","last_unstale":"2024-04-25T13:39:31.490264+0000","last_undegraded":"2024-04-25T13:39:31.490264+0000","last_fullsized":"2024-04-25T13:39:31.490264+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:01.442189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367693+0000","last_change":"2024-04-25T13:38:45.362599+0000","last_active":"2024-04-25T13:40:00.367693+0000","last_peered":"2024-04-25T13:40:00.367693+0000","last_clean":"2024-04-25T13:40:00.367693+0000","last_became_active":"2024-04-25T13:38:45.361893+0000","last_became_peered":"2024-04-25T13:38:45.361893+0000","last_unstale":"2024-04-25T13:40:00.367693+0000","last_undegraded":"2024-04-25T13:40:00.367693+0000","last_fullsized":"2024-04-25T13:40:00.367693+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:52.414826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491185+0000","last_change":"2024-04-25T13:38:44.353063+0000","last_active":"2024-04-25T13:39:31.491185+0000","last_peered":"2024-04-25T13:39:31.491185+0000","last_clean":"2024-04-25T13:39:31.491185+0000","last_became_active":"2024-04-25T13:38:44.352403+0000","last_became_peered":"2024-04-25T13:38:44.352403+0000","last_unstale":"2024-04-25T13:39:31.491185+0000","last_undegraded":"2024-04-25T13:39:31.491185+0000","last_fullsized":"2024-04-25T13:39:31.491185+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:09.670589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369475+0000","last_change":"2024-04-25T13:38:45.359078+0000","last_active":"2024-04-25T13:40:00.369475+0000","last_peered":"2024-04-25T13:40:00.369475+0000","last_clean":"2024-04-25T13:40:00.369475+0000","last_became_active":"2024-04-25T13:38:45.354063+0000","last_became_peered":"2024-04-25T13:38:45.354063+0000","last_unstale":"2024-04-25T13:40:00.369475+0000","last_undegraded":"2024-04-25T13:40:00.369475+0000","last_fullsized":"2024-04-25T13:40:00.369475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:16.694340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492390+0000","last_change":"2024-04-25T13:38:44.349132+0000","last_active":"2024-04-25T13:39:31.492390+0000","last_peered":"2024-04-25T13:39:31.492390+0000","last_clean":"2024-04-25T13:39:31.492390+0000","last_became_active":"2024-04-25T13:38:44.348762+0000","last_became_peered":"2024-04-25T13:38:44.348762+0000","last_unstale":"2024-04-25T13:39:31.492390+0000","last_undegraded":"2024-04-25T13:39:31.492390+0000","last_fullsized":"2024-04-25T13:39:31.492390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:12.990304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368999+0000","last_change":"2024-04-25T13:38:45.369116+0000","last_active":"2024-04-25T13:40:00.368999+0000","last_peered":"2024-04-25T13:40:00.368999+0000","last_clean":"2024-04-25T13:40:00.368999+0000","last_became_active":"2024-04-25T13:38:45.368380+0000","last_became_peered":"2024-04-25T13:38:45.368380+0000","last_unstale":"2024-04-25T13:40:00.368999+0000","last_undegraded":"2024-04-25T13:40:00.368999+0000","last_fullsized":"2024-04-25T13:40:00.368999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:02.259401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485437+0000","last_change":"2024-04-25T13:38:44.351873+0000","last_active":"2024-04-25T13:39:31.485437+0000","last_peered":"2024-04-25T13:39:31.485437+0000","last_clean":"2024-04-25T13:39:31.485437+0000","last_became_active":"2024-04-25T13:38:44.351518+0000","last_became_peered":"2024-04-25T13:38:44.351518+0000","last_unstale":"2024-04-25T13:39:31.485437+0000","last_undegraded":"2024-04-25T13:39:31.485437+0000","last_fullsized":"2024-04-25T13:39:31.485437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:06.839053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342392+0000","last_change":"2024-04-25T13:38:45.351369+0000","last_active":"2024-04-25T13:40:00.342392+0000","last_peered":"2024-04-25T13:40:00.342392+0000","last_clean":"2024-04-25T13:40:00.342392+0000","last_became_active":"2024-04-25T13:38:45.351018+0000","last_became_peered":"2024-04-25T13:38:45.351018+0000","last_unstale":"2024-04-25T13:40:00.342392+0000","last_undegraded":"2024-04-25T13:40:00.342392+0000","last_fullsized":"2024-04-25T13:40:00.342392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040292+0000","last_change":"2024-04-25T13:38:44.345986+0000","last_active":"2024-04-25T13:39:32.040292+0000","last_peered":"2024-04-25T13:39:32.040292+0000","last_clean":"2024-04-25T13:39:32.040292+0000","last_became_active":"2024-04-25T13:38:44.345600+0000","last_became_peered":"2024-04-25T13:38:44.345600+0000","last_unstale":"2024-04-25T13:39:32.040292+0000","last_undegraded":"2024-04-25T13:39:32.040292+0000","last_fullsized":"2024-04-25T13:39:32.040292+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:52.749192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365960+0000","last_change":"2024-04-25T13:38:45.368641+0000","last_active":"2024-04-25T13:40:00.365960+0000","last_peered":"2024-04-25T13:40:00.365960+0000","last_clean":"2024-04-25T13:40:00.365960+0000","last_became_active":"2024-04-25T13:38:45.368209+0000","last_became_peered":"2024-04-25T13:38:45.368209+0000","last_unstale":"2024-04-25T13:40:00.365960+0000","last_undegraded":"2024-04-25T13:40:00.365960+0000","last_fullsized":"2024-04-25T13:40:00.365960+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:47.990808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484783+0000","last_change":"2024-04-25T13:38:44.339927+0000","last_active":"2024-04-25T13:39:31.484783+0000","last_peered":"2024-04-25T13:39:31.484783+0000","last_clean":"2024-04-25T13:39:31.484783+0000","last_became_active":"2024-04-25T13:38:44.339588+0000","last_became_peered":"2024-04-25T13:38:44.339588+0000","last_unstale":"2024-04-25T13:39:31.484783+0000","last_undegraded":"2024-04-25T13:39:31.484783+0000","last_fullsized":"2024-04-25T13:39:31.484783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:51.707750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365330+0000","last_change":"2024-04-25T13:38:45.357323+0000","last_active":"2024-04-25T13:40:00.365330+0000","last_peered":"2024-04-25T13:40:00.365330+0000","last_clean":"2024-04-25T13:40:00.365330+0000","last_became_active":"2024-04-25T13:38:45.356870+0000","last_became_peered":"2024-04-25T13:38:45.356870+0000","last_unstale":"2024-04-25T13:40:00.365330+0000","last_undegraded":"2024-04-25T13:40:00.365330+0000","last_fullsized":"2024-04-25T13:40:00.365330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:36.838709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485179+0000","last_change":"2024-04-25T13:38:44.351294+0000","last_active":"2024-04-25T13:39:31.485179+0000","last_peered":"2024-04-25T13:39:31.485179+0000","last_clean":"2024-04-25T13:39:31.485179+0000","last_became_active":"2024-04-25T13:38:44.350971+0000","last_became_peered":"2024-04-25T13:38:44.350971+0000","last_unstale":"2024-04-25T13:39:31.485179+0000","last_undegraded":"2024-04-25T13:39:31.485179+0000","last_fullsized":"2024-04-25T13:39:31.485179+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:22.452217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349813+0000","last_change":"2024-04-25T13:38:45.345943+0000","last_active":"2024-04-25T13:40:00.349813+0000","last_peered":"2024-04-25T13:40:00.349813+0000","last_clean":"2024-04-25T13:40:00.349813+0000","last_became_active":"2024-04-25T13:38:45.345528+0000","last_became_peered":"2024-04-25T13:38:45.345528+0000","last_unstale":"2024-04-25T13:40:00.349813+0000","last_undegraded":"2024-04-25T13:40:00.349813+0000","last_fullsized":"2024-04-25T13:40:00.349813+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:57.327303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040662+0000","last_change":"2024-04-25T13:38:44.343032+0000","last_active":"2024-04-25T13:39:32.040662+0000","last_peered":"2024-04-25T13:39:32.040662+0000","last_clean":"2024-04-25T13:39:32.040662+0000","last_became_active":"2024-04-25T13:38:44.342613+0000","last_became_peered":"2024-04-25T13:38:44.342613+0000","last_unstale":"2024-04-25T13:39:32.040662+0000","last_undegraded":"2024-04-25T13:39:32.040662+0000","last_fullsized":"2024-04-25T13:39:32.040662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:33.957150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369787+0000","last_change":"2024-04-25T13:38:45.354534+0000","last_active":"2024-04-25T13:40:00.369787+0000","last_peered":"2024-04-25T13:40:00.369787+0000","last_clean":"2024-04-25T13:40:00.369787+0000","last_became_active":"2024-04-25T13:38:45.354147+0000","last_became_peered":"2024-04-25T13:38:45.354147+0000","last_unstale":"2024-04-25T13:40:00.369787+0000","last_undegraded":"2024-04-25T13:40:00.369787+0000","last_fullsized":"2024-04-25T13:40:00.369787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:31.537339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487809+0000","last_change":"2024-04-25T13:38:44.356103+0000","last_active":"2024-04-25T13:39:31.487809+0000","last_peered":"2024-04-25T13:39:31.487809+0000","last_clean":"2024-04-25T13:39:31.487809+0000","last_became_active":"2024-04-25T13:38:44.355702+0000","last_became_peered":"2024-04-25T13:38:44.355702+0000","last_unstale":"2024-04-25T13:39:31.487809+0000","last_undegraded":"2024-04-25T13:39:31.487809+0000","last_fullsized":"2024-04-25T13:39:31.487809+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:47.572922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"78'9","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.344967+0000","last_change":"2024-04-25T13:39:31.494580+0000","last_active":"2024-04-25T13:40:00.344967+0000","last_peered":"2024-04-25T13:40:00.344967+0000","last_clean":"2024-04-25T13:40:00.344967+0000","last_became_active":"2024-04-25T13:39:31.494102+0000","last_became_peered":"2024-04-25T13:39:31.494102+0000","last_unstale":"2024-04-25T13:40:00.344967+0000","last_undegraded":"2024-04-25T13:40:00.344967+0000","last_fullsized":"2024-04-25T13:40:00.344967+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.056136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485841+0000","last_change":"2024-04-25T13:38:44.355474+0000","last_active":"2024-04-25T13:39:31.485841+0000","last_peered":"2024-04-25T13:39:31.485841+0000","last_clean":"2024-04-25T13:39:31.485841+0000","last_became_active":"2024-04-25T13:38:44.355113+0000","last_became_peered":"2024-04-25T13:38:44.355113+0000","last_unstale":"2024-04-25T13:39:31.485841+0000","last_undegraded":"2024-04-25T13:39:31.485841+0000","last_fullsized":"2024-04-25T13:39:31.485841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.242974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.370871+0000","last_change":"2024-04-25T13:38:45.369618+0000","last_active":"2024-04-25T13:40:00.370871+0000","last_peered":"2024-04-25T13:40:00.370871+0000","last_clean":"2024-04-25T13:40:00.370871+0000","last_became_active":"2024-04-25T13:38:45.369252+0000","last_became_peered":"2024-04-25T13:38:45.369252+0000","last_unstale":"2024-04-25T13:40:00.370871+0000","last_undegraded":"2024-04-25T13:40:00.370871+0000","last_fullsized":"2024-04-25T13:40:00.370871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:40.336573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"78'29","reported_seq":212,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:05.532787+0000","last_change":"2024-04-25T13:38:44.349850+0000","last_active":"2024-04-25T13:40:05.532787+0000","last_peered":"2024-04-25T13:40:05.532787+0000","last_clean":"2024-04-25T13:40:05.532787+0000","last_became_active":"2024-04-25T13:38:44.349404+0000","last_became_peered":"2024-04-25T13:38:44.349404+0000","last_unstale":"2024-04-25T13:40:05.532787+0000","last_undegraded":"2024-04-25T13:40:05.532787+0000","last_fullsized":"2024-04-25T13:40:05.532787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:40.025684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":164,"num_read_kb":141,"num_write":29,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.348329+0000","last_change":"2024-04-25T13:38:45.371554+0000","last_active":"2024-04-25T13:40:00.348329+0000","last_peered":"2024-04-25T13:40:00.348329+0000","last_clean":"2024-04-25T13:40:00.348329+0000","last_became_active":"2024-04-25T13:38:45.371042+0000","last_became_peered":"2024-04-25T13:38:45.371042+0000","last_unstale":"2024-04-25T13:40:00.348329+0000","last_undegraded":"2024-04-25T13:40:00.348329+0000","last_fullsized":"2024-04-25T13:40:00.348329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:59.240536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494618+0000","last_change":"2024-04-25T13:38:44.345979+0000","last_active":"2024-04-25T13:39:31.494618+0000","last_peered":"2024-04-25T13:39:31.494618+0000","last_clean":"2024-04-25T13:39:31.494618+0000","last_became_active":"2024-04-25T13:38:44.345596+0000","last_became_peered":"2024-04-25T13:38:44.345596+0000","last_unstale":"2024-04-25T13:39:31.494618+0000","last_undegraded":"2024-04-25T13:39:31.494618+0000","last_fullsized":"2024-04-25T13:39:31.494618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:22:51.539024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367445+0000","last_change":"2024-04-25T13:38:45.364794+0000","last_active":"2024-04-25T13:40:00.367445+0000","last_peered":"2024-04-25T13:40:00.367445+0000","last_clean":"2024-04-25T13:40:00.367445+0000","last_became_active":"2024-04-25T13:38:45.364381+0000","last_became_peered":"2024-04-25T13:38:45.364381+0000","last_unstale":"2024-04-25T13:40:00.367445+0000","last_undegraded":"2024-04-25T13:40:00.367445+0000","last_fullsized":"2024-04-25T13:40:00.367445+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:16.856980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485286+0000","last_change":"2024-04-25T13:38:44.351269+0000","last_active":"2024-04-25T13:39:31.485286+0000","last_peered":"2024-04-25T13:39:31.485286+0000","last_clean":"2024-04-25T13:39:31.485286+0000","last_became_active":"2024-04-25T13:38:44.350933+0000","last_became_peered":"2024-04-25T13:38:44.350933+0000","last_unstale":"2024-04-25T13:39:31.485286+0000","last_undegraded":"2024-04-25T13:39:31.485286+0000","last_fullsized":"2024-04-25T13:39:31.485286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:02.677849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.372710+0000","last_change":"2024-04-25T13:38:45.362732+0000","last_active":"2024-04-25T13:40:00.372710+0000","last_peered":"2024-04-25T13:40:00.372710+0000","last_clean":"2024-04-25T13:40:00.372710+0000","last_became_active":"2024-04-25T13:38:45.361957+0000","last_became_peered":"2024-04-25T13:38:45.361957+0000","last_unstale":"2024-04-25T13:40:00.372710+0000","last_undegraded":"2024-04-25T13:40:00.372710+0000","last_fullsized":"2024-04-25T13:40:00.372710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:50.011442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.776862+0000","last_change":"2024-04-25T13:38:44.349378+0000","last_active":"2024-04-25T13:39:31.776862+0000","last_peered":"2024-04-25T13:39:31.776862+0000","last_clean":"2024-04-25T13:39:31.776862+0000","last_became_active":"2024-04-25T13:38:44.348887+0000","last_became_peered":"2024-04-25T13:38:44.348887+0000","last_unstale":"2024-04-25T13:39:31.776862+0000","last_undegraded":"2024-04-25T13:39:31.776862+0000","last_fullsized":"2024-04-25T13:39:31.776862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.747205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375453+0000","last_change":"2024-04-25T13:38:45.349941+0000","last_active":"2024-04-25T13:40:00.375453+0000","last_peered":"2024-04-25T13:40:00.375453+0000","last_clean":"2024-04-25T13:40:00.375453+0000","last_became_active":"2024-04-25T13:38:45.349590+0000","last_became_peered":"2024-04-25T13:38:45.349590+0000","last_unstale":"2024-04-25T13:40:00.375453+0000","last_undegraded":"2024-04-25T13:40:00.375453+0000","last_fullsized":"2024-04-25T13:40:00.375453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:42.904406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.791483+0000","last_change":"2024-04-25T13:39:31.790997+0000","last_active":"2024-04-25T13:39:31.791483+0000","last_peered":"2024-04-25T13:39:31.791483+0000","last_clean":"2024-04-25T13:39:31.791483+0000","last_became_active":"2024-04-25T13:39:31.778451+0000","last_became_peered":"2024-04-25T13:39:31.778451+0000","last_unstale":"2024-04-25T13:39:31.791483+0000","last_undegraded":"2024-04-25T13:39:31.791483+0000","last_fullsized":"2024-04-25T13:39:31.791483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:17.735970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368028+0000","last_change":"2024-04-25T13:38:45.368589+0000","last_active":"2024-04-25T13:40:00.368028+0000","last_peered":"2024-04-25T13:40:00.368028+0000","last_clean":"2024-04-25T13:40:00.368028+0000","last_became_active":"2024-04-25T13:38:45.368079+0000","last_became_peered":"2024-04-25T13:38:45.368079+0000","last_unstale":"2024-04-25T13:40:00.368028+0000","last_undegraded":"2024-04-25T13:40:00.368028+0000","last_fullsized":"2024-04-25T13:40:00.368028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:10.180264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491662+0000","last_change":"2024-04-25T13:38:44.345148+0000","last_active":"2024-04-25T13:39:31.491662+0000","last_peered":"2024-04-25T13:39:31.491662+0000","last_clean":"2024-04-25T13:39:31.491662+0000","last_became_active":"2024-04-25T13:38:44.344816+0000","last_became_peered":"2024-04-25T13:38:44.344816+0000","last_unstale":"2024-04-25T13:39:31.491662+0000","last_undegraded":"2024-04-25T13:39:31.491662+0000","last_fullsized":"2024-04-25T13:39:31.491662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:03:38.134169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351253+0000","last_change":"2024-04-25T13:38:45.354340+0000","last_active":"2024-04-25T13:40:00.351253+0000","last_peered":"2024-04-25T13:40:00.351253+0000","last_clean":"2024-04-25T13:40:00.351253+0000","last_became_active":"2024-04-25T13:38:45.353933+0000","last_became_peered":"2024-04-25T13:38:45.353933+0000","last_unstale":"2024-04-25T13:40:00.351253+0000","last_undegraded":"2024-04-25T13:40:00.351253+0000","last_fullsized":"2024-04-25T13:40:00.351253+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:07.619051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487938+0000","last_change":"2024-04-25T13:38:44.351183+0000","last_active":"2024-04-25T13:39:31.487938+0000","last_peered":"2024-04-25T13:39:31.487938+0000","last_clean":"2024-04-25T13:39:31.487938+0000","last_became_active":"2024-04-25T13:38:44.350785+0000","last_became_peered":"2024-04-25T13:38:44.350785+0000","last_unstale":"2024-04-25T13:39:31.487938+0000","last_undegraded":"2024-04-25T13:39:31.487938+0000","last_fullsized":"2024-04-25T13:39:31.487938+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:37:57.465012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360705+0000","last_change":"2024-04-25T13:38:45.362355+0000","last_active":"2024-04-25T13:40:00.360705+0000","last_peered":"2024-04-25T13:40:00.360705+0000","last_clean":"2024-04-25T13:40:00.360705+0000","last_became_active":"2024-04-25T13:38:45.361996+0000","last_became_peered":"2024-04-25T13:38:45.361996+0000","last_unstale":"2024-04-25T13:40:00.360705+0000","last_undegraded":"2024-04-25T13:40:00.360705+0000","last_fullsized":"2024-04-25T13:40:00.360705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:53:24.790450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487406+0000","last_change":"2024-04-25T13:38:44.356079+0000","last_active":"2024-04-25T13:39:31.487406+0000","last_peered":"2024-04-25T13:39:31.487406+0000","last_clean":"2024-04-25T13:39:31.487406+0000","last_became_active":"2024-04-25T13:38:44.355660+0000","last_became_peered":"2024-04-25T13:38:44.355660+0000","last_unstale":"2024-04-25T13:39:31.487406+0000","last_undegraded":"2024-04-25T13:39:31.487406+0000","last_fullsized":"2024-04-25T13:39:31.487406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:35.161337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.353301+0000","last_change":"2024-04-25T13:38:45.356578+0000","last_active":"2024-04-25T13:40:00.353301+0000","last_peered":"2024-04-25T13:40:00.353301+0000","last_clean":"2024-04-25T13:40:00.353301+0000","last_became_active":"2024-04-25T13:38:45.356232+0000","last_became_peered":"2024-04-25T13:38:45.356232+0000","last_unstale":"2024-04-25T13:40:00.353301+0000","last_undegraded":"2024-04-25T13:40:00.353301+0000","last_fullsized":"2024-04-25T13:40:00.353301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.476890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"78'8","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360561+0000","last_change":"2024-04-25T13:39:31.778577+0000","last_active":"2024-04-25T13:40:00.360561+0000","last_peered":"2024-04-25T13:40:00.360561+0000","last_clean":"2024-04-25T13:40:00.360561+0000","last_became_active":"2024-04-25T13:39:31.778178+0000","last_became_peered":"2024-04-25T13:39:31.778178+0000","last_unstale":"2024-04-25T13:40:00.360561+0000","last_undegraded":"2024-04-25T13:40:00.360561+0000","last_fullsized":"2024-04-25T13:40:00.360561+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.383670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484649+0000","last_change":"2024-04-25T13:38:44.351113+0000","last_active":"2024-04-25T13:39:31.484649+0000","last_peered":"2024-04-25T13:39:31.484649+0000","last_clean":"2024-04-25T13:39:31.484649+0000","last_became_active":"2024-04-25T13:38:44.350730+0000","last_became_peered":"2024-04-25T13:38:44.350730+0000","last_unstale":"2024-04-25T13:39:31.484649+0000","last_undegraded":"2024-04-25T13:39:31.484649+0000","last_fullsized":"2024-04-25T13:39:31.484649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:38:22.747930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362720+0000","last_change":"2024-04-25T13:38:45.356731+0000","last_active":"2024-04-25T13:40:00.362720+0000","last_peered":"2024-04-25T13:40:00.362720+0000","last_clean":"2024-04-25T13:40:00.362720+0000","last_became_active":"2024-04-25T13:38:45.355531+0000","last_became_peered":"2024-04-25T13:38:45.355531+0000","last_unstale":"2024-04-25T13:40:00.362720+0000","last_undegraded":"2024-04-25T13:40:00.362720+0000","last_fullsized":"2024-04-25T13:40:00.362720+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:15.482469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777207+0000","last_change":"2024-04-25T13:38:44.345941+0000","last_active":"2024-04-25T13:39:31.777207+0000","last_peered":"2024-04-25T13:39:31.777207+0000","last_clean":"2024-04-25T13:39:31.777207+0000","last_became_active":"2024-04-25T13:38:44.344634+0000","last_became_peered":"2024-04-25T13:38:44.344634+0000","last_unstale":"2024-04-25T13:39:31.777207+0000","last_undegraded":"2024-04-25T13:39:31.777207+0000","last_fullsized":"2024-04-25T13:39:31.777207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:42.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":93,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487278+0000","last_change":"2024-04-25T13:37:27.903345+0000","last_active":"2024-04-25T13:39:31.487278+0000","last_peered":"2024-04-25T13:39:31.487278+0000","last_clean":"2024-04-25T13:39:31.487278+0000","last_became_active":"2024-04-25T13:37:27.897986+0000","last_became_peered":"2024-04-25T13:37:27.897986+0000","last_unstale":"2024-04-25T13:39:31.487278+0000","last_undegraded":"2024-04-25T13:39:31.487278+0000","last_fullsized":"2024-04-25T13:39:31.487278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_clean_scrub_stamp":"2024-04-25T13:33:56.569417+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:56.390458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360997+0000","last_change":"2024-04-25T13:38:45.369390+0000","last_active":"2024-04-25T13:40:00.360997+0000","last_peered":"2024-04-25T13:40:00.360997+0000","last_clean":"2024-04-25T13:40:00.360997+0000","last_became_active":"2024-04-25T13:38:45.368938+0000","last_became_peered":"2024-04-25T13:38:45.368938+0000","last_unstale":"2024-04-25T13:40:00.360997+0000","last_undegraded":"2024-04-25T13:40:00.360997+0000","last_fullsized":"2024-04-25T13:40:00.360997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:11.942204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487420+0000","last_change":"2024-04-25T13:38:44.355585+0000","last_active":"2024-04-25T13:39:31.487420+0000","last_peered":"2024-04-25T13:39:31.487420+0000","last_clean":"2024-04-25T13:39:31.487420+0000","last_became_active":"2024-04-25T13:38:44.355131+0000","last_became_peered":"2024-04-25T13:38:44.355131+0000","last_unstale":"2024-04-25T13:39:31.487420+0000","last_undegraded":"2024-04-25T13:39:31.487420+0000","last_fullsized":"2024-04-25T13:39:31.487420+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:05.240247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492418+0000","last_change":"2024-04-25T13:39:31.491742+0000","last_active":"2024-04-25T13:39:31.492418+0000","last_peered":"2024-04-25T13:39:31.492418+0000","last_clean":"2024-04-25T13:39:31.492418+0000","last_became_active":"2024-04-25T13:39:31.491266+0000","last_became_peered":"2024-04-25T13:39:31.491266+0000","last_unstale":"2024-04-25T13:39:31.492418+0000","last_undegraded":"2024-04-25T13:39:31.492418+0000","last_fullsized":"2024-04-25T13:39:31.492418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:21.932333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.355121+0000","last_change":"2024-04-25T13:38:45.366187+0000","last_active":"2024-04-25T13:40:00.355121+0000","last_peered":"2024-04-25T13:40:00.355121+0000","last_clean":"2024-04-25T13:40:00.355121+0000","last_became_active":"2024-04-25T13:38:45.365744+0000","last_became_peered":"2024-04-25T13:38:45.365744+0000","last_unstale":"2024-04-25T13:40:00.355121+0000","last_undegraded":"2024-04-25T13:40:00.355121+0000","last_fullsized":"2024-04-25T13:40:00.355121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:07.970735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"78'7","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487865+0000","last_change":"2024-04-25T13:38:44.359250+0000","last_active":"2024-04-25T13:39:31.487865+0000","last_peered":"2024-04-25T13:39:31.487865+0000","last_clean":"2024-04-25T13:39:31.487865+0000","last_became_active":"2024-04-25T13:38:44.358874+0000","last_became_peered":"2024-04-25T13:38:44.358874+0000","last_unstale":"2024-04-25T13:39:31.487865+0000","last_undegraded":"2024-04-25T13:39:31.487865+0000","last_fullsized":"2024-04-25T13:39:31.487865+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:28.820465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366411+0000","last_change":"2024-04-25T13:38:45.370080+0000","last_active":"2024-04-25T13:40:00.366411+0000","last_peered":"2024-04-25T13:40:00.366411+0000","last_clean":"2024-04-25T13:40:00.366411+0000","last_became_active":"2024-04-25T13:38:45.369668+0000","last_became_peered":"2024-04-25T13:38:45.369668+0000","last_unstale":"2024-04-25T13:40:00.366411+0000","last_undegraded":"2024-04-25T13:40:00.366411+0000","last_fullsized":"2024-04-25T13:40:00.366411+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:03.197468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777370+0000","last_change":"2024-04-25T13:38:44.358907+0000","last_active":"2024-04-25T13:39:31.777370+0000","last_peered":"2024-04-25T13:39:31.777370+0000","last_clean":"2024-04-25T13:39:31.777370+0000","last_became_active":"2024-04-25T13:38:44.358455+0000","last_became_peered":"2024-04-25T13:38:44.358455+0000","last_unstale":"2024-04-25T13:39:31.777370+0000","last_undegraded":"2024-04-25T13:39:31.777370+0000","last_fullsized":"2024-04-25T13:39:31.777370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:50:52.916433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365642+0000","last_change":"2024-04-25T13:38:45.373050+0000","last_active":"2024-04-25T13:40:00.365642+0000","last_peered":"2024-04-25T13:40:00.365642+0000","last_clean":"2024-04-25T13:40:00.365642+0000","last_became_active":"2024-04-25T13:38:45.372621+0000","last_became_peered":"2024-04-25T13:38:45.372621+0000","last_unstale":"2024-04-25T13:40:00.365642+0000","last_undegraded":"2024-04-25T13:40:00.365642+0000","last_fullsized":"2024-04-25T13:40:00.365642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:29:43.259296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485534+0000","last_change":"2024-04-25T13:38:44.355768+0000","last_active":"2024-04-25T13:39:31.485534+0000","last_peered":"2024-04-25T13:39:31.485534+0000","last_clean":"2024-04-25T13:39:31.485534+0000","last_became_active":"2024-04-25T13:38:44.355400+0000","last_became_peered":"2024-04-25T13:38:44.355400+0000","last_unstale":"2024-04-25T13:39:31.485534+0000","last_undegraded":"2024-04-25T13:39:31.485534+0000","last_fullsized":"2024-04-25T13:39:31.485534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:38.533717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342472+0000","last_change":"2024-04-25T13:38:45.350582+0000","last_active":"2024-04-25T13:40:00.342472+0000","last_peered":"2024-04-25T13:40:00.342472+0000","last_clean":"2024-04-25T13:40:00.342472+0000","last_became_active":"2024-04-25T13:38:45.350220+0000","last_became_peered":"2024-04-25T13:38:45.350220+0000","last_unstale":"2024-04-25T13:40:00.342472+0000","last_undegraded":"2024-04-25T13:40:00.342472+0000","last_fullsized":"2024-04-25T13:40:00.342472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.463519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.489729+0000","last_change":"2024-04-25T13:38:44.360459+0000","last_active":"2024-04-25T13:39:31.489729+0000","last_peered":"2024-04-25T13:39:31.489729+0000","last_clean":"2024-04-25T13:39:31.489729+0000","last_became_active":"2024-04-25T13:38:44.359709+0000","last_became_peered":"2024-04-25T13:38:44.359709+0000","last_unstale":"2024-04-25T13:39:31.489729+0000","last_undegraded":"2024-04-25T13:39:31.489729+0000","last_fullsized":"2024-04-25T13:39:31.489729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:53.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356628+0000","last_change":"2024-04-25T13:38:45.357291+0000","last_active":"2024-04-25T13:40:00.356628+0000","last_peered":"2024-04-25T13:40:00.356628+0000","last_clean":"2024-04-25T13:40:00.356628+0000","last_became_active":"2024-04-25T13:38:45.356929+0000","last_became_peered":"2024-04-25T13:38:45.356929+0000","last_unstale":"2024-04-25T13:40:00.356628+0000","last_undegraded":"2024-04-25T13:40:00.356628+0000","last_fullsized":"2024-04-25T13:40:00.356628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:02:08.619208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.493732+0000","last_change":"2024-04-25T13:38:44.343885+0000","last_active":"2024-04-25T13:39:31.493732+0000","last_peered":"2024-04-25T13:39:31.493732+0000","last_clean":"2024-04-25T13:39:31.493732+0000","last_became_active":"2024-04-25T13:38:44.343541+0000","last_became_peered":"2024-04-25T13:38:44.343541+0000","last_unstale":"2024-04-25T13:39:31.493732+0000","last_undegraded":"2024-04-25T13:39:31.493732+0000","last_fullsized":"2024-04-25T13:39:31.493732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:50:01.904593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365216+0000","last_change":"2024-04-25T13:38:45.371508+0000","last_active":"2024-04-25T13:40:00.365216+0000","last_peered":"2024-04-25T13:40:00.365216+0000","last_clean":"2024-04-25T13:40:00.365216+0000","last_became_active":"2024-04-25T13:38:45.370912+0000","last_became_peered":"2024-04-25T13:38:45.370912+0000","last_unstale":"2024-04-25T13:40:00.365216+0000","last_undegraded":"2024-04-25T13:40:00.365216+0000","last_fullsized":"2024-04-25T13:40:00.365216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:07.075511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'2","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.498142+0000","last_change":"2024-04-25T13:39:31.498142+0000","last_active":"2024-04-25T13:39:31.498142+0000","last_peered":"2024-04-25T13:39:31.498142+0000","last_clean":"2024-04-25T13:39:31.498142+0000","last_became_active":"2024-04-25T13:39:31.494646+0000","last_became_peered":"2024-04-25T13:39:31.494646+0000","last_unstale":"2024-04-25T13:39:31.498142+0000","last_undegraded":"2024-04-25T13:39:31.498142+0000","last_fullsized":"2024-04-25T13:39:31.498142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:46.084354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375168+0000","last_change":"2024-04-25T13:38:45.363322+0000","last_active":"2024-04-25T13:40:00.375168+0000","last_peered":"2024-04-25T13:40:00.375168+0000","last_clean":"2024-04-25T13:40:00.375168+0000","last_became_active":"2024-04-25T13:38:45.362948+0000","last_became_peered":"2024-04-25T13:38:45.362948+0000","last_unstale":"2024-04-25T13:40:00.375168+0000","last_undegraded":"2024-04-25T13:40:00.375168+0000","last_fullsized":"2024-04-25T13:40:00.375168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:20.815774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494630+0000","last_change":"2024-04-25T13:38:44.358588+0000","last_active":"2024-04-25T13:39:31.494630+0000","last_peered":"2024-04-25T13:39:31.494630+0000","last_clean":"2024-04-25T13:39:31.494630+0000","last_became_active":"2024-04-25T13:38:44.358269+0000","last_became_peered":"2024-04-25T13:38:44.358269+0000","last_unstale":"2024-04-25T13:39:31.494630+0000","last_undegraded":"2024-04-25T13:39:31.494630+0000","last_fullsized":"2024-04-25T13:39:31.494630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:35.405572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375021+0000","last_change":"2024-04-25T13:38:45.369272+0000","last_active":"2024-04-25T13:40:00.375021+0000","last_peered":"2024-04-25T13:40:00.375021+0000","last_clean":"2024-04-25T13:40:00.375021+0000","last_became_active":"2024-04-25T13:38:45.368774+0000","last_became_peered":"2024-04-25T13:38:45.368774+0000","last_unstale":"2024-04-25T13:40:00.375021+0000","last_undegraded":"2024-04-25T13:40:00.375021+0000","last_fullsized":"2024-04-25T13:40:00.375021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:39.149557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485416+0000","last_change":"2024-04-25T13:38:44.349479+0000","last_active":"2024-04-25T13:39:31.485416+0000","last_peered":"2024-04-25T13:39:31.485416+0000","last_clean":"2024-04-25T13:39:31.485416+0000","last_became_active":"2024-04-25T13:38:44.349104+0000","last_became_peered":"2024-04-25T13:38:44.349104+0000","last_unstale":"2024-04-25T13:39:31.485416+0000","last_undegraded":"2024-04-25T13:39:31.485416+0000","last_fullsized":"2024-04-25T13:39:31.485416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:39.257472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"78'7","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375501+0000","last_change":"2024-04-25T13:39:31.495349+0000","last_active":"2024-04-25T13:40:00.375501+0000","last_peered":"2024-04-25T13:40:00.375501+0000","last_clean":"2024-04-25T13:40:00.375501+0000","last_became_active":"2024-04-25T13:39:31.490741+0000","last_became_peered":"2024-04-25T13:39:31.490741+0000","last_unstale":"2024-04-25T13:40:00.375501+0000","last_undegraded":"2024-04-25T13:40:00.375501+0000","last_fullsized":"2024-04-25T13:40:00.375501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.474725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484923+0000","last_change":"2024-04-25T13:38:44.349502+0000","last_active":"2024-04-25T13:39:31.484923+0000","last_peered":"2024-04-25T13:39:31.484923+0000","last_clean":"2024-04-25T13:39:31.484923+0000","last_became_active":"2024-04-25T13:38:44.349127+0000","last_became_peered":"2024-04-25T13:38:44.349127+0000","last_unstale":"2024-04-25T13:39:31.484923+0000","last_undegraded":"2024-04-25T13:39:31.484923+0000","last_fullsized":"2024-04-25T13:39:31.484923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:34:50.673352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.371319+0000","last_change":"2024-04-25T13:38:45.364746+0000","last_active":"2024-04-25T13:40:00.371319+0000","last_peered":"2024-04-25T13:40:00.371319+0000","last_clean":"2024-04-25T13:40:00.371319+0000","last_became_active":"2024-04-25T13:38:45.364364+0000","last_became_peered":"2024-04-25T13:38:45.364364+0000","last_unstale":"2024-04-25T13:40:00.371319+0000","last_undegraded":"2024-04-25T13:40:00.371319+0000","last_fullsized":"2024-04-25T13:40:00.371319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:42.145637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357194+0000","last_change":"2024-04-25T13:38:45.356683+0000","last_active":"2024-04-25T13:40:00.357194+0000","last_peered":"2024-04-25T13:40:00.357194+0000","last_clean":"2024-04-25T13:40:00.357194+0000","last_became_active":"2024-04-25T13:38:45.356048+0000","last_became_peered":"2024-04-25T13:38:45.356048+0000","last_unstale":"2024-04-25T13:40:00.357194+0000","last_undegraded":"2024-04-25T13:40:00.357194+0000","last_fullsized":"2024-04-25T13:40:00.357194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:49.578317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"78'12","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362649+0000","last_change":"2024-04-25T13:38:45.358298+0000","last_active":"2024-04-25T13:40:00.362649+0000","last_peered":"2024-04-25T13:40:00.362649+0000","last_clean":"2024-04-25T13:40:00.362649+0000","last_became_active":"2024-04-25T13:38:45.357986+0000","last_became_peered":"2024-04-25T13:38:45.357986+0000","last_unstale":"2024-04-25T13:40:00.362649+0000","last_undegraded":"2024-04-25T13:40:00.362649+0000","last_fullsized":"2024-04-25T13:40:00.362649+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:33.093156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.355496+0000","last_change":"2024-04-25T13:38:45.363043+0000","last_active":"2024-04-25T13:40:00.355496+0000","last_peered":"2024-04-25T13:40:00.355496+0000","last_clean":"2024-04-25T13:40:00.355496+0000","last_became_active":"2024-04-25T13:38:45.362686+0000","last_became_peered":"2024-04-25T13:38:45.362686+0000","last_unstale":"2024-04-25T13:40:00.355496+0000","last_undegraded":"2024-04-25T13:40:00.355496+0000","last_fullsized":"2024-04-25T13:40:00.355496+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:56.602134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362067+0000","last_change":"2024-04-25T13:38:45.373256+0000","last_active":"2024-04-25T13:40:00.362067+0000","last_peered":"2024-04-25T13:40:00.362067+0000","last_clean":"2024-04-25T13:40:00.362067+0000","last_became_active":"2024-04-25T13:38:45.372844+0000","last_became_peered":"2024-04-25T13:38:45.372844+0000","last_unstale":"2024-04-25T13:40:00.362067+0000","last_undegraded":"2024-04-25T13:40:00.362067+0000","last_fullsized":"2024-04-25T13:40:00.362067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:39:34.590536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.364083+0000","last_change":"2024-04-25T13:38:45.370007+0000","last_active":"2024-04-25T13:40:00.364083+0000","last_peered":"2024-04-25T13:40:00.364083+0000","last_clean":"2024-04-25T13:40:00.364083+0000","last_became_active":"2024-04-25T13:38:45.369606+0000","last_became_peered":"2024-04-25T13:38:45.369606+0000","last_unstale":"2024-04-25T13:40:00.364083+0000","last_undegraded":"2024-04-25T13:40:00.364083+0000","last_fullsized":"2024-04-25T13:40:00.364083+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:45.514756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.350193+0000","last_change":"2024-04-25T13:38:45.362075+0000","last_active":"2024-04-25T13:40:00.350193+0000","last_peered":"2024-04-25T13:40:00.350193+0000","last_clean":"2024-04-25T13:40:00.350193+0000","last_became_active":"2024-04-25T13:38:45.361720+0000","last_became_peered":"2024-04-25T13:38:45.361720+0000","last_unstale":"2024-04-25T13:40:00.350193+0000","last_undegraded":"2024-04-25T13:40:00.350193+0000","last_fullsized":"2024-04-25T13:40:00.350193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:36.169185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354853+0000","last_change":"2024-04-25T13:38:45.352101+0000","last_active":"2024-04-25T13:40:00.354853+0000","last_peered":"2024-04-25T13:40:00.354853+0000","last_clean":"2024-04-25T13:40:00.354853+0000","last_became_active":"2024-04-25T13:38:45.351739+0000","last_became_peered":"2024-04-25T13:38:45.351739+0000","last_unstale":"2024-04-25T13:40:00.354853+0000","last_undegraded":"2024-04-25T13:40:00.354853+0000","last_fullsized":"2024-04-25T13:40:00.354853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:05.612527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362759+0000","last_change":"2024-04-25T13:38:45.363855+0000","last_active":"2024-04-25T13:40:00.362759+0000","last_peered":"2024-04-25T13:40:00.362759+0000","last_clean":"2024-04-25T13:40:00.362759+0000","last_became_active":"2024-04-25T13:38:45.363377+0000","last_became_peered":"2024-04-25T13:38:45.363377+0000","last_unstale":"2024-04-25T13:40:00.362759+0000","last_undegraded":"2024-04-25T13:40:00.362759+0000","last_fullsized":"2024-04-25T13:40:00.362759+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:26.828215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365565+0000","last_change":"2024-04-25T13:38:45.359844+0000","last_active":"2024-04-25T13:40:00.365565+0000","last_peered":"2024-04-25T13:40:00.365565+0000","last_clean":"2024-04-25T13:40:00.365565+0000","last_became_active":"2024-04-25T13:38:45.359428+0000","last_became_peered":"2024-04-25T13:38:45.359428+0000","last_unstale":"2024-04-25T13:40:00.365565+0000","last_undegraded":"2024-04-25T13:40:00.365565+0000","last_fullsized":"2024-04-25T13:40:00.365565+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:01.370409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.359173+0000","last_change":"2024-04-25T13:38:45.370415+0000","last_active":"2024-04-25T13:40:00.359173+0000","last_peered":"2024-04-25T13:40:00.359173+0000","last_clean":"2024-04-25T13:40:00.359173+0000","last_became_active":"2024-04-25T13:38:45.369525+0000","last_became_peered":"2024-04-25T13:38:45.369525+0000","last_unstale":"2024-04-25T13:40:00.359173+0000","last_undegraded":"2024-04-25T13:40:00.359173+0000","last_fullsized":"2024-04-25T13:40:00.359173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:39.532380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.376082+0000","last_change":"2024-04-25T13:38:45.365309+0000","last_active":"2024-04-25T13:40:00.376082+0000","last_peered":"2024-04-25T13:40:00.376082+0000","last_clean":"2024-04-25T13:40:00.376082+0000","last_became_active":"2024-04-25T13:38:45.364893+0000","last_became_peered":"2024-04-25T13:38:45.364893+0000","last_unstale":"2024-04-25T13:40:00.376082+0000","last_undegraded":"2024-04-25T13:40:00.376082+0000","last_fullsized":"2024-04-25T13:40:00.376082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:04:54.454443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368064+0000","last_change":"2024-04-25T13:38:45.365150+0000","last_active":"2024-04-25T13:40:00.368064+0000","last_peered":"2024-04-25T13:40:00.368064+0000","last_clean":"2024-04-25T13:40:00.368064+0000","last_became_active":"2024-04-25T13:38:45.364649+0000","last_became_peered":"2024-04-25T13:38:45.364649+0000","last_unstale":"2024-04-25T13:40:00.368064+0000","last_undegraded":"2024-04-25T13:40:00.368064+0000","last_fullsized":"2024-04-25T13:40:00.368064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:25.035789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354459+0000","last_change":"2024-04-25T13:38:45.352108+0000","last_active":"2024-04-25T13:40:00.354459+0000","last_peered":"2024-04-25T13:40:00.354459+0000","last_clean":"2024-04-25T13:40:00.354459+0000","last_became_active":"2024-04-25T13:38:45.349887+0000","last_became_peered":"2024-04-25T13:38:45.349887+0000","last_unstale":"2024-04-25T13:40:00.354459+0000","last_undegraded":"2024-04-25T13:40:00.354459+0000","last_fullsized":"2024-04-25T13:40:00.354459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:09.339685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285228,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62093,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":170,"num_read_kb":147,"num_write":80,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":186279,"data_compressed":5526,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678046,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":403844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874275,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":1055818,"data_compressed":38103,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037801,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234031,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430259,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":888832,"data_stored":994911,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593785,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":465139,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":403934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":465173,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":843776,"data_stored":404065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953540,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790014,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":994511,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61295,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61329,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61489,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T13:40:10.561 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T13:40:10.561 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T13:40:10.562 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T13:40:10.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T13:40:10.816 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:11.211 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:10 smithi027 ceph-mon[29140]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:11.229 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:10 smithi045 ceph-mon[32731]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:11.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:10 smithi122 ceph-mon[32843]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:12.007 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:11 smithi027 ceph-mon[29140]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:12.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:11 smithi122 ceph-mon[32843]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:12.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:11 smithi045 ceph-mon[32731]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:12.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 -- 172.21.15.27:0/1563395387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908103b70 msgr2=0x7f2908103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:12.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 --2- 172.21.15.27:0/1563395387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908103b70 0x7f2908103f80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f28f80099c0 tx=0x7f28f8009cd0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 -- 172.21.15.27:0/1563395387 shutdown_connections 2024-04-25T13:40:12.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 --2- 172.21.15.27:0/1563395387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2908108630 0x7f2908110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 --2- 172.21.15.27:0/1563395387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908103b70 0x7f2908103f80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 --2- 172.21.15.27:0/1563395387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2908107ce0 0x7f29081080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 -- 172.21.15.27:0/1563395387 >> 172.21.15.27:0/1563395387 conn(0x7f29080fe5c0 msgr2=0x7f29081009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:12.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 -- 172.21.15.27:0/1563395387 shutdown_connections 2024-04-25T13:40:12.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.646+0000 7f290ea39700 1 -- 172.21.15.27:0/1563395387 wait complete. 2024-04-25T13:40:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290ea39700 1 Processor -- start 2024-04-25T13:40:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290ea39700 1 -- start start 2024-04-25T13:40:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290ea39700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2908103b70 0x7f290819bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290ea39700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2908107ce0 0x7f290819c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290ea39700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908108630 0x7f290819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290ea39700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2908113250 con 0x7f2908108630 2024-04-25T13:40:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290ea39700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f29081130d0 con 0x7f2908103b70 2024-04-25T13:40:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290ea39700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f29081133d0 con 0x7f2908107ce0 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290cfd6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908108630 0x7f290819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290cfd6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908108630 0x7f290819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39346/0 (socket says 172.21.15.27:39346) 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f290cfd6700 1 -- 172.21.15.27:0/3986111598 learned_addr learned my addr 172.21.15.27:0/3986111598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f29077fe700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2908107ce0 0x7f290819c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.647+0000 7f2907fff700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2908103b70 0x7f290819bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f290cfd6700 1 -- 172.21.15.27:0/3986111598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2908107ce0 msgr2=0x7f290819c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f290cfd6700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2908107ce0 0x7f290819c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f290cfd6700 1 -- 172.21.15.27:0/3986111598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2908103b70 msgr2=0x7f290819bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f290cfd6700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2908103b70 0x7f290819bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f290cfd6700 1 -- 172.21.15.27:0/3986111598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29081a6e50 con 0x7f2908108630 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f290cfd6700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908108630 0x7f290819c630 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2900007d70 tx=0x7f290000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f29057fa700 1 -- 172.21.15.27:0/3986111598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900016020 con 0x7f2908108630 2024-04-25T13:40:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29081a7140 con 0x7f2908108630 2024-04-25T13:40:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f29057fa700 1 -- 172.21.15.27:0/3986111598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f290000ac90 con 0x7f2908108630 2024-04-25T13:40:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f29057fa700 1 -- 172.21.15.27:0/3986111598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29000115e0 con 0x7f2908108630 2024-04-25T13:40:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.648+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29081a7680 con 0x7f2908108630 2024-04-25T13:40:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.649+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2908051190 con 0x7f2908108630 2024-04-25T13:40:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.651+0000 7f29057fa700 1 -- 172.21.15.27:0/3986111598 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2900019020 con 0x7f2908108630 2024-04-25T13:40:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.652+0000 7f29057fa700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f28f00afb80 0x7f28f00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.652+0000 7f2907fff700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f28f00afb80 0x7f28f00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.652+0000 7f29057fa700 1 -- 172.21.15.27:0/3986111598 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f29000def70 con 0x7f2908108630 2024-04-25T13:40:12.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.653+0000 7f2907fff700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f28f00afb80 0x7f28f00b2090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f28f8003760 tx=0x7f28f801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.656+0000 7f29057fa700 1 -- 172.21.15.27:0/3986111598 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2900020020 con 0x7f2908108630 2024-04-25T13:40:12.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.908+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f290802e510 con 0x7f2908108630 2024-04-25T13:40:12.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.909+0000 7f29057fa700 1 -- 172.21.15.27:0/3986111598 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f29000a7b80 con 0x7f2908108630 2024-04-25T13:40:12.909 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:12.909 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f28f00afb80 msgr2=0x7f28f00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f28f00afb80 0x7f28f00b2090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f28f8003760 tx=0x7f28f801a040 comp rx=0 tx=0).stop 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908108630 msgr2=0x7f290819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908108630 0x7f290819c630 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2900007d70 tx=0x7f290000a340 comp rx=0 tx=0).stop 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 shutdown_connections 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f28f00afb80 0x7f28f00b2090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2908108630 0x7f290819c630 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2908103b70 0x7f290819bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 --2- 172.21.15.27:0/3986111598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2908107ce0 0x7f290819c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 >> 172.21.15.27:0/3986111598 conn(0x7f29080fe5c0 msgr2=0x7f2908107450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:12.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 shutdown_connections 2024-04-25T13:40:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:12.911+0000 7f290ea39700 1 -- 172.21.15.27:0/3986111598 wait complete. 2024-04-25T13:40:13.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:12 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3986111598' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T13:40:13.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:12 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3986111598' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T13:40:13.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:12 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3986111598' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T13:40:13.784 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T13:40:13.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:13.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/3290253877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd4108920 msgr2=0x7fcbd410ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:13.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/3290253877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd4108920 0x7fcbd410ad40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009ca0 tx=0x7fcbcc02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:13.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/3290253877 shutdown_connections 2024-04-25T13:40:13.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/3290253877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd4108920 0x7fcbd410ad40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/3290253877 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbd4103c20 0x7fcbd4104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/3290253877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbd41029e0 0x7fcbd41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/3290253877 >> 172.21.15.27:0/3290253877 conn(0x7fcbd40fdbd0 msgr2=0x7fcbd4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/3290253877 shutdown_connections 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.907+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/3290253877 wait complete. 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbdc2f2640 1 Processor -- start 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbdc2f2640 1 -- start start 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbdc2f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd41029e0 0x7fcbd419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbdc2f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbd4103c20 0x7fcbd419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbdc2f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbd4108920 0x7fcbd41a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbdc2f2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd410e4f0 con 0x7fcbd41029e0 2024-04-25T13:40:13.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbdc2f2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd410e370 con 0x7fcbd4103c20 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbdc2f2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd410e670 con 0x7fcbd4108920 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd41029e0 0x7fcbd419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd41029e0 0x7fcbd419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38226/0 (socket says 172.21.15.27:38226) 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 -- 172.21.15.27:0/2696311140 learned_addr learned my addr 172.21.15.27:0/2696311140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbd988d640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbd4103c20 0x7fcbd419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda88f640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbd4108920 0x7fcbd41a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 -- 172.21.15.27:0/2696311140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbd4108920 msgr2=0x7fcbd41a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbd4108920 0x7fcbd41a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 -- 172.21.15.27:0/2696311140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbd4103c20 msgr2=0x7fcbd419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbd4103c20 0x7fcbd419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 -- 172.21.15.27:0/2696311140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd41a1d50 con 0x7fcbd41029e0 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda88f640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbd4108920 0x7fcbd41a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbd988d640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbd4103c20 0x7fcbd419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:13.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.908+0000 7fcbda08e640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd41029e0 0x7fcbd419a210 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000d740 tx=0x7fcbd000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:13.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.909+0000 7fcbcb7fe640 1 -- 172.21.15.27:0/2696311140 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0017070 con 0x7fcbd41029e0 2024-04-25T13:40:13.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.909+0000 7fcbcb7fe640 1 -- 172.21.15.27:0/2696311140 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd000de50 con 0x7fcbd41029e0 2024-04-25T13:40:13.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.909+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd41a1fe0 con 0x7fcbd41029e0 2024-04-25T13:40:13.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.909+0000 7fcbcb7fe640 1 -- 172.21.15.27:0/2696311140 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0003c20 con 0x7fcbd41029e0 2024-04-25T13:40:13.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.909+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd41a24a0 con 0x7fcbd41029e0 2024-04-25T13:40:13.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.910+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd41a2700 con 0x7fcbd41029e0 2024-04-25T13:40:13.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.912+0000 7fcbcb7fe640 1 -- 172.21.15.27:0/2696311140 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcbd0003dc0 con 0x7fcbd41029e0 2024-04-25T13:40:13.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.912+0000 7fcbcb7fe640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcbc00afaf0 0x7fcbc00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:13.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.913+0000 7fcbd988d640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcbc00afaf0 0x7fcbc00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:13.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.913+0000 7fcbcb7fe640 1 -- 172.21.15.27:0/2696311140 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcbd00deb00 con 0x7fcbd41029e0 2024-04-25T13:40:13.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.913+0000 7fcbd988d640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcbc00afaf0 0x7fcbc00b2000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4005fd0 tx=0x7fcbc4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:13.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:13.916+0000 7fcbcb7fe640 1 -- 172.21.15.27:0/2696311140 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbd00a76b0 con 0x7fcbd41029e0 2024-04-25T13:40:14.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.104+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbd41a2970 con 0x7fcbd41029e0 2024-04-25T13:40:14.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.106+0000 7fcbcb7fe640 1 -- 172.21.15.27:0/2696311140 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7fcbd00a6fb0 con 0x7fcbd41029e0 2024-04-25T13:40:14.107 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:14.107 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcbc00afaf0 msgr2=0x7fcbc00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcbc00afaf0 0x7fcbc00b2000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4005fd0 tx=0x7fcbc4005850 comp rx=0 tx=0).stop 2024-04-25T13:40:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd41029e0 msgr2=0x7fcbd419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd41029e0 0x7fcbd419a210 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000d740 tx=0x7fcbd000dc00 comp rx=0 tx=0).stop 2024-04-25T13:40:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 shutdown_connections 2024-04-25T13:40:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcbc00afaf0 0x7fcbc00b2000 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcbd4108920 0x7fcbd41a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbd4103c20 0x7fcbd419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 --2- 172.21.15.27:0/2696311140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbd41029e0 0x7fcbd419a210 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 >> 172.21.15.27:0/2696311140 conn(0x7fcbd40fdbd0 msgr2=0x7fcbd4109e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:14.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 shutdown_connections 2024-04-25T13:40:14.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.108+0000 7fcbdc2f2640 1 -- 172.21.15.27:0/2696311140 wait complete. 2024-04-25T13:40:14.118 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:13 smithi027 ceph-mon[29140]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:14.120 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:40:14.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:40:14.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.236+0000 7f11a118f640 1 -- 172.21.15.27:0/1379159331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c108920 msgr2=0x7f119c10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.236+0000 7f11a118f640 1 --2- 172.21.15.27:0/1379159331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c108920 0x7f119c10ad40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1194009ca0 tx=0x7f119402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.237+0000 7f11a118f640 1 -- 172.21.15.27:0/1379159331 shutdown_connections 2024-04-25T13:40:14.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.237+0000 7f11a118f640 1 --2- 172.21.15.27:0/1379159331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c108920 0x7f119c10ad40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.237+0000 7f11a118f640 1 --2- 172.21.15.27:0/1379159331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f119c103c20 0x7f119c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.237+0000 7f11a118f640 1 --2- 172.21.15.27:0/1379159331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c1029e0 0x7f119c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.237+0000 7f11a118f640 1 -- 172.21.15.27:0/1379159331 >> 172.21.15.27:0/1379159331 conn(0x7f119c0fdbd0 msgr2=0x7f119c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:14.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.237+0000 7f11a118f640 1 -- 172.21.15.27:0/1379159331 shutdown_connections 2024-04-25T13:40:14.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.237+0000 7f11a118f640 1 -- 172.21.15.27:0/1379159331 wait complete. 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.237+0000 7f11a118f640 1 Processor -- start 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f11a118f640 1 -- start start 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f11a118f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c1029e0 0x7f119c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f11a118f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c103c20 0x7f119c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f11a118f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f119c108920 0x7f119c1a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f11a118f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10f610 con 0x7f119c1029e0 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f11a118f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10f490 con 0x7f119c108920 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f11a118f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10f790 con 0x7f119c103c20 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c1029e0 0x7f119c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c1029e0 0x7f119c19a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38242/0 (socket says 172.21.15.27:38242) 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119ad9d640 1 -- 172.21.15.27:0/2859231047 learned_addr learned my addr 172.21.15.27:0/2859231047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119a59c640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c103c20 0x7f119c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119b59e640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f119c108920 0x7f119c1a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119ad9d640 1 -- 172.21.15.27:0/2859231047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c103c20 msgr2=0x7f119c19aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119ad9d640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c103c20 0x7f119c19aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119ad9d640 1 -- 172.21.15.27:0/2859231047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f119c108920 msgr2=0x7f119c1a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119ad9d640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f119c108920 0x7f119c1a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119ad9d640 1 -- 172.21.15.27:0/2859231047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c1a2000 con 0x7f119c1029e0 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.238+0000 7f119a59c640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c103c20 0x7f119c19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.239+0000 7f119ad9d640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c1029e0 0x7f119c19a4c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1190005fd0 tx=0x7f119000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.239+0000 7f1183fff640 1 -- 172.21.15.27:0/2859231047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1190017070 con 0x7f119c1029e0 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.239+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119c1a22f0 con 0x7f119c1029e0 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.239+0000 7f1183fff640 1 -- 172.21.15.27:0/2859231047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1190009c30 con 0x7f119c1029e0 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.239+0000 7f1183fff640 1 -- 172.21.15.27:0/2859231047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1190003d10 con 0x7f119c1029e0 2024-04-25T13:40:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.239+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119c1a2890 con 0x7f119c1029e0 2024-04-25T13:40:14.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.240+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119c107730 con 0x7f119c1029e0 2024-04-25T13:40:14.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.241+0000 7f1183fff640 1 -- 172.21.15.27:0/2859231047 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1190003eb0 con 0x7f119c1029e0 2024-04-25T13:40:14.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.242+0000 7f1183fff640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11840afaf0 0x7f11840b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.242+0000 7f119a59c640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11840afaf0 0x7f11840b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.242+0000 7f1183fff640 1 -- 172.21.15.27:0/2859231047 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f11900de980 con 0x7f119c1029e0 2024-04-25T13:40:14.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.243+0000 7f119a59c640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11840afaf0 0x7f11840b2000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f118c005fd0 tx=0x7f118c00b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:14.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.247+0000 7f1183fff640 1 -- 172.21.15.27:0/2859231047 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1190020030 con 0x7f119c1029e0 2024-04-25T13:40:14.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:13 smithi122 ceph-mon[32843]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:14.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:13 smithi045 ceph-mon[32731]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-25T13:40:14.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.451+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f119c02dd10 con 0x7f119c1029e0 2024-04-25T13:40:14.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.451+0000 7f1183fff640 1 -- 172.21.15.27:0/2859231047 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f1190009de0 con 0x7f119c1029e0 2024-04-25T13:40:14.451 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11840afaf0 msgr2=0x7f11840b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11840afaf0 0x7f11840b2000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f118c005fd0 tx=0x7f118c00b5f0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c1029e0 msgr2=0x7f119c19a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c1029e0 0x7f119c19a4c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1190005fd0 tx=0x7f119000dde0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 shutdown_connections 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11840afaf0 0x7f11840b2000 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f119c108920 0x7f119c1a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119c103c20 0x7f119c19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 --2- 172.21.15.27:0/2859231047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f119c1029e0 0x7f119c19a4c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.453+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 >> 172.21.15.27:0/2859231047 conn(0x7f119c0fdbd0 msgr2=0x7f119c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.454+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 shutdown_connections 2024-04-25T13:40:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.454+0000 7f11a118f640 1 -- 172.21.15.27:0/2859231047 wait complete. 2024-04-25T13:40:14.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 -- 172.21.15.27:0/3109265455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8104d20 msgr2=0x7fb8b8105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 --2- 172.21.15.27:0/3109265455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8104d20 0x7fb8b8105130 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009900 tx=0x7fb8ac02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 -- 172.21.15.27:0/3109265455 shutdown_connections 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 --2- 172.21.15.27:0/3109265455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8b8109a20 0x7fb8b810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 --2- 172.21.15.27:0/3109265455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8104d20 0x7fb8b8105130 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 --2- 172.21.15.27:0/3109265455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b8103ae0 0x7fb8b81047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 -- 172.21.15.27:0/3109265455 >> 172.21.15.27:0/3109265455 conn(0x7fb8b80fecd0 msgr2=0x7fb8b8101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 -- 172.21.15.27:0/3109265455 shutdown_connections 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.583+0000 7fb8be172640 1 -- 172.21.15.27:0/3109265455 wait complete. 2024-04-25T13:40:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.584+0000 7fb8be172640 1 Processor -- start 2024-04-25T13:40:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.584+0000 7fb8be172640 1 -- start start 2024-04-25T13:40:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.584+0000 7fb8be172640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b8103ae0 0x7fb8b819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.584+0000 7fb8be172640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8b8104d20 0x7fb8b819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.584+0000 7fb8be172640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8109a20 0x7fb8b81a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.584+0000 7fb8be172640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b810f610 con 0x7fb8b8109a20 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.584+0000 7fb8be172640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b810f490 con 0x7fb8b8103ae0 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.584+0000 7fb8be172640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b810f790 con 0x7fb8b8104d20 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8109a20 0x7fb8b81a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8109a20 0x7fb8b81a16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38258/0 (socket says 172.21.15.27:38258) 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 -- 172.21.15.27:0/1392311624 learned_addr learned my addr 172.21.15.27:0/1392311624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b77fe640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b8103ae0 0x7fb8b819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b6ffd640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8b8104d20 0x7fb8b819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 -- 172.21.15.27:0/1392311624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8b8104d20 msgr2=0x7fb8b819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8b8104d20 0x7fb8b819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 -- 172.21.15.27:0/1392311624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b8103ae0 msgr2=0x7fb8b819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b8103ae0 0x7fb8b819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 -- 172.21.15.27:0/1392311624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b81a1d30 con 0x7fb8b8109a20 2024-04-25T13:40:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b7fff640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8109a20 0x7fb8b81a16c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000eaa0 tx=0x7fb8b000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b4ff9640 1 -- 172.21.15.27:0/1392311624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000cdb0 con 0x7fb8b8109a20 2024-04-25T13:40:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b4ff9640 1 -- 172.21.15.27:0/1392311624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8b0005020 con 0x7fb8b8109a20 2024-04-25T13:40:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8b4ff9640 1 -- 172.21.15.27:0/1392311624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b00116d0 con 0x7fb8b8109a20 2024-04-25T13:40:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b81a1fc0 con 0x7fb8b8109a20 2024-04-25T13:40:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.585+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b81a24d0 con 0x7fb8b8109a20 2024-04-25T13:40:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.586+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b8108830 con 0x7fb8b8109a20 2024-04-25T13:40:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.588+0000 7fb8b4ff9640 1 -- 172.21.15.27:0/1392311624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb8b00051d0 con 0x7fb8b8109a20 2024-04-25T13:40:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.588+0000 7fb8b4ff9640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8a00afb70 0x7fb8a00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.588+0000 7fb8b77fe640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8a00afb70 0x7fb8a00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.588+0000 7fb8b4ff9640 1 -- 172.21.15.27:0/1392311624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb8b00defb0 con 0x7fb8b8109a20 2024-04-25T13:40:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.589+0000 7fb8b77fe640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8a00afb70 0x7fb8a00b2080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb8b81046e0 tx=0x7fb8a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:14.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.592+0000 7fb8b4ff9640 1 -- 172.21.15.27:0/1392311624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8b0016030 con 0x7fb8b8109a20 2024-04-25T13:40:14.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.778+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb8b80637a0 con 0x7fb8b8109a20 2024-04-25T13:40:14.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.779+0000 7fb8b4ff9640 1 -- 172.21.15.27:0/1392311624 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7fb8b00a75f0 con 0x7fb8b8109a20 2024-04-25T13:40:14.779 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:40:14.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8a00afb70 msgr2=0x7fb8a00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8a00afb70 0x7fb8a00b2080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb8b81046e0 tx=0x7fb8a8005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:14.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8109a20 msgr2=0x7fb8b81a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8109a20 0x7fb8b81a16c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000eaa0 tx=0x7fb8b000ef60 comp rx=0 tx=0).stop 2024-04-25T13:40:14.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 shutdown_connections 2024-04-25T13:40:14.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8a00afb70 0x7fb8a00b2080 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b8109a20 0x7fb8b81a16c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8b8104d20 0x7fb8b819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 --2- 172.21.15.27:0/1392311624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8b8103ae0 0x7fb8b819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 >> 172.21.15.27:0/1392311624 conn(0x7fb8b80fecd0 msgr2=0x7fb8b8105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 shutdown_connections 2024-04-25T13:40:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.781+0000 7fb8be172640 1 -- 172.21.15.27:0/1392311624 wait complete. 2024-04-25T13:40:14.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:40:14.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.909+0000 7f5d85215640 1 -- 172.21.15.27:0/3938631128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d80103c20 msgr2=0x7f5d80104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.909+0000 7f5d85215640 1 --2- 172.21.15.27:0/3938631128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d80103c20 0x7f5d80104030 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009900 tx=0x7f5d6802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:14.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 -- 172.21.15.27:0/3938631128 shutdown_connections 2024-04-25T13:40:14.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 --2- 172.21.15.27:0/3938631128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80108920 0x7f5d8010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 --2- 172.21.15.27:0/3938631128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d80103c20 0x7f5d80104030 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 --2- 172.21.15.27:0/3938631128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d801029e0 0x7f5d801036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 -- 172.21.15.27:0/3938631128 >> 172.21.15.27:0/3938631128 conn(0x7f5d800fdbd0 msgr2=0x7f5d80100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:14.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 -- 172.21.15.27:0/3938631128 shutdown_connections 2024-04-25T13:40:14.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 -- 172.21.15.27:0/3938631128 wait complete. 2024-04-25T13:40:14.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 Processor -- start 2024-04-25T13:40:14.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.910+0000 7f5d85215640 1 -- start start 2024-04-25T13:40:14.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d85215640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d801029e0 0x7f5d8006a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d85215640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80103c20 0x7f5d8019f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d85215640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d80108920 0x7f5d801a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d7ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d801029e0 0x7f5d8006a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d7ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d801029e0 0x7f5d8006a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:49846/0 (socket says 172.21.15.27:49846) 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d85215640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8010e4e0 con 0x7f5d80108920 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d7ed9d640 1 -- 172.21.15.27:0/409288708 learned_addr learned my addr 172.21.15.27:0/409288708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8010e360 con 0x7f5d80103c20 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8010e660 con 0x7f5d801029e0 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d7f59e640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d80108920 0x7f5d801a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.911+0000 7f5d77fff640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80103c20 0x7f5d8019f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7ed9d640 1 -- 172.21.15.27:0/409288708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80103c20 msgr2=0x7f5d8019f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7ed9d640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80103c20 0x7f5d8019f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7ed9d640 1 -- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d80108920 msgr2=0x7f5d801a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7ed9d640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d80108920 0x7f5d801a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:14.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7ed9d640 1 -- 172.21.15.27:0/409288708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d801a2060 con 0x7f5d801029e0 2024-04-25T13:40:14.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7ed9d640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d801029e0 0x7f5d8006a8f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c970 tx=0x7f5d7000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:14.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7cd99640 1 -- 172.21.15.27:0/409288708 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70004220 con 0x7f5d801029e0 2024-04-25T13:40:14.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d801a2350 con 0x7f5d801029e0 2024-04-25T13:40:14.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7cd99640 1 -- 172.21.15.27:0/409288708 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d700043c0 con 0x7f5d801029e0 2024-04-25T13:40:14.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d7cd99640 1 -- 172.21.15.27:0/409288708 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7000f450 con 0x7f5d801029e0 2024-04-25T13:40:14.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.912+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d801a28f0 con 0x7f5d801029e0 2024-04-25T13:40:14.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.913+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d80107730 con 0x7f5d801029e0 2024-04-25T13:40:14.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.915+0000 7f5d7cd99640 1 -- 172.21.15.27:0/409288708 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5d700079b0 con 0x7f5d801029e0 2024-04-25T13:40:14.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.916+0000 7f5d7cd99640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5d6c0afaf0 0x7f5d6c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:14.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.916+0000 7f5d77fff640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5d6c0afaf0 0x7f5d6c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:14.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.916+0000 7f5d7cd99640 1 -- 172.21.15.27:0/409288708 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5d700decb0 con 0x7f5d801029e0 2024-04-25T13:40:14.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.916+0000 7f5d77fff640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5d6c0afaf0 0x7f5d6c0b2000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009230 tx=0x7f5d68031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:14.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:14.919+0000 7f5d7cd99640 1 -- 172.21.15.27:0/409288708 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d70014030 con 0x7f5d801029e0 2024-04-25T13:40:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.107+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5d8006c2a0 con 0x7f5d801029e0 2024-04-25T13:40:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.108+0000 7f5d7cd99640 1 -- 172.21.15.27:0/409288708 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f5d700a7270 con 0x7f5d801029e0 2024-04-25T13:40:15.108 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:40:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5d6c0afaf0 msgr2=0x7f5d6c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5d6c0afaf0 0x7f5d6c0b2000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009230 tx=0x7f5d68031400 comp rx=0 tx=0).stop 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d801029e0 msgr2=0x7f5d8006a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d801029e0 0x7f5d8006a8f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c970 tx=0x7f5d7000ce30 comp rx=0 tx=0).stop 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 shutdown_connections 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5d6c0afaf0 0x7f5d6c0b2000 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d80108920 0x7f5d801a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d80103c20 0x7f5d8019f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 --2- 172.21.15.27:0/409288708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d801029e0 0x7f5d8006a8f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 >> 172.21.15.27:0/409288708 conn(0x7f5d800fdbd0 msgr2=0x7f5d80104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 shutdown_connections 2024-04-25T13:40:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.110+0000 7f5d85215640 1 -- 172.21.15.27:0/409288708 wait complete. 2024-04-25T13:40:15.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:14 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2696311140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:15.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:14 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2859231047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:15.121 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:14 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1392311624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:15.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.233+0000 7ff363834640 1 -- 172.21.15.27:0/4200801503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 msgr2=0x7ff35c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.233+0000 7ff363834640 1 --2- 172.21.15.27:0/4200801503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 0x7ff35c109330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009900 tx=0x7ff34c02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.233+0000 7ff363834640 1 -- 172.21.15.27:0/4200801503 shutdown_connections 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.233+0000 7ff363834640 1 --2- 172.21.15.27:0/4200801503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff35c109870 0x7ff35c10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.233+0000 7ff363834640 1 --2- 172.21.15.27:0/4200801503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 0x7ff35c109330 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.233+0000 7ff363834640 1 --2- 172.21.15.27:0/4200801503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35c1045b0 0x7ff35c1069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.234+0000 7ff363834640 1 -- 172.21.15.27:0/4200801503 >> 172.21.15.27:0/4200801503 conn(0x7ff35c0fdbd0 msgr2=0x7ff35c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.234+0000 7ff363834640 1 -- 172.21.15.27:0/4200801503 shutdown_connections 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.234+0000 7ff363834640 1 -- 172.21.15.27:0/4200801503 wait complete. 2024-04-25T13:40:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.234+0000 7ff363834640 1 Processor -- start 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.234+0000 7ff363834640 1 -- start start 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.234+0000 7ff363834640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35c1045b0 0x7ff35c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.234+0000 7ff363834640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 0x7ff35c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff363834640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff35c109870 0x7ff35c1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff363834640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c10f5f0 con 0x7ff35c106f10 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff363834640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c10f470 con 0x7ff35c1045b0 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff363834640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c10f770 con 0x7ff35c109870 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 0x7ff35c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 0x7ff35c19a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38304/0 (socket says 172.21.15.27:38304) 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 -- 172.21.15.27:0/2328797788 learned_addr learned my addr 172.21.15.27:0/2328797788 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff361dd1640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff35c109870 0x7ff35c1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff3615d0640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35c1045b0 0x7ff35c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 -- 172.21.15.27:0/2328797788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff35c109870 msgr2=0x7ff35c1a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff35c109870 0x7ff35c1a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 -- 172.21.15.27:0/2328797788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35c1045b0 msgr2=0x7ff35c19a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35c1045b0 0x7ff35c19a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 -- 172.21.15.27:0/2328797788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35c1a1d60 con 0x7ff35c106f10 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff3615d0640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35c1045b0 0x7ff35c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff360dcf640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 0x7ff35c19a760 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006010 tx=0x7ff34c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff3527fc640 1 -- 172.21.15.27:0/2328797788 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c03f030 con 0x7ff35c106f10 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff3527fc640 1 -- 172.21.15.27:0/2328797788 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff34c031910 con 0x7ff35c106f10 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35c1a1ff0 con 0x7ff35c106f10 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff3527fc640 1 -- 172.21.15.27:0/2328797788 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c035070 con 0x7ff35c106f10 2024-04-25T13:40:15.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.235+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff35c1a2400 con 0x7ff35c106f10 2024-04-25T13:40:15.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.236+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff35c0ff6f0 con 0x7ff35c106f10 2024-04-25T13:40:15.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.238+0000 7ff3527fc640 1 -- 172.21.15.27:0/2328797788 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff34c042030 con 0x7ff35c106f10 2024-04-25T13:40:15.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.239+0000 7ff3527fc640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff3480afaf0 0x7ff3480b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.239+0000 7ff3615d0640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff3480afaf0 0x7ff3480b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.239+0000 7ff3527fc640 1 -- 172.21.15.27:0/2328797788 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff34c0ff7f0 con 0x7ff35c106f10 2024-04-25T13:40:15.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.240+0000 7ff3615d0640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff3480afaf0 0x7ff3480b2000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff35c06a8b0 tx=0x7ff358005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.243+0000 7ff3527fc640 1 -- 172.21.15.27:0/2328797788 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff34c0c8400 con 0x7ff35c106f10 2024-04-25T13:40:15.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:14 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2696311140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:15.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:14 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2859231047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:15.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:14 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1392311624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:15.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:14 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2696311140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:15.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:14 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2859231047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:15.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:14 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1392311624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:15.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.470+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff35c02db50 con 0x7ff35c106f10 2024-04-25T13:40:15.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.470+0000 7ff3527fc640 1 -- 172.21.15.27:0/2328797788 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7ff34c0c7d00 con 0x7ff35c106f10 2024-04-25T13:40:15.471 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:15.471 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:40:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:40:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff3480afaf0 msgr2=0x7ff3480b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff3480afaf0 0x7ff3480b2000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff35c06a8b0 tx=0x7ff358005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 msgr2=0x7ff35c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 0x7ff35c19a760 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006010 tx=0x7ff34c0042f0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 shutdown_connections 2024-04-25T13:40:15.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff3480afaf0 0x7ff3480b2000 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff35c109870 0x7ff35c1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c106f10 0x7ff35c19a760 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 --2- 172.21.15.27:0/2328797788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35c1045b0 0x7ff35c19a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 >> 172.21.15.27:0/2328797788 conn(0x7ff35c0fdbd0 msgr2=0x7ff35c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:15.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 shutdown_connections 2024-04-25T13:40:15.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.473+0000 7ff363834640 1 -- 172.21.15.27:0/2328797788 wait complete. 2024-04-25T13:40:15.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:15.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4137238409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef58103c20 msgr2=0x7fef58104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4137238409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef58103c20 0x7fef58104030 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009900 tx=0x7fef4c02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:15.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4137238409 shutdown_connections 2024-04-25T13:40:15.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4137238409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef58108920 0x7fef5810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4137238409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef58103c20 0x7fef58104030 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4137238409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef581029e0 0x7fef581036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4137238409 >> 172.21.15.27:0/4137238409 conn(0x7fef580fdbd0 msgr2=0x7fef58100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:15.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4137238409 shutdown_connections 2024-04-25T13:40:15.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.597+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4137238409 wait complete. 2024-04-25T13:40:15.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5cc9f640 1 Processor -- start 2024-04-25T13:40:15.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5cc9f640 1 -- start start 2024-04-25T13:40:15.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5cc9f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef581029e0 0x7fef581980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5cc9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef58103c20 0x7fef581985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5cc9f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef58108920 0x7fef5819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5cc9f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fef5810d400 con 0x7fef581029e0 2024-04-25T13:40:15.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5cc9f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef5810d280 con 0x7fef58103c20 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5cc9f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fef5810d580 con 0x7fef58108920 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef581029e0 0x7fef581980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef581029e0 0x7fef581980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38330/0 (socket says 172.21.15.27:38330) 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef5659c640 1 -- 172.21.15.27:0/4167034188 learned_addr learned my addr 172.21.15.27:0/4167034188 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.598+0000 7fef56d9d640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef58108920 0x7fef5819f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef55d9b640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef58103c20 0x7fef581985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef5659c640 1 -- 172.21.15.27:0/4167034188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef58108920 msgr2=0x7fef5819f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef5659c640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef58108920 0x7fef5819f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef5659c640 1 -- 172.21.15.27:0/4167034188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef58103c20 msgr2=0x7fef581985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef5659c640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef58103c20 0x7fef581985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef5659c640 1 -- 172.21.15.27:0/4167034188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5819fb90 con 0x7fef581029e0 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef56d9d640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef58108920 0x7fef5819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef5659c640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef581029e0 0x7fef581980a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fef50009db0 tx=0x7fef5000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef477fe640 1 -- 172.21.15.27:0/4167034188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5000fdc0 con 0x7fef581029e0 2024-04-25T13:40:15.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef477fe640 1 -- 172.21.15.27:0/4167034188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef500035f0 con 0x7fef581029e0 2024-04-25T13:40:15.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef477fe640 1 -- 172.21.15.27:0/4167034188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50017660 con 0x7fef581029e0 2024-04-25T13:40:15.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5819fe80 con 0x7fef581029e0 2024-04-25T13:40:15.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.599+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef581a02e0 con 0x7fef581029e0 2024-04-25T13:40:15.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.600+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef5806a1c0 con 0x7fef581029e0 2024-04-25T13:40:15.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.601+0000 7fef477fe640 1 -- 172.21.15.27:0/4167034188 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fef50004000 con 0x7fef581029e0 2024-04-25T13:40:15.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.602+0000 7fef477fe640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef400afaf0 0x7fef400b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.602+0000 7fef55d9b640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef400afaf0 0x7fef400b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.602+0000 7fef477fe640 1 -- 172.21.15.27:0/4167034188 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fef500e0fe0 con 0x7fef581029e0 2024-04-25T13:40:15.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.603+0000 7fef55d9b640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef400afaf0 0x7fef400b2000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fef4c031500 tx=0x7fef4c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.606+0000 7fef477fe640 1 -- 172.21.15.27:0/4167034188 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef5806a1c0 con 0x7fef581029e0 2024-04-25T13:40:15.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.795+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef5802dc10 con 0x7fef581029e0 2024-04-25T13:40:15.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.796+0000 7fef477fe640 1 -- 172.21.15.27:0/4167034188 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7fef50012070 con 0x7fef581029e0 2024-04-25T13:40:15.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:15.797 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef400afaf0 msgr2=0x7fef400b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef400afaf0 0x7fef400b2000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fef4c031500 tx=0x7fef4c000bc0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef581029e0 msgr2=0x7fef581980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef581029e0 0x7fef581980a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fef50009db0 tx=0x7fef5000f4a0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 shutdown_connections 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef400afaf0 0x7fef400b2000 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef58108920 0x7fef5819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef58103c20 0x7fef581985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 --2- 172.21.15.27:0/4167034188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef581029e0 0x7fef581980a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 >> 172.21.15.27:0/4167034188 conn(0x7fef580fdbd0 msgr2=0x7fef58104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 shutdown_connections 2024-04-25T13:40:15.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.799+0000 7fef5cc9f640 1 -- 172.21.15.27:0/4167034188 wait complete. 2024-04-25T13:40:15.810 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:40:15.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 -- 172.21.15.27:0/3923415663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 msgr2=0x7f8f5c1036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 --2- 172.21.15.27:0/3923415663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 0x7f8f5c1036f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009900 tx=0x7f8f4c02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 -- 172.21.15.27:0/3923415663 shutdown_connections 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 --2- 172.21.15.27:0/3923415663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c108930 0x7f8f5c10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 --2- 172.21.15.27:0/3923415663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f5c103c30 0x7f8f5c104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 --2- 172.21.15.27:0/3923415663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 0x7f8f5c1036f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 -- 172.21.15.27:0/3923415663 >> 172.21.15.27:0/3923415663 conn(0x7f8f5c0fdbc0 msgr2=0x7f8f5c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 -- 172.21.15.27:0/3923415663 shutdown_connections 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.925+0000 7f8f625e3640 1 -- 172.21.15.27:0/3923415663 wait complete. 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.926+0000 7f8f625e3640 1 Processor -- start 2024-04-25T13:40:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.926+0000 7f8f625e3640 1 -- start start 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.926+0000 7f8f625e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 0x7f8f5c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.926+0000 7f8f625e3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c103c30 0x7f8f5c19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.926+0000 7f8f625e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f5c108930 0x7f8f5c1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.926+0000 7f8f625e3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c10f6a0 con 0x7f8f5c1029f0 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.926+0000 7f8f625e3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c10f520 con 0x7f8f5c108930 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.926+0000 7f8f625e3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c10f820 con 0x7f8f5c103c30 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 0x7f8f5c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 0x7f8f5c19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38356/0 (socket says 172.21.15.27:38356) 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 -- 172.21.15.27:0/2365050214 learned_addr learned my addr 172.21.15.27:0/2365050214 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f60b80640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f5c108930 0x7f8f5c1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5b7fe640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c103c30 0x7f8f5c19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 -- 172.21.15.27:0/2365050214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c103c30 msgr2=0x7f8f5c19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c103c30 0x7f8f5c19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 -- 172.21.15.27:0/2365050214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f5c108930 msgr2=0x7f8f5c1a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f5c108930 0x7f8f5c1a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 -- 172.21.15.27:0/2365050214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c1a1df0 con 0x7f8f5c1029f0 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f60b80640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f5c108930 0x7f8f5c1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f5bfff640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 0x7f8f5c19a2b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c031440 tx=0x7f8f4c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f597fa640 1 -- 172.21.15.27:0/2365050214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c03f070 con 0x7f8f5c1029f0 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f597fa640 1 -- 172.21.15.27:0/2365050214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f4c03b460 con 0x7f8f5c1029f0 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f597fa640 1 -- 172.21.15.27:0/2365050214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c035030 con 0x7f8f5c1029f0 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5c1a2080 con 0x7f8f5c1029f0 2024-04-25T13:40:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.927+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5c1a2640 con 0x7f8f5c1029f0 2024-04-25T13:40:15.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.928+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f5c107740 con 0x7f8f5c1029f0 2024-04-25T13:40:15.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.930+0000 7f8f597fa640 1 -- 172.21.15.27:0/2365050214 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8f4c042070 con 0x7f8f5c1029f0 2024-04-25T13:40:15.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.930+0000 7f8f597fa640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f440afb70 0x7f8f440b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:15.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.931+0000 7f8f5b7fe640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f440afb70 0x7f8f440b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:15.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.931+0000 7f8f597fa640 1 -- 172.21.15.27:0/2365050214 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8f4c0ffa30 con 0x7f8f5c1029f0 2024-04-25T13:40:15.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.931+0000 7f8f5b7fe640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f440afb70 0x7f8f440b2080 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8f50005fd0 tx=0x7f8f50005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:15.935+0000 7f8f597fa640 1 -- 172.21.15.27:0/2365050214 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f4c0c8640 con 0x7f8f5c1029f0 2024-04-25T13:40:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.122+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8f5c0637a0 con 0x7f8f5c1029f0 2024-04-25T13:40:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.122+0000 7f8f597fa640 1 -- 172.21.15.27:0/2365050214 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f8f4c0c7f40 con 0x7f8f5c1029f0 2024-04-25T13:40:16.122 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:40:16.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f440afb70 msgr2=0x7f8f440b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f440afb70 0x7f8f440b2080 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8f50005fd0 tx=0x7f8f50005850 comp rx=0 tx=0).stop 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 msgr2=0x7f8f5c19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 0x7f8f5c19a2b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c031440 tx=0x7f8f4c003d50 comp rx=0 tx=0).stop 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 shutdown_connections 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f440afb70 0x7f8f440b2080 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f5c108930 0x7f8f5c1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c103c30 0x7f8f5c19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 --2- 172.21.15.27:0/2365050214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c1029f0 0x7f8f5c19a2b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 >> 172.21.15.27:0/2365050214 conn(0x7f8f5c0fdbc0 msgr2=0x7f8f5c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 shutdown_connections 2024-04-25T13:40:16.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.124+0000 7f8f625e3640 1 -- 172.21.15.27:0/2365050214 wait complete. 2024-04-25T13:40:16.134 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:15 smithi027 ceph-mon[29140]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-25T13:40:16.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:15 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/409288708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:16.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:15 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2328797788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:16.135 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:15 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/4167034188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:16.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:40:16.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 -- 172.21.15.27:0/4265814717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c108920 msgr2=0x7f215c10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 --2- 172.21.15.27:0/4265814717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c108920 0x7f215c10ad40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2154009ca0 tx=0x7f215402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 -- 172.21.15.27:0/4265814717 shutdown_connections 2024-04-25T13:40:16.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 --2- 172.21.15.27:0/4265814717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c108920 0x7f215c10ad40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 --2- 172.21.15.27:0/4265814717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f215c103c20 0x7f215c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 --2- 172.21.15.27:0/4265814717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f215c1029e0 0x7f215c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 -- 172.21.15.27:0/4265814717 >> 172.21.15.27:0/4265814717 conn(0x7f215c0fdbd0 msgr2=0x7f215c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:16.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 -- 172.21.15.27:0/4265814717 shutdown_connections 2024-04-25T13:40:16.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.250+0000 7f21609f3640 1 -- 172.21.15.27:0/4265814717 wait complete. 2024-04-25T13:40:16.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f21609f3640 1 Processor -- start 2024-04-25T13:40:16.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f21609f3640 1 -- start start 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f21609f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c1029e0 0x7f215c19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f21609f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f215c103c20 0x7f215c19aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f21609f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f215c108920 0x7f215c1a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f21609f3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f215c10f610 con 0x7f215c1029e0 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f21609f3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f215c10f490 con 0x7f215c108920 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f21609f3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f215c10f790 con 0x7f215c103c20 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c1029e0 0x7f215c19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c1029e0 0x7f215c19a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38384/0 (socket says 172.21.15.27:38384) 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215a59c640 1 -- 172.21.15.27:0/2854047980 learned_addr learned my addr 172.21.15.27:0/2854047980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f2159d9b640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f215c103c20 0x7f215c19aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215ad9d640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f215c108920 0x7f215c1a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215a59c640 1 -- 172.21.15.27:0/2854047980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f215c103c20 msgr2=0x7f215c19aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215a59c640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f215c103c20 0x7f215c19aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215a59c640 1 -- 172.21.15.27:0/2854047980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f215c108920 msgr2=0x7f215c1a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215a59c640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f215c108920 0x7f215c1a19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.251+0000 7f215a59c640 1 -- 172.21.15.27:0/2854047980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215c1a2040 con 0x7f215c1029e0 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.252+0000 7f215a59c640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c1029e0 0x7f215c19a500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2144005fd0 tx=0x7f214400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.252+0000 7f21537fe640 1 -- 172.21.15.27:0/2854047980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144017070 con 0x7f215c1029e0 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.252+0000 7f21537fe640 1 -- 172.21.15.27:0/2854047980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2144009c30 con 0x7f215c1029e0 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.252+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215c1a2330 con 0x7f215c1029e0 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.252+0000 7f21537fe640 1 -- 172.21.15.27:0/2854047980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2144003d10 con 0x7f215c1029e0 2024-04-25T13:40:16.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.252+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f215c1a2880 con 0x7f215c1029e0 2024-04-25T13:40:16.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.253+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f215c107730 con 0x7f215c1029e0 2024-04-25T13:40:16.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.254+0000 7f21537fe640 1 -- 172.21.15.27:0/2854047980 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2144003eb0 con 0x7f215c1029e0 2024-04-25T13:40:16.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.255+0000 7f21537fe640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f21480afaf0 0x7f21480b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.255+0000 7f2159d9b640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f21480afaf0 0x7f21480b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.255+0000 7f21537fe640 1 -- 172.21.15.27:0/2854047980 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f21440de900 con 0x7f215c1029e0 2024-04-25T13:40:16.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.256+0000 7f2159d9b640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f21480afaf0 0x7f21480b2000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f215c19b5a0 tx=0x7f214c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:16.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.260+0000 7f21537fe640 1 -- 172.21.15.27:0/2854047980 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2144020030 con 0x7f215c1029e0 2024-04-25T13:40:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:15 smithi122 ceph-mon[32843]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-25T13:40:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:15 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/409288708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:15 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2328797788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:15 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/4167034188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:16.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:15 smithi045 ceph-mon[32731]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-25T13:40:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:15 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/409288708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:15 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2328797788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:16.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:15 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/4167034188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:16.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.447+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f215c02dca0 con 0x7f215c1029e0 2024-04-25T13:40:16.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.447+0000 7f21537fe640 1 -- 172.21.15.27:0/2854047980 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f2144009de0 con 0x7f215c1029e0 2024-04-25T13:40:16.447 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:40:16.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f21480afaf0 msgr2=0x7f21480b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f21480afaf0 0x7f21480b2000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f215c19b5a0 tx=0x7f214c00b390 comp rx=0 tx=0).stop 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c1029e0 msgr2=0x7f215c19a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c1029e0 0x7f215c19a500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2144005fd0 tx=0x7f214400dde0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 shutdown_connections 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f21480afaf0 0x7f21480b2000 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f215c108920 0x7f215c1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f215c103c20 0x7f215c19aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 --2- 172.21.15.27:0/2854047980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f215c1029e0 0x7f215c19a500 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 >> 172.21.15.27:0/2854047980 conn(0x7f215c0fdbd0 msgr2=0x7f215c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 shutdown_connections 2024-04-25T13:40:16.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.449+0000 7f21609f3640 1 -- 172.21.15.27:0/2854047980 wait complete. 2024-04-25T13:40:16.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.574+0000 7f72f3f15640 1 -- 172.21.15.27:0/1521460282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec1043e0 msgr2=0x7f72ec102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.574+0000 7f72f3f15640 1 --2- 172.21.15.27:0/1521460282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec1043e0 0x7f72ec102de0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009900 tx=0x7f72dc02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.574+0000 7f72f3f15640 1 -- 172.21.15.27:0/1521460282 shutdown_connections 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.574+0000 7f72f3f15640 1 --2- 172.21.15.27:0/1521460282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72ec108a30 0x7f72ec10ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.574+0000 7f72f3f15640 1 --2- 172.21.15.27:0/1521460282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec1043e0 0x7f72ec102de0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.574+0000 7f72f3f15640 1 --2- 172.21.15.27:0/1521460282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72ec103f00 0x7f72ec1027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.574+0000 7f72f3f15640 1 -- 172.21.15.27:0/1521460282 >> 172.21.15.27:0/1521460282 conn(0x7f72ec0fd380 msgr2=0x7f72ec0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.574+0000 7f72f3f15640 1 -- 172.21.15.27:0/1521460282 shutdown_connections 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 -- 172.21.15.27:0/1521460282 wait complete. 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 Processor -- start 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 -- start start 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec103f00 0x7f72ec199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72ec108a30 0x7f72ec19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72ec19f2f0 0x7f72ec1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec10f6a0 con 0x7f72ec103f00 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec10f520 con 0x7f72ec19f2f0 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f3f15640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec10f820 con 0x7f72ec108a30 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f1cb1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec103f00 0x7f72ec199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f1cb1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec103f00 0x7f72ec199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38400/0 (socket says 172.21.15.27:38400) 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.575+0000 7f72f1cb1640 1 -- 172.21.15.27:0/115801308 learned_addr learned my addr 172.21.15.27:0/115801308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f24b2640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72ec19f2f0 0x7f72ec1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f14b0640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72ec108a30 0x7f72ec19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f1cb1640 1 -- 172.21.15.27:0/115801308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72ec108a30 msgr2=0x7f72ec19a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f1cb1640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72ec108a30 0x7f72ec19a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f1cb1640 1 -- 172.21.15.27:0/115801308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72ec19f2f0 msgr2=0x7f72ec1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f1cb1640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72ec19f2f0 0x7f72ec1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f1cb1640 1 -- 172.21.15.27:0/115801308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ec1a1d80 con 0x7f72ec103f00 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f14b0640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72ec108a30 0x7f72ec19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f24b2640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72ec19f2f0 0x7f72ec1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f1cb1640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec103f00 0x7f72ec199e30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f72e800d740 tx=0x7f72e800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72e2ffd640 1 -- 172.21.15.27:0/115801308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8017070 con 0x7f72ec103f00 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72e2ffd640 1 -- 172.21.15.27:0/115801308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72e800de50 con 0x7f72ec103f00 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ec1a2070 con 0x7f72ec103f00 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72e2ffd640 1 -- 172.21.15.27:0/115801308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8004ce0 con 0x7f72ec103f00 2024-04-25T13:40:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.576+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ec1a2570 con 0x7f72ec103f00 2024-04-25T13:40:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.577+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72ec107130 con 0x7f72ec103f00 2024-04-25T13:40:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.579+0000 7f72e2ffd640 1 -- 172.21.15.27:0/115801308 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f72e8004e80 con 0x7f72ec103f00 2024-04-25T13:40:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.579+0000 7f72e2ffd640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f72d80afaf0 0x7f72d80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.580+0000 7f72f14b0640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f72d80afaf0 0x7f72d80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.580+0000 7f72e2ffd640 1 -- 172.21.15.27:0/115801308 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f72e80dea70 con 0x7f72ec103f00 2024-04-25T13:40:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.580+0000 7f72f14b0640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f72d80afaf0 0x7f72d80b2000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72dc006010 tx=0x7f72dc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.583+0000 7f72e2ffd640 1 -- 172.21.15.27:0/115801308 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72e80a7680 con 0x7f72ec103f00 2024-04-25T13:40:16.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.771+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f72ec0637a0 con 0x7f72ec103f00 2024-04-25T13:40:16.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.771+0000 7f72e2ffd640 1 -- 172.21.15.27:0/115801308 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f72e80a6f80 con 0x7f72ec103f00 2024-04-25T13:40:16.771 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:40:16.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f72d80afaf0 msgr2=0x7f72d80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f72d80afaf0 0x7f72d80b2000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72dc006010 tx=0x7f72dc000bc0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec103f00 msgr2=0x7f72ec199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec103f00 0x7f72ec199e30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f72e800d740 tx=0x7f72e800dc00 comp rx=0 tx=0).stop 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 shutdown_connections 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f72d80afaf0 0x7f72d80b2000 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72ec19f2f0 0x7f72ec1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72ec108a30 0x7f72ec19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 --2- 172.21.15.27:0/115801308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec103f00 0x7f72ec199e30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 >> 172.21.15.27:0/115801308 conn(0x7f72ec0fd380 msgr2=0x7f72ec0ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 shutdown_connections 2024-04-25T13:40:16.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.773+0000 7f72f3f15640 1 -- 172.21.15.27:0/115801308 wait complete. 2024-04-25T13:40:16.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 -- 172.21.15.27:0/670143212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374103c00 msgr2=0x7f2374104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 --2- 172.21.15.27:0/670143212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374103c00 0x7f2374104010 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2370009900 tx=0x7f237002a410 comp rx=0 tx=0).stop 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 -- 172.21.15.27:0/670143212 shutdown_connections 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 --2- 172.21.15.27:0/670143212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2374108900 0x7f237410ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 --2- 172.21.15.27:0/670143212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2374103c00 0x7f2374104010 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 --2- 172.21.15.27:0/670143212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23741029c0 0x7f23741036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 -- 172.21.15.27:0/670143212 >> 172.21.15.27:0/670143212 conn(0x7f23740fdbd0 msgr2=0x7f2374100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 -- 172.21.15.27:0/670143212 shutdown_connections 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.896+0000 7f237c3b7640 1 -- 172.21.15.27:0/670143212 wait complete. 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237c3b7640 1 Processor -- start 2024-04-25T13:40:16.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237c3b7640 1 -- start start 2024-04-25T13:40:16.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237c3b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23741029c0 0x7f237419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237c3b7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2374103c00 0x7f237419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237c3b7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2374108900 0x7f23741a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237c3b7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f237410f5f0 con 0x7f23741029c0 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237c3b7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f237410f470 con 0x7f2374103c00 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237c3b7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f237410f770 con 0x7f2374108900 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237a153640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23741029c0 0x7f237419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237a153640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23741029c0 0x7f237419a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38412/0 (socket says 172.21.15.27:38412) 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.897+0000 7f237a153640 1 -- 172.21.15.27:0/3006660248 learned_addr learned my addr 172.21.15.27:0/3006660248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237a954640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2374108900 0x7f23741a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f2379952640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2374103c00 0x7f237419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237a153640 1 -- 172.21.15.27:0/3006660248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2374108900 msgr2=0x7f23741a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237a153640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2374108900 0x7f23741a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237a153640 1 -- 172.21.15.27:0/3006660248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2374103c00 msgr2=0x7f237419a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237a153640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2374103c00 0x7f237419a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237a153640 1 -- 172.21.15.27:0/3006660248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23741a1d30 con 0x7f23741029c0 2024-04-25T13:40:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f2379952640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2374103c00 0x7f237419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237a153640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23741029c0 0x7f237419a260 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f236400e9f0 tx=0x7f236400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f236b7fe640 1 -- 172.21.15.27:0/3006660248 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236400ccc0 con 0x7f23741029c0 2024-04-25T13:40:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23741a1fc0 con 0x7f23741029c0 2024-04-25T13:40:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f236b7fe640 1 -- 172.21.15.27:0/3006660248 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23640035f0 con 0x7f23741029c0 2024-04-25T13:40:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f236b7fe640 1 -- 172.21.15.27:0/3006660248 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236400ce60 con 0x7f23741029c0 2024-04-25T13:40:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.898+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23741a2430 con 0x7f23741029c0 2024-04-25T13:40:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.899+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2374107710 con 0x7f23741029c0 2024-04-25T13:40:16.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.901+0000 7f236b7fe640 1 -- 172.21.15.27:0/3006660248 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2364003aa0 con 0x7f23741029c0 2024-04-25T13:40:16.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.901+0000 7f236b7fe640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23600afaf0 0x7f23600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:16.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.901+0000 7f2379952640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23600afaf0 0x7f23600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:16.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.902+0000 7f236b7fe640 1 -- 172.21.15.27:0/3006660248 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f23640df110 con 0x7f23741029c0 2024-04-25T13:40:16.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.902+0000 7f2379952640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23600afaf0 0x7f23600b2000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2370006010 tx=0x7f2370000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:16.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:16.905+0000 7f236b7fe640 1 -- 172.21.15.27:0/3006660248 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23640a7d20 con 0x7f23741029c0 2024-04-25T13:40:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.135+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f237402db50 con 0x7f23741029c0 2024-04-25T13:40:17.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.136+0000 7f236b7fe640 1 -- 172.21.15.27:0/3006660248 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f23640a7620 con 0x7f23741029c0 2024-04-25T13:40:17.136 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:17.136 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:40:17.136 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:40:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23600afaf0 msgr2=0x7f23600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23600afaf0 0x7f23600b2000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2370006010 tx=0x7f2370000bc0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23741029c0 msgr2=0x7f237419a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23741029c0 0x7f237419a260 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f236400e9f0 tx=0x7f236400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 shutdown_connections 2024-04-25T13:40:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23600afaf0 0x7f23600b2000 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2374108900 0x7f23741a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2374103c00 0x7f237419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 --2- 172.21.15.27:0/3006660248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23741029c0 0x7f237419a260 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 >> 172.21.15.27:0/3006660248 conn(0x7f23740fdbd0 msgr2=0x7f23741048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 shutdown_connections 2024-04-25T13:40:17.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.138+0000 7f237c3b7640 1 -- 172.21.15.27:0/3006660248 wait complete. 2024-04-25T13:40:17.148 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:16 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2365050214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.148 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:16 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2854047980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.148 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:16 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/115801308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 -- 172.21.15.27:0/3150814764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8108940 msgr2=0x7f0af810ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 --2- 172.21.15.27:0/3150814764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8108940 0x7f0af810ad60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009ca0 tx=0x7f0af002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 -- 172.21.15.27:0/3150814764 shutdown_connections 2024-04-25T13:40:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 --2- 172.21.15.27:0/3150814764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8108940 0x7f0af810ad60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 --2- 172.21.15.27:0/3150814764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af8103c40 0x7f0af8104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 --2- 172.21.15.27:0/3150814764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af8102a00 0x7f0af8103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 -- 172.21.15.27:0/3150814764 >> 172.21.15.27:0/3150814764 conn(0x7f0af80fdbf0 msgr2=0x7f0af81000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 -- 172.21.15.27:0/3150814764 shutdown_connections 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.261+0000 7f0affb9e640 1 -- 172.21.15.27:0/3150814764 wait complete. 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0affb9e640 1 Processor -- start 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0affb9e640 1 -- start start 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0affb9e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af8102a00 0x7f0af819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0affb9e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8103c40 0x7f0af819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0affb9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af8108940 0x7f0af81a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0affb9e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0af810e530 con 0x7f0af8103c40 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0affb9e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0af810e3b0 con 0x7f0af8102a00 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0affb9e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0af810e6b0 con 0x7f0af8108940 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8103c40 0x7f0af819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8103c40 0x7f0af819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38424/0 (socket says 172.21.15.27:38424) 2024-04-25T13:40:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 -- 172.21.15.27:0/1850752341 learned_addr learned my addr 172.21.15.27:0/1850752341 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afe13b640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af8108940 0x7f0af81a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 -- 172.21.15.27:0/1850752341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af8108940 msgr2=0x7f0af81a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af8108940 0x7f0af81a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 -- 172.21.15.27:0/1850752341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af8102a00 msgr2=0x7f0af819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd93a640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af8102a00 0x7f0af819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af8102a00 0x7f0af819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 -- 172.21.15.27:0/1850752341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af81a1c30 con 0x7f0af8103c40 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.262+0000 7f0afd139640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8103c40 0x7f0af819a6e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0ae800c920 tx=0x7f0ae800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.263+0000 7f0aeeffd640 1 -- 172.21.15.27:0/1850752341 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8007c70 con 0x7f0af8103c40 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.263+0000 7f0aeeffd640 1 -- 172.21.15.27:0/1850752341 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ae8007e10 con 0x7f0af8103c40 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.263+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af81a1f20 con 0x7f0af8103c40 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.263+0000 7f0aeeffd640 1 -- 172.21.15.27:0/1850752341 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8018820 con 0x7f0af8103c40 2024-04-25T13:40:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.263+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af81a2390 con 0x7f0af8103c40 2024-04-25T13:40:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.264+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0af8108040 con 0x7f0af8103c40 2024-04-25T13:40:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.266+0000 7f0aeeffd640 1 -- 172.21.15.27:0/1850752341 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0ae800f450 con 0x7f0af8103c40 2024-04-25T13:40:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.266+0000 7f0aeeffd640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ae40afaf0 0x7f0ae40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.266+0000 7f0afd93a640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ae40afaf0 0x7f0ae40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.267+0000 7f0aeeffd640 1 -- 172.21.15.27:0/1850752341 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0ae80df090 con 0x7f0af8103c40 2024-04-25T13:40:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.267+0000 7f0afd93a640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ae40afaf0 0x7f0ae40b2000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0af8103600 tx=0x7f0af4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.270+0000 7f0aeeffd640 1 -- 172.21.15.27:0/1850752341 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ae8014030 con 0x7f0af8103c40 2024-04-25T13:40:17.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:16 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2365050214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:16 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2854047980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:16 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/115801308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:16 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2365050214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:16 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2854047980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:16 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/115801308' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:17.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.458+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0af802dda0 con 0x7f0af8103c40 2024-04-25T13:40:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.459+0000 7f0aeeffd640 1 -- 172.21.15.27:0/1850752341 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7f0ae80a7d30 con 0x7f0af8103c40 2024-04-25T13:40:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:17.460 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:17.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ae40afaf0 msgr2=0x7f0ae40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ae40afaf0 0x7f0ae40b2000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0af8103600 tx=0x7f0af4005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8103c40 msgr2=0x7f0af819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8103c40 0x7f0af819a6e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0ae800c920 tx=0x7f0ae800cde0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 shutdown_connections 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ae40afaf0 0x7f0ae40b2000 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af8108940 0x7f0af81a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0af8103c40 0x7f0af819a6e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 --2- 172.21.15.27:0/1850752341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0af8102a00 0x7f0af819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 >> 172.21.15.27:0/1850752341 conn(0x7f0af80fdbf0 msgr2=0x7f0af8101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 shutdown_connections 2024-04-25T13:40:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.461+0000 7f0affb9e640 1 -- 172.21.15.27:0/1850752341 wait complete. 2024-04-25T13:40:17.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.584+0000 7f53ffbe3640 1 -- 172.21.15.27:0/3981910622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8109a40 msgr2=0x7f53f810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.584+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/3981910622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8109a40 0x7f53f810be60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009ca0 tx=0x7f53f002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 -- 172.21.15.27:0/3981910622 shutdown_connections 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/3981910622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8109a40 0x7f53f810be60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/3981910622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53f8104d40 0x7f53f8105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/3981910622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53f8103b00 0x7f53f8104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 -- 172.21.15.27:0/3981910622 >> 172.21.15.27:0/3981910622 conn(0x7f53f80fed10 msgr2=0x7f53f81011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 -- 172.21.15.27:0/3981910622 shutdown_connections 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 -- 172.21.15.27:0/3981910622 wait complete. 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 Processor -- start 2024-04-25T13:40:17.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 -- start start 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8103b00 0x7f53f819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53f8104d40 0x7f53f819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53f8109a40 0x7f53f81a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53f810f610 con 0x7f53f8103b00 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53f810f490 con 0x7f53f8109a40 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.585+0000 7f53ffbe3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f53f810f790 con 0x7f53f8104d40 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8103b00 0x7f53f819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8103b00 0x7f53f819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38444/0 (socket says 172.21.15.27:38444) 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 -- 172.21.15.27:0/378980667 learned_addr learned my addr 172.21.15.27:0/378980667 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fe180640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53f8109a40 0x7f53f81a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd17e640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53f8104d40 0x7f53f819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 -- 172.21.15.27:0/378980667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53f8104d40 msgr2=0x7f53f819a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53f8104d40 0x7f53f819a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 -- 172.21.15.27:0/378980667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53f8109a40 msgr2=0x7f53f81a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53f8109a40 0x7f53f81a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 -- 172.21.15.27:0/378980667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f81a1bb0 con 0x7f53f8103b00 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd17e640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53f8104d40 0x7f53f819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53fd97f640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8103b00 0x7f53f819a120 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f53f400c970 tx=0x7f53f400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53eeffd640 1 -- 172.21.15.27:0/378980667 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f4007c70 con 0x7f53f8103b00 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53eeffd640 1 -- 172.21.15.27:0/378980667 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53f4007e10 con 0x7f53f8103b00 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53eeffd640 1 -- 172.21.15.27:0/378980667 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f40187b0 con 0x7f53f8103b00 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f81a1ea0 con 0x7f53f8103b00 2024-04-25T13:40:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.586+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f81a2300 con 0x7f53f8103b00 2024-04-25T13:40:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.587+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f806a190 con 0x7f53f8103b00 2024-04-25T13:40:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.589+0000 7f53eeffd640 1 -- 172.21.15.27:0/378980667 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f53f400fab0 con 0x7f53f8103b00 2024-04-25T13:40:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.589+0000 7f53eeffd640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53e40afaf0 0x7f53e40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.589+0000 7f53fd17e640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53e40afaf0 0x7f53e40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.590+0000 7f53eeffd640 1 -- 172.21.15.27:0/378980667 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f53f40ded80 con 0x7f53f8103b00 2024-04-25T13:40:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.590+0000 7f53fd17e640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53e40afaf0 0x7f53e40b2000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f53f819b1c0 tx=0x7f53e8005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.593+0000 7f53eeffd640 1 -- 172.21.15.27:0/378980667 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f53f4014030 con 0x7f53f8103b00 2024-04-25T13:40:17.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.820+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53f802dc30 con 0x7f53f8103b00 2024-04-25T13:40:17.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.820+0000 7f53eeffd640 1 -- 172.21.15.27:0/378980667 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f53f40a7a20 con 0x7f53f8103b00 2024-04-25T13:40:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:17.821 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:40:17.821 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53e40afaf0 msgr2=0x7f53e40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53e40afaf0 0x7f53e40b2000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f53f819b1c0 tx=0x7f53e8005db0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8103b00 msgr2=0x7f53f819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8103b00 0x7f53f819a120 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f53f400c970 tx=0x7f53f400ce30 comp rx=0 tx=0).stop 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 shutdown_connections 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53e40afaf0 0x7f53e40b2000 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53f8109a40 0x7f53f81a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53f8104d40 0x7f53f819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 --2- 172.21.15.27:0/378980667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53f8103b00 0x7f53f819a120 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.822+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 >> 172.21.15.27:0/378980667 conn(0x7f53f80fed10 msgr2=0x7f53f8105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.823+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 shutdown_connections 2024-04-25T13:40:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.823+0000 7f53ffbe3640 1 -- 172.21.15.27:0/378980667 wait complete. 2024-04-25T13:40:17.834 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T13:38:44.716138+0000', 'modified': '2024-04-25T13:38:48.756882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24517': {'gid': 24517, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.122:6839/3397333136', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6838', 'nonce': 3397333136}, {'type': 'v1', 'addr': '172.21.15.122:6839', 'nonce': 3397333136}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6839/3661968964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6838', 'nonce': 3661968964}, {'type': 'v1', 'addr': '172.21.15.45:6839', 'nonce': 3661968964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24560, 'qdb_cluster': [24560]} 2024-04-25T13:40:17.834 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T13:40:17.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T13:40:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 -- 172.21.15.27:0/2358563420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1045e0 msgr2=0x7f716c106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 --2- 172.21.15.27:0/2358563420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1045e0 0x7f716c106a00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f715c009900 tx=0x7f715c02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 -- 172.21.15.27:0/2358563420 shutdown_connections 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 --2- 172.21.15.27:0/2358563420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f716c1098a0 0x7f716c10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 --2- 172.21.15.27:0/2358563420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f716c106f40 0x7f716c109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 --2- 172.21.15.27:0/2358563420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1045e0 0x7f716c106a00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 -- 172.21.15.27:0/2358563420 >> 172.21.15.27:0/2358563420 conn(0x7f716c0fdbc0 msgr2=0x7f716c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 -- 172.21.15.27:0/2358563420 shutdown_connections 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.948+0000 7f7171508640 1 -- 172.21.15.27:0/2358563420 wait complete. 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.949+0000 7f7171508640 1 Processor -- start 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.949+0000 7f7171508640 1 -- start start 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.949+0000 7f7171508640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f716c1045e0 0x7f716c103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.949+0000 7f7171508640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f716c106f40 0x7f716c1020d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.949+0000 7f7171508640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1098a0 0x7f716c102610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.949+0000 7f7171508640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f716c10f640 con 0x7f716c1098a0 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.949+0000 7f7171508640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f716c10f4c0 con 0x7f716c106f40 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.949+0000 7f7171508640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f716c10f7c0 con 0x7f716c1045e0 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1098a0 0x7f716c102610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1098a0 0x7f716c102610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38460/0 (socket says 172.21.15.27:38460) 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 -- 172.21.15.27:0/3430137459 learned_addr learned my addr 172.21.15.27:0/3430137459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716affd640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f716c1045e0 0x7f716c103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f71627fc640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f716c106f40 0x7f716c1020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 -- 172.21.15.27:0/3430137459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f716c1045e0 msgr2=0x7f716c103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f716c1045e0 0x7f716c103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 -- 172.21.15.27:0/3430137459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f716c106f40 msgr2=0x7f716c1020d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f716c106f40 0x7f716c1020d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 -- 172.21.15.27:0/3430137459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716c102d40 con 0x7f716c1098a0 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f716b7fe640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1098a0 0x7f716c102610 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f716400eb40 tx=0x7f716400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f7168ff9640 1 -- 172.21.15.27:0/3430137459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164004b20 con 0x7f716c1098a0 2024-04-25T13:40:17.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f7171508640 1 -- 172.21.15.27:0/3430137459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716c1a4310 con 0x7f716c1098a0 2024-04-25T13:40:17.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f7168ff9640 1 -- 172.21.15.27:0/3430137459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7164004cc0 con 0x7f716c1098a0 2024-04-25T13:40:17.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f7168ff9640 1 -- 172.21.15.27:0/3430137459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164010720 con 0x7f716c1098a0 2024-04-25T13:40:17.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.950+0000 7f7171508640 1 -- 172.21.15.27:0/3430137459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716c1a4800 con 0x7f716c1098a0 2024-04-25T13:40:17.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.953+0000 7f71783b7740 1 Processor -- start 2024-04-25T13:40:17.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f7168ff9640 1 -- 172.21.15.27:0/3430137459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f71640108c0 con 0x7f716c1098a0 2024-04-25T13:40:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71783b7740 1 -- start start 2024-04-25T13:40:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71783b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 0x5580fb0ad370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71783b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580face33b0 0x5580fb0ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71783b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faceb590 0x5580fb0b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71783b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5580fb012740 con 0x5580face33b0 2024-04-25T13:40:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71783b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5580fafae3a0 con 0x5580fad0c270 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71783b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5580fb00e700 con 0x5580faceb590 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f716affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580face33b0 0x5580fb0ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f716affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580face33b0 0x5580fb0ad8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38466/0 (socket says 172.21.15.27:38466) 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f716affd640 1 -- 172.21.15.27:0/164818873 learned_addr learned my addr 172.21.15.27:0/164818873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71627fc640 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 0x5580fb0ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.954+0000 7f71627fc640 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faceb590 0x5580fb0b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f716affd640 1 -- 172.21.15.27:0/164818873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faceb590 msgr2=0x5580fb0b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f716affd640 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faceb590 0x5580fb0b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f716affd640 1 -- 172.21.15.27:0/164818873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 msgr2=0x5580fb0ad370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f716affd640 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 0x5580fb0ad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f7168ff9640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71580afaf0 0x7f71580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f716affd640 1 -- 172.21.15.27:0/164818873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580fafca750 con 0x5580face33b0 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f71627fc640 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faceb590 0x5580fb0b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f71627fc640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71580afaf0 0x7f71580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f716affd640 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580face33b0 0x5580fb0ad8f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f716c0757f0 tx=0x7f715c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd118d9fc28ef171 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f7168ff9640 1 -- 172.21.15.27:0/3430137459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f71640df300 con 0x7f716c1098a0 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f71617fa640 1 -- 172.21.15.27:0/164818873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c035030 con 0x5580face33b0 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f71617fa640 1 -- 172.21.15.27:0/164818873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f715c040070 con 0x5580face33b0 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.955+0000 7f71617fa640 1 -- 172.21.15.27:0/164818873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c0318a0 con 0x5580face33b0 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 -- 172.21.15.27:0/164818873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580face33b0 msgr2=0x5580fb0ad8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71627fc640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71580afaf0 0x7f71580b2000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x5580fb0ae540 tx=0x7f7154012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580face33b0 0x5580fb0ad8f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f716c0757f0 tx=0x7f715c004040 comp rx=0 tx=0).stop 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 -- 172.21.15.27:0/164818873 shutdown_connections 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faceb590 0x5580fb0b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580face33b0 0x5580fb0ad8f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 --2- 172.21.15.27:0/164818873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 0x5580fb0ad370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 -- 172.21.15.27:0/164818873 >> 172.21.15.27:0/164818873 conn(0x5580fad003c0 msgr2=0x5580fb0a8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 -- 172.21.15.27:0/164818873 shutdown_connections 2024-04-25T13:40:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 -- 172.21.15.27:0/164818873 wait complete. 2024-04-25T13:40:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.956+0000 7f71783b7740 1 -- start start 2024-04-25T13:40:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71783b7740 1 Processor -- start 2024-04-25T13:40:17.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71783b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580face33b0 0x5580fb0b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71783b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71783b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 0x5580fb0be690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71783b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5580fb0b51e0 con 0x5580faceb590 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71783b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5580faf99fe0 con 0x5580fad0c270 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71783b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5580fb0b5360 con 0x5580face33b0 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71783b7740 10 client.0 _ll_register_callbacks cb 0x5580fafb0150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38470/0 (socket says 172.21.15.27:38470) 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 -- 172.21.15.27:0/1844603956 learned_addr learned my addr 172.21.15.27:0/1844603956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f716affd640 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580face33b0 0x5580fb0b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f716b7fe640 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 0x5580fb0be690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 -- 172.21.15.27:0/1844603956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580face33b0 msgr2=0x5580fb0b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580face33b0 0x5580fb0b71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 -- 172.21.15.27:0/1844603956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 msgr2=0x5580fb0be690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 0x5580fb0be690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 -- 172.21.15.27:0/1844603956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580fad242d0 con 0x5580faceb590 2024-04-25T13:40:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f716affd640 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580face33b0 0x5580fb0b71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71627fc640 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b7730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f715401bbf0 tx=0x7f7154005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daad21c34e563dcf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.957+0000 7f71527fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71527fc640 1 -- 172.21.15.27:0/1844603956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715400fcd0 con 0x5580faceb590 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71527fc640 1 -- 172.21.15.27:0/1844603956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f71540253f0 con 0x5580faceb590 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 1 -- 172.21.15.27:0/1844603956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f715c040070 con 0x5580faceb590 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71527fc640 1 -- 172.21.15.27:0/1844603956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f71540255f0 con 0x5580faceb590 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 10 client.15051 fetch_fsmap learned FSMap version 19 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 1 -- 172.21.15.27:0/1844603956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5580fadee6b0 con 0x5580faceb590 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71527fc640 1 -- 172.21.15.27:0/1844603956 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f715c040070 con 0x5580faceb590 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71527fc640 1 -- 172.21.15.27:0/1844603956 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f715402e6d0 con 0x5580faceb590 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 10 client.15051 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 10 client.15051 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 20 client.15051 populate_metadata read hostname 'smithi027' 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580faccae40 0x5580fb0c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 4 client.15051 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.958+0000 7f71783b7740 1 -- 172.21.15.27:0/1844603956 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580fb020350 con 0x5580faccae40 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.959+0000 7f716affd640 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580faccae40 0x5580fb0c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.959+0000 7f716affd640 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580faccae40 0x5580fb0c4fb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a9684add568d83a server_cookie=112418626ce9c735 in_seq=0 out_seq=0 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.959+0000 7f71527fc640 10 client.15051 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71527fc640 1 -- 172.21.15.27:0/1844603956 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5580fb020350 con 0x5580faccae40 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71527fc640 10 client.15051 handle_command_reply: tid=0 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 client.15051 shutdown 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 -- 172.21.15.27:0/1844603956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 msgr2=0x5580fb0b7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b7730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f715401bbf0 tx=0x7f7154005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 -- 172.21.15.27:0/1844603956 shutdown_connections 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580faccae40 0x5580fb0c4fb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580fad0c270 0x5580fb0be690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71527fc640 0 client.15051 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b7730 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 --2- 172.21.15.27:0/1844603956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580face33b0 0x5580fb0b71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 -- 172.21.15.27:0/1844603956 >> 172.21.15.27:0/1844603956 conn(0x5580fad003c0 msgr2=0x5580fb0a7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 -- 172.21.15.27:0/1844603956 shutdown_connections 2024-04-25T13:40:17.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 1 -- 172.21.15.27:0/1844603956 wait complete. 2024-04-25T13:40:17.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.960+0000 7f71783b7740 20 client.15051 trim_cache size 0 max 16384 2024-04-25T13:40:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.979+0000 7f71783b7740 1 Processor -- start 2024-04-25T13:40:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.979+0000 7f71783b7740 1 -- start start 2024-04-25T13:40:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f71783b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0c28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f71783b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 0x5580fb0b5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f71783b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f71783b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5580fb0c32f0 con 0x5580faceb590 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f71783b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5580fb0c39c0 con 0x5580face33b0 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f71783b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5580fb0c4090 con 0x5580faccae40 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38472/0 (socket says 172.21.15.27:38472) 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 -- 172.21.15.27:0/491337662 learned_addr learned my addr 172.21.15.27:0/491337662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f71627fc640 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 0x5580fb0b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716affd640 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0c28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 -- 172.21.15.27:0/491337662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 msgr2=0x5580fb0c28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0c28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 -- 172.21.15.27:0/491337662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 msgr2=0x5580fb0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 0x5580fb0b5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 -- 172.21.15.27:0/491337662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580fb0a4430 con 0x5580faceb590 2024-04-25T13:40:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f716b7fe640 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b5c70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f71640a3890 tx=0x7f71640a37e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebf801f01eb9936 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f7160ff9640 1 -- 172.21.15.27:0/491337662 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640a7810 con 0x5580faceb590 2024-04-25T13:40:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.980+0000 7f7160ff9640 1 -- 172.21.15.27:0/491337662 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71640abbc0 con 0x5580faceb590 2024-04-25T13:40:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f7160ff9640 1 -- 172.21.15.27:0/491337662 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640b01e0 con 0x5580faceb590 2024-04-25T13:40:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 -- 172.21.15.27:0/491337662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 msgr2=0x5580fb0b5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b5c70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f71640a3890 tx=0x7f71640a37e0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 -- 172.21.15.27:0/491337662 shutdown_connections 2024-04-25T13:40:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0b5c70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 0x5580fb0b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 --2- 172.21.15.27:0/491337662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0c28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 -- 172.21.15.27:0/491337662 >> 172.21.15.27:0/491337662 conn(0x5580fad003c0 msgr2=0x5580fb0a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 -- 172.21.15.27:0/491337662 shutdown_connections 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.981+0000 7f71783b7740 1 -- 172.21.15.27:0/491337662 wait complete. 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 1 -- start start 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 1 Processor -- start 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0b9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 0x5580fb0b9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0bc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5580fb0c4e30 con 0x5580faceb590 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5580fb0c4760 con 0x5580face33b0 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5580fb0c5500 con 0x5580faccae40 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71783b7740 10 client.0 _ll_register_callbacks cb 0x7f7154009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0bc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0bc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38478/0 (socket says 172.21.15.27:38478) 2024-04-25T13:40:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716b7fe640 1 -- 172.21.15.27:0/1388260158 learned_addr learned my addr 172.21.15.27:0/1388260158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716affd640 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0b9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f71627fc640 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 0x5580fb0b9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716b7fe640 1 -- 172.21.15.27:0/1388260158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 msgr2=0x5580fb0b9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716b7fe640 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0b9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716b7fe640 1 -- 172.21.15.27:0/1388260158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 msgr2=0x5580fb0b9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716b7fe640 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 0x5580fb0b9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716b7fe640 1 -- 172.21.15.27:0/1388260158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580fb0b20c0 con 0x5580faceb590 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.982+0000 7f716affd640 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0b9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f716b7fe640 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0bc970 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f71640a4a80 tx=0x7f71640b0410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b3bde74c22c1b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71527fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71527fc640 1 -- 172.21.15.27:0/1388260158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640c0070 con 0x5580faceb590 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71527fc640 1 -- 172.21.15.27:0/1388260158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f7164010bb0 con 0x5580faceb590 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 1 -- 172.21.15.27:0/1388260158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71640abbc0 con 0x5580faceb590 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71527fc640 1 -- 172.21.15.27:0/1388260158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f7164087050 con 0x5580faceb590 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 10 client.15057 fetch_fsmap learned FSMap version 19 2024-04-25T13:40:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 1 -- 172.21.15.27:0/1388260158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5580fb0bdfe0 con 0x5580faceb590 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71527fc640 1 -- 172.21.15.27:0/1388260158 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71640abbc0 con 0x5580faceb590 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71527fc640 1 -- 172.21.15.27:0/1388260158 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f71640de070 con 0x5580faceb590 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 10 client.15057 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 10 client.15057 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 20 client.15057 populate_metadata read hostname 'smithi027' 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580fad0c270 0x5580fb1311d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 4 client.15057 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.983+0000 7f71783b7740 1 -- 172.21.15.27:0/1388260158 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "status"}) -- 0x5580fb01a1c0 con 0x5580fad0c270 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.984+0000 7f716affd640 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580fad0c270 0x5580fb1311d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.984+0000 7f716affd640 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580fad0c270 0x5580fb1311d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=660c2bc0b9e9d810 server_cookie=a3b3daf0db091020 in_seq=0 out_seq=0 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.984+0000 7f71527fc640 10 client.15057 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.984+0000 7f71527fc640 1 -- 172.21.15.27:0/1388260158 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5580fb01a1c0 con 0x5580fad0c270 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.984+0000 7f71527fc640 10 client.15057 handle_command_reply: tid=0 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.984+0000 7f71783b7740 1 client.15057 shutdown 2024-04-25T13:40:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.984+0000 7f71783b7740 1 -- 172.21.15.27:0/1388260158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 msgr2=0x5580fb0bc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.984+0000 7f71783b7740 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0bc970 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f71640a4a80 tx=0x7f71640b0410 comp rx=0 tx=0).stop 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 1 -- 172.21.15.27:0/1388260158 shutdown_connections 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580fad0c270 0x5580fb1311d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580faceb590 0x5580fb0bc970 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580face33b0 0x5580fb0b9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71527fc640 0 client.15057 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 1 --2- 172.21.15.27:0/1388260158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580faccae40 0x5580fb0b9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 1 -- 172.21.15.27:0/1388260158 >> 172.21.15.27:0/1388260158 conn(0x5580fad003c0 msgr2=0x5580fb0a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 1 -- 172.21.15.27:0/1388260158 shutdown_connections 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 1 -- 172.21.15.27:0/1388260158 wait complete. 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f71783b7740 20 client.15057 trim_cache size 0 max 16384 2024-04-25T13:40:17.990 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "db01fc44-0307-11ef-bc93-c7b262605968", 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24560, 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 92.628836337999999, 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 19, 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 78, 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 76, 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 99.212795864 2024-04-25T13:40:17.991 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:40:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 -- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71580afaf0 msgr2=0x7f71580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71580afaf0 0x7f71580b2000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x5580fb0ae540 tx=0x7f7154012040 comp rx=0 tx=0).stop 2024-04-25T13:40:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 -- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1098a0 msgr2=0x7f716c102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1098a0 0x7f716c102610 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f716400eb40 tx=0x7f716400c2d0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 -- 172.21.15.27:0/3430137459 shutdown_connections 2024-04-25T13:40:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71580afaf0 0x7f71580b2000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f716c1098a0 0x7f716c102610 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f716c106f40 0x7f716c1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 --2- 172.21.15.27:0/3430137459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f716c1045e0 0x7f716c103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 -- 172.21.15.27:0/3430137459 >> 172.21.15.27:0/3430137459 conn(0x7f716c0fdbc0 msgr2=0x7f716c1055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 -- 172.21.15.27:0/3430137459 shutdown_connections 2024-04-25T13:40:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:17.985+0000 7f7171508640 1 -- 172.21.15.27:0/3430137459 wait complete. 2024-04-25T13:40:17.997 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T13:40:17.997 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T13:40:18.007 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T13:40:18.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T13:40:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:17 smithi027 ceph-mon[29140]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-25T13:40:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:17 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3006660248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:17 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1850752341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:17 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/378980667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:17 smithi122 ceph-mon[32843]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-25T13:40:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:17 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3006660248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:17 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1850752341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:17 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/378980667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:17 smithi045 ceph-mon[32731]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-25T13:40:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:17 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3006660248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:18.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:17 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1850752341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:18.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:17 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/378980667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:20.052 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:19 smithi027 ceph-mon[29140]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 -- 172.21.15.27:0/730691854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104bb0 msgr2=0x7f9af810b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 --2- 172.21.15.27:0/730691854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104bb0 0x7f9af810b460 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9af000b0a0 tx=0x7f9af000b3b0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 -- 172.21.15.27:0/730691854 shutdown_connections 2024-04-25T13:40:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 --2- 172.21.15.27:0/730691854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af81030e0 0x7f9af81034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 --2- 172.21.15.27:0/730691854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104bb0 0x7f9af810b460 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 --2- 172.21.15.27:0/730691854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af8104210 0x7f9af8104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 -- 172.21.15.27:0/730691854 >> 172.21.15.27:0/730691854 conn(0x7f9af80fe740 msgr2=0x7f9af8100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 -- 172.21.15.27:0/730691854 shutdown_connections 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 -- 172.21.15.27:0/730691854 wait complete. 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 Processor -- start 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.173+0000 7f9b001d2700 1 -- start start 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9b001d2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af81030e0 0x7f9af8199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9b001d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104210 0x7f9af8199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9b001d2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af8104bb0 0x7f9af81a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9b001d2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9af810eb60 con 0x7f9af8104210 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9b001d2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9af810e9e0 con 0x7f9af8104bb0 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9b001d2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9af810ece0 con 0x7f9af81030e0 2024-04-25T13:40:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104210 0x7f9af8199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104210 0x7f9af8199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38502/0 (socket says 172.21.15.27:38502) 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 -- 172.21.15.27:0/452910651 learned_addr learned my addr 172.21.15.27:0/452910651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afe76f700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af8104bb0 0x7f9af81a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afdf6e700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af81030e0 0x7f9af8199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 -- 172.21.15.27:0/452910651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af81030e0 msgr2=0x7f9af8199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af81030e0 0x7f9af8199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 -- 172.21.15.27:0/452910651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af8104bb0 msgr2=0x7f9af81a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af8104bb0 0x7f9af81a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 -- 172.21.15.27:0/452910651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af81a1180 con 0x7f9af8104210 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afdf6e700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af81030e0 0x7f9af8199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.174+0000 7f9afd76d700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104210 0x7f9af8199b60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9af400d980 tx=0x7f9af400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.175+0000 7f9aeeffd700 1 -- 172.21.15.27:0/452910651 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af4016070 con 0x7f9af8104210 2024-04-25T13:40:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.175+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af81a1470 con 0x7f9af8104210 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.175+0000 7f9aeeffd700 1 -- 172.21.15.27:0/452910651 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9af4004830 con 0x7f9af8104210 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.175+0000 7f9aeeffd700 1 -- 172.21.15.27:0/452910651 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af40054c0 con 0x7f9af8104210 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.175+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af81a19b0 con 0x7f9af8104210 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.176+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9af8051190 con 0x7f9af8104210 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.177+0000 7f9aeeffd700 1 -- 172.21.15.27:0/452910651 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9af4004220 con 0x7f9af8104210 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.178+0000 7f9aeeffd700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ae40afb80 0x7f9ae40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.178+0000 7f9afdf6e700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ae40afb80 0x7f9ae40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.179+0000 7f9aeeffd700 1 -- 172.21.15.27:0/452910651 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9af40deea0 con 0x7f9af8104210 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.179+0000 7f9afdf6e700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ae40afb80 0x7f9ae40b2090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9af8104110 tx=0x7f9ae800b670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.182+0000 7f9aeeffd700 1 -- 172.21.15.27:0/452910651 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9af401f020 con 0x7f9af8104210 2024-04-25T13:40:20.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:19 smithi122 ceph-mon[32843]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:20.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:19 smithi045 ceph-mon[32731]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.410+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f9af8063ad0 con 0x7f9ae40afb80 2024-04-25T13:40:20.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.412+0000 7f9aeeffd700 1 -- 172.21.15.27:0/452910651 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f9af8063ad0 con 0x7f9ae40afb80 2024-04-25T13:40:20.411 INFO:teuthology.orchestra.run.smithi027.stdout:Backend: cephadm 2024-04-25T13:40:20.412 INFO:teuthology.orchestra.run.smithi027.stdout:Available: Yes 2024-04-25T13:40:20.412 INFO:teuthology.orchestra.run.smithi027.stdout:Paused: No 2024-04-25T13:40:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ae40afb80 msgr2=0x7f9ae40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ae40afb80 0x7f9ae40b2090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9af8104110 tx=0x7f9ae800b670 comp rx=0 tx=0).stop 2024-04-25T13:40:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104210 msgr2=0x7f9af8199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104210 0x7f9af8199b60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9af400d980 tx=0x7f9af400dc90 comp rx=0 tx=0).stop 2024-04-25T13:40:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 shutdown_connections 2024-04-25T13:40:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ae40afb80 0x7f9ae40b2090 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af81030e0 0x7f9af8199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9af8104bb0 0x7f9af81a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 --2- 172.21.15.27:0/452910651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9af8104210 0x7f9af8199b60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 >> 172.21.15.27:0/452910651 conn(0x7f9af80fe740 msgr2=0x7f9af8107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 shutdown_connections 2024-04-25T13:40:20.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:20.414+0000 7f9b001d2700 1 -- 172.21.15.27:0/452910651 wait complete. 2024-04-25T13:40:21.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T13:40:21.404 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:22.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:21 smithi027 ceph-mon[29140]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:22.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:21 smithi027 ceph-mon[29140]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:22.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:21 smithi122 ceph-mon[32843]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:22.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:21 smithi122 ceph-mon[32843]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:22.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:21 smithi045 ceph-mon[32731]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:22.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:21 smithi045 ceph-mon[32731]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 -- 172.21.15.27:0/2986085041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103d80 msgr2=0x7f396c104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 --2- 172.21.15.27:0/2986085041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103d80 0x7f396c104190 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f396400b0a0 tx=0x7f396400b3b0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 -- 172.21.15.27:0/2986085041 shutdown_connections 2024-04-25T13:40:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 --2- 172.21.15.27:0/2986085041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103d80 0x7f396c104190 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 --2- 172.21.15.27:0/2986085041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396c102ae0 0x7f396c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 --2- 172.21.15.27:0/2986085041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396c103430 0x7f396c103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 -- 172.21.15.27:0/2986085041 >> 172.21.15.27:0/2986085041 conn(0x7f396c0fe730 msgr2=0x7f396c100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 -- 172.21.15.27:0/2986085041 shutdown_connections 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.379+0000 7f39726b1700 1 -- 172.21.15.27:0/2986085041 wait complete. 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f39726b1700 1 Processor -- start 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f39726b1700 1 -- start start 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f39726b1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396c102ae0 0x7f396c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f39726b1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103430 0x7f396c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f39726b1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396c103d80 0x7f396c1a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f39726b1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f396c111e50 con 0x7f396c103430 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f39726b1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f396c111cd0 con 0x7f396c102ae0 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f39726b1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f396c111fd0 con 0x7f396c103d80 2024-04-25T13:40:23.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f396b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103430 0x7f396c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f396b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103430 0x7f396c19ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38512/0 (socket says 172.21.15.27:38512) 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f396b7fe700 1 -- 172.21.15.27:0/514876479 learned_addr learned my addr 172.21.15.27:0/514876479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f3970c4e700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396c103d80 0x7f396c1a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f396bfff700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396c102ae0 0x7f396c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f396b7fe700 1 -- 172.21.15.27:0/514876479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396c103d80 msgr2=0x7f396c1a3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f396b7fe700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396c103d80 0x7f396c1a3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f396b7fe700 1 -- 172.21.15.27:0/514876479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396c102ae0 msgr2=0x7f396c19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.380+0000 7f396b7fe700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396c102ae0 0x7f396c19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.381+0000 7f396b7fe700 1 -- 172.21.15.27:0/514876479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c1a43a0 con 0x7f396c103430 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.381+0000 7f396b7fe700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103430 0x7f396c19ce10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f396000c9b0 tx=0x7f396000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.381+0000 7f39697fa700 1 -- 172.21.15.27:0/514876479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960007b00 con 0x7f396c103430 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.381+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c1a4690 con 0x7f396c103430 2024-04-25T13:40:23.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.381+0000 7f39697fa700 1 -- 172.21.15.27:0/514876479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3960007ca0 con 0x7f396c103430 2024-04-25T13:40:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.381+0000 7f39697fa700 1 -- 172.21.15.27:0/514876479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960018730 con 0x7f396c103430 2024-04-25T13:40:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.381+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c1a4c20 con 0x7f396c103430 2024-04-25T13:40:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.382+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c051190 con 0x7f396c103430 2024-04-25T13:40:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.383+0000 7f39697fa700 1 -- 172.21.15.27:0/514876479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f396000f450 con 0x7f396c103430 2024-04-25T13:40:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.384+0000 7f39697fa700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39540afb80 0x7f39540b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:23.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.385+0000 7f396bfff700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39540afb80 0x7f39540b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.385+0000 7f39697fa700 1 -- 172.21.15.27:0/514876479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f39600deca0 con 0x7f396c103430 2024-04-25T13:40:23.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.385+0000 7f396bfff700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39540afb80 0x7f39540b2090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f395c005fd0 tx=0x7f395c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:23.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.389+0000 7f39697fa700 1 -- 172.21.15.27:0/514876479 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3960014030 con 0x7f396c103430 2024-04-25T13:40:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.608+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f396c063ad0 con 0x7f39540afb80 2024-04-25T13:40:23.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.622+0000 7f39697fa700 1 -- 172.21.15.27:0/514876479 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f396c063ad0 con 0x7f39540afb80 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.a smithi027 running (108s) 17s ago 107s 81.6M - 19.0.0-3291-g994e2f20 d53c96779a29 e939bfeeb51a 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.b smithi045 running (105s) 38s ago 104s 96.0M - 19.0.0-3291-g994e2f20 d53c96779a29 172f6a43af02 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.c smithi122 running (119s) 37s ago 119s 80.4M - 19.0.0-3291-g994e2f20 d53c96779a29 83de7562972e 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.d smithi027 running (2m) 17s ago 2m 81.0M - 19.0.0-3291-g994e2f20 d53c96779a29 1ee835dc342c 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.e smithi045 running (2m) 38s ago 2m 82.9M - 19.0.0-3291-g994e2f20 d53c96779a29 fae0190ee89d 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.f smithi122 running (2m) 37s ago 2m 83.7M - 19.0.0-3291-g994e2f20 d53c96779a29 f8e414b7cf09 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.g smithi027 running (111s) 17s ago 110s 80.7M - 19.0.0-3291-g994e2f20 d53c96779a29 c73ebae03d8e 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.h smithi045 running (116s) 38s ago 116s 80.5M - 19.0.0-3291-g994e2f20 d53c96779a29 e41878edd41f 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.i smithi122 running (114s) 37s ago 113s 84.7M - 19.0.0-3291-g994e2f20 d53c96779a29 115a3ae05213 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.j smithi027 running (2m) 17s ago 2m 82.2M - 19.0.0-3291-g994e2f20 d53c96779a29 54604d426a1e 2024-04-25T13:40:23.622 INFO:teuthology.orchestra.run.smithi027.stdout:mds.k smithi045 running (2m) 38s ago 2m 82.9M - 19.0.0-3291-g994e2f20 d53c96779a29 4235b5a6ead6 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:mds.l smithi122 running (2m) 37s ago 2m 80.5M - 19.0.0-3291-g994e2f20 d53c96779a29 d2d867b1f50e 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.x smithi027 *:9283,8765 running (9m) 17s ago 9m 583M - 19.0.0-3291-g994e2f20 d53c96779a29 e2e0c7abc523 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.y smithi045 *:8443,8765 running (7m) 38s ago 7m 520M - 19.0.0-3291-g994e2f20 d53c96779a29 d464e8b57aed 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.z smithi122 *:8443,8765 running (7m) 37s ago 7m 519M - 19.0.0-3291-g994e2f20 d53c96779a29 2068ea4a75cf 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a smithi027 running (9m) 17s ago 9m 119M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 d3ac28c1a0c9 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b smithi045 running (8m) 38s ago 8m 106M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 b396fa368b63 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c smithi122 running (8m) 37s ago 8m 107M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 3705a47103b4 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (7m) 17s ago 7m 113M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 a8527e9e426e 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi045 running (7m) 38s ago 7m 126M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 fb9efe77cef8 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi122 running (6m) 37s ago 6m 113M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 68daadcd3a72 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi027 running (6m) 17s ago 6m 121M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 74f63193d562 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi045 running (5m) 38s ago 5m 123M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 2b8f3d445f64 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (5m) 37s ago 5m 110M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 9082a834a57b 2024-04-25T13:40:23.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 smithi027 running (4m) 17s ago 4m 125M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 dd37d4fcf847 2024-04-25T13:40:23.624 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 smithi045 running (4m) 38s ago 4m 121M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 5273e10d557d 2024-04-25T13:40:23.624 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 smithi122 running (3m) 37s ago 3m 119M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 833f4ece13f2 2024-04-25T13:40:23.624 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 smithi027 running (3m) 17s ago 3m 124M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 81a7e02bbb16 2024-04-25T13:40:23.624 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 smithi045 running (3m) 38s ago 3m 113M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 0e8867f9af34 2024-04-25T13:40:23.624 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 smithi122 running (2m) 37s ago 2m 119M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 0c1a60953590 2024-04-25T13:40:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39540afb80 msgr2=0x7f39540b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39540afb80 0x7f39540b2090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f395c005fd0 tx=0x7f395c00a560 comp rx=0 tx=0).stop 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103430 msgr2=0x7f396c19ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103430 0x7f396c19ce10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f396000c9b0 tx=0x7f396000cd70 comp rx=0 tx=0).stop 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 shutdown_connections 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39540afb80 0x7f39540b2090 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396c103d80 0x7f396c1a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f396c102ae0 0x7f396c19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 --2- 172.21.15.27:0/514876479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f396c103430 0x7f396c19ce10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 >> 172.21.15.27:0/514876479 conn(0x7f396c0fe730 msgr2=0x7f396c100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 shutdown_connections 2024-04-25T13:40:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:23.624+0000 7f39726b1700 1 -- 172.21.15.27:0/514876479 wait complete. 2024-04-25T13:40:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:23 smithi122 ceph-mon[32843]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:23 smithi045 ceph-mon[32731]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:23 smithi027 ceph-mon[29140]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:24.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T13:40:24.722 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:25.100 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:24 smithi027 ceph-mon[29140]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:25.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:24 smithi122 ceph-mon[32843]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:25.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:24 smithi045 ceph-mon[32731]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:26.306 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:25 smithi027 ceph-mon[29140]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:25 smithi122 ceph-mon[32843]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:25 smithi045 ceph-mon[32731]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 -- 172.21.15.27:0/4079952860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d01045c0 msgr2=0x7f55d01049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/4079952860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d01045c0 0x7f55d01049d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0099c0 tx=0x7f55bc009cd0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 -- 172.21.15.27:0/4079952860 shutdown_connections 2024-04-25T13:40:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/4079952860 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55d0104f10 0x7f55d010b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/4079952860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55d01036c0 0x7f55d0103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/4079952860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d01045c0 0x7f55d01049d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 -- 172.21.15.27:0/4079952860 >> 172.21.15.27:0/4079952860 conn(0x7f55d00fded0 msgr2=0x7f55d0100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 -- 172.21.15.27:0/4079952860 shutdown_connections 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 -- 172.21.15.27:0/4079952860 wait complete. 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 Processor -- start 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.558+0000 7f55d6fe4700 1 -- start start 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d6fe4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55d01036c0 0x7f55d0102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d6fe4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55d01045c0 0x7f55d019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d6fe4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d0104f10 0x7f55d01a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d6fe4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f55d010ed90 con 0x7f55d0104f10 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d6fe4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f55d010ec10 con 0x7f55d01036c0 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d6fe4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f55d010ef10 con 0x7f55d01045c0 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d5581700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d0104f10 0x7f55d01a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d5581700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d0104f10 0x7f55d01a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51790/0 (socket says 172.21.15.27:51790) 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d5581700 1 -- 172.21.15.27:0/544566269 learned_addr learned my addr 172.21.15.27:0/544566269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d4d80700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55d01036c0 0x7f55d0102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55cffff700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55d01045c0 0x7f55d019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:26.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d5581700 1 -- 172.21.15.27:0/544566269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55d01045c0 msgr2=0x7f55d019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d5581700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55d01045c0 0x7f55d019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d5581700 1 -- 172.21.15.27:0/544566269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55d01036c0 msgr2=0x7f55d0102d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d5581700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55d01036c0 0x7f55d0102d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.559+0000 7f55d5581700 1 -- 172.21.15.27:0/544566269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d01a2510 con 0x7f55d0104f10 2024-04-25T13:40:26.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.560+0000 7f55d5581700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d0104f10 0x7f55d01a1ce0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f55c8007ee0 tx=0x7f55c800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.560+0000 7f55cdffb700 1 -- 172.21.15.27:0/544566269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8004bb0 con 0x7f55d0104f10 2024-04-25T13:40:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.560+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d01a2800 con 0x7f55d0104f10 2024-04-25T13:40:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.560+0000 7f55cdffb700 1 -- 172.21.15.27:0/544566269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55c8004d50 con 0x7f55d0104f10 2024-04-25T13:40:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.560+0000 7f55cdffb700 1 -- 172.21.15.27:0/544566269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8005920 con 0x7f55d0104f10 2024-04-25T13:40:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.560+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d01a2d40 con 0x7f55d0104f10 2024-04-25T13:40:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.561+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d0051190 con 0x7f55d0104f10 2024-04-25T13:40:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.563+0000 7f55cdffb700 1 -- 172.21.15.27:0/544566269 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f55c801a020 con 0x7f55d0104f10 2024-04-25T13:40:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.563+0000 7f55cdffb700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55c00afc00 0x7f55c00b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.563+0000 7f55d4d80700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55c00afc00 0x7f55c00b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.563+0000 7f55cdffb700 1 -- 172.21.15.27:0/544566269 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f55c80de890 con 0x7f55d0104f10 2024-04-25T13:40:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.564+0000 7f55d4d80700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55c00afc00 0x7f55c00b2110 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f55c4005950 tx=0x7f55c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.567+0000 7f55cdffb700 1 -- 172.21.15.27:0/544566269 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f55c8010070 con 0x7f55d0104f10 2024-04-25T13:40:26.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.786+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f55d002e7d0 con 0x7f55c00afc00 2024-04-25T13:40:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.793+0000 7f55cdffb700 1 -- 172.21.15.27:0/544566269 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f55d002e7d0 con 0x7f55c00afc00 2024-04-25T13:40:26.793 INFO:teuthology.orchestra.run.smithi027.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T13:40:26.793 INFO:teuthology.orchestra.run.smithi027.stdout:mds.all 12/12 41s ago 2m smithi027=a;smithi027=d;smithi027=g;smithi027=j;smithi045=b;smithi045=e;smithi045=h;smithi045=k;smithi122=c;smithi122=f;smithi122=i;smithi122=l;count:12 2024-04-25T13:40:26.793 INFO:teuthology.orchestra.run.smithi027.stdout:mgr 3/3 41s ago 7m smithi027=x;smithi045=y;smithi122=z;count:3 2024-04-25T13:40:26.793 INFO:teuthology.orchestra.run.smithi027.stdout:mon 3/3 41s ago 8m smithi027:172.21.15.27=a;smithi045:172.21.15.45=b;smithi122:172.21.15.122=c;count:3 2024-04-25T13:40:26.793 INFO:teuthology.orchestra.run.smithi027.stdout:osd 12 41s ago - 2024-04-25T13:40:26.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55c00afc00 msgr2=0x7f55c00b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:26.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55c00afc00 0x7f55c00b2110 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f55c4005950 tx=0x7f55c400b410 comp rx=0 tx=0).stop 2024-04-25T13:40:26.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d0104f10 msgr2=0x7f55d01a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:26.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d0104f10 0x7f55d01a1ce0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f55c8007ee0 tx=0x7f55c800d3b0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 shutdown_connections 2024-04-25T13:40:26.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f55c00afc00 0x7f55c00b2110 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d0104f10 0x7f55d01a1ce0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55d01036c0 0x7f55d0102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 --2- 172.21.15.27:0/544566269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55d01045c0 0x7f55d019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 >> 172.21.15.27:0/544566269 conn(0x7f55d00fded0 msgr2=0x7f55d01077f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 shutdown_connections 2024-04-25T13:40:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:26.794+0000 7f55d6fe4700 1 -- 172.21.15.27:0/544566269 wait complete. 2024-04-25T13:40:27.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T13:40:27.886 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:28.244 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:27 smithi027 ceph-mon[29140]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:28.244 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:27 smithi027 ceph-mon[29140]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:28.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:27 smithi122 ceph-mon[32843]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:28.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:27 smithi122 ceph-mon[32843]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:28.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:27 smithi045 ceph-mon[32731]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:28.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:27 smithi045 ceph-mon[32731]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 -- 172.21.15.27:0/3133893732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614107380 msgr2=0x7f6614109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 --2- 172.21.15.27:0/3133893732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614107380 0x7f6614109840 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6600009990 tx=0x7f6600009ca0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 -- 172.21.15.27:0/3133893732 shutdown_connections 2024-04-25T13:40:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 --2- 172.21.15.27:0/3133893732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6614109ed0 0x7f661410c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 --2- 172.21.15.27:0/3133893732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6614076560 0x7f6614106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 --2- 172.21.15.27:0/3133893732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614107380 0x7f6614109840 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 -- 172.21.15.27:0/3133893732 >> 172.21.15.27:0/3133893732 conn(0x7f66140fe760 msgr2=0x7f6614100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 -- 172.21.15.27:0/3133893732 shutdown_connections 2024-04-25T13:40:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.741+0000 7f661b215700 1 -- 172.21.15.27:0/3133893732 wait complete. 2024-04-25T13:40:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f661b215700 1 Processor -- start 2024-04-25T13:40:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f661b215700 1 -- start start 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f661b215700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6614076560 0x7f66141045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f661b215700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6614107380 0x7f6614102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f661b215700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614109ed0 0x7f66141031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f661b215700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f661410fc70 con 0x7f6614109ed0 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f661b215700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f661410faf0 con 0x7f6614107380 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f661b215700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f661410fdf0 con 0x7f6614076560 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f66197b2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614109ed0 0x7f66141031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f6618fb1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6614076560 0x7f66141045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f6618fb1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6614076560 0x7f66141045b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39378/0 (socket says 172.21.15.27:39378) 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f6618fb1700 1 -- 172.21.15.27:0/1677606862 learned_addr learned my addr 172.21.15.27:0/1677606862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.742+0000 7f6613fff700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6614107380 0x7f6614102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f66197b2700 1 -- 172.21.15.27:0/1677606862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6614076560 msgr2=0x7f66141045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f66197b2700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6614076560 0x7f66141045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f66197b2700 1 -- 172.21.15.27:0/1677606862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6614107380 msgr2=0x7f6614102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f66197b2700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6614107380 0x7f6614102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f66197b2700 1 -- 172.21.15.27:0/1677606862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6614103870 con 0x7f6614109ed0 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f66197b2700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614109ed0 0x7f66141031f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f660c00cc50 tx=0x7f660c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f6611ffb700 1 -- 172.21.15.27:0/1677606862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c016020 con 0x7f6614109ed0 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66141a48b0 con 0x7f6614109ed0 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f6611ffb700 1 -- 172.21.15.27:0/1677606862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f660c012420 con 0x7f6614109ed0 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f6611ffb700 1 -- 172.21.15.27:0/1677606862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c0115a0 con 0x7f6614109ed0 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.743+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66141a4c50 con 0x7f6614109ed0 2024-04-25T13:40:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.744+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6614051190 con 0x7f6614109ed0 2024-04-25T13:40:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.746+0000 7f6611ffb700 1 -- 172.21.15.27:0/1677606862 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f660c019020 con 0x7f6614109ed0 2024-04-25T13:40:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.747+0000 7f6611ffb700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66040afb80 0x7f66040b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.747+0000 7f6618fb1700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66040afb80 0x7f66040b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.747+0000 7f6611ffb700 1 -- 172.21.15.27:0/1677606862 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f660c0de9d0 con 0x7f6614109ed0 2024-04-25T13:40:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.747+0000 7f6618fb1700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66040afb80 0x7f66040b2090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6608005c50 tx=0x7f6608007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.752+0000 7f6611ffb700 1 -- 172.21.15.27:0/1677606862 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f660c020020 con 0x7f6614109ed0 2024-04-25T13:40:29.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.969+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f661402e970 con 0x7f66040afb80 2024-04-25T13:40:29.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.970+0000 7f6611ffb700 1 -- 172.21.15.27:0/1677606862 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f6608007e20 con 0x7f66040afb80 2024-04-25T13:40:29.970 INFO:teuthology.orchestra.run.smithi027.stdout:HOST ADDR LABELS STATUS 2024-04-25T13:40:29.971 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 172.21.15.27 2024-04-25T13:40:29.971 INFO:teuthology.orchestra.run.smithi027.stdout:smithi045 172.21.15.45 2024-04-25T13:40:29.971 INFO:teuthology.orchestra.run.smithi027.stdout:smithi122 172.21.15.122 2024-04-25T13:40:29.971 INFO:teuthology.orchestra.run.smithi027.stdout:3 hosts in cluster 2024-04-25T13:40:29.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.972+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66040afb80 msgr2=0x7f66040b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:29.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.972+0000 7f661b215700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66040afb80 0x7f66040b2090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6608005c50 tx=0x7f6608007480 comp rx=0 tx=0).stop 2024-04-25T13:40:29.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614109ed0 msgr2=0x7f66141031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:29.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614109ed0 0x7f66141031f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f660c00cc50 tx=0x7f660c00cf60 comp rx=0 tx=0).stop 2024-04-25T13:40:29.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 shutdown_connections 2024-04-25T13:40:29.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f66040afb80 0x7f66040b2090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6614076560 0x7f66141045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6614109ed0 0x7f66141031f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 --2- 172.21.15.27:0/1677606862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6614107380 0x7f6614102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:29.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 >> 172.21.15.27:0/1677606862 conn(0x7f66140fe760 msgr2=0x7f66141057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:29.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 shutdown_connections 2024-04-25T13:40:29.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:29.973+0000 7f661b215700 1 -- 172.21.15.27:0/1677606862 wait complete. 2024-04-25T13:40:30.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:29 smithi122 ceph-mon[32843]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:30.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:29 smithi045 ceph-mon[32731]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:30.391 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:29 smithi027 ceph-mon[29140]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:30.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T13:40:30.990 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:31.261 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:30 smithi027 ceph-mon[29140]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:31.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:30 smithi122 ceph-mon[32843]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:31.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:30 smithi045 ceph-mon[32731]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:32.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:31 smithi027 ceph-mon[29140]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:32.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:31 smithi122 ceph-mon[32843]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:31 smithi045 ceph-mon[32731]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:32.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 -- 172.21.15.27:0/902031012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec109fc0 msgr2=0x7f2eec10c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:32.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/902031012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec109fc0 0x7f2eec10c3e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400b0a0 tx=0x7f2ee400b3b0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 -- 172.21.15.27:0/902031012 shutdown_connections 2024-04-25T13:40:32.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/902031012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec109fc0 0x7f2eec10c3e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/902031012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eec104d00 0x7f2eec107120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/902031012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2eec107660 0x7f2eec109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 -- 172.21.15.27:0/902031012 >> 172.21.15.27:0/902031012 conn(0x7f2eec0fe740 msgr2=0x7f2eec100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:32.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 -- 172.21.15.27:0/902031012 shutdown_connections 2024-04-25T13:40:32.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.713+0000 7f2ef29c2700 1 -- 172.21.15.27:0/902031012 wait complete. 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2ef29c2700 1 Processor -- start 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2ef29c2700 1 -- start start 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2ef29c2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec104d00 0x7f2eec19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2ef29c2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2eec107660 0x7f2eec19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2ef29c2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eec109fc0 0x7f2eec1a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2ef29c2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10fc50 con 0x7f2eec104d00 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2ef29c2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10fad0 con 0x7f2eec109fc0 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2ef29c2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10fdd0 con 0x7f2eec107660 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2eebfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec104d00 0x7f2eec19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2eebfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec104d00 0x7f2eec19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51826/0 (socket says 172.21.15.27:51826) 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2eebfff700 1 -- 172.21.15.27:0/2020698875 learned_addr learned my addr 172.21.15.27:0/2020698875 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:32.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.714+0000 7f2eeb7fe700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2eec107660 0x7f2eec19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2ef0f5f700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eec109fc0 0x7f2eec1a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2eebfff700 1 -- 172.21.15.27:0/2020698875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2eec107660 msgr2=0x7f2eec19ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2eebfff700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2eec107660 0x7f2eec19ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2eebfff700 1 -- 172.21.15.27:0/2020698875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eec109fc0 msgr2=0x7f2eec1a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2eebfff700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eec109fc0 0x7f2eec1a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2eebfff700 1 -- 172.21.15.27:0/2020698875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec1a2260 con 0x7f2eec104d00 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2ef0f5f700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eec109fc0 0x7f2eec1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2eebfff700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec104d00 0x7f2eec19a700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2edc00d980 tx=0x7f2edc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2ee97fa700 1 -- 172.21.15.27:0/2020698875 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc017070 con 0x7f2eec104d00 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2ee97fa700 1 -- 172.21.15.27:0/2020698875 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2edc009de0 con 0x7f2eec104d00 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec1a2550 con 0x7f2eec104d00 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2ee97fa700 1 -- 172.21.15.27:0/2020698875 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc00f3c0 con 0x7f2eec104d00 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.715+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec1a2a90 con 0x7f2eec104d00 2024-04-25T13:40:32.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.716+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec051190 con 0x7f2eec104d00 2024-04-25T13:40:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.718+0000 7f2ee97fa700 1 -- 172.21.15.27:0/2020698875 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2edc009930 con 0x7f2eec104d00 2024-04-25T13:40:32.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.718+0000 7f2ee97fa700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2ed40afb80 0x7f2ed40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:32.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.719+0000 7f2eeb7fe700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2ed40afb80 0x7f2ed40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:32.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.719+0000 7f2ee97fa700 1 -- 172.21.15.27:0/2020698875 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2edc0de6b0 con 0x7f2eec104d00 2024-04-25T13:40:32.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.719+0000 7f2eeb7fe700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2ed40afb80 0x7f2ed40b2090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2eec19b7b0 tx=0x7f2ee000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:32.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.722+0000 7f2ee97fa700 1 -- 172.21.15.27:0/2020698875 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2edc0a72c0 con 0x7f2eec104d00 2024-04-25T13:40:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.939+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f2eec063ad0 con 0x7f2ed40afb80 2024-04-25T13:40:32.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.941+0000 7f2ee97fa700 1 -- 172.21.15.27:0/2020698875 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f2eec063ad0 con 0x7f2ed40afb80 2024-04-25T13:40:32.941 INFO:teuthology.orchestra.run.smithi027.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T13:40:32.941 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T13:40:32.941 INFO:teuthology.orchestra.run.smithi027.stdout:smithi045 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75120098480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-25T13:40:32.941 INFO:teuthology.orchestra.run.smithi027.stdout:smithi122 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010U400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T13:40:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2ed40afb80 msgr2=0x7f2ed40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2ed40afb80 0x7f2ed40b2090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2eec19b7b0 tx=0x7f2ee000b410 comp rx=0 tx=0).stop 2024-04-25T13:40:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec104d00 msgr2=0x7f2eec19a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec104d00 0x7f2eec19a700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2edc00d980 tx=0x7f2edc00dc90 comp rx=0 tx=0).stop 2024-04-25T13:40:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 shutdown_connections 2024-04-25T13:40:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2ed40afb80 0x7f2ed40b2090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2eec109fc0 0x7f2eec1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec104d00 0x7f2eec19a700 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 --2- 172.21.15.27:0/2020698875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2eec107660 0x7f2eec19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 >> 172.21.15.27:0/2020698875 conn(0x7f2eec0fe740 msgr2=0x7f2eec105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 shutdown_connections 2024-04-25T13:40:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:32.943+0000 7f2ef29c2700 1 -- 172.21.15.27:0/2020698875 wait complete. 2024-04-25T13:40:33.787 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T13:40:33.798 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T13:40:33.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T13:40:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:34.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:33 smithi027 ceph-mon[29140]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:34.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:33 smithi027 ceph-mon[29140]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:33 smithi122 ceph-mon[32843]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:33 smithi122 ceph-mon[32843]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:34.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:33 smithi045 ceph-mon[32731]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:34.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:33 smithi045 ceph-mon[32731]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T13:40:35.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.888+0000 7f680d44b700 1 -- 172.21.15.27:0/3636368491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 msgr2=0x7f6808105c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:35.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.888+0000 7f680d44b700 1 --2- 172.21.15.27:0/3636368491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 0x7f6808105c90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009990 tx=0x7f67f8009ca0 comp rx=0 tx=0).stop 2024-04-25T13:40:35.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.889+0000 7f680d44b700 1 -- 172.21.15.27:0/3636368491 shutdown_connections 2024-04-25T13:40:35.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.889+0000 7f680d44b700 1 --2- 172.21.15.27:0/3636368491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68081048b0 0x7f680810c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:35.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.889+0000 7f680d44b700 1 --2- 172.21.15.27:0/3636368491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 0x7f6808105c90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:35.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.889+0000 7f680d44b700 1 --2- 172.21.15.27:0/3636368491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6808103e30 0x7f6808104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:35.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.889+0000 7f680d44b700 1 -- 172.21.15.27:0/3636368491 >> 172.21.15.27:0/3636368491 conn(0x7f68080ff8a0 msgr2=0x7f6808101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:35.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.889+0000 7f680d44b700 1 -- 172.21.15.27:0/3636368491 shutdown_connections 2024-04-25T13:40:35.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.889+0000 7f680d44b700 1 -- 172.21.15.27:0/3636368491 wait complete. 2024-04-25T13:40:35.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.889+0000 7f680d44b700 1 Processor -- start 2024-04-25T13:40:35.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f680d44b700 1 -- start start 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f680d44b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6808103e30 0x7f680819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f680d44b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68081048b0 0x7f680819ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f680d44b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 0x7f68081a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f680d44b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f680810fda0 con 0x7f6808105880 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f680d44b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f680810fc20 con 0x7f68081048b0 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f680d44b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f680810ff20 con 0x7f6808103e30 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 0x7f68081a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 0x7f68081a1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33056/0 (socket says 172.21.15.27:33056) 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68077fe700 1 -- 172.21.15.27:0/2517359001 learned_addr learned my addr 172.21.15.27:0/2517359001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:35.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68067fc700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68081048b0 0x7f680819ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f6806ffd700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6808103e30 0x7f680819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68077fe700 1 -- 172.21.15.27:0/2517359001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6808103e30 msgr2=0x7f680819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68077fe700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6808103e30 0x7f680819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68077fe700 1 -- 172.21.15.27:0/2517359001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68081048b0 msgr2=0x7f680819ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68077fe700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68081048b0 0x7f680819ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.890+0000 7f68077fe700 1 -- 172.21.15.27:0/2517359001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68081a2450 con 0x7f6808105880 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.891+0000 7f68077fe700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 0x7f68081a1dd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6800007ee0 tx=0x7f680000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.891+0000 7f67effff700 1 -- 172.21.15.27:0/2517359001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680000dbd0 con 0x7f6808105880 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.891+0000 7f67effff700 1 -- 172.21.15.27:0/2517359001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f680000dd70 con 0x7f6808105880 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.891+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68081a2740 con 0x7f6808105880 2024-04-25T13:40:35.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.891+0000 7f67effff700 1 -- 172.21.15.27:0/2517359001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6800005990 con 0x7f6808105880 2024-04-25T13:40:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.891+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68081a2c80 con 0x7f6808105880 2024-04-25T13:40:35.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.892+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6808051190 con 0x7f6808105880 2024-04-25T13:40:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.893+0000 7f67effff700 1 -- 172.21.15.27:0/2517359001 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f680001a020 con 0x7f6808105880 2024-04-25T13:40:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.894+0000 7f67effff700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67f00afc00 0x7f67f00b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.894+0000 7f6806ffd700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67f00afc00 0x7f67f00b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.894+0000 7f67effff700 1 -- 172.21.15.27:0/2517359001 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f68000de330 con 0x7f6808105880 2024-04-25T13:40:35.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.895+0000 7f6806ffd700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67f00afc00 0x7f67f00b2110 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f67f80038a0 tx=0x7f67f801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:35.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:35.899+0000 7f67effff700 1 -- 172.21.15.27:0/2517359001 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6800010070 con 0x7f6808105880 2024-04-25T13:40:36.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:35 smithi027 ceph-mon[29140]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.151+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f68080642a0 con 0x7f6808105880 2024-04-25T13:40:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.151+0000 7f67effff700 1 -- 172.21.15.27:0/2517359001 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) ==== 76+0+3008 (secure 0 0 0) 0x7f68000a6f40 con 0x7f6808105880 2024-04-25T13:40:36.151 INFO:teuthology.orchestra.run.smithi027.stdout:e19 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 19 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-25T13:38:44.716138+0000 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-25T13:38:48.756882+0000 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-25T13:40:36.152 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 300 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 0 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=24560} 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-25T13:40:36.153 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 0 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 24560 members: 24560 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b{0:24560} state up:active seq 3 addr [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.i{0:24517} state up:standby-replay seq 1 addr [v2:172.21.15.122:6838/3397333136,v1:172.21.15.122:6839/3397333136] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:36.154 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.27:6834/3166244728,v1:172.21.15.27:6835/3166244728] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d{-1:14556} state up:standby seq 1 addr [v2:172.21.15.27:6836/481277630,v1:172.21.15.27:6837/481277630] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.g{-1:14619} state up:standby seq 1 addr [v2:172.21.15.27:6838/2432690809,v1:172.21.15.27:6839/2432690809] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a{-1:14631} state up:standby seq 1 addr [v2:172.21.15.27:6840/1048885133,v1:172.21.15.27:6841/1048885133] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.l{-1:24430} state up:standby seq 1 addr [v2:172.21.15.122:6832/3322601046,v1:172.21.15.122:6833/3322601046] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.k{-1:24434} state up:standby seq 1 addr [v2:172.21.15.45:6832/3792078836,v1:172.21.15.45:6833/3792078836] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.e{-1:24476} state up:standby seq 1 addr [v2:172.21.15.45:6834/4225728799,v1:172.21.15.45:6835/4225728799] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.f{-1:24481} state up:standby seq 1 addr [v2:172.21.15.122:6834/1340027477,v1:172.21.15.122:6835/1340027477] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c{-1:24493} state up:standby seq 1 addr [v2:172.21.15.122:6836/1997914176,v1:172.21.15.122:6837/1997914176] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.h{-1:24512} state up:standby seq 1 addr [v2:172.21.15.45:6836/4218693179,v1:172.21.15.45:6837/4218693179] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T13:40:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67f00afc00 msgr2=0x7f67f00b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67f00afc00 0x7f67f00b2110 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f67f80038a0 tx=0x7f67f801a040 comp rx=0 tx=0).stop 2024-04-25T13:40:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 msgr2=0x7f68081a1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 0x7f68081a1dd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6800007ee0 tx=0x7f680000d3b0 comp rx=0 tx=0).stop 2024-04-25T13:40:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 shutdown_connections 2024-04-25T13:40:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67f00afc00 0x7f67f00b2110 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6808103e30 0x7f680819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6808105880 0x7f68081a1dd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 --2- 172.21.15.27:0/2517359001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68081048b0 0x7f680819ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 >> 172.21.15.27:0/2517359001 conn(0x7f68080ff8a0 msgr2=0x7f68081085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 shutdown_connections 2024-04-25T13:40:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:36.154+0000 7f680d44b700 1 -- 172.21.15.27:0/2517359001 wait complete. 2024-04-25T13:40:36.157 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:40:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:35 smithi122 ceph-mon[32843]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:36.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:35 smithi045 ceph-mon[32731]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:37.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T13:40:37.250 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:37.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:37 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2517359001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T13:40:37.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:37 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2517359001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T13:40:37.372 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:37 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2517359001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T13:40:38.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:38 smithi122 ceph-mon[32843]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:38.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:38 smithi045 ceph-mon[32731]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:38.396 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:38 smithi027 ceph-mon[29140]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 -- 172.21.15.27:0/1855704833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 msgr2=0x7f389010c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 --2- 172.21.15.27:0/1855704833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 0x7f389010c580 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f388800b0a0 tx=0x7f388800b3b0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 -- 172.21.15.27:0/1855704833 shutdown_connections 2024-04-25T13:40:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 --2- 172.21.15.27:0/1855704833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 0x7f389010c580 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 --2- 172.21.15.27:0/1855704833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3890104200 0x7f3890104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 --2- 172.21.15.27:0/1855704833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3890105330 0x7f3890105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 -- 172.21.15.27:0/1855704833 >> 172.21.15.27:0/1855704833 conn(0x7f38900ff840 msgr2=0x7f3890101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:39.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 -- 172.21.15.27:0/1855704833 shutdown_connections 2024-04-25T13:40:39.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 -- 172.21.15.27:0/1855704833 wait complete. 2024-04-25T13:40:39.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.223+0000 7f3897327700 1 Processor -- start 2024-04-25T13:40:39.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f3897327700 1 -- start start 2024-04-25T13:40:39.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f3897327700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3890104200 0x7f389019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:39.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f3897327700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3890105330 0x7f389019ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f3897327700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 0x7f38901a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f3897327700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f389010fc60 con 0x7f3890105cd0 2024-04-25T13:40:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f3897327700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f389010fae0 con 0x7f3890105330 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f3897327700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f389010fde0 con 0x7f3890104200 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38958c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 0x7f38901a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38958c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 0x7f38901a1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33074/0 (socket says 172.21.15.27:33074) 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38958c4700 1 -- 172.21.15.27:0/2037120329 learned_addr learned my addr 172.21.15.27:0/2037120329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38950c3700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3890104200 0x7f389019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38948c2700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3890105330 0x7f389019ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38958c4700 1 -- 172.21.15.27:0/2037120329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3890104200 msgr2=0x7f389019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38958c4700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3890104200 0x7f389019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38958c4700 1 -- 172.21.15.27:0/2037120329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3890105330 msgr2=0x7f389019ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38958c4700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3890105330 0x7f389019ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.224+0000 7f38958c4700 1 -- 172.21.15.27:0/2037120329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38901a2270 con 0x7f3890105cd0 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.225+0000 7f38958c4700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 0x7f38901a1bf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f38880062a0 tx=0x7f3888004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.225+0000 7f38867fc700 1 -- 172.21.15.27:0/2037120329 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888021070 con 0x7f3890105cd0 2024-04-25T13:40:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.225+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38901a2500 con 0x7f3890105cd0 2024-04-25T13:40:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.225+0000 7f38867fc700 1 -- 172.21.15.27:0/2037120329 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3888007b80 con 0x7f3890105cd0 2024-04-25T13:40:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.225+0000 7f38867fc700 1 -- 172.21.15.27:0/2037120329 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888004670 con 0x7f3890105cd0 2024-04-25T13:40:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.225+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38901a2a90 con 0x7f3890105cd0 2024-04-25T13:40:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.226+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3890051190 con 0x7f3890105cd0 2024-04-25T13:40:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.227+0000 7f38867fc700 1 -- 172.21.15.27:0/2037120329 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3888003a00 con 0x7f3890105cd0 2024-04-25T13:40:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.228+0000 7f38867fc700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f387c0afb80 0x7f387c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.228+0000 7f38950c3700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f387c0afb80 0x7f387c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.228+0000 7f38867fc700 1 -- 172.21.15.27:0/2037120329 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f388800f020 con 0x7f3890105cd0 2024-04-25T13:40:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.229+0000 7f38950c3700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f387c0afb80 0x7f387c0b2090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3890105230 tx=0x7f388c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.232+0000 7f38867fc700 1 -- 172.21.15.27:0/2037120329 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f388802a020 con 0x7f3890105cd0 2024-04-25T13:40:39.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.441+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f389002e510 con 0x7f3890105cd0 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.442+0000 7f38867fc700 1 -- 172.21.15.27:0/2037120329 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v78) ==== 56+0+5246 (secure 0 0 0) 0x7f38880a9260 con 0x7f3890105cd0 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 78 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:fsid db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-25T13:30:33.620592+0000 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-25T13:39:31.481282+0000 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:crush_version 26 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:full_ratio 0.95 2024-04-25T13:40:39.442 INFO:teuthology.orchestra.run.smithi027.stdout:backfillfull_ratio 0.9 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:nearfull_ratio 0.85 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:require_min_compat_client luminous 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:min_compat_client luminous 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:require_osd_release squid 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:stretch_mode_enabled false 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:max_osd 12 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 up in weight 1 up_from 8 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6802/342047280,v1:172.21.15.27:6803/342047280] [v2:172.21.15.27:6804/342047280,v1:172.21.15.27:6805/342047280] exists,up 24cfe666-151a-4e3b-b9eb-448622218f5e 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 up in weight 1 up_from 13 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6800/4202033932,v1:172.21.15.45:6801/4202033932] [v2:172.21.15.45:6802/4202033932,v1:172.21.15.45:6803/4202033932] exists,up 6f666f7a-74de-468f-8d21-f9aaeab6d18e 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 up in weight 1 up_from 18 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6800/1296998909,v1:172.21.15.122:6801/1296998909] [v2:172.21.15.122:6802/1296998909,v1:172.21.15.122:6803/1296998909] exists,up a9efabc5-f820-4cd2-9e55-956219ccb103 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 up in weight 1 up_from 26 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6810/1699526500,v1:172.21.15.27:6811/1699526500] [v2:172.21.15.27:6812/1699526500,v1:172.21.15.27:6813/1699526500] exists,up d0b3dc7e-9e5d-4cee-8395-0a83219ee52d 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 up in weight 1 up_from 32 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6808/2006568001,v1:172.21.15.45:6809/2006568001] [v2:172.21.15.45:6810/2006568001,v1:172.21.15.45:6811/2006568001] exists,up 75840f44-2190-409f-802e-0479ec705f0a 2024-04-25T13:40:39.443 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 up in weight 1 up_from 37 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6808/3058238339,v1:172.21.15.122:6809/3058238339] [v2:172.21.15.122:6810/3058238339,v1:172.21.15.122:6811/3058238339] exists,up 0e2af772-371f-40d5-8ff8-541551a67e38 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 up in weight 1 up_from 43 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6818/100381961,v1:172.21.15.27:6819/100381961] [v2:172.21.15.27:6820/100381961,v1:172.21.15.27:6821/100381961] exists,up 8e18badd-2f57-42b1-a367-e88ecaa3c0e8 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 up in weight 1 up_from 48 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6816/1966399238,v1:172.21.15.45:6817/1966399238] [v2:172.21.15.45:6818/1966399238,v1:172.21.15.45:6819/1966399238] exists,up 30034bf9-dc47-49be-a9c5-e8ff56a32d8e 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 up in weight 1 up_from 54 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6816/1308119675,v1:172.21.15.122:6817/1308119675] [v2:172.21.15.122:6818/1308119675,v1:172.21.15.122:6819/1308119675] exists,up 1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 up in weight 1 up_from 60 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.27:6826/3822838727,v1:172.21.15.27:6827/3822838727] [v2:172.21.15.27:6828/3822838727,v1:172.21.15.27:6829/3822838727] exists,up b43d7183-1100-4873-9968-72ba295cef58 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 up in weight 1 up_from 65 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6824/374944878,v1:172.21.15.45:6825/374944878] [v2:172.21.15.45:6826/374944878,v1:172.21.15.45:6827/374944878] exists,up 8438652f-3c76-4b50-8995-cd190d70f017 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 up in weight 1 up_from 71 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6824/3308484009,v1:172.21.15.122:6825/3308484009] [v2:172.21.15.122:6826/3308484009,v1:172.21.15.122:6827/3308484009] exists,up 23f3a823-363d-42ca-b03c-64cca2ba75d8 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.5 [8,11] 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.b [2,1] 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 2.10 [2,4,8,11] 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.0 [5,9] 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.c [3,9] 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.17 [4,8] 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_items 3.34 [4,9] 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/4203555014 expires 2024-04-26T13:31:30.235539+0000 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/1748672481 expires 2024-04-26T13:31:30.235539+0000 2024-04-25T13:40:39.444 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/4198333021 expires 2024-04-26T13:31:03.359904+0000 2024-04-25T13:40:39.445 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/2249397755 expires 2024-04-26T13:31:30.235539+0000 2024-04-25T13:40:39.445 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/4149968369 expires 2024-04-26T13:31:03.359904+0000 2024-04-25T13:40:39.445 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/1628315942 expires 2024-04-26T13:31:03.359904+0000 2024-04-25T13:40:39.445 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/3607663760 expires 2024-04-26T13:31:03.359904+0000 2024-04-25T13:40:39.445 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6800/1748672481 expires 2024-04-26T13:31:30.235539+0000 2024-04-25T13:40:39.445 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:6801/4149968369 expires 2024-04-26T13:31:03.359904+0000 2024-04-25T13:40:39.445 INFO:teuthology.orchestra.run.smithi027.stdout:blocklist 172.21.15.27:0/3971294279 expires 2024-04-26T13:31:30.235539+0000 2024-04-25T13:40:39.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f387c0afb80 msgr2=0x7f387c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f387c0afb80 0x7f387c0b2090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3890105230 tx=0x7f388c00b410 comp rx=0 tx=0).stop 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 msgr2=0x7f38901a1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 0x7f38901a1bf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f38880062a0 tx=0x7f3888004e40 comp rx=0 tx=0).stop 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 shutdown_connections 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f387c0afb80 0x7f387c0b2090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3890104200 0x7f389019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3890105cd0 0x7f38901a1bf0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 --2- 172.21.15.27:0/2037120329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3890105330 0x7f389019ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 >> 172.21.15.27:0/2037120329 conn(0x7f38900ff840 msgr2=0x7f3890108560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 shutdown_connections 2024-04-25T13:40:39.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:39.444+0000 7f3897327700 1 -- 172.21.15.27:0/2037120329 wait complete. 2024-04-25T13:40:40.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:40 smithi027 ceph-mon[29140]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:40.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:40 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2037120329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T13:40:40.257 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T13:40:40.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:40.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:40 smithi122 ceph-mon[32843]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:40.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:40 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2037120329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:40 smithi045 ceph-mon[32731]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:40.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:40 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2037120329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T13:40:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 -- 172.21.15.27:0/2036009485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac4108920 msgr2=0x7fbac410ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 --2- 172.21.15.27:0/2036009485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac4108920 0x7fbac410ad40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009ca0 tx=0x7fbabc02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 -- 172.21.15.27:0/2036009485 shutdown_connections 2024-04-25T13:40:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 --2- 172.21.15.27:0/2036009485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac4108920 0x7fbac410ad40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 --2- 172.21.15.27:0/2036009485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac4103c20 0x7fbac4104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 --2- 172.21.15.27:0/2036009485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac41029e0 0x7fbac41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 -- 172.21.15.27:0/2036009485 >> 172.21.15.27:0/2036009485 conn(0x7fbac40fdbd0 msgr2=0x7fbac4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 -- 172.21.15.27:0/2036009485 shutdown_connections 2024-04-25T13:40:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.429+0000 7fbacbe42640 1 -- 172.21.15.27:0/2036009485 wait complete. 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.430+0000 7fbacbe42640 1 Processor -- start 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.430+0000 7fbacbe42640 1 -- start start 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.430+0000 7fbacbe42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41029e0 0x7fbac419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.430+0000 7fbacbe42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac4103c20 0x7fbac419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.430+0000 7fbacbe42640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac4108920 0x7fbac41a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.430+0000 7fbacbe42640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbac410f600 con 0x7fbac41029e0 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.430+0000 7fbacbe42640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbac410f480 con 0x7fbac4103c20 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbacbe42640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbac410f780 con 0x7fbac4108920 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41029e0 0x7fbac419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41029e0 0x7fbac419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33080/0 (socket says 172.21.15.27:33080) 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 -- 172.21.15.27:0/1227262935 learned_addr learned my addr 172.21.15.27:0/1227262935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac93dd640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac4103c20 0x7fbac419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 -- 172.21.15.27:0/1227262935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac4108920 msgr2=0x7fbac41a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac4108920 0x7fbac41a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 -- 172.21.15.27:0/1227262935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac4103c20 msgr2=0x7fbac419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac4103c20 0x7fbac419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 -- 172.21.15.27:0/1227262935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac41a1ce0 con 0x7fbac41029e0 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbac9bde640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41029e0 0x7fbac419a1a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbac000c970 tx=0x7fbac000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbabaffd640 1 -- 172.21.15.27:0/1227262935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0004220 con 0x7fbac41029e0 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac41a1fd0 con 0x7fbac41029e0 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbabaffd640 1 -- 172.21.15.27:0/1227262935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbac00043c0 con 0x7fbac41029e0 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbabaffd640 1 -- 172.21.15.27:0/1227262935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac000f450 con 0x7fbac41029e0 2024-04-25T13:40:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.431+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac41a2440 con 0x7fbac41029e0 2024-04-25T13:40:40.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.432+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac4107730 con 0x7fbac41029e0 2024-04-25T13:40:40.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.434+0000 7fbabaffd640 1 -- 172.21.15.27:0/1227262935 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbac00079b0 con 0x7fbac41029e0 2024-04-25T13:40:40.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.434+0000 7fbabaffd640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afaf0 0x7fbab00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.435+0000 7fbac93dd640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afaf0 0x7fbab00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.435+0000 7fbabaffd640 1 -- 172.21.15.27:0/1227262935 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbac00dee10 con 0x7fbac41029e0 2024-04-25T13:40:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.435+0000 7fbac93dd640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afaf0 0x7fbab00b2000 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbab4005fd0 tx=0x7fbab4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.438+0000 7fbabaffd640 1 -- 172.21.15.27:0/1227262935 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbac0014030 con 0x7fbac41029e0 2024-04-25T13:40:40.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.628+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbac402dd20 con 0x7fbac41029e0 2024-04-25T13:40:40.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.630+0000 7fbabaffd640 1 -- 172.21.15.27:0/1227262935 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7fbac00a7a20 con 0x7fbac41029e0 2024-04-25T13:40:40.630 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:40.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:40.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afaf0 msgr2=0x7fbab00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afaf0 0x7fbab00b2000 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbab4005fd0 tx=0x7fbab4005850 comp rx=0 tx=0).stop 2024-04-25T13:40:40.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41029e0 msgr2=0x7fbac419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41029e0 0x7fbac419a1a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbac000c970 tx=0x7fbac000ce30 comp rx=0 tx=0).stop 2024-04-25T13:40:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 shutdown_connections 2024-04-25T13:40:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afaf0 0x7fbab00b2000 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac4108920 0x7fbac41a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac4103c20 0x7fbac419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 --2- 172.21.15.27:0/1227262935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac41029e0 0x7fbac419a1a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 >> 172.21.15.27:0/1227262935 conn(0x7fbac40fdbd0 msgr2=0x7fbac4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 shutdown_connections 2024-04-25T13:40:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.632+0000 7fbacbe42640 1 -- 172.21.15.27:0/1227262935 wait complete. 2024-04-25T13:40:40.644 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:40:40.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:40:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 -- 172.21.15.27:0/1106082379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c104d40 msgr2=0x7f460c105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 --2- 172.21.15.27:0/1106082379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c104d40 0x7f460c105150 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4600009900 tx=0x7f460002a410 comp rx=0 tx=0).stop 2024-04-25T13:40:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 -- 172.21.15.27:0/1106082379 shutdown_connections 2024-04-25T13:40:40.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 --2- 172.21.15.27:0/1106082379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f460c109a40 0x7f460c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 --2- 172.21.15.27:0/1106082379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c104d40 0x7f460c105150 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 --2- 172.21.15.27:0/1106082379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f460c103b00 0x7f460c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 -- 172.21.15.27:0/1106082379 >> 172.21.15.27:0/1106082379 conn(0x7f460c0fecf0 msgr2=0x7f460c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 -- 172.21.15.27:0/1106082379 shutdown_connections 2024-04-25T13:40:40.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.760+0000 7f4613508640 1 -- 172.21.15.27:0/1106082379 wait complete. 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4613508640 1 Processor -- start 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4613508640 1 -- start start 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4613508640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c103b00 0x7f460c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4613508640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f460c104d40 0x7f460c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4613508640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f460c109a40 0x7f460c1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4613508640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f460c10f630 con 0x7f460c103b00 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4613508640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f460c10f4b0 con 0x7f460c109a40 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4613508640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f460c10f7b0 con 0x7f460c104d40 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f46112a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c103b00 0x7f460c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:40.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f46112a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c103b00 0x7f460c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33086/0 (socket says 172.21.15.27:33086) 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f46112a4640 1 -- 172.21.15.27:0/1723335519 learned_addr learned my addr 172.21.15.27:0/1723335519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f4610aa3640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f460c104d40 0x7f460c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f46112a4640 1 -- 172.21.15.27:0/1723335519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f460c104d40 msgr2=0x7f460c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.761+0000 7f46112a4640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f460c104d40 0x7f460c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f46112a4640 1 -- 172.21.15.27:0/1723335519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f460c109a40 msgr2=0x7f460c1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f46112a4640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f460c109a40 0x7f460c1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f46112a4640 1 -- 172.21.15.27:0/1723335519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460c1a1d50 con 0x7f460c103b00 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f4610aa3640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f460c104d40 0x7f460c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f46112a4640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c103b00 0x7f460c19a210 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f460400e9f0 tx=0x7f460400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f460a7fc640 1 -- 172.21.15.27:0/1723335519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460400ccc0 con 0x7f460c103b00 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f460a7fc640 1 -- 172.21.15.27:0/1723335519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46040035f0 con 0x7f460c103b00 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f460a7fc640 1 -- 172.21.15.27:0/1723335519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460400ce60 con 0x7f460c103b00 2024-04-25T13:40:40.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f460c1a1fe0 con 0x7f460c103b00 2024-04-25T13:40:40.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.762+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f460c1a24a0 con 0x7f460c103b00 2024-04-25T13:40:40.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.763+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f460c108850 con 0x7f460c103b00 2024-04-25T13:40:40.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.764+0000 7f460a7fc640 1 -- 172.21.15.27:0/1723335519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4604003aa0 con 0x7f460c103b00 2024-04-25T13:40:40.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.765+0000 7f460a7fc640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45f40afaf0 0x7f45f40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:40.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.765+0000 7f4610aa3640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45f40afaf0 0x7f45f40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:40.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.765+0000 7f460a7fc640 1 -- 172.21.15.27:0/1723335519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f46040df0f0 con 0x7f460c103b00 2024-04-25T13:40:40.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.766+0000 7f4610aa3640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45f40afaf0 0x7f45f40b2000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4600006010 tx=0x7f4600000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:40.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.769+0000 7f460a7fc640 1 -- 172.21.15.27:0/1723335519 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46040a7ca0 con 0x7f460c103b00 2024-04-25T13:40:40.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.956+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f460c02de00 con 0x7f460c103b00 2024-04-25T13:40:40.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.956+0000 7f460a7fc640 1 -- 172.21.15.27:0/1723335519 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f46040a75a0 con 0x7f460c103b00 2024-04-25T13:40:40.956 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:40:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45f40afaf0 msgr2=0x7f45f40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45f40afaf0 0x7f45f40b2000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4600006010 tx=0x7f4600000bc0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c103b00 msgr2=0x7f460c19a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c103b00 0x7f460c19a210 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f460400e9f0 tx=0x7f460400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 shutdown_connections 2024-04-25T13:40:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f45f40afaf0 0x7f45f40b2000 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f460c109a40 0x7f460c1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f460c104d40 0x7f460c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 --2- 172.21.15.27:0/1723335519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f460c103b00 0x7f460c19a210 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 >> 172.21.15.27:0/1723335519 conn(0x7f460c0fecf0 msgr2=0x7f460c105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 shutdown_connections 2024-04-25T13:40:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:40.958+0000 7f4613508640 1 -- 172.21.15.27:0/1723335519 wait complete. 2024-04-25T13:40:40.970 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 -- 172.21.15.27:0/3273775578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc109790 msgr2=0x7f64bc10bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 --2- 172.21.15.27:0/3273775578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc109790 0x7f64bc10bca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009ca0 tx=0x7f64b402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 -- 172.21.15.27:0/3273775578 shutdown_connections 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 --2- 172.21.15.27:0/3273775578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc109790 0x7f64bc10bca0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 --2- 172.21.15.27:0/3273775578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64bc102370 0x7f64bc109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 --2- 172.21.15.27:0/3273775578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101e70 0x7f64bc106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 -- 172.21.15.27:0/3273775578 >> 172.21.15.27:0/3273775578 conn(0x7f64bc0fd380 msgr2=0x7f64bc0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 -- 172.21.15.27:0/3273775578 shutdown_connections 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.088+0000 7f64c43d5640 1 -- 172.21.15.27:0/3273775578 wait complete. 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c43d5640 1 Processor -- start 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c43d5640 1 -- start start 2024-04-25T13:40:41.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c43d5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101e70 0x7f64bc104340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c43d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc102370 0x7f64bc1029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c43d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64bc109790 0x7f64bc102f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c43d5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc10f600 con 0x7f64bc102370 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c43d5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc10f480 con 0x7f64bc109790 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c43d5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc10f780 con 0x7f64bc101e70 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c1970640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc102370 0x7f64bc1029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c1970640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc102370 0x7f64bc1029e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33114/0 (socket says 172.21.15.27:33114) 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c1970640 1 -- 172.21.15.27:0/1879669244 learned_addr learned my addr 172.21.15.27:0/1879669244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c2171640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101e70 0x7f64bc104340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.089+0000 7f64c2972640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64bc109790 0x7f64bc102f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c1970640 1 -- 172.21.15.27:0/1879669244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101e70 msgr2=0x7f64bc104340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c1970640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101e70 0x7f64bc104340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c1970640 1 -- 172.21.15.27:0/1879669244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64bc109790 msgr2=0x7f64bc102f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c1970640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64bc109790 0x7f64bc102f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c1970640 1 -- 172.21.15.27:0/1879669244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64bc103650 con 0x7f64bc102370 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c2972640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64bc109790 0x7f64bc102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c1970640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc102370 0x7f64bc1029e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00d950 tx=0x7f64ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64b37fe640 1 -- 172.21.15.27:0/1879669244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac017070 con 0x7f64bc102370 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64bc1a4260 con 0x7f64bc102370 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64b37fe640 1 -- 172.21.15.27:0/1879669244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64ac009c30 con 0x7f64bc102370 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64b37fe640 1 -- 172.21.15.27:0/1879669244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00f620 con 0x7f64bc102370 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.090+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64bc1a4700 con 0x7f64bc102370 2024-04-25T13:40:41.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.091+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64bc0feea0 con 0x7f64bc102370 2024-04-25T13:40:41.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.093+0000 7f64b37fe640 1 -- 172.21.15.27:0/1879669244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f64ac010460 con 0x7f64bc102370 2024-04-25T13:40:41.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.094+0000 7f64b37fe640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64a80afaf0 0x7f64a80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.094+0000 7f64c2171640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64a80afaf0 0x7f64a80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.094+0000 7f64b37fe640 1 -- 172.21.15.27:0/1879669244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f64ac0de830 con 0x7f64bc102370 2024-04-25T13:40:41.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.095+0000 7f64c2171640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64a80afaf0 0x7f64a80b2000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f64b8005fd0 tx=0x7f64b800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:41.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.098+0000 7f64b37fe640 1 -- 172.21.15.27:0/1879669244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64ac0a73e0 con 0x7f64bc102370 2024-04-25T13:40:41.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.285+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f64bc02dce0 con 0x7f64bc102370 2024-04-25T13:40:41.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.286+0000 7f64b37fe640 1 -- 172.21.15.27:0/1879669244 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f64bc02dce0 con 0x7f64bc102370 2024-04-25T13:40:41.286 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64a80afaf0 msgr2=0x7f64a80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64a80afaf0 0x7f64a80b2000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f64b8005fd0 tx=0x7f64b800b390 comp rx=0 tx=0).stop 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc102370 msgr2=0x7f64bc1029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc102370 0x7f64bc1029e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00d950 tx=0x7f64ac00de10 comp rx=0 tx=0).stop 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 shutdown_connections 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64a80afaf0 0x7f64a80b2000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64bc109790 0x7f64bc102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64bc102370 0x7f64bc1029e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 --2- 172.21.15.27:0/1879669244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101e70 0x7f64bc104340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 >> 172.21.15.27:0/1879669244 conn(0x7f64bc0fd380 msgr2=0x7f64bc1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 shutdown_connections 2024-04-25T13:40:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.288+0000 7f64c43d5640 1 -- 172.21.15.27:0/1879669244 wait complete. 2024-04-25T13:40:41.298 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:41 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1227262935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:41.298 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:41 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1723335519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:41.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:40:41.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:41 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1227262935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:41.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:41 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1723335519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:41 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1227262935' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:41.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:41 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1723335519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:41.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 -- 172.21.15.27:0/1374767410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95901098a0 msgr2=0x7f959010bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 --2- 172.21.15.27:0/1374767410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95901098a0 0x7f959010bcc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9588009ca0 tx=0x7f958802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 -- 172.21.15.27:0/1374767410 shutdown_connections 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 --2- 172.21.15.27:0/1374767410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95901098a0 0x7f959010bcc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 --2- 172.21.15.27:0/1374767410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9590106f40 0x7f9590109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 --2- 172.21.15.27:0/1374767410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95901045e0 0x7f9590106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 -- 172.21.15.27:0/1374767410 >> 172.21.15.27:0/1374767410 conn(0x7f95900fdbc0 msgr2=0x7f9590100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 -- 172.21.15.27:0/1374767410 shutdown_connections 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.413+0000 7f9597a9a640 1 -- 172.21.15.27:0/1374767410 wait complete. 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9597a9a640 1 Processor -- start 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9597a9a640 1 -- start start 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9597a9a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95901045e0 0x7f959019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9597a9a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9590106f40 0x7f959019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9597a9a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95901098a0 0x7f95901a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9597a9a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f959010f620 con 0x7f9590106f40 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9597a9a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f959010f4a0 con 0x7f95901045e0 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9597a9a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f959010f7a0 con 0x7f95901098a0 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9595035640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9590106f40 0x7f959019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9595035640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9590106f40 0x7f959019a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33130/0 (socket says 172.21.15.27:33130) 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.414+0000 7f9595035640 1 -- 172.21.15.27:0/569235000 learned_addr learned my addr 172.21.15.27:0/569235000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9595836640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95901045e0 0x7f959019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9595035640 1 -- 172.21.15.27:0/569235000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95901098a0 msgr2=0x7f95901a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9595035640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95901098a0 0x7f95901a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9595035640 1 -- 172.21.15.27:0/569235000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95901045e0 msgr2=0x7f959019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9595035640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95901045e0 0x7f959019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9595035640 1 -- 172.21.15.27:0/569235000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95901a1d00 con 0x7f9590106f40 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9595035640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9590106f40 0x7f959019a700 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f958000c970 tx=0x7f958000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9586ffd640 1 -- 172.21.15.27:0/569235000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580007bb0 con 0x7f9590106f40 2024-04-25T13:40:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9586ffd640 1 -- 172.21.15.27:0/569235000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9580007d50 con 0x7f9590106f40 2024-04-25T13:40:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95901a1ff0 con 0x7f9590106f40 2024-04-25T13:40:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9586ffd640 1 -- 172.21.15.27:0/569235000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958000f450 con 0x7f9590106f40 2024-04-25T13:40:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.415+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95901a2460 con 0x7f9590106f40 2024-04-25T13:40:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.416+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95900ff6e0 con 0x7f9590106f40 2024-04-25T13:40:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.417+0000 7f9586ffd640 1 -- 172.21.15.27:0/569235000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9580004f00 con 0x7f9590106f40 2024-04-25T13:40:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.418+0000 7f9586ffd640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f957c0afaf0 0x7f957c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.419+0000 7f9595836640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f957c0afaf0 0x7f957c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.419+0000 7f9586ffd640 1 -- 172.21.15.27:0/569235000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f95800ded30 con 0x7f9590106f40 2024-04-25T13:40:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.419+0000 7f9595836640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f957c0afaf0 0x7f957c0b2000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f95900757f0 tx=0x7f958c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:41.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.423+0000 7f9586ffd640 1 -- 172.21.15.27:0/569235000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9580014030 con 0x7f9590106f40 2024-04-25T13:40:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.610+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f959002dce0 con 0x7f9590106f40 2024-04-25T13:40:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.610+0000 7f9586ffd640 1 -- 172.21.15.27:0/569235000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f95800a7940 con 0x7f9590106f40 2024-04-25T13:40:41.610 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.612+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f957c0afaf0 msgr2=0x7f957c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.612+0000 7f9597a9a640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f957c0afaf0 0x7f957c0b2000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f95900757f0 tx=0x7f958c00b390 comp rx=0 tx=0).stop 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9590106f40 msgr2=0x7f959019a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9590106f40 0x7f959019a700 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f958000c970 tx=0x7f958000ce30 comp rx=0 tx=0).stop 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 shutdown_connections 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f957c0afaf0 0x7f957c0b2000 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f95901098a0 0x7f95901a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9590106f40 0x7f959019a700 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 --2- 172.21.15.27:0/569235000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95901045e0 0x7f959019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 >> 172.21.15.27:0/569235000 conn(0x7f95900fdbc0 msgr2=0x7f95901051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 shutdown_connections 2024-04-25T13:40:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.613+0000 7f9597a9a640 1 -- 172.21.15.27:0/569235000 wait complete. 2024-04-25T13:40:41.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:40:41.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 -- 172.21.15.27:0/1734621702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5148103c20 msgr2=0x7f5148104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 --2- 172.21.15.27:0/1734621702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5148103c20 0x7f5148104030 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5130009900 tx=0x7f513002a410 comp rx=0 tx=0).stop 2024-04-25T13:40:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 -- 172.21.15.27:0/1734621702 shutdown_connections 2024-04-25T13:40:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 --2- 172.21.15.27:0/1734621702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5148108920 0x7f514810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 --2- 172.21.15.27:0/1734621702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5148103c20 0x7f5148104030 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 --2- 172.21.15.27:0/1734621702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51481029e0 0x7f51481036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 -- 172.21.15.27:0/1734621702 >> 172.21.15.27:0/1734621702 conn(0x7f51480fdbd0 msgr2=0x7f5148100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 -- 172.21.15.27:0/1734621702 shutdown_connections 2024-04-25T13:40:41.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.735+0000 7f514cd27640 1 -- 172.21.15.27:0/1734621702 wait complete. 2024-04-25T13:40:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.736+0000 7f514cd27640 1 Processor -- start 2024-04-25T13:40:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.736+0000 7f514cd27640 1 -- start start 2024-04-25T13:40:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514cd27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51481029e0 0x7f514819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514cd27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5148103c20 0x7f514819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514cd27640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5148108920 0x7f51481a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514cd27640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f514810e510 con 0x7f51481029e0 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514cd27640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f514810e390 con 0x7f5148103c20 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514cd27640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f514810e690 con 0x7f5148108920 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51481029e0 0x7f514819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51481029e0 0x7f514819a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33144/0 (socket says 172.21.15.27:33144) 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 -- 172.21.15.27:0/3633815239 learned_addr learned my addr 172.21.15.27:0/3633815239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f513dd9b640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5148103c20 0x7f514819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f5146d9d640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5148108920 0x7f51481a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 -- 172.21.15.27:0/3633815239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5148108920 msgr2=0x7f51481a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5148108920 0x7f51481a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 -- 172.21.15.27:0/3633815239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5148103c20 msgr2=0x7f514819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5148103c20 0x7f514819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 -- 172.21.15.27:0/3633815239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51481a1db0 con 0x7f51481029e0 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f5146d9d640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5148108920 0x7f51481a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f514659c640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51481029e0 0x7f514819a270 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f513800d740 tx=0x7f513800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.737+0000 7f513ffff640 1 -- 172.21.15.27:0/3633815239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138017070 con 0x7f51481029e0 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.738+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51481a20a0 con 0x7f51481029e0 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.738+0000 7f513ffff640 1 -- 172.21.15.27:0/3633815239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f513800de50 con 0x7f51481029e0 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.738+0000 7f513ffff640 1 -- 172.21.15.27:0/3633815239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138003c30 con 0x7f51481029e0 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.738+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51481a2640 con 0x7f51481029e0 2024-04-25T13:40:41.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.739+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f514806a190 con 0x7f51481029e0 2024-04-25T13:40:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.740+0000 7f513ffff640 1 -- 172.21.15.27:0/3633815239 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5138003dd0 con 0x7f51481029e0 2024-04-25T13:40:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.741+0000 7f513ffff640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51340afaf0 0x7f51340b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.741+0000 7f513dd9b640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51340afaf0 0x7f51340b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.741+0000 7f513ffff640 1 -- 172.21.15.27:0/3633815239 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f51380de9c0 con 0x7f51481029e0 2024-04-25T13:40:41.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.741+0000 7f513dd9b640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51340afaf0 0x7f51340b2000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5130009230 tx=0x7f5130031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.744+0000 7f513ffff640 1 -- 172.21.15.27:0/3633815239 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51380a75d0 con 0x7f51481029e0 2024-04-25T13:40:41.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.970+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51481a2960 con 0x7f51481029e0 2024-04-25T13:40:41.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.970+0000 7f513ffff640 1 -- 172.21.15.27:0/3633815239 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f51380a6ed0 con 0x7f51481029e0 2024-04-25T13:40:41.971 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:41.971 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:40:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51340afaf0 msgr2=0x7f51340b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51340afaf0 0x7f51340b2000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5130009230 tx=0x7f5130031400 comp rx=0 tx=0).stop 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51481029e0 msgr2=0x7f514819a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51481029e0 0x7f514819a270 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f513800d740 tx=0x7f513800dc00 comp rx=0 tx=0).stop 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 shutdown_connections 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51340afaf0 0x7f51340b2000 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5148108920 0x7f51481a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5148103c20 0x7f514819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 --2- 172.21.15.27:0/3633815239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51481029e0 0x7f514819a270 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.972+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 >> 172.21.15.27:0/3633815239 conn(0x7f51480fdbd0 msgr2=0x7f5148109f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.973+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 shutdown_connections 2024-04-25T13:40:41.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:41.973+0000 7f514cd27640 1 -- 172.21.15.27:0/3633815239 wait complete. 2024-04-25T13:40:41.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 -- 172.21.15.27:0/1813243178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63881029e0 msgr2=0x7f63881036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 --2- 172.21.15.27:0/1813243178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63881029e0 0x7f63881036e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6378009900 tx=0x7f637802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 -- 172.21.15.27:0/1813243178 shutdown_connections 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 --2- 172.21.15.27:0/1813243178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6388108920 0x7f638810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 --2- 172.21.15.27:0/1813243178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6388103c20 0x7f6388104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 --2- 172.21.15.27:0/1813243178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63881029e0 0x7f63881036e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 -- 172.21.15.27:0/1813243178 >> 172.21.15.27:0/1813243178 conn(0x7f63880fdbd0 msgr2=0x7f6388100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 -- 172.21.15.27:0/1813243178 shutdown_connections 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.096+0000 7f638da42640 1 -- 172.21.15.27:0/1813243178 wait complete. 2024-04-25T13:40:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.097+0000 7f638da42640 1 Processor -- start 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.097+0000 7f638da42640 1 -- start start 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f638da42640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63881029e0 0x7f638819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f638da42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388103c20 0x7f638819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f638da42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6388108920 0x7f63881a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f638da42640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63880758c0 con 0x7f6388103c20 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f638da42640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6388075740 con 0x7f6388108920 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f638da42640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6388075a40 con 0x7f63881029e0 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388103c20 0x7f638819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388103c20 0x7f638819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33170/0 (socket says 172.21.15.27:33170) 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 -- 172.21.15.27:0/2474247927 learned_addr learned my addr 172.21.15.27:0/2474247927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63877fe640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6388108920 0x7f63881a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f6386ffd640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63881029e0 0x7f638819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 -- 172.21.15.27:0/2474247927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63881029e0 msgr2=0x7f638819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63881029e0 0x7f638819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 -- 172.21.15.27:0/2474247927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6388108920 msgr2=0x7f63881a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6388108920 0x7f63881a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 -- 172.21.15.27:0/2474247927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63881a1de0 con 0x7f6388103c20 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f6386ffd640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63881029e0 0x7f638819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f63867fc640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388103c20 0x7f638819a7e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f637c00c970 tx=0x7f637c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f638ca40640 1 -- 172.21.15.27:0/2474247927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c007bb0 con 0x7f6388103c20 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.098+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63881a20d0 con 0x7f6388103c20 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.099+0000 7f638ca40640 1 -- 172.21.15.27:0/2474247927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f637c007d50 con 0x7f6388103c20 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.099+0000 7f638ca40640 1 -- 172.21.15.27:0/2474247927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c00f450 con 0x7f6388103c20 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.099+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63881a2620 con 0x7f6388103c20 2024-04-25T13:40:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.099+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6388107730 con 0x7f6388103c20 2024-04-25T13:40:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.102+0000 7f638ca40640 1 -- 172.21.15.27:0/2474247927 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f637c004f00 con 0x7f6388103c20 2024-04-25T13:40:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.102+0000 7f638ca40640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63700afaf0 0x7f63700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.102+0000 7f6386ffd640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63700afaf0 0x7f63700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.102+0000 7f638ca40640 1 -- 172.21.15.27:0/2474247927 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f637c0ded30 con 0x7f6388103c20 2024-04-25T13:40:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.103+0000 7f6386ffd640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63700afaf0 0x7f63700b2000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f637802aa00 tx=0x7f6378031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:42.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.106+0000 7f638ca40640 1 -- 172.21.15.27:0/2474247927 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f637c014030 con 0x7f6388103c20 2024-04-25T13:40:42.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.294+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f638802dcd0 con 0x7f6388103c20 2024-04-25T13:40:42.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.296+0000 7f638ca40640 1 -- 172.21.15.27:0/2474247927 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7f637c0a7940 con 0x7f6388103c20 2024-04-25T13:40:42.296 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:42.296 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:42.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63700afaf0 msgr2=0x7f63700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63700afaf0 0x7f63700b2000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f637802aa00 tx=0x7f6378031400 comp rx=0 tx=0).stop 2024-04-25T13:40:42.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388103c20 msgr2=0x7f638819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388103c20 0x7f638819a7e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f637c00c970 tx=0x7f637c00ce30 comp rx=0 tx=0).stop 2024-04-25T13:40:42.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 shutdown_connections 2024-04-25T13:40:42.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63700afaf0 0x7f63700b2000 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6388108920 0x7f63881a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388103c20 0x7f638819a7e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 --2- 172.21.15.27:0/2474247927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63881029e0 0x7f638819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 >> 172.21.15.27:0/2474247927 conn(0x7f63880fdbd0 msgr2=0x7f6388104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:42.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 shutdown_connections 2024-04-25T13:40:42.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.298+0000 7f638da42640 1 -- 172.21.15.27:0/2474247927 wait complete. 2024-04-25T13:40:42.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:42 smithi122 ceph-mon[32843]: pgmap v343: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:42.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:42 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1879669244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:42.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:42 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/569235000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:42.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:42 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3633815239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:42.309 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:42 smithi027 ceph-mon[29140]: pgmap v343: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:42.309 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:42 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1879669244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:42.309 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:42 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/569235000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:42.309 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:42 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3633815239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:42.310 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:40:42.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:40:42.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:42 smithi045 ceph-mon[32731]: pgmap v343: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:42.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:42 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1879669244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:42.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:42 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/569235000' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:42.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:42 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3633815239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:42.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 -- 172.21.15.27:0/1768387109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858104d30 msgr2=0x7fb858105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 --2- 172.21.15.27:0/1768387109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858104d30 0x7fb858105140 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb848009900 tx=0x7fb84802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:42.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 -- 172.21.15.27:0/1768387109 shutdown_connections 2024-04-25T13:40:42.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 --2- 172.21.15.27:0/1768387109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb858109a30 0x7fb85810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 --2- 172.21.15.27:0/1768387109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858104d30 0x7fb858105140 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 --2- 172.21.15.27:0/1768387109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb858103af0 0x7fb8581047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 -- 172.21.15.27:0/1768387109 >> 172.21.15.27:0/1768387109 conn(0x7fb8580fece0 msgr2=0x7fb858101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 -- 172.21.15.27:0/1768387109 shutdown_connections 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.425+0000 7fb85fb57640 1 -- 172.21.15.27:0/1768387109 wait complete. 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85fb57640 1 Processor -- start 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85fb57640 1 -- start start 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85fb57640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858103af0 0x7fb85819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85fb57640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb858104d30 0x7fb85819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85fb57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb858109a30 0x7fb8581a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85fb57640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb85810f620 con 0x7fb858103af0 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85fb57640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb85810f4a0 con 0x7fb858109a30 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85fb57640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb85810f7a0 con 0x7fb858104d30 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85d8f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858103af0 0x7fb85819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85d8f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858103af0 0x7fb85819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33182/0 (socket says 172.21.15.27:33182) 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85d8f3640 1 -- 172.21.15.27:0/2023611292 learned_addr learned my addr 172.21.15.27:0/2023611292 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:42.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.426+0000 7fb85e0f4640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb858109a30 0x7fb8581a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85d0f2640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb858104d30 0x7fb85819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85d8f3640 1 -- 172.21.15.27:0/2023611292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb858104d30 msgr2=0x7fb85819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85d8f3640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb858104d30 0x7fb85819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85d8f3640 1 -- 172.21.15.27:0/2023611292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb858109a30 msgr2=0x7fb8581a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85d8f3640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb858109a30 0x7fb8581a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85d8f3640 1 -- 172.21.15.27:0/2023611292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8581a1d60 con 0x7fb858103af0 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85d8f3640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858103af0 0x7fb85819a220 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb854005fd0 tx=0x7fb85400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb84effd640 1 -- 172.21.15.27:0/2023611292 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854017070 con 0x7fb858103af0 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb84effd640 1 -- 172.21.15.27:0/2023611292 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb854009c30 con 0x7fb858103af0 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8581a1ff0 con 0x7fb858103af0 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb84effd640 1 -- 172.21.15.27:0/2023611292 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb854003d10 con 0x7fb858103af0 2024-04-25T13:40:42.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.427+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8581a2500 con 0x7fb858103af0 2024-04-25T13:40:42.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.428+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb858108840 con 0x7fb858103af0 2024-04-25T13:40:42.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.429+0000 7fb84effd640 1 -- 172.21.15.27:0/2023611292 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb854003eb0 con 0x7fb858103af0 2024-04-25T13:40:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.430+0000 7fb84effd640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8440afb70 0x7fb8440b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.430+0000 7fb85d0f2640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8440afb70 0x7fb8440b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.430+0000 7fb84effd640 1 -- 172.21.15.27:0/2023611292 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb8540de870 con 0x7fb858103af0 2024-04-25T13:40:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.431+0000 7fb85d0f2640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8440afb70 0x7fb8440b2080 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb848000c00 tx=0x7fb848031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:42.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.435+0000 7fb84effd640 1 -- 172.21.15.27:0/2023611292 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb854020030 con 0x7fb858103af0 2024-04-25T13:40:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.621+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb8580637a0 con 0x7fb858103af0 2024-04-25T13:40:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.621+0000 7fb84effd640 1 -- 172.21.15.27:0/2023611292 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7fb854009de0 con 0x7fb858103af0 2024-04-25T13:40:42.621 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8440afb70 msgr2=0x7fb8440b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8440afb70 0x7fb8440b2080 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb848000c00 tx=0x7fb848031400 comp rx=0 tx=0).stop 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858103af0 msgr2=0x7fb85819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858103af0 0x7fb85819a220 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb854005fd0 tx=0x7fb85400dde0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 shutdown_connections 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8440afb70 0x7fb8440b2080 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb858109a30 0x7fb8581a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb858104d30 0x7fb85819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 --2- 172.21.15.27:0/2023611292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb858103af0 0x7fb85819a220 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 >> 172.21.15.27:0/2023611292 conn(0x7fb8580fece0 msgr2=0x7fb858105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 shutdown_connections 2024-04-25T13:40:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.624+0000 7fb85fb57640 1 -- 172.21.15.27:0/2023611292 wait complete. 2024-04-25T13:40:42.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:40:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 -- 172.21.15.27:0/606160939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f8106c30 msgr2=0x7f70f8109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 --2- 172.21.15.27:0/606160939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f8106c30 0x7f70f8109110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009900 tx=0x7f70e802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:42.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 -- 172.21.15.27:0/606160939 shutdown_connections 2024-04-25T13:40:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 --2- 172.21.15.27:0/606160939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70f81097a0 0x7f70f810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 --2- 172.21.15.27:0/606160939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f8106c30 0x7f70f8109110 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 --2- 172.21.15.27:0/606160939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70f8101ea0 0x7f70f81022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 -- 172.21.15.27:0/606160939 >> 172.21.15.27:0/606160939 conn(0x7f70f80fd350 msgr2=0x7f70f80ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 -- 172.21.15.27:0/606160939 shutdown_connections 2024-04-25T13:40:42.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.750+0000 7f70ffab1640 1 -- 172.21.15.27:0/606160939 wait complete. 2024-04-25T13:40:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70ffab1640 1 Processor -- start 2024-04-25T13:40:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70ffab1640 1 -- start start 2024-04-25T13:40:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70ffab1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70f8101ea0 0x7f70f819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70ffab1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70f8106c30 0x7f70f819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70ffab1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f81097a0 0x7f70f81a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70ffab1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f70f810f630 con 0x7f70f81097a0 2024-04-25T13:40:42.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70ffab1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70f810f4b0 con 0x7f70f8106c30 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70ffab1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f70f810f7b0 con 0x7f70f8101ea0 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70fe04e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f81097a0 0x7f70f81a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70fe04e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f81097a0 0x7f70f81a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33198/0 (socket says 172.21.15.27:33198) 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70fe04e640 1 -- 172.21.15.27:0/126885494 learned_addr learned my addr 172.21.15.27:0/126885494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70fd04c640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70f8106c30 0x7f70f819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70fd84d640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70f8101ea0 0x7f70f819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.751+0000 7f70fe04e640 1 -- 172.21.15.27:0/126885494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70f8101ea0 msgr2=0x7f70f819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70fe04e640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70f8101ea0 0x7f70f819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70fe04e640 1 -- 172.21.15.27:0/126885494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70f8106c30 msgr2=0x7f70f819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70fe04e640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70f8106c30 0x7f70f819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70fe04e640 1 -- 172.21.15.27:0/126885494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f81a1d40 con 0x7f70f81097a0 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70fe04e640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f81097a0 0x7f70f81a16d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f70f000cad0 tx=0x7f70f000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70eeffd640 1 -- 172.21.15.27:0/126885494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f000ae20 con 0x7f70f81097a0 2024-04-25T13:40:42.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70eeffd640 1 -- 172.21.15.27:0/126885494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70f0005020 con 0x7f70f81097a0 2024-04-25T13:40:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f81a1fd0 con 0x7f70f81097a0 2024-04-25T13:40:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70eeffd640 1 -- 172.21.15.27:0/126885494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f000f6b0 con 0x7f70f81097a0 2024-04-25T13:40:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.752+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f81a2490 con 0x7f70f81097a0 2024-04-25T13:40:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.753+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70f80fee70 con 0x7f70f81097a0 2024-04-25T13:40:42.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.755+0000 7f70eeffd640 1 -- 172.21.15.27:0/126885494 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f70f00051d0 con 0x7f70f81097a0 2024-04-25T13:40:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.755+0000 7f70eeffd640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70e40afaf0 0x7f70e40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.756+0000 7f70fd84d640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70e40afaf0 0x7f70e40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.756+0000 7f70eeffd640 1 -- 172.21.15.27:0/126885494 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f70f00def90 con 0x7f70f81097a0 2024-04-25T13:40:42.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.756+0000 7f70fd84d640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70e40afaf0 0x7f70e40b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f70f806b520 tx=0x7f70f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:42.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.759+0000 7f70eeffd640 1 -- 172.21.15.27:0/126885494 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f70f0014030 con 0x7f70f81097a0 2024-04-25T13:40:42.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.945+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f70f80637a0 con 0x7f70f81097a0 2024-04-25T13:40:42.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.946+0000 7f70eeffd640 1 -- 172.21.15.27:0/126885494 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f70f00a7c40 con 0x7f70f81097a0 2024-04-25T13:40:42.946 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70e40afaf0 msgr2=0x7f70e40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70e40afaf0 0x7f70e40b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f70f806b520 tx=0x7f70f4005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f81097a0 msgr2=0x7f70f81a16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f81097a0 0x7f70f81a16d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f70f000cad0 tx=0x7f70f000cf90 comp rx=0 tx=0).stop 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 shutdown_connections 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70e40afaf0 0x7f70e40b2000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f81097a0 0x7f70f81a16d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70f8106c30 0x7f70f819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 --2- 172.21.15.27:0/126885494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f70f8101ea0 0x7f70f819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 >> 172.21.15.27:0/126885494 conn(0x7f70f80fd350 msgr2=0x7f70f810a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:42.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 shutdown_connections 2024-04-25T13:40:42.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:42.948+0000 7f70ffab1640 1 -- 172.21.15.27:0/126885494 wait complete. 2024-04-25T13:40:42.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:40:43.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 -- 172.21.15.27:0/987700613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38081029f0 msgr2=0x7f38081036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 --2- 172.21.15.27:0/987700613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38081029f0 0x7f38081036f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009900 tx=0x7f37fc02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:43.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 -- 172.21.15.27:0/987700613 shutdown_connections 2024-04-25T13:40:43.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 --2- 172.21.15.27:0/987700613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3808108930 0x7f380810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 --2- 172.21.15.27:0/987700613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3808103c30 0x7f3808104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 --2- 172.21.15.27:0/987700613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38081029f0 0x7f38081036f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 -- 172.21.15.27:0/987700613 >> 172.21.15.27:0/987700613 conn(0x7f38080fdbc0 msgr2=0x7f3808100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 -- 172.21.15.27:0/987700613 shutdown_connections 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.074+0000 7f380ff1f640 1 -- 172.21.15.27:0/987700613 wait complete. 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380ff1f640 1 Processor -- start 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380ff1f640 1 -- start start 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380ff1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38081029f0 0x7f3808075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380ff1f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3808103c30 0x7f380819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380ff1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3808108930 0x7f38081a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380ff1f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f380810e540 con 0x7f3808108930 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380ff1f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f380810e3c0 con 0x7f38081029f0 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380ff1f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f380810e6c0 con 0x7f3808103c30 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380e4bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3808108930 0x7f38081a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380e4bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3808108930 0x7f38081a1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33220/0 (socket says 172.21.15.27:33220) 2024-04-25T13:40:43.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380e4bc640 1 -- 172.21.15.27:0/3546098644 learned_addr learned my addr 172.21.15.27:0/3546098644 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380dcbb640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38081029f0 0x7f3808075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380d4ba640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3808103c30 0x7f380819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380e4bc640 1 -- 172.21.15.27:0/3546098644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3808103c30 msgr2=0x7f380819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380e4bc640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3808103c30 0x7f380819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380e4bc640 1 -- 172.21.15.27:0/3546098644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38081029f0 msgr2=0x7f3808075860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380e4bc640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38081029f0 0x7f3808075860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.075+0000 7f380e4bc640 1 -- 172.21.15.27:0/3546098644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38081a1d30 con 0x7f3808108930 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.076+0000 7f380e4bc640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3808108930 0x7f38081a1510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f380000eb10 tx=0x7f380000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.076+0000 7f37faffd640 1 -- 172.21.15.27:0/3546098644 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800004b20 con 0x7f3808108930 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.076+0000 7f37faffd640 1 -- 172.21.15.27:0/3546098644 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3800004cc0 con 0x7f3808108930 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.076+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38081a1fc0 con 0x7f3808108930 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.076+0000 7f37faffd640 1 -- 172.21.15.27:0/3546098644 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800010670 con 0x7f3808108930 2024-04-25T13:40:43.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.076+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38081a2430 con 0x7f3808108930 2024-04-25T13:40:43.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.077+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3808107740 con 0x7f3808108930 2024-04-25T13:40:43.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.078+0000 7f37faffd640 1 -- 172.21.15.27:0/3546098644 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3800005000 con 0x7f3808108930 2024-04-25T13:40:43.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.079+0000 7f37faffd640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37f40afaf0 0x7f37f40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.079+0000 7f37faffd640 1 -- 172.21.15.27:0/3546098644 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f38000df360 con 0x7f3808108930 2024-04-25T13:40:43.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.079+0000 7f380dcbb640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37f40afaf0 0x7f37f40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.080+0000 7f380dcbb640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37f40afaf0 0x7f37f40b2000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f37fc02aa00 tx=0x7f37fc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:43.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.083+0000 7f37faffd640 1 -- 172.21.15.27:0/3546098644 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3800053ff0 con 0x7f3808108930 2024-04-25T13:40:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.268+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f38080637a0 con 0x7f3808108930 2024-04-25T13:40:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.269+0000 7f37faffd640 1 -- 172.21.15.27:0/3546098644 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f38000a7f70 con 0x7f3808108930 2024-04-25T13:40:43.269 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:40:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37f40afaf0 msgr2=0x7f37f40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37f40afaf0 0x7f37f40b2000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f37fc02aa00 tx=0x7f37fc0314a0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3808108930 msgr2=0x7f38081a1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3808108930 0x7f38081a1510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f380000eb10 tx=0x7f380000c2d0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 shutdown_connections 2024-04-25T13:40:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f37f40afaf0 0x7f37f40b2000 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3808108930 0x7f38081a1510 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3808103c30 0x7f380819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 --2- 172.21.15.27:0/3546098644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38081029f0 0x7f3808075860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 >> 172.21.15.27:0/3546098644 conn(0x7f38080fdbc0 msgr2=0x7f3808104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 shutdown_connections 2024-04-25T13:40:43.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.271+0000 7f380ff1f640 1 -- 172.21.15.27:0/3546098644 wait complete. 2024-04-25T13:40:43.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:43 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2474247927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:43.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:43 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2023611292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:43.282 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:43 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/126885494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:43.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:40:43.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:43 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2474247927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:43.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:43 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2023611292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:43.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:43 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/126885494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:43.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:43 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2474247927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:43.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:43 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2023611292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:43.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:43 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/126885494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.395+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283807100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 msgr2=0x7fbce01047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283807100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 0x7fbce01047e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4009900 tx=0x7fbcd402a410 comp rx=0 tx=0).stop 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283807100 shutdown_connections 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283807100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbce0109a20 0x7fbce010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283807100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbce0104d20 0x7fbce0105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283807100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 0x7fbce01047e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283807100 >> 172.21.15.27:0/4283807100 conn(0x7fbce00fecd0 msgr2=0x7fbce0101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283807100 shutdown_connections 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283807100 wait complete. 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 Processor -- start 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 -- start start 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 0x7fbce019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbce0104d20 0x7fbce019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.396+0000 7fbce6b4d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbce0109a20 0x7fbce01a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce6b4d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbce010f5e0 con 0x7fbce0103ae0 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce6b4d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbce010f460 con 0x7fbce0104d20 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce6b4d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbce010f760 con 0x7fbce0109a20 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce48e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 0x7fbce019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce48e9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 0x7fbce019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33244/0 (socket says 172.21.15.27:33244) 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce48e9640 1 -- 172.21.15.27:0/4283047986 learned_addr learned my addr 172.21.15.27:0/4283047986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcdffff640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbce0104d20 0x7fbce019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce50ea640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbce0109a20 0x7fbce01a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcdffff640 1 -- 172.21.15.27:0/4283047986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbce0109a20 msgr2=0x7fbce01a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcdffff640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbce0109a20 0x7fbce01a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcdffff640 1 -- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 msgr2=0x7fbce019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcdffff640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 0x7fbce019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcdffff640 1 -- 172.21.15.27:0/4283047986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce01a1d50 con 0x7fbce0104d20 2024-04-25T13:40:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce48e9640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 0x7fbce019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:40:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcdffff640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbce0104d20 0x7fbce019a750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00e9f0 tx=0x7fbccc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcddffb640 1 -- 172.21.15.27:0/4283047986 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc00cd80 con 0x7fbce0104d20 2024-04-25T13:40:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce01a1fe0 con 0x7fbce0104d20 2024-04-25T13:40:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.397+0000 7fbcddffb640 1 -- 172.21.15.27:0/4283047986 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbccc0035f0 con 0x7fbce0104d20 2024-04-25T13:40:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.398+0000 7fbcddffb640 1 -- 172.21.15.27:0/4283047986 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc010690 con 0x7fbce0104d20 2024-04-25T13:40:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.398+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce01a24f0 con 0x7fbce0104d20 2024-04-25T13:40:43.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.399+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce0108830 con 0x7fbce0104d20 2024-04-25T13:40:43.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.400+0000 7fbcddffb640 1 -- 172.21.15.27:0/4283047986 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbccc0037a0 con 0x7fbce0104d20 2024-04-25T13:40:43.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.401+0000 7fbcddffb640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcd00afaf0 0x7fbcd00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.401+0000 7fbce48e9640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcd00afaf0 0x7fbcd00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.401+0000 7fbcddffb640 1 -- 172.21.15.27:0/4283047986 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbccc0deed0 con 0x7fbce0104d20 2024-04-25T13:40:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.402+0000 7fbce48e9640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcd00afaf0 0x7fbcd00b2000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4031500 tx=0x7fbcd40046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:43.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.410+0000 7fbcddffb640 1 -- 172.21.15.27:0/4283047986 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbccc0a7a80 con 0x7fbce0104d20 2024-04-25T13:40:43.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.659+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbce002db50 con 0x7fbce0104d20 2024-04-25T13:40:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.660+0000 7fbcddffb640 1 -- 172.21.15.27:0/4283047986 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fbccc0a7380 con 0x7fbce0104d20 2024-04-25T13:40:43.660 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:43.660 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:40:43.660 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:40:43.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcd00afaf0 msgr2=0x7fbcd00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcd00afaf0 0x7fbcd00b2000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4031500 tx=0x7fbcd40046c0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbce0104d20 msgr2=0x7fbce019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbce0104d20 0x7fbce019a750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00e9f0 tx=0x7fbccc00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 shutdown_connections 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbcd00afaf0 0x7fbcd00b2000 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbce0109a20 0x7fbce01a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbce0104d20 0x7fbce019a750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 --2- 172.21.15.27:0/4283047986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbce0103ae0 0x7fbce019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 >> 172.21.15.27:0/4283047986 conn(0x7fbce00fecd0 msgr2=0x7fbce0105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 shutdown_connections 2024-04-25T13:40:43.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.662+0000 7fbce6b4d640 1 -- 172.21.15.27:0/4283047986 wait complete. 2024-04-25T13:40:43.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 -- 172.21.15.27:0/864881434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 msgr2=0x7fc440104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 --2- 172.21.15.27:0/864881434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 0x7fc440104800 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc430009900 tx=0x7fc43002a410 comp rx=0 tx=0).stop 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 -- 172.21.15.27:0/864881434 shutdown_connections 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 --2- 172.21.15.27:0/864881434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc440109a40 0x7fc44010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 --2- 172.21.15.27:0/864881434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc440104d40 0x7fc440105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 --2- 172.21.15.27:0/864881434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 0x7fc440104800 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 -- 172.21.15.27:0/864881434 >> 172.21.15.27:0/864881434 conn(0x7fc4400fed10 msgr2=0x7fc4401011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 -- 172.21.15.27:0/864881434 shutdown_connections 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.803+0000 7fc4468e7640 1 -- 172.21.15.27:0/864881434 wait complete. 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc4468e7640 1 Processor -- start 2024-04-25T13:40:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc4468e7640 1 -- start start 2024-04-25T13:40:43.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc4468e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 0x7fc44019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc4468e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc440104d40 0x7fc44019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc4468e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc440109a40 0x7fc4401a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc4468e7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010f600 con 0x7fc440103b00 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc4468e7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010f480 con 0x7fc440109a40 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc4468e7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc44010f780 con 0x7fc440104d40 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc43ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 0x7fc44019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc43ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 0x7fc44019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54106/0 (socket says 172.21.15.27:54106) 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc43ffff640 1 -- 172.21.15.27:0/1809184204 learned_addr learned my addr 172.21.15.27:0/1809184204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc43f7fe640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc440104d40 0x7fc44019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.804+0000 7fc444e84640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc440109a40 0x7fc4401a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43ffff640 1 -- 172.21.15.27:0/1809184204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc440104d40 msgr2=0x7fc44019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43ffff640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc440104d40 0x7fc44019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43ffff640 1 -- 172.21.15.27:0/1809184204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc440109a40 msgr2=0x7fc4401a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43ffff640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc440109a40 0x7fc4401a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43ffff640 1 -- 172.21.15.27:0/1809184204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4401a1d60 con 0x7fc440103b00 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43f7fe640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc440104d40 0x7fc44019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43ffff640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 0x7fc44019a220 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc430006010 tx=0x7fc430031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43d7fa640 1 -- 172.21.15.27:0/1809184204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43003f030 con 0x7fc440103b00 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43d7fa640 1 -- 172.21.15.27:0/1809184204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc430003aa0 con 0x7fc440103b00 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc43d7fa640 1 -- 172.21.15.27:0/1809184204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430035070 con 0x7fc440103b00 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4401a1ff0 con 0x7fc440103b00 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.805+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4401a23a0 con 0x7fc440103b00 2024-04-25T13:40:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.806+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc440108850 con 0x7fc440103b00 2024-04-25T13:40:43.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.809+0000 7fc43d7fa640 1 -- 172.21.15.27:0/1809184204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc430042030 con 0x7fc440103b00 2024-04-25T13:40:43.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.809+0000 7fc43d7fa640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc42c0afaf0 0x7fc42c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:43.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.809+0000 7fc43f7fe640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc42c0afaf0 0x7fc42c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:43.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.809+0000 7fc43d7fa640 1 -- 172.21.15.27:0/1809184204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc4300ff230 con 0x7fc440103b00 2024-04-25T13:40:43.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.810+0000 7fc43f7fe640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc42c0afaf0 0x7fc42c0b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc428005fd0 tx=0x7fc428005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:43.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:43.814+0000 7fc43d7fa640 1 -- 172.21.15.27:0/1809184204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4300c7e40 con 0x7fc440103b00 2024-04-25T13:40:44.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.005+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc4400637a0 con 0x7fc440103b00 2024-04-25T13:40:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.007+0000 7fc43d7fa640 1 -- 172.21.15.27:0/1809184204 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7fc4300c7740 con 0x7fc440103b00 2024-04-25T13:40:44.007 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:44.007 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.009+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc42c0afaf0 msgr2=0x7fc42c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.009+0000 7fc4468e7640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc42c0afaf0 0x7fc42c0b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc428005fd0 tx=0x7fc428005850 comp rx=0 tx=0).stop 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.009+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 msgr2=0x7fc44019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.009+0000 7fc4468e7640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 0x7fc44019a220 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc430006010 tx=0x7fc430031a50 comp rx=0 tx=0).stop 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.010+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 shutdown_connections 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.010+0000 7fc4468e7640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc42c0afaf0 0x7fc42c0b2000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.010+0000 7fc4468e7640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc440109a40 0x7fc4401a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.010+0000 7fc4468e7640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc440104d40 0x7fc44019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.010+0000 7fc4468e7640 1 --2- 172.21.15.27:0/1809184204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc440103b00 0x7fc44019a220 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.010+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 >> 172.21.15.27:0/1809184204 conn(0x7fc4400fed10 msgr2=0x7fc440105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.010+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 shutdown_connections 2024-04-25T13:40:44.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.010+0000 7fc4468e7640 1 -- 172.21.15.27:0/1809184204 wait complete. 2024-04-25T13:40:44.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T13:38:44.716138+0000', 'modified': '2024-04-25T13:38:48.756882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24517': {'gid': 24517, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.122:6839/3397333136', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6838', 'nonce': 3397333136}, {'type': 'v1', 'addr': '172.21.15.122:6839', 'nonce': 3397333136}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.45:6839/3661968964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6838', 'nonce': 3661968964}, {'type': 'v1', 'addr': '172.21.15.45:6839', 'nonce': 3661968964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24560, 'qdb_cluster': [24560]} 2024-04-25T13:40:44.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T13:40:44.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T13:40:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 -- 172.21.15.27:0/848896187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac1029e0 msgr2=0x7f8fac1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 --2- 172.21.15.27:0/848896187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac1029e0 0x7f8fac1036e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009900 tx=0x7f8fa802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 -- 172.21.15.27:0/848896187 shutdown_connections 2024-04-25T13:40:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 --2- 172.21.15.27:0/848896187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac108920 0x7f8fac10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 --2- 172.21.15.27:0/848896187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fac103c20 0x7f8fac104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 --2- 172.21.15.27:0/848896187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac1029e0 0x7f8fac1036e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 -- 172.21.15.27:0/848896187 >> 172.21.15.27:0/848896187 conn(0x7f8fac0fdbd0 msgr2=0x7f8fac100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:44.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 -- 172.21.15.27:0/848896187 shutdown_connections 2024-04-25T13:40:44.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.140+0000 7f8fb3830640 1 -- 172.21.15.27:0/848896187 wait complete. 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb3830640 1 Processor -- start 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb3830640 1 -- start start 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb3830640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fac1029e0 0x7f8fac06a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb3830640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac103c20 0x7f8fac19f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb3830640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac108920 0x7f8fac1a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb3830640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac10e4e0 con 0x7f8fac103c20 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb3830640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac10e360 con 0x7f8fac108920 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb3830640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac10e660 con 0x7f8fac1029e0 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb0dcb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac103c20 0x7f8fac19f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb0dcb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac103c20 0x7f8fac19f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54134/0 (socket says 172.21.15.27:54134) 2024-04-25T13:40:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb0dcb640 1 -- 172.21.15.27:0/809729194 learned_addr learned my addr 172.21.15.27:0/809729194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fac1029e0 0x7f8fac06a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac108920 0x7f8fac1a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb0dcb640 1 -- 172.21.15.27:0/809729194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fac1029e0 msgr2=0x7f8fac06a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.141+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fac1029e0 0x7f8fac06a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fb0dcb640 1 -- 172.21.15.27:0/809729194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac108920 msgr2=0x7f8fac1a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac108920 0x7f8fac1a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fb0dcb640 1 -- 172.21.15.27:0/809729194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac1a2090 con 0x7f8fac103c20 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac103c20 0x7f8fac19f320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00d770 tx=0x7f8f9c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fa27fc640 1 -- 172.21.15.27:0/809729194 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c017070 con 0x7f8fac103c20 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fb3830640 1 -- 172.21.15.27:0/809729194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac1a2380 con 0x7f8fac103c20 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fa27fc640 1 -- 172.21.15.27:0/809729194 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f9c0054d0 con 0x7f8fac103c20 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fa27fc640 1 -- 172.21.15.27:0/809729194 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c004c30 con 0x7f8fac103c20 2024-04-25T13:40:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.142+0000 7f8fb3830640 1 -- 172.21.15.27:0/809729194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac1a28d0 con 0x7f8fac103c20 2024-04-25T13:40:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.144+0000 7f8fa27fc640 1 -- 172.21.15.27:0/809729194 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8f9c004e10 con 0x7f8fac103c20 2024-04-25T13:40:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fba7b2740 1 Processor -- start 2024-04-25T13:40:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fba7b2740 1 -- start start 2024-04-25T13:40:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025db270 0x55cf02980780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025b23b0 0x55cf02980d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf02987c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cf028e1740 con 0x55cf025db270 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cf0287d3a0 con 0x55cf025b23b0 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55cf028dd700 con 0x55cf025ba590 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fa27fc640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f980afaf0 0x7f8f980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fb1dcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025db270 0x55cf02980780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fb1dcd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025db270 0x55cf02980780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54140/0 (socket says 172.21.15.27:54140) 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fb1dcd640 1 -- 172.21.15.27:0/383399036 learned_addr learned my addr 172.21.15.27:0/383399036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025b23b0 0x55cf02980d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fa27fc640 1 -- 172.21.15.27:0/809729194 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8f9c0de6b0 con 0x7f8fac103c20 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf02987c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.145+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f980afaf0 0x7f8f980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8fb1dcd640 1 -- 172.21.15.27:0/383399036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 msgr2=0x55cf02987c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf02987c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8fb1dcd640 1 -- 172.21.15.27:0/383399036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025b23b0 msgr2=0x55cf02980d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025b23b0 0x55cf02980d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8fb1dcd640 1 -- 172.21.15.27:0/383399036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf02899750 con 0x55cf025db270 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025db270 0x55cf02980780 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4013a70 tx=0x7f8fa4013f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f244ae072cebadaa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8f977fe640 1 -- 172.21.15.27:0/383399036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4013540 con 0x55cf025db270 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8f977fe640 1 -- 172.21.15.27:0/383399036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa4022070 con 0x55cf025db270 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.146+0000 7f8f977fe640 1 -- 172.21.15.27:0/383399036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa401cb80 con 0x55cf025db270 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 -- 172.21.15.27:0/383399036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025db270 msgr2=0x55cf02980780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025db270 0x55cf02980780 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4013a70 tx=0x7f8fa4013f30 comp rx=0 tx=0).stop 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f980afaf0 0x7f8f980b2000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4021000 tx=0x7f8fa401f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 -- 172.21.15.27:0/383399036 shutdown_connections 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf02987c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025b23b0 0x55cf02980d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/383399036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025db270 0x55cf02980780 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 -- 172.21.15.27:0/383399036 >> 172.21.15.27:0/383399036 conn(0x55cf025cf3c0 msgr2=0x55cf02977d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 -- 172.21.15.27:0/383399036 shutdown_connections 2024-04-25T13:40:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 -- 172.21.15.27:0/383399036 wait complete. 2024-04-25T13:40:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 -- start start 2024-04-25T13:40:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.147+0000 7f8fba7b2740 1 Processor -- start 2024-04-25T13:40:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf0298abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 0x55cf02991b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cf02988650 con 0x55cf025b23b0 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cf02868fe0 con 0x55cf025db270 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55cf029887d0 con 0x55cf025ba590 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fba7b2740 10 client.0 _ll_register_callbacks cb 0x55cf0287f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54154/0 (socket says 172.21.15.27:54154) 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 -- 172.21.15.27:0/525489801 learned_addr learned my addr 172.21.15.27:0/525489801 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf0298abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 0x55cf02991b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 -- 172.21.15.27:0/525489801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 msgr2=0x55cf0298abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf0298abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 -- 172.21.15.27:0/525489801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 msgr2=0x55cf02991b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 0x55cf02991b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 -- 172.21.15.27:0/525489801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf025f32d0 con 0x55cf025b23b0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.148+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298a6a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009230 tx=0x7f8fa8031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6f281a03aaab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8f94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8f94ff9640 1 -- 172.21.15.27:0/525489801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8035040 con 0x55cf025b23b0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8f94ff9640 1 -- 172.21.15.27:0/525489801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8fa8004240 con 0x55cf025b23b0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 1 -- 172.21.15.27:0/525489801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa4022070 con 0x55cf025b23b0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8f94ff9640 1 -- 172.21.15.27:0/525489801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f8fa804d070 con 0x55cf025b23b0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 10 client.15177 fetch_fsmap learned FSMap version 19 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 1 -- 172.21.15.27:0/525489801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55cf026bd6b0 con 0x55cf025b23b0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8f94ff9640 1 -- 172.21.15.27:0/525489801 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa4022070 con 0x55cf025b23b0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8f94ff9640 1 -- 172.21.15.27:0/525489801 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f8fa8044020 con 0x55cf025b23b0 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 10 client.15177 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:40:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 10 client.15177 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 20 client.15177 populate_metadata read hostname 'smithi027' 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf02599e40 0x55cf02998390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 4 client.15177 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.149+0000 7f8fba7b2740 1 -- 172.21.15.27:0/525489801 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf028ef350 con 0x55cf02599e40 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.150+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf02599e40 0x55cf02998390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.150+0000 7f8f94ff9640 10 client.15177 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.150+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf02599e40 0x55cf02998390 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=888e05a8a5f533d2 server_cookie=b0a2baa65492bc3 in_seq=0 out_seq=0 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8f94ff9640 1 -- 172.21.15.27:0/525489801 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cf028ef350 con 0x55cf02599e40 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8f94ff9640 10 client.15177 handle_command_reply: tid=0 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 client.15177 shutdown 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 -- 172.21.15.27:0/525489801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 msgr2=0x55cf0298a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298a6a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009230 tx=0x7f8fa8031f50 comp rx=0 tx=0).stop 2024-04-25T13:40:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 -- 172.21.15.27:0/525489801 shutdown_connections 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf02599e40 0x55cf02998390 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 0x55cf02991b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf0298abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/525489801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298a6a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8f94ff9640 0 client.15177 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 -- 172.21.15.27:0/525489801 >> 172.21.15.27:0/525489801 conn(0x55cf025cf3c0 msgr2=0x55cf02976d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 -- 172.21.15.27:0/525489801 shutdown_connections 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.151+0000 7f8fba7b2740 1 -- 172.21.15.27:0/525489801 wait complete. 2024-04-25T13:40:44.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.152+0000 7f8fba7b2740 20 client.15177 trim_cache size 0 max 16384 2024-04-25T13:40:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.170+0000 7f8fba7b2740 1 Processor -- start 2024-04-25T13:40:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.170+0000 7f8fba7b2740 1 -- start start 2024-04-25T13:40:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.170+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf02977740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.170+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025ba590 0x55cf02991a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025db270 0x55cf02977c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cf029966d0 con 0x55cf025b23b0 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cf02996da0 con 0x55cf025ba590 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55cf02997470 con 0x55cf025db270 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf02977740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf02977740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54156/0 (socket says 172.21.15.27:54156) 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 -- 172.21.15.27:0/3948697738 learned_addr learned my addr 172.21.15.27:0/3948697738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025ba590 0x55cf02991a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025db270 0x55cf02977c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 -- 172.21.15.27:0/3948697738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025db270 msgr2=0x55cf02977c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025db270 0x55cf02977c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 -- 172.21.15.27:0/3948697738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025ba590 msgr2=0x55cf02991a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025ba590 0x55cf02991a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 -- 172.21.15.27:0/3948697738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf02985540 con 0x55cf025b23b0 2024-04-25T13:40:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025ba590 0x55cf02991a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf02977740 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8fa802a930 tx=0x7f8fa8009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abdbc331a26b7e48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8f96ffd640 1 -- 172.21.15.27:0/3948697738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8035040 con 0x55cf025b23b0 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8f96ffd640 1 -- 172.21.15.27:0/3948697738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa8046440 con 0x55cf025b23b0 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.171+0000 7f8f96ffd640 1 -- 172.21.15.27:0/3948697738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa803b640 con 0x55cf025b23b0 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 -- 172.21.15.27:0/3948697738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 msgr2=0x55cf02977740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf02977740 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8fa802a930 tx=0x7f8fa8009610 comp rx=0 tx=0).stop 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 -- 172.21.15.27:0/3948697738 shutdown_connections 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025db270 0x55cf02977c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025ba590 0x55cf02991a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/3948697738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf02977740 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 -- 172.21.15.27:0/3948697738 >> 172.21.15.27:0/3948697738 conn(0x55cf025cf3c0 msgr2=0x55cf02988cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 -- 172.21.15.27:0/3948697738 shutdown_connections 2024-04-25T13:40:44.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 -- 172.21.15.27:0/3948697738 wait complete. 2024-04-25T13:40:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.172+0000 7f8fba7b2740 1 -- start start 2024-04-25T13:40:44.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fba7b2740 1 Processor -- start 2024-04-25T13:40:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf0298d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fba7b2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 0x55cf029905f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cf02998210 con 0x55cf025b23b0 2024-04-25T13:40:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cf02997b40 con 0x55cf025db270 2024-04-25T13:40:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fba7b2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55cf029988e0 con 0x55cf025ba590 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fba7b2740 10 client.0 _ll_register_callbacks cb 0x7f8fa8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb15cc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb15cc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54166/0 (socket says 172.21.15.27:54166) 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb15cc640 1 -- 172.21.15.27:0/2543494885 learned_addr learned my addr 172.21.15.27:0/2543494885 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf0298d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb1dcd640 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 0x55cf029905f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb15cc640 1 -- 172.21.15.27:0/2543494885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 msgr2=0x55cf0298d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf0298d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb15cc640 1 -- 172.21.15.27:0/2543494885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 msgr2=0x55cf029905f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 0x55cf029905f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.173+0000 7f8fb15cc640 1 -- 172.21.15.27:0/2543494885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf02990bb0 con 0x55cf025b23b0 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8fb15cc640 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298d170 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8fa803c4c0 tx=0x7f8fa8031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ace57881671ed2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8f94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:40:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8f94ff9640 1 -- 172.21.15.27:0/2543494885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8035040 con 0x55cf025b23b0 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8f94ff9640 1 -- 172.21.15.27:0/2543494885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8fa803ccf0 con 0x55cf025b23b0 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8fba7b2740 1 -- 172.21.15.27:0/2543494885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa8046440 con 0x55cf025b23b0 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8f94ff9640 1 -- 172.21.15.27:0/2543494885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f8fa80465f0 con 0x55cf025b23b0 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8fba7b2740 10 client.15183 fetch_fsmap learned FSMap version 19 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8fba7b2740 1 -- 172.21.15.27:0/2543494885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55cf02a01210 con 0x55cf025b23b0 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8f94ff9640 1 -- 172.21.15.27:0/2543494885 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa8046440 con 0x55cf025b23b0 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8f94ff9640 1 -- 172.21.15.27:0/2543494885 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f8fa8044040 con 0x55cf025b23b0 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.174+0000 7f8fba7b2740 10 client.15183 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8fba7b2740 10 client.15183 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8fba7b2740 20 client.15183 populate_metadata read hostname 'smithi027' 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf0256f280 0x55cf02a04730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8fba7b2740 4 client.15183 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8fba7b2740 1 -- 172.21.15.27:0/2543494885 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "status"}) -- 0x55cf028e91c0 con 0x55cf0256f280 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf0256f280 0x55cf02a04730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8fb0dcb640 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf0256f280 0x55cf02a04730 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efb747a614a87002 server_cookie=908a602197334dfd in_seq=0 out_seq=0 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8f94ff9640 10 client.15183 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8f94ff9640 1 -- 172.21.15.27:0/2543494885 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cf028e91c0 con 0x55cf0256f280 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8f94ff9640 10 client.15183 handle_command_reply: tid=0 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.175+0000 7f8fba7b2740 1 client.15183 shutdown 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 -- 172.21.15.27:0/2543494885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 msgr2=0x55cf0298d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298d170 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8fa803c4c0 tx=0x7f8fa8031630 comp rx=0 tx=0).stop 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 -- 172.21.15.27:0/2543494885 shutdown_connections 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf0256f280 0x55cf02a04730 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf025db270 0x55cf029905f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf025ba590 0x55cf0298d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8f94ff9640 0 client.15183 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 --2- 172.21.15.27:0/2543494885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf025b23b0 0x55cf0298d170 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 -- 172.21.15.27:0/2543494885 >> 172.21.15.27:0/2543494885 conn(0x55cf025cf3c0 msgr2=0x55cf02977740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 -- 172.21.15.27:0/2543494885 shutdown_connections 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 1 -- 172.21.15.27:0/2543494885 wait complete. 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fba7b2740 20 client.15183 trim_cache size 0 max 16384 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "db01fc44-0307-11ef-bc93-c7b262605968", 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 24560, 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-04-25T13:40:44.182 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-04-25T13:40:44.183 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 118.819929648, 2024-04-25T13:40:44.183 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 19, 2024-04-25T13:40:44.183 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 78, 2024-04-25T13:40:44.183 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 76, 2024-04-25T13:40:44.183 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 125.40388770200001 2024-04-25T13:40:44.183 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fb3830640 1 -- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f980afaf0 msgr2=0x7f8f980b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fb3830640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f980afaf0 0x7f8f980b2000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4021000 tx=0x7f8fa401f000 comp rx=0 tx=0).stop 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fb3830640 1 -- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac103c20 msgr2=0x7f8fac19f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.176+0000 7f8fb3830640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac103c20 0x7f8fac19f320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00d770 tx=0x7f8f9c00dc30 comp rx=0 tx=0).stop 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.177+0000 7f8fb3830640 1 -- 172.21.15.27:0/809729194 shutdown_connections 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.177+0000 7f8fb3830640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8f980afaf0 0x7f8f980b2000 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.177+0000 7f8fb3830640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fac108920 0x7f8fac1a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.177+0000 7f8fb3830640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fac103c20 0x7f8fac19f320 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.177+0000 7f8fb3830640 1 --2- 172.21.15.27:0/809729194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fac1029e0 0x7f8fac06a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.177+0000 7f8fb3830640 1 -- 172.21.15.27:0/809729194 >> 172.21.15.27:0/809729194 conn(0x7f8fac0fdbd0 msgr2=0x7f8fac104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.177+0000 7f8fb3830640 1 -- 172.21.15.27:0/809729194 shutdown_connections 2024-04-25T13:40:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:44.177+0000 7f8fb3830640 1 -- 172.21.15.27:0/809729194 wait complete. 2024-04-25T13:40:44.188 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T13:40:44.197 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T13:40:44.198 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T13:40:44.198 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T13:40:44.198 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T13:40:44.262 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T13:40:44.263 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T13:40:44.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:44.263 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T13:40:44.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:44.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-04-25T13:40:44.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:44 smithi122 ceph-mon[32843]: pgmap v344: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:44.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:44 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3546098644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:44.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:44 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/4283047986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:44.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:44 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1809184204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:44.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:44 smithi045 ceph-mon[32731]: pgmap v344: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:44.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:44 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3546098644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:44.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:44 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/4283047986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:44.329 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:44 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1809184204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:44.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:44 smithi027 ceph-mon[29140]: pgmap v344: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:44.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:44 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3546098644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:44.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:44 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/4283047986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:44.351 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:44 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1809184204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:44.357 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-04-25T13:40:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:40:44.359 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v2', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-04-25T13:40:44.359 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T13:40:44.359 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-25T13:40:44.359 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-25T13:40:44.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid db01fc44-0307-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T13:40:44.649 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T13:40:46.108 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:46 smithi027 ceph-mon[29140]: pgmap v345: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:46.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:46 smithi122 ceph-mon[32843]: pgmap v345: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:46.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:46 smithi045 ceph-mon[32731]: pgmap v345: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 -- 172.21.15.27:0/1192702190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24108610 msgr2=0x7f9b2410ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 --2- 172.21.15.27:0/1192702190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24108610 0x7f9b2410ff70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00b0d0 tx=0x7f9b1c00b3e0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 -- 172.21.15.27:0/1192702190 shutdown_connections 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 --2- 172.21.15.27:0/1192702190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24108610 0x7f9b2410ff70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 --2- 172.21.15.27:0/1192702190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b24103390 0x7f9b241037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 --2- 172.21.15.27:0/1192702190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b24107cc0 0x7f9b241080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 -- 172.21.15.27:0/1192702190 >> 172.21.15.27:0/1192702190 conn(0x7f9b240fec90 msgr2=0x7f9b241010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 -- 172.21.15.27:0/1192702190 shutdown_connections 2024-04-25T13:40:46.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.353+0000 7f9b28813700 1 -- 172.21.15.27:0/1192702190 wait complete. 2024-04-25T13:40:46.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b28813700 1 Processor -- start 2024-04-25T13:40:46.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b28813700 1 -- start start 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b28813700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b24103390 0x7f9b24110950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b28813700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24107cc0 0x7f9b24110e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b28813700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b24108610 0x7f9b241113d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b28813700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b24114390 con 0x7f9b24107cc0 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b28813700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b24114210 con 0x7f9b24108610 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b28813700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9b24114510 con 0x7f9b24103390 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b21d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24107cc0 0x7f9b24110e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b21d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24107cc0 0x7f9b24110e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54196/0 (socket says 172.21.15.27:54196) 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b21d9b700 1 -- 172.21.15.27:0/339104799 learned_addr learned my addr 172.21.15.27:0/339104799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b2259c700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b24103390 0x7f9b24110950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b22d9d700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b24108610 0x7f9b241113d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:46.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b21d9b700 1 -- 172.21.15.27:0/339104799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b24103390 msgr2=0x7f9b24110950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b21d9b700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b24103390 0x7f9b24110950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b21d9b700 1 -- 172.21.15.27:0/339104799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b24108610 msgr2=0x7f9b241113d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b21d9b700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b24108610 0x7f9b241113d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.354+0000 7f9b21d9b700 1 -- 172.21.15.27:0/339104799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b241a9000 con 0x7f9b24107cc0 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.355+0000 7f9b21d9b700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24107cc0 0x7f9b24110e90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800d980 tx=0x7f9b1800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.355+0000 7f9b137fe700 1 -- 172.21.15.27:0/339104799 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b18017070 con 0x7f9b24107cc0 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.355+0000 7f9b137fe700 1 -- 172.21.15.27:0/339104799 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b1800fd40 con 0x7f9b24107cc0 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.355+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b241a92f0 con 0x7f9b24107cc0 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.355+0000 7f9b137fe700 1 -- 172.21.15.27:0/339104799 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1800e3b0 con 0x7f9b24107cc0 2024-04-25T13:40:46.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.355+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b241a9880 con 0x7f9b24107cc0 2024-04-25T13:40:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.356+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b24051190 con 0x7f9b24107cc0 2024-04-25T13:40:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.358+0000 7f9b137fe700 1 -- 172.21.15.27:0/339104799 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9b18010610 con 0x7f9b24107cc0 2024-04-25T13:40:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.358+0000 7f9b137fe700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b0c0afc00 0x7f9b0c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.358+0000 7f9b2259c700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b0c0afc00 0x7f9b0c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:46.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.359+0000 7f9b137fe700 1 -- 172.21.15.27:0/339104799 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9b180de560 con 0x7f9b24107cc0 2024-04-25T13:40:46.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.359+0000 7f9b2259c700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b0c0afc00 0x7f9b0c0b2110 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9b14007be0 tx=0x7f9b140073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:46.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.363+0000 7f9b137fe700 1 -- 172.21.15.27:0/339104799 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b180a71f0 con 0x7f9b24107cc0 2024-04-25T13:40:46.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.619+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9b24111e80 con 0x7f9b24107cc0 2024-04-25T13:40:46.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.619+0000 7f9b137fe700 1 -- 172.21.15.27:0/339104799 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) ==== 53+0+73 (secure 0 0 0) 0x7f9b180a6af0 con 0x7f9b24107cc0 2024-04-25T13:40:46.620 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T13:40:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b0c0afc00 msgr2=0x7f9b0c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b0c0afc00 0x7f9b0c0b2110 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9b14007be0 tx=0x7f9b140073d0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24107cc0 msgr2=0x7f9b24110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24107cc0 0x7f9b24110e90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800d980 tx=0x7f9b1800dc90 comp rx=0 tx=0).stop 2024-04-25T13:40:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 shutdown_connections 2024-04-25T13:40:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b0c0afc00 0x7f9b0c0b2110 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b24103390 0x7f9b24110950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b24108610 0x7f9b241113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 --2- 172.21.15.27:0/339104799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b24107cc0 0x7f9b24110e90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 >> 172.21.15.27:0/339104799 conn(0x7f9b240fec90 msgr2=0x7f9b24107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 shutdown_connections 2024-04-25T13:40:46.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:46.622+0000 7f9b28813700 1 -- 172.21.15.27:0/339104799 wait complete. 2024-04-25T13:40:47.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:47 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/339104799' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T13:40:47.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:47 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/339104799' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T13:40:47.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:47 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/339104799' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T13:40:47.344 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-25T13:40:47.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:47.344 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-04-25T13:40:47.361 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T13:40:47.361 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-04-25T13:40:47.362 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T13:40:47.363 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42144sec preferred_lft 42144sec 2024-04-25T13:40:47.363 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-25T13:40:47.363 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-25T13:40:47.363 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T13:40:47.363 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-04-25T13:40:47.363 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-04-25T13:40:47.363 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T13:40:47.363 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T13:40:47.364 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T13:40:47.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-04-25T13:40:47.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-04-25T13:40:47.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-04-25T13:40:47.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T13:40:47.364 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T13:40:47.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T13:40:47.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:47.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:47.547 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T13:40:47.572 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-04-25T13:40:47.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:47.575 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-04-25T13:40:47.644 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-04-25T13:40:47.644 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T13:40:47.644 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T13:40:47.644 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T13:40:47.644 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T13:40:47.644 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T13:40:47.644 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T13:40:47.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T13:40:47.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T13:40:47.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T13:40:47.645 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T13:40:47.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T13:40:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:47.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:47.931 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T13:40:47.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:47.947 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-04-25T13:40:48.002 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T13:40:48.002 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T13:40:48.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-25T13:40:48.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-25T13:40:48.002 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T13:40:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T13:40:48.098 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:48 smithi027 ceph-mon[29140]: pgmap v346: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:48.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:48.120 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-25T13:40:48.120 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T13:40:48.120 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T13:40:48.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-25T13:40:48.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T13:40:48.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-25T13:40:48.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-25T13:40:48.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-25T13:40:48.120 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T13:40:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T13:40:48.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:48.250 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T13:40:48.250 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-25T13:40:48.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-04-25T13:40:48.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T13:40:48.250 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-25T13:40:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T13:40:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:48 smithi122 ceph-mon[32843]: pgmap v346: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:48.326 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T13:40:48.326 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /tmp/cephfs 2024-04-25T13:40:48.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:48 smithi045 ceph-mon[32731]: pgmap v346: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T13:40:48.341 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-25T13:40:48.341 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T13:40:48.342 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /tmp/cephfs 2024-04-25T13:40:48.396 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T13:40:48.397 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T13:40:48.397 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/aa501c8f-0208-4329-abf5-aee6b1b25d7c 2024-04-25T13:40:48.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:48.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/aa501c8f-0208-4329-abf5-aee6b1b25d7c /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context=system_u:object_r:postgresql_db_t:s0,nofallback 2024-04-25T13:40:48.493 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-25T13:40:48.493 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-25T13:40:48.595 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-25T13:40:48.596 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-25T13:40:48.596 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.45:3300,172.21.15.122:3300" 2024-04-25T13:40:48.596 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with new device syntax: 0@db01fc44-0307-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/aa501c8f-0208-4329-abf5-aee6b1b25d7c 2024-04-25T13:40:48.596 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",name=0,key=0,mon_addr=172.21.15.27:3300/172.21.15.45:3300/172.21.15.122:3300" will pass to kernel 2024-04-25T13:40:48.598 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T13:40:48.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:48.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /tmp/cephfs 2024-04-25T13:40:48.630 DEBUG:teuthology.orchestra.run.smithi027:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T13:40:48.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T13:40:48.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:48.729 INFO:teuthology.orchestra.run.smithi027.stdout:client15192 2024-04-25T13:40:48.729 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:40:48.730 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T13:40:48.746 DEBUG:teuthology.orchestra.run.smithi027:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T13:40:48.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T13:40:48.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:48.844 INFO:teuthology.orchestra.run.smithi027.stdout:client15192 2024-04-25T13:40:48.844 DEBUG:teuthology.orchestra.run.smithi027:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T13:40:48.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T13:40:48.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:48.901 INFO:teuthology.orchestra.run.smithi027.stdout:client15192 2024-04-25T13:40:48.901 DEBUG:teuthology.orchestra.run.smithi027:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/db01fc44-0307-11ef-bc93-c7b262605968.client15192/status') 2024-04-25T13:40:48.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T13:40:48.960 INFO:teuthology.orchestra.run.smithi027.stdout:instance: client.15192 (3)192.168.0.1:0/4044858050 2024-04-25T13:40:48.960 INFO:teuthology.orchestra.run.smithi027.stdout:blocklisted: false 2024-04-25T13:40:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:0+1 records in 2024-04-25T13:40:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:0+1 records out 2024-04-25T13:40:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:70 bytes copied, 2.8398e-05 s, 2.5 MB/s 2024-04-25T13:40:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T13:40:48.968 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T13:40:48.980 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T13:40:49.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 -- 172.21.15.27:0/1809176863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0109a50 msgr2=0x7f8bc010be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/1809176863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0109a50 0x7f8bc010be70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009900 tx=0x7f8bb802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 -- 172.21.15.27:0/1809176863 shutdown_connections 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/1809176863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0109a50 0x7f8bc010be70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/1809176863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bc0104d50 0x7f8bc0105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/1809176863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bc0103b10 0x7f8bc0104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 -- 172.21.15.27:0/1809176863 >> 172.21.15.27:0/1809176863 conn(0x7f8bc00fed40 msgr2=0x7f8bc01011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 -- 172.21.15.27:0/1809176863 shutdown_connections 2024-04-25T13:40:49.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.120+0000 7f8bc5d82640 1 -- 172.21.15.27:0/1809176863 wait complete. 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bc5d82640 1 Processor -- start 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bc5d82640 1 -- start start 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bc5d82640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bc0103b10 0x7f8bc019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bc5d82640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0104d50 0x7f8bc019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bc5d82640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bc0109a50 0x7f8bc01a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bc5d82640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc010f640 con 0x7f8bc0104d50 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bc5d82640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc010f4c0 con 0x7f8bc0109a50 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bc5d82640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc010f7c0 con 0x7f8bc0103b10 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0104d50 0x7f8bc019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0104d50 0x7f8bc019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54220/0 (socket says 172.21.15.27:54220) 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbeffd640 1 -- 172.21.15.27:0/3692188325 learned_addr learned my addr 172.21.15.27:0/3692188325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbf7fe640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bc0103b10 0x7f8bc019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbffff640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bc0109a50 0x7f8bc01a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbeffd640 1 -- 172.21.15.27:0/3692188325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bc0103b10 msgr2=0x7f8bc019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbeffd640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bc0103b10 0x7f8bc019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbeffd640 1 -- 172.21.15.27:0/3692188325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bc0109a50 msgr2=0x7f8bc01a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbeffd640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bc0109a50 0x7f8bc01a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbeffd640 1 -- 172.21.15.27:0/3692188325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc01a1d00 con 0x7f8bc0104d50 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.121+0000 7f8bbf7fe640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bc0103b10 0x7f8bc019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.122+0000 7f8bbeffd640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0104d50 0x7f8bc019a750 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800e9f0 tx=0x7f8ba800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.122+0000 7f8bbcff9640 1 -- 172.21.15.27:0/3692188325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba800cd80 con 0x7f8bc0104d50 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.122+0000 7f8bbcff9640 1 -- 172.21.15.27:0/3692188325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ba80054d0 con 0x7f8bc0104d50 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.122+0000 7f8bbcff9640 1 -- 172.21.15.27:0/3692188325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8010690 con 0x7f8bc0104d50 2024-04-25T13:40:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.122+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc01a1ff0 con 0x7f8bc0104d50 2024-04-25T13:40:49.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.122+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc01a2460 con 0x7f8bc0104d50 2024-04-25T13:40:49.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.123+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc0108860 con 0x7f8bc0104d50 2024-04-25T13:40:49.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.125+0000 7f8bbcff9640 1 -- 172.21.15.27:0/3692188325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8ba8004e30 con 0x7f8bc0104d50 2024-04-25T13:40:49.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.125+0000 7f8bbcff9640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8bac0afaf0 0x7f8bac0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.125+0000 7f8bbf7fe640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8bac0afaf0 0x7f8bac0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.125+0000 7f8bbcff9640 1 -- 172.21.15.27:0/3692188325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8ba80df350 con 0x7f8bc0104d50 2024-04-25T13:40:49.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.126+0000 7f8bbf7fe640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8bac0afaf0 0x7f8bac0b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0104710 tx=0x7f8bb0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:49.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.129+0000 7f8bbcff9640 1 -- 172.21.15.27:0/3692188325 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ba8053ff0 con 0x7f8bc0104d50 2024-04-25T13:40:49.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.316+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8bc002dda0 con 0x7f8bc0104d50 2024-04-25T13:40:49.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.318+0000 7f8bbcff9640 1 -- 172.21.15.27:0/3692188325 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7f8ba80a7f60 con 0x7f8bc0104d50 2024-04-25T13:40:49.318 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:49.318 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.319+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8bac0afaf0 msgr2=0x7f8bac0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.319+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8bac0afaf0 0x7f8bac0b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0104710 tx=0x7f8bb0009500 comp rx=0 tx=0).stop 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.319+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0104d50 msgr2=0x7f8bc019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.319+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0104d50 0x7f8bc019a750 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800e9f0 tx=0x7f8ba800edf0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.320+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 shutdown_connections 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.320+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8bac0afaf0 0x7f8bac0b2000 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.320+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bc0109a50 0x7f8bc01a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.320+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bc0104d50 0x7f8bc019a750 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.320+0000 7f8bc5d82640 1 --2- 172.21.15.27:0/3692188325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bc0103b10 0x7f8bc019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.320+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 >> 172.21.15.27:0/3692188325 conn(0x7f8bc00fed40 msgr2=0x7f8bc0105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:49.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.320+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 shutdown_connections 2024-04-25T13:40:49.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.320+0000 7f8bc5d82640 1 -- 172.21.15.27:0/3692188325 wait complete. 2024-04-25T13:40:49.331 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:40:49.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.447+0000 7f779d784640 1 -- 172.21.15.27:0/1356523652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798106f50 msgr2=0x7f7798107360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.447+0000 7f779d784640 1 --2- 172.21.15.27:0/1356523652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798106f50 0x7f7798107360 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f778c009900 tx=0x7f778c02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.447+0000 7f779d784640 1 -- 172.21.15.27:0/1356523652 shutdown_connections 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.447+0000 7f779d784640 1 --2- 172.21.15.27:0/1356523652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f779810bc50 0x7f779810e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.447+0000 7f779d784640 1 --2- 172.21.15.27:0/1356523652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798106f50 0x7f7798107360 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.447+0000 7f779d784640 1 --2- 172.21.15.27:0/1356523652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7798105d10 0x7f7798106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.447+0000 7f779d784640 1 -- 172.21.15.27:0/1356523652 >> 172.21.15.27:0/1356523652 conn(0x7f7798100f20 msgr2=0x7f77981033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.447+0000 7f779d784640 1 -- 172.21.15.27:0/1356523652 shutdown_connections 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.448+0000 7f779d784640 1 -- 172.21.15.27:0/1356523652 wait complete. 2024-04-25T13:40:49.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.448+0000 7f779d784640 1 Processor -- start 2024-04-25T13:40:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.448+0000 7f779d784640 1 -- start start 2024-04-25T13:40:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.448+0000 7f779d784640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798105d10 0x7f779819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.448+0000 7f779d784640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7798106f50 0x7f779819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.448+0000 7f779d784640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f779810bc50 0x7f77981a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.448+0000 7f779d784640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7798111880 con 0x7f7798105d10 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f779d784640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7798111700 con 0x7f779810bc50 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f779d784640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7798111a00 con 0x7f7798106f50 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798105d10 0x7f779819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798105d10 0x7f779819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54224/0 (socket says 172.21.15.27:54224) 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 -- 172.21.15.27:0/1194363028 learned_addr learned my addr 172.21.15.27:0/1194363028 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f77977fe640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f779810bc50 0x7f77981a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f77967fc640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7798106f50 0x7f779819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 -- 172.21.15.27:0/1194363028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7798106f50 msgr2=0x7f779819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7798106f50 0x7f779819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 -- 172.21.15.27:0/1194363028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f779810bc50 msgr2=0x7f77981a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f779810bc50 0x7f77981a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 -- 172.21.15.27:0/1194363028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77981a4000 con 0x7f7798105d10 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f7796ffd640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798105d10 0x7f779819c4c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7788005d00 tx=0x7f778800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f777ffff640 1 -- 172.21.15.27:0/1194363028 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788017070 con 0x7f7798105d10 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f777ffff640 1 -- 172.21.15.27:0/1194363028 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7788009c30 con 0x7f7798105d10 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77981a42f0 con 0x7f7798105d10 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f777ffff640 1 -- 172.21.15.27:0/1194363028 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778800f620 con 0x7f7798105d10 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.449+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77981a47f0 con 0x7f7798105d10 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.450+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f779810aa60 con 0x7f7798105d10 2024-04-25T13:40:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.453+0000 7f777ffff640 1 -- 172.21.15.27:0/1194363028 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f778801a070 con 0x7f7798105d10 2024-04-25T13:40:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.453+0000 7f777ffff640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f77800afaf0 0x7f77800b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.453+0000 7f77967fc640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f77800afaf0 0x7f77800b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.454+0000 7f777ffff640 1 -- 172.21.15.27:0/1194363028 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f77880de660 con 0x7f7798105d10 2024-04-25T13:40:49.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.454+0000 7f77967fc640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f77800afaf0 0x7f77800b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f778c006010 tx=0x7f778c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.457+0000 7f777ffff640 1 -- 172.21.15.27:0/1194363028 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7788009de0 con 0x7f7798105d10 2024-04-25T13:40:49.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.646+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77980637a0 con 0x7f7798105d10 2024-04-25T13:40:49.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.646+0000 7f777ffff640 1 -- 172.21.15.27:0/1194363028 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f77880c7e20 con 0x7f7798105d10 2024-04-25T13:40:49.646 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:40:49.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f77800afaf0 msgr2=0x7f77800b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f77800afaf0 0x7f77800b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f778c006010 tx=0x7f778c031400 comp rx=0 tx=0).stop 2024-04-25T13:40:49.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798105d10 msgr2=0x7f779819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798105d10 0x7f779819c4c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7788005d00 tx=0x7f778800dde0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 shutdown_connections 2024-04-25T13:40:49.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f77800afaf0 0x7f77800b2000 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f779810bc50 0x7f77981a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7798106f50 0x7f779819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 --2- 172.21.15.27:0/1194363028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798105d10 0x7f779819c4c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 >> 172.21.15.27:0/1194363028 conn(0x7f7798100f20 msgr2=0x7f7798107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:49.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 shutdown_connections 2024-04-25T13:40:49.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.648+0000 7f779d784640 1 -- 172.21.15.27:0/1194363028 wait complete. 2024-04-25T13:40:49.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:40:49.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.774+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/2588002961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac106c60 msgr2=0x7f4bac109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.774+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/2588002961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac106c60 0x7f4bac109100 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c009900 tx=0x7f4b9c02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.774+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/2588002961 shutdown_connections 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.774+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/2588002961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac109790 0x7f4bac10bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.774+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/2588002961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac106c60 0x7f4bac109100 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.774+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/2588002961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bac104270 0x7f4bac106690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.774+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/2588002961 >> 172.21.15.27:0/2588002961 conn(0x7f4bac0fdbf0 msgr2=0x7f4bac1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.775+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/2588002961 shutdown_connections 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.775+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/2588002961 wait complete. 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.775+0000 7f4bb1bc7640 1 Processor -- start 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.775+0000 7f4bb1bc7640 1 -- start start 2024-04-25T13:40:49.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.775+0000 7f4bb1bc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac104270 0x7f4bac19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bb1bc7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bac106c60 0x7f4bac19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bb1bc7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac109790 0x7f4bac1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bb1bc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac10f600 con 0x7f4bac104270 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bb1bc7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac10f480 con 0x7f4bac109790 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bb1bc7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac10f780 con 0x7f4bac106c60 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac104270 0x7f4bac19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac104270 0x7f4bac19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54240/0 (socket says 172.21.15.27:54240) 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 -- 172.21.15.27:0/1309171540 learned_addr learned my addr 172.21.15.27:0/1309171540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4baaffd640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bac106c60 0x7f4bac19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4babfff640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac109790 0x7f4bac1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 -- 172.21.15.27:0/1309171540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bac106c60 msgr2=0x7f4bac19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bac106c60 0x7f4bac19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 -- 172.21.15.27:0/1309171540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac109790 msgr2=0x7f4bac1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac109790 0x7f4bac1a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 -- 172.21.15.27:0/1309171540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac1a1d40 con 0x7f4bac104270 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bab7fe640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac104270 0x7f4bac19a200 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0005fd0 tx=0x7f4ba000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4ba8ff9640 1 -- 172.21.15.27:0/1309171540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0017070 con 0x7f4bac104270 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4ba8ff9640 1 -- 172.21.15.27:0/1309171540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ba0009c30 con 0x7f4bac104270 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac1a1fd0 con 0x7f4bac104270 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4ba8ff9640 1 -- 172.21.15.27:0/1309171540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0003d30 con 0x7f4bac104270 2024-04-25T13:40:49.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.776+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bac1a2490 con 0x7f4bac104270 2024-04-25T13:40:49.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.777+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bac0ff710 con 0x7f4bac104270 2024-04-25T13:40:49.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.779+0000 7f4ba8ff9640 1 -- 172.21.15.27:0/1309171540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4ba0003ed0 con 0x7f4bac104270 2024-04-25T13:40:49.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.779+0000 7f4ba8ff9640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4b940afaf0 0x7f4b940b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:49.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.780+0000 7f4baaffd640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4b940afaf0 0x7f4b940b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:49.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.780+0000 7f4ba8ff9640 1 -- 172.21.15.27:0/1309171540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4ba00df0f0 con 0x7f4bac104270 2024-04-25T13:40:49.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.780+0000 7f4baaffd640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4b940afaf0 0x7f4b940b2000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c000c00 tx=0x7f4b9c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:49.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.783+0000 7f4ba8ff9640 1 -- 172.21.15.27:0/1309171540 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ba0020030 con 0x7f4bac104270 2024-04-25T13:40:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.970+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4bac02dce0 con 0x7f4bac104270 2024-04-25T13:40:49.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.971+0000 7f4ba8ff9640 1 -- 172.21.15.27:0/1309171540 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f4ba0009de0 con 0x7f4bac104270 2024-04-25T13:40:49.971 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:40:49.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4b940afaf0 msgr2=0x7f4b940b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4b940afaf0 0x7f4b940b2000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c000c00 tx=0x7f4b9c031400 comp rx=0 tx=0).stop 2024-04-25T13:40:49.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac104270 msgr2=0x7f4bac19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:49.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac104270 0x7f4bac19a200 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0005fd0 tx=0x7f4ba000dde0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 shutdown_connections 2024-04-25T13:40:49.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4b940afaf0 0x7f4b940b2000 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4bac109790 0x7f4bac1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bac106c60 0x7f4bac19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 --2- 172.21.15.27:0/1309171540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bac104270 0x7f4bac19a200 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:49.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 >> 172.21.15.27:0/1309171540 conn(0x7f4bac0fdbf0 msgr2=0x7f4bac106250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:49.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 shutdown_connections 2024-04-25T13:40:49.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:49.973+0000 7f4bb1bc7640 1 -- 172.21.15.27:0/1309171540 wait complete. 2024-04-25T13:40:49.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:40:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 -- 172.21.15.27:0/3052100790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 msgr2=0x7f4ab01047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3052100790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 0x7f4ab01047f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009900 tx=0x7f4a9802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 -- 172.21.15.27:0/3052100790 shutdown_connections 2024-04-25T13:40:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3052100790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab0109a30 0x7f4ab010be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3052100790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab0104d30 0x7f4ab0105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3052100790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 0x7f4ab01047f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 -- 172.21.15.27:0/3052100790 >> 172.21.15.27:0/3052100790 conn(0x7f4ab00fece0 msgr2=0x7f4ab0101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:50.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 -- 172.21.15.27:0/3052100790 shutdown_connections 2024-04-25T13:40:50.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.099+0000 7f4ab516c640 1 -- 172.21.15.27:0/3052100790 wait complete. 2024-04-25T13:40:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4ab516c640 1 Processor -- start 2024-04-25T13:40:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4ab516c640 1 -- start start 2024-04-25T13:40:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4ab516c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 0x7f4ab019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4ab516c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab0104d30 0x7f4ab019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4ab516c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab0109a30 0x7f4ab01a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4ab516c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab010f5d0 con 0x7f4ab0103af0 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4ab516c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab010f450 con 0x7f4ab0104d30 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4ab516c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab010f750 con 0x7f4ab0109a30 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4aaed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 0x7f4ab019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4aaed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 0x7f4ab019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54248/0 (socket says 172.21.15.27:54248) 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4aaed9d640 1 -- 172.21.15.27:0/3672154723 learned_addr learned my addr 172.21.15.27:0/3672154723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.100+0000 7f4aaf59e640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab0109a30 0x7f4ab01a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aae59c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab0104d30 0x7f4ab019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aaed9d640 1 -- 172.21.15.27:0/3672154723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab0109a30 msgr2=0x7f4ab01a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aaed9d640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab0109a30 0x7f4ab01a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aaed9d640 1 -- 172.21.15.27:0/3672154723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab0104d30 msgr2=0x7f4ab019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aaed9d640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab0104d30 0x7f4ab019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aaed9d640 1 -- 172.21.15.27:0/3672154723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab01a1d80 con 0x7f4ab0103af0 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aae59c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab0104d30 0x7f4ab019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aaed9d640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 0x7f4ab019a290 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4a98006010 tx=0x7f4a98004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aa7fff640 1 -- 172.21.15.27:0/3672154723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9803f030 con 0x7f4ab0103af0 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aa7fff640 1 -- 172.21.15.27:0/3672154723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a98003aa0 con 0x7f4ab0103af0 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab01a2010 con 0x7f4ab0103af0 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4aa7fff640 1 -- 172.21.15.27:0/3672154723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a98035070 con 0x7f4ab0103af0 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.101+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab01a2500 con 0x7f4ab0103af0 2024-04-25T13:40:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.102+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab0108840 con 0x7f4ab0103af0 2024-04-25T13:40:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.104+0000 7f4aa7fff640 1 -- 172.21.15.27:0/3672154723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4a98042030 con 0x7f4ab0103af0 2024-04-25T13:40:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.104+0000 7f4aa7fff640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a9c0afaf0 0x7f4a9c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.104+0000 7f4aae59c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a9c0afaf0 0x7f4a9c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.104+0000 7f4aa7fff640 1 -- 172.21.15.27:0/3672154723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4a980ff6a0 con 0x7f4ab0103af0 2024-04-25T13:40:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.105+0000 7f4aae59c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a9c0afaf0 0x7f4a9c0b2000 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0005fd0 tx=0x7f4aa0005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:50.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.108+0000 7f4aa7fff640 1 -- 172.21.15.27:0/3672154723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a980c82b0 con 0x7f4ab0103af0 2024-04-25T13:40:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.294+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4ab00637a0 con 0x7f4ab0103af0 2024-04-25T13:40:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.294+0000 7f4aa7fff640 1 -- 172.21.15.27:0/3672154723 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f4a980c7bb0 con 0x7f4ab0103af0 2024-04-25T13:40:50.294 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:40:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a9c0afaf0 msgr2=0x7f4a9c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a9c0afaf0 0x7f4a9c0b2000 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0005fd0 tx=0x7f4aa0005ec0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 msgr2=0x7f4ab019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 0x7f4ab019a290 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4a98006010 tx=0x7f4a98004270 comp rx=0 tx=0).stop 2024-04-25T13:40:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 shutdown_connections 2024-04-25T13:40:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4a9c0afaf0 0x7f4a9c0b2000 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ab0109a30 0x7f4ab01a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ab0104d30 0x7f4ab019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 --2- 172.21.15.27:0/3672154723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ab0103af0 0x7f4ab019a290 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 >> 172.21.15.27:0/3672154723 conn(0x7f4ab00fece0 msgr2=0x7f4ab0105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 shutdown_connections 2024-04-25T13:40:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.296+0000 7f4ab516c640 1 -- 172.21.15.27:0/3672154723 wait complete. 2024-04-25T13:40:50.306 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:50 smithi027 ceph-mon[29140]: pgmap v347: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:50.306 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:50 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3692188325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:50.307 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:50 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1194363028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:50.307 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:50 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1309171540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:50.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:50.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:50 smithi122 ceph-mon[32843]: pgmap v347: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:50.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:50 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3692188325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:50.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:50 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1194363028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:50.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:50 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1309171540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:50.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:50 smithi045 ceph-mon[32731]: pgmap v347: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-25T13:40:50.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:50 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3692188325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:50.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:50 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1194363028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:50.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:50 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1309171540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.420+0000 7fa8707c5640 1 -- 172.21.15.27:0/2253700235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868108920 msgr2=0x7fa86810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.420+0000 7fa8707c5640 1 --2- 172.21.15.27:0/2253700235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868108920 0x7fa86810ad40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa860009ca0 tx=0x7fa86002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.420+0000 7fa8707c5640 1 -- 172.21.15.27:0/2253700235 shutdown_connections 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.420+0000 7fa8707c5640 1 --2- 172.21.15.27:0/2253700235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868108920 0x7fa86810ad40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.420+0000 7fa8707c5640 1 --2- 172.21.15.27:0/2253700235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa868103c20 0x7fa868104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.420+0000 7fa8707c5640 1 --2- 172.21.15.27:0/2253700235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8681029e0 0x7fa8681036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.420+0000 7fa8707c5640 1 -- 172.21.15.27:0/2253700235 >> 172.21.15.27:0/2253700235 conn(0x7fa8680fdbd0 msgr2=0x7fa868100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 -- 172.21.15.27:0/2253700235 shutdown_connections 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 -- 172.21.15.27:0/2253700235 wait complete. 2024-04-25T13:40:50.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 Processor -- start 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 -- start start 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8681029e0 0x7fa868075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868103c20 0x7fa86819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa868108920 0x7fa8681a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810e510 con 0x7fa868103c20 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810e390 con 0x7fa8681029e0 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa8707c5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810e690 con 0x7fa868108920 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.421+0000 7fa86dd60640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868103c20 0x7fa86819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86dd60640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868103c20 0x7fa86819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54276/0 (socket says 172.21.15.27:54276) 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86dd60640 1 -- 172.21.15.27:0/4003161358 learned_addr learned my addr 172.21.15.27:0/4003161358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86e561640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8681029e0 0x7fa868075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86ed62640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa868108920 0x7fa8681a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86dd60640 1 -- 172.21.15.27:0/4003161358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa868108920 msgr2=0x7fa8681a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86dd60640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa868108920 0x7fa8681a1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86dd60640 1 -- 172.21.15.27:0/4003161358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8681029e0 msgr2=0x7fa868075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86dd60640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8681029e0 0x7fa868075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86dd60640 1 -- 172.21.15.27:0/4003161358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8681a1cf0 con 0x7fa868103c20 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa86dd60640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868103c20 0x7fa86819f010 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa85800c920 tx=0x7fa85800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa85f7fe640 1 -- 172.21.15.27:0/4003161358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858007c70 con 0x7fa868103c20 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa85f7fe640 1 -- 172.21.15.27:0/4003161358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa858007e10 con 0x7fa868103c20 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8681a1fe0 con 0x7fa868103c20 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa85f7fe640 1 -- 172.21.15.27:0/4003161358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa858018820 con 0x7fa868103c20 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.422+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8681a2450 con 0x7fa868103c20 2024-04-25T13:40:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.423+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa868107730 con 0x7fa868103c20 2024-04-25T13:40:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.425+0000 7fa85f7fe640 1 -- 172.21.15.27:0/4003161358 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa85800f450 con 0x7fa868103c20 2024-04-25T13:40:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.425+0000 7fa85f7fe640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8540afaf0 0x7fa8540b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.425+0000 7fa86e561640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8540afaf0 0x7fa8540b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.425+0000 7fa85f7fe640 1 -- 172.21.15.27:0/4003161358 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa8580defd0 con 0x7fa868103c20 2024-04-25T13:40:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.426+0000 7fa86e561640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8540afaf0 0x7fa8540b2000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa864005fd0 tx=0x7fa86400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.429+0000 7fa85f7fe640 1 -- 172.21.15.27:0/4003161358 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa858014030 con 0x7fa868103c20 2024-04-25T13:40:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.617+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa86802dda0 con 0x7fa868103c20 2024-04-25T13:40:50.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.619+0000 7fa85f7fe640 1 -- 172.21.15.27:0/4003161358 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7fa8580a7c70 con 0x7fa868103c20 2024-04-25T13:40:50.619 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:50.619 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8540afaf0 msgr2=0x7fa8540b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8540afaf0 0x7fa8540b2000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa864005fd0 tx=0x7fa86400b390 comp rx=0 tx=0).stop 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868103c20 msgr2=0x7fa86819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868103c20 0x7fa86819f010 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa85800c920 tx=0x7fa85800cde0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 shutdown_connections 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8540afaf0 0x7fa8540b2000 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa868108920 0x7fa8681a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa868103c20 0x7fa86819f010 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 --2- 172.21.15.27:0/4003161358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8681029e0 0x7fa868075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 >> 172.21.15.27:0/4003161358 conn(0x7fa8680fdbd0 msgr2=0x7fa868104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:50.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 shutdown_connections 2024-04-25T13:40:50.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.621+0000 7fa8707c5640 1 -- 172.21.15.27:0/4003161358 wait complete. 2024-04-25T13:40:50.632 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:40:50.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:40:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2556989604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 msgr2=0x7f64dc1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2556989604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 0x7f64dc1036e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009900 tx=0x7f64cc02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2556989604 shutdown_connections 2024-04-25T13:40:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2556989604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64dc108920 0x7f64dc10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2556989604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64dc103c20 0x7f64dc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2556989604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 0x7f64dc1036e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2556989604 >> 172.21.15.27:0/2556989604 conn(0x7f64dc0fdbd0 msgr2=0x7f64dc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2556989604 shutdown_connections 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.748+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2556989604 wait complete. 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e3ce9640 1 Processor -- start 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e3ce9640 1 -- start start 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e3ce9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 0x7f64dc199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e3ce9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64dc103c20 0x7f64dc199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e3ce9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64dc108920 0x7f64dc1a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e3ce9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10e4e0 con 0x7f64dc1029e0 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e3ce9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10e360 con 0x7f64dc103c20 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e3ce9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10e660 con 0x7f64dc108920 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 0x7f64dc199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 0x7f64dc199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54298/0 (socket says 172.21.15.27:54298) 2024-04-25T13:40:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 -- 172.21.15.27:0/2294714660 learned_addr learned my addr 172.21.15.27:0/2294714660 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1284640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64dc103c20 0x7f64dc199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e2286640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64dc108920 0x7f64dc1a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 -- 172.21.15.27:0/2294714660 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64dc108920 msgr2=0x7f64dc1a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64dc108920 0x7f64dc1a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 -- 172.21.15.27:0/2294714660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64dc103c20 msgr2=0x7f64dc199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64dc103c20 0x7f64dc199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 -- 172.21.15.27:0/2294714660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc1a0f50 con 0x7f64dc1029e0 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.749+0000 7f64e1a85640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 0x7f64dc199410 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f64cc006010 tx=0x7f64cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.750+0000 7f64d2ffd640 1 -- 172.21.15.27:0/2294714660 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc035070 con 0x7f64dc1029e0 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.750+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc069e90 con 0x7f64dc1029e0 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.750+0000 7f64d2ffd640 1 -- 172.21.15.27:0/2294714660 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc003aa0 con 0x7f64dc1029e0 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.750+0000 7f64d2ffd640 1 -- 172.21.15.27:0/2294714660 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc03bbd0 con 0x7f64dc1029e0 2024-04-25T13:40:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.750+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64dc06a380 con 0x7f64dc1029e0 2024-04-25T13:40:50.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.751+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64dc06c3c0 con 0x7f64dc1029e0 2024-04-25T13:40:50.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.752+0000 7f64d2ffd640 1 -- 172.21.15.27:0/2294714660 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f64cc003c50 con 0x7f64dc1029e0 2024-04-25T13:40:50.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.753+0000 7f64d2ffd640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64c80afaf0 0x7f64c80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:50.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.753+0000 7f64e1284640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64c80afaf0 0x7f64c80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:50.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.753+0000 7f64d2ffd640 1 -- 172.21.15.27:0/2294714660 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f64cc101790 con 0x7f64dc1029e0 2024-04-25T13:40:50.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.754+0000 7f64e1284640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64c80afaf0 0x7f64c80b2000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f64dc19a4b0 tx=0x7f64d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:50.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.757+0000 7f64d2ffd640 1 -- 172.21.15.27:0/2294714660 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64cc0ca3a0 con 0x7f64dc1029e0 2024-04-25T13:40:50.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.942+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f64dc06af30 con 0x7f64dc1029e0 2024-04-25T13:40:50.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.942+0000 7f64d2ffd640 1 -- 172.21.15.27:0/2294714660 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f64dc06af30 con 0x7f64dc1029e0 2024-04-25T13:40:50.942 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.944+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64c80afaf0 msgr2=0x7f64c80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.944+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64c80afaf0 0x7f64c80b2000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f64dc19a4b0 tx=0x7f64d8005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.944+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 msgr2=0x7f64dc199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.944+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 0x7f64dc199410 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f64cc006010 tx=0x7f64cc004270 comp rx=0 tx=0).stop 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.945+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 shutdown_connections 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.945+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64c80afaf0 0x7f64c80b2000 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.945+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64dc108920 0x7f64dc1a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.945+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64dc103c20 0x7f64dc199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.945+0000 7f64e3ce9640 1 --2- 172.21.15.27:0/2294714660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc1029e0 0x7f64dc199410 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.945+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 >> 172.21.15.27:0/2294714660 conn(0x7f64dc0fdbd0 msgr2=0x7f64dc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.945+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 shutdown_connections 2024-04-25T13:40:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:50.945+0000 7f64e3ce9640 1 -- 172.21.15.27:0/2294714660 wait complete. 2024-04-25T13:40:50.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:40:51.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 -- 172.21.15.27:0/3447760505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41080b0 msgr2=0x7febb410a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 --2- 172.21.15.27:0/3447760505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41080b0 0x7febb410a4d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7febac009ca0 tx=0x7febac02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 -- 172.21.15.27:0/3447760505 shutdown_connections 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 --2- 172.21.15.27:0/3447760505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41080b0 0x7febb410a4d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 --2- 172.21.15.27:0/3447760505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb41033b0 0x7febb41037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 --2- 172.21.15.27:0/3447760505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb4102170 0x7febb4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 -- 172.21.15.27:0/3447760505 >> 172.21.15.27:0/3447760505 conn(0x7febb40fd380 msgr2=0x7febb40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 -- 172.21.15.27:0/3447760505 shutdown_connections 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.071+0000 7febbb584640 1 -- 172.21.15.27:0/3447760505 wait complete. 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febbb584640 1 Processor -- start 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febbb584640 1 -- start start 2024-04-25T13:40:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febbb584640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb4102170 0x7febb419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febbb584640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41033b0 0x7febb419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febbb584640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41080b0 0x7febb41a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febbb584640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7febb410d400 con 0x7febb41033b0 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febbb584640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7febb410d280 con 0x7febb41080b0 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febbb584640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7febb410d580 con 0x7febb4102170 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febb8b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41033b0 0x7febb419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febb8b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41033b0 0x7febb419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54310/0 (socket says 172.21.15.27:54310) 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febb8b1f640 1 -- 172.21.15.27:0/2452749122 learned_addr learned my addr 172.21.15.27:0/2452749122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.072+0000 7febb9320640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb4102170 0x7febb419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febb8b1f640 1 -- 172.21.15.27:0/2452749122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb4102170 msgr2=0x7febb419a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febb9b21640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41080b0 0x7febb41a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febb8b1f640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb4102170 0x7febb419a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febb8b1f640 1 -- 172.21.15.27:0/2452749122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41080b0 msgr2=0x7febb41a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febb8b1f640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41080b0 0x7febb41a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febb8b1f640 1 -- 172.21.15.27:0/2452749122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb41a1cd0 con 0x7febb41033b0 2024-04-25T13:40:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febb9b21640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41080b0 0x7febb41a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febb8b1f640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41033b0 0x7febb419a6d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feba400d950 tx=0x7feba400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febaa7fc640 1 -- 172.21.15.27:0/2452749122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba4017030 con 0x7febb41033b0 2024-04-25T13:40:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febaa7fc640 1 -- 172.21.15.27:0/2452749122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feba4004020 con 0x7febb41033b0 2024-04-25T13:40:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb41a1fc0 con 0x7febb41033b0 2024-04-25T13:40:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febaa7fc640 1 -- 172.21.15.27:0/2452749122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba40047a0 con 0x7febb41033b0 2024-04-25T13:40:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.073+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb41a2430 con 0x7febb41033b0 2024-04-25T13:40:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.074+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb4106ec0 con 0x7febb41033b0 2024-04-25T13:40:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.076+0000 7febaa7fc640 1 -- 172.21.15.27:0/2452749122 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7feba4009a30 con 0x7febb41033b0 2024-04-25T13:40:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.076+0000 7febaa7fc640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feba00afaf0 0x7feba00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.076+0000 7febb9320640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feba00afaf0 0x7feba00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.076+0000 7febaa7fc640 1 -- 172.21.15.27:0/2452749122 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feba40deca0 con 0x7febb41033b0 2024-04-25T13:40:51.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.077+0000 7febb9320640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feba00afaf0 0x7feba00b2000 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7febb4102d70 tx=0x7febb000b490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.080+0000 7febaa7fc640 1 -- 172.21.15.27:0/2452749122 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feba40a78b0 con 0x7febb41033b0 2024-04-25T13:40:51.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.266+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7febb402dda0 con 0x7febb41033b0 2024-04-25T13:40:51.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.266+0000 7febaa7fc640 1 -- 172.21.15.27:0/2452749122 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7feba40a71b0 con 0x7febb41033b0 2024-04-25T13:40:51.266 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:40:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feba00afaf0 msgr2=0x7feba00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feba00afaf0 0x7feba00b2000 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7febb4102d70 tx=0x7febb000b490 comp rx=0 tx=0).stop 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41033b0 msgr2=0x7febb419a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41033b0 0x7febb419a6d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feba400d950 tx=0x7feba400de10 comp rx=0 tx=0).stop 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 shutdown_connections 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feba00afaf0 0x7feba00b2000 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41080b0 0x7febb41a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febb41033b0 0x7febb419a6d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 --2- 172.21.15.27:0/2452749122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7febb4102170 0x7febb419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 >> 172.21.15.27:0/2452749122 conn(0x7febb40fd380 msgr2=0x7febb4104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 shutdown_connections 2024-04-25T13:40:51.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.268+0000 7febbb584640 1 -- 172.21.15.27:0/2452749122 wait complete. 2024-04-25T13:40:51.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:51 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3672154723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:51.278 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:51 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/4003161358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:51.279 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:51 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2294714660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:51.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:40:51.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:51 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3672154723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:51.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:51 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/4003161358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:51.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:51 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2294714660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:51.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:51 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3672154723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:51.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:51 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/4003161358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:51.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:51 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2294714660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 -- 172.21.15.27:0/2053978908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 msgr2=0x7f22701069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 --2- 172.21.15.27:0/2053978908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 0x7f22701069f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2268009ca0 tx=0x7f226802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 -- 172.21.15.27:0/2053978908 shutdown_connections 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 --2- 172.21.15.27:0/2053978908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2270109890 0x7f227010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 --2- 172.21.15.27:0/2053978908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2270106f30 0x7f2270109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 --2- 172.21.15.27:0/2053978908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 0x7f22701069f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 -- 172.21.15.27:0/2053978908 >> 172.21.15.27:0/2053978908 conn(0x7f22700fdbd0 msgr2=0x7f2270100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 -- 172.21.15.27:0/2053978908 shutdown_connections 2024-04-25T13:40:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.394+0000 7f2276af9640 1 -- 172.21.15.27:0/2053978908 wait complete. 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2276af9640 1 Processor -- start 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2276af9640 1 -- start start 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2276af9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 0x7f2270103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2276af9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2270106f30 0x7f22701020c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2276af9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2270109890 0x7f2270102600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2276af9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f227010f5e0 con 0x7f22701045d0 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2276af9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f227010f460 con 0x7f2270106f30 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2276af9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f227010f760 con 0x7f2270109890 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 0x7f2270103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 0x7f2270103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54328/0 (socket says 172.21.15.27:54328) 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 -- 172.21.15.27:0/59078526 learned_addr learned my addr 172.21.15.27:0/59078526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f226ffff640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2270106f30 0x7f22701020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2275096640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2270109890 0x7f2270102600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 -- 172.21.15.27:0/59078526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2270109890 msgr2=0x7f2270102600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2270109890 0x7f2270102600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 -- 172.21.15.27:0/59078526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2270106f30 msgr2=0x7f22701020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2270106f30 0x7f22701020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 -- 172.21.15.27:0/59078526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2270102d30 con 0x7f22701045d0 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f226ffff640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2270106f30 0x7f22701020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2275096640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2270109890 0x7f2270102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.395+0000 7f2274895640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 0x7f2270103a20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f226802ade0 tx=0x7f2268003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.396+0000 7f226dffb640 1 -- 172.21.15.27:0/59078526 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226802f380 con 0x7f22701045d0 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.396+0000 7f226dffb640 1 -- 172.21.15.27:0/59078526 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22680317c0 con 0x7f22701045d0 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.396+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22701a4650 con 0x7f22701045d0 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.396+0000 7f226dffb640 1 -- 172.21.15.27:0/59078526 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226802fc60 con 0x7f22701045d0 2024-04-25T13:40:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.396+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22701a4b10 con 0x7f22701045d0 2024-04-25T13:40:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.397+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227006c2a0 con 0x7f22701045d0 2024-04-25T13:40:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.398+0000 7f226dffb640 1 -- 172.21.15.27:0/59078526 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2268004040 con 0x7f22701045d0 2024-04-25T13:40:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.399+0000 7f226dffb640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f225c0afd10 0x7f225c0b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.399+0000 7f226ffff640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f225c0afd10 0x7f225c0b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.399+0000 7f226dffb640 1 -- 172.21.15.27:0/59078526 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2268100350 con 0x7f22701045d0 2024-04-25T13:40:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.400+0000 7f226ffff640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f225c0afd10 0x7f225c0b2220 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f22701033c0 tx=0x7f2260005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.403+0000 7f226dffb640 1 -- 172.21.15.27:0/59078526 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22680c8f60 con 0x7f22701045d0 2024-04-25T13:40:51.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.591+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22701a4d40 con 0x7f22701045d0 2024-04-25T13:40:51.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.591+0000 7f226dffb640 1 -- 172.21.15.27:0/59078526 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f22680c8860 con 0x7f22701045d0 2024-04-25T13:40:51.591 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f225c0afd10 msgr2=0x7f225c0b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f225c0afd10 0x7f225c0b2220 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f22701033c0 tx=0x7f2260005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 msgr2=0x7f2270103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 0x7f2270103a20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f226802ade0 tx=0x7f2268003ba0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 shutdown_connections 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f225c0afd10 0x7f225c0b2220 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2270109890 0x7f2270102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2270106f30 0x7f22701020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 --2- 172.21.15.27:0/59078526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22701045d0 0x7f2270103a20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 >> 172.21.15.27:0/59078526 conn(0x7f22700fdbd0 msgr2=0x7f2270104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:51.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 shutdown_connections 2024-04-25T13:40:51.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.594+0000 7f2276af9640 1 -- 172.21.15.27:0/59078526 wait complete. 2024-04-25T13:40:51.605 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T13:40:51.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 -- 172.21.15.27:0/2453296063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 msgr2=0x7fce181036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 --2- 172.21.15.27:0/2453296063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 0x7fce181036c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fce08009900 tx=0x7fce0802a3f0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 -- 172.21.15.27:0/2453296063 shutdown_connections 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 --2- 172.21.15.27:0/2453296063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce18108900 0x7fce1810ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 --2- 172.21.15.27:0/2453296063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce18103c00 0x7fce18104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 --2- 172.21.15.27:0/2453296063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 0x7fce181036c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 -- 172.21.15.27:0/2453296063 >> 172.21.15.27:0/2453296063 conn(0x7fce180fdbd0 msgr2=0x7fce18100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 -- 172.21.15.27:0/2453296063 shutdown_connections 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.717+0000 7fce1fd26640 1 -- 172.21.15.27:0/2453296063 wait complete. 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1fd26640 1 Processor -- start 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1fd26640 1 -- start start 2024-04-25T13:40:51.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1fd26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 0x7fce1819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1fd26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce18103c00 0x7fce1819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1fd26640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce18108900 0x7fce181a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1fd26640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810f5b0 con 0x7fce181029c0 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1fd26640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810f430 con 0x7fce18103c00 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1fd26640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fce1810f730 con 0x7fce18108900 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 0x7fce1819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 0x7fce1819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54346/0 (socket says 172.21.15.27:54346) 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 -- 172.21.15.27:0/3937807007 learned_addr learned my addr 172.21.15.27:0/3937807007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1e2c3640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce18108900 0x7fce181a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1d2c1640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce18103c00 0x7fce1819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 -- 172.21.15.27:0/3937807007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce18108900 msgr2=0x7fce181a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce18108900 0x7fce181a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 -- 172.21.15.27:0/3937807007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce18103c00 msgr2=0x7fce1819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce18103c00 0x7fce1819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 -- 172.21.15.27:0/3937807007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce181a1c20 con 0x7fce181029c0 2024-04-25T13:40:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.718+0000 7fce1dac2640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 0x7fce1819a190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fce08032fd0 tx=0x7fce080042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.719+0000 7fce0effd640 1 -- 172.21.15.27:0/3937807007 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0803f070 con 0x7fce181029c0 2024-04-25T13:40:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.719+0000 7fce0effd640 1 -- 172.21.15.27:0/3937807007 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce080044d0 con 0x7fce181029c0 2024-04-25T13:40:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.719+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce181a1eb0 con 0x7fce181029c0 2024-04-25T13:40:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.719+0000 7fce0effd640 1 -- 172.21.15.27:0/3937807007 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08035070 con 0x7fce181029c0 2024-04-25T13:40:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.719+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce181a2310 con 0x7fce181029c0 2024-04-25T13:40:51.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.720+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce18107710 con 0x7fce181029c0 2024-04-25T13:40:51.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.721+0000 7fce0effd640 1 -- 172.21.15.27:0/3937807007 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fce0803b5f0 con 0x7fce181029c0 2024-04-25T13:40:51.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.722+0000 7fce0effd640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce040afb70 0x7fce040b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:51.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.722+0000 7fce1d2c1640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce040afb70 0x7fce040b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:51.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.722+0000 7fce0effd640 1 -- 172.21.15.27:0/3937807007 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fce080ff860 con 0x7fce181029c0 2024-04-25T13:40:51.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.723+0000 7fce1d2c1640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce040afb70 0x7fce040b2080 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fce1819b230 tx=0x7fce140082b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:51.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.726+0000 7fce0effd640 1 -- 172.21.15.27:0/3937807007 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce080c8470 con 0x7fce181029c0 2024-04-25T13:40:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.950+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce1802db50 con 0x7fce181029c0 2024-04-25T13:40:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.951+0000 7fce0effd640 1 -- 172.21.15.27:0/3937807007 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fce080c7d70 con 0x7fce181029c0 2024-04-25T13:40:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:51.951 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:40:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.953+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce040afb70 msgr2=0x7fce040b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.953+0000 7fce1fd26640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce040afb70 0x7fce040b2080 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fce1819b230 tx=0x7fce140082b0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.953+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 msgr2=0x7fce1819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.953+0000 7fce1fd26640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 0x7fce1819a190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fce08032fd0 tx=0x7fce080042e0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.953+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 shutdown_connections 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.953+0000 7fce1fd26640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce040afb70 0x7fce040b2080 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.954+0000 7fce1fd26640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce18108900 0x7fce181a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.954+0000 7fce1fd26640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce18103c00 0x7fce1819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.954+0000 7fce1fd26640 1 --2- 172.21.15.27:0/3937807007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce181029c0 0x7fce1819a190 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.954+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 >> 172.21.15.27:0/3937807007 conn(0x7fce180fdbd0 msgr2=0x7fce181048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.954+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 shutdown_connections 2024-04-25T13:40:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:51.954+0000 7fce1fd26640 1 -- 172.21.15.27:0/3937807007 wait complete. 2024-04-25T13:40:51.965 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T13:40:51.965 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T13:40:51.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T13:40:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 -- 172.21.15.27:0/3554274010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 msgr2=0x7f32e010bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/3554274010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 0x7f32e010bc80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009ca0 tx=0x7f32d802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 -- 172.21.15.27:0/3554274010 shutdown_connections 2024-04-25T13:40:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/3554274010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 0x7f32e010bc80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/3554274010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32e0106c00 0x7f32e01090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/3554274010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32e0101e20 0x7f32e0102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 -- 172.21.15.27:0/3554274010 >> 172.21.15.27:0/3554274010 conn(0x7f32e00fd330 msgr2=0x7f32e00ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 -- 172.21.15.27:0/3554274010 shutdown_connections 2024-04-25T13:40:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.077+0000 7f32e4fe3640 1 -- 172.21.15.27:0/3554274010 wait complete. 2024-04-25T13:40:52.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32e4fe3640 1 Processor -- start 2024-04-25T13:40:52.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32e4fe3640 1 -- start start 2024-04-25T13:40:52.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32e4fe3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32e0101e20 0x7f32e01042f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32e4fe3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32e0106c00 0x7f32e0102990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32e4fe3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 0x7f32e0102ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32e4fe3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32e010f600 con 0x7f32e0109770 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32e4fe3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f32e010f480 con 0x7f32e0101e20 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32e4fe3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f32e010f780 con 0x7f32e0106c00 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32ded9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 0x7f32e0102ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32ded9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 0x7f32e0102ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54362/0 (socket says 172.21.15.27:54362) 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32ded9d640 1 -- 172.21.15.27:0/919593060 learned_addr learned my addr 172.21.15.27:0/919593060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32de59c640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32e0101e20 0x7f32e01042f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.078+0000 7f32ddd9b640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32e0106c00 0x7f32e0102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32ded9d640 1 -- 172.21.15.27:0/919593060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32e0106c00 msgr2=0x7f32e0102990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32ded9d640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32e0106c00 0x7f32e0102990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32ded9d640 1 -- 172.21.15.27:0/919593060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32e0101e20 msgr2=0x7f32e01042f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32ded9d640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32e0101e20 0x7f32e01042f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32ded9d640 1 -- 172.21.15.27:0/919593060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e0103570 con 0x7f32e0109770 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32ded9d640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 0x7f32e0102ed0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f32d80097b0 tx=0x7f32d8031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32cf7fe640 1 -- 172.21.15.27:0/919593060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8003bf0 con 0x7f32e0109770 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32cf7fe640 1 -- 172.21.15.27:0/919593060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32d8003d90 con 0x7f32e0109770 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e01a4260 con 0x7f32e0109770 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32cf7fe640 1 -- 172.21.15.27:0/919593060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d802fb70 con 0x7f32e0109770 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.079+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e01a4610 con 0x7f32e0109770 2024-04-25T13:40:52.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.080+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e006a190 con 0x7f32e0109770 2024-04-25T13:40:52.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.081+0000 7f32cf7fe640 1 -- 172.21.15.27:0/919593060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f32d802fd50 con 0x7f32e0109770 2024-04-25T13:40:52.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.082+0000 7f32cf7fe640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f32c80afaf0 0x7f32c80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.082+0000 7f32de59c640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f32c80afaf0 0x7f32c80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.082+0000 7f32cf7fe640 1 -- 172.21.15.27:0/919593060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f32d8100390 con 0x7f32e0109770 2024-04-25T13:40:52.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.083+0000 7f32de59c640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f32c80afaf0 0x7f32c80b2000 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f32e0104af0 tx=0x7f32d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:52.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.086+0000 7f32cf7fe640 1 -- 172.21.15.27:0/919593060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32d80c8fa0 con 0x7f32e0109770 2024-04-25T13:40:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.275+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f32e0062fe0 con 0x7f32c80afaf0 2024-04-25T13:40:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.283+0000 7f32cf7fe640 1 -- 172.21.15.27:0/919593060 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258957 (secure 0 0 0) 0x7f32e0062fe0 con 0x7f32c80afaf0 2024-04-25T13:40:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:52.287 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T13:40:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f32c80afaf0 msgr2=0x7f32c80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f32c80afaf0 0x7f32c80b2000 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f32e0104af0 tx=0x7f32d0005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 msgr2=0x7f32e0102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 0x7f32e0102ed0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f32d80097b0 tx=0x7f32d8031840 comp rx=0 tx=0).stop 2024-04-25T13:40:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 shutdown_connections 2024-04-25T13:40:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f32c80afaf0 0x7f32c80b2000 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e0109770 0x7f32e0102ed0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32e0106c00 0x7f32e0102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.285+0000 7f32e4fe3640 1 --2- 172.21.15.27:0/919593060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32e0101e20 0x7f32e01042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.286+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 >> 172.21.15.27:0/919593060 conn(0x7f32e00fd330 msgr2=0x7f32e0107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.286+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 shutdown_connections 2024-04-25T13:40:52.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.286+0000 7f32e4fe3640 1 -- 172.21.15.27:0/919593060 wait complete. 2024-04-25T13:40:52.296 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:52 smithi027 ceph-mon[29140]: pgmap v348: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-25T13:40:52.296 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2452749122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:52.296 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/59078526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:52.296 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3937807007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:52.297 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-04-25T13:40:50.381423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652695,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":326,"num_read_kb":251,"num_write":207,"num_write_kb":2239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951957,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":663,"ondisk_log_size":663,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375956,"kb_used_data":10164,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124565868,"statfs":{"total":1151940427776,"available":1151555448832,"internally_reserved":0,"allocated":10407936,"data_stored":6804213,"data_compressed":114309,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19138,"internal_metadata":374453566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003217"},"pg_stats":[{"pgid":"3.3f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368465+0000","last_change":"2024-04-25T13:38:45.356552+0000","last_active":"2024-04-25T13:40:00.368465+0000","last_peered":"2024-04-25T13:40:00.368465+0000","last_clean":"2024-04-25T13:40:00.368465+0000","last_became_active":"2024-04-25T13:38:45.356122+0000","last_became_peered":"2024-04-25T13:38:45.356122+0000","last_unstale":"2024-04-25T13:40:00.368465+0000","last_undegraded":"2024-04-25T13:40:00.368465+0000","last_fullsized":"2024-04-25T13:40:00.368465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:55.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354236+0000","last_change":"2024-04-25T13:38:45.358577+0000","last_active":"2024-04-25T13:40:00.354236+0000","last_peered":"2024-04-25T13:40:00.354236+0000","last_clean":"2024-04-25T13:40:00.354236+0000","last_became_active":"2024-04-25T13:38:45.358122+0000","last_became_peered":"2024-04-25T13:38:45.358122+0000","last_unstale":"2024-04-25T13:40:00.354236+0000","last_undegraded":"2024-04-25T13:40:00.354236+0000","last_fullsized":"2024-04-25T13:40:00.354236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:52.508459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368109+0000","last_change":"2024-04-25T13:38:45.362539+0000","last_active":"2024-04-25T13:40:00.368109+0000","last_peered":"2024-04-25T13:40:00.368109+0000","last_clean":"2024-04-25T13:40:00.368109+0000","last_became_active":"2024-04-25T13:38:45.361983+0000","last_became_peered":"2024-04-25T13:38:45.361983+0000","last_unstale":"2024-04-25T13:40:00.368109+0000","last_undegraded":"2024-04-25T13:40:00.368109+0000","last_fullsized":"2024-04-25T13:40:00.368109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:13.267692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"78'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.337438+0000","last_change":"2024-04-25T13:38:45.360535+0000","last_active":"2024-04-25T13:40:00.337438+0000","last_peered":"2024-04-25T13:40:00.337438+0000","last_clean":"2024-04-25T13:40:00.337438+0000","last_became_active":"2024-04-25T13:38:45.357722+0000","last_became_peered":"2024-04-25T13:38:45.357722+0000","last_unstale":"2024-04-25T13:40:00.337438+0000","last_undegraded":"2024-04-25T13:40:00.337438+0000","last_fullsized":"2024-04-25T13:40:00.337438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:35.347434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365993+0000","last_change":"2024-04-25T13:38:45.353653+0000","last_active":"2024-04-25T13:40:00.365993+0000","last_peered":"2024-04-25T13:40:00.365993+0000","last_clean":"2024-04-25T13:40:00.365993+0000","last_became_active":"2024-04-25T13:38:45.353254+0000","last_became_peered":"2024-04-25T13:38:45.353254+0000","last_unstale":"2024-04-25T13:40:00.365993+0000","last_undegraded":"2024-04-25T13:40:00.365993+0000","last_fullsized":"2024-04-25T13:40:00.365993+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:57.275732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356027+0000","last_change":"2024-04-25T13:38:45.358241+0000","last_active":"2024-04-25T13:40:00.356027+0000","last_peered":"2024-04-25T13:40:00.356027+0000","last_clean":"2024-04-25T13:40:00.356027+0000","last_became_active":"2024-04-25T13:38:45.357843+0000","last_became_peered":"2024-04-25T13:38:45.357843+0000","last_unstale":"2024-04-25T13:40:00.356027+0000","last_undegraded":"2024-04-25T13:40:00.356027+0000","last_fullsized":"2024-04-25T13:40:00.356027+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:24.560357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375607+0000","last_change":"2024-04-25T13:38:45.350710+0000","last_active":"2024-04-25T13:40:00.375607+0000","last_peered":"2024-04-25T13:40:00.375607+0000","last_clean":"2024-04-25T13:40:00.375607+0000","last_became_active":"2024-04-25T13:38:45.350171+0000","last_became_peered":"2024-04-25T13:38:45.350171+0000","last_unstale":"2024-04-25T13:40:00.375607+0000","last_undegraded":"2024-04-25T13:40:00.375607+0000","last_fullsized":"2024-04-25T13:40:00.375607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:24.933631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363691+0000","last_change":"2024-04-25T13:38:45.356221+0000","last_active":"2024-04-25T13:40:00.363691+0000","last_peered":"2024-04-25T13:40:00.363691+0000","last_clean":"2024-04-25T13:40:00.363691+0000","last_became_active":"2024-04-25T13:38:45.355823+0000","last_became_peered":"2024-04-25T13:38:45.355823+0000","last_unstale":"2024-04-25T13:40:00.363691+0000","last_undegraded":"2024-04-25T13:40:00.363691+0000","last_fullsized":"2024-04-25T13:40:00.363691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:25:02.040407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366879+0000","last_change":"2024-04-25T13:38:45.360940+0000","last_active":"2024-04-25T13:40:00.366879+0000","last_peered":"2024-04-25T13:40:00.366879+0000","last_clean":"2024-04-25T13:40:00.366879+0000","last_became_active":"2024-04-25T13:38:45.360241+0000","last_became_peered":"2024-04-25T13:38:45.360241+0000","last_unstale":"2024-04-25T13:40:00.366879+0000","last_undegraded":"2024-04-25T13:40:00.366879+0000","last_fullsized":"2024-04-25T13:40:00.366879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:05.251236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"78'16","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362862+0000","last_change":"2024-04-25T13:38:45.357908+0000","last_active":"2024-04-25T13:40:00.362862+0000","last_peered":"2024-04-25T13:40:00.362862+0000","last_clean":"2024-04-25T13:40:00.362862+0000","last_became_active":"2024-04-25T13:38:45.357491+0000","last_became_peered":"2024-04-25T13:38:45.357491+0000","last_unstale":"2024-04-25T13:40:00.362862+0000","last_undegraded":"2024-04-25T13:40:00.362862+0000","last_fullsized":"2024-04-25T13:40:00.362862+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:30.931611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374765+0000","last_change":"2024-04-25T13:38:45.368137+0000","last_active":"2024-04-25T13:40:00.374765+0000","last_peered":"2024-04-25T13:40:00.374765+0000","last_clean":"2024-04-25T13:40:00.374765+0000","last_became_active":"2024-04-25T13:38:45.367355+0000","last_became_peered":"2024-04-25T13:38:45.367355+0000","last_unstale":"2024-04-25T13:40:00.374765+0000","last_undegraded":"2024-04-25T13:40:00.374765+0000","last_fullsized":"2024-04-25T13:40:00.374765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:09.502056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"78'7","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349152+0000","last_change":"2024-04-25T13:39:32.041604+0000","last_active":"2024-04-25T13:40:00.349152+0000","last_peered":"2024-04-25T13:40:00.349152+0000","last_clean":"2024-04-25T13:40:00.349152+0000","last_became_active":"2024-04-25T13:39:32.041313+0000","last_became_peered":"2024-04-25T13:39:32.041313+0000","last_unstale":"2024-04-25T13:40:00.349152+0000","last_undegraded":"2024-04-25T13:40:00.349152+0000","last_fullsized":"2024-04-25T13:40:00.349152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:32.390348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357001+0000","last_change":"2024-04-25T13:38:45.353972+0000","last_active":"2024-04-25T13:40:00.357001+0000","last_peered":"2024-04-25T13:40:00.357001+0000","last_clean":"2024-04-25T13:40:00.357001+0000","last_became_active":"2024-04-25T13:38:45.353458+0000","last_became_peered":"2024-04-25T13:38:45.353458+0000","last_unstale":"2024-04-25T13:40:00.357001+0000","last_undegraded":"2024-04-25T13:40:00.357001+0000","last_fullsized":"2024-04-25T13:40:00.357001+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:17.087850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.347216+0000","last_change":"2024-04-25T13:38:45.358876+0000","last_active":"2024-04-25T13:40:00.347216+0000","last_peered":"2024-04-25T13:40:00.347216+0000","last_clean":"2024-04-25T13:40:00.347216+0000","last_became_active":"2024-04-25T13:38:45.358435+0000","last_became_peered":"2024-04-25T13:38:45.358435+0000","last_unstale":"2024-04-25T13:40:00.347216+0000","last_undegraded":"2024-04-25T13:40:00.347216+0000","last_fullsized":"2024-04-25T13:40:00.347216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:18:47.287112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362043+0000","last_change":"2024-04-25T13:38:45.373557+0000","last_active":"2024-04-25T13:40:00.362043+0000","last_peered":"2024-04-25T13:40:00.362043+0000","last_clean":"2024-04-25T13:40:00.362043+0000","last_became_active":"2024-04-25T13:38:45.373179+0000","last_became_peered":"2024-04-25T13:38:45.373179+0000","last_unstale":"2024-04-25T13:40:00.362043+0000","last_undegraded":"2024-04-25T13:40:00.362043+0000","last_fullsized":"2024-04-25T13:40:00.362043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:01.790266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374896+0000","last_change":"2024-04-25T13:38:45.358282+0000","last_active":"2024-04-25T13:40:00.374896+0000","last_peered":"2024-04-25T13:40:00.374896+0000","last_clean":"2024-04-25T13:40:00.374896+0000","last_became_active":"2024-04-25T13:38:45.357923+0000","last_became_peered":"2024-04-25T13:38:45.357923+0000","last_unstale":"2024-04-25T13:40:00.374896+0000","last_undegraded":"2024-04-25T13:40:00.374896+0000","last_fullsized":"2024-04-25T13:40:00.374896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:47.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368746+0000","last_change":"2024-04-25T13:38:45.357865+0000","last_active":"2024-04-25T13:40:00.368746+0000","last_peered":"2024-04-25T13:40:00.368746+0000","last_clean":"2024-04-25T13:40:00.368746+0000","last_became_active":"2024-04-25T13:38:45.355399+0000","last_became_peered":"2024-04-25T13:38:45.355399+0000","last_unstale":"2024-04-25T13:40:00.368746+0000","last_undegraded":"2024-04-25T13:40:00.368746+0000","last_fullsized":"2024-04-25T13:40:00.368746+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:33:41.382462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349318+0000","last_change":"2024-04-25T13:38:45.350724+0000","last_active":"2024-04-25T13:40:00.349318+0000","last_peered":"2024-04-25T13:40:00.349318+0000","last_clean":"2024-04-25T13:40:00.349318+0000","last_became_active":"2024-04-25T13:38:45.350284+0000","last_became_peered":"2024-04-25T13:38:45.350284+0000","last_unstale":"2024-04-25T13:40:00.349318+0000","last_undegraded":"2024-04-25T13:40:00.349318+0000","last_fullsized":"2024-04-25T13:40:00.349318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:30.960843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351696+0000","last_change":"2024-04-25T13:38:45.362966+0000","last_active":"2024-04-25T13:40:00.351696+0000","last_peered":"2024-04-25T13:40:00.351696+0000","last_clean":"2024-04-25T13:40:00.351696+0000","last_became_active":"2024-04-25T13:38:45.362585+0000","last_became_peered":"2024-04-25T13:38:45.362585+0000","last_unstale":"2024-04-25T13:40:00.351696+0000","last_undegraded":"2024-04-25T13:40:00.351696+0000","last_fullsized":"2024-04-25T13:40:00.351696+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:46:10.825028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'4","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.488075+0000","last_change":"2024-04-25T13:38:44.354887+0000","last_active":"2024-04-25T13:39:31.488075+0000","last_peered":"2024-04-25T13:39:31.488075+0000","last_clean":"2024-04-25T13:39:31.488075+0000","last_became_active":"2024-04-25T13:38:44.354510+0000","last_became_peered":"2024-04-25T13:38:44.354510+0000","last_unstale":"2024-04-25T13:39:31.488075+0000","last_undegraded":"2024-04-25T13:39:31.488075+0000","last_fullsized":"2024-04-25T13:39:31.488075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:46.457087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"78'5","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363062+0000","last_change":"2024-04-25T13:38:45.364056+0000","last_active":"2024-04-25T13:40:00.363062+0000","last_peered":"2024-04-25T13:40:00.363062+0000","last_clean":"2024-04-25T13:40:00.363062+0000","last_became_active":"2024-04-25T13:38:45.363572+0000","last_became_peered":"2024-04-25T13:38:45.363572+0000","last_unstale":"2024-04-25T13:40:00.363062+0000","last_undegraded":"2024-04-25T13:40:00.363062+0000","last_fullsized":"2024-04-25T13:40:00.363062+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:49:12.136637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.490480+0000","last_change":"2024-04-25T13:38:44.362371+0000","last_active":"2024-04-25T13:39:31.490480+0000","last_peered":"2024-04-25T13:39:31.490480+0000","last_clean":"2024-04-25T13:39:31.490480+0000","last_became_active":"2024-04-25T13:38:44.361262+0000","last_became_peered":"2024-04-25T13:38:44.361262+0000","last_unstale":"2024-04-25T13:39:31.490480+0000","last_undegraded":"2024-04-25T13:39:31.490480+0000","last_fullsized":"2024-04-25T13:39:31.490480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:01.442189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367693+0000","last_change":"2024-04-25T13:38:45.362599+0000","last_active":"2024-04-25T13:40:00.367693+0000","last_peered":"2024-04-25T13:40:00.367693+0000","last_clean":"2024-04-25T13:40:00.367693+0000","last_became_active":"2024-04-25T13:38:45.361893+0000","last_became_peered":"2024-04-25T13:38:45.361893+0000","last_unstale":"2024-04-25T13:40:00.367693+0000","last_undegraded":"2024-04-25T13:40:00.367693+0000","last_fullsized":"2024-04-25T13:40:00.367693+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:52.414826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491185+0000","last_change":"2024-04-25T13:38:44.353063+0000","last_active":"2024-04-25T13:39:31.491185+0000","last_peered":"2024-04-25T13:39:31.491185+0000","last_clean":"2024-04-25T13:39:31.491185+0000","last_became_active":"2024-04-25T13:38:44.352403+0000","last_became_peered":"2024-04-25T13:38:44.352403+0000","last_unstale":"2024-04-25T13:39:31.491185+0000","last_undegraded":"2024-04-25T13:39:31.491185+0000","last_fullsized":"2024-04-25T13:39:31.491185+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:09.670589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369475+0000","last_change":"2024-04-25T13:38:45.359078+0000","last_active":"2024-04-25T13:40:00.369475+0000","last_peered":"2024-04-25T13:40:00.369475+0000","last_clean":"2024-04-25T13:40:00.369475+0000","last_became_active":"2024-04-25T13:38:45.354063+0000","last_became_peered":"2024-04-25T13:38:45.354063+0000","last_unstale":"2024-04-25T13:40:00.369475+0000","last_undegraded":"2024-04-25T13:40:00.369475+0000","last_fullsized":"2024-04-25T13:40:00.369475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:16.694340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492390+0000","last_change":"2024-04-25T13:38:44.349132+0000","last_active":"2024-04-25T13:39:31.492390+0000","last_peered":"2024-04-25T13:39:31.492390+0000","last_clean":"2024-04-25T13:39:31.492390+0000","last_became_active":"2024-04-25T13:38:44.348762+0000","last_became_peered":"2024-04-25T13:38:44.348762+0000","last_unstale":"2024-04-25T13:39:31.492390+0000","last_undegraded":"2024-04-25T13:39:31.492390+0000","last_fullsized":"2024-04-25T13:39:31.492390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:12.990304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368999+0000","last_change":"2024-04-25T13:38:45.369116+0000","last_active":"2024-04-25T13:40:00.368999+0000","last_peered":"2024-04-25T13:40:00.368999+0000","last_clean":"2024-04-25T13:40:00.368999+0000","last_became_active":"2024-04-25T13:38:45.368380+0000","last_became_peered":"2024-04-25T13:38:45.368380+0000","last_unstale":"2024-04-25T13:40:00.368999+0000","last_undegraded":"2024-04-25T13:40:00.368999+0000","last_fullsized":"2024-04-25T13:40:00.368999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:02.259401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485437+0000","last_change":"2024-04-25T13:38:44.351873+0000","last_active":"2024-04-25T13:39:31.485437+0000","last_peered":"2024-04-25T13:39:31.485437+0000","last_clean":"2024-04-25T13:39:31.485437+0000","last_became_active":"2024-04-25T13:38:44.351518+0000","last_became_peered":"2024-04-25T13:38:44.351518+0000","last_unstale":"2024-04-25T13:39:31.485437+0000","last_undegraded":"2024-04-25T13:39:31.485437+0000","last_fullsized":"2024-04-25T13:39:31.485437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:06.839053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342392+0000","last_change":"2024-04-25T13:38:45.351369+0000","last_active":"2024-04-25T13:40:00.342392+0000","last_peered":"2024-04-25T13:40:00.342392+0000","last_clean":"2024-04-25T13:40:00.342392+0000","last_became_active":"2024-04-25T13:38:45.351018+0000","last_became_peered":"2024-04-25T13:38:45.351018+0000","last_unstale":"2024-04-25T13:40:00.342392+0000","last_undegraded":"2024-04-25T13:40:00.342392+0000","last_fullsized":"2024-04-25T13:40:00.342392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040292+0000","last_change":"2024-04-25T13:38:44.345986+0000","last_active":"2024-04-25T13:39:32.040292+0000","last_peered":"2024-04-25T13:39:32.040292+0000","last_clean":"2024-04-25T13:39:32.040292+0000","last_became_active":"2024-04-25T13:38:44.345600+0000","last_became_peered":"2024-04-25T13:38:44.345600+0000","last_unstale":"2024-04-25T13:39:32.040292+0000","last_undegraded":"2024-04-25T13:39:32.040292+0000","last_fullsized":"2024-04-25T13:39:32.040292+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:52.749192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365960+0000","last_change":"2024-04-25T13:38:45.368641+0000","last_active":"2024-04-25T13:40:00.365960+0000","last_peered":"2024-04-25T13:40:00.365960+0000","last_clean":"2024-04-25T13:40:00.365960+0000","last_became_active":"2024-04-25T13:38:45.368209+0000","last_became_peered":"2024-04-25T13:38:45.368209+0000","last_unstale":"2024-04-25T13:40:00.365960+0000","last_undegraded":"2024-04-25T13:40:00.365960+0000","last_fullsized":"2024-04-25T13:40:00.365960+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:47.990808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484783+0000","last_change":"2024-04-25T13:38:44.339927+0000","last_active":"2024-04-25T13:39:31.484783+0000","last_peered":"2024-04-25T13:39:31.484783+0000","last_clean":"2024-04-25T13:39:31.484783+0000","last_became_active":"2024-04-25T13:38:44.339588+0000","last_became_peered":"2024-04-25T13:38:44.339588+0000","last_unstale":"2024-04-25T13:39:31.484783+0000","last_undegraded":"2024-04-25T13:39:31.484783+0000","last_fullsized":"2024-04-25T13:39:31.484783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:51.707750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365330+0000","last_change":"2024-04-25T13:38:45.357323+0000","last_active":"2024-04-25T13:40:00.365330+0000","last_peered":"2024-04-25T13:40:00.365330+0000","last_clean":"2024-04-25T13:40:00.365330+0000","last_became_active":"2024-04-25T13:38:45.356870+0000","last_became_peered":"2024-04-25T13:38:45.356870+0000","last_unstale":"2024-04-25T13:40:00.365330+0000","last_undegraded":"2024-04-25T13:40:00.365330+0000","last_fullsized":"2024-04-25T13:40:00.365330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:36.838709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485179+0000","last_change":"2024-04-25T13:38:44.351294+0000","last_active":"2024-04-25T13:39:31.485179+0000","last_peered":"2024-04-25T13:39:31.485179+0000","last_clean":"2024-04-25T13:39:31.485179+0000","last_became_active":"2024-04-25T13:38:44.350971+0000","last_became_peered":"2024-04-25T13:38:44.350971+0000","last_unstale":"2024-04-25T13:39:31.485179+0000","last_undegraded":"2024-04-25T13:39:31.485179+0000","last_fullsized":"2024-04-25T13:39:31.485179+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:22.452217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349813+0000","last_change":"2024-04-25T13:38:45.345943+0000","last_active":"2024-04-25T13:40:00.349813+0000","last_peered":"2024-04-25T13:40:00.349813+0000","last_clean":"2024-04-25T13:40:00.349813+0000","last_became_active":"2024-04-25T13:38:45.345528+0000","last_became_peered":"2024-04-25T13:38:45.345528+0000","last_unstale":"2024-04-25T13:40:00.349813+0000","last_undegraded":"2024-04-25T13:40:00.349813+0000","last_fullsized":"2024-04-25T13:40:00.349813+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:57.327303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040662+0000","last_change":"2024-04-25T13:38:44.343032+0000","last_active":"2024-04-25T13:39:32.040662+0000","last_peered":"2024-04-25T13:39:32.040662+0000","last_clean":"2024-04-25T13:39:32.040662+0000","last_became_active":"2024-04-25T13:38:44.342613+0000","last_became_peered":"2024-04-25T13:38:44.342613+0000","last_unstale":"2024-04-25T13:39:32.040662+0000","last_undegraded":"2024-04-25T13:39:32.040662+0000","last_fullsized":"2024-04-25T13:39:32.040662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:33.957150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369787+0000","last_change":"2024-04-25T13:38:45.354534+0000","last_active":"2024-04-25T13:40:00.369787+0000","last_peered":"2024-04-25T13:40:00.369787+0000","last_clean":"2024-04-25T13:40:00.369787+0000","last_became_active":"2024-04-25T13:38:45.354147+0000","last_became_peered":"2024-04-25T13:38:45.354147+0000","last_unstale":"2024-04-25T13:40:00.369787+0000","last_undegraded":"2024-04-25T13:40:00.369787+0000","last_fullsized":"2024-04-25T13:40:00.369787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:31.537339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487809+0000","last_change":"2024-04-25T13:38:44.356103+0000","last_active":"2024-04-25T13:39:31.487809+0000","last_peered":"2024-04-25T13:39:31.487809+0000","last_clean":"2024-04-25T13:39:31.487809+0000","last_became_active":"2024-04-25T13:38:44.355702+0000","last_became_peered":"2024-04-25T13:38:44.355702+0000","last_unstale":"2024-04-25T13:39:31.487809+0000","last_undegraded":"2024-04-25T13:39:31.487809+0000","last_fullsized":"2024-04-25T13:39:31.487809+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:47.572922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"78'9","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.344967+0000","last_change":"2024-04-25T13:39:31.494580+0000","last_active":"2024-04-25T13:40:00.344967+0000","last_peered":"2024-04-25T13:40:00.344967+0000","last_clean":"2024-04-25T13:40:00.344967+0000","last_became_active":"2024-04-25T13:39:31.494102+0000","last_became_peered":"2024-04-25T13:39:31.494102+0000","last_unstale":"2024-04-25T13:40:00.344967+0000","last_undegraded":"2024-04-25T13:40:00.344967+0000","last_fullsized":"2024-04-25T13:40:00.344967+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.056136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485841+0000","last_change":"2024-04-25T13:38:44.355474+0000","last_active":"2024-04-25T13:39:31.485841+0000","last_peered":"2024-04-25T13:39:31.485841+0000","last_clean":"2024-04-25T13:39:31.485841+0000","last_became_active":"2024-04-25T13:38:44.355113+0000","last_became_peered":"2024-04-25T13:38:44.355113+0000","last_unstale":"2024-04-25T13:39:31.485841+0000","last_undegraded":"2024-04-25T13:39:31.485841+0000","last_fullsized":"2024-04-25T13:39:31.485841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.242974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.370871+0000","last_change":"2024-04-25T13:38:45.369618+0000","last_active":"2024-04-25T13:40:00.370871+0000","last_peered":"2024-04-25T13:40:00.370871+0000","last_clean":"2024-04-25T13:40:00.370871+0000","last_became_active":"2024-04-25T13:38:45.369252+0000","last_became_peered":"2024-04-25T13:38:45.369252+0000","last_unstale":"2024-04-25T13:40:00.370871+0000","last_undegraded":"2024-04-25T13:40:00.370871+0000","last_fullsized":"2024-04-25T13:40:00.370871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:40.336573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"78'29","reported_seq":292,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:45.598036+0000","last_change":"2024-04-25T13:38:44.349850+0000","last_active":"2024-04-25T13:40:45.598036+0000","last_peered":"2024-04-25T13:40:45.598036+0000","last_clean":"2024-04-25T13:40:45.598036+0000","last_became_active":"2024-04-25T13:38:44.349404+0000","last_became_peered":"2024-04-25T13:38:44.349404+0000","last_unstale":"2024-04-25T13:40:45.598036+0000","last_undegraded":"2024-04-25T13:40:45.598036+0000","last_fullsized":"2024-04-25T13:40:45.598036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:40.025684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":244,"num_read_kb":181,"num_write":29,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.348329+0000","last_change":"2024-04-25T13:38:45.371554+0000","last_active":"2024-04-25T13:40:00.348329+0000","last_peered":"2024-04-25T13:40:00.348329+0000","last_clean":"2024-04-25T13:40:00.348329+0000","last_became_active":"2024-04-25T13:38:45.371042+0000","last_became_peered":"2024-04-25T13:38:45.371042+0000","last_unstale":"2024-04-25T13:40:00.348329+0000","last_undegraded":"2024-04-25T13:40:00.348329+0000","last_fullsized":"2024-04-25T13:40:00.348329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:59.240536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494618+0000","last_change":"2024-04-25T13:38:44.345979+0000","last_active":"2024-04-25T13:39:31.494618+0000","last_peered":"2024-04-25T13:39:31.494618+0000","last_clean":"2024-04-25T13:39:31.494618+0000","last_became_active":"2024-04-25T13:38:44.345596+0000","last_became_peered":"2024-04-25T13:38:44.345596+0000","last_unstale":"2024-04-25T13:39:31.494618+0000","last_undegraded":"2024-04-25T13:39:31.494618+0000","last_fullsized":"2024-04-25T13:39:31.494618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:22:51.539024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367445+0000","last_change":"2024-04-25T13:38:45.364794+0000","last_active":"2024-04-25T13:40:00.367445+0000","last_peered":"2024-04-25T13:40:00.367445+0000","last_clean":"2024-04-25T13:40:00.367445+0000","last_became_active":"2024-04-25T13:38:45.364381+0000","last_became_peered":"2024-04-25T13:38:45.364381+0000","last_unstale":"2024-04-25T13:40:00.367445+0000","last_undegraded":"2024-04-25T13:40:00.367445+0000","last_fullsized":"2024-04-25T13:40:00.367445+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:16.856980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485286+0000","last_change":"2024-04-25T13:38:44.351269+0000","last_active":"2024-04-25T13:39:31.485286+0000","last_peered":"2024-04-25T13:39:31.485286+0000","last_clean":"2024-04-25T13:39:31.485286+0000","last_became_active":"2024-04-25T13:38:44.350933+0000","last_became_peered":"2024-04-25T13:38:44.350933+0000","last_unstale":"2024-04-25T13:39:31.485286+0000","last_undegraded":"2024-04-25T13:39:31.485286+0000","last_fullsized":"2024-04-25T13:39:31.485286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:02.677849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.372710+0000","last_change":"2024-04-25T13:38:45.362732+0000","last_active":"2024-04-25T13:40:00.372710+0000","last_peered":"2024-04-25T13:40:00.372710+0000","last_clean":"2024-04-25T13:40:00.372710+0000","last_became_active":"2024-04-25T13:38:45.361957+0000","last_became_peered":"2024-04-25T13:38:45.361957+0000","last_unstale":"2024-04-25T13:40:00.372710+0000","last_undegraded":"2024-04-25T13:40:00.372710+0000","last_fullsized":"2024-04-25T13:40:00.372710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:50.011442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.776862+0000","last_change":"2024-04-25T13:38:44.349378+0000","last_active":"2024-04-25T13:39:31.776862+0000","last_peered":"2024-04-25T13:39:31.776862+0000","last_clean":"2024-04-25T13:39:31.776862+0000","last_became_active":"2024-04-25T13:38:44.348887+0000","last_became_peered":"2024-04-25T13:38:44.348887+0000","last_unstale":"2024-04-25T13:39:31.776862+0000","last_undegraded":"2024-04-25T13:39:31.776862+0000","last_fullsized":"2024-04-25T13:39:31.776862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.747205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375453+0000","last_change":"2024-04-25T13:38:45.349941+0000","last_active":"2024-04-25T13:40:00.375453+0000","last_peered":"2024-04-25T13:40:00.375453+0000","last_clean":"2024-04-25T13:40:00.375453+0000","last_became_active":"2024-04-25T13:38:45.349590+0000","last_became_peered":"2024-04-25T13:38:45.349590+0000","last_unstale":"2024-04-25T13:40:00.375453+0000","last_undegraded":"2024-04-25T13:40:00.375453+0000","last_fullsized":"2024-04-25T13:40:00.375453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:42.904406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.791483+0000","last_change":"2024-04-25T13:39:31.790997+0000","last_active":"2024-04-25T13:39:31.791483+0000","last_peered":"2024-04-25T13:39:31.791483+0000","last_clean":"2024-04-25T13:39:31.791483+0000","last_became_active":"2024-04-25T13:39:31.778451+0000","last_became_peered":"2024-04-25T13:39:31.778451+0000","last_unstale":"2024-04-25T13:39:31.791483+0000","last_undegraded":"2024-04-25T13:39:31.791483+0000","last_fullsized":"2024-04-25T13:39:31.791483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:17.735970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368028+0000","last_change":"2024-04-25T13:38:45.368589+0000","last_active":"2024-04-25T13:40:00.368028+0000","last_peered":"2024-04-25T13:40:00.368028+0000","last_clean":"2024-04-25T13:40:00.368028+0000","last_became_active":"2024-04-25T13:38:45.368079+0000","last_became_peered":"2024-04-25T13:38:45.368079+0000","last_unstale":"2024-04-25T13:40:00.368028+0000","last_undegraded":"2024-04-25T13:40:00.368028+0000","last_fullsized":"2024-04-25T13:40:00.368028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:10.180264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491662+0000","last_change":"2024-04-25T13:38:44.345148+0000","last_active":"2024-04-25T13:39:31.491662+0000","last_peered":"2024-04-25T13:39:31.491662+0000","last_clean":"2024-04-25T13:39:31.491662+0000","last_became_active":"2024-04-25T13:38:44.344816+0000","last_became_peered":"2024-04-25T13:38:44.344816+0000","last_unstale":"2024-04-25T13:39:31.491662+0000","last_undegraded":"2024-04-25T13:39:31.491662+0000","last_fullsized":"2024-04-25T13:39:31.491662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:03:38.134169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351253+0000","last_change":"2024-04-25T13:38:45.354340+0000","last_active":"2024-04-25T13:40:00.351253+0000","last_peered":"2024-04-25T13:40:00.351253+0000","last_clean":"2024-04-25T13:40:00.351253+0000","last_became_active":"2024-04-25T13:38:45.353933+0000","last_became_peered":"2024-04-25T13:38:45.353933+0000","last_unstale":"2024-04-25T13:40:00.351253+0000","last_undegraded":"2024-04-25T13:40:00.351253+0000","last_fullsized":"2024-04-25T13:40:00.351253+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:07.619051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.488024+0000","last_change":"2024-04-25T13:38:44.351183+0000","last_active":"2024-04-25T13:39:31.488024+0000","last_peered":"2024-04-25T13:39:31.488024+0000","last_clean":"2024-04-25T13:39:31.488024+0000","last_became_active":"2024-04-25T13:38:44.350785+0000","last_became_peered":"2024-04-25T13:38:44.350785+0000","last_unstale":"2024-04-25T13:39:31.488024+0000","last_undegraded":"2024-04-25T13:39:31.488024+0000","last_fullsized":"2024-04-25T13:39:31.488024+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:37:57.465012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360705+0000","last_change":"2024-04-25T13:38:45.362355+0000","last_active":"2024-04-25T13:40:00.360705+0000","last_peered":"2024-04-25T13:40:00.360705+0000","last_clean":"2024-04-25T13:40:00.360705+0000","last_became_active":"2024-04-25T13:38:45.361996+0000","last_became_peered":"2024-04-25T13:38:45.361996+0000","last_unstale":"2024-04-25T13:40:00.360705+0000","last_undegraded":"2024-04-25T13:40:00.360705+0000","last_fullsized":"2024-04-25T13:40:00.360705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:53:24.790450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487641+0000","last_change":"2024-04-25T13:38:44.356079+0000","last_active":"2024-04-25T13:39:31.487641+0000","last_peered":"2024-04-25T13:39:31.487641+0000","last_clean":"2024-04-25T13:39:31.487641+0000","last_became_active":"2024-04-25T13:38:44.355660+0000","last_became_peered":"2024-04-25T13:38:44.355660+0000","last_unstale":"2024-04-25T13:39:31.487641+0000","last_undegraded":"2024-04-25T13:39:31.487641+0000","last_fullsized":"2024-04-25T13:39:31.487641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:35.161337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.353301+0000","last_change":"2024-04-25T13:38:45.356578+0000","last_active":"2024-04-25T13:40:00.353301+0000","last_peered":"2024-04-25T13:40:00.353301+0000","last_clean":"2024-04-25T13:40:00.353301+0000","last_became_active":"2024-04-25T13:38:45.356232+0000","last_became_peered":"2024-04-25T13:38:45.356232+0000","last_unstale":"2024-04-25T13:40:00.353301+0000","last_undegraded":"2024-04-25T13:40:00.353301+0000","last_fullsized":"2024-04-25T13:40:00.353301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.476890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"78'8","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360561+0000","last_change":"2024-04-25T13:39:31.778577+0000","last_active":"2024-04-25T13:40:00.360561+0000","last_peered":"2024-04-25T13:40:00.360561+0000","last_clean":"2024-04-25T13:40:00.360561+0000","last_became_active":"2024-04-25T13:39:31.778178+0000","last_became_peered":"2024-04-25T13:39:31.778178+0000","last_unstale":"2024-04-25T13:40:00.360561+0000","last_undegraded":"2024-04-25T13:40:00.360561+0000","last_fullsized":"2024-04-25T13:40:00.360561+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.383670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485143+0000","last_change":"2024-04-25T13:38:44.351113+0000","last_active":"2024-04-25T13:39:31.485143+0000","last_peered":"2024-04-25T13:39:31.485143+0000","last_clean":"2024-04-25T13:39:31.485143+0000","last_became_active":"2024-04-25T13:38:44.350730+0000","last_became_peered":"2024-04-25T13:38:44.350730+0000","last_unstale":"2024-04-25T13:39:31.485143+0000","last_undegraded":"2024-04-25T13:39:31.485143+0000","last_fullsized":"2024-04-25T13:39:31.485143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:38:22.747930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362720+0000","last_change":"2024-04-25T13:38:45.356731+0000","last_active":"2024-04-25T13:40:00.362720+0000","last_peered":"2024-04-25T13:40:00.362720+0000","last_clean":"2024-04-25T13:40:00.362720+0000","last_became_active":"2024-04-25T13:38:45.355531+0000","last_became_peered":"2024-04-25T13:38:45.355531+0000","last_unstale":"2024-04-25T13:40:00.362720+0000","last_undegraded":"2024-04-25T13:40:00.362720+0000","last_fullsized":"2024-04-25T13:40:00.362720+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:15.482469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777207+0000","last_change":"2024-04-25T13:38:44.345941+0000","last_active":"2024-04-25T13:39:31.777207+0000","last_peered":"2024-04-25T13:39:31.777207+0000","last_clean":"2024-04-25T13:39:31.777207+0000","last_became_active":"2024-04-25T13:38:44.344634+0000","last_became_peered":"2024-04-25T13:38:44.344634+0000","last_unstale":"2024-04-25T13:39:31.777207+0000","last_undegraded":"2024-04-25T13:39:31.777207+0000","last_fullsized":"2024-04-25T13:39:31.777207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:42.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":94,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487340+0000","last_change":"2024-04-25T13:37:27.903345+0000","last_active":"2024-04-25T13:39:31.487340+0000","last_peered":"2024-04-25T13:39:31.487340+0000","last_clean":"2024-04-25T13:39:31.487340+0000","last_became_active":"2024-04-25T13:37:27.897986+0000","last_became_peered":"2024-04-25T13:37:27.897986+0000","last_unstale":"2024-04-25T13:39:31.487340+0000","last_undegraded":"2024-04-25T13:39:31.487340+0000","last_fullsized":"2024-04-25T13:39:31.487340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_clean_scrub_stamp":"2024-04-25T13:33:56.569417+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:56.390458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"78'5","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.364075+0000","last_change":"2024-04-25T13:38:45.369390+0000","last_active":"2024-04-25T13:40:00.364075+0000","last_peered":"2024-04-25T13:40:00.364075+0000","last_clean":"2024-04-25T13:40:00.364075+0000","last_became_active":"2024-04-25T13:38:45.368938+0000","last_became_peered":"2024-04-25T13:38:45.368938+0000","last_unstale":"2024-04-25T13:40:00.364075+0000","last_undegraded":"2024-04-25T13:40:00.364075+0000","last_fullsized":"2024-04-25T13:40:00.364075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:11.942204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487500+0000","last_change":"2024-04-25T13:38:44.355585+0000","last_active":"2024-04-25T13:39:31.487500+0000","last_peered":"2024-04-25T13:39:31.487500+0000","last_clean":"2024-04-25T13:39:31.487500+0000","last_became_active":"2024-04-25T13:38:44.355131+0000","last_became_peered":"2024-04-25T13:38:44.355131+0000","last_unstale":"2024-04-25T13:39:31.487500+0000","last_undegraded":"2024-04-25T13:39:31.487500+0000","last_fullsized":"2024-04-25T13:39:31.487500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:05.240247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492418+0000","last_change":"2024-04-25T13:39:31.491742+0000","last_active":"2024-04-25T13:39:31.492418+0000","last_peered":"2024-04-25T13:39:31.492418+0000","last_clean":"2024-04-25T13:39:31.492418+0000","last_became_active":"2024-04-25T13:39:31.491266+0000","last_became_peered":"2024-04-25T13:39:31.491266+0000","last_unstale":"2024-04-25T13:39:31.492418+0000","last_undegraded":"2024-04-25T13:39:31.492418+0000","last_fullsized":"2024-04-25T13:39:31.492418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:21.932333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"78'9","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356400+0000","last_change":"2024-04-25T13:38:45.366187+0000","last_active":"2024-04-25T13:40:00.356400+0000","last_peered":"2024-04-25T13:40:00.356400+0000","last_clean":"2024-04-25T13:40:00.356400+0000","last_became_active":"2024-04-25T13:38:45.365744+0000","last_became_peered":"2024-04-25T13:38:45.365744+0000","last_unstale":"2024-04-25T13:40:00.356400+0000","last_undegraded":"2024-04-25T13:40:00.356400+0000","last_fullsized":"2024-04-25T13:40:00.356400+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:07.970735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"78'8","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.378057+0000","last_change":"2024-04-25T13:38:44.359250+0000","last_active":"2024-04-25T13:40:00.378057+0000","last_peered":"2024-04-25T13:40:00.378057+0000","last_clean":"2024-04-25T13:40:00.378057+0000","last_became_active":"2024-04-25T13:38:44.358874+0000","last_became_peered":"2024-04-25T13:38:44.358874+0000","last_unstale":"2024-04-25T13:40:00.378057+0000","last_undegraded":"2024-04-25T13:40:00.378057+0000","last_fullsized":"2024-04-25T13:40:00.378057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:28.820465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366411+0000","last_change":"2024-04-25T13:38:45.370080+0000","last_active":"2024-04-25T13:40:00.366411+0000","last_peered":"2024-04-25T13:40:00.366411+0000","last_clean":"2024-04-25T13:40:00.366411+0000","last_became_active":"2024-04-25T13:38:45.369668+0000","last_became_peered":"2024-04-25T13:38:45.369668+0000","last_unstale":"2024-04-25T13:40:00.366411+0000","last_undegraded":"2024-04-25T13:40:00.366411+0000","last_fullsized":"2024-04-25T13:40:00.366411+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:03.197468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777370+0000","last_change":"2024-04-25T13:38:44.358907+0000","last_active":"2024-04-25T13:39:31.777370+0000","last_peered":"2024-04-25T13:39:31.777370+0000","last_clean":"2024-04-25T13:39:31.777370+0000","last_became_active":"2024-04-25T13:38:44.358455+0000","last_became_peered":"2024-04-25T13:38:44.358455+0000","last_unstale":"2024-04-25T13:39:31.777370+0000","last_undegraded":"2024-04-25T13:39:31.777370+0000","last_fullsized":"2024-04-25T13:39:31.777370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:50:52.916433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365642+0000","last_change":"2024-04-25T13:38:45.373050+0000","last_active":"2024-04-25T13:40:00.365642+0000","last_peered":"2024-04-25T13:40:00.365642+0000","last_clean":"2024-04-25T13:40:00.365642+0000","last_became_active":"2024-04-25T13:38:45.372621+0000","last_became_peered":"2024-04-25T13:38:45.372621+0000","last_unstale":"2024-04-25T13:40:00.365642+0000","last_undegraded":"2024-04-25T13:40:00.365642+0000","last_fullsized":"2024-04-25T13:40:00.365642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:29:43.259296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485534+0000","last_change":"2024-04-25T13:38:44.355768+0000","last_active":"2024-04-25T13:39:31.485534+0000","last_peered":"2024-04-25T13:39:31.485534+0000","last_clean":"2024-04-25T13:39:31.485534+0000","last_became_active":"2024-04-25T13:38:44.355400+0000","last_became_peered":"2024-04-25T13:38:44.355400+0000","last_unstale":"2024-04-25T13:39:31.485534+0000","last_undegraded":"2024-04-25T13:39:31.485534+0000","last_fullsized":"2024-04-25T13:39:31.485534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:38.533717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342472+0000","last_change":"2024-04-25T13:38:45.350582+0000","last_active":"2024-04-25T13:40:00.342472+0000","last_peered":"2024-04-25T13:40:00.342472+0000","last_clean":"2024-04-25T13:40:00.342472+0000","last_became_active":"2024-04-25T13:38:45.350220+0000","last_became_peered":"2024-04-25T13:38:45.350220+0000","last_unstale":"2024-04-25T13:40:00.342472+0000","last_undegraded":"2024-04-25T13:40:00.342472+0000","last_fullsized":"2024-04-25T13:40:00.342472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.463519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.489729+0000","last_change":"2024-04-25T13:38:44.360459+0000","last_active":"2024-04-25T13:39:31.489729+0000","last_peered":"2024-04-25T13:39:31.489729+0000","last_clean":"2024-04-25T13:39:31.489729+0000","last_became_active":"2024-04-25T13:38:44.359709+0000","last_became_peered":"2024-04-25T13:38:44.359709+0000","last_unstale":"2024-04-25T13:39:31.489729+0000","last_undegraded":"2024-04-25T13:39:31.489729+0000","last_fullsized":"2024-04-25T13:39:31.489729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:53.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356628+0000","last_change":"2024-04-25T13:38:45.357291+0000","last_active":"2024-04-25T13:40:00.356628+0000","last_peered":"2024-04-25T13:40:00.356628+0000","last_clean":"2024-04-25T13:40:00.356628+0000","last_became_active":"2024-04-25T13:38:45.356929+0000","last_became_peered":"2024-04-25T13:38:45.356929+0000","last_unstale":"2024-04-25T13:40:00.356628+0000","last_undegraded":"2024-04-25T13:40:00.356628+0000","last_fullsized":"2024-04-25T13:40:00.356628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:02:08.619208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.493732+0000","last_change":"2024-04-25T13:38:44.343885+0000","last_active":"2024-04-25T13:39:31.493732+0000","last_peered":"2024-04-25T13:39:31.493732+0000","last_clean":"2024-04-25T13:39:31.493732+0000","last_became_active":"2024-04-25T13:38:44.343541+0000","last_became_peered":"2024-04-25T13:38:44.343541+0000","last_unstale":"2024-04-25T13:39:31.493732+0000","last_undegraded":"2024-04-25T13:39:31.493732+0000","last_fullsized":"2024-04-25T13:39:31.493732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:50:01.904593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365216+0000","last_change":"2024-04-25T13:38:45.371508+0000","last_active":"2024-04-25T13:40:00.365216+0000","last_peered":"2024-04-25T13:40:00.365216+0000","last_clean":"2024-04-25T13:40:00.365216+0000","last_became_active":"2024-04-25T13:38:45.370912+0000","last_became_peered":"2024-04-25T13:38:45.370912+0000","last_unstale":"2024-04-25T13:40:00.365216+0000","last_undegraded":"2024-04-25T13:40:00.365216+0000","last_fullsized":"2024-04-25T13:40:00.365216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:07.075511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'2","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.498142+0000","last_change":"2024-04-25T13:39:31.498142+0000","last_active":"2024-04-25T13:39:31.498142+0000","last_peered":"2024-04-25T13:39:31.498142+0000","last_clean":"2024-04-25T13:39:31.498142+0000","last_became_active":"2024-04-25T13:39:31.494646+0000","last_became_peered":"2024-04-25T13:39:31.494646+0000","last_unstale":"2024-04-25T13:39:31.498142+0000","last_undegraded":"2024-04-25T13:39:31.498142+0000","last_fullsized":"2024-04-25T13:39:31.498142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:46.084354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375168+0000","last_change":"2024-04-25T13:38:45.363322+0000","last_active":"2024-04-25T13:40:00.375168+0000","last_peered":"2024-04-25T13:40:00.375168+0000","last_clean":"2024-04-25T13:40:00.375168+0000","last_became_active":"2024-04-25T13:38:45.362948+0000","last_became_peered":"2024-04-25T13:38:45.362948+0000","last_unstale":"2024-04-25T13:40:00.375168+0000","last_undegraded":"2024-04-25T13:40:00.375168+0000","last_fullsized":"2024-04-25T13:40:00.375168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:20.815774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494630+0000","last_change":"2024-04-25T13:38:44.358588+0000","last_active":"2024-04-25T13:39:31.494630+0000","last_peered":"2024-04-25T13:39:31.494630+0000","last_clean":"2024-04-25T13:39:31.494630+0000","last_became_active":"2024-04-25T13:38:44.358269+0000","last_became_peered":"2024-04-25T13:38:44.358269+0000","last_unstale":"2024-04-25T13:39:31.494630+0000","last_undegraded":"2024-04-25T13:39:31.494630+0000","last_fullsized":"2024-04-25T13:39:31.494630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:35.405572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375021+0000","last_change":"2024-04-25T13:38:45.369272+0000","last_active":"2024-04-25T13:40:00.375021+0000","last_peered":"2024-04-25T13:40:00.375021+0000","last_clean":"2024-04-25T13:40:00.375021+0000","last_became_active":"2024-04-25T13:38:45.368774+0000","last_became_peered":"2024-04-25T13:38:45.368774+0000","last_unstale":"2024-04-25T13:40:00.375021+0000","last_undegraded":"2024-04-25T13:40:00.375021+0000","last_fullsized":"2024-04-25T13:40:00.375021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:39.149557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485416+0000","last_change":"2024-04-25T13:38:44.349479+0000","last_active":"2024-04-25T13:39:31.485416+0000","last_peered":"2024-04-25T13:39:31.485416+0000","last_clean":"2024-04-25T13:39:31.485416+0000","last_became_active":"2024-04-25T13:38:44.349104+0000","last_became_peered":"2024-04-25T13:38:44.349104+0000","last_unstale":"2024-04-25T13:39:31.485416+0000","last_undegraded":"2024-04-25T13:39:31.485416+0000","last_fullsized":"2024-04-25T13:39:31.485416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:39.257472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"78'7","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375501+0000","last_change":"2024-04-25T13:39:31.495349+0000","last_active":"2024-04-25T13:40:00.375501+0000","last_peered":"2024-04-25T13:40:00.375501+0000","last_clean":"2024-04-25T13:40:00.375501+0000","last_became_active":"2024-04-25T13:39:31.490741+0000","last_became_peered":"2024-04-25T13:39:31.490741+0000","last_unstale":"2024-04-25T13:40:00.375501+0000","last_undegraded":"2024-04-25T13:40:00.375501+0000","last_fullsized":"2024-04-25T13:40:00.375501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.474725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484923+0000","last_change":"2024-04-25T13:38:44.349502+0000","last_active":"2024-04-25T13:39:31.484923+0000","last_peered":"2024-04-25T13:39:31.484923+0000","last_clean":"2024-04-25T13:39:31.484923+0000","last_became_active":"2024-04-25T13:38:44.349127+0000","last_became_peered":"2024-04-25T13:38:44.349127+0000","last_unstale":"2024-04-25T13:39:31.484923+0000","last_undegraded":"2024-04-25T13:39:31.484923+0000","last_fullsized":"2024-04-25T13:39:31.484923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:34:50.673352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.371319+0000","last_change":"2024-04-25T13:38:45.364746+0000","last_active":"2024-04-25T13:40:00.371319+0000","last_peered":"2024-04-25T13:40:00.371319+0000","last_clean":"2024-04-25T13:40:00.371319+0000","last_became_active":"2024-04-25T13:38:45.364364+0000","last_became_peered":"2024-04-25T13:38:45.364364+0000","last_unstale":"2024-04-25T13:40:00.371319+0000","last_undegraded":"2024-04-25T13:40:00.371319+0000","last_fullsized":"2024-04-25T13:40:00.371319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:42.145637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357194+0000","last_change":"2024-04-25T13:38:45.356683+0000","last_active":"2024-04-25T13:40:00.357194+0000","last_peered":"2024-04-25T13:40:00.357194+0000","last_clean":"2024-04-25T13:40:00.357194+0000","last_became_active":"2024-04-25T13:38:45.356048+0000","last_became_peered":"2024-04-25T13:38:45.356048+0000","last_unstale":"2024-04-25T13:40:00.357194+0000","last_undegraded":"2024-04-25T13:40:00.357194+0000","last_fullsized":"2024-04-25T13:40:00.357194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:49.578317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"78'12","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362649+0000","last_change":"2024-04-25T13:38:45.358298+0000","last_active":"2024-04-25T13:40:00.362649+0000","last_peered":"2024-04-25T13:40:00.362649+0000","last_clean":"2024-04-25T13:40:00.362649+0000","last_became_active":"2024-04-25T13:38:45.357986+0000","last_became_peered":"2024-04-25T13:38:45.357986+0000","last_unstale":"2024-04-25T13:40:00.362649+0000","last_undegraded":"2024-04-25T13:40:00.362649+0000","last_fullsized":"2024-04-25T13:40:00.362649+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:33.093156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.355496+0000","last_change":"2024-04-25T13:38:45.363043+0000","last_active":"2024-04-25T13:40:00.355496+0000","last_peered":"2024-04-25T13:40:00.355496+0000","last_clean":"2024-04-25T13:40:00.355496+0000","last_became_active":"2024-04-25T13:38:45.362686+0000","last_became_peered":"2024-04-25T13:38:45.362686+0000","last_unstale":"2024-04-25T13:40:00.355496+0000","last_undegraded":"2024-04-25T13:40:00.355496+0000","last_fullsized":"2024-04-25T13:40:00.355496+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:56.602134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362067+0000","last_change":"2024-04-25T13:38:45.373256+0000","last_active":"2024-04-25T13:40:00.362067+0000","last_peered":"2024-04-25T13:40:00.362067+0000","last_clean":"2024-04-25T13:40:00.362067+0000","last_became_active":"2024-04-25T13:38:45.372844+0000","last_became_peered":"2024-04-25T13:38:45.372844+0000","last_unstale":"2024-04-25T13:40:00.362067+0000","last_undegraded":"2024-04-25T13:40:00.362067+0000","last_fullsized":"2024-04-25T13:40:00.362067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:39:34.590536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.364083+0000","last_change":"2024-04-25T13:38:45.370007+0000","last_active":"2024-04-25T13:40:00.364083+0000","last_peered":"2024-04-25T13:40:00.364083+0000","last_clean":"2024-04-25T13:40:00.364083+0000","last_became_active":"2024-04-25T13:38:45.369606+0000","last_became_peered":"2024-04-25T13:38:45.369606+0000","last_unstale":"2024-04-25T13:40:00.364083+0000","last_undegraded":"2024-04-25T13:40:00.364083+0000","last_fullsized":"2024-04-25T13:40:00.364083+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:45.514756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.350193+0000","last_change":"2024-04-25T13:38:45.362075+0000","last_active":"2024-04-25T13:40:00.350193+0000","last_peered":"2024-04-25T13:40:00.350193+0000","last_clean":"2024-04-25T13:40:00.350193+0000","last_became_active":"2024-04-25T13:38:45.361720+0000","last_became_peered":"2024-04-25T13:38:45.361720+0000","last_unstale":"2024-04-25T13:40:00.350193+0000","last_undegraded":"2024-04-25T13:40:00.350193+0000","last_fullsized":"2024-04-25T13:40:00.350193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:36.169185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354853+0000","last_change":"2024-04-25T13:38:45.352101+0000","last_active":"2024-04-25T13:40:00.354853+0000","last_peered":"2024-04-25T13:40:00.354853+0000","last_clean":"2024-04-25T13:40:00.354853+0000","last_became_active":"2024-04-25T13:38:45.351739+0000","last_became_peered":"2024-04-25T13:38:45.351739+0000","last_unstale":"2024-04-25T13:40:00.354853+0000","last_undegraded":"2024-04-25T13:40:00.354853+0000","last_fullsized":"2024-04-25T13:40:00.354853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:05.612527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362759+0000","last_change":"2024-04-25T13:38:45.363855+0000","last_active":"2024-04-25T13:40:00.362759+0000","last_peered":"2024-04-25T13:40:00.362759+0000","last_clean":"2024-04-25T13:40:00.362759+0000","last_became_active":"2024-04-25T13:38:45.363377+0000","last_became_peered":"2024-04-25T13:38:45.363377+0000","last_unstale":"2024-04-25T13:40:00.362759+0000","last_undegraded":"2024-04-25T13:40:00.362759+0000","last_fullsized":"2024-04-25T13:40:00.362759+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:26.828215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365565+0000","last_change":"2024-04-25T13:38:45.359844+0000","last_active":"2024-04-25T13:40:00.365565+0000","last_peered":"2024-04-25T13:40:00.365565+0000","last_clean":"2024-04-25T13:40:00.365565+0000","last_became_active":"2024-04-25T13:38:45.359428+0000","last_became_peered":"2024-04-25T13:38:45.359428+0000","last_unstale":"2024-04-25T13:40:00.365565+0000","last_undegraded":"2024-04-25T13:40:00.365565+0000","last_fullsized":"2024-04-25T13:40:00.365565+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:01.370409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.359173+0000","last_change":"2024-04-25T13:38:45.370415+0000","last_active":"2024-04-25T13:40:00.359173+0000","last_peered":"2024-04-25T13:40:00.359173+0000","last_clean":"2024-04-25T13:40:00.359173+0000","last_became_active":"2024-04-25T13:38:45.369525+0000","last_became_peered":"2024-04-25T13:38:45.369525+0000","last_unstale":"2024-04-25T13:40:00.359173+0000","last_undegraded":"2024-04-25T13:40:00.359173+0000","last_fullsized":"2024-04-25T13:40:00.359173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:39.532380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.376082+0000","last_change":"2024-04-25T13:38:45.365309+0000","last_active":"2024-04-25T13:40:00.376082+0000","last_peered":"2024-04-25T13:40:00.376082+0000","last_clean":"2024-04-25T13:40:00.376082+0000","last_became_active":"2024-04-25T13:38:45.364893+0000","last_became_peered":"2024-04-25T13:38:45.364893+0000","last_unstale":"2024-04-25T13:40:00.376082+0000","last_undegraded":"2024-04-25T13:40:00.376082+0000","last_fullsized":"2024-04-25T13:40:00.376082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:04:54.454443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368064+0000","last_change":"2024-04-25T13:38:45.365150+0000","last_active":"2024-04-25T13:40:00.368064+0000","last_peered":"2024-04-25T13:40:00.368064+0000","last_clean":"2024-04-25T13:40:00.368064+0000","last_became_active":"2024-04-25T13:38:45.364649+0000","last_became_peered":"2024-04-25T13:38:45.364649+0000","last_unstale":"2024-04-25T13:40:00.368064+0000","last_undegraded":"2024-04-25T13:40:00.368064+0000","last_fullsized":"2024-04-25T13:40:00.368064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:25.035789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354459+0000","last_change":"2024-04-25T13:38:45.352108+0000","last_active":"2024-04-25T13:40:00.354459+0000","last_peered":"2024-04-25T13:40:00.354459+0000","last_clean":"2024-04-25T13:40:00.354459+0000","last_became_active":"2024-04-25T13:38:45.349887+0000","last_became_peered":"2024-04-25T13:38:45.349887+0000","last_unstale":"2024-04-25T13:40:00.354459+0000","last_undegraded":"2024-04-25T13:40:00.354459+0000","last_fullsized":"2024-04-25T13:40:00.354459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:09.339685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285228,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62093,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":250,"num_read_kb":187,"num_write":82,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":186279,"data_compressed":5526,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678055,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":403844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874283,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":1055818,"data_compressed":38103,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037810,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234039,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430268,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":888832,"data_stored":994911,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593794,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":465139,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":403934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":465173,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":843776,"data_stored":404065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790022,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":994511,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61295,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61329,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61489,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T13:40:52.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:52 smithi122 ceph-mon[32843]: pgmap v348: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-25T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2452749122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/59078526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3937807007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:52 smithi045 ceph-mon[32731]: pgmap v348: 97 pgs: 97 active+clean; 637 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-25T13:40:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2452749122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/59078526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:52.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3937807007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 -- 172.21.15.27:0/558485013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0108940 msgr2=0x7f4ad010ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/558485013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0108940 0x7f4ad010ad60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009ca0 tx=0x7f4ac802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 -- 172.21.15.27:0/558485013 shutdown_connections 2024-04-25T13:40:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/558485013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0108940 0x7f4ad010ad60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/558485013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ad0103c40 0x7f4ad0104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/558485013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad0102a00 0x7f4ad0103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 -- 172.21.15.27:0/558485013 >> 172.21.15.27:0/558485013 conn(0x7f4ad00fdbf0 msgr2=0x7f4ad01000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 -- 172.21.15.27:0/558485013 shutdown_connections 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 -- 172.21.15.27:0/558485013 wait complete. 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 Processor -- start 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.451+0000 7f4ad6dee640 1 -- start start 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad6dee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ad0102a00 0x7f4ad019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad6dee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0103c40 0x7f4ad019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad6dee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad0108940 0x7f4ad01a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad6dee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad010f630 con 0x7f4ad0103c40 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad6dee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad010f4b0 con 0x7f4ad0102a00 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad6dee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad010f7b0 con 0x7f4ad0108940 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0103c40 0x7f4ad019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0103c40 0x7f4ad019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54370/0 (socket says 172.21.15.27:54370) 2024-04-25T13:40:52.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 -- 172.21.15.27:0/3554794953 learned_addr learned my addr 172.21.15.27:0/3554794953 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad538b640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad0108940 0x7f4ad01a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad4b8a640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ad0102a00 0x7f4ad019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 -- 172.21.15.27:0/3554794953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad0108940 msgr2=0x7f4ad01a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad0108940 0x7f4ad01a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 -- 172.21.15.27:0/3554794953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ad0102a00 msgr2=0x7f4ad019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ad0102a00 0x7f4ad019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 -- 172.21.15.27:0/3554794953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad01a1cc0 con 0x7f4ad0103c40 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4ad538b640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad0108940 0x7f4ad01a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.452+0000 7f4acffff640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0103c40 0x7f4ad019a6c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00c890 tx=0x7f4abc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.453+0000 7f4acdffb640 1 -- 172.21.15.27:0/3554794953 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc004220 con 0x7f4ad0103c40 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.453+0000 7f4acdffb640 1 -- 172.21.15.27:0/3554794953 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4abc0043c0 con 0x7f4ad0103c40 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.453+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad01a1fb0 con 0x7f4ad0103c40 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.453+0000 7f4acdffb640 1 -- 172.21.15.27:0/3554794953 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc004a00 con 0x7f4ad0103c40 2024-04-25T13:40:52.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.453+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad01a2420 con 0x7f4ad0103c40 2024-04-25T13:40:52.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.454+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ad0107750 con 0x7f4ad0103c40 2024-04-25T13:40:52.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.455+0000 7f4acdffb640 1 -- 172.21.15.27:0/3554794953 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4abc0079b0 con 0x7f4ad0103c40 2024-04-25T13:40:52.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.456+0000 7f4acdffb640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4ac00afaf0 0x7f4ac00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.456+0000 7f4acdffb640 1 -- 172.21.15.27:0/3554794953 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4abc0de960 con 0x7f4ad0103c40 2024-04-25T13:40:52.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.456+0000 7f4ad4b8a640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4ac00afaf0 0x7f4ac00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.457+0000 7f4ad4b8a640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4ac00afaf0 0x7f4ac00b2000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0103600 tx=0x7f4ac400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:52.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.461+0000 7f4acdffb640 1 -- 172.21.15.27:0/3554794953 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4abc014030 con 0x7f4ad0103c40 2024-04-25T13:40:52.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.643+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ad0062fe0 con 0x7f4ac00afaf0 2024-04-25T13:40:52.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.651+0000 7f4acdffb640 1 -- 172.21.15.27:0/3554794953 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258960 (secure 0 0 0) 0x7f4ad0062fe0 con 0x7f4ac00afaf0 2024-04-25T13:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:52.655 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T13:40:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4ac00afaf0 msgr2=0x7f4ac00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4ac00afaf0 0x7f4ac00b2000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0103600 tx=0x7f4ac400b390 comp rx=0 tx=0).stop 2024-04-25T13:40:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0103c40 msgr2=0x7f4ad019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0103c40 0x7f4ad019a6c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00c890 tx=0x7f4abc00cd50 comp rx=0 tx=0).stop 2024-04-25T13:40:52.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 shutdown_connections 2024-04-25T13:40:52.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4ac00afaf0 0x7f4ac00b2000 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ad0108940 0x7f4ad01a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad0103c40 0x7f4ad019a6c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 --2- 172.21.15.27:0/3554794953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ad0102a00 0x7f4ad019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 >> 172.21.15.27:0/3554794953 conn(0x7f4ad00fdbf0 msgr2=0x7f4ad0104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:52.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 shutdown_connections 2024-04-25T13:40:52.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.654+0000 7f4ad6dee640 1 -- 172.21.15.27:0/3554794953 wait complete. 2024-04-25T13:40:52.666 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-25T13:40:52.381916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658376,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":338,"num_read_kb":263,"num_write":210,"num_write_kb":2247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951957,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":666,"ondisk_log_size":666,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375980,"kb_used_data":10188,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124565844,"statfs":{"total":1151940427776,"available":1151555424256,"internally_reserved":0,"allocated":10432512,"data_stored":6821256,"data_compressed":114309,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19138,"internal_metadata":374453566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5681,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003253"},"pg_stats":[{"pgid":"3.3f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368465+0000","last_change":"2024-04-25T13:38:45.356552+0000","last_active":"2024-04-25T13:40:00.368465+0000","last_peered":"2024-04-25T13:40:00.368465+0000","last_clean":"2024-04-25T13:40:00.368465+0000","last_became_active":"2024-04-25T13:38:45.356122+0000","last_became_peered":"2024-04-25T13:38:45.356122+0000","last_unstale":"2024-04-25T13:40:00.368465+0000","last_undegraded":"2024-04-25T13:40:00.368465+0000","last_fullsized":"2024-04-25T13:40:00.368465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:55.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"78'4","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354435+0000","last_change":"2024-04-25T13:38:45.358577+0000","last_active":"2024-04-25T13:40:00.354435+0000","last_peered":"2024-04-25T13:40:00.354435+0000","last_clean":"2024-04-25T13:40:00.354435+0000","last_became_active":"2024-04-25T13:38:45.358122+0000","last_became_peered":"2024-04-25T13:38:45.358122+0000","last_unstale":"2024-04-25T13:40:00.354435+0000","last_undegraded":"2024-04-25T13:40:00.354435+0000","last_fullsized":"2024-04-25T13:40:00.354435+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:52.508459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"78'7","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368247+0000","last_change":"2024-04-25T13:38:45.362539+0000","last_active":"2024-04-25T13:40:00.368247+0000","last_peered":"2024-04-25T13:40:00.368247+0000","last_clean":"2024-04-25T13:40:00.368247+0000","last_became_active":"2024-04-25T13:38:45.361983+0000","last_became_peered":"2024-04-25T13:38:45.361983+0000","last_unstale":"2024-04-25T13:40:00.368247+0000","last_undegraded":"2024-04-25T13:40:00.368247+0000","last_fullsized":"2024-04-25T13:40:00.368247+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:13.267692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"78'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.337438+0000","last_change":"2024-04-25T13:38:45.360535+0000","last_active":"2024-04-25T13:40:00.337438+0000","last_peered":"2024-04-25T13:40:00.337438+0000","last_clean":"2024-04-25T13:40:00.337438+0000","last_became_active":"2024-04-25T13:38:45.357722+0000","last_became_peered":"2024-04-25T13:38:45.357722+0000","last_unstale":"2024-04-25T13:40:00.337438+0000","last_undegraded":"2024-04-25T13:40:00.337438+0000","last_fullsized":"2024-04-25T13:40:00.337438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:35.347434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365993+0000","last_change":"2024-04-25T13:38:45.353653+0000","last_active":"2024-04-25T13:40:00.365993+0000","last_peered":"2024-04-25T13:40:00.365993+0000","last_clean":"2024-04-25T13:40:00.365993+0000","last_became_active":"2024-04-25T13:38:45.353254+0000","last_became_peered":"2024-04-25T13:38:45.353254+0000","last_unstale":"2024-04-25T13:40:00.365993+0000","last_undegraded":"2024-04-25T13:40:00.365993+0000","last_fullsized":"2024-04-25T13:40:00.365993+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:57.275732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356027+0000","last_change":"2024-04-25T13:38:45.358241+0000","last_active":"2024-04-25T13:40:00.356027+0000","last_peered":"2024-04-25T13:40:00.356027+0000","last_clean":"2024-04-25T13:40:00.356027+0000","last_became_active":"2024-04-25T13:38:45.357843+0000","last_became_peered":"2024-04-25T13:38:45.357843+0000","last_unstale":"2024-04-25T13:40:00.356027+0000","last_undegraded":"2024-04-25T13:40:00.356027+0000","last_fullsized":"2024-04-25T13:40:00.356027+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:24.560357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375607+0000","last_change":"2024-04-25T13:38:45.350710+0000","last_active":"2024-04-25T13:40:00.375607+0000","last_peered":"2024-04-25T13:40:00.375607+0000","last_clean":"2024-04-25T13:40:00.375607+0000","last_became_active":"2024-04-25T13:38:45.350171+0000","last_became_peered":"2024-04-25T13:38:45.350171+0000","last_unstale":"2024-04-25T13:40:00.375607+0000","last_undegraded":"2024-04-25T13:40:00.375607+0000","last_fullsized":"2024-04-25T13:40:00.375607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:24.933631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363691+0000","last_change":"2024-04-25T13:38:45.356221+0000","last_active":"2024-04-25T13:40:00.363691+0000","last_peered":"2024-04-25T13:40:00.363691+0000","last_clean":"2024-04-25T13:40:00.363691+0000","last_became_active":"2024-04-25T13:38:45.355823+0000","last_became_peered":"2024-04-25T13:38:45.355823+0000","last_unstale":"2024-04-25T13:40:00.363691+0000","last_undegraded":"2024-04-25T13:40:00.363691+0000","last_fullsized":"2024-04-25T13:40:00.363691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:25:02.040407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366879+0000","last_change":"2024-04-25T13:38:45.360940+0000","last_active":"2024-04-25T13:40:00.366879+0000","last_peered":"2024-04-25T13:40:00.366879+0000","last_clean":"2024-04-25T13:40:00.366879+0000","last_became_active":"2024-04-25T13:38:45.360241+0000","last_became_peered":"2024-04-25T13:38:45.360241+0000","last_unstale":"2024-04-25T13:40:00.366879+0000","last_undegraded":"2024-04-25T13:40:00.366879+0000","last_fullsized":"2024-04-25T13:40:00.366879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:05.251236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"78'16","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362862+0000","last_change":"2024-04-25T13:38:45.357908+0000","last_active":"2024-04-25T13:40:00.362862+0000","last_peered":"2024-04-25T13:40:00.362862+0000","last_clean":"2024-04-25T13:40:00.362862+0000","last_became_active":"2024-04-25T13:38:45.357491+0000","last_became_peered":"2024-04-25T13:38:45.357491+0000","last_unstale":"2024-04-25T13:40:00.362862+0000","last_undegraded":"2024-04-25T13:40:00.362862+0000","last_fullsized":"2024-04-25T13:40:00.362862+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:30.931611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374765+0000","last_change":"2024-04-25T13:38:45.368137+0000","last_active":"2024-04-25T13:40:00.374765+0000","last_peered":"2024-04-25T13:40:00.374765+0000","last_clean":"2024-04-25T13:40:00.374765+0000","last_became_active":"2024-04-25T13:38:45.367355+0000","last_became_peered":"2024-04-25T13:38:45.367355+0000","last_unstale":"2024-04-25T13:40:00.374765+0000","last_undegraded":"2024-04-25T13:40:00.374765+0000","last_fullsized":"2024-04-25T13:40:00.374765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:09.502056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"78'7","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349152+0000","last_change":"2024-04-25T13:39:32.041604+0000","last_active":"2024-04-25T13:40:00.349152+0000","last_peered":"2024-04-25T13:40:00.349152+0000","last_clean":"2024-04-25T13:40:00.349152+0000","last_became_active":"2024-04-25T13:39:32.041313+0000","last_became_peered":"2024-04-25T13:39:32.041313+0000","last_unstale":"2024-04-25T13:40:00.349152+0000","last_undegraded":"2024-04-25T13:40:00.349152+0000","last_fullsized":"2024-04-25T13:40:00.349152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:32.390348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357001+0000","last_change":"2024-04-25T13:38:45.353972+0000","last_active":"2024-04-25T13:40:00.357001+0000","last_peered":"2024-04-25T13:40:00.357001+0000","last_clean":"2024-04-25T13:40:00.357001+0000","last_became_active":"2024-04-25T13:38:45.353458+0000","last_became_peered":"2024-04-25T13:38:45.353458+0000","last_unstale":"2024-04-25T13:40:00.357001+0000","last_undegraded":"2024-04-25T13:40:00.357001+0000","last_fullsized":"2024-04-25T13:40:00.357001+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:17.087850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.347216+0000","last_change":"2024-04-25T13:38:45.358876+0000","last_active":"2024-04-25T13:40:00.347216+0000","last_peered":"2024-04-25T13:40:00.347216+0000","last_clean":"2024-04-25T13:40:00.347216+0000","last_became_active":"2024-04-25T13:38:45.358435+0000","last_became_peered":"2024-04-25T13:38:45.358435+0000","last_unstale":"2024-04-25T13:40:00.347216+0000","last_undegraded":"2024-04-25T13:40:00.347216+0000","last_fullsized":"2024-04-25T13:40:00.347216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:18:47.287112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362043+0000","last_change":"2024-04-25T13:38:45.373557+0000","last_active":"2024-04-25T13:40:00.362043+0000","last_peered":"2024-04-25T13:40:00.362043+0000","last_clean":"2024-04-25T13:40:00.362043+0000","last_became_active":"2024-04-25T13:38:45.373179+0000","last_became_peered":"2024-04-25T13:38:45.373179+0000","last_unstale":"2024-04-25T13:40:00.362043+0000","last_undegraded":"2024-04-25T13:40:00.362043+0000","last_fullsized":"2024-04-25T13:40:00.362043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:01.790266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374896+0000","last_change":"2024-04-25T13:38:45.358282+0000","last_active":"2024-04-25T13:40:00.374896+0000","last_peered":"2024-04-25T13:40:00.374896+0000","last_clean":"2024-04-25T13:40:00.374896+0000","last_became_active":"2024-04-25T13:38:45.357923+0000","last_became_peered":"2024-04-25T13:38:45.357923+0000","last_unstale":"2024-04-25T13:40:00.374896+0000","last_undegraded":"2024-04-25T13:40:00.374896+0000","last_fullsized":"2024-04-25T13:40:00.374896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:47.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368746+0000","last_change":"2024-04-25T13:38:45.357865+0000","last_active":"2024-04-25T13:40:00.368746+0000","last_peered":"2024-04-25T13:40:00.368746+0000","last_clean":"2024-04-25T13:40:00.368746+0000","last_became_active":"2024-04-25T13:38:45.355399+0000","last_became_peered":"2024-04-25T13:38:45.355399+0000","last_unstale":"2024-04-25T13:40:00.368746+0000","last_undegraded":"2024-04-25T13:40:00.368746+0000","last_fullsized":"2024-04-25T13:40:00.368746+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:33:41.382462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349318+0000","last_change":"2024-04-25T13:38:45.350724+0000","last_active":"2024-04-25T13:40:00.349318+0000","last_peered":"2024-04-25T13:40:00.349318+0000","last_clean":"2024-04-25T13:40:00.349318+0000","last_became_active":"2024-04-25T13:38:45.350284+0000","last_became_peered":"2024-04-25T13:38:45.350284+0000","last_unstale":"2024-04-25T13:40:00.349318+0000","last_undegraded":"2024-04-25T13:40:00.349318+0000","last_fullsized":"2024-04-25T13:40:00.349318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:30.960843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351696+0000","last_change":"2024-04-25T13:38:45.362966+0000","last_active":"2024-04-25T13:40:00.351696+0000","last_peered":"2024-04-25T13:40:00.351696+0000","last_clean":"2024-04-25T13:40:00.351696+0000","last_became_active":"2024-04-25T13:38:45.362585+0000","last_became_peered":"2024-04-25T13:38:45.362585+0000","last_unstale":"2024-04-25T13:40:00.351696+0000","last_undegraded":"2024-04-25T13:40:00.351696+0000","last_fullsized":"2024-04-25T13:40:00.351696+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:46:10.825028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'4","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.488075+0000","last_change":"2024-04-25T13:38:44.354887+0000","last_active":"2024-04-25T13:39:31.488075+0000","last_peered":"2024-04-25T13:39:31.488075+0000","last_clean":"2024-04-25T13:39:31.488075+0000","last_became_active":"2024-04-25T13:38:44.354510+0000","last_became_peered":"2024-04-25T13:38:44.354510+0000","last_unstale":"2024-04-25T13:39:31.488075+0000","last_undegraded":"2024-04-25T13:39:31.488075+0000","last_fullsized":"2024-04-25T13:39:31.488075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:46.457087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"78'5","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363062+0000","last_change":"2024-04-25T13:38:45.364056+0000","last_active":"2024-04-25T13:40:00.363062+0000","last_peered":"2024-04-25T13:40:00.363062+0000","last_clean":"2024-04-25T13:40:00.363062+0000","last_became_active":"2024-04-25T13:38:45.363572+0000","last_became_peered":"2024-04-25T13:38:45.363572+0000","last_unstale":"2024-04-25T13:40:00.363062+0000","last_undegraded":"2024-04-25T13:40:00.363062+0000","last_fullsized":"2024-04-25T13:40:00.363062+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:49:12.136637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.490480+0000","last_change":"2024-04-25T13:38:44.362371+0000","last_active":"2024-04-25T13:39:31.490480+0000","last_peered":"2024-04-25T13:39:31.490480+0000","last_clean":"2024-04-25T13:39:31.490480+0000","last_became_active":"2024-04-25T13:38:44.361262+0000","last_became_peered":"2024-04-25T13:38:44.361262+0000","last_unstale":"2024-04-25T13:39:31.490480+0000","last_undegraded":"2024-04-25T13:39:31.490480+0000","last_fullsized":"2024-04-25T13:39:31.490480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:01.442189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"78'6","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367818+0000","last_change":"2024-04-25T13:38:45.362599+0000","last_active":"2024-04-25T13:40:00.367818+0000","last_peered":"2024-04-25T13:40:00.367818+0000","last_clean":"2024-04-25T13:40:00.367818+0000","last_became_active":"2024-04-25T13:38:45.361893+0000","last_became_peered":"2024-04-25T13:38:45.361893+0000","last_unstale":"2024-04-25T13:40:00.367818+0000","last_undegraded":"2024-04-25T13:40:00.367818+0000","last_fullsized":"2024-04-25T13:40:00.367818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:52.414826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491185+0000","last_change":"2024-04-25T13:38:44.353063+0000","last_active":"2024-04-25T13:39:31.491185+0000","last_peered":"2024-04-25T13:39:31.491185+0000","last_clean":"2024-04-25T13:39:31.491185+0000","last_became_active":"2024-04-25T13:38:44.352403+0000","last_became_peered":"2024-04-25T13:38:44.352403+0000","last_unstale":"2024-04-25T13:39:31.491185+0000","last_undegraded":"2024-04-25T13:39:31.491185+0000","last_fullsized":"2024-04-25T13:39:31.491185+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:09.670589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369475+0000","last_change":"2024-04-25T13:38:45.359078+0000","last_active":"2024-04-25T13:40:00.369475+0000","last_peered":"2024-04-25T13:40:00.369475+0000","last_clean":"2024-04-25T13:40:00.369475+0000","last_became_active":"2024-04-25T13:38:45.354063+0000","last_became_peered":"2024-04-25T13:38:45.354063+0000","last_unstale":"2024-04-25T13:40:00.369475+0000","last_undegraded":"2024-04-25T13:40:00.369475+0000","last_fullsized":"2024-04-25T13:40:00.369475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:16.694340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492390+0000","last_change":"2024-04-25T13:38:44.349132+0000","last_active":"2024-04-25T13:39:31.492390+0000","last_peered":"2024-04-25T13:39:31.492390+0000","last_clean":"2024-04-25T13:39:31.492390+0000","last_became_active":"2024-04-25T13:38:44.348762+0000","last_became_peered":"2024-04-25T13:38:44.348762+0000","last_unstale":"2024-04-25T13:39:31.492390+0000","last_undegraded":"2024-04-25T13:39:31.492390+0000","last_fullsized":"2024-04-25T13:39:31.492390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:12.990304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368999+0000","last_change":"2024-04-25T13:38:45.369116+0000","last_active":"2024-04-25T13:40:00.368999+0000","last_peered":"2024-04-25T13:40:00.368999+0000","last_clean":"2024-04-25T13:40:00.368999+0000","last_became_active":"2024-04-25T13:38:45.368380+0000","last_became_peered":"2024-04-25T13:38:45.368380+0000","last_unstale":"2024-04-25T13:40:00.368999+0000","last_undegraded":"2024-04-25T13:40:00.368999+0000","last_fullsized":"2024-04-25T13:40:00.368999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:02.259401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485437+0000","last_change":"2024-04-25T13:38:44.351873+0000","last_active":"2024-04-25T13:39:31.485437+0000","last_peered":"2024-04-25T13:39:31.485437+0000","last_clean":"2024-04-25T13:39:31.485437+0000","last_became_active":"2024-04-25T13:38:44.351518+0000","last_became_peered":"2024-04-25T13:38:44.351518+0000","last_unstale":"2024-04-25T13:39:31.485437+0000","last_undegraded":"2024-04-25T13:39:31.485437+0000","last_fullsized":"2024-04-25T13:39:31.485437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:06.839053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342392+0000","last_change":"2024-04-25T13:38:45.351369+0000","last_active":"2024-04-25T13:40:00.342392+0000","last_peered":"2024-04-25T13:40:00.342392+0000","last_clean":"2024-04-25T13:40:00.342392+0000","last_became_active":"2024-04-25T13:38:45.351018+0000","last_became_peered":"2024-04-25T13:38:45.351018+0000","last_unstale":"2024-04-25T13:40:00.342392+0000","last_undegraded":"2024-04-25T13:40:00.342392+0000","last_fullsized":"2024-04-25T13:40:00.342392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040292+0000","last_change":"2024-04-25T13:38:44.345986+0000","last_active":"2024-04-25T13:39:32.040292+0000","last_peered":"2024-04-25T13:39:32.040292+0000","last_clean":"2024-04-25T13:39:32.040292+0000","last_became_active":"2024-04-25T13:38:44.345600+0000","last_became_peered":"2024-04-25T13:38:44.345600+0000","last_unstale":"2024-04-25T13:39:32.040292+0000","last_undegraded":"2024-04-25T13:39:32.040292+0000","last_fullsized":"2024-04-25T13:39:32.040292+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:52.749192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365960+0000","last_change":"2024-04-25T13:38:45.368641+0000","last_active":"2024-04-25T13:40:00.365960+0000","last_peered":"2024-04-25T13:40:00.365960+0000","last_clean":"2024-04-25T13:40:00.365960+0000","last_became_active":"2024-04-25T13:38:45.368209+0000","last_became_peered":"2024-04-25T13:38:45.368209+0000","last_unstale":"2024-04-25T13:40:00.365960+0000","last_undegraded":"2024-04-25T13:40:00.365960+0000","last_fullsized":"2024-04-25T13:40:00.365960+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:47.990808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484783+0000","last_change":"2024-04-25T13:38:44.339927+0000","last_active":"2024-04-25T13:39:31.484783+0000","last_peered":"2024-04-25T13:39:31.484783+0000","last_clean":"2024-04-25T13:39:31.484783+0000","last_became_active":"2024-04-25T13:38:44.339588+0000","last_became_peered":"2024-04-25T13:38:44.339588+0000","last_unstale":"2024-04-25T13:39:31.484783+0000","last_undegraded":"2024-04-25T13:39:31.484783+0000","last_fullsized":"2024-04-25T13:39:31.484783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:51.707750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365330+0000","last_change":"2024-04-25T13:38:45.357323+0000","last_active":"2024-04-25T13:40:00.365330+0000","last_peered":"2024-04-25T13:40:00.365330+0000","last_clean":"2024-04-25T13:40:00.365330+0000","last_became_active":"2024-04-25T13:38:45.356870+0000","last_became_peered":"2024-04-25T13:38:45.356870+0000","last_unstale":"2024-04-25T13:40:00.365330+0000","last_undegraded":"2024-04-25T13:40:00.365330+0000","last_fullsized":"2024-04-25T13:40:00.365330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:36.838709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485179+0000","last_change":"2024-04-25T13:38:44.351294+0000","last_active":"2024-04-25T13:39:31.485179+0000","last_peered":"2024-04-25T13:39:31.485179+0000","last_clean":"2024-04-25T13:39:31.485179+0000","last_became_active":"2024-04-25T13:38:44.350971+0000","last_became_peered":"2024-04-25T13:38:44.350971+0000","last_unstale":"2024-04-25T13:39:31.485179+0000","last_undegraded":"2024-04-25T13:39:31.485179+0000","last_fullsized":"2024-04-25T13:39:31.485179+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:22.452217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349813+0000","last_change":"2024-04-25T13:38:45.345943+0000","last_active":"2024-04-25T13:40:00.349813+0000","last_peered":"2024-04-25T13:40:00.349813+0000","last_clean":"2024-04-25T13:40:00.349813+0000","last_became_active":"2024-04-25T13:38:45.345528+0000","last_became_peered":"2024-04-25T13:38:45.345528+0000","last_unstale":"2024-04-25T13:40:00.349813+0000","last_undegraded":"2024-04-25T13:40:00.349813+0000","last_fullsized":"2024-04-25T13:40:00.349813+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:57.327303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040662+0000","last_change":"2024-04-25T13:38:44.343032+0000","last_active":"2024-04-25T13:39:32.040662+0000","last_peered":"2024-04-25T13:39:32.040662+0000","last_clean":"2024-04-25T13:39:32.040662+0000","last_became_active":"2024-04-25T13:38:44.342613+0000","last_became_peered":"2024-04-25T13:38:44.342613+0000","last_unstale":"2024-04-25T13:39:32.040662+0000","last_undegraded":"2024-04-25T13:39:32.040662+0000","last_fullsized":"2024-04-25T13:39:32.040662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:33.957150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369787+0000","last_change":"2024-04-25T13:38:45.354534+0000","last_active":"2024-04-25T13:40:00.369787+0000","last_peered":"2024-04-25T13:40:00.369787+0000","last_clean":"2024-04-25T13:40:00.369787+0000","last_became_active":"2024-04-25T13:38:45.354147+0000","last_became_peered":"2024-04-25T13:38:45.354147+0000","last_unstale":"2024-04-25T13:40:00.369787+0000","last_undegraded":"2024-04-25T13:40:00.369787+0000","last_fullsized":"2024-04-25T13:40:00.369787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:31.537339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487809+0000","last_change":"2024-04-25T13:38:44.356103+0000","last_active":"2024-04-25T13:39:31.487809+0000","last_peered":"2024-04-25T13:39:31.487809+0000","last_clean":"2024-04-25T13:39:31.487809+0000","last_became_active":"2024-04-25T13:38:44.355702+0000","last_became_peered":"2024-04-25T13:38:44.355702+0000","last_unstale":"2024-04-25T13:39:31.487809+0000","last_undegraded":"2024-04-25T13:39:31.487809+0000","last_fullsized":"2024-04-25T13:39:31.487809+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:47.572922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"78'9","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.344967+0000","last_change":"2024-04-25T13:39:31.494580+0000","last_active":"2024-04-25T13:40:00.344967+0000","last_peered":"2024-04-25T13:40:00.344967+0000","last_clean":"2024-04-25T13:40:00.344967+0000","last_became_active":"2024-04-25T13:39:31.494102+0000","last_became_peered":"2024-04-25T13:39:31.494102+0000","last_unstale":"2024-04-25T13:40:00.344967+0000","last_undegraded":"2024-04-25T13:40:00.344967+0000","last_fullsized":"2024-04-25T13:40:00.344967+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.056136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485841+0000","last_change":"2024-04-25T13:38:44.355474+0000","last_active":"2024-04-25T13:39:31.485841+0000","last_peered":"2024-04-25T13:39:31.485841+0000","last_clean":"2024-04-25T13:39:31.485841+0000","last_became_active":"2024-04-25T13:38:44.355113+0000","last_became_peered":"2024-04-25T13:38:44.355113+0000","last_unstale":"2024-04-25T13:39:31.485841+0000","last_undegraded":"2024-04-25T13:39:31.485841+0000","last_fullsized":"2024-04-25T13:39:31.485841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.242974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.370871+0000","last_change":"2024-04-25T13:38:45.369618+0000","last_active":"2024-04-25T13:40:00.370871+0000","last_peered":"2024-04-25T13:40:00.370871+0000","last_clean":"2024-04-25T13:40:00.370871+0000","last_became_active":"2024-04-25T13:38:45.369252+0000","last_became_peered":"2024-04-25T13:38:45.369252+0000","last_unstale":"2024-04-25T13:40:00.370871+0000","last_undegraded":"2024-04-25T13:40:00.370871+0000","last_fullsized":"2024-04-25T13:40:00.370871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:40.336573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"78'32","reported_seq":307,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:50.608632+0000","last_change":"2024-04-25T13:38:44.349850+0000","last_active":"2024-04-25T13:40:50.608632+0000","last_peered":"2024-04-25T13:40:50.608632+0000","last_clean":"2024-04-25T13:40:50.608632+0000","last_became_active":"2024-04-25T13:38:44.349404+0000","last_became_peered":"2024-04-25T13:38:44.349404+0000","last_unstale":"2024-04-25T13:40:50.608632+0000","last_undegraded":"2024-04-25T13:40:50.608632+0000","last_fullsized":"2024-04-25T13:40:50.608632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:40.025684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66976,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":256,"num_read_kb":193,"num_write":32,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.348329+0000","last_change":"2024-04-25T13:38:45.371554+0000","last_active":"2024-04-25T13:40:00.348329+0000","last_peered":"2024-04-25T13:40:00.348329+0000","last_clean":"2024-04-25T13:40:00.348329+0000","last_became_active":"2024-04-25T13:38:45.371042+0000","last_became_peered":"2024-04-25T13:38:45.371042+0000","last_unstale":"2024-04-25T13:40:00.348329+0000","last_undegraded":"2024-04-25T13:40:00.348329+0000","last_fullsized":"2024-04-25T13:40:00.348329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:59.240536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494618+0000","last_change":"2024-04-25T13:38:44.345979+0000","last_active":"2024-04-25T13:39:31.494618+0000","last_peered":"2024-04-25T13:39:31.494618+0000","last_clean":"2024-04-25T13:39:31.494618+0000","last_became_active":"2024-04-25T13:38:44.345596+0000","last_became_peered":"2024-04-25T13:38:44.345596+0000","last_unstale":"2024-04-25T13:39:31.494618+0000","last_undegraded":"2024-04-25T13:39:31.494618+0000","last_fullsized":"2024-04-25T13:39:31.494618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:22:51.539024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367445+0000","last_change":"2024-04-25T13:38:45.364794+0000","last_active":"2024-04-25T13:40:00.367445+0000","last_peered":"2024-04-25T13:40:00.367445+0000","last_clean":"2024-04-25T13:40:00.367445+0000","last_became_active":"2024-04-25T13:38:45.364381+0000","last_became_peered":"2024-04-25T13:38:45.364381+0000","last_unstale":"2024-04-25T13:40:00.367445+0000","last_undegraded":"2024-04-25T13:40:00.367445+0000","last_fullsized":"2024-04-25T13:40:00.367445+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:16.856980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485286+0000","last_change":"2024-04-25T13:38:44.351269+0000","last_active":"2024-04-25T13:39:31.485286+0000","last_peered":"2024-04-25T13:39:31.485286+0000","last_clean":"2024-04-25T13:39:31.485286+0000","last_became_active":"2024-04-25T13:38:44.350933+0000","last_became_peered":"2024-04-25T13:38:44.350933+0000","last_unstale":"2024-04-25T13:39:31.485286+0000","last_undegraded":"2024-04-25T13:39:31.485286+0000","last_fullsized":"2024-04-25T13:39:31.485286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:02.677849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.372710+0000","last_change":"2024-04-25T13:38:45.362732+0000","last_active":"2024-04-25T13:40:00.372710+0000","last_peered":"2024-04-25T13:40:00.372710+0000","last_clean":"2024-04-25T13:40:00.372710+0000","last_became_active":"2024-04-25T13:38:45.361957+0000","last_became_peered":"2024-04-25T13:38:45.361957+0000","last_unstale":"2024-04-25T13:40:00.372710+0000","last_undegraded":"2024-04-25T13:40:00.372710+0000","last_fullsized":"2024-04-25T13:40:00.372710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:50.011442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.776862+0000","last_change":"2024-04-25T13:38:44.349378+0000","last_active":"2024-04-25T13:39:31.776862+0000","last_peered":"2024-04-25T13:39:31.776862+0000","last_clean":"2024-04-25T13:39:31.776862+0000","last_became_active":"2024-04-25T13:38:44.348887+0000","last_became_peered":"2024-04-25T13:38:44.348887+0000","last_unstale":"2024-04-25T13:39:31.776862+0000","last_undegraded":"2024-04-25T13:39:31.776862+0000","last_fullsized":"2024-04-25T13:39:31.776862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.747205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375453+0000","last_change":"2024-04-25T13:38:45.349941+0000","last_active":"2024-04-25T13:40:00.375453+0000","last_peered":"2024-04-25T13:40:00.375453+0000","last_clean":"2024-04-25T13:40:00.375453+0000","last_became_active":"2024-04-25T13:38:45.349590+0000","last_became_peered":"2024-04-25T13:38:45.349590+0000","last_unstale":"2024-04-25T13:40:00.375453+0000","last_undegraded":"2024-04-25T13:40:00.375453+0000","last_fullsized":"2024-04-25T13:40:00.375453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:42.904406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.791483+0000","last_change":"2024-04-25T13:39:31.790997+0000","last_active":"2024-04-25T13:39:31.791483+0000","last_peered":"2024-04-25T13:39:31.791483+0000","last_clean":"2024-04-25T13:39:31.791483+0000","last_became_active":"2024-04-25T13:39:31.778451+0000","last_became_peered":"2024-04-25T13:39:31.778451+0000","last_unstale":"2024-04-25T13:39:31.791483+0000","last_undegraded":"2024-04-25T13:39:31.791483+0000","last_fullsized":"2024-04-25T13:39:31.791483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:17.735970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368028+0000","last_change":"2024-04-25T13:38:45.368589+0000","last_active":"2024-04-25T13:40:00.368028+0000","last_peered":"2024-04-25T13:40:00.368028+0000","last_clean":"2024-04-25T13:40:00.368028+0000","last_became_active":"2024-04-25T13:38:45.368079+0000","last_became_peered":"2024-04-25T13:38:45.368079+0000","last_unstale":"2024-04-25T13:40:00.368028+0000","last_undegraded":"2024-04-25T13:40:00.368028+0000","last_fullsized":"2024-04-25T13:40:00.368028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:10.180264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491662+0000","last_change":"2024-04-25T13:38:44.345148+0000","last_active":"2024-04-25T13:39:31.491662+0000","last_peered":"2024-04-25T13:39:31.491662+0000","last_clean":"2024-04-25T13:39:31.491662+0000","last_became_active":"2024-04-25T13:38:44.344816+0000","last_became_peered":"2024-04-25T13:38:44.344816+0000","last_unstale":"2024-04-25T13:39:31.491662+0000","last_undegraded":"2024-04-25T13:39:31.491662+0000","last_fullsized":"2024-04-25T13:39:31.491662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:03:38.134169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351253+0000","last_change":"2024-04-25T13:38:45.354340+0000","last_active":"2024-04-25T13:40:00.351253+0000","last_peered":"2024-04-25T13:40:00.351253+0000","last_clean":"2024-04-25T13:40:00.351253+0000","last_became_active":"2024-04-25T13:38:45.353933+0000","last_became_peered":"2024-04-25T13:38:45.353933+0000","last_unstale":"2024-04-25T13:40:00.351253+0000","last_undegraded":"2024-04-25T13:40:00.351253+0000","last_fullsized":"2024-04-25T13:40:00.351253+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:07.619051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.488024+0000","last_change":"2024-04-25T13:38:44.351183+0000","last_active":"2024-04-25T13:39:31.488024+0000","last_peered":"2024-04-25T13:39:31.488024+0000","last_clean":"2024-04-25T13:39:31.488024+0000","last_became_active":"2024-04-25T13:38:44.350785+0000","last_became_peered":"2024-04-25T13:38:44.350785+0000","last_unstale":"2024-04-25T13:39:31.488024+0000","last_undegraded":"2024-04-25T13:39:31.488024+0000","last_fullsized":"2024-04-25T13:39:31.488024+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:37:57.465012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"78'5","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360834+0000","last_change":"2024-04-25T13:38:45.362355+0000","last_active":"2024-04-25T13:40:00.360834+0000","last_peered":"2024-04-25T13:40:00.360834+0000","last_clean":"2024-04-25T13:40:00.360834+0000","last_became_active":"2024-04-25T13:38:45.361996+0000","last_became_peered":"2024-04-25T13:38:45.361996+0000","last_unstale":"2024-04-25T13:40:00.360834+0000","last_undegraded":"2024-04-25T13:40:00.360834+0000","last_fullsized":"2024-04-25T13:40:00.360834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:53:24.790450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487641+0000","last_change":"2024-04-25T13:38:44.356079+0000","last_active":"2024-04-25T13:39:31.487641+0000","last_peered":"2024-04-25T13:39:31.487641+0000","last_clean":"2024-04-25T13:39:31.487641+0000","last_became_active":"2024-04-25T13:38:44.355660+0000","last_became_peered":"2024-04-25T13:38:44.355660+0000","last_unstale":"2024-04-25T13:39:31.487641+0000","last_undegraded":"2024-04-25T13:39:31.487641+0000","last_fullsized":"2024-04-25T13:39:31.487641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:35.161337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"78'7","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363216+0000","last_change":"2024-04-25T13:38:45.356578+0000","last_active":"2024-04-25T13:40:00.363216+0000","last_peered":"2024-04-25T13:40:00.363216+0000","last_clean":"2024-04-25T13:40:00.363216+0000","last_became_active":"2024-04-25T13:38:45.356232+0000","last_became_peered":"2024-04-25T13:38:45.356232+0000","last_unstale":"2024-04-25T13:40:00.363216+0000","last_undegraded":"2024-04-25T13:40:00.363216+0000","last_fullsized":"2024-04-25T13:40:00.363216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.476890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"78'8","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360561+0000","last_change":"2024-04-25T13:39:31.778577+0000","last_active":"2024-04-25T13:40:00.360561+0000","last_peered":"2024-04-25T13:40:00.360561+0000","last_clean":"2024-04-25T13:40:00.360561+0000","last_became_active":"2024-04-25T13:39:31.778178+0000","last_became_peered":"2024-04-25T13:39:31.778178+0000","last_unstale":"2024-04-25T13:40:00.360561+0000","last_undegraded":"2024-04-25T13:40:00.360561+0000","last_fullsized":"2024-04-25T13:40:00.360561+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.383670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485143+0000","last_change":"2024-04-25T13:38:44.351113+0000","last_active":"2024-04-25T13:39:31.485143+0000","last_peered":"2024-04-25T13:39:31.485143+0000","last_clean":"2024-04-25T13:39:31.485143+0000","last_became_active":"2024-04-25T13:38:44.350730+0000","last_became_peered":"2024-04-25T13:38:44.350730+0000","last_unstale":"2024-04-25T13:39:31.485143+0000","last_undegraded":"2024-04-25T13:39:31.485143+0000","last_fullsized":"2024-04-25T13:39:31.485143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:38:22.747930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"78'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362961+0000","last_change":"2024-04-25T13:38:45.356731+0000","last_active":"2024-04-25T13:40:00.362961+0000","last_peered":"2024-04-25T13:40:00.362961+0000","last_clean":"2024-04-25T13:40:00.362961+0000","last_became_active":"2024-04-25T13:38:45.355531+0000","last_became_peered":"2024-04-25T13:38:45.355531+0000","last_unstale":"2024-04-25T13:40:00.362961+0000","last_undegraded":"2024-04-25T13:40:00.362961+0000","last_fullsized":"2024-04-25T13:40:00.362961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:15.482469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777207+0000","last_change":"2024-04-25T13:38:44.345941+0000","last_active":"2024-04-25T13:39:31.777207+0000","last_peered":"2024-04-25T13:39:31.777207+0000","last_clean":"2024-04-25T13:39:31.777207+0000","last_became_active":"2024-04-25T13:38:44.344634+0000","last_became_peered":"2024-04-25T13:38:44.344634+0000","last_unstale":"2024-04-25T13:39:31.777207+0000","last_undegraded":"2024-04-25T13:39:31.777207+0000","last_fullsized":"2024-04-25T13:39:31.777207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:42.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":94,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487340+0000","last_change":"2024-04-25T13:37:27.903345+0000","last_active":"2024-04-25T13:39:31.487340+0000","last_peered":"2024-04-25T13:39:31.487340+0000","last_clean":"2024-04-25T13:39:31.487340+0000","last_became_active":"2024-04-25T13:37:27.897986+0000","last_became_peered":"2024-04-25T13:37:27.897986+0000","last_unstale":"2024-04-25T13:39:31.487340+0000","last_undegraded":"2024-04-25T13:39:31.487340+0000","last_fullsized":"2024-04-25T13:39:31.487340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_clean_scrub_stamp":"2024-04-25T13:33:56.569417+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:56.390458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"78'5","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.364075+0000","last_change":"2024-04-25T13:38:45.369390+0000","last_active":"2024-04-25T13:40:00.364075+0000","last_peered":"2024-04-25T13:40:00.364075+0000","last_clean":"2024-04-25T13:40:00.364075+0000","last_became_active":"2024-04-25T13:38:45.368938+0000","last_became_peered":"2024-04-25T13:38:45.368938+0000","last_unstale":"2024-04-25T13:40:00.364075+0000","last_undegraded":"2024-04-25T13:40:00.364075+0000","last_fullsized":"2024-04-25T13:40:00.364075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:11.942204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487500+0000","last_change":"2024-04-25T13:38:44.355585+0000","last_active":"2024-04-25T13:39:31.487500+0000","last_peered":"2024-04-25T13:39:31.487500+0000","last_clean":"2024-04-25T13:39:31.487500+0000","last_became_active":"2024-04-25T13:38:44.355131+0000","last_became_peered":"2024-04-25T13:38:44.355131+0000","last_unstale":"2024-04-25T13:39:31.487500+0000","last_undegraded":"2024-04-25T13:39:31.487500+0000","last_fullsized":"2024-04-25T13:39:31.487500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:05.240247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492418+0000","last_change":"2024-04-25T13:39:31.491742+0000","last_active":"2024-04-25T13:39:31.492418+0000","last_peered":"2024-04-25T13:39:31.492418+0000","last_clean":"2024-04-25T13:39:31.492418+0000","last_became_active":"2024-04-25T13:39:31.491266+0000","last_became_peered":"2024-04-25T13:39:31.491266+0000","last_unstale":"2024-04-25T13:39:31.492418+0000","last_undegraded":"2024-04-25T13:39:31.492418+0000","last_fullsized":"2024-04-25T13:39:31.492418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:21.932333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"78'9","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356400+0000","last_change":"2024-04-25T13:38:45.366187+0000","last_active":"2024-04-25T13:40:00.356400+0000","last_peered":"2024-04-25T13:40:00.356400+0000","last_clean":"2024-04-25T13:40:00.356400+0000","last_became_active":"2024-04-25T13:38:45.365744+0000","last_became_peered":"2024-04-25T13:38:45.365744+0000","last_unstale":"2024-04-25T13:40:00.356400+0000","last_undegraded":"2024-04-25T13:40:00.356400+0000","last_fullsized":"2024-04-25T13:40:00.356400+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:07.970735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"78'8","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.378057+0000","last_change":"2024-04-25T13:38:44.359250+0000","last_active":"2024-04-25T13:40:00.378057+0000","last_peered":"2024-04-25T13:40:00.378057+0000","last_clean":"2024-04-25T13:40:00.378057+0000","last_became_active":"2024-04-25T13:38:44.358874+0000","last_became_peered":"2024-04-25T13:38:44.358874+0000","last_unstale":"2024-04-25T13:40:00.378057+0000","last_undegraded":"2024-04-25T13:40:00.378057+0000","last_fullsized":"2024-04-25T13:40:00.378057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:28.820465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366411+0000","last_change":"2024-04-25T13:38:45.370080+0000","last_active":"2024-04-25T13:40:00.366411+0000","last_peered":"2024-04-25T13:40:00.366411+0000","last_clean":"2024-04-25T13:40:00.366411+0000","last_became_active":"2024-04-25T13:38:45.369668+0000","last_became_peered":"2024-04-25T13:38:45.369668+0000","last_unstale":"2024-04-25T13:40:00.366411+0000","last_undegraded":"2024-04-25T13:40:00.366411+0000","last_fullsized":"2024-04-25T13:40:00.366411+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:03.197468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777370+0000","last_change":"2024-04-25T13:38:44.358907+0000","last_active":"2024-04-25T13:39:31.777370+0000","last_peered":"2024-04-25T13:39:31.777370+0000","last_clean":"2024-04-25T13:39:31.777370+0000","last_became_active":"2024-04-25T13:38:44.358455+0000","last_became_peered":"2024-04-25T13:38:44.358455+0000","last_unstale":"2024-04-25T13:39:31.777370+0000","last_undegraded":"2024-04-25T13:39:31.777370+0000","last_fullsized":"2024-04-25T13:39:31.777370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:50:52.916433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365642+0000","last_change":"2024-04-25T13:38:45.373050+0000","last_active":"2024-04-25T13:40:00.365642+0000","last_peered":"2024-04-25T13:40:00.365642+0000","last_clean":"2024-04-25T13:40:00.365642+0000","last_became_active":"2024-04-25T13:38:45.372621+0000","last_became_peered":"2024-04-25T13:38:45.372621+0000","last_unstale":"2024-04-25T13:40:00.365642+0000","last_undegraded":"2024-04-25T13:40:00.365642+0000","last_fullsized":"2024-04-25T13:40:00.365642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:29:43.259296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485534+0000","last_change":"2024-04-25T13:38:44.355768+0000","last_active":"2024-04-25T13:39:31.485534+0000","last_peered":"2024-04-25T13:39:31.485534+0000","last_clean":"2024-04-25T13:39:31.485534+0000","last_became_active":"2024-04-25T13:38:44.355400+0000","last_became_peered":"2024-04-25T13:38:44.355400+0000","last_unstale":"2024-04-25T13:39:31.485534+0000","last_undegraded":"2024-04-25T13:39:31.485534+0000","last_fullsized":"2024-04-25T13:39:31.485534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:38.533717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342472+0000","last_change":"2024-04-25T13:38:45.350582+0000","last_active":"2024-04-25T13:40:00.342472+0000","last_peered":"2024-04-25T13:40:00.342472+0000","last_clean":"2024-04-25T13:40:00.342472+0000","last_became_active":"2024-04-25T13:38:45.350220+0000","last_became_peered":"2024-04-25T13:38:45.350220+0000","last_unstale":"2024-04-25T13:40:00.342472+0000","last_undegraded":"2024-04-25T13:40:00.342472+0000","last_fullsized":"2024-04-25T13:40:00.342472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.463519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.489729+0000","last_change":"2024-04-25T13:38:44.360459+0000","last_active":"2024-04-25T13:39:31.489729+0000","last_peered":"2024-04-25T13:39:31.489729+0000","last_clean":"2024-04-25T13:39:31.489729+0000","last_became_active":"2024-04-25T13:38:44.359709+0000","last_became_peered":"2024-04-25T13:38:44.359709+0000","last_unstale":"2024-04-25T13:39:31.489729+0000","last_undegraded":"2024-04-25T13:39:31.489729+0000","last_fullsized":"2024-04-25T13:39:31.489729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:53.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356628+0000","last_change":"2024-04-25T13:38:45.357291+0000","last_active":"2024-04-25T13:40:00.356628+0000","last_peered":"2024-04-25T13:40:00.356628+0000","last_clean":"2024-04-25T13:40:00.356628+0000","last_became_active":"2024-04-25T13:38:45.356929+0000","last_became_peered":"2024-04-25T13:38:45.356929+0000","last_unstale":"2024-04-25T13:40:00.356628+0000","last_undegraded":"2024-04-25T13:40:00.356628+0000","last_fullsized":"2024-04-25T13:40:00.356628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:02:08.619208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.493732+0000","last_change":"2024-04-25T13:38:44.343885+0000","last_active":"2024-04-25T13:39:31.493732+0000","last_peered":"2024-04-25T13:39:31.493732+0000","last_clean":"2024-04-25T13:39:31.493732+0000","last_became_active":"2024-04-25T13:38:44.343541+0000","last_became_peered":"2024-04-25T13:38:44.343541+0000","last_unstale":"2024-04-25T13:39:31.493732+0000","last_undegraded":"2024-04-25T13:39:31.493732+0000","last_fullsized":"2024-04-25T13:39:31.493732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:50:01.904593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365216+0000","last_change":"2024-04-25T13:38:45.371508+0000","last_active":"2024-04-25T13:40:00.365216+0000","last_peered":"2024-04-25T13:40:00.365216+0000","last_clean":"2024-04-25T13:40:00.365216+0000","last_became_active":"2024-04-25T13:38:45.370912+0000","last_became_peered":"2024-04-25T13:38:45.370912+0000","last_unstale":"2024-04-25T13:40:00.365216+0000","last_undegraded":"2024-04-25T13:40:00.365216+0000","last_fullsized":"2024-04-25T13:40:00.365216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:07.075511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'2","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.498142+0000","last_change":"2024-04-25T13:39:31.498142+0000","last_active":"2024-04-25T13:39:31.498142+0000","last_peered":"2024-04-25T13:39:31.498142+0000","last_clean":"2024-04-25T13:39:31.498142+0000","last_became_active":"2024-04-25T13:39:31.494646+0000","last_became_peered":"2024-04-25T13:39:31.494646+0000","last_unstale":"2024-04-25T13:39:31.498142+0000","last_undegraded":"2024-04-25T13:39:31.498142+0000","last_fullsized":"2024-04-25T13:39:31.498142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:46.084354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375168+0000","last_change":"2024-04-25T13:38:45.363322+0000","last_active":"2024-04-25T13:40:00.375168+0000","last_peered":"2024-04-25T13:40:00.375168+0000","last_clean":"2024-04-25T13:40:00.375168+0000","last_became_active":"2024-04-25T13:38:45.362948+0000","last_became_peered":"2024-04-25T13:38:45.362948+0000","last_unstale":"2024-04-25T13:40:00.375168+0000","last_undegraded":"2024-04-25T13:40:00.375168+0000","last_fullsized":"2024-04-25T13:40:00.375168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:20.815774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494630+0000","last_change":"2024-04-25T13:38:44.358588+0000","last_active":"2024-04-25T13:39:31.494630+0000","last_peered":"2024-04-25T13:39:31.494630+0000","last_clean":"2024-04-25T13:39:31.494630+0000","last_became_active":"2024-04-25T13:38:44.358269+0000","last_became_peered":"2024-04-25T13:38:44.358269+0000","last_unstale":"2024-04-25T13:39:31.494630+0000","last_undegraded":"2024-04-25T13:39:31.494630+0000","last_fullsized":"2024-04-25T13:39:31.494630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:35.405572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375021+0000","last_change":"2024-04-25T13:38:45.369272+0000","last_active":"2024-04-25T13:40:00.375021+0000","last_peered":"2024-04-25T13:40:00.375021+0000","last_clean":"2024-04-25T13:40:00.375021+0000","last_became_active":"2024-04-25T13:38:45.368774+0000","last_became_peered":"2024-04-25T13:38:45.368774+0000","last_unstale":"2024-04-25T13:40:00.375021+0000","last_undegraded":"2024-04-25T13:40:00.375021+0000","last_fullsized":"2024-04-25T13:40:00.375021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:39.149557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485416+0000","last_change":"2024-04-25T13:38:44.349479+0000","last_active":"2024-04-25T13:39:31.485416+0000","last_peered":"2024-04-25T13:39:31.485416+0000","last_clean":"2024-04-25T13:39:31.485416+0000","last_became_active":"2024-04-25T13:38:44.349104+0000","last_became_peered":"2024-04-25T13:38:44.349104+0000","last_unstale":"2024-04-25T13:39:31.485416+0000","last_undegraded":"2024-04-25T13:39:31.485416+0000","last_fullsized":"2024-04-25T13:39:31.485416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:39.257472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"78'7","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375501+0000","last_change":"2024-04-25T13:39:31.495349+0000","last_active":"2024-04-25T13:40:00.375501+0000","last_peered":"2024-04-25T13:40:00.375501+0000","last_clean":"2024-04-25T13:40:00.375501+0000","last_became_active":"2024-04-25T13:39:31.490741+0000","last_became_peered":"2024-04-25T13:39:31.490741+0000","last_unstale":"2024-04-25T13:40:00.375501+0000","last_undegraded":"2024-04-25T13:40:00.375501+0000","last_fullsized":"2024-04-25T13:40:00.375501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.474725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484923+0000","last_change":"2024-04-25T13:38:44.349502+0000","last_active":"2024-04-25T13:39:31.484923+0000","last_peered":"2024-04-25T13:39:31.484923+0000","last_clean":"2024-04-25T13:39:31.484923+0000","last_became_active":"2024-04-25T13:38:44.349127+0000","last_became_peered":"2024-04-25T13:38:44.349127+0000","last_unstale":"2024-04-25T13:39:31.484923+0000","last_undegraded":"2024-04-25T13:39:31.484923+0000","last_fullsized":"2024-04-25T13:39:31.484923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:34:50.673352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.371319+0000","last_change":"2024-04-25T13:38:45.364746+0000","last_active":"2024-04-25T13:40:00.371319+0000","last_peered":"2024-04-25T13:40:00.371319+0000","last_clean":"2024-04-25T13:40:00.371319+0000","last_became_active":"2024-04-25T13:38:45.364364+0000","last_became_peered":"2024-04-25T13:38:45.364364+0000","last_unstale":"2024-04-25T13:40:00.371319+0000","last_undegraded":"2024-04-25T13:40:00.371319+0000","last_fullsized":"2024-04-25T13:40:00.371319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:42.145637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357194+0000","last_change":"2024-04-25T13:38:45.356683+0000","last_active":"2024-04-25T13:40:00.357194+0000","last_peered":"2024-04-25T13:40:00.357194+0000","last_clean":"2024-04-25T13:40:00.357194+0000","last_became_active":"2024-04-25T13:38:45.356048+0000","last_became_peered":"2024-04-25T13:38:45.356048+0000","last_unstale":"2024-04-25T13:40:00.357194+0000","last_undegraded":"2024-04-25T13:40:00.357194+0000","last_fullsized":"2024-04-25T13:40:00.357194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:49.578317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"78'12","reported_seq":30,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363111+0000","last_change":"2024-04-25T13:38:45.358298+0000","last_active":"2024-04-25T13:40:00.363111+0000","last_peered":"2024-04-25T13:40:00.363111+0000","last_clean":"2024-04-25T13:40:00.363111+0000","last_became_active":"2024-04-25T13:38:45.357986+0000","last_became_peered":"2024-04-25T13:38:45.357986+0000","last_unstale":"2024-04-25T13:40:00.363111+0000","last_undegraded":"2024-04-25T13:40:00.363111+0000","last_fullsized":"2024-04-25T13:40:00.363111+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:33.093156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"78'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356232+0000","last_change":"2024-04-25T13:38:45.363043+0000","last_active":"2024-04-25T13:40:00.356232+0000","last_peered":"2024-04-25T13:40:00.356232+0000","last_clean":"2024-04-25T13:40:00.356232+0000","last_became_active":"2024-04-25T13:38:45.362686+0000","last_became_peered":"2024-04-25T13:38:45.362686+0000","last_unstale":"2024-04-25T13:40:00.356232+0000","last_undegraded":"2024-04-25T13:40:00.356232+0000","last_fullsized":"2024-04-25T13:40:00.356232+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:56.602134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"78'7","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362173+0000","last_change":"2024-04-25T13:38:45.373256+0000","last_active":"2024-04-25T13:40:00.362173+0000","last_peered":"2024-04-25T13:40:00.362173+0000","last_clean":"2024-04-25T13:40:00.362173+0000","last_became_active":"2024-04-25T13:38:45.372844+0000","last_became_peered":"2024-04-25T13:38:45.372844+0000","last_unstale":"2024-04-25T13:40:00.362173+0000","last_undegraded":"2024-04-25T13:40:00.362173+0000","last_fullsized":"2024-04-25T13:40:00.362173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:39:34.590536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.364083+0000","last_change":"2024-04-25T13:38:45.370007+0000","last_active":"2024-04-25T13:40:00.364083+0000","last_peered":"2024-04-25T13:40:00.364083+0000","last_clean":"2024-04-25T13:40:00.364083+0000","last_became_active":"2024-04-25T13:38:45.369606+0000","last_became_peered":"2024-04-25T13:38:45.369606+0000","last_unstale":"2024-04-25T13:40:00.364083+0000","last_undegraded":"2024-04-25T13:40:00.364083+0000","last_fullsized":"2024-04-25T13:40:00.364083+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:45.514756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.350193+0000","last_change":"2024-04-25T13:38:45.362075+0000","last_active":"2024-04-25T13:40:00.350193+0000","last_peered":"2024-04-25T13:40:00.350193+0000","last_clean":"2024-04-25T13:40:00.350193+0000","last_became_active":"2024-04-25T13:38:45.361720+0000","last_became_peered":"2024-04-25T13:38:45.361720+0000","last_unstale":"2024-04-25T13:40:00.350193+0000","last_undegraded":"2024-04-25T13:40:00.350193+0000","last_fullsized":"2024-04-25T13:40:00.350193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:36.169185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354853+0000","last_change":"2024-04-25T13:38:45.352101+0000","last_active":"2024-04-25T13:40:00.354853+0000","last_peered":"2024-04-25T13:40:00.354853+0000","last_clean":"2024-04-25T13:40:00.354853+0000","last_became_active":"2024-04-25T13:38:45.351739+0000","last_became_peered":"2024-04-25T13:38:45.351739+0000","last_unstale":"2024-04-25T13:40:00.354853+0000","last_undegraded":"2024-04-25T13:40:00.354853+0000","last_fullsized":"2024-04-25T13:40:00.354853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:05.612527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362759+0000","last_change":"2024-04-25T13:38:45.363855+0000","last_active":"2024-04-25T13:40:00.362759+0000","last_peered":"2024-04-25T13:40:00.362759+0000","last_clean":"2024-04-25T13:40:00.362759+0000","last_became_active":"2024-04-25T13:38:45.363377+0000","last_became_peered":"2024-04-25T13:38:45.363377+0000","last_unstale":"2024-04-25T13:40:00.362759+0000","last_undegraded":"2024-04-25T13:40:00.362759+0000","last_fullsized":"2024-04-25T13:40:00.362759+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:26.828215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365565+0000","last_change":"2024-04-25T13:38:45.359844+0000","last_active":"2024-04-25T13:40:00.365565+0000","last_peered":"2024-04-25T13:40:00.365565+0000","last_clean":"2024-04-25T13:40:00.365565+0000","last_became_active":"2024-04-25T13:38:45.359428+0000","last_became_peered":"2024-04-25T13:38:45.359428+0000","last_unstale":"2024-04-25T13:40:00.365565+0000","last_undegraded":"2024-04-25T13:40:00.365565+0000","last_fullsized":"2024-04-25T13:40:00.365565+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:01.370409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.359173+0000","last_change":"2024-04-25T13:38:45.370415+0000","last_active":"2024-04-25T13:40:00.359173+0000","last_peered":"2024-04-25T13:40:00.359173+0000","last_clean":"2024-04-25T13:40:00.359173+0000","last_became_active":"2024-04-25T13:38:45.369525+0000","last_became_peered":"2024-04-25T13:38:45.369525+0000","last_unstale":"2024-04-25T13:40:00.359173+0000","last_undegraded":"2024-04-25T13:40:00.359173+0000","last_fullsized":"2024-04-25T13:40:00.359173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:39.532380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.376082+0000","last_change":"2024-04-25T13:38:45.365309+0000","last_active":"2024-04-25T13:40:00.376082+0000","last_peered":"2024-04-25T13:40:00.376082+0000","last_clean":"2024-04-25T13:40:00.376082+0000","last_became_active":"2024-04-25T13:38:45.364893+0000","last_became_peered":"2024-04-25T13:38:45.364893+0000","last_unstale":"2024-04-25T13:40:00.376082+0000","last_undegraded":"2024-04-25T13:40:00.376082+0000","last_fullsized":"2024-04-25T13:40:00.376082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:04:54.454443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368064+0000","last_change":"2024-04-25T13:38:45.365150+0000","last_active":"2024-04-25T13:40:00.368064+0000","last_peered":"2024-04-25T13:40:00.368064+0000","last_clean":"2024-04-25T13:40:00.368064+0000","last_became_active":"2024-04-25T13:38:45.364649+0000","last_became_peered":"2024-04-25T13:38:45.364649+0000","last_unstale":"2024-04-25T13:40:00.368064+0000","last_undegraded":"2024-04-25T13:40:00.368064+0000","last_fullsized":"2024-04-25T13:40:00.368064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:25.035789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354459+0000","last_change":"2024-04-25T13:38:45.352108+0000","last_active":"2024-04-25T13:40:00.354459+0000","last_peered":"2024-04-25T13:40:00.354459+0000","last_clean":"2024-04-25T13:40:00.354459+0000","last_became_active":"2024-04-25T13:38:45.349887+0000","last_became_peered":"2024-04-25T13:38:45.349887+0000","last_unstale":"2024-04-25T13:40:00.354459+0000","last_undegraded":"2024-04-25T13:40:00.354459+0000","last_fullsized":"2024-04-25T13:40:00.354459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:09.339685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285228,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67774,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":262,"num_read_kb":199,"num_write":85,"num_write_kb":95,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":203322,"data_compressed":5526,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678055,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":403844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874284,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":1061499,"data_compressed":38103,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037810,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234039,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430268,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":888832,"data_stored":994911,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593794,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36220,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708932,"statfs":{"total":95995035648,"available":95957946368,"internally_reserved":0,"allocated":905216,"data_stored":470820,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":403934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":470854,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":843776,"data_stored":404065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790023,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":994511,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":66976,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67010,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67170,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T13:40:52.670 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T13:40:52.670 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T13:40:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.817+0000 7f4196e34640 1 -- 172.21.15.27:0/258096502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190109a40 msgr2=0x7f419010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.817+0000 7f4196e34640 1 --2- 172.21.15.27:0/258096502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190109a40 0x7f419010be60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4188009ca0 tx=0x7f418802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 -- 172.21.15.27:0/258096502 shutdown_connections 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 --2- 172.21.15.27:0/258096502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190109a40 0x7f419010be60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 --2- 172.21.15.27:0/258096502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4190104d40 0x7f4190105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 --2- 172.21.15.27:0/258096502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4190103b00 0x7f4190104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 -- 172.21.15.27:0/258096502 >> 172.21.15.27:0/258096502 conn(0x7f41900fed10 msgr2=0x7f41901011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 -- 172.21.15.27:0/258096502 shutdown_connections 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 -- 172.21.15.27:0/258096502 wait complete. 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 Processor -- start 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 -- start start 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4190103b00 0x7f419019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190104d40 0x7f419019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.818+0000 7f4196e34640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4190109a40 0x7f41901a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4196e34640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f419010f630 con 0x7f4190104d40 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4196e34640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f419010f4b0 con 0x7f4190103b00 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4196e34640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f419010f7b0 con 0x7f4190109a40 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190104d40 0x7f419019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190104d40 0x7f419019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54384/0 (socket says 172.21.15.27:54384) 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 -- 172.21.15.27:0/1708356996 learned_addr learned my addr 172.21.15.27:0/1708356996 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4194bd0640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4190103b00 0x7f419019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f41953d1640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4190109a40 0x7f41901a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 -- 172.21.15.27:0/1708356996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4190109a40 msgr2=0x7f41901a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4190109a40 0x7f41901a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 -- 172.21.15.27:0/1708356996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4190103b00 msgr2=0x7f419019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:52.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4190103b00 0x7f419019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:52.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 -- 172.21.15.27:0/1708356996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41901a1d50 con 0x7f4190104d40 2024-04-25T13:40:52.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4194bd0640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4190103b00 0x7f419019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:52.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4187fff640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190104d40 0x7f419019a750 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f418000e9f0 tx=0x7f418000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:52.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4185ffb640 1 -- 172.21.15.27:0/1708356996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418000cd80 con 0x7f4190104d40 2024-04-25T13:40:52.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4185ffb640 1 -- 172.21.15.27:0/1708356996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41800108e0 con 0x7f4190104d40 2024-04-25T13:40:52.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4185ffb640 1 -- 172.21.15.27:0/1708356996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180018980 con 0x7f4190104d40 2024-04-25T13:40:52.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.819+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41901a1fe0 con 0x7f4190104d40 2024-04-25T13:40:52.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.820+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41901a2520 con 0x7f4190104d40 2024-04-25T13:40:52.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.821+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4190108850 con 0x7f4190104d40 2024-04-25T13:40:52.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.822+0000 7f4185ffb640 1 -- 172.21.15.27:0/1708356996 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4180010430 con 0x7f4190104d40 2024-04-25T13:40:52.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.823+0000 7f4185ffb640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41780afb70 0x7f41780b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:52.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.823+0000 7f4194bd0640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41780afb70 0x7f41780b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:52.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.823+0000 7f4185ffb640 1 -- 172.21.15.27:0/1708356996 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f41800defa0 con 0x7f4190104d40 2024-04-25T13:40:52.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.823+0000 7f4194bd0640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41780afb70 0x7f41780b2080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4190104700 tx=0x7f418c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:52.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:52.827+0000 7f4185ffb640 1 -- 172.21.15.27:0/1708356996 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41800a7bb0 con 0x7f4190104d40 2024-04-25T13:40:53.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.010+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 --> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4190062fe0 con 0x7f41780afb70 2024-04-25T13:40:53.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.017+0000 7f4185ffb640 1 -- 172.21.15.27:0/1708356996 <== mgr.14152 v2:172.21.15.27:6800/1716439151 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258960 (secure 0 0 0) 0x7f4190062fe0 con 0x7f41780afb70 2024-04-25T13:40:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:53.021 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-25T13:40:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41780afb70 msgr2=0x7f41780b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41780afb70 0x7f41780b2080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4190104700 tx=0x7f418c005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190104d40 msgr2=0x7f419019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190104d40 0x7f419019a750 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f418000e9f0 tx=0x7f418000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 shutdown_connections 2024-04-25T13:40:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41780afb70 0x7f41780b2080 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4190109a40 0x7f41901a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4190104d40 0x7f419019a750 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 --2- 172.21.15.27:0/1708356996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4190103b00 0x7f419019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 >> 172.21.15.27:0/1708356996 conn(0x7f41900fed10 msgr2=0x7f4190105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 shutdown_connections 2024-04-25T13:40:53.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.020+0000 7f4196e34640 1 -- 172.21.15.27:0/1708356996 wait complete. 2024-04-25T13:40:53.034 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-25T13:40:52.381916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658376,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":338,"num_read_kb":263,"num_write":210,"num_write_kb":2247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951957,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":666,"ondisk_log_size":666,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375980,"kb_used_data":10188,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124565844,"statfs":{"total":1151940427776,"available":1151555424256,"internally_reserved":0,"allocated":10432512,"data_stored":6821256,"data_compressed":114309,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19138,"internal_metadata":374453566},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5681,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003253"},"pg_stats":[{"pgid":"3.3f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368465+0000","last_change":"2024-04-25T13:38:45.356552+0000","last_active":"2024-04-25T13:40:00.368465+0000","last_peered":"2024-04-25T13:40:00.368465+0000","last_clean":"2024-04-25T13:40:00.368465+0000","last_became_active":"2024-04-25T13:38:45.356122+0000","last_became_peered":"2024-04-25T13:38:45.356122+0000","last_unstale":"2024-04-25T13:40:00.368465+0000","last_undegraded":"2024-04-25T13:40:00.368465+0000","last_fullsized":"2024-04-25T13:40:00.368465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:55.877283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"78'4","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354435+0000","last_change":"2024-04-25T13:38:45.358577+0000","last_active":"2024-04-25T13:40:00.354435+0000","last_peered":"2024-04-25T13:40:00.354435+0000","last_clean":"2024-04-25T13:40:00.354435+0000","last_became_active":"2024-04-25T13:38:45.358122+0000","last_became_peered":"2024-04-25T13:38:45.358122+0000","last_unstale":"2024-04-25T13:40:00.354435+0000","last_undegraded":"2024-04-25T13:40:00.354435+0000","last_fullsized":"2024-04-25T13:40:00.354435+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:52.508459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"78'7","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368247+0000","last_change":"2024-04-25T13:38:45.362539+0000","last_active":"2024-04-25T13:40:00.368247+0000","last_peered":"2024-04-25T13:40:00.368247+0000","last_clean":"2024-04-25T13:40:00.368247+0000","last_became_active":"2024-04-25T13:38:45.361983+0000","last_became_peered":"2024-04-25T13:38:45.361983+0000","last_unstale":"2024-04-25T13:40:00.368247+0000","last_undegraded":"2024-04-25T13:40:00.368247+0000","last_fullsized":"2024-04-25T13:40:00.368247+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:13.267692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"78'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.337438+0000","last_change":"2024-04-25T13:38:45.360535+0000","last_active":"2024-04-25T13:40:00.337438+0000","last_peered":"2024-04-25T13:40:00.337438+0000","last_clean":"2024-04-25T13:40:00.337438+0000","last_became_active":"2024-04-25T13:38:45.357722+0000","last_became_peered":"2024-04-25T13:38:45.357722+0000","last_unstale":"2024-04-25T13:40:00.337438+0000","last_undegraded":"2024-04-25T13:40:00.337438+0000","last_fullsized":"2024-04-25T13:40:00.337438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:35.347434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365993+0000","last_change":"2024-04-25T13:38:45.353653+0000","last_active":"2024-04-25T13:40:00.365993+0000","last_peered":"2024-04-25T13:40:00.365993+0000","last_clean":"2024-04-25T13:40:00.365993+0000","last_became_active":"2024-04-25T13:38:45.353254+0000","last_became_peered":"2024-04-25T13:38:45.353254+0000","last_unstale":"2024-04-25T13:40:00.365993+0000","last_undegraded":"2024-04-25T13:40:00.365993+0000","last_fullsized":"2024-04-25T13:40:00.365993+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:57.275732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356027+0000","last_change":"2024-04-25T13:38:45.358241+0000","last_active":"2024-04-25T13:40:00.356027+0000","last_peered":"2024-04-25T13:40:00.356027+0000","last_clean":"2024-04-25T13:40:00.356027+0000","last_became_active":"2024-04-25T13:38:45.357843+0000","last_became_peered":"2024-04-25T13:38:45.357843+0000","last_unstale":"2024-04-25T13:40:00.356027+0000","last_undegraded":"2024-04-25T13:40:00.356027+0000","last_fullsized":"2024-04-25T13:40:00.356027+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:24.560357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375607+0000","last_change":"2024-04-25T13:38:45.350710+0000","last_active":"2024-04-25T13:40:00.375607+0000","last_peered":"2024-04-25T13:40:00.375607+0000","last_clean":"2024-04-25T13:40:00.375607+0000","last_became_active":"2024-04-25T13:38:45.350171+0000","last_became_peered":"2024-04-25T13:38:45.350171+0000","last_unstale":"2024-04-25T13:40:00.375607+0000","last_undegraded":"2024-04-25T13:40:00.375607+0000","last_fullsized":"2024-04-25T13:40:00.375607+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:24.933631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363691+0000","last_change":"2024-04-25T13:38:45.356221+0000","last_active":"2024-04-25T13:40:00.363691+0000","last_peered":"2024-04-25T13:40:00.363691+0000","last_clean":"2024-04-25T13:40:00.363691+0000","last_became_active":"2024-04-25T13:38:45.355823+0000","last_became_peered":"2024-04-25T13:38:45.355823+0000","last_unstale":"2024-04-25T13:40:00.363691+0000","last_undegraded":"2024-04-25T13:40:00.363691+0000","last_fullsized":"2024-04-25T13:40:00.363691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:25:02.040407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366879+0000","last_change":"2024-04-25T13:38:45.360940+0000","last_active":"2024-04-25T13:40:00.366879+0000","last_peered":"2024-04-25T13:40:00.366879+0000","last_clean":"2024-04-25T13:40:00.366879+0000","last_became_active":"2024-04-25T13:38:45.360241+0000","last_became_peered":"2024-04-25T13:38:45.360241+0000","last_unstale":"2024-04-25T13:40:00.366879+0000","last_undegraded":"2024-04-25T13:40:00.366879+0000","last_fullsized":"2024-04-25T13:40:00.366879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:05.251236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"78'16","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362862+0000","last_change":"2024-04-25T13:38:45.357908+0000","last_active":"2024-04-25T13:40:00.362862+0000","last_peered":"2024-04-25T13:40:00.362862+0000","last_clean":"2024-04-25T13:40:00.362862+0000","last_became_active":"2024-04-25T13:38:45.357491+0000","last_became_peered":"2024-04-25T13:38:45.357491+0000","last_unstale":"2024-04-25T13:40:00.362862+0000","last_undegraded":"2024-04-25T13:40:00.362862+0000","last_fullsized":"2024-04-25T13:40:00.362862+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:30.931611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374765+0000","last_change":"2024-04-25T13:38:45.368137+0000","last_active":"2024-04-25T13:40:00.374765+0000","last_peered":"2024-04-25T13:40:00.374765+0000","last_clean":"2024-04-25T13:40:00.374765+0000","last_became_active":"2024-04-25T13:38:45.367355+0000","last_became_peered":"2024-04-25T13:38:45.367355+0000","last_unstale":"2024-04-25T13:40:00.374765+0000","last_undegraded":"2024-04-25T13:40:00.374765+0000","last_fullsized":"2024-04-25T13:40:00.374765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:09.502056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"78'7","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349152+0000","last_change":"2024-04-25T13:39:32.041604+0000","last_active":"2024-04-25T13:40:00.349152+0000","last_peered":"2024-04-25T13:40:00.349152+0000","last_clean":"2024-04-25T13:40:00.349152+0000","last_became_active":"2024-04-25T13:39:32.041313+0000","last_became_peered":"2024-04-25T13:39:32.041313+0000","last_unstale":"2024-04-25T13:40:00.349152+0000","last_undegraded":"2024-04-25T13:40:00.349152+0000","last_fullsized":"2024-04-25T13:40:00.349152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:32.390348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357001+0000","last_change":"2024-04-25T13:38:45.353972+0000","last_active":"2024-04-25T13:40:00.357001+0000","last_peered":"2024-04-25T13:40:00.357001+0000","last_clean":"2024-04-25T13:40:00.357001+0000","last_became_active":"2024-04-25T13:38:45.353458+0000","last_became_peered":"2024-04-25T13:38:45.353458+0000","last_unstale":"2024-04-25T13:40:00.357001+0000","last_undegraded":"2024-04-25T13:40:00.357001+0000","last_fullsized":"2024-04-25T13:40:00.357001+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:17.087850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.347216+0000","last_change":"2024-04-25T13:38:45.358876+0000","last_active":"2024-04-25T13:40:00.347216+0000","last_peered":"2024-04-25T13:40:00.347216+0000","last_clean":"2024-04-25T13:40:00.347216+0000","last_became_active":"2024-04-25T13:38:45.358435+0000","last_became_peered":"2024-04-25T13:38:45.358435+0000","last_unstale":"2024-04-25T13:40:00.347216+0000","last_undegraded":"2024-04-25T13:40:00.347216+0000","last_fullsized":"2024-04-25T13:40:00.347216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:18:47.287112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362043+0000","last_change":"2024-04-25T13:38:45.373557+0000","last_active":"2024-04-25T13:40:00.362043+0000","last_peered":"2024-04-25T13:40:00.362043+0000","last_clean":"2024-04-25T13:40:00.362043+0000","last_became_active":"2024-04-25T13:38:45.373179+0000","last_became_peered":"2024-04-25T13:38:45.373179+0000","last_unstale":"2024-04-25T13:40:00.362043+0000","last_undegraded":"2024-04-25T13:40:00.362043+0000","last_fullsized":"2024-04-25T13:40:00.362043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:01.790266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.374896+0000","last_change":"2024-04-25T13:38:45.358282+0000","last_active":"2024-04-25T13:40:00.374896+0000","last_peered":"2024-04-25T13:40:00.374896+0000","last_clean":"2024-04-25T13:40:00.374896+0000","last_became_active":"2024-04-25T13:38:45.357923+0000","last_became_peered":"2024-04-25T13:38:45.357923+0000","last_unstale":"2024-04-25T13:40:00.374896+0000","last_undegraded":"2024-04-25T13:40:00.374896+0000","last_fullsized":"2024-04-25T13:40:00.374896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:47.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368746+0000","last_change":"2024-04-25T13:38:45.357865+0000","last_active":"2024-04-25T13:40:00.368746+0000","last_peered":"2024-04-25T13:40:00.368746+0000","last_clean":"2024-04-25T13:40:00.368746+0000","last_became_active":"2024-04-25T13:38:45.355399+0000","last_became_peered":"2024-04-25T13:38:45.355399+0000","last_unstale":"2024-04-25T13:40:00.368746+0000","last_undegraded":"2024-04-25T13:40:00.368746+0000","last_fullsized":"2024-04-25T13:40:00.368746+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:33:41.382462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349318+0000","last_change":"2024-04-25T13:38:45.350724+0000","last_active":"2024-04-25T13:40:00.349318+0000","last_peered":"2024-04-25T13:40:00.349318+0000","last_clean":"2024-04-25T13:40:00.349318+0000","last_became_active":"2024-04-25T13:38:45.350284+0000","last_became_peered":"2024-04-25T13:38:45.350284+0000","last_unstale":"2024-04-25T13:40:00.349318+0000","last_undegraded":"2024-04-25T13:40:00.349318+0000","last_fullsized":"2024-04-25T13:40:00.349318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:12:30.960843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351696+0000","last_change":"2024-04-25T13:38:45.362966+0000","last_active":"2024-04-25T13:40:00.351696+0000","last_peered":"2024-04-25T13:40:00.351696+0000","last_clean":"2024-04-25T13:40:00.351696+0000","last_became_active":"2024-04-25T13:38:45.362585+0000","last_became_peered":"2024-04-25T13:38:45.362585+0000","last_unstale":"2024-04-25T13:40:00.351696+0000","last_undegraded":"2024-04-25T13:40:00.351696+0000","last_fullsized":"2024-04-25T13:40:00.351696+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:46:10.825028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'4","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.488075+0000","last_change":"2024-04-25T13:38:44.354887+0000","last_active":"2024-04-25T13:39:31.488075+0000","last_peered":"2024-04-25T13:39:31.488075+0000","last_clean":"2024-04-25T13:39:31.488075+0000","last_became_active":"2024-04-25T13:38:44.354510+0000","last_became_peered":"2024-04-25T13:38:44.354510+0000","last_unstale":"2024-04-25T13:39:31.488075+0000","last_undegraded":"2024-04-25T13:39:31.488075+0000","last_fullsized":"2024-04-25T13:39:31.488075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:46.457087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"78'5","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363062+0000","last_change":"2024-04-25T13:38:45.364056+0000","last_active":"2024-04-25T13:40:00.363062+0000","last_peered":"2024-04-25T13:40:00.363062+0000","last_clean":"2024-04-25T13:40:00.363062+0000","last_became_active":"2024-04-25T13:38:45.363572+0000","last_became_peered":"2024-04-25T13:38:45.363572+0000","last_unstale":"2024-04-25T13:40:00.363062+0000","last_undegraded":"2024-04-25T13:40:00.363062+0000","last_fullsized":"2024-04-25T13:40:00.363062+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:49:12.136637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.490480+0000","last_change":"2024-04-25T13:38:44.362371+0000","last_active":"2024-04-25T13:39:31.490480+0000","last_peered":"2024-04-25T13:39:31.490480+0000","last_clean":"2024-04-25T13:39:31.490480+0000","last_became_active":"2024-04-25T13:38:44.361262+0000","last_became_peered":"2024-04-25T13:38:44.361262+0000","last_unstale":"2024-04-25T13:39:31.490480+0000","last_undegraded":"2024-04-25T13:39:31.490480+0000","last_fullsized":"2024-04-25T13:39:31.490480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:01.442189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"78'6","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367818+0000","last_change":"2024-04-25T13:38:45.362599+0000","last_active":"2024-04-25T13:40:00.367818+0000","last_peered":"2024-04-25T13:40:00.367818+0000","last_clean":"2024-04-25T13:40:00.367818+0000","last_became_active":"2024-04-25T13:38:45.361893+0000","last_became_peered":"2024-04-25T13:38:45.361893+0000","last_unstale":"2024-04-25T13:40:00.367818+0000","last_undegraded":"2024-04-25T13:40:00.367818+0000","last_fullsized":"2024-04-25T13:40:00.367818+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:52.414826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491185+0000","last_change":"2024-04-25T13:38:44.353063+0000","last_active":"2024-04-25T13:39:31.491185+0000","last_peered":"2024-04-25T13:39:31.491185+0000","last_clean":"2024-04-25T13:39:31.491185+0000","last_became_active":"2024-04-25T13:38:44.352403+0000","last_became_peered":"2024-04-25T13:38:44.352403+0000","last_unstale":"2024-04-25T13:39:31.491185+0000","last_undegraded":"2024-04-25T13:39:31.491185+0000","last_fullsized":"2024-04-25T13:39:31.491185+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:09.670589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369475+0000","last_change":"2024-04-25T13:38:45.359078+0000","last_active":"2024-04-25T13:40:00.369475+0000","last_peered":"2024-04-25T13:40:00.369475+0000","last_clean":"2024-04-25T13:40:00.369475+0000","last_became_active":"2024-04-25T13:38:45.354063+0000","last_became_peered":"2024-04-25T13:38:45.354063+0000","last_unstale":"2024-04-25T13:40:00.369475+0000","last_undegraded":"2024-04-25T13:40:00.369475+0000","last_fullsized":"2024-04-25T13:40:00.369475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:16.694340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492390+0000","last_change":"2024-04-25T13:38:44.349132+0000","last_active":"2024-04-25T13:39:31.492390+0000","last_peered":"2024-04-25T13:39:31.492390+0000","last_clean":"2024-04-25T13:39:31.492390+0000","last_became_active":"2024-04-25T13:38:44.348762+0000","last_became_peered":"2024-04-25T13:38:44.348762+0000","last_unstale":"2024-04-25T13:39:31.492390+0000","last_undegraded":"2024-04-25T13:39:31.492390+0000","last_fullsized":"2024-04-25T13:39:31.492390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:12.990304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368999+0000","last_change":"2024-04-25T13:38:45.369116+0000","last_active":"2024-04-25T13:40:00.368999+0000","last_peered":"2024-04-25T13:40:00.368999+0000","last_clean":"2024-04-25T13:40:00.368999+0000","last_became_active":"2024-04-25T13:38:45.368380+0000","last_became_peered":"2024-04-25T13:38:45.368380+0000","last_unstale":"2024-04-25T13:40:00.368999+0000","last_undegraded":"2024-04-25T13:40:00.368999+0000","last_fullsized":"2024-04-25T13:40:00.368999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:02.259401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485437+0000","last_change":"2024-04-25T13:38:44.351873+0000","last_active":"2024-04-25T13:39:31.485437+0000","last_peered":"2024-04-25T13:39:31.485437+0000","last_clean":"2024-04-25T13:39:31.485437+0000","last_became_active":"2024-04-25T13:38:44.351518+0000","last_became_peered":"2024-04-25T13:38:44.351518+0000","last_unstale":"2024-04-25T13:39:31.485437+0000","last_undegraded":"2024-04-25T13:39:31.485437+0000","last_fullsized":"2024-04-25T13:39:31.485437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:06.839053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342392+0000","last_change":"2024-04-25T13:38:45.351369+0000","last_active":"2024-04-25T13:40:00.342392+0000","last_peered":"2024-04-25T13:40:00.342392+0000","last_clean":"2024-04-25T13:40:00.342392+0000","last_became_active":"2024-04-25T13:38:45.351018+0000","last_became_peered":"2024-04-25T13:38:45.351018+0000","last_unstale":"2024-04-25T13:40:00.342392+0000","last_undegraded":"2024-04-25T13:40:00.342392+0000","last_fullsized":"2024-04-25T13:40:00.342392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:55.750407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040292+0000","last_change":"2024-04-25T13:38:44.345986+0000","last_active":"2024-04-25T13:39:32.040292+0000","last_peered":"2024-04-25T13:39:32.040292+0000","last_clean":"2024-04-25T13:39:32.040292+0000","last_became_active":"2024-04-25T13:38:44.345600+0000","last_became_peered":"2024-04-25T13:38:44.345600+0000","last_unstale":"2024-04-25T13:39:32.040292+0000","last_undegraded":"2024-04-25T13:39:32.040292+0000","last_fullsized":"2024-04-25T13:39:32.040292+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:52.749192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365960+0000","last_change":"2024-04-25T13:38:45.368641+0000","last_active":"2024-04-25T13:40:00.365960+0000","last_peered":"2024-04-25T13:40:00.365960+0000","last_clean":"2024-04-25T13:40:00.365960+0000","last_became_active":"2024-04-25T13:38:45.368209+0000","last_became_peered":"2024-04-25T13:38:45.368209+0000","last_unstale":"2024-04-25T13:40:00.365960+0000","last_undegraded":"2024-04-25T13:40:00.365960+0000","last_fullsized":"2024-04-25T13:40:00.365960+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:47.990808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484783+0000","last_change":"2024-04-25T13:38:44.339927+0000","last_active":"2024-04-25T13:39:31.484783+0000","last_peered":"2024-04-25T13:39:31.484783+0000","last_clean":"2024-04-25T13:39:31.484783+0000","last_became_active":"2024-04-25T13:38:44.339588+0000","last_became_peered":"2024-04-25T13:38:44.339588+0000","last_unstale":"2024-04-25T13:39:31.484783+0000","last_undegraded":"2024-04-25T13:39:31.484783+0000","last_fullsized":"2024-04-25T13:39:31.484783+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:51.707750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365330+0000","last_change":"2024-04-25T13:38:45.357323+0000","last_active":"2024-04-25T13:40:00.365330+0000","last_peered":"2024-04-25T13:40:00.365330+0000","last_clean":"2024-04-25T13:40:00.365330+0000","last_became_active":"2024-04-25T13:38:45.356870+0000","last_became_peered":"2024-04-25T13:38:45.356870+0000","last_unstale":"2024-04-25T13:40:00.365330+0000","last_undegraded":"2024-04-25T13:40:00.365330+0000","last_fullsized":"2024-04-25T13:40:00.365330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:36.838709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485179+0000","last_change":"2024-04-25T13:38:44.351294+0000","last_active":"2024-04-25T13:39:31.485179+0000","last_peered":"2024-04-25T13:39:31.485179+0000","last_clean":"2024-04-25T13:39:31.485179+0000","last_became_active":"2024-04-25T13:38:44.350971+0000","last_became_peered":"2024-04-25T13:38:44.350971+0000","last_unstale":"2024-04-25T13:39:31.485179+0000","last_undegraded":"2024-04-25T13:39:31.485179+0000","last_fullsized":"2024-04-25T13:39:31.485179+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:22.452217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.349813+0000","last_change":"2024-04-25T13:38:45.345943+0000","last_active":"2024-04-25T13:40:00.349813+0000","last_peered":"2024-04-25T13:40:00.349813+0000","last_clean":"2024-04-25T13:40:00.349813+0000","last_became_active":"2024-04-25T13:38:45.345528+0000","last_became_peered":"2024-04-25T13:38:45.345528+0000","last_unstale":"2024-04-25T13:40:00.349813+0000","last_undegraded":"2024-04-25T13:40:00.349813+0000","last_fullsized":"2024-04-25T13:40:00.349813+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:57.327303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:32.040662+0000","last_change":"2024-04-25T13:38:44.343032+0000","last_active":"2024-04-25T13:39:32.040662+0000","last_peered":"2024-04-25T13:39:32.040662+0000","last_clean":"2024-04-25T13:39:32.040662+0000","last_became_active":"2024-04-25T13:38:44.342613+0000","last_became_peered":"2024-04-25T13:38:44.342613+0000","last_unstale":"2024-04-25T13:39:32.040662+0000","last_undegraded":"2024-04-25T13:39:32.040662+0000","last_fullsized":"2024-04-25T13:39:32.040662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:33.957150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.369787+0000","last_change":"2024-04-25T13:38:45.354534+0000","last_active":"2024-04-25T13:40:00.369787+0000","last_peered":"2024-04-25T13:40:00.369787+0000","last_clean":"2024-04-25T13:40:00.369787+0000","last_became_active":"2024-04-25T13:38:45.354147+0000","last_became_peered":"2024-04-25T13:38:45.354147+0000","last_unstale":"2024-04-25T13:40:00.369787+0000","last_undegraded":"2024-04-25T13:40:00.369787+0000","last_fullsized":"2024-04-25T13:40:00.369787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:31.537339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487809+0000","last_change":"2024-04-25T13:38:44.356103+0000","last_active":"2024-04-25T13:39:31.487809+0000","last_peered":"2024-04-25T13:39:31.487809+0000","last_clean":"2024-04-25T13:39:31.487809+0000","last_became_active":"2024-04-25T13:38:44.355702+0000","last_became_peered":"2024-04-25T13:38:44.355702+0000","last_unstale":"2024-04-25T13:39:31.487809+0000","last_undegraded":"2024-04-25T13:39:31.487809+0000","last_fullsized":"2024-04-25T13:39:31.487809+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:47.572922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"78'9","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.344967+0000","last_change":"2024-04-25T13:39:31.494580+0000","last_active":"2024-04-25T13:40:00.344967+0000","last_peered":"2024-04-25T13:40:00.344967+0000","last_clean":"2024-04-25T13:40:00.344967+0000","last_became_active":"2024-04-25T13:39:31.494102+0000","last_became_peered":"2024-04-25T13:39:31.494102+0000","last_unstale":"2024-04-25T13:40:00.344967+0000","last_undegraded":"2024-04-25T13:40:00.344967+0000","last_fullsized":"2024-04-25T13:40:00.344967+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.056136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485841+0000","last_change":"2024-04-25T13:38:44.355474+0000","last_active":"2024-04-25T13:39:31.485841+0000","last_peered":"2024-04-25T13:39:31.485841+0000","last_clean":"2024-04-25T13:39:31.485841+0000","last_became_active":"2024-04-25T13:38:44.355113+0000","last_became_peered":"2024-04-25T13:38:44.355113+0000","last_unstale":"2024-04-25T13:39:31.485841+0000","last_undegraded":"2024-04-25T13:39:31.485841+0000","last_fullsized":"2024-04-25T13:39:31.485841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.242974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.370871+0000","last_change":"2024-04-25T13:38:45.369618+0000","last_active":"2024-04-25T13:40:00.370871+0000","last_peered":"2024-04-25T13:40:00.370871+0000","last_clean":"2024-04-25T13:40:00.370871+0000","last_became_active":"2024-04-25T13:38:45.369252+0000","last_became_peered":"2024-04-25T13:38:45.369252+0000","last_unstale":"2024-04-25T13:40:00.370871+0000","last_undegraded":"2024-04-25T13:40:00.370871+0000","last_fullsized":"2024-04-25T13:40:00.370871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:40.336573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"78'32","reported_seq":307,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:50.608632+0000","last_change":"2024-04-25T13:38:44.349850+0000","last_active":"2024-04-25T13:40:50.608632+0000","last_peered":"2024-04-25T13:40:50.608632+0000","last_clean":"2024-04-25T13:40:50.608632+0000","last_became_active":"2024-04-25T13:38:44.349404+0000","last_became_peered":"2024-04-25T13:38:44.349404+0000","last_unstale":"2024-04-25T13:40:50.608632+0000","last_undegraded":"2024-04-25T13:40:50.608632+0000","last_fullsized":"2024-04-25T13:40:50.608632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:40.025684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66976,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":256,"num_read_kb":193,"num_write":32,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.348329+0000","last_change":"2024-04-25T13:38:45.371554+0000","last_active":"2024-04-25T13:40:00.348329+0000","last_peered":"2024-04-25T13:40:00.348329+0000","last_clean":"2024-04-25T13:40:00.348329+0000","last_became_active":"2024-04-25T13:38:45.371042+0000","last_became_peered":"2024-04-25T13:38:45.371042+0000","last_unstale":"2024-04-25T13:40:00.348329+0000","last_undegraded":"2024-04-25T13:40:00.348329+0000","last_fullsized":"2024-04-25T13:40:00.348329+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:59.240536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494618+0000","last_change":"2024-04-25T13:38:44.345979+0000","last_active":"2024-04-25T13:39:31.494618+0000","last_peered":"2024-04-25T13:39:31.494618+0000","last_clean":"2024-04-25T13:39:31.494618+0000","last_became_active":"2024-04-25T13:38:44.345596+0000","last_became_peered":"2024-04-25T13:38:44.345596+0000","last_unstale":"2024-04-25T13:39:31.494618+0000","last_undegraded":"2024-04-25T13:39:31.494618+0000","last_fullsized":"2024-04-25T13:39:31.494618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:22:51.539024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.367445+0000","last_change":"2024-04-25T13:38:45.364794+0000","last_active":"2024-04-25T13:40:00.367445+0000","last_peered":"2024-04-25T13:40:00.367445+0000","last_clean":"2024-04-25T13:40:00.367445+0000","last_became_active":"2024-04-25T13:38:45.364381+0000","last_became_peered":"2024-04-25T13:38:45.364381+0000","last_unstale":"2024-04-25T13:40:00.367445+0000","last_undegraded":"2024-04-25T13:40:00.367445+0000","last_fullsized":"2024-04-25T13:40:00.367445+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:16.856980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485286+0000","last_change":"2024-04-25T13:38:44.351269+0000","last_active":"2024-04-25T13:39:31.485286+0000","last_peered":"2024-04-25T13:39:31.485286+0000","last_clean":"2024-04-25T13:39:31.485286+0000","last_became_active":"2024-04-25T13:38:44.350933+0000","last_became_peered":"2024-04-25T13:38:44.350933+0000","last_unstale":"2024-04-25T13:39:31.485286+0000","last_undegraded":"2024-04-25T13:39:31.485286+0000","last_fullsized":"2024-04-25T13:39:31.485286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:02.677849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.372710+0000","last_change":"2024-04-25T13:38:45.362732+0000","last_active":"2024-04-25T13:40:00.372710+0000","last_peered":"2024-04-25T13:40:00.372710+0000","last_clean":"2024-04-25T13:40:00.372710+0000","last_became_active":"2024-04-25T13:38:45.361957+0000","last_became_peered":"2024-04-25T13:38:45.361957+0000","last_unstale":"2024-04-25T13:40:00.372710+0000","last_undegraded":"2024-04-25T13:40:00.372710+0000","last_fullsized":"2024-04-25T13:40:00.372710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:50.011442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.776862+0000","last_change":"2024-04-25T13:38:44.349378+0000","last_active":"2024-04-25T13:39:31.776862+0000","last_peered":"2024-04-25T13:39:31.776862+0000","last_clean":"2024-04-25T13:39:31.776862+0000","last_became_active":"2024-04-25T13:38:44.348887+0000","last_became_peered":"2024-04-25T13:38:44.348887+0000","last_unstale":"2024-04-25T13:39:31.776862+0000","last_undegraded":"2024-04-25T13:39:31.776862+0000","last_fullsized":"2024-04-25T13:39:31.776862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.747205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375453+0000","last_change":"2024-04-25T13:38:45.349941+0000","last_active":"2024-04-25T13:40:00.375453+0000","last_peered":"2024-04-25T13:40:00.375453+0000","last_clean":"2024-04-25T13:40:00.375453+0000","last_became_active":"2024-04-25T13:38:45.349590+0000","last_became_peered":"2024-04-25T13:38:45.349590+0000","last_unstale":"2024-04-25T13:40:00.375453+0000","last_undegraded":"2024-04-25T13:40:00.375453+0000","last_fullsized":"2024-04-25T13:40:00.375453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:42.904406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.791483+0000","last_change":"2024-04-25T13:39:31.790997+0000","last_active":"2024-04-25T13:39:31.791483+0000","last_peered":"2024-04-25T13:39:31.791483+0000","last_clean":"2024-04-25T13:39:31.791483+0000","last_became_active":"2024-04-25T13:39:31.778451+0000","last_became_peered":"2024-04-25T13:39:31.778451+0000","last_unstale":"2024-04-25T13:39:31.791483+0000","last_undegraded":"2024-04-25T13:39:31.791483+0000","last_fullsized":"2024-04-25T13:39:31.791483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:17.735970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368028+0000","last_change":"2024-04-25T13:38:45.368589+0000","last_active":"2024-04-25T13:40:00.368028+0000","last_peered":"2024-04-25T13:40:00.368028+0000","last_clean":"2024-04-25T13:40:00.368028+0000","last_became_active":"2024-04-25T13:38:45.368079+0000","last_became_peered":"2024-04-25T13:38:45.368079+0000","last_unstale":"2024-04-25T13:40:00.368028+0000","last_undegraded":"2024-04-25T13:40:00.368028+0000","last_fullsized":"2024-04-25T13:40:00.368028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:10.180264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.491662+0000","last_change":"2024-04-25T13:38:44.345148+0000","last_active":"2024-04-25T13:39:31.491662+0000","last_peered":"2024-04-25T13:39:31.491662+0000","last_clean":"2024-04-25T13:39:31.491662+0000","last_became_active":"2024-04-25T13:38:44.344816+0000","last_became_peered":"2024-04-25T13:38:44.344816+0000","last_unstale":"2024-04-25T13:39:31.491662+0000","last_undegraded":"2024-04-25T13:39:31.491662+0000","last_fullsized":"2024-04-25T13:39:31.491662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:03:38.134169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.351253+0000","last_change":"2024-04-25T13:38:45.354340+0000","last_active":"2024-04-25T13:40:00.351253+0000","last_peered":"2024-04-25T13:40:00.351253+0000","last_clean":"2024-04-25T13:40:00.351253+0000","last_became_active":"2024-04-25T13:38:45.353933+0000","last_became_peered":"2024-04-25T13:38:45.353933+0000","last_unstale":"2024-04-25T13:40:00.351253+0000","last_undegraded":"2024-04-25T13:40:00.351253+0000","last_fullsized":"2024-04-25T13:40:00.351253+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:07.619051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.488024+0000","last_change":"2024-04-25T13:38:44.351183+0000","last_active":"2024-04-25T13:39:31.488024+0000","last_peered":"2024-04-25T13:39:31.488024+0000","last_clean":"2024-04-25T13:39:31.488024+0000","last_became_active":"2024-04-25T13:38:44.350785+0000","last_became_peered":"2024-04-25T13:38:44.350785+0000","last_unstale":"2024-04-25T13:39:31.488024+0000","last_undegraded":"2024-04-25T13:39:31.488024+0000","last_fullsized":"2024-04-25T13:39:31.488024+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:37:57.465012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"78'5","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360834+0000","last_change":"2024-04-25T13:38:45.362355+0000","last_active":"2024-04-25T13:40:00.360834+0000","last_peered":"2024-04-25T13:40:00.360834+0000","last_clean":"2024-04-25T13:40:00.360834+0000","last_became_active":"2024-04-25T13:38:45.361996+0000","last_became_peered":"2024-04-25T13:38:45.361996+0000","last_unstale":"2024-04-25T13:40:00.360834+0000","last_undegraded":"2024-04-25T13:40:00.360834+0000","last_fullsized":"2024-04-25T13:40:00.360834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:53:24.790450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487641+0000","last_change":"2024-04-25T13:38:44.356079+0000","last_active":"2024-04-25T13:39:31.487641+0000","last_peered":"2024-04-25T13:39:31.487641+0000","last_clean":"2024-04-25T13:39:31.487641+0000","last_became_active":"2024-04-25T13:38:44.355660+0000","last_became_peered":"2024-04-25T13:38:44.355660+0000","last_unstale":"2024-04-25T13:39:31.487641+0000","last_undegraded":"2024-04-25T13:39:31.487641+0000","last_fullsized":"2024-04-25T13:39:31.487641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:35.161337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"78'7","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363216+0000","last_change":"2024-04-25T13:38:45.356578+0000","last_active":"2024-04-25T13:40:00.363216+0000","last_peered":"2024-04-25T13:40:00.363216+0000","last_clean":"2024-04-25T13:40:00.363216+0000","last_became_active":"2024-04-25T13:38:45.356232+0000","last_became_peered":"2024-04-25T13:38:45.356232+0000","last_unstale":"2024-04-25T13:40:00.363216+0000","last_undegraded":"2024-04-25T13:40:00.363216+0000","last_fullsized":"2024-04-25T13:40:00.363216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.476890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"78'8","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.360561+0000","last_change":"2024-04-25T13:39:31.778577+0000","last_active":"2024-04-25T13:40:00.360561+0000","last_peered":"2024-04-25T13:40:00.360561+0000","last_clean":"2024-04-25T13:40:00.360561+0000","last_became_active":"2024-04-25T13:39:31.778178+0000","last_became_peered":"2024-04-25T13:39:31.778178+0000","last_unstale":"2024-04-25T13:40:00.360561+0000","last_undegraded":"2024-04-25T13:40:00.360561+0000","last_fullsized":"2024-04-25T13:40:00.360561+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:16.383670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485143+0000","last_change":"2024-04-25T13:38:44.351113+0000","last_active":"2024-04-25T13:39:31.485143+0000","last_peered":"2024-04-25T13:39:31.485143+0000","last_clean":"2024-04-25T13:39:31.485143+0000","last_became_active":"2024-04-25T13:38:44.350730+0000","last_became_peered":"2024-04-25T13:38:44.350730+0000","last_unstale":"2024-04-25T13:39:31.485143+0000","last_undegraded":"2024-04-25T13:39:31.485143+0000","last_fullsized":"2024-04-25T13:39:31.485143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:38:22.747930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"78'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362961+0000","last_change":"2024-04-25T13:38:45.356731+0000","last_active":"2024-04-25T13:40:00.362961+0000","last_peered":"2024-04-25T13:40:00.362961+0000","last_clean":"2024-04-25T13:40:00.362961+0000","last_became_active":"2024-04-25T13:38:45.355531+0000","last_became_peered":"2024-04-25T13:38:45.355531+0000","last_unstale":"2024-04-25T13:40:00.362961+0000","last_undegraded":"2024-04-25T13:40:00.362961+0000","last_fullsized":"2024-04-25T13:40:00.362961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:15.482469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777207+0000","last_change":"2024-04-25T13:38:44.345941+0000","last_active":"2024-04-25T13:39:31.777207+0000","last_peered":"2024-04-25T13:39:31.777207+0000","last_clean":"2024-04-25T13:39:31.777207+0000","last_became_active":"2024-04-25T13:38:44.344634+0000","last_became_peered":"2024-04-25T13:38:44.344634+0000","last_unstale":"2024-04-25T13:39:31.777207+0000","last_undegraded":"2024-04-25T13:39:31.777207+0000","last_fullsized":"2024-04-25T13:39:31.777207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:42.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":94,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487340+0000","last_change":"2024-04-25T13:37:27.903345+0000","last_active":"2024-04-25T13:39:31.487340+0000","last_peered":"2024-04-25T13:39:31.487340+0000","last_clean":"2024-04-25T13:39:31.487340+0000","last_became_active":"2024-04-25T13:37:27.897986+0000","last_became_peered":"2024-04-25T13:37:27.897986+0000","last_unstale":"2024-04-25T13:39:31.487340+0000","last_undegraded":"2024-04-25T13:39:31.487340+0000","last_fullsized":"2024-04-25T13:39:31.487340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:33:56.569417+0000","last_clean_scrub_stamp":"2024-04-25T13:33:56.569417+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:56.390458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"78'5","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.364075+0000","last_change":"2024-04-25T13:38:45.369390+0000","last_active":"2024-04-25T13:40:00.364075+0000","last_peered":"2024-04-25T13:40:00.364075+0000","last_clean":"2024-04-25T13:40:00.364075+0000","last_became_active":"2024-04-25T13:38:45.368938+0000","last_became_peered":"2024-04-25T13:38:45.368938+0000","last_unstale":"2024-04-25T13:40:00.364075+0000","last_undegraded":"2024-04-25T13:40:00.364075+0000","last_fullsized":"2024-04-25T13:40:00.364075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:11.942204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.487500+0000","last_change":"2024-04-25T13:38:44.355585+0000","last_active":"2024-04-25T13:39:31.487500+0000","last_peered":"2024-04-25T13:39:31.487500+0000","last_clean":"2024-04-25T13:39:31.487500+0000","last_became_active":"2024-04-25T13:38:44.355131+0000","last_became_peered":"2024-04-25T13:38:44.355131+0000","last_unstale":"2024-04-25T13:39:31.487500+0000","last_undegraded":"2024-04-25T13:39:31.487500+0000","last_fullsized":"2024-04-25T13:39:31.487500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:05.240247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.492418+0000","last_change":"2024-04-25T13:39:31.491742+0000","last_active":"2024-04-25T13:39:31.492418+0000","last_peered":"2024-04-25T13:39:31.492418+0000","last_clean":"2024-04-25T13:39:31.492418+0000","last_became_active":"2024-04-25T13:39:31.491266+0000","last_became_peered":"2024-04-25T13:39:31.491266+0000","last_unstale":"2024-04-25T13:39:31.492418+0000","last_undegraded":"2024-04-25T13:39:31.492418+0000","last_fullsized":"2024-04-25T13:39:31.492418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:21.932333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"78'9","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356400+0000","last_change":"2024-04-25T13:38:45.366187+0000","last_active":"2024-04-25T13:40:00.356400+0000","last_peered":"2024-04-25T13:40:00.356400+0000","last_clean":"2024-04-25T13:40:00.356400+0000","last_became_active":"2024-04-25T13:38:45.365744+0000","last_became_peered":"2024-04-25T13:38:45.365744+0000","last_unstale":"2024-04-25T13:40:00.356400+0000","last_undegraded":"2024-04-25T13:40:00.356400+0000","last_fullsized":"2024-04-25T13:40:00.356400+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:07.970735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"78'8","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.378057+0000","last_change":"2024-04-25T13:38:44.359250+0000","last_active":"2024-04-25T13:40:00.378057+0000","last_peered":"2024-04-25T13:40:00.378057+0000","last_clean":"2024-04-25T13:40:00.378057+0000","last_became_active":"2024-04-25T13:38:44.358874+0000","last_became_peered":"2024-04-25T13:38:44.358874+0000","last_unstale":"2024-04-25T13:40:00.378057+0000","last_undegraded":"2024-04-25T13:40:00.378057+0000","last_fullsized":"2024-04-25T13:40:00.378057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:28.820465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.366411+0000","last_change":"2024-04-25T13:38:45.370080+0000","last_active":"2024-04-25T13:40:00.366411+0000","last_peered":"2024-04-25T13:40:00.366411+0000","last_clean":"2024-04-25T13:40:00.366411+0000","last_became_active":"2024-04-25T13:38:45.369668+0000","last_became_peered":"2024-04-25T13:38:45.369668+0000","last_unstale":"2024-04-25T13:40:00.366411+0000","last_undegraded":"2024-04-25T13:40:00.366411+0000","last_fullsized":"2024-04-25T13:40:00.366411+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:03.197468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.777370+0000","last_change":"2024-04-25T13:38:44.358907+0000","last_active":"2024-04-25T13:39:31.777370+0000","last_peered":"2024-04-25T13:39:31.777370+0000","last_clean":"2024-04-25T13:39:31.777370+0000","last_became_active":"2024-04-25T13:38:44.358455+0000","last_became_peered":"2024-04-25T13:38:44.358455+0000","last_unstale":"2024-04-25T13:39:31.777370+0000","last_undegraded":"2024-04-25T13:39:31.777370+0000","last_fullsized":"2024-04-25T13:39:31.777370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:50:52.916433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365642+0000","last_change":"2024-04-25T13:38:45.373050+0000","last_active":"2024-04-25T13:40:00.365642+0000","last_peered":"2024-04-25T13:40:00.365642+0000","last_clean":"2024-04-25T13:40:00.365642+0000","last_became_active":"2024-04-25T13:38:45.372621+0000","last_became_peered":"2024-04-25T13:38:45.372621+0000","last_unstale":"2024-04-25T13:40:00.365642+0000","last_undegraded":"2024-04-25T13:40:00.365642+0000","last_fullsized":"2024-04-25T13:40:00.365642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:29:43.259296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485534+0000","last_change":"2024-04-25T13:38:44.355768+0000","last_active":"2024-04-25T13:39:31.485534+0000","last_peered":"2024-04-25T13:39:31.485534+0000","last_clean":"2024-04-25T13:39:31.485534+0000","last_became_active":"2024-04-25T13:38:44.355400+0000","last_became_peered":"2024-04-25T13:38:44.355400+0000","last_unstale":"2024-04-25T13:39:31.485534+0000","last_undegraded":"2024-04-25T13:39:31.485534+0000","last_fullsized":"2024-04-25T13:39:31.485534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:38.533717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.342472+0000","last_change":"2024-04-25T13:38:45.350582+0000","last_active":"2024-04-25T13:40:00.342472+0000","last_peered":"2024-04-25T13:40:00.342472+0000","last_clean":"2024-04-25T13:40:00.342472+0000","last_became_active":"2024-04-25T13:38:45.350220+0000","last_became_peered":"2024-04-25T13:38:45.350220+0000","last_unstale":"2024-04-25T13:40:00.342472+0000","last_undegraded":"2024-04-25T13:40:00.342472+0000","last_fullsized":"2024-04-25T13:40:00.342472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.463519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.489729+0000","last_change":"2024-04-25T13:38:44.360459+0000","last_active":"2024-04-25T13:39:31.489729+0000","last_peered":"2024-04-25T13:39:31.489729+0000","last_clean":"2024-04-25T13:39:31.489729+0000","last_became_active":"2024-04-25T13:38:44.359709+0000","last_became_peered":"2024-04-25T13:38:44.359709+0000","last_unstale":"2024-04-25T13:39:31.489729+0000","last_undegraded":"2024-04-25T13:39:31.489729+0000","last_fullsized":"2024-04-25T13:39:31.489729+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:53.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356628+0000","last_change":"2024-04-25T13:38:45.357291+0000","last_active":"2024-04-25T13:40:00.356628+0000","last_peered":"2024-04-25T13:40:00.356628+0000","last_clean":"2024-04-25T13:40:00.356628+0000","last_became_active":"2024-04-25T13:38:45.356929+0000","last_became_peered":"2024-04-25T13:38:45.356929+0000","last_unstale":"2024-04-25T13:40:00.356628+0000","last_undegraded":"2024-04-25T13:40:00.356628+0000","last_fullsized":"2024-04-25T13:40:00.356628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:02:08.619208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.493732+0000","last_change":"2024-04-25T13:38:44.343885+0000","last_active":"2024-04-25T13:39:31.493732+0000","last_peered":"2024-04-25T13:39:31.493732+0000","last_clean":"2024-04-25T13:39:31.493732+0000","last_became_active":"2024-04-25T13:38:44.343541+0000","last_became_peered":"2024-04-25T13:38:44.343541+0000","last_unstale":"2024-04-25T13:39:31.493732+0000","last_undegraded":"2024-04-25T13:39:31.493732+0000","last_fullsized":"2024-04-25T13:39:31.493732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:50:01.904593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365216+0000","last_change":"2024-04-25T13:38:45.371508+0000","last_active":"2024-04-25T13:40:00.365216+0000","last_peered":"2024-04-25T13:40:00.365216+0000","last_clean":"2024-04-25T13:40:00.365216+0000","last_became_active":"2024-04-25T13:38:45.370912+0000","last_became_peered":"2024-04-25T13:38:45.370912+0000","last_unstale":"2024-04-25T13:40:00.365216+0000","last_undegraded":"2024-04-25T13:40:00.365216+0000","last_fullsized":"2024-04-25T13:40:00.365216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:07.075511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'2","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.498142+0000","last_change":"2024-04-25T13:39:31.498142+0000","last_active":"2024-04-25T13:39:31.498142+0000","last_peered":"2024-04-25T13:39:31.498142+0000","last_clean":"2024-04-25T13:39:31.498142+0000","last_became_active":"2024-04-25T13:39:31.494646+0000","last_became_peered":"2024-04-25T13:39:31.494646+0000","last_unstale":"2024-04-25T13:39:31.498142+0000","last_undegraded":"2024-04-25T13:39:31.498142+0000","last_fullsized":"2024-04-25T13:39:31.498142+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:46.084354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375168+0000","last_change":"2024-04-25T13:38:45.363322+0000","last_active":"2024-04-25T13:40:00.375168+0000","last_peered":"2024-04-25T13:40:00.375168+0000","last_clean":"2024-04-25T13:40:00.375168+0000","last_became_active":"2024-04-25T13:38:45.362948+0000","last_became_peered":"2024-04-25T13:38:45.362948+0000","last_unstale":"2024-04-25T13:40:00.375168+0000","last_undegraded":"2024-04-25T13:40:00.375168+0000","last_fullsized":"2024-04-25T13:40:00.375168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:20.815774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.494630+0000","last_change":"2024-04-25T13:38:44.358588+0000","last_active":"2024-04-25T13:39:31.494630+0000","last_peered":"2024-04-25T13:39:31.494630+0000","last_clean":"2024-04-25T13:39:31.494630+0000","last_became_active":"2024-04-25T13:38:44.358269+0000","last_became_peered":"2024-04-25T13:38:44.358269+0000","last_unstale":"2024-04-25T13:39:31.494630+0000","last_undegraded":"2024-04-25T13:39:31.494630+0000","last_fullsized":"2024-04-25T13:39:31.494630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:35.405572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375021+0000","last_change":"2024-04-25T13:38:45.369272+0000","last_active":"2024-04-25T13:40:00.375021+0000","last_peered":"2024-04-25T13:40:00.375021+0000","last_clean":"2024-04-25T13:40:00.375021+0000","last_became_active":"2024-04-25T13:38:45.368774+0000","last_became_peered":"2024-04-25T13:38:45.368774+0000","last_unstale":"2024-04-25T13:40:00.375021+0000","last_undegraded":"2024-04-25T13:40:00.375021+0000","last_fullsized":"2024-04-25T13:40:00.375021+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:39.149557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.485416+0000","last_change":"2024-04-25T13:38:44.349479+0000","last_active":"2024-04-25T13:39:31.485416+0000","last_peered":"2024-04-25T13:39:31.485416+0000","last_clean":"2024-04-25T13:39:31.485416+0000","last_became_active":"2024-04-25T13:38:44.349104+0000","last_became_peered":"2024-04-25T13:38:44.349104+0000","last_unstale":"2024-04-25T13:39:31.485416+0000","last_undegraded":"2024-04-25T13:39:31.485416+0000","last_fullsized":"2024-04-25T13:39:31.485416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:32:39.257472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"78'7","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.375501+0000","last_change":"2024-04-25T13:39:31.495349+0000","last_active":"2024-04-25T13:40:00.375501+0000","last_peered":"2024-04-25T13:40:00.375501+0000","last_clean":"2024-04-25T13:40:00.375501+0000","last_became_active":"2024-04-25T13:39:31.490741+0000","last_became_peered":"2024-04-25T13:39:31.490741+0000","last_unstale":"2024-04-25T13:40:00.375501+0000","last_undegraded":"2024-04-25T13:40:00.375501+0000","last_fullsized":"2024-04-25T13:40:00.375501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.474725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:39:31.484923+0000","last_change":"2024-04-25T13:38:44.349502+0000","last_active":"2024-04-25T13:39:31.484923+0000","last_peered":"2024-04-25T13:39:31.484923+0000","last_clean":"2024-04-25T13:39:31.484923+0000","last_became_active":"2024-04-25T13:38:44.349127+0000","last_became_peered":"2024-04-25T13:38:44.349127+0000","last_unstale":"2024-04-25T13:39:31.484923+0000","last_undegraded":"2024-04-25T13:39:31.484923+0000","last_fullsized":"2024-04-25T13:39:31.484923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:43.321589+0000","last_clean_scrub_stamp":"2024-04-25T13:38:43.321589+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:34:50.673352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.371319+0000","last_change":"2024-04-25T13:38:45.364746+0000","last_active":"2024-04-25T13:40:00.371319+0000","last_peered":"2024-04-25T13:40:00.371319+0000","last_clean":"2024-04-25T13:40:00.371319+0000","last_became_active":"2024-04-25T13:38:45.364364+0000","last_became_peered":"2024-04-25T13:38:45.364364+0000","last_unstale":"2024-04-25T13:40:00.371319+0000","last_undegraded":"2024-04-25T13:40:00.371319+0000","last_fullsized":"2024-04-25T13:40:00.371319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:42.145637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.357194+0000","last_change":"2024-04-25T13:38:45.356683+0000","last_active":"2024-04-25T13:40:00.357194+0000","last_peered":"2024-04-25T13:40:00.357194+0000","last_clean":"2024-04-25T13:40:00.357194+0000","last_became_active":"2024-04-25T13:38:45.356048+0000","last_became_peered":"2024-04-25T13:38:45.356048+0000","last_unstale":"2024-04-25T13:40:00.357194+0000","last_undegraded":"2024-04-25T13:40:00.357194+0000","last_fullsized":"2024-04-25T13:40:00.357194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:49.578317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"78'12","reported_seq":30,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.363111+0000","last_change":"2024-04-25T13:38:45.358298+0000","last_active":"2024-04-25T13:40:00.363111+0000","last_peered":"2024-04-25T13:40:00.363111+0000","last_clean":"2024-04-25T13:40:00.363111+0000","last_became_active":"2024-04-25T13:38:45.357986+0000","last_became_peered":"2024-04-25T13:38:45.357986+0000","last_unstale":"2024-04-25T13:40:00.363111+0000","last_undegraded":"2024-04-25T13:40:00.363111+0000","last_fullsized":"2024-04-25T13:40:00.363111+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:12:33.093156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"78'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.356232+0000","last_change":"2024-04-25T13:38:45.363043+0000","last_active":"2024-04-25T13:40:00.356232+0000","last_peered":"2024-04-25T13:40:00.356232+0000","last_clean":"2024-04-25T13:40:00.356232+0000","last_became_active":"2024-04-25T13:38:45.362686+0000","last_became_peered":"2024-04-25T13:38:45.362686+0000","last_unstale":"2024-04-25T13:40:00.356232+0000","last_undegraded":"2024-04-25T13:40:00.356232+0000","last_fullsized":"2024-04-25T13:40:00.356232+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:56.602134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"78'7","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362173+0000","last_change":"2024-04-25T13:38:45.373256+0000","last_active":"2024-04-25T13:40:00.362173+0000","last_peered":"2024-04-25T13:40:00.362173+0000","last_clean":"2024-04-25T13:40:00.362173+0000","last_became_active":"2024-04-25T13:38:45.372844+0000","last_became_peered":"2024-04-25T13:38:45.372844+0000","last_unstale":"2024-04-25T13:40:00.362173+0000","last_undegraded":"2024-04-25T13:40:00.362173+0000","last_fullsized":"2024-04-25T13:40:00.362173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:39:34.590536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.364083+0000","last_change":"2024-04-25T13:38:45.370007+0000","last_active":"2024-04-25T13:40:00.364083+0000","last_peered":"2024-04-25T13:40:00.364083+0000","last_clean":"2024-04-25T13:40:00.364083+0000","last_became_active":"2024-04-25T13:38:45.369606+0000","last_became_peered":"2024-04-25T13:38:45.369606+0000","last_unstale":"2024-04-25T13:40:00.364083+0000","last_undegraded":"2024-04-25T13:40:00.364083+0000","last_fullsized":"2024-04-25T13:40:00.364083+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:45.514756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.350193+0000","last_change":"2024-04-25T13:38:45.362075+0000","last_active":"2024-04-25T13:40:00.350193+0000","last_peered":"2024-04-25T13:40:00.350193+0000","last_clean":"2024-04-25T13:40:00.350193+0000","last_became_active":"2024-04-25T13:38:45.361720+0000","last_became_peered":"2024-04-25T13:38:45.361720+0000","last_unstale":"2024-04-25T13:40:00.350193+0000","last_undegraded":"2024-04-25T13:40:00.350193+0000","last_fullsized":"2024-04-25T13:40:00.350193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:36.169185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354853+0000","last_change":"2024-04-25T13:38:45.352101+0000","last_active":"2024-04-25T13:40:00.354853+0000","last_peered":"2024-04-25T13:40:00.354853+0000","last_clean":"2024-04-25T13:40:00.354853+0000","last_became_active":"2024-04-25T13:38:45.351739+0000","last_became_peered":"2024-04-25T13:38:45.351739+0000","last_unstale":"2024-04-25T13:40:00.354853+0000","last_undegraded":"2024-04-25T13:40:00.354853+0000","last_fullsized":"2024-04-25T13:40:00.354853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:05.612527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.362759+0000","last_change":"2024-04-25T13:38:45.363855+0000","last_active":"2024-04-25T13:40:00.362759+0000","last_peered":"2024-04-25T13:40:00.362759+0000","last_clean":"2024-04-25T13:40:00.362759+0000","last_became_active":"2024-04-25T13:38:45.363377+0000","last_became_peered":"2024-04-25T13:38:45.363377+0000","last_unstale":"2024-04-25T13:40:00.362759+0000","last_undegraded":"2024-04-25T13:40:00.362759+0000","last_fullsized":"2024-04-25T13:40:00.362759+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:26.828215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.365565+0000","last_change":"2024-04-25T13:38:45.359844+0000","last_active":"2024-04-25T13:40:00.365565+0000","last_peered":"2024-04-25T13:40:00.365565+0000","last_clean":"2024-04-25T13:40:00.365565+0000","last_became_active":"2024-04-25T13:38:45.359428+0000","last_became_peered":"2024-04-25T13:38:45.359428+0000","last_unstale":"2024-04-25T13:40:00.365565+0000","last_undegraded":"2024-04-25T13:40:00.365565+0000","last_fullsized":"2024-04-25T13:40:00.365565+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:01.370409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.359173+0000","last_change":"2024-04-25T13:38:45.370415+0000","last_active":"2024-04-25T13:40:00.359173+0000","last_peered":"2024-04-25T13:40:00.359173+0000","last_clean":"2024-04-25T13:40:00.359173+0000","last_became_active":"2024-04-25T13:38:45.369525+0000","last_became_peered":"2024-04-25T13:38:45.369525+0000","last_unstale":"2024-04-25T13:40:00.359173+0000","last_undegraded":"2024-04-25T13:40:00.359173+0000","last_fullsized":"2024-04-25T13:40:00.359173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:39.532380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.376082+0000","last_change":"2024-04-25T13:38:45.365309+0000","last_active":"2024-04-25T13:40:00.376082+0000","last_peered":"2024-04-25T13:40:00.376082+0000","last_clean":"2024-04-25T13:40:00.376082+0000","last_became_active":"2024-04-25T13:38:45.364893+0000","last_became_peered":"2024-04-25T13:38:45.364893+0000","last_unstale":"2024-04-25T13:40:00.376082+0000","last_undegraded":"2024-04-25T13:40:00.376082+0000","last_fullsized":"2024-04-25T13:40:00.376082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:04:54.454443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.368064+0000","last_change":"2024-04-25T13:38:45.365150+0000","last_active":"2024-04-25T13:40:00.368064+0000","last_peered":"2024-04-25T13:40:00.368064+0000","last_clean":"2024-04-25T13:40:00.368064+0000","last_became_active":"2024-04-25T13:38:45.364649+0000","last_became_peered":"2024-04-25T13:38:45.364649+0000","last_unstale":"2024-04-25T13:40:00.368064+0000","last_undegraded":"2024-04-25T13:40:00.368064+0000","last_fullsized":"2024-04-25T13:40:00.368064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:25.035789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T13:40:00.354459+0000","last_change":"2024-04-25T13:38:45.352108+0000","last_active":"2024-04-25T13:40:00.354459+0000","last_peered":"2024-04-25T13:40:00.354459+0000","last_clean":"2024-04-25T13:40:00.354459+0000","last_became_active":"2024-04-25T13:38:45.349887+0000","last_became_peered":"2024-04-25T13:38:45.349887+0000","last_unstale":"2024-04-25T13:40:00.354459+0000","last_undegraded":"2024-04-25T13:40:00.354459+0000","last_fullsized":"2024-04-25T13:40:00.354459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T13:38:44.328044+0000","last_clean_scrub_stamp":"2024-04-25T13:38:44.328044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:13:09.339685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285228,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67774,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":262,"num_read_kb":199,"num_write":85,"num_write_kb":95,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":203322,"data_compressed":5526,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678055,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":403844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874284,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":1061499,"data_compressed":38103,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037810,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234039,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430268,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":888832,"data_stored":994911,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593794,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36220,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708932,"statfs":{"total":95995035648,"available":95957946368,"internally_reserved":0,"allocated":905216,"data_stored":470820,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":403934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":470854,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":843776,"data_stored":404065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":403866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790023,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":994511,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34641,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36261,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":66976,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67010,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67170,"data_compressed":1842,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T13:40:53.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:53.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.187+0000 7f320f984640 1 -- 172.21.15.27:0/2600538393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32081098b0 msgr2=0x7f320810bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.187+0000 7f320f984640 1 --2- 172.21.15.27:0/2600538393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32081098b0 0x7f320810bcd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3200009ca0 tx=0x7f320002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.187+0000 7f320f984640 1 -- 172.21.15.27:0/2600538393 shutdown_connections 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.187+0000 7f320f984640 1 --2- 172.21.15.27:0/2600538393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32081098b0 0x7f320810bcd0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.187+0000 7f320f984640 1 --2- 172.21.15.27:0/2600538393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3208106f50 0x7f3208109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.187+0000 7f320f984640 1 --2- 172.21.15.27:0/2600538393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32081045f0 0x7f3208106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 -- 172.21.15.27:0/2600538393 >> 172.21.15.27:0/2600538393 conn(0x7f32080fdbf0 msgr2=0x7f32081000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 -- 172.21.15.27:0/2600538393 shutdown_connections 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 -- 172.21.15.27:0/2600538393 wait complete. 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 Processor -- start 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 -- start start 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32081045f0 0x7f3208103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208106f50 0x7f32081020e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32081098b0 0x7f3208102620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f320810f610 con 0x7f3208106f50 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f320810f490 con 0x7f32081098b0 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.188+0000 7f320f984640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f320810f790 con 0x7f32081045f0 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208106f50 0x7f32081020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208106f50 0x7f32081020e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54400/0 (socket says 172.21.15.27:54400) 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 -- 172.21.15.27:0/3795174247 learned_addr learned my addr 172.21.15.27:0/3795174247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320df21640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32081098b0 0x7f3208102620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320d720640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32081045f0 0x7f3208103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 -- 172.21.15.27:0/3795174247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32081045f0 msgr2=0x7f3208103a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32081045f0 0x7f3208103a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 -- 172.21.15.27:0/3795174247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32081098b0 msgr2=0x7f3208102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32081098b0 0x7f3208102620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 -- 172.21.15.27:0/3795174247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3208102d50 con 0x7f3208106f50 2024-04-25T13:40:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320cf1f640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208106f50 0x7f32081020e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f31f800d950 tx=0x7f31f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f31fe7fc640 1 -- 172.21.15.27:0/3795174247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8017070 con 0x7f3208106f50 2024-04-25T13:40:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f31fe7fc640 1 -- 172.21.15.27:0/3795174247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31f8009c30 con 0x7f3208106f50 2024-04-25T13:40:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32081a4240 con 0x7f3208106f50 2024-04-25T13:40:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f31fe7fc640 1 -- 172.21.15.27:0/3795174247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f800f620 con 0x7f3208106f50 2024-04-25T13:40:53.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.189+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32081a4700 con 0x7f3208106f50 2024-04-25T13:40:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.191+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32080ff710 con 0x7f3208106f50 2024-04-25T13:40:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.192+0000 7f31fe7fc640 1 -- 172.21.15.27:0/3795174247 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f31f8010460 con 0x7f3208106f50 2024-04-25T13:40:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.192+0000 7f31fe7fc640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31f40afaf0 0x7f31f40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.193+0000 7f320d720640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31f40afaf0 0x7f31f40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.193+0000 7f31fe7fc640 1 -- 172.21.15.27:0/3795174247 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f31f80de7b0 con 0x7f3208106f50 2024-04-25T13:40:53.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.193+0000 7f320d720640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31f40afaf0 0x7f31f40b2000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f32080757f0 tx=0x7f320400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:53.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.197+0000 7f31fe7fc640 1 -- 172.21.15.27:0/3795174247 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31f8009de0 con 0x7f3208106f50 2024-04-25T13:40:53.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.387+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32080637a0 con 0x7f3208106f50 2024-04-25T13:40:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.388+0000 7f31fe7fc640 1 -- 172.21.15.27:0/3795174247 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7f31f80a6d70 con 0x7f3208106f50 2024-04-25T13:40:53.389 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:53.389 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:53.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31f40afaf0 msgr2=0x7f31f40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31f40afaf0 0x7f31f40b2000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f32080757f0 tx=0x7f320400b390 comp rx=0 tx=0).stop 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208106f50 msgr2=0x7f32081020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208106f50 0x7f32081020e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f31f800d950 tx=0x7f31f800de10 comp rx=0 tx=0).stop 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 shutdown_connections 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31f40afaf0 0x7f31f40b2000 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32081098b0 0x7f3208102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208106f50 0x7f32081020e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 --2- 172.21.15.27:0/3795174247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32081045f0 0x7f3208103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.390+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 >> 172.21.15.27:0/3795174247 conn(0x7f32080fdbf0 msgr2=0x7f32081054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.391+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 shutdown_connections 2024-04-25T13:40:53.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.391+0000 7f320f984640 1 -- 172.21.15.27:0/3795174247 wait complete. 2024-04-25T13:40:53.403 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:40:53.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:40:53.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 -- 172.21.15.27:0/1199516752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 msgr2=0x7fc3d0104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 --2- 172.21.15.27:0/1199516752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 0x7fc3d0104030 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009900 tx=0x7fc3b802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:53.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 -- 172.21.15.27:0/1199516752 shutdown_connections 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 --2- 172.21.15.27:0/1199516752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d0108920 0x7fc3d010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 --2- 172.21.15.27:0/1199516752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 0x7fc3d0104030 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 --2- 172.21.15.27:0/1199516752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3d01029e0 0x7fc3d01036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 -- 172.21.15.27:0/1199516752 >> 172.21.15.27:0/1199516752 conn(0x7fc3d00fdbd0 msgr2=0x7fc3d0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 -- 172.21.15.27:0/1199516752 shutdown_connections 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.518+0000 7fc3d6934640 1 -- 172.21.15.27:0/1199516752 wait complete. 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d6934640 1 Processor -- start 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d6934640 1 -- start start 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d6934640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d01029e0 0x7fc3d019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d6934640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 0x7fc3d019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d6934640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3d0108920 0x7fc3d01a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d6934640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d010f610 con 0x7fc3d0103c20 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d6934640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d010f490 con 0x7fc3d01029e0 2024-04-25T13:40:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d6934640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d010f790 con 0x7fc3d0108920 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 0x7fc3d019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 0x7fc3d019aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54428/0 (socket says 172.21.15.27:54428) 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3cf7fe640 1 -- 172.21.15.27:0/2139011149 learned_addr learned my addr 172.21.15.27:0/2139011149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3cffff640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d01029e0 0x7fc3d019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.519+0000 7fc3d4ed1640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3d0108920 0x7fc3d01a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cf7fe640 1 -- 172.21.15.27:0/2139011149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3d0108920 msgr2=0x7fc3d01a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cf7fe640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3d0108920 0x7fc3d01a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cf7fe640 1 -- 172.21.15.27:0/2139011149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d01029e0 msgr2=0x7fc3d019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cf7fe640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d01029e0 0x7fc3d019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cf7fe640 1 -- 172.21.15.27:0/2139011149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d01a2080 con 0x7fc3d0103c20 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cf7fe640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 0x7fc3d019aa80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8006010 tx=0x7fc3b8004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cd7fa640 1 -- 172.21.15.27:0/2139011149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b803f030 con 0x7fc3d0103c20 2024-04-25T13:40:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d01a2310 con 0x7fc3d0103c20 2024-04-25T13:40:53.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cd7fa640 1 -- 172.21.15.27:0/2139011149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b8003cb0 con 0x7fc3d0103c20 2024-04-25T13:40:53.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3cd7fa640 1 -- 172.21.15.27:0/2139011149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8035070 con 0x7fc3d0103c20 2024-04-25T13:40:53.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.520+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d01a2850 con 0x7fc3d0103c20 2024-04-25T13:40:53.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.521+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3d0107730 con 0x7fc3d0103c20 2024-04-25T13:40:53.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.523+0000 7fc3cd7fa640 1 -- 172.21.15.27:0/2139011149 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc3b8042030 con 0x7fc3d0103c20 2024-04-25T13:40:53.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.523+0000 7fc3cd7fa640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3bc0afaf0 0x7fc3bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.523+0000 7fc3cffff640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3bc0afaf0 0x7fc3bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.523+0000 7fc3cd7fa640 1 -- 172.21.15.27:0/2139011149 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc3b80ff830 con 0x7fc3d0103c20 2024-04-25T13:40:53.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.524+0000 7fc3cffff640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3bc0afaf0 0x7fc3bc0b2000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc3d01035e0 tx=0x7fc3c000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.527+0000 7fc3cd7fa640 1 -- 172.21.15.27:0/2139011149 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3b8003e60 con 0x7fc3d0103c20 2024-04-25T13:40:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.713+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc3d002dcd0 con 0x7fc3d0103c20 2024-04-25T13:40:53.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.714+0000 7fc3cd7fa640 1 -- 172.21.15.27:0/2139011149 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7fc3b80c7df0 con 0x7fc3d0103c20 2024-04-25T13:40:53.714 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3bc0afaf0 msgr2=0x7fc3bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3bc0afaf0 0x7fc3bc0b2000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc3d01035e0 tx=0x7fc3c000b390 comp rx=0 tx=0).stop 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 msgr2=0x7fc3d019aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 0x7fc3d019aa80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8006010 tx=0x7fc3b8004340 comp rx=0 tx=0).stop 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 shutdown_connections 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc3bc0afaf0 0x7fc3bc0b2000 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3d0108920 0x7fc3d01a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d0103c20 0x7fc3d019aa80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 --2- 172.21.15.27:0/2139011149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d01029e0 0x7fc3d019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 >> 172.21.15.27:0/2139011149 conn(0x7fc3d00fdbd0 msgr2=0x7fc3d0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:53.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 shutdown_connections 2024-04-25T13:40:53.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.716+0000 7fc3d6934640 1 -- 172.21.15.27:0/2139011149 wait complete. 2024-04-25T13:40:53.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:40:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 -- 172.21.15.27:0/3205389999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448103c40 msgr2=0x7f7448104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 --2- 172.21.15.27:0/3205389999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448103c40 0x7f7448104050 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f743c009900 tx=0x7f743c02a410 comp rx=0 tx=0).stop 2024-04-25T13:40:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 -- 172.21.15.27:0/3205389999 shutdown_connections 2024-04-25T13:40:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 --2- 172.21.15.27:0/3205389999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7448108940 0x7f744810ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 --2- 172.21.15.27:0/3205389999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448103c40 0x7f7448104050 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 --2- 172.21.15.27:0/3205389999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7448102a00 0x7f7448103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 -- 172.21.15.27:0/3205389999 >> 172.21.15.27:0/3205389999 conn(0x7f74480fdbf0 msgr2=0x7f74481000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:53.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 -- 172.21.15.27:0/3205389999 shutdown_connections 2024-04-25T13:40:53.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 -- 172.21.15.27:0/3205389999 wait complete. 2024-04-25T13:40:53.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 Processor -- start 2024-04-25T13:40:53.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.843+0000 7f744db14640 1 -- start start 2024-04-25T13:40:53.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f744db14640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448102a00 0x7f7448075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f744db14640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7448103c40 0x7f744819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f744db14640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7448108940 0x7f74481a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f744db14640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f744810e530 con 0x7f7448102a00 2024-04-25T13:40:53.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f744db14640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f744810e3b0 con 0x7f7448103c40 2024-04-25T13:40:53.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f744db14640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f744810e6b0 con 0x7f7448108940 2024-04-25T13:40:53.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448102a00 0x7f7448075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448102a00 0x7f7448075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35106/0 (socket says 172.21.15.27:35106) 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 -- 172.21.15.27:0/3577482226 learned_addr learned my addr 172.21.15.27:0/3577482226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f7446ffd640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7448103c40 0x7f744819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f7447fff640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7448108940 0x7f74481a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 -- 172.21.15.27:0/3577482226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7448108940 msgr2=0x7f74481a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7448108940 0x7f74481a14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 -- 172.21.15.27:0/3577482226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7448103c40 msgr2=0x7f744819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7448103c40 0x7f744819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 -- 172.21.15.27:0/3577482226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74481a1d00 con 0x7f7448102a00 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f74477fe640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448102a00 0x7f7448075840 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7438005fd0 tx=0x7f743800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.844+0000 7f7444ff9640 1 -- 172.21.15.27:0/3577482226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438017070 con 0x7f7448102a00 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.845+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74481a1ff0 con 0x7f7448102a00 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.845+0000 7f7444ff9640 1 -- 172.21.15.27:0/3577482226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7438009c30 con 0x7f7448102a00 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.845+0000 7f7444ff9640 1 -- 172.21.15.27:0/3577482226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438003d10 con 0x7f7448102a00 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.845+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74481a2450 con 0x7f7448102a00 2024-04-25T13:40:53.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.846+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f744806a160 con 0x7f7448102a00 2024-04-25T13:40:53.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.847+0000 7f7444ff9640 1 -- 172.21.15.27:0/3577482226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7438003eb0 con 0x7f7448102a00 2024-04-25T13:40:53.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.848+0000 7f7444ff9640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f74300afaf0 0x7f74300b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:53.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.848+0000 7f7444ff9640 1 -- 172.21.15.27:0/3577482226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f74380df080 con 0x7f7448102a00 2024-04-25T13:40:53.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.848+0000 7f7446ffd640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f74300afaf0 0x7f74300b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:53.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.848+0000 7f7446ffd640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f74300afaf0 0x7f74300b2000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f743c009230 tx=0x7f743c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:53.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:53.851+0000 7f7444ff9640 1 -- 172.21.15.27:0/3577482226 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7438020030 con 0x7f7448102a00 2024-04-25T13:40:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.037+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f744802dbf0 con 0x7f7448102a00 2024-04-25T13:40:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.038+0000 7f7444ff9640 1 -- 172.21.15.27:0/3577482226 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f7438009de0 con 0x7f7448102a00 2024-04-25T13:40:54.038 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f74300afaf0 msgr2=0x7f74300b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f74300afaf0 0x7f74300b2000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f743c009230 tx=0x7f743c031400 comp rx=0 tx=0).stop 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448102a00 msgr2=0x7f7448075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448102a00 0x7f7448075840 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7438005fd0 tx=0x7f743800dde0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 shutdown_connections 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f74300afaf0 0x7f74300b2000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7448108940 0x7f74481a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7448103c40 0x7f744819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 --2- 172.21.15.27:0/3577482226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7448102a00 0x7f7448075840 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 >> 172.21.15.27:0/3577482226 conn(0x7f74480fdbf0 msgr2=0x7f7448104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 shutdown_connections 2024-04-25T13:40:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.040+0000 7f744db14640 1 -- 172.21.15.27:0/3577482226 wait complete. 2024-04-25T13:40:54.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:40:54.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 -- 172.21.15.27:0/3914484184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a01045d0 msgr2=0x7fb0a01069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3914484184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a01045d0 0x7fb0a01069f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb090009ca0 tx=0x7fb09002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 -- 172.21.15.27:0/3914484184 shutdown_connections 2024-04-25T13:40:54.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3914484184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a0109890 0x7fb0a010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3914484184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0a0106f30 0x7fb0a0109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3914484184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a01045d0 0x7fb0a01069f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 -- 172.21.15.27:0/3914484184 >> 172.21.15.27:0/3914484184 conn(0x7fb0a00fdbd0 msgr2=0x7fb0a0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 -- 172.21.15.27:0/3914484184 shutdown_connections 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 -- 172.21.15.27:0/3914484184 wait complete. 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.170+0000 7fb0a610d640 1 Processor -- start 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb0a610d640 1 -- start start 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb0a610d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0a01045d0 0x7fb0a019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb0a610d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a0106f30 0x7fb0a019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb0a610d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a0109890 0x7fb0a01a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb0a610d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a010f5c0 con 0x7fb0a0109890 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb0a610d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a010f440 con 0x7fb0a01045d0 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb0a610d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a010f740 con 0x7fb0a0106f30 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a0109890 0x7fb0a01a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a0109890 0x7fb0a01a1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35122/0 (socket says 172.21.15.27:35122) 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09ffff640 1 -- 172.21.15.27:0/3302346730 learned_addr learned my addr 172.21.15.27:0/3302346730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09f7fe640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0a01045d0 0x7fb0a019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09effd640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a0106f30 0x7fb0a019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09ffff640 1 -- 172.21.15.27:0/3302346730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a0106f30 msgr2=0x7fb0a019aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09ffff640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a0106f30 0x7fb0a019aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09ffff640 1 -- 172.21.15.27:0/3302346730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0a01045d0 msgr2=0x7fb0a019a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09ffff640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0a01045d0 0x7fb0a019a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.171+0000 7fb09ffff640 1 -- 172.21.15.27:0/3302346730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a01a2030 con 0x7fb0a0109890 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.172+0000 7fb09ffff640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a0109890 0x7fb0a01a1a70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb09800d720 tx=0x7fb09800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.172+0000 7fb09cff9640 1 -- 172.21.15.27:0/3302346730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb098017070 con 0x7fb0a0109890 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.172+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a01a2320 con 0x7fb0a0109890 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.172+0000 7fb09cff9640 1 -- 172.21.15.27:0/3302346730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb09800de30 con 0x7fb0a0109890 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.172+0000 7fb09cff9640 1 -- 172.21.15.27:0/3302346730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb098004d00 con 0x7fb0a0109890 2024-04-25T13:40:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.172+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0a01a2870 con 0x7fb0a0109890 2024-04-25T13:40:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.173+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0a006c2a0 con 0x7fb0a0109890 2024-04-25T13:40:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.175+0000 7fb09cff9640 1 -- 172.21.15.27:0/3302346730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb098004ea0 con 0x7fb0a0109890 2024-04-25T13:40:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.176+0000 7fb09cff9640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb08c0afaf0 0x7fb08c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.176+0000 7fb09f7fe640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb08c0afaf0 0x7fb08c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.176+0000 7fb09cff9640 1 -- 172.21.15.27:0/3302346730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb0980dea60 con 0x7fb0a0109890 2024-04-25T13:40:54.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.176+0000 7fb09f7fe640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb08c0afaf0 0x7fb08c0b2000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb090031b50 tx=0x7fb090004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:54.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.179+0000 7fb09cff9640 1 -- 172.21.15.27:0/3302346730 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0980a7610 con 0x7fb0a0109890 2024-04-25T13:40:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:54 smithi045 ceph-mon[32731]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:54 smithi045 ceph-mon[32731]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-25T13:40:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:54 smithi045 ceph-mon[32731]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:54 smithi045 ceph-mon[32731]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:54 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3795174247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:54 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2139011149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:54.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:54 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3577482226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:54.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.366+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb0a01a2b90 con 0x7fb0a0109890 2024-04-25T13:40:54.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.366+0000 7fb09cff9640 1 -- 172.21.15.27:0/3302346730 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7fb0980a6f10 con 0x7fb0a0109890 2024-04-25T13:40:54.367 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:40:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb08c0afaf0 msgr2=0x7fb08c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb08c0afaf0 0x7fb08c0b2000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb090031b50 tx=0x7fb090004a40 comp rx=0 tx=0).stop 2024-04-25T13:40:54.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a0109890 msgr2=0x7fb0a01a1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a0109890 0x7fb0a01a1a70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb09800d720 tx=0x7fb09800dbe0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 shutdown_connections 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb08c0afaf0 0x7fb08c0b2000 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a0109890 0x7fb0a01a1a70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0a0106f30 0x7fb0a019aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 --2- 172.21.15.27:0/3302346730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0a01045d0 0x7fb0a019a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 >> 172.21.15.27:0/3302346730 conn(0x7fb0a00fdbd0 msgr2=0x7fb0a0104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 shutdown_connections 2024-04-25T13:40:54.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.368+0000 7fb0a610d640 1 -- 172.21.15.27:0/3302346730 wait complete. 2024-04-25T13:40:54.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:54 smithi027 ceph-mon[29140]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:54 smithi027 ceph-mon[29140]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-25T13:40:54.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:54 smithi027 ceph-mon[29140]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:54 smithi027 ceph-mon[29140]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3795174247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:54.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2139011149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:54.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3577482226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:54.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:40:54.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 -- 172.21.15.27:0/1984605669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50103b00 msgr2=0x7fbb50104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 --2- 172.21.15.27:0/1984605669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50103b00 0x7fbb50104800 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009900 tx=0x7fbb3802a410 comp rx=0 tx=0).stop 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 -- 172.21.15.27:0/1984605669 shutdown_connections 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 --2- 172.21.15.27:0/1984605669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50109a40 0x7fbb5010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 --2- 172.21.15.27:0/1984605669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb50104d40 0x7fbb50105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 --2- 172.21.15.27:0/1984605669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50103b00 0x7fbb50104800 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 -- 172.21.15.27:0/1984605669 >> 172.21.15.27:0/1984605669 conn(0x7fbb500fed10 msgr2=0x7fbb501011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 -- 172.21.15.27:0/1984605669 shutdown_connections 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.492+0000 7fbb5675f640 1 -- 172.21.15.27:0/1984605669 wait complete. 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb5675f640 1 Processor -- start 2024-04-25T13:40:54.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb5675f640 1 -- start start 2024-04-25T13:40:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb5675f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb50103b00 0x7fbb5019c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb5675f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50104d40 0x7fbb5019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb5675f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50109a40 0x7fbb501a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb5675f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010f5e0 con 0x7fbb50104d40 2024-04-25T13:40:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb5675f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010f460 con 0x7fbb50109a40 2024-04-25T13:40:54.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb5675f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010f760 con 0x7fbb50103b00 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50104d40 0x7fbb5019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50104d40 0x7fbb5019c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35144/0 (socket says 172.21.15.27:35144) 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4f7fe640 1 -- 172.21.15.27:0/2955710504 learned_addr learned my addr 172.21.15.27:0/2955710504 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4ffff640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb50103b00 0x7fbb5019c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb54cfc640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50109a40 0x7fbb501a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4f7fe640 1 -- 172.21.15.27:0/2955710504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb50103b00 msgr2=0x7fbb5019c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4f7fe640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb50103b00 0x7fbb5019c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4f7fe640 1 -- 172.21.15.27:0/2955710504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50109a40 msgr2=0x7fbb501a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4f7fe640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50109a40 0x7fbb501a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.493+0000 7fbb4f7fe640 1 -- 172.21.15.27:0/2955710504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb501a3f20 con 0x7fbb50104d40 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.494+0000 7fbb54cfc640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50109a40 0x7fbb501a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.494+0000 7fbb4f7fe640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50104d40 0x7fbb5019c9d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000c920 tx=0x7fbb4000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.494+0000 7fbb4d7fa640 1 -- 172.21.15.27:0/2955710504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40007c70 con 0x7fbb50104d40 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.494+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb501a4210 con 0x7fbb50104d40 2024-04-25T13:40:54.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.494+0000 7fbb4d7fa640 1 -- 172.21.15.27:0/2955710504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb40007e10 con 0x7fbb50104d40 2024-04-25T13:40:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.494+0000 7fbb4d7fa640 1 -- 172.21.15.27:0/2955710504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40018820 con 0x7fbb50104d40 2024-04-25T13:40:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.494+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb501a47b0 con 0x7fbb50104d40 2024-04-25T13:40:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.495+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb501a4ad0 con 0x7fbb50104d40 2024-04-25T13:40:54.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.496+0000 7fbb4d7fa640 1 -- 172.21.15.27:0/2955710504 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbb4000f450 con 0x7fbb50104d40 2024-04-25T13:40:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.497+0000 7fbb4d7fa640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb3c0afaf0 0x7fbb3c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.497+0000 7fbb4ffff640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb3c0afaf0 0x7fbb3c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.497+0000 7fbb4d7fa640 1 -- 172.21.15.27:0/2955710504 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbb400defc0 con 0x7fbb50104d40 2024-04-25T13:40:54.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.498+0000 7fbb4ffff640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb3c0afaf0 0x7fbb3c0b2000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbb38005ab0 tx=0x7fbb38031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.501+0000 7fbb4d7fa640 1 -- 172.21.15.27:0/2955710504 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb40014030 con 0x7fbb50104d40 2024-04-25T13:40:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:54 smithi122 ceph-mon[32843]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:54 smithi122 ceph-mon[32843]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-25T13:40:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:54 smithi122 ceph-mon[32843]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:54 smithi122 ceph-mon[32843]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T13:40:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:54 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3795174247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:54 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2139011149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:40:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:54 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3577482226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:40:54.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.728+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb501a4ca0 con 0x7fbb50104d40 2024-04-25T13:40:54.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.729+0000 7fbb4d7fa640 1 -- 172.21.15.27:0/2955710504 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fbb400a7c60 con 0x7fbb50104d40 2024-04-25T13:40:54.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:54.729 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:40:54.729 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:40:54.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb3c0afaf0 msgr2=0x7fbb3c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb3c0afaf0 0x7fbb3c0b2000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbb38005ab0 tx=0x7fbb38031400 comp rx=0 tx=0).stop 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50104d40 msgr2=0x7fbb5019c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50104d40 0x7fbb5019c9d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000c920 tx=0x7fbb4000cde0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 shutdown_connections 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb3c0afaf0 0x7fbb3c0b2000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50109a40 0x7fbb501a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb50104d40 0x7fbb5019c9d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 --2- 172.21.15.27:0/2955710504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb50103b00 0x7fbb5019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 >> 172.21.15.27:0/2955710504 conn(0x7fbb500fed10 msgr2=0x7fbb500ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 shutdown_connections 2024-04-25T13:40:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.731+0000 7fbb5675f640 1 -- 172.21.15.27:0/2955710504 wait complete. 2024-04-25T13:40:54.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:40:54.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 -- 172.21.15.27:0/2470873554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 msgr2=0x7f0fb010ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/2470873554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 0x7f0fb010ad20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009ca0 tx=0x7f0fa802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 -- 172.21.15.27:0/2470873554 shutdown_connections 2024-04-25T13:40:54.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/2470873554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 0x7f0fb010ad20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/2470873554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fb0103c00 0x7f0fb0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/2470873554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fb01029c0 0x7f0fb01036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 -- 172.21.15.27:0/2470873554 >> 172.21.15.27:0/2470873554 conn(0x7f0fb00fdbd0 msgr2=0x7f0fb0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 -- 172.21.15.27:0/2470873554 shutdown_connections 2024-04-25T13:40:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.855+0000 7f0fb4a92640 1 -- 172.21.15.27:0/2470873554 wait complete. 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fb4a92640 1 Processor -- start 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fb4a92640 1 -- start start 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fb4a92640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fb01029c0 0x7f0fb01990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fb4a92640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fb0103c00 0x7f0fb01995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fb4a92640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 0x7f0fb01a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fb4a92640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb010e4d0 con 0x7f0fb0108900 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fb4a92640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb010e350 con 0x7f0fb0103c00 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fb4a92640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb010e650 con 0x7f0fb01029c0 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0faed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 0x7f0fb01a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0faed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 0x7f0fb01a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35170/0 (socket says 172.21.15.27:35170) 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0faed9d640 1 -- 172.21.15.27:0/1302536589 learned_addr learned my addr 172.21.15.27:0/1302536589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fae59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fb01029c0 0x7f0fb01990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0fadd9b640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fb0103c00 0x7f0fb01995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0faed9d640 1 -- 172.21.15.27:0/1302536589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fb01029c0 msgr2=0x7f0fb01990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0faed9d640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fb01029c0 0x7f0fb01990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0faed9d640 1 -- 172.21.15.27:0/1302536589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fb0103c00 msgr2=0x7f0fb01995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0faed9d640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fb0103c00 0x7f0fb01995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.856+0000 7f0faed9d640 1 -- 172.21.15.27:0/1302536589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb01a0be0 con 0x7f0fb0108900 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.857+0000 7f0fadd9b640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fb0103c00 0x7f0fb01995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.857+0000 7f0faed9d640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 0x7f0fb01a0570 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80097b0 tx=0x7f0fa8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.857+0000 7f0fa77fe640 1 -- 172.21.15.27:0/1302536589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8004240 con 0x7f0fb0108900 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.857+0000 7f0fa77fe640 1 -- 172.21.15.27:0/1302536589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa802f380 con 0x7f0fb0108900 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.857+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb01a0e70 con 0x7f0fb0108900 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.857+0000 7f0fa77fe640 1 -- 172.21.15.27:0/1302536589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa802fc30 con 0x7f0fb0108900 2024-04-25T13:40:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.857+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb01a12d0 con 0x7f0fb0108900 2024-04-25T13:40:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.858+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb006afa0 con 0x7f0fb0108900 2024-04-25T13:40:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.859+0000 7f0fa77fe640 1 -- 172.21.15.27:0/1302536589 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0fa80043e0 con 0x7f0fb0108900 2024-04-25T13:40:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.860+0000 7f0fa77fe640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f9c0afaf0 0x7f0f9c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.860+0000 7f0fae59c640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f9c0afaf0 0x7f0f9c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.860+0000 7f0fa77fe640 1 -- 172.21.15.27:0/1302536589 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0fa8100660 con 0x7f0fb0108900 2024-04-25T13:40:54.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.861+0000 7f0fae59c640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f9c0afaf0 0x7f0f9c0b2000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0fb01035c0 tx=0x7f0f9800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:54.864+0000 7f0fa77fe640 1 -- 172.21.15.27:0/1302536589 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0fa80c9270 con 0x7f0fb0108900 2024-04-25T13:40:55.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.050+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fb002dc10 con 0x7f0fb0108900 2024-04-25T13:40:55.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.051+0000 7f0fa77fe640 1 -- 172.21.15.27:0/1302536589 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7f0fa80e9e20 con 0x7f0fb0108900 2024-04-25T13:40:55.051 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:55.052 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:40:55.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f9c0afaf0 msgr2=0x7f0f9c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f9c0afaf0 0x7f0f9c0b2000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0fb01035c0 tx=0x7f0f9800b390 comp rx=0 tx=0).stop 2024-04-25T13:40:55.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 msgr2=0x7f0fb01a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 0x7f0fb01a0570 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80097b0 tx=0x7f0fa8003c20 comp rx=0 tx=0).stop 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 shutdown_connections 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0f9c0afaf0 0x7f0f9c0b2000 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fb0108900 0x7f0fb01a0570 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fb0103c00 0x7f0fb01995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 --2- 172.21.15.27:0/1302536589 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0fb01029c0 0x7f0fb01990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 >> 172.21.15.27:0/1302536589 conn(0x7f0fb00fdbd0 msgr2=0x7f0fb01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 shutdown_connections 2024-04-25T13:40:55.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.053+0000 7f0fb4a92640 1 -- 172.21.15.27:0/1302536589 wait complete. 2024-04-25T13:40:55.066 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T13:40:55.066 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T13:40:55.076 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T13:40:55.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T13:40:55.076 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T13:40:55.077 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:40:55.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:40:55.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:40:55.079 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T13:40:55.088 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T13:40:55.088 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T13:40:55.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-25T13:40:55.122 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:40:55.134 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-25T13:40:55.134 INFO:teuthology.orchestra.run.smithi027.stdout:0@db01fc44-0307-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/aa501c8f-0208-4329-abf5-aee6b1b25d7c 340G 0 340G 0% /tmp/cephfs 2024-04-25T13:40:55.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-25T13:40:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 -- 172.21.15.27:0/4251634677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8108920 msgr2=0x7fa5d810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4251634677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8108920 0x7fa5d810ad40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009ca0 tx=0x7fa5d002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 -- 172.21.15.27:0/4251634677 shutdown_connections 2024-04-25T13:40:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4251634677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8108920 0x7fa5d810ad40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4251634677 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d8103c20 0x7fa5d8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4251634677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5d81029e0 0x7fa5d81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 -- 172.21.15.27:0/4251634677 >> 172.21.15.27:0/4251634677 conn(0x7fa5d80fdbd0 msgr2=0x7fa5d8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 -- 172.21.15.27:0/4251634677 shutdown_connections 2024-04-25T13:40:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.195+0000 7fa5de17d640 1 -- 172.21.15.27:0/4251634677 wait complete. 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5de17d640 1 Processor -- start 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5de17d640 1 -- start start 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5de17d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5d81029e0 0x7fa5d819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5de17d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8103c20 0x7fa5d819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5de17d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d8108920 0x7fa5d81a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5de17d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810e510 con 0x7fa5d8103c20 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5de17d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810e390 con 0x7fa5d81029e0 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5de17d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810e690 con 0x7fa5d8108920 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8103c20 0x7fa5d819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8103c20 0x7fa5d819aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35198/0 (socket says 172.21.15.27:35198) 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/4038291192 learned_addr learned my addr 172.21.15.27:0/4038291192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.196+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d8108920 0x7fa5d81a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5d81029e0 0x7fa5d819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/4038291192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d8108920 msgr2=0x7fa5d81a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d8108920 0x7fa5d81a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/4038291192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5d81029e0 msgr2=0x7fa5d819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5d81029e0 0x7fa5d819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/4038291192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d81a2030 con 0x7fa5d8103c20 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d8108920 0x7fa5d81a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5d81029e0 0x7fa5d819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8103c20 0x7fa5d819aa30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800e9f0 tx=0x7fa5c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d4ff9640 1 -- 172.21.15.27:0/4038291192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c800cd80 con 0x7fa5d8103c20 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5de17d640 1 -- 172.21.15.27:0/4038291192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d81a2320 con 0x7fa5d8103c20 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d4ff9640 1 -- 172.21.15.27:0/4038291192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c80054d0 con 0x7fa5d8103c20 2024-04-25T13:40:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5d4ff9640 1 -- 172.21.15.27:0/4038291192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8010690 con 0x7fa5d8103c20 2024-04-25T13:40:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.197+0000 7fa5de17d640 1 -- 172.21.15.27:0/4038291192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d81a2820 con 0x7fa5d8103c20 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5e502c740 1 Processor -- start 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5e502c740 1 -- start start 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5d4ff9640 1 -- 172.21.15.27:0/4038291192 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa5c8004e90 con 0x7fa5d8103c20 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b86410 0x556a86f274a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b5d550 0x556a86f27a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b65730 0x556a86f2e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5e502c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556a86e8cb00 con 0x556a86b65730 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5e502c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556a86e28920 con 0x556a86b5d550 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5e502c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556a86e88ac0 con 0x556a86b86410 2024-04-25T13:40:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.200+0000 7fa5d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b65730 0x556a86f2e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b65730 0x556a86f2e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35212/0 (socket says 172.21.15.27:35212) 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/4094756551 learned_addr learned my addr 172.21.15.27:0/4094756551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b5d550 0x556a86f27a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b86410 0x556a86f274a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/4094756551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b86410 msgr2=0x556a86f274a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b86410 0x556a86f274a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/4094756551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b5d550 msgr2=0x556a86f27a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b5d550 0x556a86f27a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d4ff9640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5c00afaf0 0x7fa5c00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/4094756551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a86c667e0 con 0x556a86b65730 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5c00afaf0 0x7fa5c00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b65730 0x556a86f2e920 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80a8b10 tx=0x7fa5c80a3d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81af9b155920e3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5d4ff9640 1 -- 172.21.15.27:0/4038291192 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5c80df110 con 0x7fa5d8103c20 2024-04-25T13:40:55.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.201+0000 7fa5bdffb640 1 -- 172.21.15.27:0/4094756551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80a73f0 con 0x556a86b65730 2024-04-25T13:40:55.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.202+0000 7fa5bdffb640 1 -- 172.21.15.27:0/4094756551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c80a8950 con 0x556a86b65730 2024-04-25T13:40:55.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.202+0000 7fa5bdffb640 1 -- 172.21.15.27:0/4094756551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8005180 con 0x556a86b65730 2024-04-25T13:40:55.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5c00afaf0 0x7fa5c00b2000 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00094e0 tx=0x7fa5d0031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 -- 172.21.15.27:0/4094756551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b65730 msgr2=0x556a86f2e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b65730 0x556a86f2e920 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80a8b10 tx=0x7fa5c80a3d30 comp rx=0 tx=0).stop 2024-04-25T13:40:55.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 -- 172.21.15.27:0/4094756551 shutdown_connections 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b65730 0x556a86f2e920 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b5d550 0x556a86f27a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 --2- 172.21.15.27:0/4094756551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b86410 0x556a86f274a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 -- 172.21.15.27:0/4094756551 >> 172.21.15.27:0/4094756551 conn(0x556a86b7a560 msgr2=0x556a86f22e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 -- 172.21.15.27:0/4094756551 shutdown_connections 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 -- 172.21.15.27:0/4094756551 wait complete. 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.203+0000 7fa5e502c740 1 -- start start 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5e502c740 1 Processor -- start 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f31270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b65730 0x556a86f317b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b86410 0x556a86f38710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5e502c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556a86f2f370 con 0x556a86b5d550 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5e502c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556a86e265d0 con 0x556a86b86410 2024-04-25T13:40:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5e502c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556a86f2f4f0 con 0x556a86b65730 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5e502c740 10 client.0 _ll_register_callbacks cb 0x556a86e14080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f31270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5d77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f31270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35216/0 (socket says 172.21.15.27:35216) 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.204+0000 7fa5d77fe640 1 -- 172.21.15.27:0/1635871527 learned_addr learned my addr 172.21.15.27:0/1635871527 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b65730 0x556a86f317b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b86410 0x556a86f38710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5d77fe640 1 -- 172.21.15.27:0/1635871527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b65730 msgr2=0x556a86f317b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b65730 0x556a86f317b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5d77fe640 1 -- 172.21.15.27:0/1635871527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b86410 msgr2=0x556a86f38710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b86410 0x556a86f38710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5d77fe640 1 -- 172.21.15.27:0/1635871527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a86cc6f70 con 0x556a86b5d550 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f31270 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009c80 tx=0x7fa5cc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453b2c7ecae0e5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/1635871527 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc005220 con 0x556a86b5d550 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/1635871527 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5cc0053c0 con 0x556a86b5d550 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5e502c740 1 -- 172.21.15.27:0/1635871527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556a86e9a710 con 0x556a86b5d550 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/1635871527 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa5cc022070 con 0x556a86b5d550 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5e502c740 10 client.15312 fetch_fsmap learned FSMap version 19 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5e502c740 1 -- 172.21.15.27:0/1635871527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556a86b5e170 con 0x556a86b5d550 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.205+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/1635871527 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556a86e9a710 con 0x556a86b5d550 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.206+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/1635871527 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa5cc01a030 con 0x556a86b5d550 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.206+0000 7fa5e502c740 10 client.15312 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.206+0000 7fa5e502c740 10 client.15312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.206+0000 7fa5e502c740 20 client.15312 populate_metadata read hostname 'smithi027' 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.206+0000 7fa5e502c740 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a86b451f0 0x556a86f3ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.206+0000 7fa5e502c740 4 client.15312 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.206+0000 7fa5e502c740 1 -- 172.21.15.27:0/1635871527 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a86e94580 con 0x556a86b451f0 2024-04-25T13:40:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.206+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a86b451f0 0x556a86f3ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.207+0000 7fa5b6ffd640 10 client.15312 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.207+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a86b451f0 0x556a86f3ecd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a903a427b8348cf server_cookie=d52287bbc773db75 in_seq=0 out_seq=0 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.207+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/1635871527 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556a86e94580 con 0x556a86b451f0 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.207+0000 7fa5b6ffd640 10 client.15312 handle_command_reply: tid=0 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.207+0000 7fa5e502c740 1 client.15312 shutdown 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 -- 172.21.15.27:0/1635871527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 msgr2=0x556a86f31270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f31270 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009c80 tx=0x7fa5cc0039c0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 -- 172.21.15.27:0/1635871527 shutdown_connections 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a86b451f0 0x556a86f3ecd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b86410 0x556a86f38710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5b6ffd640 0 client.15312 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b65730 0x556a86f317b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 --2- 172.21.15.27:0/1635871527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f31270 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 -- 172.21.15.27:0/1635871527 >> 172.21.15.27:0/1635871527 conn(0x556a86b7a560 msgr2=0x556a86f21ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 -- 172.21.15.27:0/1635871527 shutdown_connections 2024-04-25T13:40:55.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 1 -- 172.21.15.27:0/1635871527 wait complete. 2024-04-25T13:40:55.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.208+0000 7fa5e502c740 20 client.15312 trim_cache size 0 max 16384 2024-04-25T13:40:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.228+0000 7fa5e502c740 1 Processor -- start 2024-04-25T13:40:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.228+0000 7fa5e502c740 1 -- start start 2024-04-25T13:40:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.228+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f31550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.228+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f38630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86f43480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5e502c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81a9120 con 0x556a86b5d550 2024-04-25T13:40:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5e502c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81ab9a0 con 0x556a86b65730 2024-04-25T13:40:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5e502c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81a9790 con 0x556a86b451f0 2024-04-25T13:40:55.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-25T13:40:55.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f31550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f38630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f31550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39670/0 (socket says 172.21.15.27:39670) 2024-04-25T13:40:55.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 -- 172.21.15.27:0/4212718738 learned_addr learned my addr 172.21.15.27:0/4212718738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/4212718738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86f43480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 -- 172.21.15.27:0/4212718738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 msgr2=0x556a86f43480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/4212718738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86f43480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 -- 172.21.15.27:0/4212718738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 msgr2=0x556a86f38630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/4212718738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f38630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 -- 172.21.15.27:0/4212718738 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a86f26fb0 con 0x556a86b451f0 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.229+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/4212718738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f31550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc000c00 tx=0x7fa5cc012890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=224bd83a3992d53e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5bd7fa640 1 -- 172.21.15.27:0/4212718738 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc005040 con 0x556a86b451f0 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5bd7fa640 1 -- 172.21.15.27:0/4212718738 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5cc02b070 con 0x556a86b451f0 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5bd7fa640 1 -- 172.21.15.27:0/4212718738 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc030470 con 0x556a86b451f0 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5e502c740 1 -- 172.21.15.27:0/4212718738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 msgr2=0x556a86f31550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5e502c740 1 --2- 172.21.15.27:0/4212718738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f31550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc000c00 tx=0x7fa5cc012890 comp rx=0 tx=0).stop 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5e502c740 1 -- 172.21.15.27:0/4212718738 shutdown_connections 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5e502c740 1 --2- 172.21.15.27:0/4212718738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86f43480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5e502c740 1 --2- 172.21.15.27:0/4212718738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f38630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5e502c740 1 --2- 172.21.15.27:0/4212718738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f31550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.230+0000 7fa5e502c740 1 -- 172.21.15.27:0/4212718738 >> 172.21.15.27:0/4212718738 conn(0x556a86b7a560 msgr2=0x556a86f30e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 -- 172.21.15.27:0/4212718738 shutdown_connections 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 -- 172.21.15.27:0/4212718738 wait complete. 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 -- start start 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 Processor -- start 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f35410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f35950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86fa5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81a9e60 con 0x556a86b5d550 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81ab2d0 con 0x556a86b65730 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81aa530 con 0x556a86b451f0 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5e502c740 10 client.0 _ll_register_callbacks cb 0x7fa5cc0098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.231+0000 7fa5d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f35950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f35950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35222/0 (socket says 172.21.15.27:35222) 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/3934905423 learned_addr learned my addr 172.21.15.27:0/3934905423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86fa5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f35410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/3934905423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 msgr2=0x556a86f35410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f35410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 msgr2=0x556a86fa5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86fa5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d6ffd640 1 -- 172.21.15.27:0/3934905423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a86f388a0 con 0x556a86b5d550 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f35410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d7fff640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86fa5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5d6ffd640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f35950 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80ca0f0 tx=0x7fa5c8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb86a8e5bd5da418 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/3934905423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80a6d90 con 0x556a86b5d550 2024-04-25T13:40:55.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/3934905423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5c80e29c0 con 0x556a86b5d550 2024-04-25T13:40:55.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5e502c740 1 -- 172.21.15.27:0/3934905423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5cc02b070 con 0x556a86b5d550 2024-04-25T13:40:55.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/3934905423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa5c80a6f70 con 0x556a86b5d550 2024-04-25T13:40:55.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5e502c740 10 client.15318 fetch_fsmap learned FSMap version 19 2024-04-25T13:40:55.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5e502c740 1 -- 172.21.15.27:0/3934905423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556a86fa6510 con 0x556a86b5d550 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.232+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/3934905423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5cc02b070 con 0x556a86b5d550 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/3934905423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa5c80a38b0 con 0x556a86b5d550 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5e502c740 10 client.15318 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5e502c740 10 client.15318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5e502c740 20 client.15318 populate_metadata read hostname 'smithi027' 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5e502c740 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a86b86410 0x556a86fa8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5e502c740 4 client.15318 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5e502c740 1 -- 172.21.15.27:0/3934905423 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556a86e0fac0 con 0x556a86b86410 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a86b86410 0x556a86fa8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5d77fe640 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a86b86410 0x556a86fa8a80 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3a906e82d6820a9 server_cookie=a97d7b93b484ee2e in_seq=0 out_seq=0 2024-04-25T13:40:55.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.233+0000 7fa5b6ffd640 10 client.15318 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/3934905423 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556a86e0fac0 con 0x556a86b86410 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5b6ffd640 10 client.15318 handle_command_reply: tid=0 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 client.15318 shutdown 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 -- 172.21.15.27:0/3934905423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 msgr2=0x556a86f35950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f35950 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80ca0f0 tx=0x7fa5c8009310 comp rx=0 tx=0).stop 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 -- 172.21.15.27:0/3934905423 shutdown_connections 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a86b86410 0x556a86fa8a80 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a86b65730 0x556a86fa5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a86b5d550 0x556a86f35950 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 --2- 172.21.15.27:0/3934905423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a86b451f0 0x556a86f35410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 -- 172.21.15.27:0/3934905423 >> 172.21.15.27:0/3934905423 conn(0x556a86b7a560 msgr2=0x556a86f25c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5b6ffd640 0 client.15318 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 -- 172.21.15.27:0/3934905423 shutdown_connections 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 1 -- 172.21.15.27:0/3934905423 wait complete. 2024-04-25T13:40:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.235+0000 7fa5e502c740 20 client.15318 trim_cache size 0 max 16384 2024-04-25T13:40:55.242 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:40:55.242 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:40:55.242 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "96e5208a-eb95-40ef-ae97-abbc06e2cf91", 2024-04-25T13:40:55.242 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:40:55.242 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 -- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5c00afaf0 msgr2=0x7fa5c00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5c00afaf0 0x7fa5c00b2000 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00094e0 tx=0x7fa5d0031af0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 -- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8103c20 msgr2=0x7fa5d819aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8103c20 0x7fa5d819aa30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800e9f0 tx=0x7fa5c800edf0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 -- 172.21.15.27:0/4038291192 shutdown_connections 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5c00afaf0 0x7fa5c00b2000 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5d8108920 0x7fa5d81a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5d8103c20 0x7fa5d819aa30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 --2- 172.21.15.27:0/4038291192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5d81029e0 0x7fa5d819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 -- 172.21.15.27:0/4038291192 >> 172.21.15.27:0/4038291192 conn(0x7fa5d80fdbd0 msgr2=0x7fa5d8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 -- 172.21.15.27:0/4038291192 shutdown_connections 2024-04-25T13:40:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.236+0000 7fa5de17d640 1 -- 172.21.15.27:0/4038291192 wait complete. 2024-04-25T13:40:55.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:40:55.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:40:55.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-25T13:40:55.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:55 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3302346730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:55.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:55 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2955710504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:55.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:55 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1302536589' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:55.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:55 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3302346730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:55.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:55 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2955710504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:55.331 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:55 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1302536589' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:55.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 -- 172.21.15.27:0/486912797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 msgr2=0x7ff9b010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 --2- 172.21.15.27:0/486912797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 0x7ff9b010ad40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009ca0 tx=0x7ff9a802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 -- 172.21.15.27:0/486912797 shutdown_connections 2024-04-25T13:40:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 --2- 172.21.15.27:0/486912797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 0x7ff9b010ad40 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 --2- 172.21.15.27:0/486912797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9b0103c20 0x7ff9b0104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 --2- 172.21.15.27:0/486912797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9b01029e0 0x7ff9b01036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 -- 172.21.15.27:0/486912797 >> 172.21.15.27:0/486912797 conn(0x7ff9b00fdbd0 msgr2=0x7ff9b0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 -- 172.21.15.27:0/486912797 shutdown_connections 2024-04-25T13:40:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.426+0000 7ff9b4900640 1 -- 172.21.15.27:0/486912797 wait complete. 2024-04-25T13:40:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9b4900640 1 Processor -- start 2024-04-25T13:40:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9b4900640 1 -- start start 2024-04-25T13:40:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9b4900640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9b01029e0 0x7ff9b0199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9b4900640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9b0103c20 0x7ff9b0199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9b4900640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 0x7ff9b01a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9b4900640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b010e510 con 0x7ff9b0108920 2024-04-25T13:40:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9b4900640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b010e390 con 0x7ff9b01029e0 2024-04-25T13:40:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9b4900640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b010e690 con 0x7ff9b0103c20 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9aed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 0x7ff9b01a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9aed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 0x7ff9b01a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35254/0 (socket says 172.21.15.27:35254) 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9aed9d640 1 -- 172.21.15.27:0/2367524470 learned_addr learned my addr 172.21.15.27:0/2367524470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9add9b640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9b0103c20 0x7ff9b0199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.427+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9b01029e0 0x7ff9b0199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9aed9d640 1 -- 172.21.15.27:0/2367524470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9b0103c20 msgr2=0x7ff9b0199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9aed9d640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9b0103c20 0x7ff9b0199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9aed9d640 1 -- 172.21.15.27:0/2367524470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9b01029e0 msgr2=0x7ff9b0199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9aed9d640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9b01029e0 0x7ff9b0199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9aed9d640 1 -- 172.21.15.27:0/2367524470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b01a0f50 con 0x7ff9b0108920 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9aed9d640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 0x7ff9b01a08e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80097b0 tx=0x7ff9a8031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9a77fe640 1 -- 172.21.15.27:0/2367524470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8003bf0 con 0x7ff9b0108920 2024-04-25T13:40:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9b4900640 1 -- 172.21.15.27:0/2367524470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b01a11e0 con 0x7ff9b0108920 2024-04-25T13:40:55.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9a77fe640 1 -- 172.21.15.27:0/2367524470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9a8003d90 con 0x7ff9b0108920 2024-04-25T13:40:55.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9b4900640 1 -- 172.21.15.27:0/2367524470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b01a1780 con 0x7ff9b0108920 2024-04-25T13:40:55.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.428+0000 7ff9a77fe640 1 -- 172.21.15.27:0/2367524470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a802fc10 con 0x7ff9b0108920 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.431+0000 7ff9a77fe640 1 -- 172.21.15.27:0/2367524470 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff9a80305c0 con 0x7ff9b0108920 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.431+0000 7ff9bb7b4740 1 Processor -- start 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.431+0000 7ff9bb7b4740 1 -- start start 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.431+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e3473390 0x5565e3814a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e344a4d0 0x5565e3814fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3815c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5565e3779d80 con 0x5565e34526b0 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5565e37338b0 con 0x5565e344a4d0 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5565e3775d40 con 0x5565e3473390 2024-04-25T13:40:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9add9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3815c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9add9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3815c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35268/0 (socket says 172.21.15.27:35268) 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9add9b640 1 -- 172.21.15.27:0/2993904631 learned_addr learned my addr 172.21.15.27:0/2993904631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e344a4d0 0x5565e3814fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9add9b640 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e3473390 0x5565e3814a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9a77fe640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff99c0afaf0 0x7ff99c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9aed9d640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff99c0afaf0 0x7ff99c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9ae59c640 1 -- 172.21.15.27:0/2993904631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e3473390 msgr2=0x5565e3814a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e3473390 0x5565e3814a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9ae59c640 1 -- 172.21.15.27:0/2993904631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 msgr2=0x5565e3815c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3815c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9ae59c640 1 -- 172.21.15.27:0/2993904631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565e3704a40 con 0x5565e344a4d0 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9add9b640 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3815c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.432+0000 7ff9a77fe640 1 -- 172.21.15.27:0/2367524470 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff9a81001a0 con 0x7ff9b0108920 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.433+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e344a4d0 0x5565e3814fe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9b01022b0 tx=0x7ff998011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e35c0e905413a0f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.433+0000 7ff9aed9d640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff99c0afaf0 0x7ff99c0b2000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80fdd20 tx=0x7ff9a808f070 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.433+0000 7ff9a4ff9640 1 -- 172.21.15.27:0/2993904631 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998004030 con 0x5565e344a4d0 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.433+0000 7ff9a4ff9640 1 -- 172.21.15.27:0/2993904631 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9980035f0 con 0x5565e344a4d0 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.433+0000 7ff9a4ff9640 1 -- 172.21.15.27:0/2993904631 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998016650 con 0x5565e344a4d0 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2993904631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e344a4d0 msgr2=0x5565e3814fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e344a4d0 0x5565e3814fe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9b01022b0 tx=0x7ff998011810 comp rx=0 tx=0).stop 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2993904631 shutdown_connections 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3815c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e344a4d0 0x5565e3814fe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2993904631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e3473390 0x5565e3814a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2993904631 >> 172.21.15.27:0/2993904631 conn(0x5565e34674e0 msgr2=0x5565e3810270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2993904631 shutdown_connections 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2993904631 wait complete. 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- start start 2024-04-25T13:40:55.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 Processor -- start 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e381e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 0x5565e381ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3825d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5565e3816780 con 0x5565e3473390 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5565e3700d10 con 0x5565e34526b0 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.434+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5565e3816900 con 0x5565e344a4d0 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9bb7b4740 10 client.0 _ll_register_callbacks cb 0x5565e372d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3825d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3825d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35280/0 (socket says 172.21.15.27:35280) 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 -- 172.21.15.27:0/2659168690 learned_addr learned my addr 172.21.15.27:0/2659168690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 0x5565e381ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e381e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 -- 172.21.15.27:0/2659168690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 msgr2=0x5565e381e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e381e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 -- 172.21.15.27:0/2659168690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 msgr2=0x5565e381ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 0x5565e381ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 -- 172.21.15.27:0/2659168690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565e35557d0 con 0x5565e3473390 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9add9b640 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3825d60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0012c60 tx=0x7ff9a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d802c225da28ff0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff995ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff995ffb640 1 -- 172.21.15.27:0/2659168690 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0004250 con 0x5565e3473390 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2659168690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9980035f0 con 0x5565e3473390 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.435+0000 7ff995ffb640 1 -- 172.21.15.27:0/2659168690 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff9a00043f0 con 0x5565e3473390 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff995ffb640 1 -- 172.21.15.27:0/2659168690 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff9a0005020 con 0x5565e3473390 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff9bb7b4740 10 client.15330 fetch_fsmap learned FSMap version 19 2024-04-25T13:40:55.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2659168690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5565e348b3f0 con 0x5565e3473390 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff995ffb640 1 -- 172.21.15.27:0/2659168690 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9980035f0 con 0x5565e3473390 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff995ffb640 1 -- 172.21.15.27:0/2659168690 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff9a001a040 con 0x5565e3473390 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff9bb7b4740 10 client.15330 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff9bb7b4740 10 client.15330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff9bb7b4740 20 client.15330 populate_metadata read hostname 'smithi027' 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5565e3431f60 0x5565e382c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff9bb7b4740 4 client.15330 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.436+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2659168690 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565e3787990 con 0x5565e3431f60 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.437+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5565e3431f60 0x5565e382c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.437+0000 7ff995ffb640 10 client.15330 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.437+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5565e3431f60 0x5565e382c1c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7db3a41ac2e909fd server_cookie=a3c5605fd7616f48 in_seq=0 out_seq=0 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff995ffb640 1 -- 172.21.15.27:0/2659168690 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5565e3787990 con 0x5565e3431f60 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff995ffb640 10 client.15330 handle_command_reply: tid=0 2024-04-25T13:40:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 client.15330 shutdown 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2659168690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 msgr2=0x5565e3825d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3825d60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0012c60 tx=0x7ff9a0004730 comp rx=0 tx=0).stop 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2659168690 shutdown_connections 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5565e3431f60 0x5565e382c1c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3825d60 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 0x5565e381ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff995ffb640 0 client.15330 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/2659168690 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e381e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2659168690 >> 172.21.15.27:0/2659168690 conn(0x5565e34674e0 msgr2=0x5565e3813180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2659168690 shutdown_connections 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/2659168690 wait complete. 2024-04-25T13:40:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.438+0000 7ff9bb7b4740 20 client.15330 trim_cache size 0 max 16384 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.456+0000 7ff9bb7b4740 1 Processor -- start 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.456+0000 7ff9bb7b4740 1 -- start start 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.456+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e380fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.456+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3810150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e3473390 0x5565e3813070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b01a6f60 con 0x5565e34526b0 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b01a97e0 con 0x5565e3473390 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b01a75d0 con 0x5565e344a4d0 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3810150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3810150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35294/0 (socket says 172.21.15.27:35294) 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 -- 172.21.15.27:0/3849855053 learned_addr learned my addr 172.21.15.27:0/3849855053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9add9b640 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e380fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9add9b640 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e3473390 0x5565e3813070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 -- 172.21.15.27:0/3849855053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 msgr2=0x5565e380fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e380fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 -- 172.21.15.27:0/3849855053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e3473390 msgr2=0x5565e3813070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e3473390 0x5565e3813070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 -- 172.21.15.27:0/3849855053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565e344b0f0 con 0x5565e34526b0 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9add9b640 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e3473390 0x5565e3813070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3810150 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x5565e3814780 tx=0x7ff998005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c505d37ceeef8e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.457+0000 7ff997fff640 1 -- 172.21.15.27:0/3849855053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9980035f0 con 0x5565e34526b0 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff997fff640 1 -- 172.21.15.27:0/3849855053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9980054f0 con 0x5565e34526b0 2024-04-25T13:40:55.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff997fff640 1 -- 172.21.15.27:0/3849855053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9980167d0 con 0x5565e34526b0 2024-04-25T13:40:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3849855053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 msgr2=0x5565e3810150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3810150 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x5565e3814780 tx=0x7ff998005f90 comp rx=0 tx=0).stop 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3849855053 shutdown_connections 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e3473390 0x5565e3813070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e34526b0 0x5565e3810150 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3849855053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e380fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3849855053 >> 172.21.15.27:0/3849855053 conn(0x5565e34674e0 msgr2=0x5565e38195e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3849855053 shutdown_connections 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.458+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3849855053 wait complete. 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 1 -- start start 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 1 Processor -- start 2024-04-25T13:40:55.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e3822990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 0x5565e3822ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3823d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b01a7ca0 con 0x5565e3473390 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b01a9110 con 0x5565e34526b0 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b01a8370 con 0x5565e344a4d0 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9bb7b4740 10 client.0 _ll_register_callbacks cb 0x7ff9a0012580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3823d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3823d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35304/0 (socket says 172.21.15.27:35304) 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 -- 172.21.15.27:0/3562627826 learned_addr learned my addr 172.21.15.27:0/3562627826 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 0x5565e3822ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e3822990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 -- 172.21.15.27:0/3562627826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 msgr2=0x5565e3822990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e3822990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 -- 172.21.15.27:0/3562627826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 msgr2=0x5565e3822ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 0x5565e3822ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.459+0000 7ff9add9b640 1 -- 172.21.15.27:0/3562627826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565e373a4b0 con 0x5565e3473390 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff9add9b640 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3823d60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0012c60 tx=0x7ff9a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27601d3590f3a6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff995ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff995ffb640 1 -- 172.21.15.27:0/3562627826 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00055d0 con 0x5565e3473390 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff995ffb640 1 -- 172.21.15.27:0/3562627826 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff9a00047a0 con 0x5565e3473390 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3562627826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9980054f0 con 0x5565e3473390 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff995ffb640 1 -- 172.21.15.27:0/3562627826 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff9a00127d0 con 0x5565e3473390 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff9bb7b4740 10 client.15336 fetch_fsmap learned FSMap version 19 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3562627826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5565e3825380 con 0x5565e3473390 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff995ffb640 1 -- 172.21.15.27:0/3562627826 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9980054f0 con 0x5565e3473390 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff995ffb640 1 -- 172.21.15.27:0/3562627826 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff9a0042a50 con 0x5565e3473390 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff9bb7b4740 10 client.15336 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:40:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff9bb7b4740 10 client.15336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.460+0000 7ff9bb7b4740 20 client.15336 populate_metadata read hostname 'smithi027' 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.461+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5565e34073a0 0x5565e3896980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.461+0000 7ff9bb7b4740 4 client.15336 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.461+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3562627826 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5565e3781800 con 0x5565e34073a0 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.461+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5565e34073a0 0x5565e3896980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.461+0000 7ff995ffb640 10 client.15336 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.461+0000 7ff9ae59c640 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5565e34073a0 0x5565e3896980 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=678961ed3ad447f4 server_cookie=21f117f8aef5bd72 in_seq=0 out_seq=0 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff995ffb640 1 -- 172.21.15.27:0/3562627826 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5565e3781800 con 0x5565e34073a0 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff995ffb640 10 client.15336 handle_command_reply: tid=0 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 client.15336 shutdown 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3562627826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 msgr2=0x5565e3823d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3823d60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0012c60 tx=0x7ff9a0004730 comp rx=0 tx=0).stop 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3562627826 shutdown_connections 2024-04-25T13:40:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5565e34073a0 0x5565e3896980 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565e3473390 0x5565e3823d60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565e34526b0 0x5565e3822ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff995ffb640 0 client.15336 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 --2- 172.21.15.27:0/3562627826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5565e344a4d0 0x5565e3822990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3562627826 >> 172.21.15.27:0/3562627826 conn(0x5565e34674e0 msgr2=0x5565e380fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3562627826 shutdown_connections 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 1 -- 172.21.15.27:0/3562627826 wait complete. 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9bb7b4740 20 client.15336 trim_cache size 0 max 16384 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:40:55.468 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:40:55.469 INFO:teuthology.orchestra.run.smithi027.stdout: "96e5208a-eb95-40ef-ae97-abbc06e2cf91": { 2024-04-25T13:40:55.469 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:40:55.469 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "96e5208a-eb95-40ef-ae97-abbc06e2cf91", 2024-04-25T13:40:55.469 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:40:55.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:40:55.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:40:55.469 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9b4900640 1 -- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff99c0afaf0 msgr2=0x7ff99c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9b4900640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff99c0afaf0 0x7ff99c0b2000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80fdd20 tx=0x7ff9a808f070 comp rx=0 tx=0).stop 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9b4900640 1 -- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 msgr2=0x7ff9b01a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.462+0000 7ff9b4900640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 0x7ff9b01a08e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80097b0 tx=0x7ff9a8031840 comp rx=0 tx=0).stop 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.463+0000 7ff9b4900640 1 -- 172.21.15.27:0/2367524470 shutdown_connections 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.463+0000 7ff9b4900640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff99c0afaf0 0x7ff99c0b2000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.463+0000 7ff9b4900640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9b0108920 0x7ff9b01a08e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.463+0000 7ff9b4900640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9b0103c20 0x7ff9b0199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.463+0000 7ff9b4900640 1 --2- 172.21.15.27:0/2367524470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9b01029e0 0x7ff9b0199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.463+0000 7ff9b4900640 1 -- 172.21.15.27:0/2367524470 >> 172.21.15.27:0/2367524470 conn(0x7ff9b00fdbd0 msgr2=0x7ff9b0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.463+0000 7ff9b4900640 1 -- 172.21.15.27:0/2367524470 shutdown_connections 2024-04-25T13:40:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:40:55.463+0000 7ff9b4900640 1 -- 172.21.15.27:0/2367524470 wait complete. 2024-04-25T13:40:55.474 INFO:tasks.cephfs.filesystem:scrub status for tag:96e5208a-eb95-40ef-ae97-abbc06e2cf91 - {'path': '/', 'tag': '96e5208a-eb95-40ef-ae97-abbc06e2cf91', 'options': 'recursive,force'} 2024-04-25T13:40:55.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:55 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3302346730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:40:55.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:55 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2955710504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:40:55.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:55 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1302536589' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:40:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:56 smithi045 ceph-mon[32731]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-25T13:40:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:56 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:40:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:56 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:40:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:56 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:40:56.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:56 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:40:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:56 smithi027 ceph-mon[29140]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-25T13:40:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:56 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:40:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:56 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:40:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:56 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:40:56.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:56 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:40:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:56 smithi122 ceph-mon[32843]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-25T13:40:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:56 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:40:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:56 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:40:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:56 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:40:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:56 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:40:56.723 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-25T13:40:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-25T13:40:56.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-25T13:40:56.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T13:40:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/: 2024-04-25T13:40:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:40:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:40:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 .. 2024-04-25T13:40:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 data 2024-04-25T13:40:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:40:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data: 2024-04-25T13:40:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:40:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:40:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 .. 2024-04-25T13:40:56.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-25T13:40:56.972 INFO:teuthology.orchestra.run.smithi027.stdout:[Service] 2024-04-25T13:40:56.972 INFO:teuthology.orchestra.run.smithi027.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-25T13:40:56.972 INFO:teuthology.orchestra.run.smithi027.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-25T13:40:56.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-25T13:40:57.099 INFO:teuthology.orchestra.run.smithi027.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-25T13:40:58.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:58 smithi045 ceph-mon[32731]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-25T13:40:58.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:58 smithi027 ceph-mon[29140]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-25T13:40:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:58 smithi122 ceph-mon[32843]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-25T13:40:59.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:40:59 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:40:59.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:40:59 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:40:59.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:40:59 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:41:00.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:00 smithi045 ceph-mon[32731]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-25T13:41:00.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:00 smithi027 ceph-mon[29140]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-25T13:41:00.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:00 smithi122 ceph-mon[32843]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-25T13:41:01.448 INFO:teuthology.orchestra.run.smithi027.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-25T13:41:01.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T13:41:01.490 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/: 2024-04-25T13:41:01.490 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 .. 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 data 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data: 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:total 35 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 . 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 .. 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 base 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 25 13:40 global 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 log 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_commit_ts 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_dynshmem 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 25 13:40 pg_hba.conf 2024-04-25T13:41:01.491 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 25 13:40 pg_ident.conf 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:41 pg_logical 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 pg_multixact 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_notify 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_replslot 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_serial 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_snapshots 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_stat 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_stat_tmp 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 pg_subtrans 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_tblspc 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_twophase 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 PG_VERSION 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 pg_wal 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 pg_xact 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 25 13:40 postgresql.auto.conf 2024-04-25T13:41:01.492 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 25 13:40 postgresql.conf 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base: 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 . 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:40 1 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:40 13493 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:41 13494 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:total 7777 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:40 . 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 .. 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 112 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 113 2024-04-25T13:41:01.493 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:40 1247 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1247_fsm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1247_vm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 13:40 1249 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1249_fsm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1249_vm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 13:40 1255 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1255_fsm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1255_vm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 13:40 1259 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1259_fsm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1259_vm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 13:40 13344 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13344_fsm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13344_vm 2024-04-25T13:41:01.494 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13346 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13348 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13349_fsm 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349_vm 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13351 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13353 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13354 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13354_fsm 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13354_vm 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13356 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13358 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13359_fsm 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359_vm 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13361 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13363 2024-04-25T13:41:01.495 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1417 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1418 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 174 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 175 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2187 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2224 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2328 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2336 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2337 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2579 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2600 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2600_fsm 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2600_vm 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2601_fsm 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601_vm 2024-04-25T13:41:01.496 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2602 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2602_fsm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2602_vm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2603 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2603_fsm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2603_vm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2604 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2605 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2605_fsm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2605_vm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2606 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2606_fsm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2606_vm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2607 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2607_fsm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2607_vm 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 13:40 2608 2024-04-25T13:41:01.497 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2608_fsm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2608_vm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 13:40 2609 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2609_fsm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2609_vm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2610 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2610_fsm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2610_vm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2611 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2612_fsm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612_vm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2613 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2615_fsm 2024-04-25T13:41:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615_vm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2616 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2616_fsm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2616_vm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 13:40 2617 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2617_fsm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2617_vm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 13:40 2618 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2618_fsm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2618_vm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 13:40 2619 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2619_fsm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2619_vm 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2620 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2650 2024-04-25T13:41:01.499 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2651 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2652 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2653 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2654 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2655 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2656 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2657 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 13:40 2658 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:40 2659 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2660 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2661 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2662 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2663 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2664 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2665 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2666 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2667 2024-04-25T13:41:01.500 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2668 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2669 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2670 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 13:40 2673 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 13:40 2674 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 13:40 2675 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2678 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2679 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2680 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2681 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2682 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2683 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2684 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2685 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2686 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2687 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2688 2024-04-25T13:41:01.501 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2689 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 13:40 2690 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 13:40 2691 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2692 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2693 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2696 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2699 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2701 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2702 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2703 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2704 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2753 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2753_fsm 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2753_vm 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2754 2024-04-25T13:41:01.502 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2755 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2756 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2757 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2830 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2831 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2832 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2833 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2834 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2835 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2836 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2837 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 13:40 2838 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2838_fsm 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2838_vm 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2839 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2840 2024-04-25T13:41:01.503 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2840_fsm 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2840_vm 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2841 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2995 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2996 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3079 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3079_fsm 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3079_vm 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3080 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3081 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 3085 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3118 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3119 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 13:40 3164 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3256 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3257 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3258 2024-04-25T13:41:01.504 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3350 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3351 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3379 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3380 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3381 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3394 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3394_fsm 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3394_vm 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3395 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3429 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3430 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3431 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3433 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3439 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3440 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 3455 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 13:40 3456 2024-04-25T13:41:01.505 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3456_fsm 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3456_vm 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3466 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3467 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3468 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3501 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3502 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3503 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3534 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3541_fsm 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541_vm 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3542 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3574 2024-04-25T13:41:01.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3575 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3576 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3596 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3597 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3598 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3599 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3600_fsm 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600_vm 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3601_fsm 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601_vm 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3602_fsm 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602_vm 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3603 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3603_fsm 2024-04-25T13:41:01.507 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3603_vm 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3604 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3605 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3606 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3607 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3608 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 3609 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3712 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3764 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3764_fsm 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3764_vm 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3766 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3767 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3997 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4143 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4144 2024-04-25T13:41:01.508 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4145 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4146 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4147 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4148 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4149 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4150 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4151 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4152 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4153 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4154 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4155 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4156 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4157 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4158 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4159 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4160 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4161 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4162 2024-04-25T13:41:01.509 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4163 2024-04-25T13:41:01.510 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4164 2024-04-25T13:41:01.510 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4165 2024-04-25T13:41:01.512 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4166 2024-04-25T13:41:01.512 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4167 2024-04-25T13:41:01.512 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4168 2024-04-25T13:41:01.512 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4169 2024-04-25T13:41:01.512 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4170 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4171 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4172 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4173 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4174 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 5002 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 548 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 549 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6102 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6104 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6106 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6110 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6111 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6112 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6113 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6117 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 826 2024-04-25T13:41:01.513 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 827 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 828 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 13:40 pg_filenode.map 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 PG_VERSION 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:total 7777 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:40 . 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 .. 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 112 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 113 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:40 1247 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1247_fsm 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1247_vm 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 13:40 1249 2024-04-25T13:41:01.514 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1249_fsm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1249_vm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 13:40 1255 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1255_fsm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1255_vm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 13:40 1259 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1259_fsm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1259_vm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 13:40 13344 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13344_fsm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13344_vm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13346 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13348 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13349_fsm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349_vm 2024-04-25T13:41:01.515 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13351 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13353 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13354 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13354_fsm 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13354_vm 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13356 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13358 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13359_fsm 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359_vm 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13361 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13363 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1417 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1418 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 174 2024-04-25T13:41:01.516 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 175 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2187 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2224 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2328 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2336 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2337 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2579 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2600 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2600_fsm 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2600_vm 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2601_fsm 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601_vm 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2602 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2602_fsm 2024-04-25T13:41:01.517 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2602_vm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2603 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2603_fsm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2603_vm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2604 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2605 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2605_fsm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2605_vm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2606 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2606_fsm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2606_vm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2607 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2607_fsm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2607_vm 2024-04-25T13:41:01.518 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 13:40 2608 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2608_fsm 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2608_vm 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 13:40 2609 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2609_fsm 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2609_vm 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2610 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2610_fsm 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2610_vm 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2611 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2612_fsm 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612_vm 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2613 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615 2024-04-25T13:41:01.519 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2615_fsm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615_vm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2616 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2616_fsm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2616_vm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 13:40 2617 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2617_fsm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2617_vm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 13:40 2618 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2618_fsm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2618_vm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 13:40 2619 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2619_fsm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2619_vm 2024-04-25T13:41:01.520 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2620 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2650 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2651 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2652 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2653 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2654 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2655 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2656 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2657 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 13:40 2658 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:40 2659 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2660 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2661 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2662 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2663 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2664 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2665 2024-04-25T13:41:01.521 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2666 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2667 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2668 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2669 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2670 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 13:40 2673 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 13:40 2674 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 13:40 2675 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2678 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2679 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2680 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2681 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2682 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2683 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2684 2024-04-25T13:41:01.522 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2685 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2686 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2687 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2688 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2689 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 13:40 2690 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 13:40 2691 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2692 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2693 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2696 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2699 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2701 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2702 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2703 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2704 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2753 2024-04-25T13:41:01.523 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2753_fsm 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2753_vm 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2754 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2755 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2756 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2757 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2830 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2831 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2832 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2833 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2834 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2835 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2836 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2837 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 13:40 2838 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2838_fsm 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2838_vm 2024-04-25T13:41:01.524 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2839 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2840 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2840_fsm 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2840_vm 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2841 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2995 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2996 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3079 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3079_fsm 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3079_vm 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3080 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3081 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 3085 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3118 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3119 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 13:40 3164 2024-04-25T13:41:01.525 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3256 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3257 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3258 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3350 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3351 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3379 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3380 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3381 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3394 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3394_fsm 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3394_vm 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3395 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3429 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3430 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3431 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3433 2024-04-25T13:41:01.526 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3439 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3440 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 3455 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 13:40 3456 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3456_fsm 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3456_vm 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3466 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3467 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3468 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3501 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3502 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3503 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3534 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3541_fsm 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541_vm 2024-04-25T13:41:01.527 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3542 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3574 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3575 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3576 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3596 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3597 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3598 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3599 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3600_fsm 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600_vm 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3601_fsm 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601_vm 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3602_fsm 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602_vm 2024-04-25T13:41:01.528 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3603 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3603_fsm 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3603_vm 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3604 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3605 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3606 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3607 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3608 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 3609 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3712 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3764 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3764_fsm 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3764_vm 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3766 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3767 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3997 2024-04-25T13:41:01.529 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4143 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4144 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4145 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4146 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4147 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4148 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4149 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4150 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4151 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4152 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4153 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4154 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4155 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4156 2024-04-25T13:41:01.530 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4157 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4158 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4159 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4160 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4161 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4162 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4163 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4164 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4165 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4166 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4167 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4168 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4169 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4170 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4171 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4172 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4173 2024-04-25T13:41:01.531 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4174 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 5002 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 548 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 549 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6102 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6104 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6106 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6110 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6111 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6112 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6113 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6117 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 826 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 827 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 828 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 13:40 pg_filenode.map 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 PG_VERSION 2024-04-25T13:41:01.532 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:total 7777 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:41 . 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 .. 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 112 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 113 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:41 1247 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1247_fsm 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1247_vm 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 13:41 1249 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1249_fsm 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1249_vm 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 13:40 1255 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 1255_fsm 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1255_vm 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 13:41 1259 2024-04-25T13:41:01.533 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 1259_fsm 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1259_vm 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 13:40 13344 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13344_fsm 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13344_vm 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13346 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13348 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13349_fsm 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13349_vm 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 13351 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13353 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13354 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13354_fsm 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13354_vm 2024-04-25T13:41:01.534 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13356 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13358 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 13359_fsm 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359_vm 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 13361 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13363 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1417 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 1418 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 174 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 175 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2187 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2224 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2328 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2336 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2337 2024-04-25T13:41:01.535 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2579 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2600 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2600_fsm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2600_vm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2601_fsm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601_vm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2602 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2602_fsm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2602_vm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2603 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2603_fsm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2603_vm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2604 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2605 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2605_fsm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2605_vm 2024-04-25T13:41:01.536 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2606 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2606_fsm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2606_vm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2607 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2607_fsm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2607_vm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 13:41 2608 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2608_fsm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2608_vm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 13:40 2609 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2609_fsm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2609_vm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2610 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2610_fsm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2610_vm 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2611 2024-04-25T13:41:01.537 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2612_fsm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612_vm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2613 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2615_fsm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615_vm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2616 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2616_fsm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2616_vm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 13:40 2617 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2617_fsm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2617_vm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 13:40 2618 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2618_fsm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2618_vm 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 13:40 2619 2024-04-25T13:41:01.538 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2619_fsm 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2619_vm 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2620 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2650 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2651 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2652 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 2653 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 2654 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2655 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2656 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2657 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 13:40 2658 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:41 2659 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2660 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2661 2024-04-25T13:41:01.539 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2662 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:41 2663 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2664 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2665 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2666 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2667 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2668 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2669 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2670 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 13:41 2673 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 13:41 2674 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 13:40 2675 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2678 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2679 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2680 2024-04-25T13:41:01.540 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2681 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2682 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2683 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2684 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2685 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2686 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2687 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2688 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2689 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 13:41 2690 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 13:41 2691 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2692 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2693 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2696 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2699 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2701 2024-04-25T13:41:01.541 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2702 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2703 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 2704 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2753 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2753_fsm 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2753_vm 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2754 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2755 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2756 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2757 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2830 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2831 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2832 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2833 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2834 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2835 2024-04-25T13:41:01.542 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2836 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2837 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 13:41 2838 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2838_fsm 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2838_vm 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2839 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2840 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2840_fsm 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2840_vm 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2841 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2995 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2996 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3079 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3079_fsm 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3079_vm 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3080 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3081 2024-04-25T13:41:01.543 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 3085 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3118 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3119 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 13:41 3164 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3256 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3257 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3258 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3350 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3351 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3379 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3380 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3381 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3394 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3394_fsm 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3394_vm 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3395 2024-04-25T13:41:01.544 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3429 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3430 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3431 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3433 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3439 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3440 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 3455 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 13:40 3456 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3456_fsm 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3456_vm 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3466 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3467 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3468 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3501 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3502 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3503 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3534 2024-04-25T13:41:01.545 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 3541_fsm 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3541_vm 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3542 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3574 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3575 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3576 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3596 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3597 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3598 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3599 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3600_fsm 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600_vm 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601 2024-04-25T13:41:01.546 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3601_fsm 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3601_vm 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 3602_fsm 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3602_vm 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 3603 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 3603_fsm 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3603_vm 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3604 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3605 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3606 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3607 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3608 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 3609 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3712 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3764 2024-04-25T13:41:01.547 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3764_fsm 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3764_vm 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3766 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3767 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3997 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4143 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4144 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4145 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4146 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4147 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4148 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4149 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4150 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4151 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4152 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4153 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4154 2024-04-25T13:41:01.548 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4155 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4156 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4157 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4158 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4159 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4160 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4161 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4162 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4163 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4164 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4165 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4166 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4167 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4168 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4169 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4170 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4171 2024-04-25T13:41:01.549 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4172 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4173 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4174 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 5002 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 548 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 549 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6102 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6104 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 6106 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 6110 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 6111 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 6112 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6113 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6117 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 826 2024-04-25T13:41:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 827 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 828 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 13:41 pg_filenode.map 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:41 PG_VERSION 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/global: 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:total 537 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 25 13:40 . 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1213 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1213_fsm 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1213_vm 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1214 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1214_fsm 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1214_vm 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 1232 2024-04-25T13:41:01.551 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 1233 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1260 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1260_fsm 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1260_vm 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1261 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1261_fsm 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1261_vm 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 1262 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1262_fsm 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1262_vm 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2396 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2396_fsm 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2396_vm 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2397 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2671 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2672 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2676 2024-04-25T13:41:01.552 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2677 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2694 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2695 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2697 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2698 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2846 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2847 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2964 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2965 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2966 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2967 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3592 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3593 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4060 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4061 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4175 2024-04-25T13:41:01.553 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4176 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4177 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4178 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4181 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4182 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4183 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4184 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4185 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4186 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6000 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6001 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6002 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6100 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6114 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6115 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 pg_control 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 13:40 pg_filenode.map 2024-04-25T13:41:01.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/log: 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 . 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.555 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:total 1 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:41 . 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 mappings 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 25 13:41 replorigin_checkpoint 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 snapshots 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:41 .. 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.556 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:41 .. 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 . 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 members 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 offsets 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:total 8 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 .. 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 0000 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.557 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:total 8 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 .. 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 0000 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.558 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-25T13:41:01.559 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:total 8 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 0000 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-25T13:41:01.560 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:total 16384 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 . 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:41 000000010000000000000001 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 archive_status 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 .. 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:01.561 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-25T13:41:01.562 INFO:teuthology.orchestra.run.smithi027.stdout:total 8 2024-04-25T13:41:01.562 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:41:01.562 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 13:41 .. 2024-04-25T13:41:01.562 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 0000 2024-04-25T13:41:01.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-25T13:41:01.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-25T13:41:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:dropping old tables... 2024-04-25T13:41:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-25T13:41:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-25T13:41:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-25T13:41:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-25T13:41:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:creating tables... 2024-04-25T13:41:01.736 INFO:teuthology.orchestra.run.smithi027.stderr:generating data (client-side)... 2024-04-25T13:41:01.811 INFO:teuthology.orchestra.run.smithi027.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.06 s, remaining 1.81 s) 2024-04-25T13:41:01.978 INFO:teuthology.orchestra.run.smithi027.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.22 s, remaining 3.37 s) 2024-04-25T13:41:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.37 s, remaining 3.57 s) 2024-04-25T13:41:02.198 INFO:teuthology.orchestra.run.smithi027.stderr:400000 of 3200000 tuples (12%) done (elapsed 0.45 s, remaining 3.12 s) 2024-04-25T13:41:02.341 INFO:teuthology.orchestra.run.smithi027.stderr:500000 of 3200000 tuples (15%) done (elapsed 0.59 s, remaining 3.18 s) 2024-04-25T13:41:02.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:02 smithi027 ceph-mon[29140]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 KiB/s wr, 5 op/s 2024-04-25T13:41:02.422 INFO:teuthology.orchestra.run.smithi027.stderr:600000 of 3200000 tuples (18%) done (elapsed 0.67 s, remaining 2.90 s) 2024-04-25T13:41:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:02 smithi122 ceph-mon[32843]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 KiB/s wr, 5 op/s 2024-04-25T13:41:02.562 INFO:teuthology.orchestra.run.smithi027.stderr:700000 of 3200000 tuples (21%) done (elapsed 0.81 s, remaining 2.89 s) 2024-04-25T13:41:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:02 smithi045 ceph-mon[32731]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 17 KiB/s wr, 5 op/s 2024-04-25T13:41:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:800000 of 3200000 tuples (25%) done (elapsed 0.99 s, remaining 2.97 s) 2024-04-25T13:41:02.837 INFO:teuthology.orchestra.run.smithi027.stderr:900000 of 3200000 tuples (28%) done (elapsed 1.08 s, remaining 2.77 s) 2024-04-25T13:41:02.972 INFO:teuthology.orchestra.run.smithi027.stderr:1000000 of 3200000 tuples (31%) done (elapsed 1.22 s, remaining 2.68 s) 2024-04-25T13:41:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:1100000 of 3200000 tuples (34%) done (elapsed 1.38 s, remaining 2.64 s) 2024-04-25T13:41:03.242 INFO:teuthology.orchestra.run.smithi027.stderr:1200000 of 3200000 tuples (37%) done (elapsed 1.49 s, remaining 2.48 s) 2024-04-25T13:41:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:1300000 of 3200000 tuples (40%) done (elapsed 1.62 s, remaining 2.37 s) 2024-04-25T13:41:03.547 INFO:teuthology.orchestra.run.smithi027.stderr:1400000 of 3200000 tuples (43%) done (elapsed 1.79 s, remaining 2.31 s) 2024-04-25T13:41:03.639 INFO:teuthology.orchestra.run.smithi027.stderr:1500000 of 3200000 tuples (46%) done (elapsed 1.89 s, remaining 2.14 s) 2024-04-25T13:41:03.770 INFO:teuthology.orchestra.run.smithi027.stderr:1600000 of 3200000 tuples (50%) done (elapsed 2.02 s, remaining 2.02 s) 2024-04-25T13:41:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:1700000 of 3200000 tuples (53%) done (elapsed 2.11 s, remaining 1.86 s) 2024-04-25T13:41:04.021 INFO:teuthology.orchestra.run.smithi027.stderr:1800000 of 3200000 tuples (56%) done (elapsed 2.27 s, remaining 1.76 s) 2024-04-25T13:41:04.149 INFO:teuthology.orchestra.run.smithi027.stderr:1900000 of 3200000 tuples (59%) done (elapsed 2.40 s, remaining 1.64 s) 2024-04-25T13:41:04.246 INFO:teuthology.orchestra.run.smithi027.stderr:2000000 of 3200000 tuples (62%) done (elapsed 2.49 s, remaining 1.50 s) 2024-04-25T13:41:04.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:04 smithi027 ceph-mon[29140]: pgmap v354: 97 pgs: 97 active+clean; 26 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.1 MiB/s wr, 39 op/s 2024-04-25T13:41:04.405 INFO:teuthology.orchestra.run.smithi027.stderr:2100000 of 3200000 tuples (65%) done (elapsed 2.65 s, remaining 1.39 s) 2024-04-25T13:41:04.554 INFO:teuthology.orchestra.run.smithi027.stderr:2200000 of 3200000 tuples (68%) done (elapsed 2.80 s, remaining 1.27 s) 2024-04-25T13:41:04.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:04 smithi122 ceph-mon[32843]: pgmap v354: 97 pgs: 97 active+clean; 26 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.1 MiB/s wr, 39 op/s 2024-04-25T13:41:04.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:04 smithi045 ceph-mon[32731]: pgmap v354: 97 pgs: 97 active+clean; 26 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 2.1 MiB/s wr, 39 op/s 2024-04-25T13:41:04.636 INFO:teuthology.orchestra.run.smithi027.stderr:2300000 of 3200000 tuples (71%) done (elapsed 2.88 s, remaining 1.13 s) 2024-04-25T13:41:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2400000 of 3200000 tuples (75%) done (elapsed 3.06 s, remaining 1.02 s) 2024-04-25T13:41:04.967 INFO:teuthology.orchestra.run.smithi027.stderr:2500000 of 3200000 tuples (78%) done (elapsed 3.21 s, remaining 0.90 s) 2024-04-25T13:41:05.057 INFO:teuthology.orchestra.run.smithi027.stderr:2600000 of 3200000 tuples (81%) done (elapsed 3.30 s, remaining 0.76 s) 2024-04-25T13:41:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2700000 of 3200000 tuples (84%) done (elapsed 3.45 s, remaining 0.64 s) 2024-04-25T13:41:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2800000 of 3200000 tuples (87%) done (elapsed 3.55 s, remaining 0.51 s) 2024-04-25T13:41:05.468 INFO:teuthology.orchestra.run.smithi027.stderr:2900000 of 3200000 tuples (90%) done (elapsed 3.72 s, remaining 0.38 s) 2024-04-25T13:41:05.613 INFO:teuthology.orchestra.run.smithi027.stderr:3000000 of 3200000 tuples (93%) done (elapsed 3.86 s, remaining 0.26 s) 2024-04-25T13:41:05.726 INFO:teuthology.orchestra.run.smithi027.stderr:3100000 of 3200000 tuples (96%) done (elapsed 3.97 s, remaining 0.13 s) 2024-04-25T13:41:05.861 INFO:teuthology.orchestra.run.smithi027.stderr:3200000 of 3200000 tuples (100%) done (elapsed 4.11 s, remaining 0.00 s) 2024-04-25T13:41:05.910 INFO:teuthology.orchestra.run.smithi027.stderr:vacuuming... 2024-04-25T13:41:06.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:06 smithi027 ceph-mon[29140]: pgmap v355: 97 pgs: 97 active+clean; 85 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 52 op/s 2024-04-25T13:41:06.545 INFO:teuthology.orchestra.run.smithi027.stderr:creating primary keys... 2024-04-25T13:41:06.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:06 smithi122 ceph-mon[32843]: pgmap v355: 97 pgs: 97 active+clean; 85 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 52 op/s 2024-04-25T13:41:06.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:06 smithi045 ceph-mon[32731]: pgmap v355: 97 pgs: 97 active+clean; 85 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 52 op/s 2024-04-25T13:41:07.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:07 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:41:07.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:07 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:41:07.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:07 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:41:07.816 INFO:teuthology.orchestra.run.smithi027.stderr:done in 6.09 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 4.17 s, vacuum 0.63 s, primary keys 1.27 s). 2024-04-25T13:41:07.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-25T13:41:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:starting vacuum...end. 2024-04-25T13:41:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.272+0000 7fd49eaaf640 1 -- 172.21.15.27:0/1510600305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd498103a80 msgr2=0x7fd498103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.272+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/1510600305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd498103a80 0x7fd498103e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd484009980 tx=0x7fd48402c650 comp rx=0 tx=0).stop 2024-04-25T13:41:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.272+0000 7fd49eaaf640 1 -- 172.21.15.27:0/1510600305 shutdown_connections 2024-04-25T13:41:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.272+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/1510600305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd498108780 0x7fd49810aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.272+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/1510600305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd498103a80 0x7fd498103e90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.272+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/1510600305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd498102840 0x7fd498103540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.272+0000 7fd49eaaf640 1 -- 172.21.15.27:0/1510600305 >> 172.21.15.27:0/1510600305 conn(0x7fd4980fda30 msgr2=0x7fd4980ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.273+0000 7fd49eaaf640 1 -- 172.21.15.27:0/1510600305 shutdown_connections 2024-04-25T13:41:08.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.273+0000 7fd49eaaf640 1 -- 172.21.15.27:0/1510600305 wait complete. 2024-04-25T13:41:08.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.274+0000 7fd49eaaf640 1 Processor -- start 2024-04-25T13:41:08.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.274+0000 7fd49eaaf640 1 -- start start 2024-04-25T13:41:08.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.274+0000 7fd49eaaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd498102840 0x7fd4981a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49eaaf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd498103a80 0x7fd4981a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49eaaf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd498108780 0x7fd4981a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49eaaf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd49810d300 con 0x7fd498102840 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49eaaf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd49810d180 con 0x7fd498108780 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49eaaf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd49810d480 con 0x7fd498103a80 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49c84b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd498102840 0x7fd4981a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49c84b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd498102840 0x7fd4981a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35200/0 (socket says 172.21.15.27:35200) 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49c84b640 1 -- 172.21.15.27:0/3031892887 learned_addr learned my addr 172.21.15.27:0/3031892887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.275+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd498108780 0x7fd4981a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.276+0000 7fd49c84b640 1 -- 172.21.15.27:0/3031892887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd498103a80 msgr2=0x7fd4981a0c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:41:08.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.276+0000 7fd49c84b640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd498103a80 0x7fd4981a0c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.276+0000 7fd49c84b640 1 -- 172.21.15.27:0/3031892887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd498108780 msgr2=0x7fd4981a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:08.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.276+0000 7fd49c84b640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd498108780 0x7fd4981a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.276+0000 7fd49c84b640 1 -- 172.21.15.27:0/3031892887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4981a8200 con 0x7fd498102840 2024-04-25T13:41:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.276+0000 7fd49c84b640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd498102840 0x7fd4981a06c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd48c00b450 tx=0x7fd48c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.277+0000 7fd495ffb640 1 -- 172.21.15.27:0/3031892887 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c015020 con 0x7fd498102840 2024-04-25T13:41:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.277+0000 7fd495ffb640 1 -- 172.21.15.27:0/3031892887 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd48c011d40 con 0x7fd498102840 2024-04-25T13:41:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.277+0000 7fd495ffb640 1 -- 172.21.15.27:0/3031892887 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00f340 con 0x7fd498102840 2024-04-25T13:41:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.277+0000 7fd49eaaf640 1 -- 172.21.15.27:0/3031892887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4981a84f0 con 0x7fd498102840 2024-04-25T13:41:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.277+0000 7fd49eaaf640 1 -- 172.21.15.27:0/3031892887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4981a8a10 con 0x7fd498102840 2024-04-25T13:41:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.281+0000 7fd4a5966740 1 Processor -- start 2024-04-25T13:41:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.281+0000 7fd4a5966740 1 -- start start 2024-04-25T13:41:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.281+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.281+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d6200022f0 0x55d6203cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.281+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d62000a4d0 0x55d6203d2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.281+0000 7fd4a5966740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d620331460 con 0x55d62002b1b0 2024-04-25T13:41:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.281+0000 7fd4a5966740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d6202eb590 con 0x55d62000a4d0 2024-04-25T13:41:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.281+0000 7fd4a5966740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d62032d420 con 0x55d6200022f0 2024-04-25T13:41:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.282+0000 7fd49d04c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.282+0000 7fd49d04c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35202/0 (socket says 172.21.15.27:35202) 2024-04-25T13:41:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.282+0000 7fd49d04c640 1 -- 172.21.15.27:0/1406399847 learned_addr learned my addr 172.21.15.27:0/1406399847 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.283+0000 7fd49d04c640 1 -- 172.21.15.27:0/1406399847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d6200022f0 msgr2=0x55d6203cc0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:41:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.283+0000 7fd49d04c640 1 --2- 172.21.15.27:0/1406399847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d6200022f0 0x55d6203cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.283+0000 7fd49d04c640 1 -- 172.21.15.27:0/1406399847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d62000a4d0 msgr2=0x55d6203d2fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:41:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.283+0000 7fd49d04c640 1 --2- 172.21.15.27:0/1406399847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d62000a4d0 0x55d6203d2fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.283+0000 7fd49d04c640 1 -- 172.21.15.27:0/1406399847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6202f80d0 con 0x55d62002b1b0 2024-04-25T13:41:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.283+0000 7fd495ffb640 1 -- 172.21.15.27:0/3031892887 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd48c018020 con 0x7fd498102840 2024-04-25T13:41:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.284+0000 7fd49d04c640 1 --2- 172.21.15.27:0/1406399847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203cbb20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd49000f490 tx=0x7fd49000a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2cc72ecf08ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.285+0000 7fd495ffb640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd4880afcf0 0x7fd4880b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.285+0000 7fd495ffb640 1 -- 172.21.15.27:0/3031892887 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd48c0de570 con 0x7fd498102840 2024-04-25T13:41:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.286+0000 7fd482ffd640 1 -- 172.21.15.27:0/1406399847 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49000c860 con 0x55d62002b1b0 2024-04-25T13:41:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.286+0000 7fd482ffd640 1 -- 172.21.15.27:0/1406399847 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd490019430 con 0x55d62002b1b0 2024-04-25T13:41:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.286+0000 7fd482ffd640 1 -- 172.21.15.27:0/1406399847 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490018640 con 0x55d62002b1b0 2024-04-25T13:41:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.286+0000 7fd49c84b640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd4880afcf0 0x7fd4880b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.287+0000 7fd4a5966740 1 -- 172.21.15.27:0/1406399847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 msgr2=0x55d6203cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:08.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.287+0000 7fd4a5966740 1 --2- 172.21.15.27:0/1406399847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203cbb20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd49000f490 tx=0x7fd49000a580 comp rx=0 tx=0).stop 2024-04-25T13:41:08.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.287+0000 7fd49c84b640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd4880afcf0 0x7fd4880b2200 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0de850 tx=0x7fd48c0de7c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.288+0000 7fd4a5966740 1 -- 172.21.15.27:0/1406399847 shutdown_connections 2024-04-25T13:41:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.288+0000 7fd4a5966740 1 --2- 172.21.15.27:0/1406399847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d62000a4d0 0x55d6203d2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.288+0000 7fd4a5966740 1 --2- 172.21.15.27:0/1406399847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d6200022f0 0x55d6203cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.288+0000 7fd4a5966740 1 --2- 172.21.15.27:0/1406399847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203cbb20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.288+0000 7fd4a5966740 1 -- 172.21.15.27:0/1406399847 >> 172.21.15.27:0/1406399847 conn(0x55d62001f300 msgr2=0x55d6203c74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.288+0000 7fd4a5966740 1 -- 172.21.15.27:0/1406399847 shutdown_connections 2024-04-25T13:41:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.288+0000 7fd4a5966740 1 -- 172.21.15.27:0/1406399847 wait complete. 2024-04-25T13:41:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.289+0000 7fd4a5966740 1 -- start start 2024-04-25T13:41:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.289+0000 7fd4a5966740 1 Processor -- start 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.289+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203e6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.289+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 0x55d6203e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.289+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203edce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.289+0000 7fd4a5966740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d6203e4a00 con 0x55d62002b1b0 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.289+0000 7fd4a5966740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d6202b8b60 con 0x55d6200022f0 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.289+0000 7fd4a5966740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d6203e4b80 con 0x55d62000a4d0 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd4a5966740 10 client.0 _ll_register_callbacks cb 0x55d6202dd2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203e6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34318/0 (socket says 172.21.15.27:34318) 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 -- 172.21.15.27:0/2610957631 learned_addr learned my addr 172.21.15.27:0/2610957631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd497fff640 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 0x55d6203e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 -- 172.21.15.27:0/2610957631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 msgr2=0x55d6203e6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 0x55d6203e6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 -- 172.21.15.27:0/2610957631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 msgr2=0x55d6203edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 -- 172.21.15.27:0/2610957631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d620043210 con 0x55d6200022f0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd49d04c640 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203e6840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd490000dc0 tx=0x7fd49000a960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d25a98392a199a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.290+0000 7fd477fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.291+0000 7fd477fff640 1 -- 172.21.15.27:0/2610957631 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49000ca70 con 0x55d6200022f0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.291+0000 7fd477fff640 1 -- 172.21.15.27:0/2610957631 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd49000cc10 con 0x55d6200022f0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.291+0000 7fd4a5966740 1 -- 172.21.15.27:0/2610957631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd490019430 con 0x55d6200022f0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.291+0000 7fd477fff640 1 -- 172.21.15.27:0/2610957631 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd49000ce10 con 0x55d6200022f0 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.291+0000 7fd4a5966740 10 client.25088 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.291+0000 7fd4a5966740 1 -- 172.21.15.27:0/2610957631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d62010f620 con 0x55d6200022f0 2024-04-25T13:41:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.292+0000 7fd477fff640 1 -- 172.21.15.27:0/2610957631 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd490019430 con 0x55d6200022f0 2024-04-25T13:41:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.292+0000 7fd477fff640 1 -- 172.21.15.27:0/2610957631 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fd490032050 con 0x55d6200022f0 2024-04-25T13:41:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.293+0000 7fd4a5966740 10 client.25088 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.293+0000 7fd4a5966740 10 client.25088 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.293+0000 7fd4a5966740 20 client.25088 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.293+0000 7fd4a5966740 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d61ffe9d80 0x55d6203f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.293+0000 7fd4a5966740 4 client.25088 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.293+0000 7fd4a5966740 1 -- 172.21.15.27:0/2610957631 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d62033f070 con 0x55d61ffe9d80 2024-04-25T13:41:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.293+0000 7fd497fff640 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d61ffe9d80 0x55d6203f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.294+0000 7fd497fff640 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d61ffe9d80 0x55d6203f41f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fc401cb9b945b9e server_cookie=c3684addda785574 in_seq=0 out_seq=0 2024-04-25T13:41:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.294+0000 7fd477fff640 10 client.25088 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd477fff640 1 -- 172.21.15.27:0/2610957631 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d62033f070 con 0x55d61ffe9d80 2024-04-25T13:41:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd477fff640 10 client.25088 handle_command_reply: tid=0 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 client.25088 shutdown 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 -- 172.21.15.27:0/2610957631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 msgr2=0x55d6203e6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203e6840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd490000dc0 tx=0x7fd49000a960 comp rx=0 tx=0).stop 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 -- 172.21.15.27:0/2610957631 shutdown_connections 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d61ffe9d80 0x55d6203f41f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62002b1b0 0x55d6203edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 0x55d6203e6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 --2- 172.21.15.27:0/2610957631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203e6840 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd477fff640 0 client.25088 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.295+0000 7fd4a5966740 1 -- 172.21.15.27:0/2610957631 >> 172.21.15.27:0/2610957631 conn(0x55d62001f300 msgr2=0x55d6203ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.296+0000 7fd4a5966740 1 -- 172.21.15.27:0/2610957631 shutdown_connections 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.296+0000 7fd4a5966740 1 -- 172.21.15.27:0/2610957631 wait complete. 2024-04-25T13:41:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.296+0000 7fd4a5966740 20 client.25088 trim_cache size 0 max 16384 2024-04-25T13:41:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.327+0000 7fd4a5966740 1 Processor -- start 2024-04-25T13:41:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.327+0000 7fd4a5966740 1 -- start start 2024-04-25T13:41:08.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:08 smithi045 ceph-mon[32731]: pgmap v356: 97 pgs: 97 active+clean; 292 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 43 MiB/s wr, 107 op/s 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d61ffe9d80 0x55d6203e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6200022f0 0x55d6203e6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 0x55d6203cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd49d04c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d61ffe9d80 0x55d6203e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd49d04c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d61ffe9d80 0x55d6203e67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34324/0 (socket says 172.21.15.27:34324) 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd49d04c640 1 -- 172.21.15.27:0/3742032594 learned_addr learned my addr 172.21.15.27:0/3742032594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd4a5966740 1 -- 172.21.15.27:0/3742032594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd498073010 con 0x55d6200022f0 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd4a5966740 1 -- 172.21.15.27:0/3742032594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd4981a9790 con 0x55d61ffe9d80 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.328+0000 7fd4a5966740 1 -- 172.21.15.27:0/3742032594 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd498073680 con 0x55d62000a4d0 2024-04-25T13:41:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.329+0000 7fd497fff640 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6200022f0 0x55d6203e6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.329+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 0x55d6203cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.329+0000 7fd49d04c640 1 -- 172.21.15.27:0/3742032594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 msgr2=0x55d6203cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.329+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 0x55d6203cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.329+0000 7fd49d04c640 1 -- 172.21.15.27:0/3742032594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6200022f0 msgr2=0x55d6203e6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.329+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6200022f0 0x55d6203e6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.329+0000 7fd49d04c640 1 -- 172.21.15.27:0/3742032594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6203d33c0 con 0x55d61ffe9d80 2024-04-25T13:41:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.329+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d61ffe9d80 0x55d6203e67e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd490000c40 tx=0x7fd4900199d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83bfc86d39020931 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.330+0000 7fd4827fc640 1 -- 172.21.15.27:0/3742032594 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49003a080 con 0x55d61ffe9d80 2024-04-25T13:41:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.330+0000 7fd4827fc640 1 -- 172.21.15.27:0/3742032594 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd490028600 con 0x55d61ffe9d80 2024-04-25T13:41:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.330+0000 7fd4827fc640 1 -- 172.21.15.27:0/3742032594 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49000caf0 con 0x55d61ffe9d80 2024-04-25T13:41:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.331+0000 7fd4a5966740 1 -- 172.21.15.27:0/3742032594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d61ffe9d80 msgr2=0x55d6203e67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.331+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d61ffe9d80 0x55d6203e67e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd490000c40 tx=0x7fd4900199d0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.332+0000 7fd4a5966740 1 -- 172.21.15.27:0/3742032594 shutdown_connections 2024-04-25T13:41:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.332+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d62000a4d0 0x55d6203cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.332+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d6200022f0 0x55d6203e6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.332+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3742032594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d61ffe9d80 0x55d6203e67e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.332+0000 7fd4a5966740 1 -- 172.21.15.27:0/3742032594 >> 172.21.15.27:0/3742032594 conn(0x55d62001f300 msgr2=0x55d6203e4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.333+0000 7fd4a5966740 1 -- 172.21.15.27:0/3742032594 shutdown_connections 2024-04-25T13:41:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.334+0000 7fd4a5966740 1 -- 172.21.15.27:0/3742032594 wait complete. 2024-04-25T13:41:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.334+0000 7fd4a5966740 1 -- start start 2024-04-25T13:41:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.334+0000 7fd4a5966740 1 Processor -- start 2024-04-25T13:41:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.335+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d61ffe9d80 0x55d6203eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.335+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203eb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.335+0000 7fd4a5966740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62000a4d0 0x55d6203ec350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.335+0000 7fd497fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.335+0000 7fd497fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203eb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34328/0 (socket says 172.21.15.27:34328) 2024-04-25T13:41:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.335+0000 7fd497fff640 1 -- 172.21.15.27:0/3788162895 learned_addr learned my addr 172.21.15.27:0/3788162895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.335+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd498073d50 con 0x55d62000a4d0 2024-04-25T13:41:08.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd4981a90c0 con 0x55d6200022f0 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d61ffe9d80 0x55d6203eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd498074420 con 0x55d61ffe9d80 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd4a5966740 10 client.0 _ll_register_callbacks cb 0x7fd4900012e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62000a4d0 0x55d6203ec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd497fff640 1 -- 172.21.15.27:0/3788162895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d61ffe9d80 msgr2=0x55d6203eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd497fff640 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d61ffe9d80 0x55d6203eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd497fff640 1 -- 172.21.15.27:0/3788162895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62000a4d0 msgr2=0x55d6203ec350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd497fff640 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62000a4d0 0x55d6203ec350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd497fff640 1 -- 172.21.15.27:0/3788162895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6203ec910 con 0x55d6200022f0 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.336+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62000a4d0 0x55d6203ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.337+0000 7fd497fff640 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203eb490 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd484009600 tx=0x7fd484005310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ade9fbf801957d45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.337+0000 7fd477fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.337+0000 7fd477fff640 1 -- 172.21.15.27:0/3788162895 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484046070 con 0x55d6200022f0 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.338+0000 7fd477fff640 1 -- 172.21.15.27:0/3788162895 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd484004240 con 0x55d6200022f0 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.338+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd490028600 con 0x55d6200022f0 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.338+0000 7fd477fff640 1 -- 172.21.15.27:0/3788162895 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd484052070 con 0x55d6200022f0 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.338+0000 7fd4a5966740 10 client.25094 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.338+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d62045c710 con 0x55d6200022f0 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.338+0000 7fd477fff640 1 -- 172.21.15.27:0/3788162895 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd490028600 con 0x55d6200022f0 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.339+0000 7fd477fff640 1 -- 172.21.15.27:0/3788162895 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fd4840036f0 con 0x55d6200022f0 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.339+0000 7fd4a5966740 10 client.25094 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.339+0000 7fd4a5966740 10 client.25094 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.339+0000 7fd4a5966740 20 client.25094 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.339+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d62002b1b0 0x55d62045f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.339+0000 7fd4a5966740 4 client.25094 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:41:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.339+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d620338ee0 con 0x55d62002b1b0 2024-04-25T13:41:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.339+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d62002b1b0 0x55d62045f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.341+0000 7fd477fff640 10 client.25094 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.341+0000 7fd49d04c640 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d62002b1b0 0x55d62045f090 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c0066e9ccc7dc04 server_cookie=c147c71d99a6fdc6 in_seq=0 out_seq=0 2024-04-25T13:41:08.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:08 smithi027 ceph-mon[29140]: pgmap v356: 97 pgs: 97 active+clean; 292 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 43 MiB/s wr, 107 op/s 2024-04-25T13:41:08.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:08 smithi122 ceph-mon[32843]: pgmap v356: 97 pgs: 97 active+clean; 292 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 43 MiB/s wr, 107 op/s 2024-04-25T13:41:08.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd477fff640 1 -- 172.21.15.27:0/3788162895 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d620338ee0 con 0x55d62002b1b0 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd477fff640 10 client.25094 handle_command_reply: tid=0 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 client.25094 shutdown 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 msgr2=0x55d6203eb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203eb490 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd484009600 tx=0x7fd484005310 comp rx=0 tx=0).stop 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 shutdown_connections 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d62002b1b0 0x55d62045f090 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d62000a4d0 0x55d6203ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6200022f0 0x55d6203eb490 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 --2- 172.21.15.27:0/3788162895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d61ffe9d80 0x55d6203eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 >> 172.21.15.27:0/3788162895 conn(0x55d62001f300 msgr2=0x55d6203e4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.594+0000 7fd477fff640 0 client.25094 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 shutdown_connections 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd4a5966740 1 -- 172.21.15.27:0/3788162895 wait complete. 2024-04-25T13:41:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd4a5966740 20 client.25094 trim_cache size 0 max 16384 2024-04-25T13:41:08.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:41:08.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:41:08.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:41:08.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 -- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd4880afcf0 msgr2=0x7fd4880b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd4880afcf0 0x7fd4880b2200 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0de850 tx=0x7fd48c0de7c0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 -- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd498102840 msgr2=0x7fd4981a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd498102840 0x7fd4981a06c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd48c00b450 tx=0x7fd48c00b910 comp rx=0 tx=0).stop 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 -- 172.21.15.27:0/3031892887 shutdown_connections 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd4880afcf0 0x7fd4880b2200 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd498108780 0x7fd4981a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd498103a80 0x7fd4981a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 --2- 172.21.15.27:0/3031892887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd498102840 0x7fd4981a06c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 -- 172.21.15.27:0/3031892887 >> 172.21.15.27:0/3031892887 conn(0x7fd4980fda30 msgr2=0x7fd498104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:08.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 -- 172.21.15.27:0/3031892887 shutdown_connections 2024-04-25T13:41:08.598 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:08.595+0000 7fd49eaaf640 1 -- 172.21.15.27:0/3031892887 wait complete. 2024-04-25T13:41:09.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:09 smithi027 ceph-mon[29140]: pgmap v357: 97 pgs: 97 active+clean; 340 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 52 MiB/s wr, 117 op/s 2024-04-25T13:41:09.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:09.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:09.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:09.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:09.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:09.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:09 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:09 smithi122 ceph-mon[32843]: pgmap v357: 97 pgs: 97 active+clean; 340 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 52 MiB/s wr, 117 op/s 2024-04-25T13:41:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:09 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:09 smithi045 ceph-mon[32731]: pgmap v357: 97 pgs: 97 active+clean; 340 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 52 MiB/s wr, 117 op/s 2024-04-25T13:41:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:09 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:10.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:41:10.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:41:10.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:10 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:11.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:41:11.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:41:11.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:10 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:10 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:41:12.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:11 smithi122 ceph-mon[32843]: pgmap v358: 97 pgs: 97 active+clean; 444 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 66 MiB/s wr, 206 op/s 2024-04-25T13:41:12.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:11 smithi045 ceph-mon[32731]: pgmap v358: 97 pgs: 97 active+clean; 444 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 66 MiB/s wr, 206 op/s 2024-04-25T13:41:12.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:11 smithi027 ceph-mon[29140]: pgmap v358: 97 pgs: 97 active+clean; 444 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 66 MiB/s wr, 206 op/s 2024-04-25T13:41:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 5.0 s, 3662.6 tps, lat 26.863 ms stddev 31.132 2024-04-25T13:41:14.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:13 smithi027 ceph-mon[29140]: pgmap v359: 97 pgs: 97 active+clean; 541 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 76 MiB/s wr, 360 op/s 2024-04-25T13:41:14.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:13 smithi122 ceph-mon[32843]: pgmap v359: 97 pgs: 97 active+clean; 541 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 76 MiB/s wr, 360 op/s 2024-04-25T13:41:14.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:13 smithi045 ceph-mon[32731]: pgmap v359: 97 pgs: 97 active+clean; 541 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 76 MiB/s wr, 360 op/s 2024-04-25T13:41:16.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:15 smithi122 ceph-mon[32843]: pgmap v360: 97 pgs: 97 active+clean; 590 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 74 MiB/s wr, 353 op/s 2024-04-25T13:41:16.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:15 smithi045 ceph-mon[32731]: pgmap v360: 97 pgs: 97 active+clean; 590 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 74 MiB/s wr, 353 op/s 2024-04-25T13:41:16.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:15 smithi027 ceph-mon[29140]: pgmap v360: 97 pgs: 97 active+clean; 590 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 74 MiB/s wr, 353 op/s 2024-04-25T13:41:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 10.0 s, 3916.6 tps, lat 25.444 ms stddev 27.004 2024-04-25T13:41:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:17 smithi122 ceph-mon[32843]: pgmap v361: 97 pgs: 97 active+clean; 894 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 69 MiB/s wr, 601 op/s 2024-04-25T13:41:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:17 smithi045 ceph-mon[32731]: pgmap v361: 97 pgs: 97 active+clean; 894 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 69 MiB/s wr, 601 op/s 2024-04-25T13:41:18.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:17 smithi027 ceph-mon[29140]: pgmap v361: 97 pgs: 97 active+clean; 894 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 69 MiB/s wr, 601 op/s 2024-04-25T13:41:20.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:19 smithi122 ceph-mon[32843]: pgmap v362: 97 pgs: 97 active+clean; 908 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 566 op/s 2024-04-25T13:41:20.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:19 smithi045 ceph-mon[32731]: pgmap v362: 97 pgs: 97 active+clean; 908 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 566 op/s 2024-04-25T13:41:20.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:19 smithi027 ceph-mon[29140]: pgmap v362: 97 pgs: 97 active+clean; 908 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 37 MiB/s wr, 566 op/s 2024-04-25T13:41:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.727+0000 7f0393de3640 1 -- 172.21.15.27:0/1976618654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c102290 msgr2=0x7f038c10bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.727+0000 7f0393de3640 1 --2- 172.21.15.27:0/1976618654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c102290 0x7f038c10bcf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0384009ca0 tx=0x7f038402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 -- 172.21.15.27:0/1976618654 shutdown_connections 2024-04-25T13:41:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 --2- 172.21.15.27:0/1976618654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c102290 0x7f038c10bcf0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 --2- 172.21.15.27:0/1976618654 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f038c101790 0x7f038c101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 --2- 172.21.15.27:0/1976618654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f038c1050e0 0x7f038c1054f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 -- 172.21.15.27:0/1976618654 >> 172.21.15.27:0/1976618654 conn(0x7f038c0fd390 msgr2=0x7f038c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 -- 172.21.15.27:0/1976618654 shutdown_connections 2024-04-25T13:41:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 -- 172.21.15.27:0/1976618654 wait complete. 2024-04-25T13:41:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 Processor -- start 2024-04-25T13:41:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.728+0000 7f0393de3640 1 -- start start 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0393de3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f038c101790 0x7f038c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0393de3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f038c102290 0x7f038c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0393de3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c1050e0 0x7f038c19f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0393de3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f038c10d4c0 con 0x7f038c1050e0 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0393de3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f038c10d340 con 0x7f038c102290 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0393de3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f038c10d640 con 0x7f038c101790 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0392380640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c1050e0 0x7f038c19f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0391b7f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f038c101790 0x7f038c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0392380640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c1050e0 0x7f038c19f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54574/0 (socket says 172.21.15.27:54574) 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0392380640 1 -- 172.21.15.27:0/3389380037 learned_addr learned my addr 172.21.15.27:0/3389380037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f039137e640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f038c102290 0x7f038c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0391b7f640 1 -- 172.21.15.27:0/3389380037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f038c102290 msgr2=0x7f038c1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0391b7f640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f038c102290 0x7f038c1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0391b7f640 1 -- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c1050e0 msgr2=0x7f038c19f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0391b7f640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c1050e0 0x7f038c19f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0391b7f640 1 -- 172.21.15.27:0/3389380037 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c19faa0 con 0x7f038c101790 2024-04-25T13:41:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.729+0000 7f0391b7f640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f038c101790 0x7f038c197f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f038c06a2f0 tx=0x7f038800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.731+0000 7f0382ffd640 1 -- 172.21.15.27:0/3389380037 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038800cc40 con 0x7f038c101790 2024-04-25T13:41:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.731+0000 7f0382ffd640 1 -- 172.21.15.27:0/3389380037 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f038800cde0 con 0x7f038c101790 2024-04-25T13:41:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.732+0000 7f0382ffd640 1 -- 172.21.15.27:0/3389380037 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0388010430 con 0x7f038c101790 2024-04-25T13:41:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.733+0000 7f0393de3640 1 -- 172.21.15.27:0/3389380037 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f038c19fd90 con 0x7f038c101790 2024-04-25T13:41:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.733+0000 7f0393de3640 1 -- 172.21.15.27:0/3389380037 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f038c1a01a0 con 0x7f038c101790 2024-04-25T13:41:21.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.738+0000 7f039ad6f740 1 Processor -- start 2024-04-25T13:41:21.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.740+0000 7f0382ffd640 1 -- 172.21.15.27:0/3389380037 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0388003e70 con 0x7f038c101790 2024-04-25T13:41:21.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.741+0000 7f039ad6f740 1 -- start start 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.742+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b52f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.742+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b52f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.742+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b536600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.742+0000 7f039ad6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a2b48c460 con 0x562a2b1654d0 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.742+0000 7f039ad6f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562a2b446590 con 0x562a2b1861b0 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.742+0000 7f039ad6f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562a2b488420 con 0x562a2b15d2f0 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.744+0000 7f039137e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b536600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.744+0000 7f0391b7f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b52f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.744+0000 7f039137e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b536600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54586/0 (socket says 172.21.15.27:54586) 2024-04-25T13:41:21.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.744+0000 7f039137e640 1 -- 172.21.15.27:0/2622374910 learned_addr learned my addr 172.21.15.27:0/2622374910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.744+0000 7f0392380640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b52f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.745+0000 7f0382ffd640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03780afa00 0x7f03780b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.745+0000 7f0392380640 1 -- 172.21.15.27:0/2622374910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 msgr2=0x562a2b52f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.745+0000 7f0392380640 1 --2- 172.21.15.27:0/2622374910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b52f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.745+0000 7f0392380640 1 -- 172.21.15.27:0/2622374910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 msgr2=0x562a2b536600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.745+0000 7f0392380640 1 --2- 172.21.15.27:0/2622374910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b536600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.745+0000 7f0392380640 1 -- 172.21.15.27:0/2622374910 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a2b4530d0 con 0x562a2b1861b0 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.745+0000 7f0392380640 1 --2- 172.21.15.27:0/2622374910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b52f180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0384006010 tx=0x7f03840093a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f74a980f2763b2c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.746+0000 7f0392380640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03780afa00 0x7f03780b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.746+0000 7f0382ffd640 1 -- 172.21.15.27:0/3389380037 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f03880df030 con 0x7f038c101790 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.746+0000 7f0377fff640 1 -- 172.21.15.27:0/2622374910 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038404a040 con 0x562a2b1861b0 2024-04-25T13:41:21.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.746+0000 7f0377fff640 1 -- 172.21.15.27:0/2622374910 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0384036030 con 0x562a2b1861b0 2024-04-25T13:41:21.748 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.747+0000 7f039ad6f740 1 -- 172.21.15.27:0/2622374910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 msgr2=0x562a2b52f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.748 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.747+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2622374910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b52f180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0384006010 tx=0x7f03840093a0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.748 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.748+0000 7f0392380640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03780afa00 0x7f03780b1f10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f03840094e0 tx=0x7f0384004910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:21.748 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.748+0000 7f039ad6f740 1 -- 172.21.15.27:0/2622374910 shutdown_connections 2024-04-25T13:41:21.748 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.748+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2622374910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b536600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.748+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2622374910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b52f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.748+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2622374910 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b52f180 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.748+0000 7f039ad6f740 1 -- 172.21.15.27:0/2622374910 >> 172.21.15.27:0/2622374910 conn(0x562a2b17a300 msgr2=0x562a2b52ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:21.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.749+0000 7f039ad6f740 1 -- 172.21.15.27:0/2622374910 shutdown_connections 2024-04-25T13:41:21.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.749+0000 7f039ad6f740 1 -- 172.21.15.27:0/2622374910 wait complete. 2024-04-25T13:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.750+0000 7f039ad6f740 1 -- start start 2024-04-25T13:41:21.750 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.750+0000 7f039ad6f740 1 Processor -- start 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.750+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b538c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.750+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1654d0 0x562a2b539160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.751+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1861b0 0x562a2b5400c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.751+0000 7f039ad6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a2b536ec0 con 0x562a2b15d2f0 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.751+0000 7f039ad6f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562a2b413b60 con 0x562a2b1654d0 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.751+0000 7f039ad6f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562a2b537040 con 0x562a2b1861b0 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.751+0000 7f039ad6f740 10 client.0 _ll_register_callbacks cb 0x562a2b4382b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.751+0000 7f039137e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1654d0 0x562a2b539160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.751+0000 7f039137e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1654d0 0x562a2b539160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37956/0 (socket says 172.21.15.27:37956) 2024-04-25T13:41:21.751 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.751+0000 7f039137e640 1 -- 172.21.15.27:0/3644357303 learned_addr learned my addr 172.21.15.27:0/3644357303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.752+0000 7f0391b7f640 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b538c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.752+0000 7f0392380640 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1861b0 0x562a2b5400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.752+0000 7f039137e640 1 -- 172.21.15.27:0/3644357303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1861b0 msgr2=0x562a2b5400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.752+0000 7f039137e640 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1861b0 0x562a2b5400c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.752+0000 7f039137e640 1 -- 172.21.15.27:0/3644357303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 msgr2=0x562a2b538c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.752+0000 7f039137e640 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b538c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.752+0000 7f039137e640 1 -- 172.21.15.27:0/3644357303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a2b19e210 con 0x562a2b1654d0 2024-04-25T13:41:21.752 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.752+0000 7f039137e640 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1654d0 0x562a2b539160 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f037c009fd0 tx=0x7f037c012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3d4335631b22c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:21.753 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.753+0000 7f03757fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:21.753 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.753+0000 7f03757fa640 1 -- 172.21.15.27:0/3644357303 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c011bb0 con 0x562a2b1654d0 2024-04-25T13:41:21.753 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.753+0000 7f03757fa640 1 -- 172.21.15.27:0/3644357303 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f037c011d50 con 0x562a2b1654d0 2024-04-25T13:41:21.756 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.756+0000 7f039ad6f740 1 -- 172.21.15.27:0/3644357303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0384036030 con 0x562a2b1654d0 2024-04-25T13:41:21.756 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.756+0000 7f03757fa640 1 -- 172.21.15.27:0/3644357303 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f037c00f460 con 0x562a2b1654d0 2024-04-25T13:41:21.756 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.757+0000 7f039ad6f740 10 client.25106 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:21.757 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.757+0000 7f039ad6f740 1 -- 172.21.15.27:0/3644357303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562a2b26a620 con 0x562a2b1654d0 2024-04-25T13:41:21.757 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.757+0000 7f03757fa640 1 -- 172.21.15.27:0/3644357303 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0384036030 con 0x562a2b1654d0 2024-04-25T13:41:21.757 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.758+0000 7f03757fa640 1 -- 172.21.15.27:0/3644357303 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f037c026850 con 0x562a2b1654d0 2024-04-25T13:41:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.758+0000 7f039ad6f740 10 client.25106 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.758+0000 7f039ad6f740 10 client.25106 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.758+0000 7f039ad6f740 20 client.25106 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.758+0000 7f039ad6f740 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a2b144d80 0x562a2b528440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.758+0000 7f039ad6f740 4 client.25106 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:21.758 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.758+0000 7f039ad6f740 1 -- 172.21.15.27:0/3644357303 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a2b49a070 con 0x562a2b144d80 2024-04-25T13:41:21.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.759+0000 7f0391b7f640 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a2b144d80 0x562a2b528440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.761+0000 7f0391b7f640 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a2b144d80 0x562a2b528440 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8367239bc33cefa server_cookie=eb57a930e2eebeb4 in_seq=0 out_seq=0 2024-04-25T13:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.761+0000 7f03757fa640 10 client.25106 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:21.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.762+0000 7f03757fa640 1 -- 172.21.15.27:0/3644357303 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562a2b49a070 con 0x562a2b144d80 2024-04-25T13:41:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.762+0000 7f03757fa640 10 client.25106 handle_command_reply: tid=0 2024-04-25T13:41:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.762+0000 7f039ad6f740 1 client.25106 shutdown 2024-04-25T13:41:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 -- 172.21.15.27:0/3644357303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1654d0 msgr2=0x562a2b539160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.762 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1654d0 0x562a2b539160 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f037c009fd0 tx=0x7f037c012b60 comp rx=0 tx=0).stop 2024-04-25T13:41:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 -- 172.21.15.27:0/3644357303 shutdown_connections 2024-04-25T13:41:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a2b144d80 0x562a2b528440 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1861b0 0x562a2b5400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1654d0 0x562a2b539160 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 --2- 172.21.15.27:0/3644357303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b538c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 -- 172.21.15.27:0/3644357303 >> 172.21.15.27:0/3644357303 conn(0x562a2b17a300 msgr2=0x562a2b521250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f03757fa640 0 client.25106 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.763+0000 7f039ad6f740 1 -- 172.21.15.27:0/3644357303 shutdown_connections 2024-04-25T13:41:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.764+0000 7f039ad6f740 1 -- 172.21.15.27:0/3644357303 wait complete. 2024-04-25T13:41:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.764+0000 7f039ad6f740 20 client.25106 trim_cache size 0 max 16384 2024-04-25T13:41:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.789+0000 7f039ad6f740 1 Processor -- start 2024-04-25T13:41:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.789+0000 7f039ad6f740 1 -- start start 2024-04-25T13:41:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b5324c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1654d0 0x562a2b53ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0391b7f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b5324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0391b7f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b5324c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54610/0 (socket says 172.21.15.27:54610) 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b537ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f039ad6f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f038c1a6d50 con 0x562a2b15d2f0 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f039ad6f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f038c1a95d0 con 0x562a2b1861b0 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f039ad6f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f038c1a73c0 con 0x562a2b1654d0 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0391b7f640 1 -- 172.21.15.27:0/80427950 learned_addr learned my addr 172.21.15.27:0/80427950 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f039137e640 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1654d0 0x562a2b53ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0392380640 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b537ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0392380640 1 -- 172.21.15.27:0/80427950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1654d0 msgr2=0x562a2b53ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0392380640 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1654d0 0x562a2b53ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0392380640 1 -- 172.21.15.27:0/80427950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 msgr2=0x562a2b5324c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0392380640 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b5324c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.790+0000 7f0392380640 1 -- 172.21.15.27:0/80427950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a2b530380 con 0x562a2b1861b0 2024-04-25T13:41:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.791+0000 7f0392380640 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b537ed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x562a2b530350 tx=0x7f0384051000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6c41d4e8fd601a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.791+0000 7f03777fe640 1 -- 172.21.15.27:0/80427950 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384057040 con 0x562a2b1861b0 2024-04-25T13:41:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.791+0000 7f03777fe640 1 -- 172.21.15.27:0/80427950 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0384036040 con 0x562a2b1861b0 2024-04-25T13:41:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.791+0000 7f03777fe640 1 -- 172.21.15.27:0/80427950 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038404a040 con 0x562a2b1861b0 2024-04-25T13:41:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.793+0000 7f039ad6f740 1 -- 172.21.15.27:0/80427950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 msgr2=0x562a2b537ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.793+0000 7f039ad6f740 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b537ed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x562a2b530350 tx=0x7f0384051000 comp rx=0 tx=0).stop 2024-04-25T13:41:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.794+0000 7f039ad6f740 1 -- 172.21.15.27:0/80427950 shutdown_connections 2024-04-25T13:41:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.794+0000 7f039ad6f740 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b537ed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.794+0000 7f039ad6f740 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b1654d0 0x562a2b53ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.794+0000 7f039ad6f740 1 --2- 172.21.15.27:0/80427950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b15d2f0 0x562a2b5324c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.794+0000 7f039ad6f740 1 -- 172.21.15.27:0/80427950 >> 172.21.15.27:0/80427950 conn(0x562a2b17a300 msgr2=0x562a2b52f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.794+0000 7f039ad6f740 1 -- 172.21.15.27:0/80427950 shutdown_connections 2024-04-25T13:41:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.794+0000 7f039ad6f740 1 -- 172.21.15.27:0/80427950 wait complete. 2024-04-25T13:41:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.795+0000 7f039ad6f740 1 -- start start 2024-04-25T13:41:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.795+0000 7f039ad6f740 1 Processor -- start 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b539f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b53a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039ad6f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b53f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039137e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b53a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039137e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b53a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54620/0 (socket says 172.21.15.27:54620) 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039137e640 1 -- 172.21.15.27:0/2685142331 learned_addr learned my addr 172.21.15.27:0/2685142331 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f038c1a7a90 con 0x562a2b1654d0 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f038c1a8f00 con 0x562a2b1861b0 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f038c1a8160 con 0x562a2b15d2f0 2024-04-25T13:41:21.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.796+0000 7f0392380640 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b53f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f0391b7f640 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b539f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f039ad6f740 10 client.0 _ll_register_callbacks cb 0x7f037c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f0391b7f640 1 -- 172.21.15.27:0/2685142331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 msgr2=0x562a2b53f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f0391b7f640 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b53f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f0391b7f640 1 -- 172.21.15.27:0/2685142331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 msgr2=0x562a2b53a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f0391b7f640 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b53a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f0391b7f640 1 -- 172.21.15.27:0/2685142331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a2b53f5f0 con 0x562a2b15d2f0 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0384036040 con 0x562a2b15d2f0 2024-04-25T13:41:21.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.797+0000 7f039137e640 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b53a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:21.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.798+0000 7f0391b7f640 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b539f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f03880039a0 tx=0x7f038800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d020e20cfd7205 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:21.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.798+0000 7f03757fa640 10 client.25120 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:41:21.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.798+0000 7f03757fa640 1 -- 172.21.15.27:0/2685142331 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03880a7be0 con 0x562a2b15d2f0 2024-04-25T13:41:21.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.798+0000 7f03757fa640 1 -- 172.21.15.27:0/2685142331 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f03880a3770 con 0x562a2b15d2f0 2024-04-25T13:41:21.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.798+0000 7f03757fa640 1 -- 172.21.15.27:0/2685142331 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f03880a74e0 con 0x562a2b15d2f0 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f039ad6f740 10 client.25120 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x562a2b5b74a0 con 0x562a2b15d2f0 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f03757fa640 1 -- 172.21.15.27:0/2685142331 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f03880e2b60 con 0x562a2b15d2f0 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f039ad6f740 10 client.25120 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f039ad6f740 10 client.25120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f039ad6f740 20 client.25120 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a2b11a1c0 0x562a2b5b9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f039ad6f740 4 client.25120 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:41:21.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.799+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a2b493ee0 con 0x562a2b11a1c0 2024-04-25T13:41:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.800+0000 7f039137e640 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a2b11a1c0 0x562a2b5b9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.800+0000 7f03757fa640 10 client.25120 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:21.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.800+0000 7f039137e640 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a2b11a1c0 0x562a2b5b9d90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19d849a326344fed server_cookie=bac80137d672cef8 in_seq=0 out_seq=0 2024-04-25T13:41:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.818+0000 7f03757fa640 1 -- 172.21.15.27:0/2685142331 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a2b493ee0 con 0x562a2b11a1c0 2024-04-25T13:41:21.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.818+0000 7f03757fa640 10 client.25120 handle_command_reply: tid=0 2024-04-25T13:41:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.819+0000 7f039ad6f740 1 client.25120 shutdown 2024-04-25T13:41:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.819+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 msgr2=0x562a2b539f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.819 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.820+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b539f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f03880039a0 tx=0x7f038800cf90 comp rx=0 tx=0).stop 2024-04-25T13:41:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.820+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 shutdown_connections 2024-04-25T13:41:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.820+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a2b11a1c0 0x562a2b5b9d90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.820+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a2b1861b0 0x562a2b53f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.820+0000 7f03757fa640 0 client.25120 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.820+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a2b1654d0 0x562a2b53a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.820+0000 7f039ad6f740 1 --2- 172.21.15.27:0/2685142331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a2b15d2f0 0x562a2b539f50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.820+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 >> 172.21.15.27:0/2685142331 conn(0x562a2b17a300 msgr2=0x562a2b537ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:21.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.822+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 shutdown_connections 2024-04-25T13:41:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.822+0000 7f039ad6f740 1 -- 172.21.15.27:0/2685142331 wait complete. 2024-04-25T13:41:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.822+0000 7f039ad6f740 20 client.25120 trim_cache size 0 max 16384 2024-04-25T13:41:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:41:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:41:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:41:21.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:41:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 -- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03780afa00 msgr2=0x7f03780b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03780afa00 0x7f03780b1f10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f03840094e0 tx=0x7f0384004910 comp rx=0 tx=0).stop 2024-04-25T13:41:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 -- 172.21.15.27:0/3389380037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f038c101790 msgr2=0x7f038c197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f038c101790 0x7f038c197f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f038c06a2f0 tx=0x7f038800ee50 comp rx=0 tx=0).stop 2024-04-25T13:41:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 -- 172.21.15.27:0/3389380037 shutdown_connections 2024-04-25T13:41:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03780afa00 0x7f03780b1f10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f038c1050e0 0x7f038c19f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f038c102290 0x7f038c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 --2- 172.21.15.27:0/3389380037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f038c101790 0x7f038c197f60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 -- 172.21.15.27:0/3389380037 >> 172.21.15.27:0/3389380037 conn(0x7f038c0fd390 msgr2=0x7f038c107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 -- 172.21.15.27:0/3389380037 shutdown_connections 2024-04-25T13:41:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:21.824+0000 7f0393de3640 1 -- 172.21.15.27:0/3389380037 wait complete. 2024-04-25T13:41:22.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:21 smithi122 ceph-mon[32843]: pgmap v363: 97 pgs: 97 active+clean; 917 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 MiB/s wr, 731 op/s 2024-04-25T13:41:22.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:21 smithi045 ceph-mon[32731]: pgmap v363: 97 pgs: 97 active+clean; 917 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 MiB/s wr, 731 op/s 2024-04-25T13:41:22.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:21 smithi027 ceph-mon[29140]: pgmap v363: 97 pgs: 97 active+clean; 917 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 MiB/s wr, 731 op/s 2024-04-25T13:41:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 15.0 s, 4017.0 tps, lat 24.885 ms stddev 24.695 2024-04-25T13:41:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:23 smithi122 ceph-mon[32843]: pgmap v364: 97 pgs: 97 active+clean; 928 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 775 op/s 2024-04-25T13:41:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:23 smithi045 ceph-mon[32731]: pgmap v364: 97 pgs: 97 active+clean; 928 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 775 op/s 2024-04-25T13:41:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:23 smithi027 ceph-mon[29140]: pgmap v364: 97 pgs: 97 active+clean; 928 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 18 MiB/s wr, 775 op/s 2024-04-25T13:41:25.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:41:25.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:41:25.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 -- 172.21.15.27:0/490122957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 msgr2=0x7f33a8111b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 --2- 172.21.15.27:0/490122957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 0x7f33a8111b80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3398009900 tx=0x7f339802a3f0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 -- 172.21.15.27:0/490122957 shutdown_connections 2024-04-25T13:41:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 --2- 172.21.15.27:0/490122957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33a81120c0 0x7f33a8114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 --2- 172.21.15.27:0/490122957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 0x7f33a8111b80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 --2- 172.21.15.27:0/490122957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a8109fc0 0x7f33a810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 -- 172.21.15.27:0/490122957 >> 172.21.15.27:0/490122957 conn(0x7f33a80704e0 msgr2=0x7f33a8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 -- 172.21.15.27:0/490122957 shutdown_connections 2024-04-25T13:41:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.648+0000 7f33ad692640 1 -- 172.21.15.27:0/490122957 wait complete. 2024-04-25T13:41:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.649+0000 7f33ad692640 1 Processor -- start 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.649+0000 7f33ad692640 1 -- start start 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33ad692640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a8109fc0 0x7f33a81a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33ad692640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 0x7f33a81a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33ad692640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33a81120c0 0x7f33a81a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33ad692640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8117e90 con 0x7f33a8109fc0 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33ad692640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8117d10 con 0x7f33a81120c0 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33ad692640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8118010 con 0x7f33a810f760 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 0x7f33a81a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 0x7f33a81a2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:41926/0 (socket says 172.21.15.27:41926) 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33a67fc640 1 -- 172.21.15.27:0/179448240 learned_addr learned my addr 172.21.15.27:0/179448240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33a77fe640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33a81120c0 0x7f33a81a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.650+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a8109fc0 0x7f33a81a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.651+0000 7f33a77fe640 1 -- 172.21.15.27:0/179448240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 msgr2=0x7f33a81a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.651+0000 7f33a77fe640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 0x7f33a81a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.651+0000 7f33a77fe640 1 -- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a8109fc0 msgr2=0x7f33a81a2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.651+0000 7f33a77fe640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a8109fc0 0x7f33a81a2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.651+0000 7f33a77fe640 1 -- 172.21.15.27:0/179448240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a810b5c0 con 0x7f33a81120c0 2024-04-25T13:41:25.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.651+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a8109fc0 0x7f33a81a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:25.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.651+0000 7f33a67fc640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 0x7f33a81a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.651+0000 7f33a77fe640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33a81120c0 0x7f33a81a9f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f339c00e970 tx=0x7f339c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.652+0000 7f338ffff640 1 -- 172.21.15.27:0/179448240 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c00ccc0 con 0x7f33a81120c0 2024-04-25T13:41:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.652+0000 7f338ffff640 1 -- 172.21.15.27:0/179448240 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f339c005020 con 0x7f33a81120c0 2024-04-25T13:41:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.652+0000 7f338ffff640 1 -- 172.21.15.27:0/179448240 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c00ccc0 con 0x7f33a81120c0 2024-04-25T13:41:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.652+0000 7f33ad692640 1 -- 172.21.15.27:0/179448240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a810b8b0 con 0x7f33a81120c0 2024-04-25T13:41:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.652+0000 7f33ad692640 1 -- 172.21.15.27:0/179448240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a810bdb0 con 0x7f33a81120c0 2024-04-25T13:41:25.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f33b4615740 1 Processor -- start 2024-04-25T13:41:25.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f338ffff640 1 -- 172.21.15.27:0/179448240 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f339c0047a0 con 0x7f33a81120c0 2024-04-25T13:41:25.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f33b4615740 1 -- start start 2024-04-25T13:41:25.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f33b4615740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdea390 0x557fac18ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f33b4615740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc14d0 0x557fac18bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f33b4615740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 0x557fac18cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f33b4615740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557fac0f0d80 con 0x557fabdc14d0 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f33b4615740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557fac0aa8b0 con 0x557fabdc96b0 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.657+0000 7f33b4615740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557fac0ecd40 con 0x557fabdea390 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc14d0 0x557fac18bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdea390 0x557fac18ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc14d0 0x557fac18bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51764/0 (socket says 172.21.15.27:51764) 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a6ffd640 1 -- 172.21.15.27:0/2909597841 learned_addr learned my addr 172.21.15.27:0/2909597841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a67fc640 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 0x557fac18cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a67fc640 1 -- 172.21.15.27:0/2909597841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 msgr2=0x557fac18cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a67fc640 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 0x557fac18cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a67fc640 1 -- 172.21.15.27:0/2909597841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc14d0 msgr2=0x557fac18bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a67fc640 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc14d0 0x557fac18bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a67fc640 1 -- 172.21.15.27:0/2909597841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fac07ba40 con 0x557fabdea390 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc14d0 0x557fac18bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f338ffff640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33900afee0 0x7f33900b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a67fc640 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdea390 0x557fac18ba60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33a81a3b10 tx=0x7f3398032ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78d1bceb404ee2c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33900afee0 0x7f33900b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f338ffff640 1 -- 172.21.15.27:0/179448240 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f339c00cac0 con 0x7f33a81120c0 2024-04-25T13:41:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f338d7fa640 1 -- 172.21.15.27:0/2909597841 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398035040 con 0x557fabdea390 2024-04-25T13:41:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f338d7fa640 1 -- 172.21.15.27:0/2909597841 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33980035f0 con 0x557fabdea390 2024-04-25T13:41:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.658+0000 7f338d7fa640 1 -- 172.21.15.27:0/2909597841 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339803d6b0 con 0x557fabdea390 2024-04-25T13:41:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.659+0000 7f33b4615740 1 -- 172.21.15.27:0/2909597841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdea390 msgr2=0x557fac18ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.659+0000 7f33b4615740 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdea390 0x557fac18ba60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f33a81a3b10 tx=0x7f3398032ed0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.659+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33900afee0 0x7f33900b23f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x557fac18b780 tx=0x7f33a0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.661+0000 7f33b4615740 1 -- 172.21.15.27:0/2909597841 shutdown_connections 2024-04-25T13:41:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.661+0000 7f33b4615740 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 0x557fac18cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.661+0000 7f33b4615740 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc14d0 0x557fac18bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.661+0000 7f33b4615740 1 --2- 172.21.15.27:0/2909597841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdea390 0x557fac18ba60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.661+0000 7f33b4615740 1 -- 172.21.15.27:0/2909597841 >> 172.21.15.27:0/2909597841 conn(0x557fabdde4e0 msgr2=0x557fac187270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:25.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.661+0000 7f33b4615740 1 -- 172.21.15.27:0/2909597841 shutdown_connections 2024-04-25T13:41:25.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.661+0000 7f33b4615740 1 -- 172.21.15.27:0/2909597841 wait complete. 2024-04-25T13:41:25.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 1 -- start start 2024-04-25T13:41:25.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 1 Processor -- start 2024-04-25T13:41:25.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc14d0 0x557fac195a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc96b0 0x557fac195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdea390 0x557fac19cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557fac18d720 con 0x557fabdea390 2024-04-25T13:41:25.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557fac077d10 con 0x557fabdc14d0 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557fac18d8a0 con 0x557fabdc96b0 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.662+0000 7f33b4615740 10 client.0 _ll_register_callbacks cb 0x557fac0a4210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc96b0 0x557fac195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc14d0 0x557fac195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc96b0 0x557fac195fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:41946/0 (socket says 172.21.15.27:41946) 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 -- 172.21.15.27:0/3733343269 learned_addr learned my addr 172.21.15.27:0/3733343269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc14d0 0x557fac195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43208/0 (socket says 172.21.15.27:43208) 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a77fe640 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdea390 0x557fac19cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 -- 172.21.15.27:0/3733343269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc14d0 msgr2=0x557fac195a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc14d0 0x557fac195a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 -- 172.21.15.27:0/3733343269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdea390 msgr2=0x557fac19cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdea390 0x557fac19cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 -- 172.21.15.27:0/3733343269 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fabecc7d0 con 0x557fabdc96b0 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc14d0 0x557fac195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33a67fc640 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc96b0 0x557fac195fa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3398031440 tx=0x7f3398009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cec69f9469168304 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:25.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f338aff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f338aff5640 1 -- 172.21.15.27:0/3733343269 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339804e070 con 0x557fabdc96b0 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f33b4615740 1 -- 172.21.15.27:0/3733343269 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33980035f0 con 0x557fabdc96b0 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.663+0000 7f338aff5640 1 -- 172.21.15.27:0/3733343269 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3398004240 con 0x557fabdc96b0 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f338aff5640 1 -- 172.21.15.27:0/3733343269 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f339805a070 con 0x557fabdc96b0 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f33b4615740 10 client.25132 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f33b4615740 1 -- 172.21.15.27:0/3733343269 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557fabe023f0 con 0x557fabdc96b0 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f338aff5640 1 -- 172.21.15.27:0/3733343269 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33980035f0 con 0x557fabdc96b0 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f338aff5640 1 -- 172.21.15.27:0/3733343269 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f339803d450 con 0x557fabdc96b0 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f33b4615740 10 client.25132 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f33b4615740 10 client.25132 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f33b4615740 20 client.25132 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f33b4615740 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fabda8f60 0x557fac1a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f33b4615740 4 client.25132 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:25.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.664+0000 7f33b4615740 1 -- 172.21.15.27:0/3733343269 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fac0fe990 con 0x557fabda8f60 2024-04-25T13:41:25.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.665+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fabda8f60 0x557fac1a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.666+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fabda8f60 0x557fac1a2fc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ef356ca37c179e8 server_cookie=236362f4165e1e21 in_seq=0 out_seq=0 2024-04-25T13:41:25.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.666+0000 7f338aff5640 10 client.25132 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:25.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.667+0000 7f338aff5640 1 -- 172.21.15.27:0/3733343269 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557fac0fe990 con 0x557fabda8f60 2024-04-25T13:41:25.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.667+0000 7f338aff5640 10 client.25132 handle_command_reply: tid=0 2024-04-25T13:41:25.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.668+0000 7f33b4615740 1 client.25132 shutdown 2024-04-25T13:41:25.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.668+0000 7f33b4615740 1 -- 172.21.15.27:0/3733343269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc96b0 msgr2=0x557fac195fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.668+0000 7f33b4615740 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc96b0 0x557fac195fa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3398031440 tx=0x7f3398009260 comp rx=0 tx=0).stop 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 1 -- 172.21.15.27:0/3733343269 shutdown_connections 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fabda8f60 0x557fac1a2fc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f338aff5640 0 client.25132 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdea390 0x557fac19cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc96b0 0x557fac195fa0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 1 --2- 172.21.15.27:0/3733343269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc14d0 0x557fac195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 1 -- 172.21.15.27:0/3733343269 >> 172.21.15.27:0/3733343269 conn(0x557fabdde4e0 msgr2=0x557fac18a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 1 -- 172.21.15.27:0/3733343269 shutdown_connections 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 1 -- 172.21.15.27:0/3733343269 wait complete. 2024-04-25T13:41:25.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.669+0000 7f33b4615740 20 client.25132 trim_cache size 0 max 16384 2024-04-25T13:41:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33b4615740 1 Processor -- start 2024-04-25T13:41:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33b4615740 1 -- start start 2024-04-25T13:41:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33b4615740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabda8f60 0x557fac194080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33b4615740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac1945c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33b4615740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc96b0 0x557fac19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33b4615740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81b3b90 con 0x557fabdc96b0 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33b4615740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81b6410 con 0x557fabda8f60 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33b4615740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81b4200 con 0x557fabdc14d0 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33a6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabda8f60 0x557fac194080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac1945c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc96b0 0x557fac19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33a6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabda8f60 0x557fac194080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43212/0 (socket says 172.21.15.27:43212) 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33a6ffd640 1 -- 172.21.15.27:0/747681554 learned_addr learned my addr 172.21.15.27:0/747681554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac1945c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:41952/0 (socket says 172.21.15.27:41952) 2024-04-25T13:41:25.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.702+0000 7f33a77fe640 1 --2- 172.21.15.27:0/747681554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc96b0 0x557fac19ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51784/0 (socket says 172.21.15.27:51784) 2024-04-25T13:41:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.703+0000 7f33a77fe640 1 -- 172.21.15.27:0/747681554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 msgr2=0x557fac1945c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.703+0000 7f33a77fe640 1 --2- 172.21.15.27:0/747681554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac1945c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.703+0000 7f33a77fe640 1 -- 172.21.15.27:0/747681554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabda8f60 msgr2=0x557fac194080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.703+0000 7f33a77fe640 1 --2- 172.21.15.27:0/747681554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabda8f60 0x557fac194080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.703+0000 7f33a77fe640 1 -- 172.21.15.27:0/747681554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fabdc20f0 con 0x557fabdc96b0 2024-04-25T13:41:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.703+0000 7f33a77fe640 1 --2- 172.21.15.27:0/747681554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc96b0 0x557fac19ce20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f339c0dc090 tx=0x7f339c0af280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d5a24895cf5fed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:25.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.704+0000 7f338cff9640 1 -- 172.21.15.27:0/747681554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c01c040 con 0x557fabdc96b0 2024-04-25T13:41:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.704+0000 7f338cff9640 1 -- 172.21.15.27:0/747681554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f339c0a6a20 con 0x557fabdc96b0 2024-04-25T13:41:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.704+0000 7f338cff9640 1 -- 172.21.15.27:0/747681554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c00ccc0 con 0x557fabdc96b0 2024-04-25T13:41:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.705+0000 7f33b4615740 1 -- 172.21.15.27:0/747681554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc96b0 msgr2=0x557fac19ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.705+0000 7f33b4615740 1 --2- 172.21.15.27:0/747681554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc96b0 0x557fac19ce20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f339c0dc090 tx=0x7f339c0af280 comp rx=0 tx=0).stop 2024-04-25T13:41:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.705+0000 7f33b4615740 1 -- 172.21.15.27:0/747681554 shutdown_connections 2024-04-25T13:41:25.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.705+0000 7f33b4615740 1 --2- 172.21.15.27:0/747681554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabdc96b0 0x557fac19ce20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.705+0000 7f33b4615740 1 --2- 172.21.15.27:0/747681554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac1945c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.705+0000 7f33b4615740 1 --2- 172.21.15.27:0/747681554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabda8f60 0x557fac194080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.705+0000 7f33b4615740 1 -- 172.21.15.27:0/747681554 >> 172.21.15.27:0/747681554 conn(0x557fabdde4e0 msgr2=0x557fac1907c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:25.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.706+0000 7f33b4615740 1 -- 172.21.15.27:0/747681554 shutdown_connections 2024-04-25T13:41:25.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.707+0000 7f33b4615740 1 -- 172.21.15.27:0/747681554 wait complete. 2024-04-25T13:41:25.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.707+0000 7f33b4615740 1 -- start start 2024-04-25T13:41:25.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33b4615740 1 Processor -- start 2024-04-25T13:41:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33b4615740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabda8f60 0x557fac19d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33b4615740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33b4615740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 0x557fac199c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33b4615740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81b48d0 con 0x557fabda8f60 2024-04-25T13:41:25.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33b4615740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81b5d40 con 0x557fabdc96b0 2024-04-25T13:41:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33b4615740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81b4fa0 con 0x557fabdc14d0 2024-04-25T13:41:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33b4615740 10 client.0 _ll_register_callbacks cb 0x7f33a0009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33a6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabda8f60 0x557fac19d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33a67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac19d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:41968/0 (socket says 172.21.15.27:41968) 2024-04-25T13:41:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33a67fc640 1 -- 172.21.15.27:0/4146745025 learned_addr learned my addr 172.21.15.27:0/4146745025 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:25.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.708+0000 7f33a77fe640 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 0x557fac199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.709+0000 7f33a67fc640 1 -- 172.21.15.27:0/4146745025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 msgr2=0x557fac199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.709+0000 7f33a67fc640 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 0x557fac199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.709+0000 7f33a67fc640 1 -- 172.21.15.27:0/4146745025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabda8f60 msgr2=0x557fac19d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.709+0000 7f33a67fc640 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabda8f60 0x557fac19d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.709+0000 7f33a67fc640 1 -- 172.21.15.27:0/4146745025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fac0b14b0 con 0x557fabdc14d0 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.710+0000 7f33a67fc640 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac19d5c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f339802a430 tx=0x7f33980095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9caf70dd117a64cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.710+0000 7f338aff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.710+0000 7f338aff5640 1 -- 172.21.15.27:0/4146745025 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339804e070 con 0x557fabdc14d0 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.710+0000 7f338aff5640 1 -- 172.21.15.27:0/4146745025 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3398005440 con 0x557fabdc14d0 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.710+0000 7f33b4615740 1 -- 172.21.15.27:0/4146745025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f339c0a6a20 con 0x557fabdc14d0 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.710+0000 7f338aff5640 1 -- 172.21.15.27:0/4146745025 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f3398035040 con 0x557fabdc14d0 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.710+0000 7f33b4615740 10 client.25138 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:25.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.710+0000 7f33b4615740 1 -- 172.21.15.27:0/4146745025 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557fac209cd0 con 0x557fabdc14d0 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f338aff5640 1 -- 172.21.15.27:0/4146745025 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f339c0a6a20 con 0x557fabdc14d0 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f338aff5640 1 -- 172.21.15.27:0/4146745025 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f3398004c00 con 0x557fabdc14d0 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f33b4615740 10 client.25138 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f33b4615740 10 client.25138 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f33b4615740 20 client.25138 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f33b4615740 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fabdea390 0x557fac20c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f33b4615740 4 client.25138 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f33b4615740 1 -- 172.21.15.27:0/4146745025 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557fac0f8800 con 0x557fabdea390 2024-04-25T13:41:25.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fabdea390 0x557fac20c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.711+0000 7f338aff5640 10 client.25138 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.712+0000 7f33a6ffd640 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fabdea390 0x557fac20c6e0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd25a0e5ae39a7d1 server_cookie=bd9bcfc00a5c5ed4 in_seq=0 out_seq=0 2024-04-25T13:41:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.712+0000 7f338aff5640 1 -- 172.21.15.27:0/4146745025 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557fac0f8800 con 0x557fabdea390 2024-04-25T13:41:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.712+0000 7f338aff5640 10 client.25138 handle_command_reply: tid=0 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.712+0000 7f33b4615740 1 client.25138 shutdown 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 -- 172.21.15.27:0/4146745025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 msgr2=0x557fac19d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac19d5c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f339802a430 tx=0x7f33980095e0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 -- 172.21.15.27:0/4146745025 shutdown_connections 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fabdea390 0x557fac20c6e0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fabdc96b0 0x557fac199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f338aff5640 0 client.25138 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fabdc14d0 0x557fac19d5c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 --2- 172.21.15.27:0/4146745025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fabda8f60 0x557fac19d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 -- 172.21.15.27:0/4146745025 >> 172.21.15.27:0/4146745025 conn(0x557fabdde4e0 msgr2=0x557fac194080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 -- 172.21.15.27:0/4146745025 shutdown_connections 2024-04-25T13:41:25.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.713+0000 7f33b4615740 1 -- 172.21.15.27:0/4146745025 wait complete. 2024-04-25T13:41:25.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.714+0000 7f33b4615740 20 client.25138 trim_cache size 0 max 16384 2024-04-25T13:41:25.715 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:41:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:41:25.715 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:41:25.715 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:41:25.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.714+0000 7f33ad692640 1 -- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33900afee0 msgr2=0x7f33900b23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.714+0000 7f33ad692640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33900afee0 0x7f33900b23f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x557fac18b780 tx=0x7f33a0009500 comp rx=0 tx=0).stop 2024-04-25T13:41:25.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.714+0000 7f33ad692640 1 -- 172.21.15.27:0/179448240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33a81120c0 msgr2=0x7f33a81a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.714+0000 7f33ad692640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33a81120c0 0x7f33a81a9f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f339c00e970 tx=0x7f339c00ee30 comp rx=0 tx=0).stop 2024-04-25T13:41:25.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.715+0000 7f33ad692640 1 -- 172.21.15.27:0/179448240 shutdown_connections 2024-04-25T13:41:25.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.715+0000 7f33ad692640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33900afee0 0x7f33900b23f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.715+0000 7f33ad692640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33a81120c0 0x7f33a81a9f40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.715+0000 7f33ad692640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33a810f760 0x7f33a81a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.715+0000 7f33ad692640 1 --2- 172.21.15.27:0/179448240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a8109fc0 0x7f33a81a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.715+0000 7f33ad692640 1 -- 172.21.15.27:0/179448240 >> 172.21.15.27:0/179448240 conn(0x7f33a80704e0 msgr2=0x7f33a81106a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:25.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.715+0000 7f33ad692640 1 -- 172.21.15.27:0/179448240 shutdown_connections 2024-04-25T13:41:25.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.715+0000 7f33ad692640 1 -- 172.21.15.27:0/179448240 wait complete. 2024-04-25T13:41:25.730 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:41:25.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:41:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 -- 172.21.15.27:0/2535824236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c4104b70 msgr2=0x7fc5c4104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/2535824236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c4104b70 0x7fc5c4104f80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009900 tx=0x7fc5b402a410 comp rx=0 tx=0).stop 2024-04-25T13:41:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 -- 172.21.15.27:0/2535824236 shutdown_connections 2024-04-25T13:41:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/2535824236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c4109870 0x7fc5c410bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/2535824236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c4104b70 0x7fc5c4104f80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/2535824236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c4103930 0x7fc5c4104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 -- 172.21.15.27:0/2535824236 >> 172.21.15.27:0/2535824236 conn(0x7fc5c40feb20 msgr2=0x7fc5c4100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 -- 172.21.15.27:0/2535824236 shutdown_connections 2024-04-25T13:41:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.899+0000 7fc5caf9d640 1 -- 172.21.15.27:0/2535824236 wait complete. 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.900+0000 7fc5caf9d640 1 Processor -- start 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.900+0000 7fc5caf9d640 1 -- start start 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.900+0000 7fc5caf9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c4103930 0x7fc5c419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.900+0000 7fc5caf9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c4104b70 0x7fc5c419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.900+0000 7fc5caf9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c4109870 0x7fc5c41a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.900+0000 7fc5caf9d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c410f4a0 con 0x7fc5c4109870 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.900+0000 7fc5caf9d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c410f320 con 0x7fc5c4103930 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.900+0000 7fc5caf9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c410f620 con 0x7fc5c4104b70 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c9f9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c4103930 0x7fc5c419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c9f9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c4103930 0x7fc5c419a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43242/0 (socket says 172.21.15.27:43242) 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c9f9b640 1 -- 172.21.15.27:0/680109694 learned_addr learned my addr 172.21.15.27:0/680109694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c979a640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c4104b70 0x7fc5c419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5ca79c640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c4109870 0x7fc5c41a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c979a640 1 -- 172.21.15.27:0/680109694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c4103930 msgr2=0x7fc5c419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c979a640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c4103930 0x7fc5c419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c979a640 1 -- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c4109870 msgr2=0x7fc5c41a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c979a640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c4109870 0x7fc5c41a14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c979a640 1 -- 172.21.15.27:0/680109694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c41a1b50 con 0x7fc5c4104b70 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.901+0000 7fc5c9f9b640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c4103930 0x7fc5c419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.902+0000 7fc5c979a640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c4104b70 0x7fc5c419a550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc5b402aa00 tx=0x7fc5b4003bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.902+0000 7fc5baffd640 1 -- 172.21.15.27:0/680109694 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b403f070 con 0x7fc5c4104b70 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.902+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c41a1de0 con 0x7fc5c4104b70 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.902+0000 7fc5baffd640 1 -- 172.21.15.27:0/680109694 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5b4031c50 con 0x7fc5c4104b70 2024-04-25T13:41:25.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.902+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c4074fb0 con 0x7fc5c4104b70 2024-04-25T13:41:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.902+0000 7fc5baffd640 1 -- 172.21.15.27:0/680109694 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4035030 con 0x7fc5c4104b70 2024-04-25T13:41:25.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.904+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c4108680 con 0x7fc5c4104b70 2024-04-25T13:41:25.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.906+0000 7fc5baffd640 1 -- 172.21.15.27:0/680109694 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc5b4042070 con 0x7fc5c4104b70 2024-04-25T13:41:25.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.907+0000 7fc5baffd640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc5b00b3f40 0x7fc5b00b6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:25.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.907+0000 7fc5baffd640 1 -- 172.21.15.27:0/680109694 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc5b40ffa70 con 0x7fc5c4104b70 2024-04-25T13:41:25.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.907+0000 7fc5c9f9b640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc5b00b3f40 0x7fc5b00b6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:25.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.908+0000 7fc5c9f9b640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc5b00b3f40 0x7fc5b00b6450 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0005d00 tx=0x7fc5c0005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:25.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:25.910+0000 7fc5baffd640 1 -- 172.21.15.27:0/680109694 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5b40c8680 con 0x7fc5c4104b70 2024-04-25T13:41:26.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:25 smithi027 ceph-mon[29140]: pgmap v365: 97 pgs: 97 active+clean; 930 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.6 MiB/s wr, 691 op/s 2024-04-25T13:41:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:25 smithi122 ceph-mon[32843]: pgmap v365: 97 pgs: 97 active+clean; 930 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.6 MiB/s wr, 691 op/s 2024-04-25T13:41:26.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.317+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5c402db50 con 0x7fc5c4104b70 2024-04-25T13:41:26.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.318+0000 7fc5baffd640 1 -- 172.21.15.27:0/680109694 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fc5b40c7f80 con 0x7fc5c4104b70 2024-04-25T13:41:26.318 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:26.318 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:41:26.319 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc5b00b3f40 msgr2=0x7fc5b00b6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc5b00b3f40 0x7fc5b00b6450 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0005d00 tx=0x7fc5c0005f50 comp rx=0 tx=0).stop 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c4104b70 msgr2=0x7fc5c419a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c4104b70 0x7fc5c419a550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc5b402aa00 tx=0x7fc5b4003bb0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 shutdown_connections 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc5b00b3f40 0x7fc5b00b6450 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c4109870 0x7fc5c41a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc5c4104b70 0x7fc5c419a550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 --2- 172.21.15.27:0/680109694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5c4103930 0x7fc5c419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 >> 172.21.15.27:0/680109694 conn(0x7fc5c40feb20 msgr2=0x7fc5c4105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:26.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 shutdown_connections 2024-04-25T13:41:26.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.322+0000 7fc5caf9d640 1 -- 172.21.15.27:0/680109694 wait complete. 2024-04-25T13:41:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:25 smithi045 ceph-mon[32731]: pgmap v365: 97 pgs: 97 active+clean; 930 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.6 MiB/s wr, 691 op/s 2024-04-25T13:41:26.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 -- 172.21.15.27:0/795356887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940104250 msgr2=0x7f6940106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 --2- 172.21.15.27:0/795356887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940104250 0x7f6940106670 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6930009900 tx=0x7f693002a410 comp rx=0 tx=0).stop 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 -- 172.21.15.27:0/795356887 shutdown_connections 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 --2- 172.21.15.27:0/795356887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6940109770 0x7f694010bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 --2- 172.21.15.27:0/795356887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6940106c40 0x7f69401090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 --2- 172.21.15.27:0/795356887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940104250 0x7f6940106670 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 -- 172.21.15.27:0/795356887 >> 172.21.15.27:0/795356887 conn(0x7f69400fdbd0 msgr2=0x7f6940100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 -- 172.21.15.27:0/795356887 shutdown_connections 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 -- 172.21.15.27:0/795356887 wait complete. 2024-04-25T13:41:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.507+0000 7f6947914640 1 Processor -- start 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6947914640 1 -- start start 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6947914640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6940104250 0x7f694019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6947914640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940106c40 0x7f694019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6947914640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6940109770 0x7f69401a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6947914640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f694010f620 con 0x7f6940106c40 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6947914640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f694010f4a0 con 0x7f6940109770 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6947914640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f694010f7a0 con 0x7f6940104250 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940106c40 0x7f694019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940106c40 0x7f694019aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51832/0 (socket says 172.21.15.27:51832) 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 -- 172.21.15.27:0/704464195 learned_addr learned my addr 172.21.15.27:0/704464195 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f69456b0640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6940104250 0x7f694019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6945eb1640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6940109770 0x7f69401a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 -- 172.21.15.27:0/704464195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6940104250 msgr2=0x7f694019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6940104250 0x7f694019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 -- 172.21.15.27:0/704464195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6940109770 msgr2=0x7f69401a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6940109770 0x7f69401a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 -- 172.21.15.27:0/704464195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69401a20d0 con 0x7f6940106c40 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.508+0000 7f6944eaf640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940106c40 0x7f694019aad0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f693c00e9f0 tx=0x7f693c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.509+0000 7f69367fc640 1 -- 172.21.15.27:0/704464195 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c00ccc0 con 0x7f6940106c40 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.509+0000 7f6947914640 1 -- 172.21.15.27:0/704464195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69401a23c0 con 0x7f6940106c40 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.509+0000 7f69367fc640 1 -- 172.21.15.27:0/704464195 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f693c00ce60 con 0x7f6940106c40 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.509+0000 7f69367fc640 1 -- 172.21.15.27:0/704464195 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c010430 con 0x7f6940106c40 2024-04-25T13:41:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.509+0000 7f6947914640 1 -- 172.21.15.27:0/704464195 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69401a2910 con 0x7f6940106c40 2024-04-25T13:41:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.512+0000 7f694e7c9740 1 Processor -- start 2024-04-25T13:41:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.512+0000 7f694e7c9740 1 -- start start 2024-04-25T13:41:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.512+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 0x564a14034c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a140351a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c6d6b0 0x564a14035e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f694e7c9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a13f94d80 con 0x564a13c6d6b0 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f694e7c9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564a13f4ea20 con 0x564a13c654d0 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f694e7c9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564a13f90d40 con 0x564a13c8e390 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a140351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f6945eb1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 0x564a14034c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a140351a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43274/0 (socket says 172.21.15.27:43274) 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 -- 172.21.15.27:0/1244143409 learned_addr learned my addr 172.21.15.27:0/1244143409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 -- 172.21.15.27:0/1244143409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 msgr2=0x564a14034c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 --2- 172.21.15.27:0/1244143409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 0x564a14034c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 -- 172.21.15.27:0/1244143409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c6d6b0 msgr2=0x564a14035e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 --2- 172.21.15.27:0/1244143409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c6d6b0 0x564a14035e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 -- 172.21.15.27:0/1244143409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a13f59c30 con 0x564a13c654d0 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69367fc640 1 -- 172.21.15.27:0/704464195 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f693c0046c0 con 0x7f6940106c40 2024-04-25T13:41:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.513+0000 7f69456b0640 1 --2- 172.21.15.27:0/1244143409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a140351a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f694006a810 tx=0x7f6930004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=569f65191ad063ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.514+0000 7f692b7fe640 1 -- 172.21.15.27:0/1244143409 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930035030 con 0x564a13c654d0 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.514+0000 7f692b7fe640 1 -- 172.21.15.27:0/1244143409 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6930040070 con 0x564a13c654d0 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.514+0000 7f69367fc640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f692c0afb70 0x7f692c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.514+0000 7f69367fc640 1 -- 172.21.15.27:0/704464195 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f693c0def70 con 0x7f6940106c40 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.514+0000 7f692b7fe640 1 -- 172.21.15.27:0/1244143409 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930035030 con 0x564a13c654d0 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.515+0000 7f6945eb1640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f692c0afb70 0x7f692c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.515+0000 7f694e7c9740 1 -- 172.21.15.27:0/1244143409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 msgr2=0x564a140351a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.515+0000 7f694e7c9740 1 --2- 172.21.15.27:0/1244143409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a140351a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f694006a810 tx=0x7f6930004040 comp rx=0 tx=0).stop 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.516+0000 7f694e7c9740 1 -- 172.21.15.27:0/1244143409 shutdown_connections 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.516+0000 7f694e7c9740 1 --2- 172.21.15.27:0/1244143409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c6d6b0 0x564a14035e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.516+0000 7f694e7c9740 1 --2- 172.21.15.27:0/1244143409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a140351a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.516+0000 7f694e7c9740 1 --2- 172.21.15.27:0/1244143409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 0x564a14034c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.516+0000 7f694e7c9740 1 -- 172.21.15.27:0/1244143409 >> 172.21.15.27:0/1244143409 conn(0x564a13c824e0 msgr2=0x564a1402c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:26.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.516+0000 7f6945eb1640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f692c0afb70 0x7f692c0b2080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x564a14035c70 tx=0x7f6938009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:26.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.517+0000 7f694e7c9740 1 -- 172.21.15.27:0/1244143409 shutdown_connections 2024-04-25T13:41:26.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.517+0000 7f694e7c9740 1 -- 172.21.15.27:0/1244143409 wait complete. 2024-04-25T13:41:26.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.517+0000 7f694e7c9740 1 -- start start 2024-04-25T13:41:26.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.517+0000 7f694e7c9740 1 Processor -- start 2024-04-25T13:41:26.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c654d0 0x564a1403e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a1403ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a14045da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f694e7c9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a14036930 con 0x564a13c8e390 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f694e7c9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564a13f1bdc0 con 0x564a13c6d6b0 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f694e7c9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564a14036ab0 con 0x564a13c654d0 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f694e7c9740 10 client.0 _ll_register_callbacks cb 0x564a13f37600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6945eb1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a14045da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6944eaf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a1403ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6945eb1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a14045da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51840/0 (socket says 172.21.15.27:51840) 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6945eb1640 1 -- 172.21.15.27:0/4086793149 learned_addr learned my addr 172.21.15.27:0/4086793149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f69456b0640 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c654d0 0x564a1403e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6944eaf640 1 -- 172.21.15.27:0/4086793149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c654d0 msgr2=0x564a1403e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6944eaf640 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c654d0 0x564a1403e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6944eaf640 1 -- 172.21.15.27:0/4086793149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 msgr2=0x564a14045da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6944eaf640 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a14045da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6944eaf640 1 -- 172.21.15.27:0/4086793149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a13ca63f0 con 0x564a13c6d6b0 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.518+0000 7f6945eb1640 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a14045da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.519+0000 7f6944eaf640 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a1403ee40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f693c0dcaf0 tx=0x7f693c0a70d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb17373bd5fd6ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.519+0000 7f6928ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.519+0000 7f6928ff9640 1 -- 172.21.15.27:0/4086793149 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0ab830 con 0x564a13c6d6b0 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.519+0000 7f6928ff9640 1 -- 172.21.15.27:0/4086793149 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f693c0b0060 con 0x564a13c6d6b0 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.519+0000 7f694e7c9740 1 -- 172.21.15.27:0/4086793149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6930040070 con 0x564a13c6d6b0 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f6928ff9640 1 -- 172.21.15.27:0/4086793149 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f693c0b02e0 con 0x564a13c6d6b0 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f694e7c9740 10 client.25148 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f694e7c9740 1 -- 172.21.15.27:0/4086793149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564a13d707d0 con 0x564a13c6d6b0 2024-04-25T13:41:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f6928ff9640 1 -- 172.21.15.27:0/4086793149 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6930040070 con 0x564a13c6d6b0 2024-04-25T13:41:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f6928ff9640 1 -- 172.21.15.27:0/4086793149 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f693c0a6150 con 0x564a13c6d6b0 2024-04-25T13:41:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f694e7c9740 10 client.25148 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f694e7c9740 10 client.25148 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:26.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f694e7c9740 20 client.25148 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.520+0000 7f694e7c9740 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a13c4cf60 0x564a1404c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.521+0000 7f694e7c9740 4 client.25148 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.521+0000 7f694e7c9740 1 -- 172.21.15.27:0/4086793149 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a13fa2990 con 0x564a13c4cf60 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.521+0000 7f69456b0640 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a13c4cf60 0x564a1404c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.521+0000 7f6928ff9640 10 client.25148 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.521+0000 7f69456b0640 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a13c4cf60 0x564a1404c6b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e397cb9077315e8 server_cookie=ad67942da0c6dcb6 in_seq=0 out_seq=0 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f6928ff9640 1 -- 172.21.15.27:0/4086793149 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564a13fa2990 con 0x564a13c4cf60 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f6928ff9640 10 client.25148 handle_command_reply: tid=0 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 client.25148 shutdown 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 -- 172.21.15.27:0/4086793149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 msgr2=0x564a1403ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a1403ee40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f693c0dcaf0 tx=0x7f693c0a70d0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 -- 172.21.15.27:0/4086793149 shutdown_connections 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a13c4cf60 0x564a1404c6b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a14045da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a1403ee40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 --2- 172.21.15.27:0/4086793149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c654d0 0x564a1403e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f6928ff9640 0 client.25148 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 -- 172.21.15.27:0/4086793149 >> 172.21.15.27:0/4086793149 conn(0x564a13c824e0 msgr2=0x564a1402ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 -- 172.21.15.27:0/4086793149 shutdown_connections 2024-04-25T13:41:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 1 -- 172.21.15.27:0/4086793149 wait complete. 2024-04-25T13:41:26.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.522+0000 7f694e7c9740 20 client.25148 trim_cache size 0 max 16384 2024-04-25T13:41:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f694e7c9740 1 Processor -- start 2024-04-25T13:41:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f694e7c9740 1 -- start start 2024-04-25T13:41:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c654d0 0x564a1403d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a14045cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 0x564a1403d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f694e7c9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69401a6f10 con 0x564a13c654d0 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f694e7c9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69401a9790 con 0x564a13c6d6b0 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f694e7c9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f69401a7580 con 0x564a13c8e390 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f69456b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c654d0 0x564a1403d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f69456b0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c654d0 0x564a1403d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51842/0 (socket says 172.21.15.27:51842) 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f6944eaf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a14045cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f69456b0640 1 -- 172.21.15.27:0/2302485041 learned_addr learned my addr 172.21.15.27:0/2302485041 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.546+0000 7f6945eb1640 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 0x564a1403d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f6944eaf640 1 -- 172.21.15.27:0/2302485041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 msgr2=0x564a1403d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f6944eaf640 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 0x564a1403d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f6944eaf640 1 -- 172.21.15.27:0/2302485041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c654d0 msgr2=0x564a1403d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f6944eaf640 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c654d0 0x564a1403d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f6944eaf640 1 -- 172.21.15.27:0/2302485041 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a1404a420 con 0x564a13c6d6b0 2024-04-25T13:41:26.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f69456b0640 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c654d0 0x564a1403d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f6944eaf640 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a14045cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f693c0a7410 tx=0x7f693c0a7cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=870b77b467944dbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f692affd640 1 -- 172.21.15.27:0/2302485041 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0a3340 con 0x564a13c6d6b0 2024-04-25T13:41:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.547+0000 7f692affd640 1 -- 172.21.15.27:0/2302485041 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f693c0ab830 con 0x564a13c6d6b0 2024-04-25T13:41:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.548+0000 7f692affd640 1 -- 172.21.15.27:0/2302485041 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0a3020 con 0x564a13c6d6b0 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 -- 172.21.15.27:0/2302485041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 msgr2=0x564a14045cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a14045cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f693c0a7410 tx=0x7f693c0a7cc0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 -- 172.21.15.27:0/2302485041 shutdown_connections 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c8e390 0x564a1403d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c6d6b0 0x564a14045cc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 --2- 172.21.15.27:0/2302485041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c654d0 0x564a1403d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 -- 172.21.15.27:0/2302485041 >> 172.21.15.27:0/2302485041 conn(0x564a13c824e0 msgr2=0x564a14039600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 -- 172.21.15.27:0/2302485041 shutdown_connections 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 -- 172.21.15.27:0/2302485041 wait complete. 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 -- start start 2024-04-25T13:41:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.549+0000 7f694e7c9740 1 Processor -- start 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a14042bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c6d6b0 0x564a14043100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f694e7c9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a140b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f694e7c9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69401a7c50 con 0x564a13c8e390 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f694e7c9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69401a90c0 con 0x564a13c654d0 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f694e7c9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f69401a8320 con 0x564a13c6d6b0 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f694e7c9740 10 client.0 _ll_register_callbacks cb 0x7f693c005680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f69456b0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a14042bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f69456b0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a14042bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43310/0 (socket says 172.21.15.27:43310) 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f69456b0640 1 -- 172.21.15.27:0/600891174 learned_addr learned my addr 172.21.15.27:0/600891174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f6944eaf640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c6d6b0 0x564a14043100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.550+0000 7f6945eb1640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a140b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f69456b0640 1 -- 172.21.15.27:0/600891174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c6d6b0 msgr2=0x564a14043100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f69456b0640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c6d6b0 0x564a14043100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f69456b0640 1 -- 172.21.15.27:0/600891174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 msgr2=0x564a140b29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f69456b0640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a140b29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f69456b0640 1 -- 172.21.15.27:0/600891174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a140460c0 con 0x564a13c654d0 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f6945eb1640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a140b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f6944eaf640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c6d6b0 0x564a14043100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f69456b0640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a14042bc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x564a14034940 tx=0x7f693002a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e17c96446eebf515 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f6928ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.551+0000 7f6928ff9640 1 -- 172.21.15.27:0/600891174 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930056070 con 0x564a13c654d0 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f6928ff9640 1 -- 172.21.15.27:0/600891174 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6930003a60 con 0x564a13c654d0 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 1 -- 172.21.15.27:0/600891174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f693c0ab830 con 0x564a13c654d0 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f6928ff9640 1 -- 172.21.15.27:0/600891174 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6930035030 con 0x564a13c654d0 2024-04-25T13:41:26.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 10 client.25154 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 1 -- 172.21.15.27:0/600891174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564a140b3ef0 con 0x564a13c654d0 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f6928ff9640 1 -- 172.21.15.27:0/600891174 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f693c0ab830 con 0x564a13c654d0 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f6928ff9640 1 -- 172.21.15.27:0/600891174 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6930059020 con 0x564a13c654d0 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 10 client.25154 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 10 client.25154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 20 client.25154 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a13c223a0 0x564a140b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 4 client.25154 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.552+0000 7f694e7c9740 1 -- 172.21.15.27:0/600891174 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564a13f9c800 con 0x564a13c223a0 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f6944eaf640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a13c223a0 0x564a140b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f6928ff9640 10 client.25154 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:26.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f6944eaf640 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a13c223a0 0x564a140b6580 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7ba7d7eea22fcf3 server_cookie=16c6e359dbd51f08 in_seq=0 out_seq=0 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f6928ff9640 1 -- 172.21.15.27:0/600891174 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564a13f9c800 con 0x564a13c223a0 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f6928ff9640 10 client.25154 handle_command_reply: tid=0 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 client.25154 shutdown 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 -- 172.21.15.27:0/600891174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 msgr2=0x564a14042bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a14042bc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x564a14034940 tx=0x7f693002a410 comp rx=0 tx=0).stop 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 -- 172.21.15.27:0/600891174 shutdown_connections 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a13c223a0 0x564a140b6580 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a13c8e390 0x564a140b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a13c6d6b0 0x564a14043100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f6928ff9640 0 client.25154 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 --2- 172.21.15.27:0/600891174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a13c654d0 0x564a14042bc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 -- 172.21.15.27:0/600891174 >> 172.21.15.27:0/600891174 conn(0x564a13c824e0 msgr2=0x564a1403d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 -- 172.21.15.27:0/600891174 shutdown_connections 2024-04-25T13:41:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.553+0000 7f694e7c9740 1 -- 172.21.15.27:0/600891174 wait complete. 2024-04-25T13:41:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.554+0000 7f694e7c9740 20 client.25154 trim_cache size 0 max 16384 2024-04-25T13:41:26.559 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:41:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 -- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f692c0afb70 msgr2=0x7f692c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f692c0afb70 0x7f692c0b2080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x564a14035c70 tx=0x7f6938009500 comp rx=0 tx=0).stop 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 -- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940106c40 msgr2=0x7f694019aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940106c40 0x7f694019aad0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f693c00e9f0 tx=0x7f693c00edf0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 -- 172.21.15.27:0/704464195 shutdown_connections 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f692c0afb70 0x7f692c0b2080 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6940109770 0x7f69401a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6940106c40 0x7f694019aad0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 --2- 172.21.15.27:0/704464195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6940104250 0x7f694019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 -- 172.21.15.27:0/704464195 >> 172.21.15.27:0/704464195 conn(0x7f69400fdbd0 msgr2=0x7f6940105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.556+0000 7f6947914640 1 -- 172.21.15.27:0/704464195 shutdown_connections 2024-04-25T13:41:26.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:26.557+0000 7f6947914640 1 -- 172.21.15.27:0/704464195 wait complete. 2024-04-25T13:41:27.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:26 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/680109694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:41:27.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:26 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/680109694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:41:27.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:26 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/680109694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:41:27.571 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:41:27.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:41:27.571 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:41:27.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 -- 172.21.15.27:0/1940882560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674104d20 msgr2=0x7f6674105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 --2- 172.21.15.27:0/1940882560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674104d20 0x7f6674105130 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6664009900 tx=0x7f666402a410 comp rx=0 tx=0).stop 2024-04-25T13:41:27.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 -- 172.21.15.27:0/1940882560 shutdown_connections 2024-04-25T13:41:27.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 --2- 172.21.15.27:0/1940882560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6674109a20 0x7f667410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 --2- 172.21.15.27:0/1940882560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674104d20 0x7f6674105130 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 --2- 172.21.15.27:0/1940882560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6674103ae0 0x7f66741047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 -- 172.21.15.27:0/1940882560 >> 172.21.15.27:0/1940882560 conn(0x7f66740fecd0 msgr2=0x7f6674101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:27.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 -- 172.21.15.27:0/1940882560 shutdown_connections 2024-04-25T13:41:27.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.747+0000 7f6679da8640 1 -- 172.21.15.27:0/1940882560 wait complete. 2024-04-25T13:41:27.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.748+0000 7f6679da8640 1 Processor -- start 2024-04-25T13:41:27.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6679da8640 1 -- start start 2024-04-25T13:41:27.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6679da8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674103ae0 0x7f6674198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6679da8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6674104d20 0x7f6674198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6679da8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6674109a20 0x7f66741950c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6679da8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f667410f660 con 0x7f6674103ae0 2024-04-25T13:41:27.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6679da8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f667410f4e0 con 0x7f6674109a20 2024-04-25T13:41:27.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6679da8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f667410f7e0 con 0x7f6674104d20 2024-04-25T13:41:27.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6673fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6674109a20 0x7f66741950c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f66737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674103ae0 0x7f6674198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6673fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6674109a20 0x7f66741950c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43316/0 (socket says 172.21.15.27:43316) 2024-04-25T13:41:27.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f6673fff640 1 -- 172.21.15.27:0/1149908115 learned_addr learned my addr 172.21.15.27:0/1149908115 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:27.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.749+0000 7f66737fe640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674103ae0 0x7f6674198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51860/0 (socket says 172.21.15.27:51860) 2024-04-25T13:41:27.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.750+0000 7f6673fff640 1 -- 172.21.15.27:0/1149908115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6674104d20 msgr2=0x7f6674198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.750+0000 7f6672ffd640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6674104d20 0x7f6674198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.750+0000 7f6673fff640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6674104d20 0x7f6674198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.750+0000 7f6673fff640 1 -- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674103ae0 msgr2=0x7f6674198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.750+0000 7f6673fff640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674103ae0 0x7f6674198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.750+0000 7f6673fff640 1 -- 172.21.15.27:0/1149908115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66741958e0 con 0x7f6674109a20 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.750+0000 7f66737fe640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674103ae0 0x7f6674198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.750+0000 7f6672ffd640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6674104d20 0x7f6674198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.751+0000 7f6673fff640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6674109a20 0x7f66741950c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f666800e970 tx=0x7f666800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.751+0000 7f6670ff9640 1 -- 172.21.15.27:0/1149908115 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666800ccc0 con 0x7f6674109a20 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.751+0000 7f6679da8640 1 -- 172.21.15.27:0/1149908115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6674195bd0 con 0x7f6674109a20 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.751+0000 7f6670ff9640 1 -- 172.21.15.27:0/1149908115 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f666800ce60 con 0x7f6674109a20 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.751+0000 7f6679da8640 1 -- 172.21.15.27:0/1149908115 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66741a8c00 con 0x7f6674109a20 2024-04-25T13:41:27.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.751+0000 7f6670ff9640 1 -- 172.21.15.27:0/1149908115 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668010430 con 0x7f6674109a20 2024-04-25T13:41:27.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6680c59740 1 Processor -- start 2024-04-25T13:41:27.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6670ff9640 1 -- 172.21.15.27:0/1149908115 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f66680105d0 con 0x7f6674109a20 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6680c59740 1 -- start start 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6680c59740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed56410 0x55b68f0f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6680c59740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f0f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6680c59740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6680c59740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b68f05cb00 con 0x55b68ed35730 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6680c59740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b68eff8920 con 0x55b68ed56410 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.754+0000 7f6680c59740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b68f058ac0 con 0x55b68ed2d550 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed56410 0x55b68f0f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f66737fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f0f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed56410 0x55b68f0f74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43326/0 (socket says 172.21.15.27:43326) 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6672ffd640 1 -- 172.21.15.27:0/1281565897 learned_addr learned my addr 172.21.15.27:0/1281565897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6672ffd640 1 --2- 172.21.15.27:0/1281565897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6670ff9640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f665c0afaf0 0x7f665c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f66737fe640 1 -- 172.21.15.27:0/1281565897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed56410 msgr2=0x55b68f0f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f66737fe640 1 --2- 172.21.15.27:0/1281565897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed56410 0x55b68f0f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f66737fe640 1 -- 172.21.15.27:0/1281565897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 msgr2=0x55b68f0fe920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f66737fe640 1 --2- 172.21.15.27:0/1281565897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f0fe920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f66737fe640 1 -- 172.21.15.27:0/1281565897 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b68ee367e0 con 0x55b68ed2d550 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6673fff640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f665c0afaf0 0x7f665c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f66737fe640 1 --2- 172.21.15.27:0/1281565897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f0f7a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66741046e0 tx=0x7f666c011c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103fd966deb4df18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6670ff9640 1 -- 172.21.15.27:0/1149908115 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f66680df090 con 0x7f6674109a20 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6659ffb640 1 -- 172.21.15.27:0/1281565897 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c003c30 con 0x55b68ed2d550 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.755+0000 7f6659ffb640 1 -- 172.21.15.27:0/1281565897 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f666c003dd0 con 0x55b68ed2d550 2024-04-25T13:41:27.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.756+0000 7f6659ffb640 1 -- 172.21.15.27:0/1281565897 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c0047e0 con 0x55b68ed2d550 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.757+0000 7f6680c59740 1 -- 172.21.15.27:0/1281565897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 msgr2=0x55b68f0f7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.757+0000 7f6680c59740 1 --2- 172.21.15.27:0/1281565897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f0f7a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66741046e0 tx=0x7f666c011c70 comp rx=0 tx=0).stop 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.757+0000 7f6673fff640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f665c0afaf0 0x7f665c0b2000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f66680a1c60 tx=0x7f66680d5cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.757+0000 7f6680c59740 1 -- 172.21.15.27:0/1281565897 shutdown_connections 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.757+0000 7f6680c59740 1 --2- 172.21.15.27:0/1281565897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f0fe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.757+0000 7f6680c59740 1 --2- 172.21.15.27:0/1281565897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f0f7a20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.757+0000 7f6680c59740 1 --2- 172.21.15.27:0/1281565897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed56410 0x55b68f0f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.757+0000 7f6680c59740 1 -- 172.21.15.27:0/1281565897 >> 172.21.15.27:0/1281565897 conn(0x55b68ed4a560 msgr2=0x55b68f0f2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.758+0000 7f6680c59740 1 -- 172.21.15.27:0/1281565897 shutdown_connections 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.758+0000 7f6680c59740 1 -- 172.21.15.27:0/1281565897 wait complete. 2024-04-25T13:41:27.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.758+0000 7f6680c59740 1 -- start start 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6680c59740 1 Processor -- start 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6680c59740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f101290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6680c59740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed35730 0x55b68f1017d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6680c59740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 0x55b68f108730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6680c59740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b68f0ff370 con 0x55b68ed56410 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6680c59740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b68eff65d0 con 0x55b68ed35730 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6680c59740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b68f0ff4f0 con 0x55b68ed2d550 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6680c59740 10 client.0 _ll_register_callbacks cb 0x55b68efe4080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f66737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed35730 0x55b68f1017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f66737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed35730 0x55b68f1017d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43340/0 (socket says 172.21.15.27:43340) 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f66737fe640 1 -- 172.21.15.27:0/3253072760 learned_addr learned my addr 172.21.15.27:0/3253072760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f6672ffd640 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 0x55b68f108730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f66737fe640 1 -- 172.21.15.27:0/3253072760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 msgr2=0x55b68f101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f66737fe640 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f66737fe640 1 -- 172.21.15.27:0/3253072760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 msgr2=0x55b68f108730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f66737fe640 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 0x55b68f108730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.759+0000 7f66737fe640 1 -- 172.21.15.27:0/3253072760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b68ee96f70 con 0x55b68ed35730 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.760+0000 7f66737fe640 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed35730 0x55b68f1017d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f666c00a010 tx=0x7f666c004160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=679bd1bf2f6110bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.760+0000 7f6652ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.760+0000 7f6652ffd640 1 -- 172.21.15.27:0/3253072760 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c027020 con 0x55b68ed35730 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.760+0000 7f6680c59740 1 -- 172.21.15.27:0/3253072760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f666c003dd0 con 0x55b68ed35730 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.760+0000 7f6652ffd640 1 -- 172.21.15.27:0/3253072760 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f666c0047b0 con 0x55b68ed35730 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.760+0000 7f6652ffd640 1 -- 172.21.15.27:0/3253072760 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f666c011050 con 0x55b68ed35730 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.760+0000 7f6680c59740 10 client.25166 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:27.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.760+0000 7f6680c59740 1 -- 172.21.15.27:0/3253072760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b68ed2e170 con 0x55b68ed35730 2024-04-25T13:41:27.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6652ffd640 1 -- 172.21.15.27:0/3253072760 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f666c003dd0 con 0x55b68ed35730 2024-04-25T13:41:27.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6652ffd640 1 -- 172.21.15.27:0/3253072760 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f666c018070 con 0x55b68ed35730 2024-04-25T13:41:27.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6680c59740 10 client.25166 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:27.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6680c59740 10 client.25166 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:27.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6680c59740 20 client.25166 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:27.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6680c59740 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b68ed151f0 0x55b68f10e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6680c59740 4 client.25166 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6680c59740 1 -- 172.21.15.27:0/3253072760 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b68f06a710 con 0x55b68ed151f0 2024-04-25T13:41:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.761+0000 7f6672ffd640 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b68ed151f0 0x55b68f10e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.762+0000 7f6652ffd640 10 client.25166 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.762+0000 7f6672ffd640 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b68ed151f0 0x55b68f10e9a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc3aa7910cb704e4 server_cookie=97fa2702f048374d in_seq=0 out_seq=0 2024-04-25T13:41:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.763+0000 7f6652ffd640 1 -- 172.21.15.27:0/3253072760 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b68f06a710 con 0x55b68ed151f0 2024-04-25T13:41:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.763+0000 7f6652ffd640 10 client.25166 handle_command_reply: tid=0 2024-04-25T13:41:27.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.763+0000 7f6680c59740 1 client.25166 shutdown 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.764+0000 7f6680c59740 1 -- 172.21.15.27:0/3253072760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed35730 msgr2=0x55b68f1017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.764+0000 7f6680c59740 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed35730 0x55b68f1017d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f666c00a010 tx=0x7f666c004160 comp rx=0 tx=0).stop 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.764+0000 7f6680c59740 1 -- 172.21.15.27:0/3253072760 shutdown_connections 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.764+0000 7f6680c59740 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b68ed151f0 0x55b68f10e9a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.764+0000 7f6652ffd640 0 client.25166 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.764+0000 7f6680c59740 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 0x55b68f108730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.765+0000 7f6680c59740 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed35730 0x55b68f1017d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.765+0000 7f6680c59740 1 --2- 172.21.15.27:0/3253072760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed2d550 0x55b68f101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.765+0000 7f6680c59740 1 -- 172.21.15.27:0/3253072760 >> 172.21.15.27:0/3253072760 conn(0x55b68ed4a560 msgr2=0x55b68f0f1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:27.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.765+0000 7f6680c59740 1 -- 172.21.15.27:0/3253072760 shutdown_connections 2024-04-25T13:41:27.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.765+0000 7f6680c59740 1 -- 172.21.15.27:0/3253072760 wait complete. 2024-04-25T13:41:27.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.766+0000 7f6680c59740 20 client.25166 trim_cache size 0 max 16384 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.792+0000 7f6680c59740 1 Processor -- start 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.792+0000 7f6680c59740 1 -- start start 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6680c59740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f0fffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6680c59740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f113150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6680c59740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed56410 0x55b68f113690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6680c59740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f66741af900 con 0x55b68ed35730 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6680c59740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f66741b2180 con 0x55b68ed2d550 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6680c59740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f66741aff70 con 0x55b68ed56410 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f66737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f113150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f0fffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f0fffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43352/0 (socket says 172.21.15.27:43352) 2024-04-25T13:41:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f66737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f113150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51892/0 (socket says 172.21.15.27:51892) 2024-04-25T13:41:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 -- 172.21.15.27:0/1357268633 learned_addr learned my addr 172.21.15.27:0/1357268633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 --2- 172.21.15.27:0/1357268633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed56410 0x55b68f113690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 -- 172.21.15.27:0/1357268633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed56410 msgr2=0x55b68f113690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:27.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 --2- 172.21.15.27:0/1357268633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed56410 0x55b68f113690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 -- 172.21.15.27:0/1357268633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 msgr2=0x55b68f113150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 --2- 172.21.15.27:0/1357268633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f113150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.793+0000 7f6672ffd640 1 -- 172.21.15.27:0/1357268633 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b68f0fc110 con 0x55b68ed2d550 2024-04-25T13:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.794+0000 7f6672ffd640 1 --2- 172.21.15.27:0/1357268633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f0fffd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f666402a9d0 tx=0x7f6664009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c18b12be9cce3d95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.794+0000 7f66597fa640 1 -- 172.21.15.27:0/1357268633 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664035040 con 0x55b68ed2d550 2024-04-25T13:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.794+0000 7f66597fa640 1 -- 172.21.15.27:0/1357268633 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66640054d0 con 0x55b68ed2d550 2024-04-25T13:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.794+0000 7f66597fa640 1 -- 172.21.15.27:0/1357268633 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666404d360 con 0x55b68ed2d550 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 -- 172.21.15.27:0/1357268633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 msgr2=0x55b68f0fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 --2- 172.21.15.27:0/1357268633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f0fffd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f666402a9d0 tx=0x7f6664009260 comp rx=0 tx=0).stop 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 -- 172.21.15.27:0/1357268633 shutdown_connections 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 --2- 172.21.15.27:0/1357268633 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed56410 0x55b68f113690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 --2- 172.21.15.27:0/1357268633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed35730 0x55b68f113150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 --2- 172.21.15.27:0/1357268633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f0fffd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 -- 172.21.15.27:0/1357268633 >> 172.21.15.27:0/1357268633 conn(0x55b68ed4a560 msgr2=0x55b68f0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 -- 172.21.15.27:0/1357268633 shutdown_connections 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.795+0000 7f6680c59740 1 -- 172.21.15.27:0/1357268633 wait complete. 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.796+0000 7f6680c59740 1 -- start start 2024-04-25T13:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.796+0000 7f6680c59740 1 Processor -- start 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.796+0000 7f6680c59740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f104ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.796+0000 7f6680c59740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed35730 0x55b68f105400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.796+0000 7f6680c59740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 0x55b68f1062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.796+0000 7f6680c59740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f66741b0640 con 0x55b68ed56410 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6680c59740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f66741b1ab0 con 0x55b68ed2d550 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6680c59740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f66741b0d10 con 0x55b68ed35730 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f104ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6680c59740 10 client.0 _ll_register_callbacks cb 0x7f666c011660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f104ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43366/0 (socket says 172.21.15.27:43366) 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 -- 172.21.15.27:0/401845928 learned_addr learned my addr 172.21.15.27:0/401845928 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 0x55b68f1062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f66737fe640 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed35730 0x55b68f105400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 -- 172.21.15.27:0/401845928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed35730 msgr2=0x55b68f105400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed35730 0x55b68f105400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 -- 172.21.15.27:0/401845928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 msgr2=0x55b68f1062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 0x55b68f1062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 -- 172.21.15.27:0/401845928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b68f106880 con 0x55b68ed2d550 2024-04-25T13:41:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f66737fe640 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed35730 0x55b68f105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:41:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6672ffd640 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f104ec0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f66640056e0 tx=0x7f6664009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=615162c747140e97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.797+0000 7f6652ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.798+0000 7f6652ffd640 1 -- 172.21.15.27:0/401845928 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664035070 con 0x55b68ed2d550 2024-04-25T13:41:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.798+0000 7f6652ffd640 1 -- 172.21.15.27:0/401845928 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f66640319b0 con 0x55b68ed2d550 2024-04-25T13:41:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.798+0000 7f6680c59740 1 -- 172.21.15.27:0/401845928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66640054d0 con 0x55b68ed2d550 2024-04-25T13:41:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.799+0000 7f6652ffd640 1 -- 172.21.15.27:0/401845928 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6664061020 con 0x55b68ed2d550 2024-04-25T13:41:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.799+0000 7f6680c59740 10 client.25172 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.799+0000 7f6680c59740 1 -- 172.21.15.27:0/401845928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b68f107b20 con 0x55b68ed2d550 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6652ffd640 1 -- 172.21.15.27:0/401845928 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6664031bb0 con 0x55b68ed2d550 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6652ffd640 1 -- 172.21.15.27:0/401845928 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f666404b070 con 0x55b68ed2d550 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6680c59740 10 client.25172 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6680c59740 10 client.25172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6680c59740 20 client.25172 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6680c59740 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b68ecea630 0x55b68f178ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6680c59740 4 client.25172 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6680c59740 1 -- 172.21.15.27:0/401845928 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f66640054d0 con 0x55b68ecea630 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f66737fe640 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b68ecea630 0x55b68f178ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.800+0000 7f6652ffd640 10 client.25172 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.801+0000 7f66737fe640 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b68ecea630 0x55b68f178ef0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d934872caa619d2 server_cookie=adbea4a0c75c774d in_seq=0 out_seq=0 2024-04-25T13:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.802+0000 7f6652ffd640 1 -- 172.21.15.27:0/401845928 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f66640054d0 con 0x55b68ecea630 2024-04-25T13:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.802+0000 7f6652ffd640 10 client.25172 handle_command_reply: tid=0 2024-04-25T13:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.802+0000 7f6680c59740 1 client.25172 shutdown 2024-04-25T13:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.802+0000 7f6680c59740 1 -- 172.21.15.27:0/401845928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 msgr2=0x55b68f104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.802+0000 7f6680c59740 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f104ec0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f66640056e0 tx=0x7f6664009880 comp rx=0 tx=0).stop 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.802+0000 7f6680c59740 1 -- 172.21.15.27:0/401845928 shutdown_connections 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.802+0000 7f6680c59740 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b68ecea630 0x55b68f178ef0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.803+0000 7f6652ffd640 0 client.25172 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.803+0000 7f6680c59740 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b68ed56410 0x55b68f1062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.803+0000 7f6680c59740 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b68ed35730 0x55b68f105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.803+0000 7f6680c59740 1 --2- 172.21.15.27:0/401845928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b68ed2d550 0x55b68f104ec0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.803+0000 7f6680c59740 1 -- 172.21.15.27:0/401845928 >> 172.21.15.27:0/401845928 conn(0x55b68ed4a560 msgr2=0x55b68f0f5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.803+0000 7f6680c59740 1 -- 172.21.15.27:0/401845928 shutdown_connections 2024-04-25T13:41:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.803+0000 7f6680c59740 1 -- 172.21.15.27:0/401845928 wait complete. 2024-04-25T13:41:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.804+0000 7f6680c59740 20 client.25172 trim_cache size 0 max 16384 2024-04-25T13:41:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:41:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:41:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "15e55c1f-7fe5-43bf-b8c6-b26734762fe4", 2024-04-25T13:41:27.804 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:41:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:41:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.805+0000 7f6679da8640 1 -- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f665c0afaf0 msgr2=0x7f665c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.805+0000 7f6679da8640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f665c0afaf0 0x7f665c0b2000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f66680a1c60 tx=0x7f66680d5cf0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.806+0000 7f6679da8640 1 -- 172.21.15.27:0/1149908115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6674109a20 msgr2=0x7f66741950c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.806+0000 7f6679da8640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6674109a20 0x7f66741950c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f666800e970 tx=0x7f666800ed70 comp rx=0 tx=0).stop 2024-04-25T13:41:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.807+0000 7f6679da8640 1 -- 172.21.15.27:0/1149908115 shutdown_connections 2024-04-25T13:41:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.807+0000 7f6679da8640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f665c0afaf0 0x7f665c0b2000 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.807+0000 7f6679da8640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6674109a20 0x7f66741950c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.807+0000 7f6679da8640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6674104d20 0x7f6674198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.807+0000 7f6679da8640 1 --2- 172.21.15.27:0/1149908115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6674103ae0 0x7f6674198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.807+0000 7f6679da8640 1 -- 172.21.15.27:0/1149908115 >> 172.21.15.27:0/1149908115 conn(0x7f66740fecd0 msgr2=0x7f66740ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.807+0000 7f6679da8640 1 -- 172.21.15.27:0/1149908115 shutdown_connections 2024-04-25T13:41:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:27.807+0000 7f6679da8640 1 -- 172.21.15.27:0/1149908115 wait complete. 2024-04-25T13:41:27.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:41:27.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:41:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 20.0 s, 3678.4 tps, lat 27.203 ms stddev 28.769 2024-04-25T13:41:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 -- 172.21.15.27:0/2301772278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 msgr2=0x7f2f1c1054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 --2- 172.21.15.27:0/2301772278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 0x7f2f1c1054b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009900 tx=0x7f2f1002a410 comp rx=0 tx=0).stop 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 -- 172.21.15.27:0/2301772278 shutdown_connections 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 --2- 172.21.15.27:0/2301772278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c102250 0x7f2f1c10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 --2- 172.21.15.27:0/2301772278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c101750 0x7f2f1c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 --2- 172.21.15.27:0/2301772278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 0x7f2f1c1054b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 -- 172.21.15.27:0/2301772278 >> 172.21.15.27:0/2301772278 conn(0x7f2f1c0fd350 msgr2=0x7f2f1c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 -- 172.21.15.27:0/2301772278 shutdown_connections 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.012+0000 7f2f21249640 1 -- 172.21.15.27:0/2301772278 wait complete. 2024-04-25T13:41:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f21249640 1 Processor -- start 2024-04-25T13:41:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f21249640 1 -- start start 2024-04-25T13:41:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f21249640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c101750 0x7f2f1c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f21249640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c102250 0x7f2f1c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f21249640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 0x7f2f1c1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f21249640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c10f5f0 con 0x7f2f1c1050a0 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f21249640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c10f470 con 0x7f2f1c102250 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f21249640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c10f770 con 0x7f2f1c101750 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f1b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 0x7f2f1c1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f1a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c102250 0x7f2f1c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.013+0000 7f2f1b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 0x7f2f1c1a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51912/0 (socket says 172.21.15.27:51912) 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1b59e640 1 -- 172.21.15.27:0/3232762084 learned_addr learned my addr 172.21.15.27:0/3232762084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c101750 0x7f2f1c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1a59c640 1 -- 172.21.15.27:0/3232762084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c101750 msgr2=0x7f2f1c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c101750 0x7f2f1c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1a59c640 1 -- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 msgr2=0x7f2f1c1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 0x7f2f1c1a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1a59c640 1 -- 172.21.15.27:0/3232762084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c1a1d90 con 0x7f2f1c102250 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 0x7f2f1c1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.014+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c102250 0x7f2f1c19a790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00e9f0 tx=0x7f2f0c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.015+0000 7f2f03fff640 1 -- 172.21.15.27:0/3232762084 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c00cd80 con 0x7f2f1c102250 2024-04-25T13:41:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.015+0000 7f2f03fff640 1 -- 172.21.15.27:0/3232762084 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f0c0035f0 con 0x7f2f1c102250 2024-04-25T13:41:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.015+0000 7f2f03fff640 1 -- 172.21.15.27:0/3232762084 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c010670 con 0x7f2f1c102250 2024-04-25T13:41:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.015+0000 7f2f21249640 1 -- 172.21.15.27:0/3232762084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1c1a2080 con 0x7f2f1c102250 2024-04-25T13:41:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.015+0000 7f2f21249640 1 -- 172.21.15.27:0/3232762084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1c1a25d0 con 0x7f2f1c102250 2024-04-25T13:41:28.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.019+0000 7f2f03fff640 1 -- 172.21.15.27:0/3232762084 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2f0c0037a0 con 0x7f2f1c102250 2024-04-25T13:41:28.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.019+0000 7f2f281d4740 1 Processor -- start 2024-04-25T13:41:28.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.019+0000 7f2f281d4740 1 -- start start 2024-04-25T13:41:28.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c4d390 0x562087ff9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c244d0 0x562087ffbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c2c6b0 0x562087ffe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f281d4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562087f53d80 con 0x562087c4d390 2024-04-25T13:41:28.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f281d4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562087f0d8b0 con 0x562087c244d0 2024-04-25T13:41:28.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f281d4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562087f4fd40 con 0x562087c2c6b0 2024-04-25T13:41:28.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f03fff640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2f040afc40 0x7f2f040b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c4d390 0x562087ff9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c2c6b0 0x562087ffe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c4d390 0x562087ff9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51916/0 (socket says 172.21.15.27:51916) 2024-04-25T13:41:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 -- 172.21.15.27:0/2555662187 learned_addr learned my addr 172.21.15.27:0/2555662187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2f040afc40 0x7f2f040b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c244d0 0x562087ffbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f03fff640 1 -- 172.21.15.27:0/3232762084 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2f0c0ded90 con 0x7f2f1c102250 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 -- 172.21.15.27:0/2555662187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c2c6b0 msgr2=0x562087ffe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c2c6b0 0x562087ffe690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 -- 172.21.15.27:0/2555662187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c244d0 msgr2=0x562087ffbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c244d0 0x562087ffbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 -- 172.21.15.27:0/2555662187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562087edea40 con 0x562087c4d390 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c2c6b0 0x562087ffe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.020+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c4d390 0x562087ff9930 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2f140115b0 tx=0x7f2f14011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ee2883a99450e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.021+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2f040afc40 0x7f2f040b2150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2f140094f0 tx=0x7f2f14018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.021+0000 7f2f017fa640 1 -- 172.21.15.27:0/2555662187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f14025070 con 0x562087c4d390 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.021+0000 7f2f017fa640 1 -- 172.21.15.27:0/2555662187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f140135d0 con 0x562087c4d390 2024-04-25T13:41:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.022+0000 7f2f017fa640 1 -- 172.21.15.27:0/2555662187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f14021430 con 0x562087c4d390 2024-04-25T13:41:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.022+0000 7f2f281d4740 1 -- 172.21.15.27:0/2555662187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c4d390 msgr2=0x562087ff9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.022+0000 7f2f281d4740 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c4d390 0x562087ff9930 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2f140115b0 tx=0x7f2f14011a70 comp rx=0 tx=0).stop 2024-04-25T13:41:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.022+0000 7f2f281d4740 1 -- 172.21.15.27:0/2555662187 shutdown_connections 2024-04-25T13:41:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.022+0000 7f2f281d4740 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c2c6b0 0x562087ffe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.022+0000 7f2f281d4740 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c244d0 0x562087ffbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.022+0000 7f2f281d4740 1 --2- 172.21.15.27:0/2555662187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c4d390 0x562087ff9930 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.023+0000 7f2f281d4740 1 -- 172.21.15.27:0/2555662187 >> 172.21.15.27:0/2555662187 conn(0x562087c414e0 msgr2=0x562087fea1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.023+0000 7f2f281d4740 1 -- 172.21.15.27:0/2555662187 shutdown_connections 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.023+0000 7f2f281d4740 1 -- 172.21.15.27:0/2555662187 wait complete. 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.023+0000 7f2f281d4740 1 -- start start 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.023+0000 7f2f281d4740 1 Processor -- start 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088000f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088001440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c4d390 0x562088008330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088000f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f281d4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562087fff1f0 con 0x562087c244d0 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f281d4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562087edad10 con 0x562087c2c6b0 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f281d4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562087fff370 con 0x562087c4d390 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088000f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51918/0 (socket says 172.21.15.27:51918) 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 -- 172.21.15.27:0/4119173676 learned_addr learned my addr 172.21.15.27:0/4119173676 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088001440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f281d4740 10 client.0 _ll_register_callbacks cb 0x562087f07210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c4d390 0x562088008330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 -- 172.21.15.27:0/4119173676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c4d390 msgr2=0x562088008330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c4d390 0x562088008330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 -- 172.21.15.27:0/4119173676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 msgr2=0x562088001440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088001440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 -- 172.21.15.27:0/4119173676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562087d2f7d0 con 0x562087c244d0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088000f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2f10005ab0 tx=0x7f2f100042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4b8380c6201b8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.024+0000 7f2efeff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.025+0000 7f2efeff5640 1 -- 172.21.15.27:0/4119173676 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f10035070 con 0x562087c244d0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.025+0000 7f2efeff5640 1 -- 172.21.15.27:0/4119173676 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2f100316d0 con 0x562087c244d0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.025+0000 7f2f281d4740 1 -- 172.21.15.27:0/4119173676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562087f61990 con 0x562087c244d0 2024-04-25T13:41:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.025+0000 7f2efeff5640 1 -- 172.21.15.27:0/4119173676 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2f100318d0 con 0x562087c244d0 2024-04-25T13:41:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.025+0000 7f2f281d4740 10 client.15441 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:28.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.025+0000 7f2f281d4740 1 -- 172.21.15.27:0/4119173676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562087c653f0 con 0x562087c244d0 2024-04-25T13:41:28.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.026+0000 7f2efeff5640 1 -- 172.21.15.27:0/4119173676 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f100035f0 con 0x562087c244d0 2024-04-25T13:41:28.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.026+0000 7f2efeff5640 1 -- 172.21.15.27:0/4119173676 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2f10046460 con 0x562087c244d0 2024-04-25T13:41:28.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.026+0000 7f2f281d4740 10 client.15441 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:28.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.026+0000 7f2f281d4740 10 client.15441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.026+0000 7f2f281d4740 20 client.15441 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.026+0000 7f2f281d4740 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562087c0bf60 0x56208800e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.026+0000 7f2f281d4740 4 client.15441 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.026+0000 7f2f281d4740 1 -- 172.21.15.27:0/4119173676 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562087f61990 con 0x562087c0bf60 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.027+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562087c0bf60 0x56208800e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.027+0000 7f2efeff5640 10 client.15441 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.027+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562087c0bf60 0x56208800e7e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78a4467cef901b20 server_cookie=217e21ad39b2ef5 in_seq=0 out_seq=0 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2efeff5640 1 -- 172.21.15.27:0/4119173676 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562087f61990 con 0x562087c0bf60 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2efeff5640 10 client.15441 handle_command_reply: tid=0 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 client.15441 shutdown 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 -- 172.21.15.27:0/4119173676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 msgr2=0x562088000f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088000f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2f10005ab0 tx=0x7f2f100042e0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 -- 172.21.15.27:0/4119173676 shutdown_connections 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562087c0bf60 0x56208800e7e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c4d390 0x562088008330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2efeff5640 0 client.15441 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088001440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 --2- 172.21.15.27:0/4119173676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088000f00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 -- 172.21.15.27:0/4119173676 >> 172.21.15.27:0/4119173676 conn(0x562087c414e0 msgr2=0x562087fe89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 -- 172.21.15.27:0/4119173676 shutdown_connections 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 1 -- 172.21.15.27:0/4119173676 wait complete. 2024-04-25T13:41:28.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.028+0000 7f2f281d4740 20 client.15441 trim_cache size 0 max 16384 2024-04-25T13:41:28.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.052+0000 7f2f281d4740 1 Processor -- start 2024-04-25T13:41:28.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.052+0000 7f2f281d4740 1 -- start start 2024-04-25T13:41:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c0bf60 0x562088000ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088001010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f1ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c0bf60 0x562088000ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f1a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088001010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f1ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c0bf60 0x562088000ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42156/0 (socket says 172.21.15.27:42156) 2024-04-25T13:41:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f1a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088001010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51932/0 (socket says 172.21.15.27:51932) 2024-04-25T13:41:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f1a59c640 1 -- 172.21.15.27:0/3406847155 learned_addr learned my addr 172.21.15.27:0/3406847155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f281d4740 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088012f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f281d4740 1 -- 172.21.15.27:0/3406847155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1a6ef0 con 0x562087c244d0 2024-04-25T13:41:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f281d4740 1 -- 172.21.15.27:0/3406847155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1a9770 con 0x562087c2c6b0 2024-04-25T13:41:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f281d4740 1 -- 172.21.15.27:0/3406847155 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1a7560 con 0x562087c0bf60 2024-04-25T13:41:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.053+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088012f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.054+0000 7f2f1a59c640 1 -- 172.21.15.27:0/3406847155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c0bf60 msgr2=0x562088000ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.054+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c0bf60 0x562088000ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.054+0000 7f2f1a59c640 1 -- 172.21.15.27:0/3406847155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 msgr2=0x562088012f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.054+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088012f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.054+0000 7f2f1a59c640 1 -- 172.21.15.27:0/3406847155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562087c250f0 con 0x562087c244d0 2024-04-25T13:41:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.054+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088012f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.054+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c0bf60 0x562088000ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:41:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.054+0000 7f2f1a59c640 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088001010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c0a3320 tx=0x7f2f0c074bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c70becbaa51a25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.056+0000 7f2f00ff9640 1 -- 172.21.15.27:0/3406847155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c0a6db0 con 0x562087c244d0 2024-04-25T13:41:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.056+0000 7f2f00ff9640 1 -- 172.21.15.27:0/3406847155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f0c0a79a0 con 0x562087c244d0 2024-04-25T13:41:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.056+0000 7f2f00ff9640 1 -- 172.21.15.27:0/3406847155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c009310 con 0x562087c244d0 2024-04-25T13:41:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 -- 172.21.15.27:0/3406847155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 msgr2=0x562088001010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088001010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c0a3320 tx=0x7f2f0c074bd0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 -- 172.21.15.27:0/3406847155 shutdown_connections 2024-04-25T13:41:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088012f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c244d0 0x562088001010 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 --2- 172.21.15.27:0/3406847155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c0bf60 0x562088000ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 -- 172.21.15.27:0/3406847155 >> 172.21.15.27:0/3406847155 conn(0x562087c414e0 msgr2=0x562087fffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 -- 172.21.15.27:0/3406847155 shutdown_connections 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 -- 172.21.15.27:0/3406847155 wait complete. 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 -- start start 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.057+0000 7f2f281d4740 1 Processor -- start 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c0bf60 0x562088002b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c244d0 0x562088003080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f281d4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088074b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f281d4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1a7c30 con 0x562087c0bf60 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f281d4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1a90a0 con 0x562087c2c6b0 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f281d4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c1a8300 con 0x562087c244d0 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c0bf60 0x562088002b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c244d0 0x562088003080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c0bf60 0x562088002b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51940/0 (socket says 172.21.15.27:51940) 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c244d0 0x562088003080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42162/0 (socket says 172.21.15.27:42162) 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1ad9d640 1 -- 172.21.15.27:0/1480422348 learned_addr learned my addr 172.21.15.27:0/1480422348 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f281d4740 10 client.0 _ll_register_callbacks cb 0x7f2f10031500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1b59e640 1 -- 172.21.15.27:0/1480422348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c244d0 msgr2=0x562088003080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c244d0 0x562088003080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1b59e640 1 -- 172.21.15.27:0/1480422348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c0bf60 msgr2=0x562088002b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c0bf60 0x562088002b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1b59e640 1 -- 172.21.15.27:0/1480422348 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562087f144b0 con 0x562087c2c6b0 2024-04-25T13:41:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.058+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c0bf60 0x562088002b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.059+0000 7f2f1b59e640 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088074b10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f14011f70 tx=0x7f2f14013790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24995bd60e53b6b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.059+0000 7f2efeff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.059+0000 7f2efeff5640 1 -- 172.21.15.27:0/1480422348 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1402f070 con 0x562087c2c6b0 2024-04-25T13:41:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.059+0000 7f2f281d4740 1 -- 172.21.15.27:0/1480422348 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f0c0a79a0 con 0x562087c2c6b0 2024-04-25T13:41:28.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.059+0000 7f2efeff5640 1 -- 172.21.15.27:0/1480422348 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2f14020ad0 con 0x562087c2c6b0 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2efeff5640 1 -- 172.21.15.27:0/1480422348 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2f1403b050 con 0x562087c2c6b0 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f281d4740 10 client.25190 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f281d4740 1 -- 172.21.15.27:0/1480422348 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562088076150 con 0x562087c2c6b0 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2efeff5640 1 -- 172.21.15.27:0/1480422348 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f0c0a79a0 con 0x562087c2c6b0 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2efeff5640 1 -- 172.21.15.27:0/1480422348 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2f1400f040 con 0x562087c2c6b0 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f281d4740 10 client.25190 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f281d4740 10 client.25190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f281d4740 20 client.25190 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f281d4740 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562087c4d390 0x562088078880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f281d4740 4 client.25190 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f281d4740 1 -- 172.21.15.27:0/1480422348 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562087f5b800 con 0x562087c4d390 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.060+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562087c4d390 0x562088078880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2efeff5640 10 client.25190 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f1ad9d640 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562087c4d390 0x562088078880 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=466f678ade311c61 server_cookie=550ab76d90f76bfb in_seq=0 out_seq=0 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2efeff5640 1 -- 172.21.15.27:0/1480422348 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562087f5b800 con 0x562087c4d390 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2efeff5640 10 client.25190 handle_command_reply: tid=0 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 client.25190 shutdown 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 -- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 msgr2=0x562088074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088074b10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f14011f70 tx=0x7f2f14013790 comp rx=0 tx=0).stop 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 -- 172.21.15.27:0/1480422348 shutdown_connections 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562087c4d390 0x562088078880 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562087c2c6b0 0x562088074b10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2efeff5640 0 client.25190 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562087c244d0 0x562088003080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 --2- 172.21.15.27:0/1480422348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562087c0bf60 0x562088002b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 -- 172.21.15.27:0/1480422348 >> 172.21.15.27:0/1480422348 conn(0x562087c414e0 msgr2=0x562087fffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 -- 172.21.15.27:0/1480422348 shutdown_connections 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 1 -- 172.21.15.27:0/1480422348 wait complete. 2024-04-25T13:41:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.061+0000 7f2f281d4740 20 client.25190 trim_cache size 0 max 16384 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: "15e55c1f-7fe5-43bf-b8c6-b26734762fe4": { 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "15e55c1f-7fe5-43bf-b8c6-b26734762fe4", 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:41:28.066 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:41:28.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 -- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2f040afc40 msgr2=0x7f2f040b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2f040afc40 0x7f2f040b2150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2f140094f0 tx=0x7f2f14018000 comp rx=0 tx=0).stop 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 -- 172.21.15.27:0/3232762084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c102250 msgr2=0x7f2f1c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c102250 0x7f2f1c19a790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00e9f0 tx=0x7f2f0c00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 -- 172.21.15.27:0/3232762084 shutdown_connections 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2f040afc40 0x7f2f040b2150 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1c1050a0 0x7f2f1c1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f1c102250 0x7f2f1c19a790 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 --2- 172.21.15.27:0/3232762084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c101750 0x7f2f1c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 -- 172.21.15.27:0/3232762084 >> 172.21.15.27:0/3232762084 conn(0x7f2f1c0fd350 msgr2=0x7f2f1c105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 -- 172.21.15.27:0/3232762084 shutdown_connections 2024-04-25T13:41:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:28.062+0000 7f2f21249640 1 -- 172.21.15.27:0/3232762084 wait complete. 2024-04-25T13:41:28.077 INFO:tasks.cephfs.filesystem:scrub status for tag:15e55c1f-7fe5-43bf-b8c6-b26734762fe4 - {'path': '/', 'tag': '15e55c1f-7fe5-43bf-b8c6-b26734762fe4', 'options': 'recursive,force'} 2024-04-25T13:41:28.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:27 smithi027 ceph-mon[29140]: pgmap v366: 97 pgs: 97 active+clean; 941 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 MiB/s wr, 898 op/s 2024-04-25T13:41:28.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:27 smithi122 ceph-mon[32843]: pgmap v366: 97 pgs: 97 active+clean; 941 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 MiB/s wr, 898 op/s 2024-04-25T13:41:28.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:27 smithi045 ceph-mon[32731]: pgmap v366: 97 pgs: 97 active+clean; 941 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 MiB/s wr, 898 op/s 2024-04-25T13:41:29.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:28 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:41:29.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:28 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:41:29.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:28 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:41:29.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:28 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:41:29.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:28 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:41:29.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:28 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:41:29.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:28 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:41:29.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:28 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:41:29.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:28 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:41:29.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:28 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:41:29.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:28 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:41:29.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:28 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:41:29.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:28 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:41:29.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:28 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:41:29.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:28 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:41:30.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:29 smithi122 ceph-mon[32843]: pgmap v367: 97 pgs: 97 active+clean; 950 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 761 op/s 2024-04-25T13:41:30.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:29 smithi045 ceph-mon[32731]: pgmap v367: 97 pgs: 97 active+clean; 950 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 761 op/s 2024-04-25T13:41:30.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:29 smithi027 ceph-mon[29140]: pgmap v367: 97 pgs: 97 active+clean; 950 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 761 op/s 2024-04-25T13:41:32.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:31 smithi122 ceph-mon[32843]: pgmap v368: 97 pgs: 97 active+clean; 951 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 862 op/s 2024-04-25T13:41:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:31 smithi045 ceph-mon[32731]: pgmap v368: 97 pgs: 97 active+clean; 951 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 862 op/s 2024-04-25T13:41:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:31 smithi027 ceph-mon[29140]: pgmap v368: 97 pgs: 97 active+clean; 951 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 862 op/s 2024-04-25T13:41:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 25.0 s, 3755.8 tps, lat 26.644 ms stddev 26.970 2024-04-25T13:41:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:33 smithi122 ceph-mon[32843]: pgmap v369: 97 pgs: 97 active+clean; 953 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 903 op/s 2024-04-25T13:41:34.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:33 smithi045 ceph-mon[32731]: pgmap v369: 97 pgs: 97 active+clean; 953 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 903 op/s 2024-04-25T13:41:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:33 smithi027 ceph-mon[29140]: pgmap v369: 97 pgs: 97 active+clean; 953 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 903 op/s 2024-04-25T13:41:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 -- 172.21.15.27:0/395293742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c106cb0 msgr2=0x7ff43c109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 --2- 172.21.15.27:0/395293742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c106cb0 0x7ff43c109150 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009900 tx=0x7ff42c02a410 comp rx=0 tx=0).stop 2024-04-25T13:41:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 -- 172.21.15.27:0/395293742 shutdown_connections 2024-04-25T13:41:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 --2- 172.21.15.27:0/395293742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff43c1097e0 0x7ff43c10bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 --2- 172.21.15.27:0/395293742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c106cb0 0x7ff43c109150 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 --2- 172.21.15.27:0/395293742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff43c1042c0 0x7ff43c1066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 -- 172.21.15.27:0/395293742 >> 172.21.15.27:0/395293742 conn(0x7ff43c0fdc60 msgr2=0x7ff43c1000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 -- 172.21.15.27:0/395293742 shutdown_connections 2024-04-25T13:41:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.968+0000 7ff443dc7640 1 -- 172.21.15.27:0/395293742 wait complete. 2024-04-25T13:41:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.969+0000 7ff443dc7640 1 Processor -- start 2024-04-25T13:41:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff443dc7640 1 -- start start 2024-04-25T13:41:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff443dc7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff43c1042c0 0x7ff43c103b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff443dc7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff43c106cb0 0x7ff43c1021d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff443dc7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c1097e0 0x7ff43c102710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff443dc7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c10f6c0 con 0x7ff43c1097e0 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff443dc7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c10f540 con 0x7ff43c1042c0 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff443dc7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c10f840 con 0x7ff43c106cb0 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff441b63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff43c1042c0 0x7ff43c103b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff441b63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff43c1042c0 0x7ff43c103b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38944/0 (socket says 172.21.15.27:38944) 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.970+0000 7ff441b63640 1 -- 172.21.15.27:0/3762302037 learned_addr learned my addr 172.21.15.27:0/3762302037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff441362640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff43c106cb0 0x7ff43c1021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff441b63640 1 -- 172.21.15.27:0/3762302037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff43c106cb0 msgr2=0x7ff43c1021d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff442364640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c1097e0 0x7ff43c102710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff441b63640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff43c106cb0 0x7ff43c1021d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff441b63640 1 -- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c1097e0 msgr2=0x7ff43c102710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff441b63640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c1097e0 0x7ff43c102710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff441b63640 1 -- 172.21.15.27:0/3762302037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c19e0a0 con 0x7ff43c1042c0 2024-04-25T13:41:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff441362640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff43c106cb0 0x7ff43c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:41:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff442364640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c1097e0 0x7ff43c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:41:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff441b63640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff43c1042c0 0x7ff43c103b30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff43800e9c0 tx=0x7ff43800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.971+0000 7ff432ffd640 1 -- 172.21.15.27:0/3762302037 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43800cd80 con 0x7ff43c1042c0 2024-04-25T13:41:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.972+0000 7ff432ffd640 1 -- 172.21.15.27:0/3762302037 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4380054d0 con 0x7ff43c1042c0 2024-04-25T13:41:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.972+0000 7ff443dc7640 1 -- 172.21.15.27:0/3762302037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43c102d70 con 0x7ff43c1042c0 2024-04-25T13:41:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.972+0000 7ff443dc7640 1 -- 172.21.15.27:0/3762302037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43c1a8ca0 con 0x7ff43c1042c0 2024-04-25T13:41:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.972+0000 7ff432ffd640 1 -- 172.21.15.27:0/3762302037 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438010430 con 0x7ff43c1042c0 2024-04-25T13:41:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.974+0000 7ff44ac73740 1 Processor -- start 2024-04-25T13:41:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.974+0000 7ff432ffd640 1 -- 172.21.15.27:0/3762302037 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff4380105d0 con 0x7ff43c1042c0 2024-04-25T13:41:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff44ac73740 1 -- start start 2024-04-25T13:41:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 0x560b9ff66970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff66ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fba54d0 0x560b9ff6ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff44ac73740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560b9fecc460 con 0x560b9fb9d2f0 2024-04-25T13:41:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff44ac73740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560b9fe86590 con 0x560b9fbc61b0 2024-04-25T13:41:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff44ac73740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x560b9fec8420 con 0x560b9fba54d0 2024-04-25T13:41:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff441362640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fba54d0 0x560b9ff6ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff442364640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 0x560b9ff66970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff441362640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fba54d0 0x560b9ff6ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40892/0 (socket says 172.21.15.27:40892) 2024-04-25T13:41:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.975+0000 7ff441362640 1 -- 172.21.15.27:0/1223076625 learned_addr learned my addr 172.21.15.27:0/1223076625 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.976+0000 7ff441362640 1 -- 172.21.15.27:0/1223076625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 msgr2=0x560b9ff66970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.976+0000 7ff441362640 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 0x560b9ff66970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.976+0000 7ff441362640 1 -- 172.21.15.27:0/1223076625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 msgr2=0x560b9ff66ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.976+0000 7ff432ffd640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4280afae0 0x7ff4280b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff442364640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4280afae0 0x7ff4280b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff432ffd640 1 -- 172.21.15.27:0/3762302037 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff4380df110 con 0x7ff43c1042c0 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff441b63640 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff66ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff441362640 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff66ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff441362640 1 -- 172.21.15.27:0/1223076625 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b9fe930d0 con 0x560b9fba54d0 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff441b63640 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff66ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff441362640 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fba54d0 0x560b9ff6ddf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff42c031500 tx=0x7ff42c004190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c7bd9857c1d8624 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff442364640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4280afae0 0x7ff4280b1ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x560b9ff67b40 tx=0x7ff434009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff427fff640 1 -- 172.21.15.27:0/1223076625 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c035030 con 0x560b9fba54d0 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff427fff640 1 -- 172.21.15.27:0/1223076625 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff42c0054f0 con 0x560b9fba54d0 2024-04-25T13:41:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.977+0000 7ff427fff640 1 -- 172.21.15.27:0/1223076625 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c003850 con 0x560b9fba54d0 2024-04-25T13:41:34.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.979+0000 7ff44ac73740 1 -- 172.21.15.27:0/1223076625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fba54d0 msgr2=0x560b9ff6ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.979+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fba54d0 0x560b9ff6ddf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff42c031500 tx=0x7ff42c004190 comp rx=0 tx=0).stop 2024-04-25T13:41:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.979+0000 7ff44ac73740 1 -- 172.21.15.27:0/1223076625 shutdown_connections 2024-04-25T13:41:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.979+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fba54d0 0x560b9ff6ddf0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff42c031500 tx=0x7ff42c004190 comp rx=0 tx=0).stop 2024-04-25T13:41:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.979+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff66ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.979+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1223076625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 0x560b9ff66970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.979+0000 7ff44ac73740 1 -- 172.21.15.27:0/1223076625 >> 172.21.15.27:0/1223076625 conn(0x560b9fbba300 msgr2=0x560b9ff62300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.980+0000 7ff44ac73740 1 -- 172.21.15.27:0/1223076625 shutdown_connections 2024-04-25T13:41:34.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.980+0000 7ff44ac73740 1 -- 172.21.15.27:0/1223076625 wait complete. 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.980+0000 7ff44ac73740 1 -- start start 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.980+0000 7ff44ac73740 1 Processor -- start 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb9d2f0 0x560b9ff70800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 0x560b9ff70d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb84d80 0x560b9ff77ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff44ac73740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560b9ff6e7e0 con 0x560b9fb84d80 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff44ac73740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560b9fe53b60 con 0x560b9fb9d2f0 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff441b63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb9d2f0 0x560b9ff70800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff442364640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb84d80 0x560b9ff77ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff441362640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 0x560b9ff70d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff441b63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb9d2f0 0x560b9ff70800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38958/0 (socket says 172.21.15.27:38958) 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff441b63640 1 -- 172.21.15.27:0/4226804020 learned_addr learned my addr 172.21.15.27:0/4226804020 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff44ac73740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x560b9ff6e960 con 0x560b9fbc61b0 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.981+0000 7ff44ac73740 10 client.0 _ll_register_callbacks cb 0x560b9fe782b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff441b63640 1 -- 172.21.15.27:0/4226804020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 msgr2=0x560b9ff70d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff441b63640 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 0x560b9ff70d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff441b63640 1 -- 172.21.15.27:0/4226804020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb84d80 msgr2=0x560b9ff77ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff441b63640 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb84d80 0x560b9ff77ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff441b63640 1 -- 172.21.15.27:0/4226804020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b9fbde210 con 0x560b9fb9d2f0 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff442364640 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb84d80 0x560b9ff77ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff441b63640 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb9d2f0 0x560b9ff70800 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4380a4890 tx=0x7ff4380a7270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2199dabd71689765 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff4257fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff4257fa640 1 -- 172.21.15.27:0/4226804020 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4380ab9d0 con 0x560b9fb9d2f0 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff44ac73740 1 -- 172.21.15.27:0/4226804020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff42c0054f0 con 0x560b9fb9d2f0 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff4257fa640 1 -- 172.21.15.27:0/4226804020 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff4380afff0 con 0x560b9fb9d2f0 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff4257fa640 1 -- 172.21.15.27:0/4226804020 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff4380b0480 con 0x560b9fb9d2f0 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff44ac73740 10 client.25202 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.982+0000 7ff44ac73740 1 -- 172.21.15.27:0/4226804020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560b9fcaa620 con 0x560b9fb9d2f0 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.983+0000 7ff4257fa640 1 -- 172.21.15.27:0/4226804020 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff42c0054f0 con 0x560b9fb9d2f0 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.983+0000 7ff4257fa640 1 -- 172.21.15.27:0/4226804020 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff4380a2f90 con 0x560b9fb9d2f0 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.983+0000 7ff44ac73740 10 client.25202 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.983+0000 7ff44ac73740 10 client.25202 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.983+0000 7ff44ac73740 20 client.25202 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:34.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.983+0000 7ff44ac73740 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560b9fb5a1c0 0x560b9ff7e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.983+0000 7ff44ac73740 4 client.25202 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.983+0000 7ff44ac73740 1 -- 172.21.15.27:0/4226804020 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b9feda070 con 0x560b9fb5a1c0 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.984+0000 7ff441362640 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560b9fb5a1c0 0x560b9ff7e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.985+0000 7ff4257fa640 10 client.25202 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.985+0000 7ff441362640 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560b9fb5a1c0 0x560b9ff7e960 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3eb84f307209394 server_cookie=7a4af93cb7380852 in_seq=0 out_seq=0 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.985+0000 7ff4257fa640 1 -- 172.21.15.27:0/4226804020 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560b9feda070 con 0x560b9fb5a1c0 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.985+0000 7ff4257fa640 10 client.25202 handle_command_reply: tid=0 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.985+0000 7ff44ac73740 1 client.25202 shutdown 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 -- 172.21.15.27:0/4226804020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb9d2f0 msgr2=0x560b9ff70800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb9d2f0 0x560b9ff70800 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4380a4890 tx=0x7ff4380a7270 comp rx=0 tx=0).stop 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 -- 172.21.15.27:0/4226804020 shutdown_connections 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560b9fb5a1c0 0x560b9ff7e960 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb84d80 0x560b9ff77ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 0x560b9ff70d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 --2- 172.21.15.27:0/4226804020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb9d2f0 0x560b9ff70800 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff4257fa640 0 client.25202 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 -- 172.21.15.27:0/4226804020 >> 172.21.15.27:0/4226804020 conn(0x560b9fbba300 msgr2=0x560b9ff61250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 -- 172.21.15.27:0/4226804020 shutdown_connections 2024-04-25T13:41:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 1 -- 172.21.15.27:0/4226804020 wait complete. 2024-04-25T13:41:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:34.986+0000 7ff44ac73740 20 client.25202 trim_cache size 0 max 16384 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.010+0000 7ff44ac73740 1 Processor -- start 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.010+0000 7ff44ac73740 1 -- start start 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.010+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb84d80 0x560b9ff6ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.010+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff77c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.010+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 0x560b9ff6f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.010+0000 7ff441362640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff77c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.010+0000 7ff441b63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb84d80 0x560b9ff6ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.010+0000 7ff441362640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff77c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54268/0 (socket says 172.21.15.27:54268) 2024-04-25T13:41:35.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff441362640 1 -- 172.21.15.27:0/2883037805 learned_addr learned my addr 172.21.15.27:0/2883037805 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff44ac73740 1 -- 172.21.15.27:0/2883037805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff42000c2f0 con 0x560b9fb9d2f0 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff44ac73740 1 -- 172.21.15.27:0/2883037805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff42000eb70 con 0x560b9fb84d80 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff44ac73740 1 -- 172.21.15.27:0/2883037805 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff42000c960 con 0x560b9fbc61b0 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff442364640 1 --2- 172.21.15.27:0/2883037805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 0x560b9ff6f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff441362640 1 -- 172.21.15.27:0/2883037805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 msgr2=0x560b9ff6f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff441362640 1 --2- 172.21.15.27:0/2883037805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 0x560b9ff6f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff441362640 1 -- 172.21.15.27:0/2883037805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb84d80 msgr2=0x560b9ff6ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff441362640 1 --2- 172.21.15.27:0/2883037805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb84d80 0x560b9ff6ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff441362640 1 -- 172.21.15.27:0/2883037805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b9ff6e030 con 0x560b9fb9d2f0 2024-04-25T13:41:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.011+0000 7ff441362640 1 --2- 172.21.15.27:0/2883037805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff77c60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff42c000c00 tx=0x7ff42c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e077142851c003f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.012+0000 7ff4277fe640 1 -- 172.21.15.27:0/2883037805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c057070 con 0x560b9fb9d2f0 2024-04-25T13:41:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.012+0000 7ff4277fe640 1 -- 172.21.15.27:0/2883037805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff42c0315a0 con 0x560b9fb9d2f0 2024-04-25T13:41:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.012+0000 7ff4277fe640 1 -- 172.21.15.27:0/2883037805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c0523c0 con 0x560b9fb9d2f0 2024-04-25T13:41:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 -- 172.21.15.27:0/2883037805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 msgr2=0x560b9ff77c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 --2- 172.21.15.27:0/2883037805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff77c60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff42c000c00 tx=0x7ff42c02a410 comp rx=0 tx=0).stop 2024-04-25T13:41:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 -- 172.21.15.27:0/2883037805 shutdown_connections 2024-04-25T13:41:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 --2- 172.21.15.27:0/2883037805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fbc61b0 0x560b9ff6f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 --2- 172.21.15.27:0/2883037805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff77c60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 --2- 172.21.15.27:0/2883037805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fb84d80 0x560b9ff6ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 -- 172.21.15.27:0/2883037805 >> 172.21.15.27:0/2883037805 conn(0x560b9fbba300 msgr2=0x560b9ff6ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 -- 172.21.15.27:0/2883037805 shutdown_connections 2024-04-25T13:41:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.013+0000 7ff44ac73740 1 -- 172.21.15.27:0/2883037805 wait complete. 2024-04-25T13:41:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.014+0000 7ff44ac73740 1 -- start start 2024-04-25T13:41:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.014+0000 7ff44ac73740 1 Processor -- start 2024-04-25T13:41:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fb84d80 0x560b9ff779d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff74550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff44ac73740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 0x560b9ff74a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff44ac73740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff42000d030 con 0x560b9fb9d2f0 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff44ac73740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff42000e4a0 con 0x560b9fbc61b0 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff44ac73740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff42000d700 con 0x560b9fb84d80 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff44ac73740 10 client.0 _ll_register_callbacks cb 0x7ff438009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441b63640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fb84d80 0x560b9ff779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441b63640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fb84d80 0x560b9ff779d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40916/0 (socket says 172.21.15.27:40916) 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441b63640 1 -- 172.21.15.27:0/1855972665 learned_addr learned my addr 172.21.15.27:0/1855972665 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff442364640 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 0x560b9ff74a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441362640 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff74550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441b63640 1 -- 172.21.15.27:0/1855972665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 msgr2=0x560b9ff74a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441b63640 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 0x560b9ff74a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441b63640 1 -- 172.21.15.27:0/1855972665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 msgr2=0x560b9ff74550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441b63640 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff74550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.015+0000 7ff441b63640 1 -- 172.21.15.27:0/1855972665 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b9ff750d0 con 0x560b9fb84d80 2024-04-25T13:41:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.016+0000 7ff441b63640 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fb84d80 0x560b9ff779d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff438020000 tx=0x7ff43808ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1317295d5edd22c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.016+0000 7ff4257fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:41:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.017+0000 7ff4257fa640 1 -- 172.21.15.27:0/1855972665 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43806c070 con 0x560b9fb84d80 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.017+0000 7ff4257fa640 1 -- 172.21.15.27:0/1855972665 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff4380b0200 con 0x560b9fb84d80 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.017+0000 7ff44ac73740 1 -- 172.21.15.27:0/1855972665 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff42c0315a0 con 0x560b9fb84d80 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.017+0000 7ff4257fa640 1 -- 172.21.15.27:0/1855972665 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff4380a6f70 con 0x560b9fb84d80 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.017+0000 7ff44ac73740 10 client.25210 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.017+0000 7ff44ac73740 1 -- 172.21.15.27:0/1855972665 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560b9ffe5e00 con 0x560b9fb84d80 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff4257fa640 1 -- 172.21.15.27:0/1855972665 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff42c0315a0 con 0x560b9fb84d80 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff4257fa640 1 -- 172.21.15.27:0/1855972665 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff43808a030 con 0x560b9fb84d80 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff44ac73740 10 client.25210 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff44ac73740 10 client.25210 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff44ac73740 20 client.25210 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560b9ffe6650 0x560b9ffe8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff44ac73740 4 client.25210 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:41:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff44ac73740 1 -- 172.21.15.27:0/1855972665 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560b9fed3ee0 con 0x560b9ffe6650 2024-04-25T13:41:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.018+0000 7ff441362640 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560b9ffe6650 0x560b9ffe8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.019+0000 7ff4257fa640 10 client.25210 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.019+0000 7ff441362640 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560b9ffe6650 0x560b9ffe8a70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=286d9433c283de27 server_cookie=21d028174ab001a1 in_seq=0 out_seq=0 2024-04-25T13:41:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.038+0000 7ff4257fa640 1 -- 172.21.15.27:0/1855972665 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560b9fed3ee0 con 0x560b9ffe6650 2024-04-25T13:41:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.038+0000 7ff4257fa640 10 client.25210 handle_command_reply: tid=0 2024-04-25T13:41:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.038+0000 7ff44ac73740 1 client.25210 shutdown 2024-04-25T13:41:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.038+0000 7ff44ac73740 1 -- 172.21.15.27:0/1855972665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fb84d80 msgr2=0x560b9ff779d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.038+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fb84d80 0x560b9ff779d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff438020000 tx=0x7ff43808ecf0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 1 -- 172.21.15.27:0/1855972665 shutdown_connections 2024-04-25T13:41:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560b9ffe6650 0x560b9ffe8a70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b9fbc61b0 0x560b9ff74a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560b9fb9d2f0 0x560b9ff74550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 1 --2- 172.21.15.27:0/1855972665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560b9fb84d80 0x560b9ff779d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff4257fa640 0 client.25210 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 1 -- 172.21.15.27:0/1855972665 >> 172.21.15.27:0/1855972665 conn(0x560b9fbba300 msgr2=0x560b9ff6ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 1 -- 172.21.15.27:0/1855972665 shutdown_connections 2024-04-25T13:41:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 1 -- 172.21.15.27:0/1855972665 wait complete. 2024-04-25T13:41:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.039+0000 7ff44ac73740 20 client.25210 trim_cache size 0 max 16384 2024-04-25T13:41:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:41:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:41:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:41:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:41:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 -- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4280afae0 msgr2=0x7ff4280b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4280afae0 0x7ff4280b1ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x560b9ff67b40 tx=0x7ff434009500 comp rx=0 tx=0).stop 2024-04-25T13:41:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 -- 172.21.15.27:0/3762302037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff43c1042c0 msgr2=0x7ff43c103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff43c1042c0 0x7ff43c103b30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff43800e9c0 tx=0x7ff43800ee80 comp rx=0 tx=0).stop 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 -- 172.21.15.27:0/3762302037 shutdown_connections 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4280afae0 0x7ff4280b1ff0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c1097e0 0x7ff43c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff43c106cb0 0x7ff43c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 --2- 172.21.15.27:0/3762302037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff43c1042c0 0x7ff43c103b30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 -- 172.21.15.27:0/3762302037 >> 172.21.15.27:0/3762302037 conn(0x7ff43c0fdc60 msgr2=0x7ff43c105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 -- 172.21.15.27:0/3762302037 shutdown_connections 2024-04-25T13:41:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:35.040+0000 7ff443dc7640 1 -- 172.21.15.27:0/3762302037 wait complete. 2024-04-25T13:41:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:36 smithi122 ceph-mon[32843]: pgmap v370: 97 pgs: 97 active+clean; 955 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 8.7 MiB/s wr, 801 op/s 2024-04-25T13:41:36.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:36 smithi045 ceph-mon[32731]: pgmap v370: 97 pgs: 97 active+clean; 955 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 8.7 MiB/s wr, 801 op/s 2024-04-25T13:41:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:36 smithi027 ceph-mon[29140]: pgmap v370: 97 pgs: 97 active+clean; 955 MiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 8.7 MiB/s wr, 801 op/s 2024-04-25T13:41:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 30.0 s, 3950.8 tps, lat 25.345 ms stddev 25.717 2024-04-25T13:41:38.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:38 smithi122 ceph-mon[32843]: pgmap v371: 97 pgs: 97 active+clean; 969 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 972 op/s 2024-04-25T13:41:38.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:38 smithi045 ceph-mon[32731]: pgmap v371: 97 pgs: 97 active+clean; 969 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 972 op/s 2024-04-25T13:41:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:38 smithi027 ceph-mon[29140]: pgmap v371: 97 pgs: 97 active+clean; 969 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 972 op/s 2024-04-25T13:41:40.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:40 smithi122 ceph-mon[32843]: pgmap v372: 97 pgs: 97 active+clean; 974 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.4 MiB/s wr, 808 op/s 2024-04-25T13:41:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:40 smithi045 ceph-mon[32731]: pgmap v372: 97 pgs: 97 active+clean; 974 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.4 MiB/s wr, 808 op/s 2024-04-25T13:41:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:40 smithi027 ceph-mon[29140]: pgmap v372: 97 pgs: 97 active+clean; 974 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.4 MiB/s wr, 808 op/s 2024-04-25T13:41:42.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:42 smithi122 ceph-mon[32843]: pgmap v373: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 10 MiB/s wr, 773 op/s 2024-04-25T13:41:42.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:42 smithi045 ceph-mon[32731]: pgmap v373: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 10 MiB/s wr, 773 op/s 2024-04-25T13:41:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:42 smithi027 ceph-mon[29140]: pgmap v373: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 10 MiB/s wr, 773 op/s 2024-04-25T13:41:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 35.0 s, 3396.0 tps, lat 29.278 ms stddev 29.797 2024-04-25T13:41:44.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:44 smithi122 ceph-mon[32843]: pgmap v374: 97 pgs: 97 active+clean; 986 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 15 MiB/s wr, 872 op/s 2024-04-25T13:41:44.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:44 smithi045 ceph-mon[32731]: pgmap v374: 97 pgs: 97 active+clean; 986 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 15 MiB/s wr, 872 op/s 2024-04-25T13:41:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:44 smithi027 ceph-mon[29140]: pgmap v374: 97 pgs: 97 active+clean; 986 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 15 MiB/s wr, 872 op/s 2024-04-25T13:41:46.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:46 smithi122 ceph-mon[32843]: pgmap v375: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-25T13:41:46.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:46 smithi045 ceph-mon[32731]: pgmap v375: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-25T13:41:46.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:46 smithi027 ceph-mon[29140]: pgmap v375: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-25T13:41:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 40.0 s, 3689.0 tps, lat 27.200 ms stddev 29.173 2024-04-25T13:41:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.191+0000 7f424aebd640 1 -- 172.21.15.27:0/100759551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4244079cb0 msgr2=0x7f424407a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.191+0000 7f424aebd640 1 --2- 172.21.15.27:0/100759551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4244079cb0 0x7f424407a0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4230009900 tx=0x7f423002a410 comp rx=0 tx=0).stop 2024-04-25T13:41:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.192+0000 7f424aebd640 1 -- 172.21.15.27:0/100759551 shutdown_connections 2024-04-25T13:41:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.192+0000 7f424aebd640 1 --2- 172.21.15.27:0/100759551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f424407a9b0 0x7f424407adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.192+0000 7f424aebd640 1 --2- 172.21.15.27:0/100759551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4244079cb0 0x7f424407a0c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.192+0000 7f424aebd640 1 --2- 172.21.15.27:0/100759551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4244078a20 0x7f4244079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.192+0000 7f424aebd640 1 -- 172.21.15.27:0/100759551 >> 172.21.15.27:0/100759551 conn(0x7f4244100f40 msgr2=0x7f42441033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.192+0000 7f424aebd640 1 -- 172.21.15.27:0/100759551 shutdown_connections 2024-04-25T13:41:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.192+0000 7f424aebd640 1 -- 172.21.15.27:0/100759551 wait complete. 2024-04-25T13:41:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.193+0000 7f424aebd640 1 Processor -- start 2024-04-25T13:41:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424aebd640 1 -- start start 2024-04-25T13:41:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424aebd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4244078a20 0x7f4244074320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424aebd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4244079cb0 0x7f4244074860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424aebd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f424407a9b0 0x7f4244072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424aebd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f424406cde0 con 0x7f4244079cb0 2024-04-25T13:41:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424aebd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f424406cc60 con 0x7f4244078a20 2024-04-25T13:41:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424aebd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f424406cf60 con 0x7f424407a9b0 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424945a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f424407a9b0 0x7f4244072b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f4243fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4244079cb0 0x7f4244074860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424945a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f424407a9b0 0x7f4244072b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58934/0 (socket says 172.21.15.27:58934) 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f424945a640 1 -- 172.21.15.27:0/2522676304 learned_addr learned my addr 172.21.15.27:0/2522676304 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.194+0000 7f4248c59640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4244078a20 0x7f4244074320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f424945a640 1 -- 172.21.15.27:0/2522676304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4244078a20 msgr2=0x7f4244074320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f424945a640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4244078a20 0x7f4244074320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f424945a640 1 -- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4244079cb0 msgr2=0x7f4244074860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f424945a640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4244079cb0 0x7f4244074860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f424945a640 1 -- 172.21.15.27:0/2522676304 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4244073320 con 0x7f424407a9b0 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f424945a640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f424407a9b0 0x7f4244072b20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f423c00eb10 tx=0x7f423c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f4241ffb640 1 -- 172.21.15.27:0/2522676304 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c004b20 con 0x7f424407a9b0 2024-04-25T13:41:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f4241ffb640 1 -- 172.21.15.27:0/2522676304 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f423c004cc0 con 0x7f424407a9b0 2024-04-25T13:41:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f424aebd640 1 -- 172.21.15.27:0/2522676304 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4244073610 con 0x7f424407a9b0 2024-04-25T13:41:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f4241ffb640 1 -- 172.21.15.27:0/2522676304 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c010670 con 0x7f424407a9b0 2024-04-25T13:41:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.195+0000 7f424aebd640 1 -- 172.21.15.27:0/2522676304 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42441b13c0 con 0x7f424407a9b0 2024-04-25T13:41:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.199+0000 7f4241ffb640 1 -- 172.21.15.27:0/2522676304 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f423c010830 con 0x7f424407a9b0 2024-04-25T13:41:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4251d74740 1 Processor -- start 2024-04-25T13:41:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4251d74740 1 -- start start 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4251d74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5bc1b0 0x55712f95c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4251d74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5932f0 0x55712f95cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4251d74740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f963df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4251d74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55712f8c2460 con 0x55712f5bc1b0 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4251d74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55712f87c590 con 0x55712f5932f0 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4251d74740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55712f8be420 con 0x55712f59b4d0 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4243fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5bc1b0 0x55712f95c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4243fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5bc1b0 0x55712f95c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59028/0 (socket says 172.21.15.27:59028) 2024-04-25T13:41:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4243fff640 1 -- 172.21.15.27:0/1623776245 learned_addr learned my addr 172.21.15.27:0/1623776245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4243fff640 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f963df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4248c59640 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5932f0 0x55712f95cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.200+0000 7f4241ffb640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f42340afb30 0x7f42340b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f4243fff640 1 -- 172.21.15.27:0/1623776245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 msgr2=0x55712f963df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f4243fff640 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f963df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f4243fff640 1 -- 172.21.15.27:0/1623776245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5932f0 msgr2=0x55712f95cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f4243fff640 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5932f0 0x55712f95cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f4243fff640 1 -- 172.21.15.27:0/1623776245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55712f8890d0 con 0x55712f5bc1b0 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f4241ffb640 1 -- 172.21.15.27:0/2522676304 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f423c0df1c0 con 0x7f424407a9b0 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f424945a640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f42340afb30 0x7f42340b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f4243fff640 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5bc1b0 0x55712f95c970 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4230009230 tx=0x7f4230005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e855ead1231c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f422effd640 1 -- 172.21.15.27:0/1623776245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230035030 con 0x55712f5bc1b0 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f422effd640 1 -- 172.21.15.27:0/1623776245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f423003d450 con 0x55712f5bc1b0 2024-04-25T13:41:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.201+0000 7f422effd640 1 -- 172.21.15.27:0/1623776245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423003c6a0 con 0x55712f5bc1b0 2024-04-25T13:41:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.202+0000 7f4251d74740 1 -- 172.21.15.27:0/1623776245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5bc1b0 msgr2=0x55712f95c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.202+0000 7f4251d74740 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5bc1b0 0x55712f95c970 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4230009230 tx=0x7f4230005fd0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.203+0000 7f424945a640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f42340afb30 0x7f42340b2040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f423c0049d0 tx=0x7f423c005810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.203+0000 7f4251d74740 1 -- 172.21.15.27:0/1623776245 shutdown_connections 2024-04-25T13:41:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.203+0000 7f4251d74740 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f963df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.203+0000 7f4251d74740 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5932f0 0x55712f95cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.203+0000 7f4251d74740 1 --2- 172.21.15.27:0/1623776245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5bc1b0 0x55712f95c970 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.203+0000 7f4251d74740 1 -- 172.21.15.27:0/1623776245 >> 172.21.15.27:0/1623776245 conn(0x55712f5b0300 msgr2=0x55712f958300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.203+0000 7f4251d74740 1 -- 172.21.15.27:0/1623776245 shutdown_connections 2024-04-25T13:41:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.204+0000 7f4251d74740 1 -- 172.21.15.27:0/1623776245 wait complete. 2024-04-25T13:41:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.204+0000 7f4251d74740 1 -- start start 2024-04-25T13:41:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.204+0000 7f4251d74740 1 Processor -- start 2024-04-25T13:41:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4251d74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f966670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4251d74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f59b4d0 0x55712f966bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4251d74740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5bc1b0 0x55712f96daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4251d74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55712f9647e0 con 0x55712f5932f0 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4251d74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55712f849b60 con 0x55712f59b4d0 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4251d74740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55712f964960 con 0x55712f5bc1b0 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4251d74740 10 client.0 _ll_register_callbacks cb 0x55712f86e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4243fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f966670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4243fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f966670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59042/0 (socket says 172.21.15.27:59042) 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4243fff640 1 -- 172.21.15.27:0/3192511379 learned_addr learned my addr 172.21.15.27:0/3192511379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4248c59640 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f59b4d0 0x55712f966bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4243fff640 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5bc1b0 0x55712f96daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4248c59640 1 -- 172.21.15.27:0/3192511379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5bc1b0 msgr2=0x55712f96daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4248c59640 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5bc1b0 0x55712f96daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4248c59640 1 -- 172.21.15.27:0/3192511379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 msgr2=0x55712f966670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4248c59640 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f966670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4248c59640 1 -- 172.21.15.27:0/3192511379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55712f5d4210 con 0x55712f59b4d0 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.205+0000 7f4243fff640 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f966670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.206+0000 7f4248c59640 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f59b4d0 0x55712f966bb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55712f95dd50 tx=0x7f4238010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=873164cc5adddd2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.206+0000 7f4223fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.206+0000 7f4223fff640 1 -- 172.21.15.27:0/3192511379 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423800fae0 con 0x55712f59b4d0 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.206+0000 7f4251d74740 1 -- 172.21.15.27:0/3192511379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f423003d450 con 0x55712f59b4d0 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.206+0000 7f4223fff640 1 -- 172.21.15.27:0/3192511379 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f423800fc80 con 0x55712f59b4d0 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.206+0000 7f4223fff640 1 -- 172.21.15.27:0/3192511379 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4238004020 con 0x55712f59b4d0 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.206+0000 7f4251d74740 10 client.25217 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.206+0000 7f4251d74740 1 -- 172.21.15.27:0/3192511379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55712f6a0620 con 0x55712f59b4d0 2024-04-25T13:41:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4223fff640 1 -- 172.21.15.27:0/3192511379 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f423003d450 con 0x55712f59b4d0 2024-04-25T13:41:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4223fff640 1 -- 172.21.15.27:0/3192511379 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f4238004940 con 0x55712f59b4d0 2024-04-25T13:41:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4251d74740 10 client.25217 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4251d74740 10 client.25217 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4251d74740 20 client.25217 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4251d74740 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55712f57ad80 0x55712f9743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4251d74740 4 client.25217 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4251d74740 1 -- 172.21.15.27:0/3192511379 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55712f8d0070 con 0x55712f57ad80 2024-04-25T13:41:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.208+0000 7f4243fff640 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55712f57ad80 0x55712f9743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.209+0000 7f4223fff640 10 client.25217 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.209+0000 7f4243fff640 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55712f57ad80 0x55712f9743a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=997f1196e4bdc3a4 server_cookie=716bfac30a8b4a4 in_seq=0 out_seq=0 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.210+0000 7f4223fff640 1 -- 172.21.15.27:0/3192511379 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55712f8d0070 con 0x55712f57ad80 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.210+0000 7f4223fff640 10 client.25217 handle_command_reply: tid=0 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.210+0000 7f4251d74740 1 client.25217 shutdown 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 -- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f59b4d0 msgr2=0x55712f966bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f59b4d0 0x55712f966bb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55712f95dd50 tx=0x7f4238010c80 comp rx=0 tx=0).stop 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 -- 172.21.15.27:0/3192511379 shutdown_connections 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55712f57ad80 0x55712f9743a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5bc1b0 0x55712f96daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f59b4d0 0x55712f966bb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 --2- 172.21.15.27:0/3192511379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f966670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4223fff640 0 client.25217 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 -- 172.21.15.27:0/3192511379 >> 172.21.15.27:0/3192511379 conn(0x55712f5b0300 msgr2=0x55712f95b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 -- 172.21.15.27:0/3192511379 shutdown_connections 2024-04-25T13:41:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 1 -- 172.21.15.27:0/3192511379 wait complete. 2024-04-25T13:41:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.211+0000 7f4251d74740 20 client.25217 trim_cache size 0 max 16384 2024-04-25T13:41:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4251d74740 1 Processor -- start 2024-04-25T13:41:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4251d74740 1 -- start start 2024-04-25T13:41:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4251d74740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5932f0 0x55712f979ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4251d74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f59b4d0 0x55712f97a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4251d74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f96d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4251d74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f42441b7ed0 con 0x55712f59b4d0 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4251d74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42441ba750 con 0x55712f5bc1b0 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4251d74740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f42441b8540 con 0x55712f5932f0 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4248c59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f59b4d0 0x55712f97a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4248c59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f59b4d0 0x55712f97a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59052/0 (socket says 172.21.15.27:59052) 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4248c59640 1 -- 172.21.15.27:0/1828297970 learned_addr learned my addr 172.21.15.27:0/1828297970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4243fff640 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5932f0 0x55712f979ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4243fff640 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f96d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4248c59640 1 -- 172.21.15.27:0/1828297970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5932f0 msgr2=0x55712f979ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4248c59640 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5932f0 0x55712f979ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4248c59640 1 -- 172.21.15.27:0/1828297970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 msgr2=0x55712f96d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4248c59640 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f96d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4248c59640 1 -- 172.21.15.27:0/1828297970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55712f96df00 con 0x55712f59b4d0 2024-04-25T13:41:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.234+0000 7f4243fff640 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f96d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:41:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.235+0000 7f4248c59640 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f59b4d0 0x55712f97a0e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4238004d30 tx=0x7f4238003800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae68086ba69ad0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.235+0000 7f422e7fc640 1 -- 172.21.15.27:0/1828297970 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238032070 con 0x55712f59b4d0 2024-04-25T13:41:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.235+0000 7f422e7fc640 1 -- 172.21.15.27:0/1828297970 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4238028040 con 0x55712f59b4d0 2024-04-25T13:41:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.235+0000 7f422e7fc640 1 -- 172.21.15.27:0/1828297970 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238027030 con 0x55712f59b4d0 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 -- 172.21.15.27:0/1828297970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f59b4d0 msgr2=0x55712f97a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f59b4d0 0x55712f97a0e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4238004d30 tx=0x7f4238003800 comp rx=0 tx=0).stop 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 -- 172.21.15.27:0/1828297970 shutdown_connections 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f96d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f59b4d0 0x55712f97a0e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 --2- 172.21.15.27:0/1828297970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f5932f0 0x55712f979ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 -- 172.21.15.27:0/1828297970 >> 172.21.15.27:0/1828297970 conn(0x55712f5b0300 msgr2=0x55712f964d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 -- 172.21.15.27:0/1828297970 shutdown_connections 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.236+0000 7f4251d74740 1 -- 172.21.15.27:0/1828297970 wait complete. 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.237+0000 7f4251d74740 1 -- start start 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.237+0000 7f4251d74740 1 Processor -- start 2024-04-25T13:41:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4251d74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f96cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4251d74740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f96d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4251d74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f9dab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4251d74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f42441b8c10 con 0x55712f5932f0 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4251d74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f42441ba080 con 0x55712f5bc1b0 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4251d74740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f42441b92e0 con 0x55712f59b4d0 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4251d74740 10 client.0 _ll_register_callbacks cb 0x7f423800a6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4248c59640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f96d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4248c59640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f96d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58974/0 (socket says 172.21.15.27:58974) 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4248c59640 1 -- 172.21.15.27:0/2912622109 learned_addr learned my addr 172.21.15.27:0/2912622109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4243fff640 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f96cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.238+0000 7f4243fff640 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f9dab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.239+0000 7f4248c59640 1 -- 172.21.15.27:0/2912622109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 msgr2=0x55712f9dab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.239+0000 7f4248c59640 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f9dab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.239+0000 7f4248c59640 1 -- 172.21.15.27:0/2912622109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 msgr2=0x55712f96cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.239+0000 7f4248c59640 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f96cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.239+0000 7f4248c59640 1 -- 172.21.15.27:0/2912622109 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55712f9db100 con 0x55712f59b4d0 2024-04-25T13:41:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.239+0000 7f4243fff640 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f96cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:41:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.239+0000 7f4248c59640 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f96d420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4238004600 tx=0x7f4238010500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0c1ca6d029db121 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.239+0000 7f4223fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:41:48.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.240+0000 7f4223fff640 1 -- 172.21.15.27:0/2912622109 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238032040 con 0x55712f59b4d0 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.240+0000 7f4223fff640 1 -- 172.21.15.27:0/2912622109 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f423800fae0 con 0x55712f59b4d0 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.240+0000 7f4251d74740 1 -- 172.21.15.27:0/2912622109 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4238028040 con 0x55712f59b4d0 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4223fff640 1 -- 172.21.15.27:0/2912622109 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4238027040 con 0x55712f59b4d0 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4251d74740 10 client.25228 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4251d74740 1 -- 172.21.15.27:0/2912622109 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55712f9dc390 con 0x55712f59b4d0 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4223fff640 1 -- 172.21.15.27:0/2912622109 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4238003d20 con 0x55712f59b4d0 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4223fff640 1 -- 172.21.15.27:0/2912622109 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f4238035020 con 0x55712f59b4d0 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4251d74740 10 client.25228 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4251d74740 10 client.25228 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4251d74740 20 client.25228 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4251d74740 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55712f5501c0 0x55712f9dec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4251d74740 4 client.25228 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:41:48.244 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.241+0000 7f4251d74740 1 -- 172.21.15.27:0/2912622109 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f4238028040 con 0x55712f5501c0 2024-04-25T13:41:48.244 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.242+0000 7f4243fff640 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55712f5501c0 0x55712f9dec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:48.244 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.243+0000 7f4243fff640 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55712f5501c0 0x55712f9dec80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7187cd8f4fbb65 server_cookie=44bb1b101cf04fb9 in_seq=0 out_seq=0 2024-04-25T13:41:48.244 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.243+0000 7f4223fff640 10 client.25228 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.268+0000 7f4223fff640 1 -- 172.21.15.27:0/2912622109 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f4238028040 con 0x55712f5501c0 2024-04-25T13:41:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.268+0000 7f4223fff640 10 client.25228 handle_command_reply: tid=0 2024-04-25T13:41:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.268+0000 7f4251d74740 1 client.25228 shutdown 2024-04-25T13:41:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.269+0000 7f4251d74740 1 -- 172.21.15.27:0/2912622109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 msgr2=0x55712f96d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.269+0000 7f4251d74740 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f96d420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4238004600 tx=0x7f4238010500 comp rx=0 tx=0).stop 2024-04-25T13:41:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.269+0000 7f4251d74740 1 -- 172.21.15.27:0/2912622109 shutdown_connections 2024-04-25T13:41:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.269+0000 7f4251d74740 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55712f5501c0 0x55712f9dec80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.269+0000 7f4251d74740 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55712f5bc1b0 0x55712f9dab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.269+0000 7f4223fff640 0 client.25228 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.269+0000 7f4251d74740 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55712f59b4d0 0x55712f96d420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.269+0000 7f4251d74740 1 --2- 172.21.15.27:0/2912622109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55712f5932f0 0x55712f96cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f4251d74740 1 -- 172.21.15.27:0/2912622109 >> 172.21.15.27:0/2912622109 conn(0x55712f5b0300 msgr2=0x55712f95b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f4251d74740 1 -- 172.21.15.27:0/2912622109 shutdown_connections 2024-04-25T13:41:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f4251d74740 1 -- 172.21.15.27:0/2912622109 wait complete. 2024-04-25T13:41:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f4251d74740 20 client.25228 trim_cache size 0 max 16384 2024-04-25T13:41:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:41:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:41:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:41:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:41:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 -- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f42340afb30 msgr2=0x7f42340b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f42340afb30 0x7f42340b2040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f423c0049d0 tx=0x7f423c005810 comp rx=0 tx=0).stop 2024-04-25T13:41:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 -- 172.21.15.27:0/2522676304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f424407a9b0 msgr2=0x7f4244072b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f424407a9b0 0x7f4244072b20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f423c00eb10 tx=0x7f423c00c2d0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 -- 172.21.15.27:0/2522676304 shutdown_connections 2024-04-25T13:41:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f42340afb30 0x7f42340b2040 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f424407a9b0 0x7f4244072b20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4244079cb0 0x7f4244074860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 --2- 172.21.15.27:0/2522676304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4244078a20 0x7f4244074320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.270+0000 7f424aebd640 1 -- 172.21.15.27:0/2522676304 >> 172.21.15.27:0/2522676304 conn(0x7f4244100f40 msgr2=0x7f424410fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.271+0000 7f424aebd640 1 -- 172.21.15.27:0/2522676304 shutdown_connections 2024-04-25T13:41:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:41:48.271+0000 7f424aebd640 1 -- 172.21.15.27:0/2522676304 wait complete. 2024-04-25T13:41:48.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:48 smithi122 ceph-mon[32843]: pgmap v376: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 944 op/s 2024-04-25T13:41:48.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:48 smithi045 ceph-mon[32731]: pgmap v376: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 944 op/s 2024-04-25T13:41:48.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:48 smithi027 ceph-mon[29140]: pgmap v376: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 944 op/s 2024-04-25T13:41:50.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:50 smithi122 ceph-mon[32843]: pgmap v377: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 716 op/s 2024-04-25T13:41:50.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:50 smithi045 ceph-mon[32731]: pgmap v377: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 716 op/s 2024-04-25T13:41:50.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:50 smithi027 ceph-mon[29140]: pgmap v377: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 716 op/s 2024-04-25T13:41:52.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:52 smithi122 ceph-mon[32843]: pgmap v378: 97 pgs: 97 active+clean; 996 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 745 op/s 2024-04-25T13:41:52.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:52 smithi045 ceph-mon[32731]: pgmap v378: 97 pgs: 97 active+clean; 996 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 745 op/s 2024-04-25T13:41:52.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:52 smithi027 ceph-mon[29140]: pgmap v378: 97 pgs: 97 active+clean; 996 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 745 op/s 2024-04-25T13:41:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 45.0 s, 3376.0 tps, lat 29.552 ms stddev 31.266 2024-04-25T13:41:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:54 smithi122 ceph-mon[32843]: pgmap v379: 97 pgs: 97 active+clean; 996 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 842 op/s 2024-04-25T13:41:54.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:54 smithi045 ceph-mon[32731]: pgmap v379: 97 pgs: 97 active+clean; 996 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 842 op/s 2024-04-25T13:41:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:54 smithi027 ceph-mon[29140]: pgmap v379: 97 pgs: 97 active+clean; 996 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 842 op/s 2024-04-25T13:41:56.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:56 smithi122 ceph-mon[32843]: pgmap v380: 97 pgs: 97 active+clean; 1004 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-04-25T13:41:56.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:56 smithi045 ceph-mon[32731]: pgmap v380: 97 pgs: 97 active+clean; 1004 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-04-25T13:41:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:56 smithi027 ceph-mon[29140]: pgmap v380: 97 pgs: 97 active+clean; 1004 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-04-25T13:41:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 50.0 s, 3428.2 tps, lat 29.226 ms stddev 30.914 2024-04-25T13:41:58.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:41:58.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 -- 172.21.15.27:0/1472223204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb524108930 msgr2=0x7fb52410ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 --2- 172.21.15.27:0/1472223204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb524108930 0x7fb52410ad50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb518009900 tx=0x7fb51802a410 comp rx=0 tx=0).stop 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 -- 172.21.15.27:0/1472223204 shutdown_connections 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 --2- 172.21.15.27:0/1472223204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb524108930 0x7fb52410ad50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 --2- 172.21.15.27:0/1472223204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb524103c30 0x7fb524104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 --2- 172.21.15.27:0/1472223204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5241029f0 0x7fb5241036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 -- 172.21.15.27:0/1472223204 >> 172.21.15.27:0/1472223204 conn(0x7fb5240fdc00 msgr2=0x7fb524100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 -- 172.21.15.27:0/1472223204 shutdown_connections 2024-04-25T13:41:58.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.246+0000 7fb52904a640 1 -- 172.21.15.27:0/1472223204 wait complete. 2024-04-25T13:41:58.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb52904a640 1 Processor -- start 2024-04-25T13:41:58.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb52904a640 1 -- start start 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb52904a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5241029f0 0x7fb52419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb52904a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb524103c30 0x7fb52419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb52904a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb524108930 0x7fb5241a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb52904a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb52410f680 con 0x7fb524103c30 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb52904a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb52410f500 con 0x7fb5241029f0 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb52904a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb52410f800 con 0x7fb524108930 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb522d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5241029f0 0x7fb52419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb522d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5241029f0 0x7fb52419a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47856/0 (socket says 172.21.15.27:47856) 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.247+0000 7fb522d9d640 1 -- 172.21.15.27:0/4059202567 learned_addr learned my addr 172.21.15.27:0/4059202567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.248+0000 7fb52259c640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb524103c30 0x7fb52419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.248+0000 7fb52359e640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb524108930 0x7fb5241a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.248+0000 7fb522d9d640 1 -- 172.21.15.27:0/4059202567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb524108930 msgr2=0x7fb5241a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.248+0000 7fb522d9d640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb524108930 0x7fb5241a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.248+0000 7fb522d9d640 1 -- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb524103c30 msgr2=0x7fb52419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.248+0000 7fb522d9d640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb524103c30 0x7fb52419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.248+0000 7fb522d9d640 1 -- 172.21.15.27:0/4059202567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5241a1cc0 con 0x7fb5241029f0 2024-04-25T13:41:58.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.249+0000 7fb522d9d640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5241029f0 0x7fb52419a230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb51c00b990 tx=0x7fb51c00be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:58.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.249+0000 7fb50bfff640 1 -- 172.21.15.27:0/4059202567 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c019070 con 0x7fb5241029f0 2024-04-25T13:41:58.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.249+0000 7fb50bfff640 1 -- 172.21.15.27:0/4059202567 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb51c00cba0 con 0x7fb5241029f0 2024-04-25T13:41:58.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.249+0000 7fb50bfff640 1 -- 172.21.15.27:0/4059202567 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c0125a0 con 0x7fb5241029f0 2024-04-25T13:41:58.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.249+0000 7fb52904a640 1 -- 172.21.15.27:0/4059202567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52410b8a0 con 0x7fb5241029f0 2024-04-25T13:41:58.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.249+0000 7fb52904a640 1 -- 172.21.15.27:0/4059202567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52410bd30 con 0x7fb5241029f0 2024-04-25T13:41:58.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.252+0000 7fb50bfff640 1 -- 172.21.15.27:0/4059202567 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb51c01c070 con 0x7fb5241029f0 2024-04-25T13:41:58.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb50bfff640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb50c0afc00 0x7fb50c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52ffd8740 1 Processor -- start 2024-04-25T13:41:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52ffd8740 1 -- start start 2024-04-25T13:41:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb50bfff640 1 -- 172.21.15.27:0/4059202567 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb51c0de160 con 0x7fb5241029f0 2024-04-25T13:41:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1dca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1dcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1ddc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55be8c141d80 con 0x55be8be3b390 2024-04-25T13:41:58.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55be8c0fb8b0 con 0x55be8be1a6b0 2024-04-25T13:41:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55be8c13dd40 con 0x55be8be124d0 2024-04-25T13:41:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1ddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1ddc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47872/0 (socket says 172.21.15.27:47872) 2024-04-25T13:41:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52359e640 1 -- 172.21.15.27:0/496869397 learned_addr learned my addr 172.21.15.27:0/496869397 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb52259c640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb50c0afc00 0x7fb50c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.253+0000 7fb522d9d640 1 --2- 172.21.15.27:0/496869397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1dca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.254+0000 7fb52359e640 1 -- 172.21.15.27:0/496869397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 msgr2=0x55be8c1dcfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.254+0000 7fb52359e640 1 --2- 172.21.15.27:0/496869397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1dcfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.254+0000 7fb52359e640 1 -- 172.21.15.27:0/496869397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 msgr2=0x55be8c1dca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.254+0000 7fb52359e640 1 --2- 172.21.15.27:0/496869397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1dca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.254+0000 7fb52359e640 1 -- 172.21.15.27:0/496869397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be8c0cca40 con 0x55be8be1a6b0 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.254+0000 7fb52359e640 1 --2- 172.21.15.27:0/496869397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1ddc70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb518005f70 tx=0x7fb518032c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2447b6002edd4483 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.254+0000 7fb5097fa640 1 -- 172.21.15.27:0/496869397 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51803f030 con 0x55be8be1a6b0 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.255+0000 7fb5097fa640 1 -- 172.21.15.27:0/496869397 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb518004040 con 0x55be8be1a6b0 2024-04-25T13:41:58.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.255+0000 7fb5097fa640 1 -- 172.21.15.27:0/496869397 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518035070 con 0x55be8be1a6b0 2024-04-25T13:41:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 -- 172.21.15.27:0/496869397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 msgr2=0x55be8c1ddc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52259c640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb50c0afc00 0x7fb50c0b2110 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55be8c1dc780 tx=0x7fb514005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/496869397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1ddc70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb518005f70 tx=0x7fb518032c50 comp rx=0 tx=0).stop 2024-04-25T13:41:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 -- 172.21.15.27:0/496869397 shutdown_connections 2024-04-25T13:41:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/496869397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1ddc70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/496869397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1dcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/496869397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1dca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 -- 172.21.15.27:0/496869397 >> 172.21.15.27:0/496869397 conn(0x55be8be2f4e0 msgr2=0x55be8c1d8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 -- 172.21.15.27:0/496869397 shutdown_connections 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 -- 172.21.15.27:0/496869397 wait complete. 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 -- start start 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.256+0000 7fb52ffd8740 1 Processor -- start 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1f7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1f7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1fede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55be8c1de720 con 0x55be8be3b390 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1fede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1f7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1fede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43858/0 (socket says 172.21.15.27:43858) 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52359e640 1 -- 172.21.15.27:0/66594042 learned_addr learned my addr 172.21.15.27:0/66594042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52259c640 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1f7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47880/0 (socket says 172.21.15.27:47880) 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55be8c0c8d10 con 0x55be8be1a6b0 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55be8c1de8a0 con 0x55be8be124d0 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb522d9d640 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52ffd8740 10 client.0 _ll_register_callbacks cb 0x55be8c0f5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52259c640 1 -- 172.21.15.27:0/66594042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 msgr2=0x55be8c1f7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52259c640 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1f7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52259c640 1 -- 172.21.15.27:0/66594042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 msgr2=0x55be8c1fede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52259c640 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1fede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52259c640 1 -- 172.21.15.27:0/66594042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be8bf1d7d0 con 0x55be8be1a6b0 2024-04-25T13:41:58.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.257+0000 7fb52359e640 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.258+0000 7fb52259c640 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1f7e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb52419b2d0 tx=0x7fb5140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff1480beb9d2cbff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.258+0000 7fb506ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.259+0000 7fb506ff5640 1 -- 172.21.15.27:0/66594042 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514021070 con 0x55be8be1a6b0 2024-04-25T13:41:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.260+0000 7fb506ff5640 1 -- 172.21.15.27:0/66594042 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb51400fd50 con 0x55be8be1a6b0 2024-04-25T13:41:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.260+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb518004040 con 0x55be8be1a6b0 2024-04-25T13:41:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.260+0000 7fb506ff5640 1 -- 172.21.15.27:0/66594042 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb51402b050 con 0x55be8be1a6b0 2024-04-25T13:41:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.260+0000 7fb52ffd8740 10 client.25232 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:58.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.260+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55be8be533f0 con 0x55be8be1a6b0 2024-04-25T13:41:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb506ff5640 1 -- 172.21.15.27:0/66594042 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb514010e40 con 0x55be8be1a6b0 2024-04-25T13:41:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb506ff5640 1 -- 172.21.15.27:0/66594042 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb514025550 con 0x55be8be1a6b0 2024-04-25T13:41:58.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb52ffd8740 10 client.25232 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb52ffd8740 10 client.25232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb52ffd8740 20 client.25232 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55be8bdf9f60 0x55be8c205380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb52ffd8740 4 client.25232 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb518004040 con 0x55be8bdf9f60 2024-04-25T13:41:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.261+0000 7fb522d9d640 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55be8bdf9f60 0x55be8c205380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.262+0000 7fb506ff5640 10 client.25232 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.262+0000 7fb522d9d640 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55be8bdf9f60 0x55be8c205380 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d1fe67b7e2b9aca server_cookie=8696fedc2c129531 in_seq=0 out_seq=0 2024-04-25T13:41:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb506ff5640 1 -- 172.21.15.27:0/66594042 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fb518004040 con 0x55be8bdf9f60 2024-04-25T13:41:58.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb506ff5640 10 client.25232 handle_command_reply: tid=0 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 client.25232 shutdown 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 msgr2=0x55be8c1f7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1f7e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb52419b2d0 tx=0x7fb5140046c0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 shutdown_connections 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55be8bdf9f60 0x55be8c205380 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be3b390 0x55be8c1fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be1a6b0 0x55be8c1f7e80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb506ff5640 0 client.25232 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/66594042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 >> 172.21.15.27:0/66594042 conn(0x55be8be2f4e0 msgr2=0x55be8c1d73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 shutdown_connections 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.263+0000 7fb52ffd8740 1 -- 172.21.15.27:0/66594042 wait complete. 2024-04-25T13:41:58.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.264+0000 7fb52ffd8740 20 client.25232 trim_cache size 0 max 16384 2024-04-25T13:41:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.293+0000 7fb52ffd8740 1 Processor -- start 2024-04-25T13:41:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.293+0000 7fb52ffd8740 1 -- start start 2024-04-25T13:41:58.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8bdf9f60 0x55be8c1e1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1f6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be1a6b0 0x55be8c1db780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb524071680 con 0x55be8be1a6b0 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be1a6b0 0x55be8c1db780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be1a6b0 0x55be8c1db780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43862/0 (socket says 172.21.15.27:43862) 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52359e640 1 -- 172.21.15.27:0/1682881935 learned_addr learned my addr 172.21.15.27:0/1682881935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb524073f00 con 0x55be8bdf9f60 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.294+0000 7fb52ffd8740 1 -- 172.21.15.27:0/1682881935 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb524071cf0 con 0x55be8be124d0 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.295+0000 7fb522d9d640 1 --2- 172.21.15.27:0/1682881935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8bdf9f60 0x55be8c1e1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.295+0000 7fb52359e640 1 -- 172.21.15.27:0/1682881935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 msgr2=0x55be8c1f6940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.295+0000 7fb52359e640 1 --2- 172.21.15.27:0/1682881935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1f6940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.295+0000 7fb52359e640 1 -- 172.21.15.27:0/1682881935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8bdf9f60 msgr2=0x55be8c1e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.295+0000 7fb52359e640 1 --2- 172.21.15.27:0/1682881935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8bdf9f60 0x55be8c1e1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.295+0000 7fb52359e640 1 -- 172.21.15.27:0/1682881935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be8be130f0 con 0x55be8be1a6b0 2024-04-25T13:41:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.295+0000 7fb52359e640 1 --2- 172.21.15.27:0/1682881935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be1a6b0 0x55be8c1db780 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb518032cc0 tx=0x7fb51802ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac6ed1408d220e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.295+0000 7fb508ff9640 1 -- 172.21.15.27:0/1682881935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51803f040 con 0x55be8be1a6b0 2024-04-25T13:41:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.296+0000 7fb508ff9640 1 -- 172.21.15.27:0/1682881935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb518048030 con 0x55be8be1a6b0 2024-04-25T13:41:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.296+0000 7fb508ff9640 1 -- 172.21.15.27:0/1682881935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518035040 con 0x55be8be1a6b0 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 -- 172.21.15.27:0/1682881935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be1a6b0 msgr2=0x55be8c1db780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/1682881935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be1a6b0 0x55be8c1db780 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb518032cc0 tx=0x7fb51802ab10 comp rx=0 tx=0).stop 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 -- 172.21.15.27:0/1682881935 shutdown_connections 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/1682881935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8be1a6b0 0x55be8c1db780 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/1682881935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be124d0 0x55be8c1f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/1682881935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8bdf9f60 0x55be8c1e1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 -- 172.21.15.27:0/1682881935 >> 172.21.15.27:0/1682881935 conn(0x55be8be2f4e0 msgr2=0x55be8c1db070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 -- 172.21.15.27:0/1682881935 shutdown_connections 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 -- 172.21.15.27:0/1682881935 wait complete. 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 -- start start 2024-04-25T13:41:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 Processor -- start 2024-04-25T13:41:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.298+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8bdf9f60 0x55be8c1fc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be124d0 0x55be8c1fccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52ffd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be1a6b0 0x55be8c1fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5240723c0 con 0x55be8bdf9f60 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb524073830 con 0x55be8be124d0 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52ffd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb524072a90 con 0x55be8be1a6b0 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52ffd8740 10 client.0 _ll_register_callbacks cb 0x7fb514014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb522d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8bdf9f60 0x55be8c1fc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be124d0 0x55be8c1fccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be124d0 0x55be8c1fccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47888/0 (socket says 172.21.15.27:47888) 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be1a6b0 0x55be8c1fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb522d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8bdf9f60 0x55be8c1fc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43874/0 (socket says 172.21.15.27:43874) 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 -- 172.21.15.27:0/334626793 learned_addr learned my addr 172.21.15.27:0/334626793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 -- 172.21.15.27:0/334626793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be1a6b0 msgr2=0x55be8c1fdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be1a6b0 0x55be8c1fdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 -- 172.21.15.27:0/334626793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8bdf9f60 msgr2=0x55be8c1fc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8bdf9f60 0x55be8c1fc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 -- 172.21.15.27:0/334626793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be8c1024b0 con 0x55be8be124d0 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb522d9d640 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8bdf9f60 0x55be8c1fc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb52259c640 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be124d0 0x55be8c1fccf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb514000c00 tx=0x7fb51400fe00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7821139f269c0e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.299+0000 7fb506ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.300+0000 7fb506ff5640 1 -- 172.21.15.27:0/334626793 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514021070 con 0x55be8be124d0 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.300+0000 7fb506ff5640 1 -- 172.21.15.27:0/334626793 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb514005030 con 0x55be8be124d0 2024-04-25T13:41:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.300+0000 7fb52ffd8740 1 -- 172.21.15.27:0/334626793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb518048030 con 0x55be8be124d0 2024-04-25T13:41:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.300+0000 7fb506ff5640 1 -- 172.21.15.27:0/334626793 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb514040050 con 0x55be8be124d0 2024-04-25T13:41:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.300+0000 7fb52ffd8740 10 client.25235 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.300+0000 7fb52ffd8740 1 -- 172.21.15.27:0/334626793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55be8c1ff330 con 0x55be8be124d0 2024-04-25T13:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.301+0000 7fb506ff5640 1 -- 172.21.15.27:0/334626793 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb518048030 con 0x55be8be124d0 2024-04-25T13:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.301+0000 7fb506ff5640 1 -- 172.21.15.27:0/334626793 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb514004890 con 0x55be8be124d0 2024-04-25T13:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.302+0000 7fb52ffd8740 10 client.25235 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.302+0000 7fb52ffd8740 10 client.25235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.302+0000 7fb52ffd8740 20 client.25235 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.302+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55be8be3b390 0x55be8c26f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.302+0000 7fb52ffd8740 4 client.25235 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.302+0000 7fb52ffd8740 1 -- 172.21.15.27:0/334626793 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55be8c14f990 con 0x55be8be3b390 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.302+0000 7fb522d9d640 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55be8be3b390 0x55be8c26f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.302+0000 7fb506ff5640 10 client.25235 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb522d9d640 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55be8be3b390 0x55be8c26f9c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f58d0c312806672c server_cookie=30b3163a3824d004 in_seq=0 out_seq=0 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb506ff5640 1 -- 172.21.15.27:0/334626793 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55be8c14f990 con 0x55be8be3b390 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb506ff5640 10 client.25235 handle_command_reply: tid=0 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 client.25235 shutdown 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 -- 172.21.15.27:0/334626793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be124d0 msgr2=0x55be8c1fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be124d0 0x55be8c1fccf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb514000c00 tx=0x7fb51400fe00 comp rx=0 tx=0).stop 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 -- 172.21.15.27:0/334626793 shutdown_connections 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55be8be3b390 0x55be8c26f9c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be8be1a6b0 0x55be8c1fdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb506ff5640 0 client.25235 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be8be124d0 0x55be8c1fccf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 --2- 172.21.15.27:0/334626793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55be8bdf9f60 0x55be8c1fc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 -- 172.21.15.27:0/334626793 >> 172.21.15.27:0/334626793 conn(0x55be8be2f4e0 msgr2=0x55be8c1dbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 -- 172.21.15.27:0/334626793 shutdown_connections 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 1 -- 172.21.15.27:0/334626793 wait complete. 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.303+0000 7fb52ffd8740 20 client.25235 trim_cache size 0 max 16384 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:41:58.306 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:41:58.307 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:41:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 -- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb50c0afc00 msgr2=0x7fb50c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb50c0afc00 0x7fb50c0b2110 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55be8c1dc780 tx=0x7fb514005da0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 -- 172.21.15.27:0/4059202567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5241029f0 msgr2=0x7fb52419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5241029f0 0x7fb52419a230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb51c00b990 tx=0x7fb51c00be50 comp rx=0 tx=0).stop 2024-04-25T13:41:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 -- 172.21.15.27:0/4059202567 shutdown_connections 2024-04-25T13:41:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb50c0afc00 0x7fb50c0b2110 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb524108930 0x7fb5241a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb524103c30 0x7fb52419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 --2- 172.21.15.27:0/4059202567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5241029f0 0x7fb52419a230 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 -- 172.21.15.27:0/4059202567 >> 172.21.15.27:0/4059202567 conn(0x7fb5240fdc00 msgr2=0x7fb524104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 -- 172.21.15.27:0/4059202567 shutdown_connections 2024-04-25T13:41:58.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.304+0000 7fb52904a640 1 -- 172.21.15.27:0/4059202567 wait complete. 2024-04-25T13:41:58.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:58 smithi122 ceph-mon[32843]: pgmap v381: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 903 op/s 2024-04-25T13:41:58.318 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:58 smithi027 ceph-mon[29140]: pgmap v381: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 903 op/s 2024-04-25T13:41:58.319 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:41:58.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:41:58.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:58 smithi045 ceph-mon[32731]: pgmap v381: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 903 op/s 2024-04-25T13:41:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.482+0000 7f6b52b97640 1 -- 172.21.15.27:0/3342000859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 msgr2=0x7f6b4c105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.482+0000 7f6b52b97640 1 --2- 172.21.15.27:0/3342000859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 0x7f6b4c105120 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009900 tx=0x7f6b3802a410 comp rx=0 tx=0).stop 2024-04-25T13:41:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.483+0000 7f6b52b97640 1 -- 172.21.15.27:0/3342000859 shutdown_connections 2024-04-25T13:41:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.483+0000 7f6b52b97640 1 --2- 172.21.15.27:0/3342000859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b4c109a10 0x7f6b4c10be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.483+0000 7f6b52b97640 1 --2- 172.21.15.27:0/3342000859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 0x7f6b4c105120 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.483+0000 7f6b52b97640 1 --2- 172.21.15.27:0/3342000859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c103ad0 0x7f6b4c1047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.483+0000 7f6b52b97640 1 -- 172.21.15.27:0/3342000859 >> 172.21.15.27:0/3342000859 conn(0x7f6b4c0fecc0 msgr2=0x7f6b4c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.483+0000 7f6b52b97640 1 -- 172.21.15.27:0/3342000859 shutdown_connections 2024-04-25T13:41:58.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.483+0000 7f6b52b97640 1 -- 172.21.15.27:0/3342000859 wait complete. 2024-04-25T13:41:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b52b97640 1 Processor -- start 2024-04-25T13:41:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b52b97640 1 -- start start 2024-04-25T13:41:58.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b52b97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b4c103ad0 0x7f6b4c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b52b97640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 0x7f6b4c19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b50933640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b4c103ad0 0x7f6b4c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b50933640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b4c103ad0 0x7f6b4c19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47912/0 (socket says 172.21.15.27:47912) 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b52b97640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c109a10 0x7f6b4c1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b50933640 1 -- 172.21.15.27:0/1088114898 learned_addr learned my addr 172.21.15.27:0/1088114898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.484+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c10f650 con 0x7f6b4c104d10 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c10f4d0 con 0x7f6b4c103ad0 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b4bfff640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 0x7f6b4c19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c10f7d0 con 0x7f6b4c109a10 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b51134640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c109a10 0x7f6b4c1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b51134640 1 -- 172.21.15.27:0/1088114898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b4c103ad0 msgr2=0x7f6b4c19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b51134640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b4c103ad0 0x7f6b4c19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b51134640 1 -- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 msgr2=0x7f6b4c19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b51134640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 0x7f6b4c19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b51134640 1 -- 172.21.15.27:0/1088114898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4c1a1d80 con 0x7f6b4c109a10 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.485+0000 7f6b50933640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b4c103ad0 0x7f6b4c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.486+0000 7f6b4bfff640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 0x7f6b4c19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.486+0000 7f6b51134640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c109a10 0x7f6b4c1a1710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009fd0 tx=0x7f6b4400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.486+0000 7f6b49ffb640 1 -- 172.21.15.27:0/1088114898 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400ce20 con 0x7f6b4c109a10 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.486+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4c074e30 con 0x7f6b4c109a10 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.486+0000 7f6b49ffb640 1 -- 172.21.15.27:0/1088114898 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b44005020 con 0x7f6b4c109a10 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.486+0000 7f6b49ffb640 1 -- 172.21.15.27:0/1088114898 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44010690 con 0x7f6b4c109a10 2024-04-25T13:41:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.486+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4c075400 con 0x7f6b4c109a10 2024-04-25T13:41:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.488+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b4c076dd0 con 0x7f6b4c109a10 2024-04-25T13:41:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.490+0000 7f6b49ffb640 1 -- 172.21.15.27:0/1088114898 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6b440054d0 con 0x7f6b4c109a10 2024-04-25T13:41:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.491+0000 7f6b49ffb640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b3c0afb70 0x7f6b3c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.491+0000 7f6b49ffb640 1 -- 172.21.15.27:0/1088114898 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6b440df4b0 con 0x7f6b4c109a10 2024-04-25T13:41:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.492+0000 7f6b50933640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b3c0afb70 0x7f6b3c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:58.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.493+0000 7f6b50933640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b3c0afb70 0x7f6b3c0b2080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6b40006fd0 tx=0x7f6b40006de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.495+0000 7f6b49ffb640 1 -- 172.21.15.27:0/1088114898 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b440a8060 con 0x7f6b4c109a10 2024-04-25T13:41:58.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.888+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b4c02db50 con 0x7f6b4c109a10 2024-04-25T13:41:58.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.889+0000 7f6b49ffb640 1 -- 172.21.15.27:0/1088114898 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f6b440a7960 con 0x7f6b4c109a10 2024-04-25T13:41:58.889 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:41:58.889 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:41:58.889 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:41:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b3c0afb70 msgr2=0x7f6b3c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b3c0afb70 0x7f6b3c0b2080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6b40006fd0 tx=0x7f6b40006de0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c109a10 msgr2=0x7f6b4c1a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c109a10 0x7f6b4c1a1710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009fd0 tx=0x7f6b4400ef10 comp rx=0 tx=0).stop 2024-04-25T13:41:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 shutdown_connections 2024-04-25T13:41:58.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b3c0afb70 0x7f6b3c0b2080 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c109a10 0x7f6b4c1a1710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c104d10 0x7f6b4c19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 --2- 172.21.15.27:0/1088114898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b4c103ad0 0x7f6b4c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.892+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 >> 172.21.15.27:0/1088114898 conn(0x7f6b4c0fecc0 msgr2=0x7f6b4c1059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.893+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 shutdown_connections 2024-04-25T13:41:58.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:58.893+0000 7f6b52b97640 1 -- 172.21.15.27:0/1088114898 wait complete. 2024-04-25T13:41:58.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:41:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.081+0000 7f64533f3640 1 -- 172.21.15.27:0/3270763499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f644c103c30 msgr2=0x7f644c104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.081+0000 7f64533f3640 1 --2- 172.21.15.27:0/3270763499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f644c103c30 0x7f644c104040 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f643c009900 tx=0x7f643c02a410 comp rx=0 tx=0).stop 2024-04-25T13:41:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.081+0000 7f64533f3640 1 -- 172.21.15.27:0/3270763499 shutdown_connections 2024-04-25T13:41:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.082+0000 7f64533f3640 1 --2- 172.21.15.27:0/3270763499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f644c108930 0x7f644c10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.082+0000 7f64533f3640 1 --2- 172.21.15.27:0/3270763499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f644c103c30 0x7f644c104040 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.082+0000 7f64533f3640 1 --2- 172.21.15.27:0/3270763499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f644c1029f0 0x7f644c1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.082+0000 7f64533f3640 1 -- 172.21.15.27:0/3270763499 >> 172.21.15.27:0/3270763499 conn(0x7f644c0fdbe0 msgr2=0x7f644c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.082+0000 7f64533f3640 1 -- 172.21.15.27:0/3270763499 shutdown_connections 2024-04-25T13:41:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.082+0000 7f64533f3640 1 -- 172.21.15.27:0/3270763499 wait complete. 2024-04-25T13:41:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.084+0000 7f64533f3640 1 Processor -- start 2024-04-25T13:41:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.084+0000 7f64533f3640 1 -- start start 2024-04-25T13:41:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f64533f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f644c1029f0 0x7f644c19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f64533f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f644c103c30 0x7f644c19ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f64533f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f644c108930 0x7f644c1a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f64533f3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10d460 con 0x7f644c103c30 2024-04-25T13:41:59.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f64533f3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10d2e0 con 0x7f644c1029f0 2024-04-25T13:41:59.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f64533f3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10d5e0 con 0x7f644c108930 2024-04-25T13:41:59.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f6451990640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f644c108930 0x7f644c1a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f6451990640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f644c108930 0x7f644c1a39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33484/0 (socket says 172.21.15.27:33484) 2024-04-25T13:41:59.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f6451990640 1 -- 172.21.15.27:0/898475755 learned_addr learned my addr 172.21.15.27:0/898475755 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:59.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.085+0000 7f645098e640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f644c103c30 0x7f644c19ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.086+0000 7f645118f640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f644c1029f0 0x7f644c19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.086+0000 7f645118f640 1 -- 172.21.15.27:0/898475755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f644c108930 msgr2=0x7f644c1a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.086+0000 7f645118f640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f644c108930 0x7f644c1a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.086+0000 7f645118f640 1 -- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f644c103c30 msgr2=0x7f644c19ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.086+0000 7f645118f640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f644c103c30 0x7f644c19ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.086+0000 7f645118f640 1 -- 172.21.15.27:0/898475755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644c1a4010 con 0x7f644c1029f0 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.086+0000 7f645118f640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f644c1029f0 0x7f644c19c4d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f644c1022c0 tx=0x7f644800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.086+0000 7f64427fc640 1 -- 172.21.15.27:0/898475755 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644800cc40 con 0x7f644c1029f0 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.087+0000 7f64427fc640 1 -- 172.21.15.27:0/898475755 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f644800cde0 con 0x7f644c1029f0 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.087+0000 7f64427fc640 1 -- 172.21.15.27:0/898475755 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6448010430 con 0x7f644c1029f0 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.087+0000 7f64533f3640 1 -- 172.21.15.27:0/898475755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644c1a4300 con 0x7f644c1029f0 2024-04-25T13:41:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.087+0000 7f64533f3640 1 -- 172.21.15.27:0/898475755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f644c1a47d0 con 0x7f644c1029f0 2024-04-25T13:41:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.089+0000 7f64427fc640 1 -- 172.21.15.27:0/898475755 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6448003eb0 con 0x7f644c1029f0 2024-04-25T13:41:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.091+0000 7f645a382740 1 Processor -- start 2024-04-25T13:41:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.091+0000 7f645a382740 1 -- start start 2024-04-25T13:41:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.092+0000 7f64427fc640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64380afb70 0x7f64380b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.092+0000 7f64427fc640 1 -- 172.21.15.27:0/898475755 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f64480df030 con 0x7f644c1029f0 2024-04-25T13:41:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.092+0000 7f6451990640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64380afb70 0x7f64380b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.092+0000 7f645a382740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74b5390 0x5644d7857810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.092+0000 7f645a382740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7857d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.092+0000 7f645a382740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74946b0 0x5644d7858a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645098e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7857d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645098e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7857d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43922/0 (socket says 172.21.15.27:43922) 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645098e640 1 -- 172.21.15.27:0/1300876537 learned_addr learned my addr 172.21.15.27:0/1300876537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645a382740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5644d77bbd80 con 0x5644d748c4d0 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645a382740 1 -- 172.21.15.27:0/1300876537 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5644d7775a20 con 0x5644d74946b0 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645a382740 1 -- 172.21.15.27:0/1300876537 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5644d77b7d40 con 0x5644d74b5390 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645118f640 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74b5390 0x5644d7857810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f6451990640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64380afb70 0x7f64380b2080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f64440049d0 tx=0x7f6444009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f6451990640 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74946b0 0x5644d7858a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645118f640 1 -- 172.21.15.27:0/1300876537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74946b0 msgr2=0x5644d7858a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645118f640 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74946b0 0x5644d7858a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645118f640 1 -- 172.21.15.27:0/1300876537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 msgr2=0x5644d7857d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645118f640 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7857d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645118f640 1 -- 172.21.15.27:0/1300876537 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644d7780c30 con 0x5644d74b5390 2024-04-25T13:41:59.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f645118f640 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74b5390 0x5644d7857810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6448000c00 tx=0x7f64480a7050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f361adccd62ff00b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:59.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f64377fe640 1 -- 172.21.15.27:0/1300876537 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64480c1070 con 0x5644d74b5390 2024-04-25T13:41:59.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f64377fe640 1 -- 172.21.15.27:0/1300876537 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64480a7540 con 0x5644d74b5390 2024-04-25T13:41:59.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.093+0000 7f64377fe640 1 -- 172.21.15.27:0/1300876537 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64480b0150 con 0x5644d74b5390 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 -- 172.21.15.27:0/1300876537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74b5390 msgr2=0x5644d7857810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74b5390 0x5644d7857810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6448000c00 tx=0x7f64480a7050 comp rx=0 tx=0).stop 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 -- 172.21.15.27:0/1300876537 shutdown_connections 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74946b0 0x5644d7858a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7857d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 --2- 172.21.15.27:0/1300876537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74b5390 0x5644d7857810 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 -- 172.21.15.27:0/1300876537 >> 172.21.15.27:0/1300876537 conn(0x5644d74a94e0 msgr2=0x5644d7853020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 -- 172.21.15.27:0/1300876537 shutdown_connections 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.094+0000 7f645a382740 1 -- 172.21.15.27:0/1300876537 wait complete. 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 1 -- start start 2024-04-25T13:41:59.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 1 Processor -- start 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7861650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7861b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74b5390 0x5644d7868af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5644d7859530 con 0x5644d748c4d0 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5644d7742dc0 con 0x5644d74b5390 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5644d78596b0 con 0x5644d74946b0 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645a382740 10 client.0 _ll_register_callbacks cb 0x5644d775e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f6451990640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74b5390 0x5644d7868af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f6451990640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74b5390 0x5644d7868af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47944/0 (socket says 172.21.15.27:47944) 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f645118f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7861650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.095+0000 7f6451990640 1 -- 172.21.15.27:0/3359062830 learned_addr learned my addr 172.21.15.27:0/3359062830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.096+0000 7f645098e640 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7861b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.096+0000 7f6451990640 1 -- 172.21.15.27:0/3359062830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 msgr2=0x5644d7861b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.096+0000 7f6451990640 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7861b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.096+0000 7f6451990640 1 -- 172.21.15.27:0/3359062830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 msgr2=0x5644d7861650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.096+0000 7f6451990640 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7861650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.096+0000 7f6451990640 1 -- 172.21.15.27:0/3359062830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644d74cd3f0 con 0x5644d74b5390 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.097+0000 7f6451990640 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74b5390 0x5644d7868af0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6444012eb0 tx=0x7f6444004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f63346996812f8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.097+0000 7f6434ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.097+0000 7f6434ff9640 1 -- 172.21.15.27:0/3359062830 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644402a680 con 0x5644d74b5390 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.097+0000 7f6434ff9640 1 -- 172.21.15.27:0/3359062830 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f644402a820 con 0x5644d74b5390 2024-04-25T13:41:59.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.097+0000 7f645a382740 1 -- 172.21.15.27:0/3359062830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64480a7540 con 0x5644d74b5390 2024-04-25T13:41:59.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f6434ff9640 1 -- 172.21.15.27:0/3359062830 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6444031070 con 0x5644d74b5390 2024-04-25T13:41:59.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f645a382740 10 client.25250 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:59.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f645a382740 1 -- 172.21.15.27:0/3359062830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5644d75977d0 con 0x5644d74b5390 2024-04-25T13:41:59.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f6434ff9640 1 -- 172.21.15.27:0/3359062830 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f644401f420 con 0x5644d74b5390 2024-04-25T13:41:59.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f6434ff9640 1 -- 172.21.15.27:0/3359062830 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f64440353d0 con 0x5644d74b5390 2024-04-25T13:41:59.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f645a382740 10 client.25250 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f645a382740 10 client.25250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f645a382740 20 client.25250 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f645a382740 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5644d7473f60 0x5644d786f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.098+0000 7f645a382740 4 client.25250 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.099+0000 7f645a382740 1 -- 172.21.15.27:0/3359062830 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f64480a7540 con 0x5644d7473f60 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.099+0000 7f645118f640 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5644d7473f60 0x5644d786f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.099+0000 7f645118f640 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5644d7473f60 0x5644d786f3c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f5c619316bb0b2 server_cookie=47128523057cb49e in_seq=0 out_seq=0 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.100+0000 7f6434ff9640 10 client.25250 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.100+0000 7f6434ff9640 1 -- 172.21.15.27:0/3359062830 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f64480a7540 con 0x5644d7473f60 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.100+0000 7f6434ff9640 10 client.25250 handle_command_reply: tid=0 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.100+0000 7f645a382740 1 client.25250 shutdown 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 -- 172.21.15.27:0/3359062830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74b5390 msgr2=0x5644d7868af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74b5390 0x5644d7868af0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6444012eb0 tx=0x7f6444004ab0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 -- 172.21.15.27:0/3359062830 shutdown_connections 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5644d7473f60 0x5644d786f3c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d74b5390 0x5644d7868af0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7861b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f6434ff9640 0 client.25250 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 --2- 172.21.15.27:0/3359062830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d7861650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 -- 172.21.15.27:0/3359062830 >> 172.21.15.27:0/3359062830 conn(0x5644d74a94e0 msgr2=0x5644d7852040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 -- 172.21.15.27:0/3359062830 shutdown_connections 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 1 -- 172.21.15.27:0/3359062830 wait complete. 2024-04-25T13:41:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.101+0000 7f645a382740 20 client.25250 trim_cache size 0 max 16384 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645a382740 1 Processor -- start 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645a382740 1 -- start start 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645a382740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d7473f60 0x5644d785c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645a382740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d785c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645a382740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7868a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645a382740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f644c1ab550 con 0x5644d748c4d0 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645a382740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f644c1addd0 con 0x5644d7473f60 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645a382740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f644c1abbc0 con 0x5644d74946b0 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645118f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d7473f60 0x5644d785c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645098e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d785c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.131+0000 7f645118f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d7473f60 0x5644d785c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47958/0 (socket says 172.21.15.27:47958) 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.132+0000 7f645118f640 1 -- 172.21.15.27:0/3535344067 learned_addr learned my addr 172.21.15.27:0/3535344067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.132+0000 7f6451990640 1 --2- 172.21.15.27:0/3535344067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7868a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.132+0000 7f645118f640 1 -- 172.21.15.27:0/3535344067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 msgr2=0x5644d7868a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.132+0000 7f645118f640 1 --2- 172.21.15.27:0/3535344067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7868a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.132+0000 7f645118f640 1 -- 172.21.15.27:0/3535344067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 msgr2=0x5644d785c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.132+0000 7f645118f640 1 --2- 172.21.15.27:0/3535344067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d785c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.132+0000 7f645118f640 1 -- 172.21.15.27:0/3535344067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644d7868f50 con 0x5644d7473f60 2024-04-25T13:41:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.132+0000 7f645118f640 1 --2- 172.21.15.27:0/3535344067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d7473f60 0x5644d785c3b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f64480a35c0 tx=0x7f6448090050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c9da9adb367d7c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.133+0000 7f6436ffd640 1 -- 172.21.15.27:0/3535344067 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64480c1040 con 0x5644d7473f60 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.133+0000 7f6436ffd640 1 -- 172.21.15.27:0/3535344067 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64480a37d0 con 0x5644d7473f60 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.133+0000 7f6436ffd640 1 -- 172.21.15.27:0/3535344067 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64480b0170 con 0x5644d7473f60 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 -- 172.21.15.27:0/3535344067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d7473f60 msgr2=0x5644d785c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 --2- 172.21.15.27:0/3535344067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d7473f60 0x5644d785c3b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f64480a35c0 tx=0x7f6448090050 comp rx=0 tx=0).stop 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 -- 172.21.15.27:0/3535344067 shutdown_connections 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 --2- 172.21.15.27:0/3535344067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7868a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 --2- 172.21.15.27:0/3535344067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d748c4d0 0x5644d785c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 --2- 172.21.15.27:0/3535344067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d7473f60 0x5644d785c3b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 -- 172.21.15.27:0/3535344067 >> 172.21.15.27:0/3535344067 conn(0x5644d74a94e0 msgr2=0x5644d785fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 -- 172.21.15.27:0/3535344067 shutdown_connections 2024-04-25T13:41:59.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 -- 172.21.15.27:0/3535344067 wait complete. 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.134+0000 7f645a382740 1 -- start start 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645a382740 1 Processor -- start 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645a382740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d7473f60 0x5644d7865c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645a382740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d748c4d0 0x5644d7866140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645a382740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7867070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645a382740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f644c1ac290 con 0x5644d7473f60 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645a382740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f644c1ad700 con 0x5644d748c4d0 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645a382740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f644c1ac960 con 0x5644d74946b0 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645a382740 10 client.0 _ll_register_callbacks cb 0x7f64440124f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645118f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d7473f60 0x5644d7865c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645118f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d7473f60 0x5644d7865c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43938/0 (socket says 172.21.15.27:43938) 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645118f640 1 -- 172.21.15.27:0/3326491200 learned_addr learned my addr 172.21.15.27:0/3326491200 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:41:59.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645098e640 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d748c4d0 0x5644d7866140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f6451990640 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7867070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645118f640 1 -- 172.21.15.27:0/3326491200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 msgr2=0x5644d7867070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645118f640 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7867070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645118f640 1 -- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d748c4d0 msgr2=0x5644d7866140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645118f640 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d748c4d0 0x5644d7866140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645118f640 1 -- 172.21.15.27:0/3326491200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644d7867630 con 0x5644d7473f60 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.135+0000 7f645098e640 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d748c4d0 0x5644d7866140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f645118f640 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d7473f60 0x5644d7865c00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f644800c6a0 tx=0x7f6448003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490b52d2e8a93b84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f6434ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f6434ff9640 1 -- 172.21.15.27:0/3326491200 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64480c1040 con 0x5644d7473f60 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f6434ff9640 1 -- 172.21.15.27:0/3326491200 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f64480aff10 con 0x5644d7473f60 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f645a382740 1 -- 172.21.15.27:0/3326491200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5644d77c9990 con 0x5644d7473f60 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f6434ff9640 1 -- 172.21.15.27:0/3326491200 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f64480b03a0 con 0x5644d7473f60 2024-04-25T13:41:59.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f645a382740 10 client.15516 fetch_fsmap learned FSMap version 19 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f645a382740 1 -- 172.21.15.27:0/3326491200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5644d78688b0 con 0x5644d7473f60 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f6434ff9640 1 -- 172.21.15.27:0/3326491200 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5644d77c9990 con 0x5644d7473f60 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f6434ff9640 1 -- 172.21.15.27:0/3326491200 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f64480a2eb0 con 0x5644d7473f60 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f645a382740 10 client.15516 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f645a382740 10 client.15516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f645a382740 20 client.15516 populate_metadata read hostname 'smithi027' 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.136+0000 7f645a382740 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5644d74b5390 0x5644d78d9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.137+0000 7f645a382740 4 client.15516 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.137+0000 7f645a382740 1 -- 172.21.15.27:0/3326491200 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644d77c3800 con 0x5644d74b5390 2024-04-25T13:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.137+0000 7f645098e640 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5644d74b5390 0x5644d78d9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:41:59.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.139+0000 7f6434ff9640 10 client.15516 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.139+0000 7f645098e640 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5644d74b5390 0x5644d78d9740 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9ad0f78baf53638 server_cookie=334e98ab8cd89c00 in_seq=0 out_seq=0 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.139+0000 7f6434ff9640 1 -- 172.21.15.27:0/3326491200 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644d77c3800 con 0x5644d74b5390 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.139+0000 7f6434ff9640 10 client.15516 handle_command_reply: tid=0 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.139+0000 7f645a382740 1 client.15516 shutdown 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 -- 172.21.15.27:0/3326491200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d7473f60 msgr2=0x5644d7865c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d7473f60 0x5644d7865c00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f644800c6a0 tx=0x7f6448003910 comp rx=0 tx=0).stop 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 -- 172.21.15.27:0/3326491200 shutdown_connections 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5644d74b5390 0x5644d78d9740 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5644d74946b0 0x5644d7867070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5644d748c4d0 0x5644d7866140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 --2- 172.21.15.27:0/3326491200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5644d7473f60 0x5644d7865c00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f6434ff9640 0 client.15516 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:41:59.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 -- 172.21.15.27:0/3326491200 >> 172.21.15.27:0/3326491200 conn(0x5644d74a94e0 msgr2=0x5644d785fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:59.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 -- 172.21.15.27:0/3326491200 shutdown_connections 2024-04-25T13:41:59.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 1 -- 172.21.15.27:0/3326491200 wait complete. 2024-04-25T13:41:59.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f645a382740 20 client.15516 trim_cache size 0 max 16384 2024-04-25T13:41:59.143 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f64533f3640 1 -- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64380afb70 msgr2=0x7f64380b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f64533f3640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64380afb70 0x7f64380b2080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f64440049d0 tx=0x7f6444009500 comp rx=0 tx=0).stop 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f64533f3640 1 -- 172.21.15.27:0/898475755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f644c1029f0 msgr2=0x7f644c19c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.140+0000 7f64533f3640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f644c1029f0 0x7f644c19c4d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f644c1022c0 tx=0x7f644800ee50 comp rx=0 tx=0).stop 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.141+0000 7f64533f3640 1 -- 172.21.15.27:0/898475755 shutdown_connections 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.141+0000 7f64533f3640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f64380afb70 0x7f64380b2080 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.141+0000 7f64533f3640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f644c108930 0x7f644c1a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.141+0000 7f64533f3640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f644c103c30 0x7f644c19ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.141+0000 7f64533f3640 1 --2- 172.21.15.27:0/898475755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f644c1029f0 0x7f644c19c4d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.141+0000 7f64533f3640 1 -- 172.21.15.27:0/898475755 >> 172.21.15.27:0/898475755 conn(0x7f644c0fdbe0 msgr2=0x7f644c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.141+0000 7f64533f3640 1 -- 172.21.15.27:0/898475755 shutdown_connections 2024-04-25T13:41:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:41:59.141+0000 7f64533f3640 1 -- 172.21.15.27:0/898475755 wait complete. 2024-04-25T13:41:59.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:41:59 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1088114898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:41:59.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:41:59 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1088114898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:41:59.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:41:59 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1088114898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:42:00.158 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:42:00.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:42:00.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:42:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:00 smithi122 ceph-mon[32843]: pgmap v382: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.4 MiB/s wr, 627 op/s 2024-04-25T13:42:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:00 smithi045 ceph-mon[32731]: pgmap v382: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.4 MiB/s wr, 627 op/s 2024-04-25T13:42:00.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 -- 172.21.15.27:0/890473149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff660103c20 msgr2=0x7ff660104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 --2- 172.21.15.27:0/890473149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff660103c20 0x7ff660104030 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009860 tx=0x7ff65c02a340 comp rx=0 tx=0).stop 2024-04-25T13:42:00.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 -- 172.21.15.27:0/890473149 shutdown_connections 2024-04-25T13:42:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 --2- 172.21.15.27:0/890473149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff660108920 0x7ff66010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 --2- 172.21.15.27:0/890473149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff660103c20 0x7ff660104030 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 --2- 172.21.15.27:0/890473149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6601029e0 0x7ff6601036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 -- 172.21.15.27:0/890473149 >> 172.21.15.27:0/890473149 conn(0x7ff6600fdbd0 msgr2=0x7ff660100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 -- 172.21.15.27:0/890473149 shutdown_connections 2024-04-25T13:42:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.329+0000 7ff667c42640 1 -- 172.21.15.27:0/890473149 wait complete. 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff667c42640 1 Processor -- start 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff667c42640 1 -- start start 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff667c42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6601029e0 0x7ff66010e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff667c42640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff660103c20 0x7ff66010ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff667c42640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff660108920 0x7ff66010d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff667c42640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010f620 con 0x7ff660103c20 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff6651dd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff660103c20 0x7ff66010ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff667c42640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010f4a0 con 0x7ff6601029e0 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff667c42640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010f7a0 con 0x7ff660108920 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff6651dd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff660103c20 0x7ff66010ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43958/0 (socket says 172.21.15.27:43958) 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.330+0000 7ff6651dd640 1 -- 172.21.15.27:0/1409760426 learned_addr learned my addr 172.21.15.27:0/1409760426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff6659de640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6601029e0 0x7ff66010e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff6661df640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff660108920 0x7ff66010d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff6651dd640 1 -- 172.21.15.27:0/1409760426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff660108920 msgr2=0x7ff66010d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff6651dd640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff660108920 0x7ff66010d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff6651dd640 1 -- 172.21.15.27:0/1409760426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6601029e0 msgr2=0x7ff66010e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff6651dd640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6601029e0 0x7ff66010e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff6651dd640 1 -- 172.21.15.27:0/1409760426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66010dae0 con 0x7ff660103c20 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff6651dd640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff660103c20 0x7ff66010ce70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff65c032d50 tx=0x7ff65c032d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff656ffd640 1 -- 172.21.15.27:0/1409760426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c037040 con 0x7ff660103c20 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff667c42640 1 -- 172.21.15.27:0/1409760426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6601a4600 con 0x7ff660103c20 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff656ffd640 1 -- 172.21.15.27:0/1409760426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff65c030c10 con 0x7ff660103c20 2024-04-25T13:42:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.331+0000 7ff667c42640 1 -- 172.21.15.27:0/1409760426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6601a49b0 con 0x7ff660103c20 2024-04-25T13:42:00.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.333+0000 7ff656ffd640 1 -- 172.21.15.27:0/1409760426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c0037f0 con 0x7ff660103c20 2024-04-25T13:42:00.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.335+0000 7ff66ebc7740 1 Processor -- start 2024-04-25T13:42:00.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff66ebc7740 1 -- start start 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 0x564bb47ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43ff730 0x564bb47d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff66ebc7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564bb4726b00 con 0x564bb4420410 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff66ebc7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564bb46c2920 con 0x564bb43f7550 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff66ebc7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564bb4722ac0 con 0x564bb43ff730 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff6661df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff6659de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 0x564bb47ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff6661df640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43962/0 (socket says 172.21.15.27:43962) 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.336+0000 7ff6661df640 1 -- 172.21.15.27:0/3648628322 learned_addr learned my addr 172.21.15.27:0/3648628322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.337+0000 7ff6661df640 1 -- 172.21.15.27:0/3648628322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43ff730 msgr2=0x564bb47d1130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:42:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.337+0000 7ff6661df640 1 --2- 172.21.15.27:0/3648628322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43ff730 0x564bb47d1130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.337+0000 7ff6661df640 1 -- 172.21.15.27:0/3648628322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 msgr2=0x564bb47ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.337+0000 7ff6661df640 1 --2- 172.21.15.27:0/3648628322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 0x564bb47ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.337+0000 7ff6661df640 1 -- 172.21.15.27:0/3648628322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bb45007e0 con 0x564bb4420410 2024-04-25T13:42:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.337+0000 7ff6661df640 1 --2- 172.21.15.27:0/3648628322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47c9cb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff650009c40 tx=0x7ff650011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9258a2efab754a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.337+0000 7ff656ffd640 1 -- 172.21.15.27:0/1409760426 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff65c043030 con 0x7ff660103c20 2024-04-25T13:42:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.338+0000 7ff64bfff640 1 -- 172.21.15.27:0/3648628322 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650004230 con 0x564bb4420410 2024-04-25T13:42:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.338+0000 7ff64bfff640 1 -- 172.21.15.27:0/3648628322 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6500043d0 con 0x564bb4420410 2024-04-25T13:42:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.338+0000 7ff64bfff640 1 -- 172.21.15.27:0/3648628322 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650019820 con 0x564bb4420410 2024-04-25T13:42:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.338+0000 7ff656ffd640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff64c0afb70 0x7ff64c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.339+0000 7ff656ffd640 1 -- 172.21.15.27:0/1409760426 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff65c0fe360 con 0x7ff660103c20 2024-04-25T13:42:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.339+0000 7ff6659de640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff64c0afb70 0x7ff64c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 -- 172.21.15.27:0/3648628322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 msgr2=0x564bb47c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/3648628322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47c9cb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff650009c40 tx=0x7ff650011810 comp rx=0 tx=0).stop 2024-04-25T13:42:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 -- 172.21.15.27:0/3648628322 shutdown_connections 2024-04-25T13:42:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/3648628322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43ff730 0x564bb47d1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/3648628322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 0x564bb47ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/3648628322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47c9cb0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 -- 172.21.15.27:0/3648628322 >> 172.21.15.27:0/3648628322 conn(0x564bb4414560 msgr2=0x564bb47c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 -- 172.21.15.27:0/3648628322 shutdown_connections 2024-04-25T13:42:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 -- 172.21.15.27:0/3648628322 wait complete. 2024-04-25T13:42:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff6659de640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff64c0afb70 0x7ff64c0b2080 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x564bb47c63d0 tx=0x7ff658009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.340+0000 7ff66ebc7740 1 -- start start 2024-04-25T13:42:00.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff66ebc7740 1 Processor -- start 2024-04-25T13:42:00.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 0x564bb47d3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 0x564bb47d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb4420410 0x564bb47db140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff66ebc7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564bb47d1b80 con 0x564bb43ff730 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff66ebc7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564bb46c05d0 con 0x564bb43f7550 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff66ebc7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564bb47d1d00 con 0x564bb4420410 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff66ebc7740 10 client.0 _ll_register_callbacks cb 0x564bb46ae080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff6661df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb4420410 0x564bb47db140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.341+0000 7ff6661df640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb4420410 0x564bb47db140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33538/0 (socket says 172.21.15.27:33538) 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6661df640 1 -- 172.21.15.27:0/1289142447 learned_addr learned my addr 172.21.15.27:0/1289142447 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6659de640 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 0x564bb47d3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6651dd640 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 0x564bb47d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6661df640 1 -- 172.21.15.27:0/1289142447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 msgr2=0x564bb47d3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6661df640 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 0x564bb47d3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6661df640 1 -- 172.21.15.27:0/1289142447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 msgr2=0x564bb47d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6661df640 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 0x564bb47d41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6661df640 1 -- 172.21.15.27:0/1289142447 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bb4560f70 con 0x564bb4420410 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6661df640 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb4420410 0x564bb47db140 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff65001b040 tx=0x7ff650003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e63a7ea0fbb8bbd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6497fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6497fa640 1 -- 172.21.15.27:0/1289142447 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650028070 con 0x564bb4420410 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff66ebc7740 1 -- 172.21.15.27:0/1289142447 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6500043d0 con 0x564bb4420410 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.342+0000 7ff6497fa640 1 -- 172.21.15.27:0/1289142447 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff650005040 con 0x564bb4420410 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff6497fa640 1 -- 172.21.15.27:0/1289142447 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff650034050 con 0x564bb4420410 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff66ebc7740 10 client.25264 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff66ebc7740 1 -- 172.21.15.27:0/1289142447 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564bb43f8170 con 0x564bb4420410 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff6497fa640 1 -- 172.21.15.27:0/1289142447 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6500043d0 con 0x564bb4420410 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff6497fa640 1 -- 172.21.15.27:0/1289142447 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff650004b20 con 0x564bb4420410 2024-04-25T13:42:00.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff66ebc7740 10 client.25264 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:00.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff66ebc7740 10 client.25264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:00.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff66ebc7740 20 client.25264 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:00.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564bb43df1f0 0x564bb47e13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff66ebc7740 4 client.25264 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:00.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff66ebc7740 1 -- 172.21.15.27:0/1289142447 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bb4734710 con 0x564bb43df1f0 2024-04-25T13:42:00.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.343+0000 7ff6659de640 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564bb43df1f0 0x564bb47e13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.344+0000 7ff6659de640 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564bb43df1f0 0x564bb47e13a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e963c38d83f4690 server_cookie=6476260d7fa197c in_seq=0 out_seq=0 2024-04-25T13:42:00.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.344+0000 7ff6497fa640 10 client.25264 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:00.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff6497fa640 1 -- 172.21.15.27:0/1289142447 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564bb4734710 con 0x564bb43df1f0 2024-04-25T13:42:00.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff6497fa640 10 client.25264 handle_command_reply: tid=0 2024-04-25T13:42:00.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 client.25264 shutdown 2024-04-25T13:42:00.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 -- 172.21.15.27:0/1289142447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb4420410 msgr2=0x564bb47db140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb4420410 0x564bb47db140 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff65001b040 tx=0x7ff650003820 comp rx=0 tx=0).stop 2024-04-25T13:42:00.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 -- 172.21.15.27:0/1289142447 shutdown_connections 2024-04-25T13:42:00.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564bb43df1f0 0x564bb47e13a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb4420410 0x564bb47db140 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 0x564bb47d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/1289142447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43f7550 0x564bb47d3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff6497fa640 0 client.25264 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:00.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 -- 172.21.15.27:0/1289142447 >> 172.21.15.27:0/1289142447 conn(0x564bb4414560 msgr2=0x564bb47bbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 -- 172.21.15.27:0/1289142447 shutdown_connections 2024-04-25T13:42:00.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 1 -- 172.21.15.27:0/1289142447 wait complete. 2024-04-25T13:42:00.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.345+0000 7ff66ebc7740 20 client.25264 trim_cache size 0 max 16384 2024-04-25T13:42:00.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.370+0000 7ff66ebc7740 1 Processor -- start 2024-04-25T13:42:00.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff66ebc7740 1 -- start start 2024-04-25T13:42:00.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43ff730 0x564bb47db060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47db5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff6659de640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff6659de640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33550/0 (socket says 172.21.15.27:33550) 2024-04-25T13:42:00.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff6659de640 1 -- 172.21.15.27:0/2620532746 learned_addr learned my addr 172.21.15.27:0/2620532746 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2620532746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a6fd0 con 0x564bb4420410 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2620532746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a9850 con 0x564bb43ff730 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.371+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2620532746 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a7640 con 0x564bb43f7550 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6651dd640 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43ff730 0x564bb47db060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6661df640 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47db5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6659de640 1 -- 172.21.15.27:0/2620532746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43ff730 msgr2=0x564bb47db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6659de640 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43ff730 0x564bb47db060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6659de640 1 -- 172.21.15.27:0/2620532746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 msgr2=0x564bb47db5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6659de640 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47db5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6659de640 1 -- 172.21.15.27:0/2620532746 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bb47c61d0 con 0x564bb43f7550 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6651dd640 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43ff730 0x564bb47db060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.372+0000 7ff6659de640 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d29d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff6601035e0 tx=0x7ff65801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6037cd255e1ce6a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.373+0000 7ff64b7fe640 1 -- 172.21.15.27:0/2620532746 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658004780 con 0x564bb43f7550 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.373+0000 7ff64b7fe640 1 -- 172.21.15.27:0/2620532746 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff658030070 con 0x564bb43f7550 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.373+0000 7ff64b7fe640 1 -- 172.21.15.27:0/2620532746 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65802a990 con 0x564bb43f7550 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2620532746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 msgr2=0x564bb47d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d29d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff6601035e0 tx=0x7ff65801a000 comp rx=0 tx=0).stop 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2620532746 shutdown_connections 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb4420410 0x564bb47db5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb43ff730 0x564bb47db060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2620532746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d29d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2620532746 >> 172.21.15.27:0/2620532746 conn(0x564bb4414560 msgr2=0x564bb47d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2620532746 shutdown_connections 2024-04-25T13:42:00.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2620532746 wait complete. 2024-04-25T13:42:00.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 -- start start 2024-04-25T13:42:00.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.374+0000 7ff66ebc7740 1 Processor -- start 2024-04-25T13:42:00.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.375+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.375+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 0x564bb47d92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.375+0000 7ff66ebc7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb4420410 0x564bb47da210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.375+0000 7ff6659de640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6651dd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 0x564bb47d92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6659de640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33562/0 (socket says 172.21.15.27:33562) 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6659de640 1 -- 172.21.15.27:0/2760570173 learned_addr learned my addr 172.21.15.27:0/2760570173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a7d10 con 0x564bb43ff730 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a9180 con 0x564bb4420410 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a83e0 con 0x564bb43f7550 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6661df640 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb4420410 0x564bb47da210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff66ebc7740 10 client.0 _ll_register_callbacks cb 0x7ff650009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6659de640 1 -- 172.21.15.27:0/2760570173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb4420410 msgr2=0x564bb47da210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6659de640 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb4420410 0x564bb47da210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6659de640 1 -- 172.21.15.27:0/2760570173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 msgr2=0x564bb47d92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6659de640 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 0x564bb47d92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6659de640 1 -- 172.21.15.27:0/2760570173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bb47da7d0 con 0x564bb43f7550 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6659de640 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d8da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff658019040 tx=0x7ff658004780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=783b19dc7f5312cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.376+0000 7ff6497fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.377+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff658030070 con 0x564bb43f7550 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.377+0000 7ff6497fa640 1 -- 172.21.15.27:0/2760570173 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658039020 con 0x564bb43f7550 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.377+0000 7ff6497fa640 1 -- 172.21.15.27:0/2760570173 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff658004c40 con 0x564bb43f7550 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.377+0000 7ff6497fa640 1 -- 172.21.15.27:0/2760570173 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff658004e40 con 0x564bb43f7550 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.377+0000 7ff66ebc7740 10 client.25270 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.377+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x564bb47db990 con 0x564bb43f7550 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.378+0000 7ff6497fa640 1 -- 172.21.15.27:0/2760570173 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff65802b3b0 con 0x564bb43f7550 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.378+0000 7ff66ebc7740 10 client.25270 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.378+0000 7ff66ebc7740 10 client.25270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.378+0000 7ff66ebc7740 20 client.25270 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.378+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564bb43b4630 0x564bb484b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.378+0000 7ff66ebc7740 4 client.25270 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.378+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564bb472e580 con 0x564bb43b4630 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.378+0000 7ff6651dd640 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564bb43b4630 0x564bb484b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.379+0000 7ff6497fa640 10 client.25270 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.379+0000 7ff6651dd640 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564bb43b4630 0x564bb484b9d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89773306bb936bed server_cookie=80b11c86d4cd61ed in_seq=0 out_seq=0 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.380+0000 7ff6497fa640 1 -- 172.21.15.27:0/2760570173 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564bb472e580 con 0x564bb43b4630 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.380+0000 7ff6497fa640 10 client.25270 handle_command_reply: tid=0 2024-04-25T13:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.380+0000 7ff66ebc7740 1 client.25270 shutdown 2024-04-25T13:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.380+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 msgr2=0x564bb47d8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.380+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d8da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff658019040 tx=0x7ff658004780 comp rx=0 tx=0).stop 2024-04-25T13:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.380+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 shutdown_connections 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.380+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564bb43b4630 0x564bb484b9d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.381+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bb4420410 0x564bb47da210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.381+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564bb43ff730 0x564bb47d92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.381+0000 7ff66ebc7740 1 --2- 172.21.15.27:0/2760570173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564bb43f7550 0x564bb47d8da0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.381+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 >> 172.21.15.27:0/2760570173 conn(0x564bb4414560 msgr2=0x564bb47c5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.381+0000 7ff6497fa640 0 client.25270 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.381+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 shutdown_connections 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.381+0000 7ff66ebc7740 1 -- 172.21.15.27:0/2760570173 wait complete. 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.381+0000 7ff66ebc7740 20 client.25270 trim_cache size 0 max 16384 2024-04-25T13:42:00.382 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:42:00.383 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:42:00.383 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "0937571e-6556-4665-bf62-23daa6f9264a", 2024-04-25T13:42:00.383 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:42:00.383 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 -- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff64c0afb70 msgr2=0x7ff64c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff64c0afb70 0x7ff64c0b2080 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x564bb47c63d0 tx=0x7ff658009500 comp rx=0 tx=0).stop 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 -- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff660103c20 msgr2=0x7ff66010ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff660103c20 0x7ff66010ce70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff65c032d50 tx=0x7ff65c032d80 comp rx=0 tx=0).stop 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 -- 172.21.15.27:0/1409760426 shutdown_connections 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff64c0afb70 0x7ff64c0b2080 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff660108920 0x7ff66010d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff660103c20 0x7ff66010ce70 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 --2- 172.21.15.27:0/1409760426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6601029e0 0x7ff66010e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 -- 172.21.15.27:0/1409760426 >> 172.21.15.27:0/1409760426 conn(0x7ff6600fdbd0 msgr2=0x7ff660104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 -- 172.21.15.27:0/1409760426 shutdown_connections 2024-04-25T13:42:00.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.382+0000 7ff667c42640 1 -- 172.21.15.27:0/1409760426 wait complete. 2024-04-25T13:42:00.395 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:00 smithi027 ceph-mon[29140]: pgmap v382: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.4 MiB/s wr, 627 op/s 2024-04-25T13:42:00.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:42:00.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 -- 172.21.15.27:0/701666868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 msgr2=0x7fbb30103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 --2- 172.21.15.27:0/701666868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 0x7fbb30103a70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009cd0 tx=0x7fbb2802a8e0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 -- 172.21.15.27:0/701666868 shutdown_connections 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 --2- 172.21.15.27:0/701666868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 0x7fbb30103a70 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 --2- 172.21.15.27:0/701666868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb30102d10 0x7fbb30103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 --2- 172.21.15.27:0/701666868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb301023c0 0x7fbb301027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 -- 172.21.15.27:0/701666868 >> 172.21.15.27:0/701666868 conn(0x7fbb300fdbd0 msgr2=0x7fbb30100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 -- 172.21.15.27:0/701666868 shutdown_connections 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.574+0000 7fbb35413640 1 -- 172.21.15.27:0/701666868 wait complete. 2024-04-25T13:42:00.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb35413640 1 Processor -- start 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb35413640 1 -- start start 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb35413640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb301023c0 0x7fbb3019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb35413640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb30102d10 0x7fbb3019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb35413640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 0x7fbb301a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb35413640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30111820 con 0x7fbb30103660 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb35413640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301116a0 con 0x7fbb30102d10 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb35413640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301119a0 con 0x7fbb301023c0 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 0x7fbb301a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb2effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb301023c0 0x7fbb3019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb2e7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb30102d10 0x7fbb3019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb2effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb301023c0 0x7fbb3019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33586/0 (socket says 172.21.15.27:33586) 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.575+0000 7fbb2effd640 1 -- 172.21.15.27:0/2294096549 learned_addr learned my addr 172.21.15.27:0/2294096549 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb2effd640 1 -- 172.21.15.27:0/2294096549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb30102d10 msgr2=0x7fbb3019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb2effd640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb30102d10 0x7fbb3019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb2effd640 1 -- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 msgr2=0x7fbb301a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb2effd640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 0x7fbb301a3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb2effd640 1 -- 172.21.15.27:0/2294096549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb301a3f70 con 0x7fbb301023c0 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 0x7fbb301a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb2effd640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb301023c0 0x7fbb3019c430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400e9c0 tx=0x7fbb2400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb17fff640 1 -- 172.21.15.27:0/2294096549 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2400cd80 con 0x7fbb301023c0 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb17fff640 1 -- 172.21.15.27:0/2294096549 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb240054d0 con 0x7fbb301023c0 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb35413640 1 -- 172.21.15.27:0/2294096549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb301a4260 con 0x7fbb301023c0 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb17fff640 1 -- 172.21.15.27:0/2294096549 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24010430 con 0x7fbb301023c0 2024-04-25T13:42:00.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.576+0000 7fbb35413640 1 -- 172.21.15.27:0/2294096549 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb301a47b0 con 0x7fbb301023c0 2024-04-25T13:42:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb3c3a0740 1 Processor -- start 2024-04-25T13:42:00.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb3c3a0740 1 -- start start 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 0x558855cdcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cddc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558855c41d80 con 0x55885593b390 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558855bfb8b0 con 0x5588559124d0 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558855c3dd40 con 0x55885591a6b0 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb2e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb2e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cddc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33598/0 (socket says 172.21.15.27:33598) 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb2e7fc640 1 -- 172.21.15.27:0/3575638372 learned_addr learned my addr 172.21.15.27:0/3575638372 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb17fff640 1 -- 172.21.15.27:0/2294096549 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbb240105d0 con 0x7fbb301023c0 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.580+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/3575638372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.581+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/3575638372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 msgr2=0x558855cddc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.581+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/3575638372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cddc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.581+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/3575638372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 msgr2=0x558855cdcfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:42:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.581+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/3575638372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 0x558855cdcfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.581+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/3575638372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558855bcca40 con 0x55885593b390 2024-04-25T13:42:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.581+0000 7fbb17fff640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb180afaf0 0x7fbb180b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.581+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/3575638372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdca60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009ca0 tx=0x7fbb28031d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7223685d1ca57f34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.582+0000 7fbb17fff640 1 -- 172.21.15.27:0/2294096549 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbb240deff0 con 0x7fbb301023c0 2024-04-25T13:42:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.582+0000 7fbb2effd640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb180afaf0 0x7fbb180b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.582+0000 7fbb157fa640 1 -- 172.21.15.27:0/3575638372 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28048070 con 0x55885593b390 2024-04-25T13:42:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.582+0000 7fbb157fa640 1 -- 172.21.15.27:0/3575638372 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb28036070 con 0x55885593b390 2024-04-25T13:42:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.582+0000 7fbb157fa640 1 -- 172.21.15.27:0/3575638372 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28004780 con 0x55885593b390 2024-04-25T13:42:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb2effd640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb180afaf0 0x7fbb180b2000 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbb240058c0 tx=0x7fbb24014000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/3575638372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 msgr2=0x558855cdca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/3575638372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdca60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009ca0 tx=0x7fbb28031d00 comp rx=0 tx=0).stop 2024-04-25T13:42:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/3575638372 shutdown_connections 2024-04-25T13:42:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/3575638372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cddc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/3575638372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 0x558855cdcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/3575638372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdca60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/3575638372 >> 172.21.15.27:0/3575638372 conn(0x55885592f4e0 msgr2=0x558855cd8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/3575638372 shutdown_connections 2024-04-25T13:42:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.584+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/3575638372 wait complete. 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 1 -- start start 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 1 Processor -- start 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5588559124d0 0x558855ce6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885591a6b0 0x558855ce6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 0x558855cedeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558855cde720 con 0x55885591a6b0 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558855bc8d10 con 0x55885593b390 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558855cde8a0 con 0x5588559124d0 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb3c3a0740 10 client.0 _ll_register_callbacks cb 0x558855bf5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb2e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885591a6b0 0x558855ce6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb2f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5588559124d0 0x558855ce6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb2f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5588559124d0 0x558855ce6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33602/0 (socket says 172.21.15.27:33602) 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.585+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/2277879312 learned_addr learned my addr 172.21.15.27:0/2277879312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 0x558855cedeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/2277879312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 msgr2=0x558855cedeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 0x558855cedeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/2277879312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885591a6b0 msgr2=0x558855ce6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885591a6b0 0x558855ce6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/2277879312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558855a1d7d0 con 0x5588559124d0 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885591a6b0 0x558855ce6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5588559124d0 0x558855ce6a10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb28004af0 tx=0x7fbb280095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd3d9be7ba99985e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.586+0000 7fbb12ff5640 1 -- 172.21.15.27:0/2277879312 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28057060 con 0x5588559124d0 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb12ff5640 1 -- 172.21.15.27:0/2277879312 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbb280305a0 con 0x5588559124d0 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/2277879312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb28036070 con 0x5588559124d0 2024-04-25T13:42:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb12ff5640 1 -- 172.21.15.27:0/2277879312 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fbb28048070 con 0x5588559124d0 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 10 client.25282 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/2277879312 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5588559533f0 con 0x5588559124d0 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb12ff5640 1 -- 172.21.15.27:0/2277879312 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb28036070 con 0x5588559124d0 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb12ff5640 1 -- 172.21.15.27:0/2277879312 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fbb28042030 con 0x5588559124d0 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 10 client.25282 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 10 client.25282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 20 client.25282 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5588558f9f60 0x558855cf3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 4 client.25282 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/2277879312 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558855c4f990 con 0x5588558f9f60 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.587+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5588558f9f60 0x558855cf3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.588+0000 7fbb12ff5640 10 client.25282 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.588+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5588558f9f60 0x558855cf3f70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8598b50fc313de server_cookie=5d31695e39353702 in_seq=0 out_seq=0 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb12ff5640 1 -- 172.21.15.27:0/2277879312 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558855c4f990 con 0x5588558f9f60 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb12ff5640 10 client.25282 handle_command_reply: tid=0 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 client.25282 shutdown 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/2277879312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5588559124d0 msgr2=0x558855ce6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5588559124d0 0x558855ce6a10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb28004af0 tx=0x7fbb280095c0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/2277879312 shutdown_connections 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5588558f9f60 0x558855cf3f70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 0x558855cedeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885591a6b0 0x558855ce6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb12ff5640 0 client.25282 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/2277879312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5588559124d0 0x558855ce6a10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/2277879312 >> 172.21.15.27:0/2277879312 conn(0x55885592f4e0 msgr2=0x558855cd73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/2277879312 shutdown_connections 2024-04-25T13:42:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/2277879312 wait complete. 2024-04-25T13:42:00.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.589+0000 7fbb3c3a0740 20 client.25282 trim_cache size 0 max 16384 2024-04-25T13:42:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.615+0000 7fbb3c3a0740 1 Processor -- start 2024-04-25T13:42:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.615+0000 7fbb3c3a0740 1 -- start start 2024-04-25T13:42:00.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 0x558855ce5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cdb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301ab350 con 0x55885593b390 2024-04-25T13:42:00.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301adbd0 con 0x5588559124d0 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301ab9c0 con 0x55885591a6b0 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb2e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cdb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb2f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 0x558855ce5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb2e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cdb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33606/0 (socket says 172.21.15.27:33606) 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb2e7fc640 1 -- 172.21.15.27:0/4047792604 learned_addr learned my addr 172.21.15.27:0/4047792604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.616+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb2e7fc640 1 -- 172.21.15.27:0/4047792604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 msgr2=0x558855ce5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 0x558855ce5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb2e7fc640 1 -- 172.21.15.27:0/4047792604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 msgr2=0x558855cdedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb2e7fc640 1 -- 172.21.15.27:0/4047792604 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588559130f0 con 0x55885591a6b0 2024-04-25T13:42:00.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 0x558855ce5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cdb830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009fd0 tx=0x7fbb20005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dd8dd4679ec6054 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb14ff9640 1 -- 172.21.15.27:0/4047792604 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20024070 con 0x55885591a6b0 2024-04-25T13:42:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb14ff9640 1 -- 172.21.15.27:0/4047792604 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb200047a0 con 0x55885591a6b0 2024-04-25T13:42:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.617+0000 7fbb14ff9640 1 -- 172.21.15.27:0/4047792604 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20020350 con 0x55885591a6b0 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/4047792604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 msgr2=0x558855cdb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cdb830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009fd0 tx=0x7fbb20005f90 comp rx=0 tx=0).stop 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/4047792604 shutdown_connections 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55885593b390 0x558855cdedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855cdb830 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/4047792604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5588559124d0 0x558855ce5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/4047792604 >> 172.21.15.27:0/4047792604 conn(0x55885592f4e0 msgr2=0x558855cdb120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/4047792604 shutdown_connections 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/4047792604 wait complete. 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 -- start start 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.619+0000 7fbb3c3a0740 1 Processor -- start 2024-04-25T13:42:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5588559124d0 0x558855ced440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855ce9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb3c3a0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 0x558855cea540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301ac090 con 0x5588559124d0 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301ad500 con 0x55885593b390 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb3c3a0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301ac760 con 0x55885591a6b0 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb3c3a0740 10 client.0 _ll_register_callbacks cb 0x7fbb28046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5588559124d0 0x558855ced440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5588559124d0 0x558855ced440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44032/0 (socket says 172.21.15.27:44032) 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/885695277 learned_addr learned my addr 172.21.15.27:0/885695277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855ce9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 0x558855cea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/885695277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 msgr2=0x558855ce9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855ce9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/885695277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 msgr2=0x558855cea540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 0x558855cea540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 -- 172.21.15.27:0/885695277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558855c024b0 con 0x5588559124d0 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb2f7fe640 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5588559124d0 0x558855ced440 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbb2802ae00 tx=0x7fbb280095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b86a37edade31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.620+0000 7fbb12ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.621+0000 7fbb12ff5640 1 -- 172.21.15.27:0/885695277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28057040 con 0x5588559124d0 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.621+0000 7fbb12ff5640 1 -- 172.21.15.27:0/885695277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbb280309e0 con 0x5588559124d0 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.621+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/885695277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb200047a0 con 0x5588559124d0 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.621+0000 7fbb12ff5640 1 -- 172.21.15.27:0/885695277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fbb28003c40 con 0x5588559124d0 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.621+0000 7fbb3c3a0740 10 client.15552 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.621+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/885695277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x558855d5b140 con 0x5588559124d0 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.621+0000 7fbb12ff5640 1 -- 172.21.15.27:0/885695277 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb200047a0 con 0x5588559124d0 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.622+0000 7fbb12ff5640 1 -- 172.21.15.27:0/885695277 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fbb28066030 con 0x5588559124d0 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.622+0000 7fbb3c3a0740 10 client.15552 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.622+0000 7fbb3c3a0740 10 client.15552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.622+0000 7fbb3c3a0740 20 client.15552 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.622+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5588558cf3a0 0x558855d5db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.622+0000 7fbb3c3a0740 4 client.15552 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.622+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/885695277 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558855c49800 con 0x5588558cf3a0 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.622+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5588558cf3a0 0x558855d5db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb2e7fc640 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5588558cf3a0 0x558855d5db60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c650f014be427a7 server_cookie=7c15724ed7b33f5c in_seq=0 out_seq=0 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb12ff5640 10 client.15552 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb12ff5640 1 -- 172.21.15.27:0/885695277 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558855c49800 con 0x5588558cf3a0 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb12ff5640 10 client.15552 handle_command_reply: tid=0 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 client.15552 shutdown 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/885695277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5588559124d0 msgr2=0x558855ced440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5588559124d0 0x558855ced440 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fbb2802ae00 tx=0x7fbb280095c0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/885695277 shutdown_connections 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5588558cf3a0 0x558855d5db60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb12ff5640 0 client.15552 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55885593b390 0x558855cea540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55885591a6b0 0x558855ce9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 --2- 172.21.15.27:0/885695277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5588559124d0 0x558855ced440 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.623+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/885695277 >> 172.21.15.27:0/885695277 conn(0x55885592f4e0 msgr2=0x558855cdb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/885695277 shutdown_connections 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb3c3a0740 1 -- 172.21.15.27:0/885695277 wait complete. 2024-04-25T13:42:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb3c3a0740 20 client.15552 trim_cache size 0 max 16384 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: "0937571e-6556-4665-bf62-23daa6f9264a": { 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "0937571e-6556-4665-bf62-23daa6f9264a", 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:42:00.628 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:42:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 -- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb180afaf0 msgr2=0x7fbb180b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb180afaf0 0x7fbb180b2000 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbb240058c0 tx=0x7fbb24014000 comp rx=0 tx=0).stop 2024-04-25T13:42:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 -- 172.21.15.27:0/2294096549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb301023c0 msgr2=0x7fbb3019c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb301023c0 0x7fbb3019c430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400e9c0 tx=0x7fbb2400ee80 comp rx=0 tx=0).stop 2024-04-25T13:42:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 -- 172.21.15.27:0/2294096549 shutdown_connections 2024-04-25T13:42:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb180afaf0 0x7fbb180b2000 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb30103660 0x7fbb301a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb30102d10 0x7fbb3019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 --2- 172.21.15.27:0/2294096549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb301023c0 0x7fbb3019c430 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 -- 172.21.15.27:0/2294096549 >> 172.21.15.27:0/2294096549 conn(0x7fbb300fdbd0 msgr2=0x7fbb300ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 -- 172.21.15.27:0/2294096549 shutdown_connections 2024-04-25T13:42:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:00.624+0000 7fbb35413640 1 -- 172.21.15.27:0/2294096549 wait complete. 2024-04-25T13:42:00.639 INFO:tasks.cephfs.filesystem:scrub status for tag:0937571e-6556-4665-bf62-23daa6f9264a - {'path': '/', 'tag': '0937571e-6556-4665-bf62-23daa6f9264a', 'options': 'recursive,force'} 2024-04-25T13:42:01.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:01 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:42:01.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:01 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:01.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:01 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:42:01.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:01 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:01.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:01 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:42:01.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:01 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:01.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:01 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:42:01.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:01 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:01.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:01 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:42:01.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:01 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:01.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:01 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:42:01.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:01 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.422+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1056992926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ec8102a20 msgr2=0x7f4ec8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1056992926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ec8102a20 0x7f4ec8102ef0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009900 tx=0x7f4eb802a410 comp rx=0 tx=0).stop 2024-04-25T13:42:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1056992926 shutdown_connections 2024-04-25T13:42:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1056992926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8103580 0x7f4ec8103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1056992926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ec8102a20 0x7f4ec8102ef0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1056992926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec8102040 0x7f4ec8102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1056992926 >> 172.21.15.27:0/1056992926 conn(0x7f4ec80fdc10 msgr2=0x7f4ec81000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1056992926 shutdown_connections 2024-04-25T13:42:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1056992926 wait complete. 2024-04-25T13:42:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.423+0000 7f4ecf33f640 1 Processor -- start 2024-04-25T13:42:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecf33f640 1 -- start start 2024-04-25T13:42:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecf33f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ec8102040 0x7f4ec819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecf33f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec8102a20 0x7f4ec819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecf33f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8103580 0x7f4ec81a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecf33f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8111850 con 0x7f4ec8102040 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecf33f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81116d0 con 0x7f4ec8103580 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecf33f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81119d0 con 0x7f4ec8102a20 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecd8dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8103580 0x7f4ec81a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecd8dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8103580 0x7f4ec81a38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48094/0 (socket says 172.21.15.27:48094) 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecc8da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec8102a20 0x7f4ec819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/1256775560 learned_addr learned my addr 172.21.15.27:0/1256775560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/1256775560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec8102a20 msgr2=0x7f4ec819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec8102a20 0x7f4ec819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ec8102040 msgr2=0x7f4ec819c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.424+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ec8102040 0x7f4ec819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.425+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/1256775560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec81a3f60 con 0x7f4ec8103580 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.425+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8103580 0x7f4ec81a38f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000eb10 tx=0x7f4ec000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.425+0000 7f4ebe7fc640 1 -- 172.21.15.27:0/1256775560 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0004b20 con 0x7f4ec8103580 2024-04-25T13:42:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.425+0000 7f4ebe7fc640 1 -- 172.21.15.27:0/1256775560 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec0004cc0 con 0x7f4ec8103580 2024-04-25T13:42:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.425+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1256775560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec81a4250 con 0x7f4ec8103580 2024-04-25T13:42:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.425+0000 7f4ebe7fc640 1 -- 172.21.15.27:0/1256775560 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0010720 con 0x7f4ec8103580 2024-04-25T13:42:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.425+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1256775560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec81a4720 con 0x7f4ec8103580 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.428+0000 7f4ebe7fc640 1 -- 172.21.15.27:0/1256775560 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4ec0005020 con 0x7f4ec8103580 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.429+0000 7f4ed61f4740 1 Processor -- start 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.429+0000 7f4ebe7fc640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4eb40afbb0 0x7f4eb40b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.429+0000 7f4ed61f4740 1 -- start start 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.429+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4eb40afbb0 0x7f4eb40b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.429+0000 7f4ebe7fc640 1 -- 172.21.15.27:0/1256775560 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4ec00df230 con 0x7f4ec8103580 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af1f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab562f0 0x557d1af1fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af26df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557d1ae85460 con 0x557d1ab7f1b0 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557d1ae3f590 con 0x557d1ab5e4d0 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557d1ae81420 con 0x557d1ab562f0 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ecc8da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab562f0 0x557d1af1fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ecc8da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab562f0 0x557d1af1fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33656/0 (socket says 172.21.15.27:33656) 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ecc8da640 1 -- 172.21.15.27:0/2238584220 learned_addr learned my addr 172.21.15.27:0/2238584220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.430+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4eb40afbb0 0x7f4eb40b20c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8031500 tx=0x7f4eb8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4ecc8da640 1 -- 172.21.15.27:0/2238584220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 msgr2=0x557d1af26df0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af26df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4ecc8da640 1 -- 172.21.15.27:0/2238584220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 msgr2=0x557d1af1f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af1f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af1f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4ecc8da640 1 -- 172.21.15.27:0/2238584220 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d1ae4c0d0 con 0x557d1ab562f0 2024-04-25T13:42:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af1f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:42:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab562f0 0x557d1af1fef0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009230 tx=0x7f4eb8045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10e97d555ea14e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4eb37fe640 1 -- 172.21.15.27:0/2238584220 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8035040 con 0x557d1ab562f0 2024-04-25T13:42:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.431+0000 7f4eb37fe640 1 -- 172.21.15.27:0/2238584220 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb8003bb0 con 0x557d1ab562f0 2024-04-25T13:42:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.432+0000 7f4eb37fe640 1 -- 172.21.15.27:0/2238584220 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8035040 con 0x557d1ab562f0 2024-04-25T13:42:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 -- 172.21.15.27:0/2238584220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab562f0 msgr2=0x557d1af1fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab562f0 0x557d1af1fef0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009230 tx=0x7f4eb8045000 comp rx=0 tx=0).stop 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 -- 172.21.15.27:0/2238584220 shutdown_connections 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af26df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab562f0 0x557d1af1fef0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/2238584220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af1f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 -- 172.21.15.27:0/2238584220 >> 172.21.15.27:0/2238584220 conn(0x557d1ab73300 msgr2=0x557d1af1b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 -- 172.21.15.27:0/2238584220 shutdown_connections 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.433+0000 7f4ed61f4740 1 -- 172.21.15.27:0/2238584220 wait complete. 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.434+0000 7f4ed61f4740 1 -- start start 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.434+0000 7f4ed61f4740 1 Processor -- start 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af296a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af29be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af30b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557d1af277e0 con 0x557d1ab562f0 2024-04-25T13:42:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557d1ae0cb60 con 0x557d1ab5e4d0 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557d1af27960 con 0x557d1ab7f1b0 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ed61f4740 10 client.0 _ll_register_callbacks cb 0x557d1ae312b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ecd8dc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af30b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ecd8dc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af30b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33670/0 (socket says 172.21.15.27:33670) 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/3274010388 learned_addr learned my addr 172.21.15.27:0/3274010388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af29be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 msgr2=0x557d1af29be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.435+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af29be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.436+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/3274010388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 msgr2=0x557d1af296a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.436+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af296a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.436+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/3274010388 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d1ab97210 con 0x557d1ab7f1b0 2024-04-25T13:42:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.436+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af29be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:42:01.439 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.436+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af30b40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00b0070 tx=0x7f4ec008ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f062d0c60d835a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.436+0000 7f4eb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.436+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/3274010388 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00c1060 con 0x557d1ab7f1b0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.437+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/3274010388 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4ec00b0110 con 0x557d1ab7f1b0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.437+0000 7f4ed61f4740 1 -- 172.21.15.27:0/3274010388 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb8003bb0 con 0x557d1ab7f1b0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.437+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/3274010388 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4ec00a7e40 con 0x557d1ab7f1b0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.437+0000 7f4ed61f4740 10 client.25300 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.437+0000 7f4ed61f4740 1 -- 172.21.15.27:0/3274010388 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557d1ac63620 con 0x557d1ab7f1b0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.437+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/3274010388 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec00c6020 con 0x557d1ab7f1b0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.437+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/3274010388 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f4ec0005310 con 0x557d1ab7f1b0 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4ed61f4740 10 client.25300 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4ed61f4740 10 client.25300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4ed61f4740 20 client.25300 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557d1ab3dd80 0x557d1af372b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4ed61f4740 4 client.25300 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4ed61f4740 1 -- 172.21.15.27:0/3274010388 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4eb8003bb0 con 0x557d1ab3dd80 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557d1ab3dd80 0x557d1af372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557d1ab3dd80 0x557d1af372b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=727db3086af4d7a4 server_cookie=bd014d064ccd6671 in_seq=0 out_seq=0 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.438+0000 7f4eb0ff9640 10 client.25300 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/3274010388 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f4eb8003bb0 con 0x557d1ab3dd80 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4eb0ff9640 10 client.25300 handle_command_reply: tid=0 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 client.25300 shutdown 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 -- 172.21.15.27:0/3274010388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 msgr2=0x557d1af30b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af30b40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00b0070 tx=0x7f4ec008ef20 comp rx=0 tx=0).stop 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 -- 172.21.15.27:0/3274010388 shutdown_connections 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557d1ab3dd80 0x557d1af372b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af30b40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af29be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/3274010388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af296a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4eb0ff9640 0 client.25300 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 -- 172.21.15.27:0/3274010388 >> 172.21.15.27:0/3274010388 conn(0x557d1ab73300 msgr2=0x557d1af1e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 -- 172.21.15.27:0/3274010388 shutdown_connections 2024-04-25T13:42:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 1 -- 172.21.15.27:0/3274010388 wait complete. 2024-04-25T13:42:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.439+0000 7f4ed61f4740 20 client.25300 trim_cache size 0 max 16384 2024-04-25T13:42:01.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ed61f4740 1 Processor -- start 2024-04-25T13:42:01.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ed61f4740 1 -- start start 2024-04-25T13:42:01.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af1ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af30a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af1b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81ab420 con 0x557d1ab562f0 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81adca0 con 0x557d1ab5e4d0 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81aba90 con 0x557d1ab7f1b0 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecd0db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af1ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecd0db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af1ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44076/0 (socket says 172.21.15.27:44076) 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecd0db640 1 -- 172.21.15.27:0/1729558628 learned_addr learned my addr 172.21.15.27:0/1729558628 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af1b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af30a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/1729558628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 msgr2=0x557d1af30a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af30a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/1729558628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 msgr2=0x557d1af1ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.461+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af1ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.462+0000 7f4ecd8dc640 1 -- 172.21.15.27:0/1729558628 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d1af26ee0 con 0x557d1ab7f1b0 2024-04-25T13:42:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.462+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af1ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.462+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af1b190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00a7440 tx=0x7f4ec00058a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c20fa0181c59f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.462+0000 7f4eb2ffd640 1 -- 172.21.15.27:0/1729558628 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00bce60 con 0x557d1ab7f1b0 2024-04-25T13:42:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.462+0000 7f4eb2ffd640 1 -- 172.21.15.27:0/1729558628 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec00a7740 con 0x557d1ab7f1b0 2024-04-25T13:42:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.462+0000 7f4eb2ffd640 1 -- 172.21.15.27:0/1729558628 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00a6eb0 con 0x557d1ab7f1b0 2024-04-25T13:42:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 -- 172.21.15.27:0/1729558628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 msgr2=0x557d1af1b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af1b190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00a7440 tx=0x7f4ec00058a0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 -- 172.21.15.27:0/1729558628 shutdown_connections 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab7f1b0 0x557d1af1b190 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab5e4d0 0x557d1af30a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/1729558628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab562f0 0x557d1af1ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 -- 172.21.15.27:0/1729558628 >> 172.21.15.27:0/1729558628 conn(0x557d1ab73300 msgr2=0x557d1af23b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 -- 172.21.15.27:0/1729558628 shutdown_connections 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 -- 172.21.15.27:0/1729558628 wait complete. 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 -- start start 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.464+0000 7f4ed61f4740 1 Processor -- start 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab562f0 0x557d1af2e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab5e4d0 0x557d1af2e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ed61f4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af2f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81ac160 con 0x557d1ab7f1b0 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81ad5d0 con 0x557d1ab562f0 2024-04-25T13:42:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ed61f4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81ac830 con 0x557d1ab5e4d0 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ed61f4740 10 client.0 _ll_register_callbacks cb 0x7f4ec00a36b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd0db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab562f0 0x557d1af2e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecc8da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab5e4d0 0x557d1af2e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd0db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab562f0 0x557d1af2e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48136/0 (socket says 172.21.15.27:48136) 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd8dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af2f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd0db640 1 -- 172.21.15.27:0/114849742 learned_addr learned my addr 172.21.15.27:0/114849742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecc8da640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab5e4d0 0x557d1af2e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33694/0 (socket says 172.21.15.27:33694) 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd0db640 1 -- 172.21.15.27:0/114849742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab5e4d0 msgr2=0x557d1af2e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab5e4d0 0x557d1af2e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd0db640 1 -- 172.21.15.27:0/114849742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 msgr2=0x557d1af2f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af2f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd0db640 1 -- 172.21.15.27:0/114849742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d1af2fd90 con 0x557d1ab562f0 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.465+0000 7f4ecd8dc640 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af2f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4ecd0db640 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab562f0 0x557d1af2e360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x557d1af20b40 tx=0x7f4ec401b600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdfbda0d4b65c710 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4eb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/114849742 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4005220 con 0x557d1ab562f0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/114849742 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4ec40053c0 con 0x557d1ab562f0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4ed61f4740 1 -- 172.21.15.27:0/114849742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ec00a7740 con 0x557d1ab562f0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/114849742 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4ec40055e0 con 0x557d1ab562f0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4ed61f4740 10 client.25301 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4ed61f4740 1 -- 172.21.15.27:0/114849742 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557d1afa7ff0 con 0x557d1ab562f0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/114849742 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec00a7740 con 0x557d1ab562f0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/114849742 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f4ec4022070 con 0x557d1ab562f0 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4ed61f4740 10 client.25301 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4ed61f4740 10 client.25301 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4ed61f4740 20 client.25301 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.466+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557d1ab131c0 0x557d1afaa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.467+0000 7f4ed61f4740 4 client.25301 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:42:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.467+0000 7f4ed61f4740 1 -- 172.21.15.27:0/114849742 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557d1ae93070 con 0x557d1ab131c0 2024-04-25T13:42:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.467+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557d1ab131c0 0x557d1afaa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.467+0000 7f4ecc8da640 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557d1ab131c0 0x557d1afaa9e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eedd95246f7c361 server_cookie=a72f0cc06722e9d3 in_seq=0 out_seq=0 2024-04-25T13:42:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.467+0000 7f4eb0ff9640 10 client.25301 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4eb0ff9640 1 -- 172.21.15.27:0/114849742 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557d1ae93070 con 0x557d1ab131c0 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4eb0ff9640 10 client.25301 handle_command_reply: tid=0 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4ed61f4740 1 client.25301 shutdown 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4ed61f4740 1 -- 172.21.15.27:0/114849742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab562f0 msgr2=0x557d1af2e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab562f0 0x557d1af2e360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x557d1af20b40 tx=0x7f4ec401b600 comp rx=0 tx=0).stop 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4ed61f4740 1 -- 172.21.15.27:0/114849742 shutdown_connections 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557d1ab131c0 0x557d1afaa9e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557d1ab7f1b0 0x557d1af2f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d1ab5e4d0 0x557d1af2e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4eb0ff9640 0 client.25301 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.481+0000 7f4ed61f4740 1 --2- 172.21.15.27:0/114849742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557d1ab562f0 0x557d1af2e360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.482+0000 7f4ed61f4740 1 -- 172.21.15.27:0/114849742 >> 172.21.15.27:0/114849742 conn(0x557d1ab73300 msgr2=0x557d1af1e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.482+0000 7f4ed61f4740 1 -- 172.21.15.27:0/114849742 shutdown_connections 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.482+0000 7f4ed61f4740 1 -- 172.21.15.27:0/114849742 wait complete. 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.482+0000 7f4ed61f4740 20 client.25301 trim_cache size 0 max 16384 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:42:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:42:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.482+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4eb40afbb0 msgr2=0x7f4eb40b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.482+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4eb40afbb0 0x7f4eb40b20c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8031500 tx=0x7f4eb8000bc0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.482+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1256775560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8103580 msgr2=0x7f4ec81a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.482+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8103580 0x7f4ec81a38f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000eb10 tx=0x7f4ec000c2d0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.483+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1256775560 shutdown_connections 2024-04-25T13:42:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.483+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4eb40afbb0 0x7f4eb40b20c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.483+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ec8103580 0x7f4ec81a38f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.483+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec8102a20 0x7f4ec819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.483+0000 7f4ecf33f640 1 --2- 172.21.15.27:0/1256775560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ec8102040 0x7f4ec819c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.483+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1256775560 >> 172.21.15.27:0/1256775560 conn(0x7f4ec80fdc10 msgr2=0x7f4ec810a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.483+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1256775560 shutdown_connections 2024-04-25T13:42:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:01.483+0000 7f4ecf33f640 1 -- 172.21.15.27:0/1256775560 wait complete. 2024-04-25T13:42:02.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:02 smithi122 ceph-mon[32843]: pgmap v383: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.3 MiB/s wr, 797 op/s 2024-04-25T13:42:02.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:02 smithi045 ceph-mon[32731]: pgmap v383: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.3 MiB/s wr, 797 op/s 2024-04-25T13:42:02.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:02 smithi027 ceph-mon[29140]: pgmap v383: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.3 MiB/s wr, 797 op/s 2024-04-25T13:42:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 55.0 s, 3855.0 tps, lat 25.993 ms stddev 27.611 2024-04-25T13:42:04.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:04 smithi122 ceph-mon[32843]: pgmap v384: 97 pgs: 97 active+clean; 1014 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.3 MiB/s wr, 844 op/s 2024-04-25T13:42:04.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:04 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:42:04.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:04 smithi045 ceph-mon[32731]: pgmap v384: 97 pgs: 97 active+clean; 1014 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.3 MiB/s wr, 844 op/s 2024-04-25T13:42:04.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:04 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:42:04.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:04 smithi027 ceph-mon[29140]: pgmap v384: 97 pgs: 97 active+clean; 1014 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.3 MiB/s wr, 844 op/s 2024-04-25T13:42:04.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:04 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:42:06.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:06 smithi122 ceph-mon[32843]: pgmap v385: 97 pgs: 97 active+clean; 1018 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 9.1 MiB/s wr, 791 op/s 2024-04-25T13:42:06.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:06 smithi045 ceph-mon[32731]: pgmap v385: 97 pgs: 97 active+clean; 1018 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 9.1 MiB/s wr, 791 op/s 2024-04-25T13:42:06.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:06 smithi027 ceph-mon[29140]: pgmap v385: 97 pgs: 97 active+clean; 1018 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 9.1 MiB/s wr, 791 op/s 2024-04-25T13:42:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 60.0 s, 4056.4 tps, lat 24.625 ms stddev 25.566 2024-04-25T13:42:08.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:08 smithi122 ceph-mon[32843]: pgmap v386: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-25T13:42:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:08 smithi045 ceph-mon[32731]: pgmap v386: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-25T13:42:08.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:08 smithi027 ceph-mon[29140]: pgmap v386: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-25T13:42:10.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:10 smithi045 ceph-mon[32731]: pgmap v387: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.3 MiB/s wr, 776 op/s 2024-04-25T13:42:10.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:10 smithi027 ceph-mon[29140]: pgmap v387: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.3 MiB/s wr, 776 op/s 2024-04-25T13:42:10.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:10 smithi122 ceph-mon[32843]: pgmap v387: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.3 MiB/s wr, 776 op/s 2024-04-25T13:42:11.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:42:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:11 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:42:11.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:11 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:42:11.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:11.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:11 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:12.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:12 smithi045 ceph-mon[32731]: pgmap v388: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 9.4 MiB/s wr, 857 op/s 2024-04-25T13:42:12.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:42:12.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:42:12.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:12.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:12 smithi027 ceph-mon[29140]: pgmap v388: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 9.4 MiB/s wr, 857 op/s 2024-04-25T13:42:12.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:42:12.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:42:12.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:12.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:12 smithi122 ceph-mon[32843]: pgmap v388: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 9.4 MiB/s wr, 857 op/s 2024-04-25T13:42:12.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:42:12.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:42:12.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:42:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 65.0 s, 3962.4 tps, lat 25.232 ms stddev 26.702 2024-04-25T13:42:14.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:14 smithi045 ceph-mon[32731]: pgmap v389: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 903 op/s 2024-04-25T13:42:14.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:14 smithi027 ceph-mon[29140]: pgmap v389: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 903 op/s 2024-04-25T13:42:14.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:14 smithi122 ceph-mon[32843]: pgmap v389: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 903 op/s 2024-04-25T13:42:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 -- 172.21.15.27:0/4143578786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1050f0 msgr2=0x7f1dcc105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 --2- 172.21.15.27:0/4143578786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1050f0 0x7f1dcc105500 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009860 tx=0x7f1dbc02a340 comp rx=0 tx=0).stop 2024-04-25T13:42:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 -- 172.21.15.27:0/4143578786 shutdown_connections 2024-04-25T13:42:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 --2- 172.21.15.27:0/4143578786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dcc1022a0 0x7f1dcc10bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 --2- 172.21.15.27:0/4143578786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dcc1017a0 0x7f1dcc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 --2- 172.21.15.27:0/4143578786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1050f0 0x7f1dcc105500 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 -- 172.21.15.27:0/4143578786 >> 172.21.15.27:0/4143578786 conn(0x7f1dcc0fd3c0 msgr2=0x7f1dcc0ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 -- 172.21.15.27:0/4143578786 shutdown_connections 2024-04-25T13:42:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.628+0000 7f1dd1227640 1 -- 172.21.15.27:0/4143578786 wait complete. 2024-04-25T13:42:14.632 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.629+0000 7f1dd1227640 1 Processor -- start 2024-04-25T13:42:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.629+0000 7f1dd1227640 1 -- start start 2024-04-25T13:42:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.629+0000 7f1dd1227640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1017a0 0x7f1dcc1a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dd1227640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dcc1022a0 0x7f1dcc1a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dd1227640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dcc1050f0 0x7f1dcc1a7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dd1227640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1046a0 con 0x7f1dcc1022a0 2024-04-25T13:42:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dd1227640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc104520 con 0x7f1dcc1050f0 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1017a0 0x7f1dcc1a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dd1227640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc104820 con 0x7f1dcc1017a0 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1017a0 0x7f1dcc1a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59146/0 (socket says 172.21.15.27:59146) 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcad9d640 1 -- 172.21.15.27:0/1269575288 learned_addr learned my addr 172.21.15.27:0/1269575288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcb59e640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dcc1050f0 0x7f1dcc1a7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dcc1022a0 0x7f1dcc1a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcad9d640 1 -- 172.21.15.27:0/1269575288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dcc1050f0 msgr2=0x7f1dcc1a7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dcc1050f0 0x7f1dcc1a7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcad9d640 1 -- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dcc1022a0 msgr2=0x7f1dcc1a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dcc1022a0 0x7f1dcc1a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.630+0000 7f1dcad9d640 1 -- 172.21.15.27:0/1269575288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dcc1a8440 con 0x7f1dcc1017a0 2024-04-25T13:42:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.631+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1017a0 0x7f1dcc1a0900 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc02a860 tx=0x7f1dbc032d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.631+0000 7f1db3fff640 1 -- 172.21.15.27:0/1269575288 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc040020 con 0x7f1dcc1017a0 2024-04-25T13:42:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.631+0000 7f1db3fff640 1 -- 172.21.15.27:0/1269575288 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dbc030d50 con 0x7f1dcc1017a0 2024-04-25T13:42:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.631+0000 7f1db3fff640 1 -- 172.21.15.27:0/1269575288 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc037040 con 0x7f1dcc1017a0 2024-04-25T13:42:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.631+0000 7f1dd1227640 1 -- 172.21.15.27:0/1269575288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dcc1a86d0 con 0x7f1dcc1017a0 2024-04-25T13:42:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.631+0000 7f1dd1227640 1 -- 172.21.15.27:0/1269575288 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dcc1a8b70 con 0x7f1dcc1017a0 2024-04-25T13:42:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1db3fff640 1 -- 172.21.15.27:0/1269575288 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1dbc043020 con 0x7f1dcc1017a0 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1dd81b9740 1 Processor -- start 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1dd81b9740 1 -- start start 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb341b0 0x5575abed4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abed4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abedbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5575abe3a460 con 0x5575abb0b2f0 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5575abdf4590 con 0x5575abb341b0 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.635+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5575abe36420 con 0x5575abb134d0 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.636+0000 7f1dcad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abed4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.636+0000 7f1dcb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb341b0 0x5575abed4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.636+0000 7f1dcb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb341b0 0x5575abed4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:58132/0 (socket says 172.21.15.27:58132) 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.636+0000 7f1dcb59e640 1 -- 172.21.15.27:0/1402088061 learned_addr learned my addr 172.21.15.27:0/1402088061 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.636+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1402088061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abedbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.637+0000 7f1dca59c640 1 -- 172.21.15.27:0/1402088061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb341b0 msgr2=0x5575abed4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.637+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1402088061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb341b0 0x5575abed4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.637+0000 7f1dca59c640 1 -- 172.21.15.27:0/1402088061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 msgr2=0x5575abed4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.637+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1402088061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abed4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.637+0000 7f1dca59c640 1 -- 172.21.15.27:0/1402088061 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575abe010d0 con 0x5575abb134d0 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.637+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1402088061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abedbdf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009970 tx=0x7f1dc0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20c4be2b262f9e57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.638+0000 7f1db17fa640 1 -- 172.21.15.27:0/1402088061 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0017070 con 0x5575abb134d0 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.638+0000 7f1db17fa640 1 -- 172.21.15.27:0/1402088061 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dc0013470 con 0x5575abb134d0 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.638+0000 7f1db17fa640 1 -- 172.21.15.27:0/1402088061 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0012630 con 0x5575abb134d0 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.638+0000 7f1db3fff640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1db40afb60 0x7f1db40b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.638+0000 7f1db3fff640 1 -- 172.21.15.27:0/1269575288 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1dbc0fed70 con 0x7f1dcc1017a0 2024-04-25T13:42:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.638+0000 7f1dcb59e640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1db40afb60 0x7f1db40b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.639+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1402088061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 msgr2=0x5575abedbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.639+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1402088061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abedbdf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009970 tx=0x7f1dc0009e30 comp rx=0 tx=0).stop 2024-04-25T13:42:14.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.640+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1402088061 shutdown_connections 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.640+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1402088061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abedbdf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.640+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1402088061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abed4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.640+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1402088061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb341b0 0x5575abed4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.640+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1402088061 >> 172.21.15.27:0/1402088061 conn(0x5575abb28300 msgr2=0x5575abed0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.640+0000 7f1dcb59e640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1db40afb60 0x7f1db40b2070 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x5575abed5b40 tx=0x7f1dc4010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.640+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1402088061 shutdown_connections 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.640+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1402088061 wait complete. 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 1 -- start start 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 1 Processor -- start 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abede700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb134d0 0x5575abedec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abee5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5575abedc840 con 0x5575abb0b2f0 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5575abdc1b60 con 0x5575abb134d0 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5575abedc9c0 con 0x5575abb341b0 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dd81b9740 10 client.0 _ll_register_callbacks cb 0x5575abde62b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dcad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abede700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dcad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abede700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48662/0 (socket says 172.21.15.27:48662) 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dcad9d640 1 -- 172.21.15.27:0/1216557568 learned_addr learned my addr 172.21.15.27:0/1216557568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dcb59e640 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abee5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.641+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb134d0 0x5575abedec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.642+0000 7f1dcb59e640 1 -- 172.21.15.27:0/1216557568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb134d0 msgr2=0x5575abedec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.642+0000 7f1dcb59e640 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb134d0 0x5575abedec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.642+0000 7f1dcb59e640 1 -- 172.21.15.27:0/1216557568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 msgr2=0x5575abede700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.642+0000 7f1dcb59e640 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abede700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.642+0000 7f1dcb59e640 1 -- 172.21.15.27:0/1216557568 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575abb4c210 con 0x5575abb341b0 2024-04-25T13:42:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.642+0000 7f1dcb59e640 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abee5ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4009fd0 tx=0x7f1dc4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ab7fdeee02d1ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.643+0000 7f1daeff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.643+0000 7f1daeff5640 1 -- 172.21.15.27:0/1216557568 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40243f0 con 0x5575abb341b0 2024-04-25T13:42:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.643+0000 7f1daeff5640 1 -- 172.21.15.27:0/1216557568 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1dc4009020 con 0x5575abb341b0 2024-04-25T13:42:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.643+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1216557568 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dc0013470 con 0x5575abb341b0 2024-04-25T13:42:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.644+0000 7f1daeff5640 1 -- 172.21.15.27:0/1216557568 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1dc4024590 con 0x5575abb341b0 2024-04-25T13:42:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.644+0000 7f1dd81b9740 10 client.25318 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.644+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1216557568 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5575abc18620 con 0x5575abb341b0 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.644+0000 7f1daeff5640 1 -- 172.21.15.27:0/1216557568 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dc0013470 con 0x5575abb341b0 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.645+0000 7f1daeff5640 1 -- 172.21.15.27:0/1216557568 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1dc400b040 con 0x5575abb341b0 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.645+0000 7f1dd81b9740 10 client.25318 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.645+0000 7f1dd81b9740 10 client.25318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.645+0000 7f1dd81b9740 20 client.25318 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.645+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5575abaf2d80 0x5575abeec9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.645+0000 7f1dd81b9740 4 client.25318 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.645+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1216557568 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575abe48070 con 0x5575abaf2d80 2024-04-25T13:42:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.645+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5575abaf2d80 0x5575abeec9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.646+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5575abaf2d80 0x5575abeec9b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a303c8ba0ee3585 server_cookie=62408928ecd90c3b in_seq=0 out_seq=0 2024-04-25T13:42:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.646+0000 7f1daeff5640 10 client.25318 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.647+0000 7f1daeff5640 1 -- 172.21.15.27:0/1216557568 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5575abe48070 con 0x5575abaf2d80 2024-04-25T13:42:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.647+0000 7f1daeff5640 10 client.25318 handle_command_reply: tid=0 2024-04-25T13:42:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.647+0000 7f1dd81b9740 1 client.25318 shutdown 2024-04-25T13:42:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1216557568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 msgr2=0x5575abee5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abee5ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4009fd0 tx=0x7f1dc4004ab0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1216557568 shutdown_connections 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5575abaf2d80 0x5575abeec9b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abee5ba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb134d0 0x5575abedec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1216557568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb0b2f0 0x5575abede700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1216557568 >> 172.21.15.27:0/1216557568 conn(0x5575abb28300 msgr2=0x5575abecf250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1daeff5640 0 client.25318 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.648+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1216557568 shutdown_connections 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.649+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1216557568 wait complete. 2024-04-25T13:42:14.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.649+0000 7f1dd81b9740 20 client.25318 trim_cache size 0 max 16384 2024-04-25T13:42:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.671+0000 7f1dd81b9740 1 Processor -- start 2024-04-25T13:42:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.671+0000 7f1dd81b9740 1 -- start start 2024-04-25T13:42:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.671+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abed0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.671+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb134d0 0x5575abee5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.671+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abece820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.671+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1af940 con 0x5575abb134d0 2024-04-25T13:42:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.671+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1b21c0 con 0x5575abb0b2f0 2024-04-25T13:42:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.671+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1affb0 con 0x5575abb341b0 2024-04-25T13:42:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dca59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb134d0 0x5575abee5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dcb59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abece820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dcb59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abece820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59174/0 (socket says 172.21.15.27:59174) 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dcb59e640 1 -- 172.21.15.27:0/1200382238 learned_addr learned my addr 172.21.15.27:0/1200382238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abed0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dca59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb134d0 0x5575abee5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48676/0 (socket says 172.21.15.27:48676) 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dca59c640 1 -- 172.21.15.27:0/1200382238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 msgr2=0x5575abece820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abece820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dca59c640 1 -- 172.21.15.27:0/1200382238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 msgr2=0x5575abed0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abed0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dca59c640 1 -- 172.21.15.27:0/1200382238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575abee6000 con 0x5575abb134d0 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.672+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abed0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.674+0000 7f1dca59c640 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb134d0 0x5575abee5ac0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0004810 tx=0x7f1dc00096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46e4add583466eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.674+0000 7f1db0ff9640 1 -- 172.21.15.27:0/1200382238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0017040 con 0x5575abb134d0 2024-04-25T13:42:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.674+0000 7f1db0ff9640 1 -- 172.21.15.27:0/1200382238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dc000ebc0 con 0x5575abb134d0 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.674+0000 7f1db0ff9640 1 -- 172.21.15.27:0/1200382238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0006740 con 0x5575abb134d0 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1200382238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb134d0 msgr2=0x5575abee5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb134d0 0x5575abee5ac0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0004810 tx=0x7f1dc00096e0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1200382238 shutdown_connections 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb341b0 0x5575abece820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb134d0 0x5575abee5ac0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/1200382238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abed0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1200382238 >> 172.21.15.27:0/1200382238 conn(0x5575abb28300 msgr2=0x5575abecfc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1200382238 shutdown_connections 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 -- 172.21.15.27:0/1200382238 wait complete. 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 -- start start 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.675+0000 7f1dd81b9740 1 Processor -- start 2024-04-25T13:42:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abee2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abee2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dd81b9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb341b0 0x5575abf52d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dca59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abee2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1b0680 con 0x5575abb341b0 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1b1af0 con 0x5575abb0b2f0 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dd81b9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1b0d50 con 0x5575abb134d0 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dca59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abee2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59186/0 (socket says 172.21.15.27:59186) 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dcb59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb341b0 0x5575abf52d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dca59c640 1 -- 172.21.15.27:0/311668258 learned_addr learned my addr 172.21.15.27:0/311668258 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abee2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dd81b9740 10 client.0 _ll_register_callbacks cb 0x7f1dc4009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dca59c640 1 -- 172.21.15.27:0/311668258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 msgr2=0x5575abee2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dca59c640 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abee2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dca59c640 1 -- 172.21.15.27:0/311668258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb341b0 msgr2=0x5575abf52d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dca59c640 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb341b0 0x5575abf52d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dca59c640 1 -- 172.21.15.27:0/311668258 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575abee34e0 con 0x5575abb134d0 2024-04-25T13:42:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.676+0000 7f1dd81b9740 1 -- 172.21.15.27:0/311668258 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dc000ebc0 con 0x5575abb134d0 2024-04-25T13:42:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.677+0000 7f1dca59c640 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abee2e70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0005bf0 tx=0x7f1dc0005c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7c1e29294ffe3a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.677+0000 7f1daeff5640 10 client.25324 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.677+0000 7f1daeff5640 1 -- 172.21.15.27:0/311668258 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc002c070 con 0x5575abb134d0 2024-04-25T13:42:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.677+0000 7f1daeff5640 1 -- 172.21.15.27:0/311668258 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1dc0005f50 con 0x5575abb134d0 2024-04-25T13:42:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.677+0000 7f1daeff5640 1 -- 172.21.15.27:0/311668258 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1dc0017070 con 0x5575abb134d0 2024-04-25T13:42:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.677+0000 7f1dd81b9740 10 client.25324 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.677+0000 7f1dd81b9740 1 -- 172.21.15.27:0/311668258 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x5575abf54200 con 0x5575abb134d0 2024-04-25T13:42:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.678+0000 7f1daeff5640 1 -- 172.21.15.27:0/311668258 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1dc0013470 con 0x5575abb134d0 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.678+0000 7f1dd81b9740 10 client.25324 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.678+0000 7f1dd81b9740 10 client.25324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.678+0000 7f1dd81b9740 20 client.25324 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.678+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5575abac81c0 0x5575abf56660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.678+0000 7f1dd81b9740 4 client.25324 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.678+0000 7f1dd81b9740 1 -- 172.21.15.27:0/311668258 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575abe41ee0 con 0x5575abac81c0 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.678+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5575abac81c0 0x5575abf56660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.679+0000 7f1daeff5640 10 client.25324 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.679+0000 7f1dcad9d640 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5575abac81c0 0x5575abf56660 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cc788806c8795d1 server_cookie=1cd78d442399428a in_seq=0 out_seq=0 2024-04-25T13:42:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.704+0000 7f1daeff5640 1 -- 172.21.15.27:0/311668258 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575abe41ee0 con 0x5575abac81c0 2024-04-25T13:42:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.704+0000 7f1daeff5640 10 client.25324 handle_command_reply: tid=0 2024-04-25T13:42:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 client.25324 shutdown 2024-04-25T13:42:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 -- 172.21.15.27:0/311668258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 msgr2=0x5575abee2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abee2e70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0005bf0 tx=0x7f1dc0005c20 comp rx=0 tx=0).stop 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 -- 172.21.15.27:0/311668258 shutdown_connections 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5575abac81c0 0x5575abf56660 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5575abb341b0 0x5575abf52d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5575abb134d0 0x5575abee2e70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1daeff5640 0 client.25324 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 --2- 172.21.15.27:0/311668258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5575abb0b2f0 0x5575abee2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 -- 172.21.15.27:0/311668258 >> 172.21.15.27:0/311668258 conn(0x5575abb28300 msgr2=0x5575abed3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 -- 172.21.15.27:0/311668258 shutdown_connections 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 1 -- 172.21.15.27:0/311668258 wait complete. 2024-04-25T13:42:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.705+0000 7f1dd81b9740 20 client.25324 trim_cache size 0 max 16384 2024-04-25T13:42:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:42:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:42:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:42:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 -- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1db40afb60 msgr2=0x7f1db40b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1db40afb60 0x7f1db40b2070 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x5575abed5b40 tx=0x7f1dc4010040 comp rx=0 tx=0).stop 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 -- 172.21.15.27:0/1269575288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1017a0 msgr2=0x7f1dcc1a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1017a0 0x7f1dcc1a0900 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc02a860 tx=0x7f1dbc032d30 comp rx=0 tx=0).stop 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 -- 172.21.15.27:0/1269575288 shutdown_connections 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1db40afb60 0x7f1db40b2070 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dcc1050f0 0x7f1dcc1a7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dcc1022a0 0x7f1dcc1a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 --2- 172.21.15.27:0/1269575288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1dcc1017a0 0x7f1dcc1a0900 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 -- 172.21.15.27:0/1269575288 >> 172.21.15.27:0/1269575288 conn(0x7f1dcc0fd3c0 msgr2=0x7f1dcc10de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 -- 172.21.15.27:0/1269575288 shutdown_connections 2024-04-25T13:42:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:14.706+0000 7f1dd1227640 1 -- 172.21.15.27:0/1269575288 wait complete. 2024-04-25T13:42:16.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:16 smithi045 ceph-mon[32731]: pgmap v390: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 842 op/s 2024-04-25T13:42:16.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:16 smithi027 ceph-mon[29140]: pgmap v390: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 842 op/s 2024-04-25T13:42:16.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:16 smithi122 ceph-mon[32843]: pgmap v390: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 842 op/s 2024-04-25T13:42:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 70.0 s, 3608.6 tps, lat 27.719 ms stddev 27.154 2024-04-25T13:42:18.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:18 smithi027 ceph-mon[29140]: pgmap v391: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 14 MiB/s wr, 949 op/s 2024-04-25T13:42:18.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:18 smithi122 ceph-mon[32843]: pgmap v391: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 14 MiB/s wr, 949 op/s 2024-04-25T13:42:18.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:18 smithi045 ceph-mon[32731]: pgmap v391: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 14 MiB/s wr, 949 op/s 2024-04-25T13:42:20.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:20 smithi027 ceph-mon[29140]: pgmap v392: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 712 op/s 2024-04-25T13:42:20.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:20 smithi122 ceph-mon[32843]: pgmap v392: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 712 op/s 2024-04-25T13:42:20.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:20 smithi045 ceph-mon[32731]: pgmap v392: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 712 op/s 2024-04-25T13:42:22.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:22 smithi027 ceph-mon[29140]: pgmap v393: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 MiB/s wr, 829 op/s 2024-04-25T13:42:22.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:22 smithi122 ceph-mon[32843]: pgmap v393: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 MiB/s wr, 829 op/s 2024-04-25T13:42:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:22 smithi045 ceph-mon[32731]: pgmap v393: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 MiB/s wr, 829 op/s 2024-04-25T13:42:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 75.0 s, 4251.6 tps, lat 23.516 ms stddev 25.436 2024-04-25T13:42:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:24 smithi027 ceph-mon[29140]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 893 op/s 2024-04-25T13:42:24.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:24 smithi122 ceph-mon[32843]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 893 op/s 2024-04-25T13:42:24.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:24 smithi045 ceph-mon[32731]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 893 op/s 2024-04-25T13:42:26.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:26 smithi027 ceph-mon[29140]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 11 MiB/s wr, 687 op/s 2024-04-25T13:42:26.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:26 smithi122 ceph-mon[32843]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 11 MiB/s wr, 687 op/s 2024-04-25T13:42:26.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:26 smithi045 ceph-mon[32731]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 11 MiB/s wr, 687 op/s 2024-04-25T13:42:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.846+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1963219686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 msgr2=0x7f5cec10ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.846+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1963219686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 0x7f5cec10ae20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009900 tx=0x7f5cdc02a410 comp rx=0 tx=0).stop 2024-04-25T13:42:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.847+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1963219686 shutdown_connections 2024-04-25T13:42:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.847+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1963219686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cec110060 0x7f5cec112480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.847+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1963219686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec10b360 0x7f5cec10b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.847+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1963219686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 0x7f5cec10ae20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.847+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1963219686 >> 172.21.15.27:0/1963219686 conn(0x7f5cec078de0 msgr2=0x7f5cec07b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.847+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1963219686 shutdown_connections 2024-04-25T13:42:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.847+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1963219686 wait complete. 2024-04-25T13:42:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.848+0000 7f5cf0f72640 1 Processor -- start 2024-04-25T13:42:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.848+0000 7f5cf0f72640 1 -- start start 2024-04-25T13:42:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.848+0000 7f5cf0f72640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 0x7f5cec1a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.848+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 0x7f5cec1a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 0x7f5cec1a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46388/0 (socket says 172.21.15.27:46388) 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5cea59c640 1 -- 172.21.15.27:0/1816835669 learned_addr learned my addr 172.21.15.27:0/1816835669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec10b360 0x7f5cec1a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cec110060 0x7f5cec1a7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec115c70 con 0x7f5cec10a320 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec115af0 con 0x7f5cec10b360 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec115df0 con 0x7f5cec110060 2024-04-25T13:42:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec10b360 0x7f5cec1a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.849+0000 7f5cead9d640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cec110060 0x7f5cec1a7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cead9d640 1 -- 172.21.15.27:0/1816835669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec10b360 msgr2=0x7f5cec1a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cead9d640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec10b360 0x7f5cec1a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cead9d640 1 -- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 msgr2=0x7f5cec1a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cead9d640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 0x7f5cec1a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cead9d640 1 -- 172.21.15.27:0/1816835669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cec1a83b0 con 0x7f5cec110060 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cead9d640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cec110060 0x7f5cec1a7d40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4005fd0 tx=0x7f5ce400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cdb7fe640 1 -- 172.21.15.27:0/1816835669 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce400cdf0 con 0x7f5cec110060 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cdb7fe640 1 -- 172.21.15.27:0/1816835669 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ce4005020 con 0x7f5cec110060 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cec1a86a0 con 0x7f5cec110060 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cdb7fe640 1 -- 172.21.15.27:0/1816835669 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4010690 con 0x7f5cec110060 2024-04-25T13:42:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.850+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cec1a8bf0 con 0x7f5cec110060 2024-04-25T13:42:27.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cf7e25740 1 Processor -- start 2024-04-25T13:42:27.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cdb7fe640 1 -- 172.21.15.27:0/1816835669 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5ce4010830 con 0x7f5cec110060 2024-04-25T13:42:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cf7e25740 1 -- start start 2024-04-25T13:42:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b15970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b15ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 0x565326b1cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565326a7b460 con 0x5653267544d0 2024-04-25T13:42:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x565326a35590 con 0x5653267751b0 2024-04-25T13:42:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.853+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x565326a77420 con 0x56532674c2f0 2024-04-25T13:42:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b15ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b15ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44514/0 (socket says 172.21.15.27:44514) 2024-04-25T13:42:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cea59c640 1 -- 172.21.15.27:0/779054244 learned_addr learned my addr 172.21.15.27:0/779054244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b15970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 0x565326b1cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cea59c640 1 -- 172.21.15.27:0/779054244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 msgr2=0x565326b15970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:27.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cea59c640 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b15970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cea59c640 1 -- 172.21.15.27:0/779054244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 msgr2=0x565326b1cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cea59c640 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 0x565326b1cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cea59c640 1 -- 172.21.15.27:0/779054244 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565326a420d0 con 0x56532674c2f0 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.854+0000 7f5cdb7fe640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5cd40afae0 0x7f5cd40b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.855+0000 7f5cdb7fe640 1 -- 172.21.15.27:0/1816835669 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5ce40df1c0 con 0x7f5cec110060 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.855+0000 7f5cea59c640 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b15ef0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc006010 tx=0x7f5cdc031b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de9785536109b973 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.855+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5cd40afae0 0x7f5cd40b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.855+0000 7f5cd8ff9640 1 -- 172.21.15.27:0/779054244 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc035030 con 0x56532674c2f0 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.855+0000 7f5cd8ff9640 1 -- 172.21.15.27:0/779054244 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cdc040070 con 0x56532674c2f0 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.855+0000 7f5cd8ff9640 1 -- 172.21.15.27:0/779054244 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc003950 con 0x56532674c2f0 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 -- 172.21.15.27:0/779054244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 msgr2=0x565326b15ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5cd40afae0 0x7f5cd40b1ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x565326b16b40 tx=0x7f5ce0004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b15ef0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc006010 tx=0x7f5cdc031b10 comp rx=0 tx=0).stop 2024-04-25T13:42:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 -- 172.21.15.27:0/779054244 shutdown_connections 2024-04-25T13:42:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 0x565326b1cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b15ef0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/779054244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b15970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 -- 172.21.15.27:0/779054244 >> 172.21.15.27:0/779054244 conn(0x565326769300 msgr2=0x565326b11300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 -- 172.21.15.27:0/779054244 shutdown_connections 2024-04-25T13:42:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.856+0000 7f5cf7e25740 1 -- 172.21.15.27:0/779054244 wait complete. 2024-04-25T13:42:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 1 -- start start 2024-04-25T13:42:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 1 Processor -- start 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b30640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 0x565326b30b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b37ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565326b2e890 con 0x5653267544d0 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x565326a02b60 con 0x5653267751b0 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x565326b2ea10 con 0x56532674c2f0 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cf7e25740 10 client.0 _ll_register_callbacks cb 0x565326a272b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b30640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5ce9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 0x565326b30b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b30640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44516/0 (socket says 172.21.15.27:44516) 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.857+0000 7f5cea59c640 1 -- 172.21.15.27:0/2720864846 learned_addr learned my addr 172.21.15.27:0/2720864846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.858+0000 7f5cead9d640 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b37ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.858+0000 7f5cea59c640 1 -- 172.21.15.27:0/2720864846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 msgr2=0x565326b37ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.858+0000 7f5cea59c640 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b37ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.858+0000 7f5cea59c640 1 -- 172.21.15.27:0/2720864846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 msgr2=0x565326b30b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.858+0000 7f5cea59c640 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 0x565326b30b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.858+0000 7f5cea59c640 1 -- 172.21.15.27:0/2720864846 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56532678d210 con 0x56532674c2f0 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.859+0000 7f5cea59c640 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b30640 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc004650 tx=0x7f5cdc031e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=484d045a4f7ea0c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.859+0000 7f5cd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.859+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/2720864846 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc035040 con 0x56532674c2f0 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.859+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/2720864846 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5cdc0315a0 con 0x56532674c2f0 2024-04-25T13:42:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.859+0000 7f5cf7e25740 1 -- 172.21.15.27:0/2720864846 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cdc040070 con 0x56532674c2f0 2024-04-25T13:42:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.859+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/2720864846 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f5cdc0317a0 con 0x56532674c2f0 2024-04-25T13:42:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.859+0000 7f5cf7e25740 10 client.25336 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.859+0000 7f5cf7e25740 1 -- 172.21.15.27:0/2720864846 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x565326859620 con 0x56532674c2f0 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.860+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/2720864846 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cdc040070 con 0x56532674c2f0 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.860+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/2720864846 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f5cdc045500 con 0x56532674c2f0 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.860+0000 7f5cf7e25740 10 client.25336 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.860+0000 7f5cf7e25740 10 client.25336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.860+0000 7f5cf7e25740 20 client.25336 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.860+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565326733d80 0x565326b3df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.860+0000 7f5cf7e25740 4 client.25336 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.861+0000 7f5cf7e25740 1 -- 172.21.15.27:0/2720864846 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565326a89070 con 0x565326733d80 2024-04-25T13:42:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.861+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565326733d80 0x565326b3df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.862+0000 7f5cd1ffb640 10 client.25336 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.862+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565326733d80 0x565326b3df30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e208b9154d3ef90f server_cookie=f7bf2887ef031f93 in_seq=0 out_seq=0 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/2720864846 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x565326a89070 con 0x565326733d80 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cd1ffb640 10 client.25336 handle_command_reply: tid=0 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 client.25336 shutdown 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 -- 172.21.15.27:0/2720864846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 msgr2=0x565326b30640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b30640 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc004650 tx=0x7f5cdc031e90 comp rx=0 tx=0).stop 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 -- 172.21.15.27:0/2720864846 shutdown_connections 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565326733d80 0x565326b3df30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b37ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cd1ffb640 0 client.25336 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267544d0 0x565326b30b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/2720864846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56532674c2f0 0x565326b30640 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 -- 172.21.15.27:0/2720864846 >> 172.21.15.27:0/2720864846 conn(0x565326769300 msgr2=0x565326b14210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 -- 172.21.15.27:0/2720864846 shutdown_connections 2024-04-25T13:42:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 1 -- 172.21.15.27:0/2720864846 wait complete. 2024-04-25T13:42:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.863+0000 7f5cf7e25740 20 client.25336 trim_cache size 0 max 16384 2024-04-25T13:42:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.887+0000 7f5cf7e25740 1 Processor -- start 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.887+0000 7f5cf7e25740 1 -- start start 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56532674c2f0 0x565326b1a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 0x565326b37a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec082dc0 con 0x56532674c2f0 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec1a97e0 con 0x5653267751b0 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec083430 con 0x5653267544d0 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56532674c2f0 0x565326b1a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56532674c2f0 0x565326b1a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46400/0 (socket says 172.21.15.27:46400) 2024-04-25T13:42:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cea59c640 1 -- 172.21.15.27:0/594615959 learned_addr learned my addr 172.21.15.27:0/594615959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.888+0000 7f5cead9d640 1 --2- 172.21.15.27:0/594615959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cea59c640 1 -- 172.21.15.27:0/594615959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 msgr2=0x565326b37a00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:42:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cea59c640 1 --2- 172.21.15.27:0/594615959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 0x565326b37a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cea59c640 1 -- 172.21.15.27:0/594615959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 msgr2=0x565326b112c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cea59c640 1 --2- 172.21.15.27:0/594615959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b112c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cea59c640 1 -- 172.21.15.27:0/594615959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565326b37f40 con 0x56532674c2f0 2024-04-25T13:42:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cea59c640 1 --2- 172.21.15.27:0/594615959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56532674c2f0 0x565326b1a0e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc031650 tx=0x7f5cdc031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbde77fe45b90b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cd3fff640 1 -- 172.21.15.27:0/594615959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc035020 con 0x56532674c2f0 2024-04-25T13:42:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cd3fff640 1 -- 172.21.15.27:0/594615959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cdc040040 con 0x56532674c2f0 2024-04-25T13:42:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.889+0000 7f5cd3fff640 1 -- 172.21.15.27:0/594615959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc045470 con 0x56532674c2f0 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.890+0000 7f5cf7e25740 1 -- 172.21.15.27:0/594615959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56532674c2f0 msgr2=0x565326b1a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.890+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/594615959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56532674c2f0 0x565326b1a0e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc031650 tx=0x7f5cdc031400 comp rx=0 tx=0).stop 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.890+0000 7f5cf7e25740 1 -- 172.21.15.27:0/594615959 shutdown_connections 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.890+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/594615959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653267751b0 0x565326b112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.890+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/594615959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 0x565326b37a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.890+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/594615959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56532674c2f0 0x565326b1a0e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.890+0000 7f5cf7e25740 1 -- 172.21.15.27:0/594615959 >> 172.21.15.27:0/594615959 conn(0x565326769300 msgr2=0x565326b10bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.891+0000 7f5cf7e25740 1 -- 172.21.15.27:0/594615959 shutdown_connections 2024-04-25T13:42:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.891+0000 7f5cf7e25740 1 -- 172.21.15.27:0/594615959 wait complete. 2024-04-25T13:42:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.891+0000 7f5cf7e25740 1 -- start start 2024-04-25T13:42:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cf7e25740 1 Processor -- start 2024-04-25T13:42:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56532674c2f0 0x565326b331a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 0x565326b336e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cf7e25740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267751b0 0x565326b36620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec083b00 con 0x5653267751b0 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec1a9110 con 0x56532674c2f0 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cf7e25740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec0841d0 con 0x5653267544d0 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cf7e25740 10 client.0 _ll_register_callbacks cb 0x7f5ce0009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.892+0000 7f5cea59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56532674c2f0 0x565326b331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267751b0 0x565326b36620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267751b0 0x565326b36620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46414/0 (socket says 172.21.15.27:46414) 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cead9d640 1 -- 172.21.15.27:0/1083356239 learned_addr learned my addr 172.21.15.27:0/1083356239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 0x565326b336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cea59c640 1 -- 172.21.15.27:0/1083356239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 msgr2=0x565326b336e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cea59c640 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 0x565326b336e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cea59c640 1 -- 172.21.15.27:0/1083356239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267751b0 msgr2=0x565326b36620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cea59c640 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267751b0 0x565326b36620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cea59c640 1 -- 172.21.15.27:0/1083356239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565326b36be0 con 0x56532674c2f0 2024-04-25T13:42:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.893+0000 7f5cead9d640 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267751b0 0x565326b36620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cea59c640 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56532674c2f0 0x565326b331a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc045950 tx=0x7f5cdc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80087285d50db9f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/1083356239 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc05e070 con 0x56532674c2f0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/1083356239 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5cdc045680 con 0x56532674c2f0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 1 -- 172.21.15.27:0/1083356239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cdc040040 con 0x56532674c2f0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/1083356239 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f5cdc0035f0 con 0x56532674c2f0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 10 client.25334 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 1 -- 172.21.15.27:0/1083356239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x565326bb71f0 con 0x56532674c2f0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/1083356239 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cdc040040 con 0x56532674c2f0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/1083356239 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f5cdc03c7e0 con 0x56532674c2f0 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 10 client.25334 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 10 client.25334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 20 client.25334 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653267091c0 0x565326bb9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 4 client.25334 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:42:27.897 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.894+0000 7f5cf7e25740 1 -- 172.21.15.27:0/1083356239 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565326a82ee0 con 0x5653267091c0 2024-04-25T13:42:27.897 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.895+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653267091c0 0x565326bb9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:27.897 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.895+0000 7f5ce9d9b640 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653267091c0 0x565326bb9a30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=685068674cac9604 server_cookie=f8a1d22e2d585e1d in_seq=0 out_seq=0 2024-04-25T13:42:27.897 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.895+0000 7f5cd1ffb640 10 client.25334 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cd1ffb640 1 -- 172.21.15.27:0/1083356239 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565326a82ee0 con 0x5653267091c0 2024-04-25T13:42:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cd1ffb640 10 client.25334 handle_command_reply: tid=0 2024-04-25T13:42:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 client.25334 shutdown 2024-04-25T13:42:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 -- 172.21.15.27:0/1083356239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56532674c2f0 msgr2=0x565326b331a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56532674c2f0 0x565326b331a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc045950 tx=0x7f5cdc005fd0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 -- 172.21.15.27:0/1083356239 shutdown_connections 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653267091c0 0x565326bb9a30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653267751b0 0x565326b36620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cd1ffb640 0 client.25334 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653267544d0 0x565326b336e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 --2- 172.21.15.27:0/1083356239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56532674c2f0 0x565326b331a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 -- 172.21.15.27:0/1083356239 >> 172.21.15.27:0/1083356239 conn(0x565326769300 msgr2=0x565326b42620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 -- 172.21.15.27:0/1083356239 shutdown_connections 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 1 -- 172.21.15.27:0/1083356239 wait complete. 2024-04-25T13:42:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.909+0000 7f5cf7e25740 20 client.25334 trim_cache size 0 max 16384 2024-04-25T13:42:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:42:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:42:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:42:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5cd40afae0 msgr2=0x7f5cd40b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5cd40afae0 0x7f5cd40b1ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x565326b16b40 tx=0x7f5ce0004730 comp rx=0 tx=0).stop 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cec110060 msgr2=0x7f5cec1a7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cec110060 0x7f5cec1a7d40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4005fd0 tx=0x7f5ce400ef10 comp rx=0 tx=0).stop 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 shutdown_connections 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5cd40afae0 0x7f5cd40b1ff0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cec110060 0x7f5cec1a7d40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cec10b360 0x7f5cec1a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 --2- 172.21.15.27:0/1816835669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cec10a320 0x7f5cec1a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 >> 172.21.15.27:0/1816835669 conn(0x7f5cec078de0 msgr2=0x7f5cec10c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 shutdown_connections 2024-04-25T13:42:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:27.910+0000 7f5cf0f72640 1 -- 172.21.15.27:0/1816835669 wait complete. 2024-04-25T13:42:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 80.0 s, 3720.0 tps, lat 26.873 ms stddev 27.266 2024-04-25T13:42:28.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:28 smithi027 ceph-mon[29140]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 874 op/s 2024-04-25T13:42:28.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:28 smithi122 ceph-mon[32843]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 874 op/s 2024-04-25T13:42:28.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:28 smithi045 ceph-mon[32731]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 874 op/s 2024-04-25T13:42:30.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:30 smithi027 ceph-mon[29140]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.7 MiB/s wr, 650 op/s 2024-04-25T13:42:30.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:30 smithi122 ceph-mon[32843]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.7 MiB/s wr, 650 op/s 2024-04-25T13:42:30.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:30 smithi045 ceph-mon[32731]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.7 MiB/s wr, 650 op/s 2024-04-25T13:42:30.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:42:30.640 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:42:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.811+0000 7f9eda18f640 1 -- 172.21.15.27:0/2106645978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed4103c40 msgr2=0x7f9ed4104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.811+0000 7f9eda18f640 1 --2- 172.21.15.27:0/2106645978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed4103c40 0x7f9ed4104050 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009ca0 tx=0x7f9ecc02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.812+0000 7f9eda18f640 1 -- 172.21.15.27:0/2106645978 shutdown_connections 2024-04-25T13:42:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.812+0000 7f9eda18f640 1 --2- 172.21.15.27:0/2106645978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed4108940 0x7f9ed410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.812+0000 7f9eda18f640 1 --2- 172.21.15.27:0/2106645978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed4103c40 0x7f9ed4104050 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.812+0000 7f9eda18f640 1 --2- 172.21.15.27:0/2106645978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ed4102a00 0x7f9ed4103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.812+0000 7f9eda18f640 1 -- 172.21.15.27:0/2106645978 >> 172.21.15.27:0/2106645978 conn(0x7f9ed40fdbf0 msgr2=0x7f9ed41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.812+0000 7f9eda18f640 1 -- 172.21.15.27:0/2106645978 shutdown_connections 2024-04-25T13:42:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.812+0000 7f9eda18f640 1 -- 172.21.15.27:0/2106645978 wait complete. 2024-04-25T13:42:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.813+0000 7f9eda18f640 1 Processor -- start 2024-04-25T13:42:30.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.814+0000 7f9eda18f640 1 -- start start 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9eda18f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed4102a00 0x7f9ed4199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9eda18f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ed4103c40 0x7f9ed4199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9ed37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed4102a00 0x7f9ed4199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9ed37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed4102a00 0x7f9ed4199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56700/0 (socket says 172.21.15.27:56700) 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9ed37fe640 1 -- 172.21.15.27:0/3582697709 learned_addr learned my addr 172.21.15.27:0/3582697709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9eda18f640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed4108940 0x7f9ed41a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed410e520 con 0x7f9ed4108940 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed410e3a0 con 0x7f9ed4102a00 2024-04-25T13:42:30.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.815+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed410e6a0 con 0x7f9ed4103c40 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.816+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ed4103c40 0x7f9ed4199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.816+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed4108940 0x7f9ed41a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.816+0000 7f9ed37fe640 1 -- 172.21.15.27:0/3582697709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ed4103c40 msgr2=0x7f9ed4199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.816+0000 7f9ed37fe640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ed4103c40 0x7f9ed4199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.816+0000 7f9ed37fe640 1 -- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed4108940 msgr2=0x7f9ed41a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.816+0000 7f9ed37fe640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed4108940 0x7f9ed41a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.816+0000 7f9ed37fe640 1 -- 172.21.15.27:0/3582697709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed4074dd0 con 0x7f9ed4102a00 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.816+0000 7f9ed37fe640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed4102a00 0x7f9ed4199120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8005d00 tx=0x7f9ec800dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.817+0000 7f9ed0ff9640 1 -- 172.21.15.27:0/3582697709 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8018070 con 0x7f9ed4102a00 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.817+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed40750c0 con 0x7f9ed4102a00 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.817+0000 7f9ed0ff9640 1 -- 172.21.15.27:0/3582697709 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ec8009c30 con 0x7f9ed4102a00 2024-04-25T13:42:30.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.817+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed4075610 con 0x7f9ed4102a00 2024-04-25T13:42:30.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.817+0000 7f9ed0ff9640 1 -- 172.21.15.27:0/3582697709 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec801c940 con 0x7f9ed4102a00 2024-04-25T13:42:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ee1040740 1 Processor -- start 2024-04-25T13:42:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ee1040740 1 -- start start 2024-04-25T13:42:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540f9390 0x55975449aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d04d0 0x55975449afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d86b0 0x55975449bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ee1040740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5597543ffd80 con 0x5597540f9390 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ee1040740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5597543b98b0 con 0x5597540d04d0 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ee1040740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5597543fbd40 con 0x5597540d86b0 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ed0ff9640 1 -- 172.21.15.27:0/3582697709 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9ec801cae0 con 0x7f9ed4102a00 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ed3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540f9390 0x55975449aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ed3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540f9390 0x55975449aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46452/0 (socket says 172.21.15.27:46452) 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.820+0000 7f9ed3fff640 1 -- 172.21.15.27:0/3242644301 learned_addr learned my addr 172.21.15.27:0/3242644301 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d86b0 0x55975449bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed37fe640 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d04d0 0x55975449afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed3fff640 1 -- 172.21.15.27:0/3242644301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d86b0 msgr2=0x55975449bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d86b0 0x55975449bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed3fff640 1 -- 172.21.15.27:0/3242644301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d04d0 msgr2=0x55975449afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:30.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d04d0 0x55975449afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed3fff640 1 -- 172.21.15.27:0/3242644301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55975438aa40 con 0x5597540f9390 2024-04-25T13:42:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed37fe640 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d04d0 0x55975449afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:42:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540f9390 0x55975449aa60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009c10 tx=0x7f9ec40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252a5b296ae4fe45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9eb9ffb640 1 -- 172.21.15.27:0/3242644301 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4012450 con 0x5597540f9390 2024-04-25T13:42:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9eb9ffb640 1 -- 172.21.15.27:0/3242644301 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ec4012aa0 con 0x5597540f9390 2024-04-25T13:42:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9eb9ffb640 1 -- 172.21.15.27:0/3242644301 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400fa50 con 0x5597540f9390 2024-04-25T13:42:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.821+0000 7f9ed0ff9640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ebc0afb70 0x7f9ebc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ed37fe640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ebc0afb70 0x7f9ebc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ed0ff9640 1 -- 172.21.15.27:0/3582697709 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9ec80e28a0 con 0x7f9ed4102a00 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ee1040740 1 -- 172.21.15.27:0/3242644301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540f9390 msgr2=0x55975449aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ed37fe640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ebc0afb70 0x7f9ebc0b2080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55975449a780 tx=0x7f9ec80e8280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540f9390 0x55975449aa60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009c10 tx=0x7f9ec40117b0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ee1040740 1 -- 172.21.15.27:0/3242644301 shutdown_connections 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d86b0 0x55975449bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d04d0 0x55975449afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3242644301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540f9390 0x55975449aa60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.822+0000 7f9ee1040740 1 -- 172.21.15.27:0/3242644301 >> 172.21.15.27:0/3242644301 conn(0x5597540ed4e0 msgr2=0x559754496270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.823+0000 7f9ee1040740 1 -- 172.21.15.27:0/3242644301 shutdown_connections 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.823+0000 7f9ee1040740 1 -- 172.21.15.27:0/3242644301 wait complete. 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.823+0000 7f9ee1040740 1 -- start start 2024-04-25T13:42:30.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.823+0000 7f9ee1040740 1 Processor -- start 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.824+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 0x5597544ad100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.824+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x5597544ad640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 0x5597544b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ee1040740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55975449c780 con 0x5597540d86b0 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ee1040740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559754386d10 con 0x5597540f9390 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ee1040740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55975449c900 con 0x5597540d04d0 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ee1040740 10 client.0 _ll_register_callbacks cb 0x5597543b3210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ed2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x5597544ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ed3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 0x5597544ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ed2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x5597544ad640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46458/0 (socket says 172.21.15.27:46458) 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ed2ffd640 1 -- 172.21.15.27:0/3717955951 learned_addr learned my addr 172.21.15.27:0/3717955951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:30.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 0x5597544b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ed2ffd640 1 -- 172.21.15.27:0/3717955951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 msgr2=0x5597544ad100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.825+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 0x5597544ad100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9ed2ffd640 1 -- 172.21.15.27:0/3717955951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 msgr2=0x5597544b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 0x5597544b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9ed2ffd640 1 -- 172.21.15.27:0/3717955951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597541db7d0 con 0x5597540d86b0 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x5597544ad640 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009ca0 tx=0x7f9ecc02ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0338aebef15491 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9eb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/3717955951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc003b80 con 0x5597540d86b0 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/3717955951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9ecc003d20 con 0x5597540d86b0 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9ee1040740 1 -- 172.21.15.27:0/3717955951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ec4012aa0 con 0x5597540d86b0 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/3717955951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9ecc004b40 con 0x5597540d86b0 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9ee1040740 10 client.15612 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:30.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.826+0000 7f9ee1040740 1 -- 172.21.15.27:0/3717955951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5597541113f0 con 0x5597540d86b0 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/3717955951 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ec4012aa0 con 0x5597540d86b0 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/3717955951 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9ecc02f660 con 0x5597540d86b0 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9ee1040740 10 client.15612 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9ee1040740 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9ee1040740 20 client.15612 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5597540b7f60 0x5597544baad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9ee1040740 4 client.15612 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9ee1040740 1 -- 172.21.15.27:0/3717955951 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55975440d990 con 0x5597540b7f60 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.827+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5597540b7f60 0x5597544baad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.828+0000 7f9eb2ffd640 10 client.15612 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.828+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5597540b7f60 0x5597544baad0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=beefa35ad1df9a59 server_cookie=797042d28ca6cd28 in_seq=0 out_seq=0 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/3717955951 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55975440d990 con 0x5597540b7f60 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9eb2ffd640 10 client.15612 handle_command_reply: tid=0 2024-04-25T13:42:30.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 client.15612 shutdown 2024-04-25T13:42:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 -- 172.21.15.27:0/3717955951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 msgr2=0x5597544ad640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x5597544ad640 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009ca0 tx=0x7f9ecc02ae60 comp rx=0 tx=0).stop 2024-04-25T13:42:30.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 -- 172.21.15.27:0/3717955951 shutdown_connections 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5597540b7f60 0x5597544baad0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 0x5597544b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x5597544ad640 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9eb2ffd640 0 client.15612 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 --2- 172.21.15.27:0/3717955951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 0x5597544ad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.829+0000 7f9ee1040740 1 -- 172.21.15.27:0/3717955951 >> 172.21.15.27:0/3717955951 conn(0x5597540ed4e0 msgr2=0x5597544953f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.830+0000 7f9ee1040740 1 -- 172.21.15.27:0/3717955951 shutdown_connections 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.830+0000 7f9ee1040740 1 -- 172.21.15.27:0/3717955951 wait complete. 2024-04-25T13:42:30.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.830+0000 7f9ee1040740 20 client.15612 trim_cache size 0 max 16384 2024-04-25T13:42:30.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.850+0000 7f9ee1040740 1 Processor -- start 2024-04-25T13:42:30.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.850+0000 7f9ee1040740 1 -- start start 2024-04-25T13:42:30.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 0x5597544a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x559754499780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 0x5597544c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x559754499780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ee1040740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec000c390 con 0x5597540d86b0 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ee1040740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec000ec10 con 0x5597540f9390 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ee1040740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec000ca00 con 0x5597540d04d0 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x559754499780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46464/0 (socket says 172.21.15.27:46464) 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 -- 172.21.15.27:0/2566312335 learned_addr learned my addr 172.21.15.27:0/2566312335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 0x5597544a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 0x5597544c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 -- 172.21.15.27:0/2566312335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 msgr2=0x5597544a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 0x5597544a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 -- 172.21.15.27:0/2566312335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 msgr2=0x5597544c02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:30.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 0x5597544c02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 -- 172.21.15.27:0/2566312335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597540d10f0 con 0x5597540d86b0 2024-04-25T13:42:30.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.851+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x559754499780 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc02ade0 tx=0x7f9ecc000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6516a617cf8bb93a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:30.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.852+0000 7f9eb97fa640 1 -- 172.21.15.27:0/2566312335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc005cc0 con 0x5597540d86b0 2024-04-25T13:42:30.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.852+0000 7f9eb97fa640 1 -- 172.21.15.27:0/2566312335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ecc036040 con 0x5597540d86b0 2024-04-25T13:42:30.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.852+0000 7f9eb97fa640 1 -- 172.21.15.27:0/2566312335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc030a70 con 0x5597540d86b0 2024-04-25T13:42:30.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.852+0000 7f9ee1040740 1 -- 172.21.15.27:0/2566312335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 msgr2=0x559754499780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.852+0000 7f9ee1040740 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x559754499780 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc02ade0 tx=0x7f9ecc000f40 comp rx=0 tx=0).stop 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 -- 172.21.15.27:0/2566312335 shutdown_connections 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540f9390 0x5597544c02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d86b0 0x559754499780 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 --2- 172.21.15.27:0/2566312335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540d04d0 0x5597544a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 -- 172.21.15.27:0/2566312335 >> 172.21.15.27:0/2566312335 conn(0x5597540ed4e0 msgr2=0x559754499070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 -- 172.21.15.27:0/2566312335 shutdown_connections 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 -- 172.21.15.27:0/2566312335 wait complete. 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 -- start start 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 Processor -- start 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d04d0 0x5597544b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d86b0 0x5597544b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540f9390 0x559754520e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec000d0d0 con 0x5597540d04d0 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec000e540 con 0x5597540d86b0 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.853+0000 7f9ee1040740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec000d7a0 con 0x5597540f9390 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ee1040740 10 client.0 _ll_register_callbacks cb 0x7f9ecc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d04d0 0x5597544b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d04d0 0x5597544b1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46480/0 (socket says 172.21.15.27:46480) 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 -- 172.21.15.27:0/444810684 learned_addr learned my addr 172.21.15.27:0/444810684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d86b0 0x5597544b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540f9390 0x559754520e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 -- 172.21.15.27:0/444810684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540f9390 msgr2=0x559754520e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540f9390 0x559754520e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 -- 172.21.15.27:0/444810684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d86b0 msgr2=0x5597544b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d86b0 0x5597544b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 -- 172.21.15.27:0/444810684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597543c04b0 con 0x5597540d04d0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ed3fff640 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d04d0 0x5597544b1e60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009bb0 tx=0x7f9ec402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c255e87422d0831 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9eb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/444810684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4005530 con 0x5597540d04d0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/444810684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9ec4020040 con 0x5597540d04d0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9ee1040740 1 -- 172.21.15.27:0/444810684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ecc036040 con 0x5597540d04d0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.854+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/444810684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9ec4012450 con 0x5597540d04d0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ee1040740 10 client.15618 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ee1040740 1 -- 172.21.15.27:0/444810684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559754522460 con 0x5597540d04d0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/444810684 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ecc036040 con 0x5597540d04d0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/444810684 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9ec4026040 con 0x5597540d04d0 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ee1040740 10 client.15618 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:30.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ee1040740 10 client.15618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ee1040740 20 client.15618 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ee1040740 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55975408d3a0 0x559754524cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ee1040740 4 client.15618 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ee1040740 1 -- 172.21.15.27:0/444810684 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559754407800 con 0x55975408d3a0 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.855+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55975408d3a0 0x559754524cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9eb2ffd640 10 client.15618 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ed2ffd640 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55975408d3a0 0x559754524cf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5e246779d7af0ec server_cookie=ca4001b8b76548c7 in_seq=0 out_seq=0 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9eb2ffd640 1 -- 172.21.15.27:0/444810684 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559754407800 con 0x55975408d3a0 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9eb2ffd640 10 client.15618 handle_command_reply: tid=0 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 client.15618 shutdown 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 -- 172.21.15.27:0/444810684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d04d0 msgr2=0x5597544b1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d04d0 0x5597544b1e60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009bb0 tx=0x7f9ec402c040 comp rx=0 tx=0).stop 2024-04-25T13:42:30.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 -- 172.21.15.27:0/444810684 shutdown_connections 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55975408d3a0 0x559754524cf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5597540f9390 0x559754520e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9eb2ffd640 0 client.15618 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5597540d86b0 0x5597544b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 --2- 172.21.15.27:0/444810684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5597540d04d0 0x5597544b1e60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 -- 172.21.15.27:0/444810684 >> 172.21.15.27:0/444810684 conn(0x5597540ed4e0 msgr2=0x5597544bf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 -- 172.21.15.27:0/444810684 shutdown_connections 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 1 -- 172.21.15.27:0/444810684 wait complete. 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.856+0000 7f9ee1040740 20 client.15618 trim_cache size 0 max 16384 2024-04-25T13:42:30.863 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:42:30.864 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:42:30.864 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:42:30.864 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ebc0afb70 msgr2=0x7f9ebc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ebc0afb70 0x7f9ebc0b2080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55975449a780 tx=0x7f9ec80e8280 comp rx=0 tx=0).stop 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed4102a00 msgr2=0x7f9ed4199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed4102a00 0x7f9ed4199120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8005d00 tx=0x7f9ec800dd20 comp rx=0 tx=0).stop 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 shutdown_connections 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9ebc0afb70 0x7f9ebc0b2080 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ed4108940 0x7f9ed41a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ed4103c40 0x7f9ed4199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 --2- 172.21.15.27:0/3582697709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed4102a00 0x7f9ed4199120 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 >> 172.21.15.27:0/3582697709 conn(0x7f9ed40fdbf0 msgr2=0x7f9ed4104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 shutdown_connections 2024-04-25T13:42:30.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:30.857+0000 7f9eda18f640 1 -- 172.21.15.27:0/3582697709 wait complete. 2024-04-25T13:42:30.873 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:42:30.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:42:31.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.044+0000 7ff7505d7640 1 -- 172.21.15.27:0/3003014851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff748103b00 msgr2=0x7ff748104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.044+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3003014851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff748103b00 0x7ff748104800 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff744009900 tx=0x7ff74402a3f0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.045+0000 7ff7505d7640 1 -- 172.21.15.27:0/3003014851 shutdown_connections 2024-04-25T13:42:31.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.045+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3003014851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff748109a40 0x7ff74810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.045+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3003014851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748104d40 0x7ff748105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.045+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3003014851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff748103b00 0x7ff748104800 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.045+0000 7ff7505d7640 1 -- 172.21.15.27:0/3003014851 >> 172.21.15.27:0/3003014851 conn(0x7ff7480fed30 msgr2=0x7ff7481011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.045+0000 7ff7505d7640 1 -- 172.21.15.27:0/3003014851 shutdown_connections 2024-04-25T13:42:31.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.045+0000 7ff7505d7640 1 -- 172.21.15.27:0/3003014851 wait complete. 2024-04-25T13:42:31.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff7505d7640 1 Processor -- start 2024-04-25T13:42:31.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff7505d7640 1 -- start start 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff7505d7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748103b00 0x7ff748076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff7505d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff748104d40 0x7ff748074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff7505d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff748109a40 0x7ff748075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff7505d7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff74810f650 con 0x7ff748103b00 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff7505d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff74810f4d0 con 0x7ff748104d40 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff7505d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff74810f7d0 con 0x7ff748109a40 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff74e373640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748103b00 0x7ff748076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff74e373640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748103b00 0x7ff748076850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46508/0 (socket says 172.21.15.27:46508) 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff74db72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff748104d40 0x7ff748074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff74e373640 1 -- 172.21.15.27:0/3182847 learned_addr learned my addr 172.21.15.27:0/3182847 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.046+0000 7ff74eb74640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff748109a40 0x7ff748075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff74db72640 1 -- 172.21.15.27:0/3182847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff748109a40 msgr2=0x7ff748075430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff74db72640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff748109a40 0x7ff748075430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff74db72640 1 -- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748103b00 msgr2=0x7ff748076850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff74db72640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748103b00 0x7ff748076850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff74db72640 1 -- 172.21.15.27:0/3182847 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff748075b60 con 0x7ff748104d40 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff74db72640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff748104d40 0x7ff748074ef0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff73800e9f0 tx=0x7ff73800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff73f7fe640 1 -- 172.21.15.27:0/3182847 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73800ccc0 con 0x7ff748104d40 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7481a42b0 con 0x7ff748104d40 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff73f7fe640 1 -- 172.21.15.27:0/3182847 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7380108e0 con 0x7ff748104d40 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff73f7fe640 1 -- 172.21.15.27:0/3182847 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738018880 con 0x7ff748104d40 2024-04-25T13:42:31.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.047+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7481a4770 con 0x7ff748104d40 2024-04-25T13:42:31.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.049+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff74806afa0 con 0x7ff748104d40 2024-04-25T13:42:31.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.050+0000 7ff73f7fe640 1 -- 172.21.15.27:0/3182847 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff738010430 con 0x7ff748104d40 2024-04-25T13:42:31.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.051+0000 7ff73f7fe640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7340afb30 0x7ff7340b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.051+0000 7ff74e373640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7340afb30 0x7ff7340b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.051+0000 7ff73f7fe640 1 -- 172.21.15.27:0/3182847 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff7380def00 con 0x7ff748104d40 2024-04-25T13:42:31.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.051+0000 7ff74e373640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7340afb30 0x7ff7340b2040 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff744000c00 tx=0x7ff744005a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:31.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.058+0000 7ff73f7fe640 1 -- 172.21.15.27:0/3182847 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7380a74c0 con 0x7ff748104d40 2024-04-25T13:42:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.418+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff748108850 con 0x7ff748104d40 2024-04-25T13:42:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.419+0000 7ff73f7fe640 1 -- 172.21.15.27:0/3182847 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7ff7380ab7c0 con 0x7ff748104d40 2024-04-25T13:42:31.420 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:42:31.420 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:42:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:42:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7340afb30 msgr2=0x7ff7340b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7340afb30 0x7ff7340b2040 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff744000c00 tx=0x7ff744005a40 comp rx=0 tx=0).stop 2024-04-25T13:42:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff748104d40 msgr2=0x7ff748074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff748104d40 0x7ff748074ef0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff73800e9f0 tx=0x7ff73800eeb0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 shutdown_connections 2024-04-25T13:42:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7340afb30 0x7ff7340b2040 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff748109a40 0x7ff748075430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff748104d40 0x7ff748074ef0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 --2- 172.21.15.27:0/3182847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff748103b00 0x7ff748076850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 >> 172.21.15.27:0/3182847 conn(0x7ff7480fed30 msgr2=0x7ff748105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 shutdown_connections 2024-04-25T13:42:31.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.423+0000 7ff7505d7640 1 -- 172.21.15.27:0/3182847 wait complete. 2024-04-25T13:42:31.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:42:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.608+0000 7fc19eb96640 1 -- 172.21.15.27:0/3774358733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc198103c30 msgr2=0x7fc198104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.608+0000 7fc19eb96640 1 --2- 172.21.15.27:0/3774358733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc198103c30 0x7fc198104040 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc188009900 tx=0x7fc18802a3f0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.609+0000 7fc19eb96640 1 -- 172.21.15.27:0/3774358733 shutdown_connections 2024-04-25T13:42:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.609+0000 7fc19eb96640 1 --2- 172.21.15.27:0/3774358733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198108930 0x7fc19810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.609+0000 7fc19eb96640 1 --2- 172.21.15.27:0/3774358733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc198103c30 0x7fc198104040 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.609+0000 7fc19eb96640 1 --2- 172.21.15.27:0/3774358733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1981029f0 0x7fc1981036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.609+0000 7fc19eb96640 1 -- 172.21.15.27:0/3774358733 >> 172.21.15.27:0/3774358733 conn(0x7fc1980fdbe0 msgr2=0x7fc198100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.609+0000 7fc19eb96640 1 -- 172.21.15.27:0/3774358733 shutdown_connections 2024-04-25T13:42:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.609+0000 7fc19eb96640 1 -- 172.21.15.27:0/3774358733 wait complete. 2024-04-25T13:42:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19eb96640 1 Processor -- start 2024-04-25T13:42:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19eb96640 1 -- start start 2024-04-25T13:42:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19eb96640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1981029f0 0x7fc19806bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19eb96640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198103c30 0x7fc1981a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19eb96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc198108930 0x7fc1981a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19c932640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1981029f0 0x7fc19806bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19c932640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1981029f0 0x7fc19806bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44640/0 (socket says 172.21.15.27:44640) 2024-04-25T13:42:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19c932640 1 -- 172.21.15.27:0/941493848 learned_addr learned my addr 172.21.15.27:0/941493848 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.610+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc19810e540 con 0x7fc198108930 2024-04-25T13:42:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.611+0000 7fc18ffff640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198103c30 0x7fc1981a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.611+0000 7fc19d133640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc198108930 0x7fc1981a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.611+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc19810e3c0 con 0x7fc198103c30 2024-04-25T13:42:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.611+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc19810e6c0 con 0x7fc1981029f0 2024-04-25T13:42:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.612+0000 7fc19c932640 1 -- 172.21.15.27:0/941493848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198103c30 msgr2=0x7fc1981a1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.612+0000 7fc19c932640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198103c30 0x7fc1981a1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.612+0000 7fc19c932640 1 -- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc198108930 msgr2=0x7fc1981a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.612+0000 7fc19c932640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc198108930 0x7fc1981a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.612+0000 7fc19c932640 1 -- 172.21.15.27:0/941493848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1981a3fe0 con 0x7fc1981029f0 2024-04-25T13:42:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.612+0000 7fc19c932640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1981029f0 0x7fc19806bd20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc19400e980 tx=0x7fc19400ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:31.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.613+0000 7fc18dffb640 1 -- 172.21.15.27:0/941493848 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19400ccc0 con 0x7fc1981029f0 2024-04-25T13:42:31.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.613+0000 7fc18dffb640 1 -- 172.21.15.27:0/941493848 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1940108e0 con 0x7fc1981029f0 2024-04-25T13:42:31.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.613+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1981a42d0 con 0x7fc1981029f0 2024-04-25T13:42:31.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.613+0000 7fc18dffb640 1 -- 172.21.15.27:0/941493848 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc194018840 con 0x7fc1981029f0 2024-04-25T13:42:31.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.613+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1981a4870 con 0x7fc1981029f0 2024-04-25T13:42:31.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.616+0000 7fc18dffb640 1 -- 172.21.15.27:0/941493848 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc19401f020 con 0x7fc1981029f0 2024-04-25T13:42:31.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc18dffb640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1800afb70 0x7fc1800b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc1a5a48740 1 Processor -- start 2024-04-25T13:42:31.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc1a5a48740 1 -- start start 2024-04-25T13:42:31.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc18dffb640 1 -- 172.21.15.27:0/941493848 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc1940ded10 con 0x7fc1981029f0 2024-04-25T13:42:31.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc18ffff640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1800afb70 0x7fc1800b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b7b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 0x55f6f3b7bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b7ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f6f3adfd80 con 0x55f6f37b86b0 2024-04-25T13:42:31.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f6f3a99a20 con 0x55f6f37b04d0 2024-04-25T13:42:31.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f6f3adbd40 con 0x55f6f37d9390 2024-04-25T13:42:31.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.619+0000 7fc19c932640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b7b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.620+0000 7fc19c932640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b7b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44646/0 (socket says 172.21.15.27:44646) 2024-04-25T13:42:31.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.620+0000 7fc19c932640 1 -- 172.21.15.27:0/4022828709 learned_addr learned my addr 172.21.15.27:0/4022828709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:31.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.620+0000 7fc19d133640 1 --2- 172.21.15.27:0/4022828709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b7ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.620+0000 7fc19c932640 1 -- 172.21.15.27:0/4022828709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 msgr2=0x55f6f3b7bd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:42:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.620+0000 7fc19c932640 1 --2- 172.21.15.27:0/4022828709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 0x55f6f3b7bd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.620+0000 7fc19c932640 1 -- 172.21.15.27:0/4022828709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 msgr2=0x55f6f3b7ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.620+0000 7fc19c932640 1 --2- 172.21.15.27:0/4022828709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b7ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.620+0000 7fc19c932640 1 -- 172.21.15.27:0/4022828709 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6f3aa4c30 con 0x55f6f37d9390 2024-04-25T13:42:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.621+0000 7fc19c932640 1 --2- 172.21.15.27:0/4022828709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b7b810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc1940a4490 tx=0x7fc1940a6d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd683b5c1685544f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.621+0000 7fc18ffff640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1800afb70 0x7fc1800b2080 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc188006010 tx=0x7fc188031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.621+0000 7fc17effd640 1 -- 172.21.15.27:0/4022828709 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1940ab5d0 con 0x55f6f37d9390 2024-04-25T13:42:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.621+0000 7fc17effd640 1 -- 172.21.15.27:0/4022828709 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1940b0080 con 0x55f6f37d9390 2024-04-25T13:42:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.621+0000 7fc17effd640 1 -- 172.21.15.27:0/4022828709 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1940afe50 con 0x55f6f37d9390 2024-04-25T13:42:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 -- 172.21.15.27:0/4022828709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 msgr2=0x55f6f3b7b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/4022828709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b7b810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc1940a4490 tx=0x7fc1940a6d30 comp rx=0 tx=0).stop 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 -- 172.21.15.27:0/4022828709 shutdown_connections 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/4022828709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b7ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/4022828709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 0x55f6f3b7bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/4022828709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b7b810 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 -- 172.21.15.27:0/4022828709 >> 172.21.15.27:0/4022828709 conn(0x55f6f37cd4e0 msgr2=0x55f6f3b77020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 -- 172.21.15.27:0/4022828709 shutdown_connections 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 -- 172.21.15.27:0/4022828709 wait complete. 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.622+0000 7fc1a5a48740 1 -- start start 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.623+0000 7fc1a5a48740 1 Processor -- start 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.623+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b85610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.623+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b85b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.624+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b8cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.624+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f6f3b7d5d0 con 0x55f6f37b04d0 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.624+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f6f3a66dc0 con 0x55f6f37b86b0 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.624+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f6f3b7d750 con 0x55f6f37d9390 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19d133640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b8cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19d133640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b8cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44656/0 (socket says 172.21.15.27:44656) 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19d133640 1 -- 172.21.15.27:0/1668494463 learned_addr learned my addr 172.21.15.27:0/1668494463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc18ffff640 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b85b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19c932640 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b85610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc1a5a48740 10 client.0 _ll_register_callbacks cb 0x55f6f3a82600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19c932640 1 -- 172.21.15.27:0/1668494463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 msgr2=0x55f6f3b8cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19c932640 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b8cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19c932640 1 -- 172.21.15.27:0/1668494463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 msgr2=0x55f6f3b85b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19c932640 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b85b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19c932640 1 -- 172.21.15.27:0/1668494463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6f37f13f0 con 0x55f6f37b04d0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19d133640 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b8cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.625+0000 7fc19c932640 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b85610 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc1940dcbb0 tx=0x7fc1940a3700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc8378d844fd66e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.626+0000 7fc177fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.626+0000 7fc177fff640 1 -- 172.21.15.27:0/1668494463 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1940ab5d0 con 0x55f6f37b04d0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.626+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1668494463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1940b0080 con 0x55f6f37b04d0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.626+0000 7fc177fff640 1 -- 172.21.15.27:0/1668494463 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc1940a3810 con 0x55f6f37b04d0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.626+0000 7fc177fff640 1 -- 172.21.15.27:0/1668494463 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc1940ab120 con 0x55f6f37b04d0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.627+0000 7fc1a5a48740 10 client.15636 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.627+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1668494463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f6f38bb7d0 con 0x55f6f37b04d0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.627+0000 7fc177fff640 1 -- 172.21.15.27:0/1668494463 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1940b0080 con 0x55f6f37b04d0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.627+0000 7fc177fff640 1 -- 172.21.15.27:0/1668494463 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc1940c1b10 con 0x55f6f37b04d0 2024-04-25T13:42:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.628+0000 7fc1a5a48740 10 client.15636 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.628+0000 7fc1a5a48740 10 client.15636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.628+0000 7fc1a5a48740 20 client.15636 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.628+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f6f3797f60 0x55f6f3b933a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.628+0000 7fc1a5a48740 4 client.15636 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.628+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1668494463 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6f3aed990 con 0x55f6f3797f60 2024-04-25T13:42:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.628+0000 7fc18ffff640 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f6f3797f60 0x55f6f3b933a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.630+0000 7fc18ffff640 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f6f3797f60 0x55f6f3b933a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c009e5049310d646 server_cookie=58b5858d87d164fe in_seq=0 out_seq=0 2024-04-25T13:42:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.630+0000 7fc177fff640 10 client.15636 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc177fff640 1 -- 172.21.15.27:0/1668494463 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f6f3aed990 con 0x55f6f3797f60 2024-04-25T13:42:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc177fff640 10 client.15636 handle_command_reply: tid=0 2024-04-25T13:42:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 client.15636 shutdown 2024-04-25T13:42:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1668494463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 msgr2=0x55f6f3b85610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b85610 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc1940dcbb0 tx=0x7fc1940a3700 comp rx=0 tx=0).stop 2024-04-25T13:42:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1668494463 shutdown_connections 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f6f3797f60 0x55f6f3b933a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b8cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b85b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1668494463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b85610 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1668494463 >> 172.21.15.27:0/1668494463 conn(0x55f6f37cd4e0 msgr2=0x55f6f3b76040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc177fff640 0 client.15636 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1668494463 shutdown_connections 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1668494463 wait complete. 2024-04-25T13:42:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.631+0000 7fc1a5a48740 20 client.15636 trim_cache size 0 max 16384 2024-04-25T13:42:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.654+0000 7fc1a5a48740 1 Processor -- start 2024-04-25T13:42:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.654+0000 7fc1a5a48740 1 -- start start 2024-04-25T13:42:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.654+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 0x55f6f3b802f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.654+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b80830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.654+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b848a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.654+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981ab480 con 0x55f6f37b86b0 2024-04-25T13:42:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.654+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981add00 con 0x55f6f37b04d0 2024-04-25T13:42:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.654+0000 7fc1a5a48740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981abaf0 con 0x55f6f37d9390 2024-04-25T13:42:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc18ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b80830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc18ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b80830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46582/0 (socket says 172.21.15.27:46582) 2024-04-25T13:42:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc19c932640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 0x55f6f3b802f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc18ffff640 1 -- 172.21.15.27:0/3928974201 learned_addr learned my addr 172.21.15.27:0/3928974201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc19c932640 1 -- 172.21.15.27:0/3928974201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 msgr2=0x55f6f3b848a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc19c932640 1 --2- 172.21.15.27:0/3928974201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b848a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc19c932640 1 -- 172.21.15.27:0/3928974201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 msgr2=0x55f6f3b80830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc19c932640 1 --2- 172.21.15.27:0/3928974201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b80830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc19c932640 1 -- 172.21.15.27:0/3928974201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6f3b7c7a0 con 0x55f6f37b04d0 2024-04-25T13:42:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.655+0000 7fc19c932640 1 --2- 172.21.15.27:0/3928974201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 0x55f6f3b802f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc194000c00 tx=0x7fc1940a6d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c9e6291523a5829 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.656+0000 7fc17e7fc640 1 -- 172.21.15.27:0/3928974201 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1940a30e0 con 0x55f6f37b04d0 2024-04-25T13:42:31.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.656+0000 7fc17e7fc640 1 -- 172.21.15.27:0/3928974201 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1940a2f00 con 0x55f6f37b04d0 2024-04-25T13:42:31.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.656+0000 7fc17e7fc640 1 -- 172.21.15.27:0/3928974201 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1940a2be0 con 0x55f6f37b04d0 2024-04-25T13:42:31.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.657+0000 7fc1a5a48740 1 -- 172.21.15.27:0/3928974201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 msgr2=0x55f6f3b802f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.657+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/3928974201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 0x55f6f3b802f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc194000c00 tx=0x7fc1940a6d30 comp rx=0 tx=0).stop 2024-04-25T13:42:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.657+0000 7fc1a5a48740 1 -- 172.21.15.27:0/3928974201 shutdown_connections 2024-04-25T13:42:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.657+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/3928974201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3b848a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.657+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/3928974201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b86b0 0x55f6f3b80830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.657+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/3928974201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b04d0 0x55f6f3b802f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.657+0000 7fc1a5a48740 1 -- 172.21.15.27:0/3928974201 >> 172.21.15.27:0/3928974201 conn(0x55f6f37cd4e0 msgr2=0x55f6f3b855b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.658+0000 7fc1a5a48740 1 -- 172.21.15.27:0/3928974201 shutdown_connections 2024-04-25T13:42:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.658+0000 7fc1a5a48740 1 -- 172.21.15.27:0/3928974201 wait complete. 2024-04-25T13:42:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.658+0000 7fc1a5a48740 1 -- start start 2024-04-25T13:42:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.658+0000 7fc1a5a48740 1 Processor -- start 2024-04-25T13:42:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.658+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b88fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b894f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc1a5a48740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3bf96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b88fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b88fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46590/0 (socket says 172.21.15.27:46590) 2024-04-25T13:42:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981ac1c0 con 0x55f6f37b04d0 2024-04-25T13:42:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981ad630 con 0x55f6f37b86b0 2024-04-25T13:42:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 -- 172.21.15.27:0/1617787459 learned_addr learned my addr 172.21.15.27:0/1617787459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981ac890 con 0x55f6f37d9390 2024-04-25T13:42:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc1a5a48740 10 client.0 _ll_register_callbacks cb 0x7fc190009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19d133640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3bf96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc18ffff640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 -- 172.21.15.27:0/1617787459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 msgr2=0x55f6f3bf96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3bf96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 -- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 msgr2=0x55f6f3b894f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b894f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 -- 172.21.15.27:0/1617787459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6f3b89b30 con 0x55f6f37b04d0 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19d133640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3bf96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc18ffff640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b894f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc19c932640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b88fb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc19400cfd0 tx=0x7fc1940c1d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06ea2a49d38f9cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.659+0000 7fc177fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.660+0000 7fc177fff640 1 -- 172.21.15.27:0/1617787459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1940a30e0 con 0x55f6f37b04d0 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.660+0000 7fc177fff640 1 -- 172.21.15.27:0/1617787459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc1940a3810 con 0x55f6f37b04d0 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.660+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1940a2f00 con 0x55f6f37b04d0 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.660+0000 7fc177fff640 1 -- 172.21.15.27:0/1617787459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc194010640 con 0x55f6f37b04d0 2024-04-25T13:42:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.660+0000 7fc1a5a48740 10 client.15642 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.660+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f6f3b89b30 con 0x55f6f37b04d0 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.660+0000 7fc177fff640 1 -- 172.21.15.27:0/1617787459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1940a2f00 con 0x55f6f37b04d0 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.661+0000 7fc177fff640 1 -- 172.21.15.27:0/1617787459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc1940a5ef0 con 0x55f6f37b04d0 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.661+0000 7fc1a5a48740 10 client.15642 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.661+0000 7fc1a5a48740 10 client.15642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.661+0000 7fc1a5a48740 20 client.15642 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.661+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f6f376d3a0 0x55f6f3bfd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.661+0000 7fc1a5a48740 4 client.15642 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.661+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6f3ae7800 con 0x55f6f376d3a0 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.661+0000 7fc18ffff640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f6f376d3a0 0x55f6f3bfd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc177fff640 10 client.15642 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc18ffff640 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f6f376d3a0 0x55f6f3bfd1f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=870a7935650fc007 server_cookie=3436901d838579e2 in_seq=0 out_seq=0 2024-04-25T13:42:31.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc177fff640 1 -- 172.21.15.27:0/1617787459 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6f3ae7800 con 0x55f6f376d3a0 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc177fff640 10 client.15642 handle_command_reply: tid=0 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 client.15642 shutdown 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 msgr2=0x55f6f3b88fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b88fb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc19400cfd0 tx=0x7fc1940c1d40 comp rx=0 tx=0).stop 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 shutdown_connections 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f6f376d3a0 0x55f6f3bfd1f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f6f37d9390 0x55f6f3bf96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f6f37b86b0 0x55f6f3b894f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 --2- 172.21.15.27:0/1617787459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f6f37b04d0 0x55f6f3b88fb0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.662+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 >> 172.21.15.27:0/1617787459 conn(0x55f6f37cd4e0 msgr2=0x55f6f3b855b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.663+0000 7fc177fff640 0 client.15642 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.663+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 shutdown_connections 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.664+0000 7fc1a5a48740 1 -- 172.21.15.27:0/1617787459 wait complete. 2024-04-25T13:42:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.664+0000 7fc1a5a48740 20 client.15642 trim_cache size 0 max 16384 2024-04-25T13:42:31.672 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1800afb70 msgr2=0x7fc1800b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1800afb70 0x7fc1800b2080 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc188006010 tx=0x7fc188031400 comp rx=0 tx=0).stop 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1981029f0 msgr2=0x7fc19806bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1981029f0 0x7fc19806bd20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc19400e980 tx=0x7fc19400ed80 comp rx=0 tx=0).stop 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 shutdown_connections 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1800afb70 0x7fc1800b2080 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc198108930 0x7fc1981a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc198103c30 0x7fc1981a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 --2- 172.21.15.27:0/941493848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1981029f0 0x7fc19806bd20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 >> 172.21.15.27:0/941493848 conn(0x7fc1980fdbe0 msgr2=0x7fc198104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 shutdown_connections 2024-04-25T13:42:31.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:31.665+0000 7fc19eb96640 1 -- 172.21.15.27:0/941493848 wait complete. 2024-04-25T13:42:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:32 smithi027 ceph-mon[29140]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9.4 MiB/s wr, 781 op/s 2024-04-25T13:42:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:32 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3182847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:42:32.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:32 smithi122 ceph-mon[32843]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9.4 MiB/s wr, 781 op/s 2024-04-25T13:42:32.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:32 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3182847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:42:32.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:32 smithi045 ceph-mon[32731]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9.4 MiB/s wr, 781 op/s 2024-04-25T13:42:32.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:32 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3182847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:42:32.679 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:42:32.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:42:32.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:42:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.851+0000 7f25f82e2640 1 -- 172.21.15.27:0/271186680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0104d20 msgr2=0x7f25f0105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.851+0000 7f25f82e2640 1 --2- 172.21.15.27:0/271186680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0104d20 0x7f25f0105130 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009900 tx=0x7f25e002a410 comp rx=0 tx=0).stop 2024-04-25T13:42:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.852+0000 7f25f82e2640 1 -- 172.21.15.27:0/271186680 shutdown_connections 2024-04-25T13:42:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.852+0000 7f25f82e2640 1 --2- 172.21.15.27:0/271186680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25f0109a20 0x7f25f010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.852+0000 7f25f82e2640 1 --2- 172.21.15.27:0/271186680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0104d20 0x7f25f0105130 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.852+0000 7f25f82e2640 1 --2- 172.21.15.27:0/271186680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f0103ae0 0x7f25f01047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.852+0000 7f25f82e2640 1 -- 172.21.15.27:0/271186680 >> 172.21.15.27:0/271186680 conn(0x7f25f00fecd0 msgr2=0x7f25f0101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.852+0000 7f25f82e2640 1 -- 172.21.15.27:0/271186680 shutdown_connections 2024-04-25T13:42:32.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.852+0000 7f25f82e2640 1 -- 172.21.15.27:0/271186680 wait complete. 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f82e2640 1 Processor -- start 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f82e2640 1 -- start start 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f82e2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f0103ae0 0x7f25f019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f82e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25f0104d20 0x7f25f019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f82e2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0109a20 0x7f25f01a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f82e2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f25f010f650 con 0x7f25f0109a20 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f82e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25f010f4d0 con 0x7f25f0104d20 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f82e2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f25f010f7d0 con 0x7f25f0103ae0 2024-04-25T13:42:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f687f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0109a20 0x7f25f01a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f607e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f0103ae0 0x7f25f019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f687f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0109a20 0x7f25f01a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46596/0 (socket says 172.21.15.27:46596) 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f687f640 1 -- 172.21.15.27:0/1966995112 learned_addr learned my addr 172.21.15.27:0/1966995112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f587d640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25f0104d20 0x7f25f019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f607e640 1 -- 172.21.15.27:0/1966995112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25f0104d20 msgr2=0x7f25f019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f607e640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25f0104d20 0x7f25f019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f607e640 1 -- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0109a20 msgr2=0x7f25f01a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f607e640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0109a20 0x7f25f01a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.853+0000 7f25f607e640 1 -- 172.21.15.27:0/1966995112 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f01a1d60 con 0x7f25f0103ae0 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.854+0000 7f25f607e640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f0103ae0 0x7f25f019a220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f25f01033b0 tx=0x7f25ec00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.854+0000 7f25e77fe640 1 -- 172.21.15.27:0/1966995112 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec00cc40 con 0x7f25f0103ae0 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.854+0000 7f25f82e2640 1 -- 172.21.15.27:0/1966995112 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f01a1ff0 con 0x7f25f0103ae0 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.854+0000 7f25e77fe640 1 -- 172.21.15.27:0/1966995112 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25ec00cde0 con 0x7f25f0103ae0 2024-04-25T13:42:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.854+0000 7f25e77fe640 1 -- 172.21.15.27:0/1966995112 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec010430 con 0x7f25f0103ae0 2024-04-25T13:42:32.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.854+0000 7f25f82e2640 1 -- 172.21.15.27:0/1966995112 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f01a2460 con 0x7f25f0103ae0 2024-04-25T13:42:32.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.858+0000 7f25ff18e740 1 Processor -- start 2024-04-25T13:42:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.858+0000 7f25ff18e740 1 -- start start 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.858+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8684f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb8684fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb86856d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25e77fe640 1 -- 172.21.15.27:0/1966995112 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f25ec0105d0 con 0x7f25f0103ae0 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25ff18e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cb867b0b00 con 0x55cb86489730 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25ff18e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cb8674c920 con 0x55cb864aa410 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25ff18e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55cb867acac0 con 0x55cb86481550 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8684f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8684f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56824/0 (socket says 172.21.15.27:56824) 2024-04-25T13:42:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 -- 172.21.15.27:0/3563355459 learned_addr learned my addr 172.21.15.27:0/3563355459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f587d640 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb86856d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f607e640 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb8684fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 -- 172.21.15.27:0/3563355459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 msgr2=0x55cb8684fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb8684fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 -- 172.21.15.27:0/3563355459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 msgr2=0x55cb86856d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb86856d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 -- 172.21.15.27:0/3563355459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb8658a7e0 con 0x55cb864aa410 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25f687f640 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8684f8b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25e80114c0 tx=0x7f25e8011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1487f8358382b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.859+0000 7f25e77fe640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f25dc0afaf0 0x7f25dc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.860+0000 7f25e4bfe640 1 -- 172.21.15.27:0/3563355459 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8004b20 con 0x55cb864aa410 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.860+0000 7f25e4bfe640 1 -- 172.21.15.27:0/3563355459 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25e8004cc0 con 0x55cb864aa410 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.860+0000 7f25e4bfe640 1 -- 172.21.15.27:0/3563355459 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8005490 con 0x55cb864aa410 2024-04-25T13:42:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.860+0000 7f25e77fe640 1 -- 172.21.15.27:0/1966995112 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f25ec0def30 con 0x7f25f0103ae0 2024-04-25T13:42:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.860+0000 7f25f607e640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f25dc0afaf0 0x7f25dc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 -- 172.21.15.27:0/3563355459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 msgr2=0x55cb8684f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25f607e640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f25dc0afaf0 0x7f25dc0b2000 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55cb86850c90 tx=0x7f25ec0e2a10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8684f8b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25e80114c0 tx=0x7f25e8011980 comp rx=0 tx=0).stop 2024-04-25T13:42:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 -- 172.21.15.27:0/3563355459 shutdown_connections 2024-04-25T13:42:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb86856d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb8684fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3563355459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8684f8b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 -- 172.21.15.27:0/3563355459 >> 172.21.15.27:0/3563355459 conn(0x55cb8649e560 msgr2=0x55cb86846e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 -- 172.21.15.27:0/3563355459 shutdown_connections 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 -- 172.21.15.27:0/3563355459 wait complete. 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 -- start start 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.861+0000 7f25ff18e740 1 Processor -- start 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86481550 0x55cb8684e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb8684ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25f687f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86481550 0x55cb8684e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb864aa410 0x55cb8684b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25ff18e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cb86857780 con 0x55cb86489730 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25ff18e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cb8674a5d0 con 0x55cb86481550 2024-04-25T13:42:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25ff18e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55cb86857900 con 0x55cb864aa410 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25f687f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86481550 0x55cb8684e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56832/0 (socket says 172.21.15.27:56832) 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25f687f640 1 -- 172.21.15.27:0/3014827086 learned_addr learned my addr 172.21.15.27:0/3014827086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25ff18e740 10 client.0 _ll_register_callbacks cb 0x55cb86738080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25f587d640 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb8684ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.862+0000 7f25f687f640 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb864aa410 0x55cb8684b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25f687f640 1 -- 172.21.15.27:0/3014827086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb864aa410 msgr2=0x55cb8684b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25f687f640 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb864aa410 0x55cb8684b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25f687f640 1 -- 172.21.15.27:0/3014827086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 msgr2=0x55cb8684ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25f687f640 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb8684ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25f687f640 1 -- 172.21.15.27:0/3014827086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb865eaf70 con 0x55cb86481550 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25f587d640 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb8684ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25f687f640 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86481550 0x55cb8684e8a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f25e801b040 tx=0x7f25e8004e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8413ecc4dfed5152 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3014827086 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e801d040 con 0x55cb86481550 2024-04-25T13:42:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25ff18e740 1 -- 172.21.15.27:0/3014827086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25e8004cc0 con 0x55cb86481550 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.863+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3014827086 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f25e8004a40 con 0x55cb86481550 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3014827086 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f25e8005190 con 0x55cb86481550 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25ff18e740 10 client.25385 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25ff18e740 1 -- 172.21.15.27:0/3014827086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55cb86482170 con 0x55cb86481550 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3014827086 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25e8004cc0 con 0x55cb86481550 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3014827086 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f25e80276f0 con 0x55cb86481550 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25ff18e740 10 client.25385 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25ff18e740 10 client.25385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25ff18e740 20 client.25385 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cb864691f0 0x55cb8686b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25ff18e740 4 client.25385 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25ff18e740 1 -- 172.21.15.27:0/3014827086 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb867be710 con 0x55cb864691f0 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.864+0000 7f25f587d640 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cb864691f0 0x55cb8686b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.865+0000 7f25f587d640 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cb864691f0 0x55cb8686b630 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46418b1b59225577 server_cookie=c856006eeb184b12 in_seq=0 out_seq=0 2024-04-25T13:42:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.865+0000 7f25d9ffb640 10 client.25385 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.866+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3014827086 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cb867be710 con 0x55cb864691f0 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.866+0000 7f25d9ffb640 10 client.25385 handle_command_reply: tid=0 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.866+0000 7f25ff18e740 1 client.25385 shutdown 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 -- 172.21.15.27:0/3014827086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86481550 msgr2=0x55cb8684e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86481550 0x55cb8684e8a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f25e801b040 tx=0x7f25e8004e70 comp rx=0 tx=0).stop 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 -- 172.21.15.27:0/3014827086 shutdown_connections 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cb864691f0 0x55cb8686b630 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb864aa410 0x55cb8684b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86489730 0x55cb8684ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3014827086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86481550 0x55cb8684e8a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 -- 172.21.15.27:0/3014827086 >> 172.21.15.27:0/3014827086 conn(0x55cb8649e560 msgr2=0x55cb86845ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25d9ffb640 0 client.25385 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 -- 172.21.15.27:0/3014827086 shutdown_connections 2024-04-25T13:42:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 1 -- 172.21.15.27:0/3014827086 wait complete. 2024-04-25T13:42:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.867+0000 7f25ff18e740 20 client.25385 trim_cache size 0 max 16384 2024-04-25T13:42:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25ff18e740 1 Processor -- start 2024-04-25T13:42:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25ff18e740 1 -- start start 2024-04-25T13:42:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb86853aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86489730 0x55cb86853fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb864aa410 0x55cb8686fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25ff18e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f25f01a6f90 con 0x55cb864aa410 2024-04-25T13:42:32.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25ff18e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25f01a9810 con 0x55cb86489730 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25ff18e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f25f01a7600 con 0x55cb86481550 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25f687f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb864aa410 0x55cb8686fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25f587d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86489730 0x55cb86853fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25f687f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb86853aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25f587d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86489730 0x55cb86853fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56836/0 (socket says 172.21.15.27:56836) 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25f587d640 1 -- 172.21.15.27:0/49737239 learned_addr learned my addr 172.21.15.27:0/49737239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.893+0000 7f25f687f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb86853aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44754/0 (socket says 172.21.15.27:44754) 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25f587d640 1 -- 172.21.15.27:0/49737239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 msgr2=0x55cb86853aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25f587d640 1 --2- 172.21.15.27:0/49737239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb86853aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25f587d640 1 -- 172.21.15.27:0/49737239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb864aa410 msgr2=0x55cb8686fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25f587d640 1 --2- 172.21.15.27:0/49737239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb864aa410 0x55cb8686fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25f587d640 1 -- 172.21.15.27:0/49737239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb8684ef10 con 0x55cb86489730 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25f687f640 1 --2- 172.21.15.27:0/49737239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb86853aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25f587d640 1 --2- 172.21.15.27:0/49737239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86489730 0x55cb86853fe0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009230 tx=0x7f25e0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31bb7a3ccfbf42f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25dbfff640 1 -- 172.21.15.27:0/49737239 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e004c070 con 0x55cb86489730 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25dbfff640 1 -- 172.21.15.27:0/49737239 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25e00469f0 con 0x55cb86489730 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.894+0000 7f25dbfff640 1 -- 172.21.15.27:0/49737239 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0035040 con 0x55cb86489730 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 -- 172.21.15.27:0/49737239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86489730 msgr2=0x55cb86853fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 --2- 172.21.15.27:0/49737239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86489730 0x55cb86853fe0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009230 tx=0x7f25e0009260 comp rx=0 tx=0).stop 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 -- 172.21.15.27:0/49737239 shutdown_connections 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 --2- 172.21.15.27:0/49737239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb864aa410 0x55cb8686fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 --2- 172.21.15.27:0/49737239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb86489730 0x55cb86853fe0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 --2- 172.21.15.27:0/49737239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86481550 0x55cb86853aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 -- 172.21.15.27:0/49737239 >> 172.21.15.27:0/49737239 conn(0x55cb8649e560 msgr2=0x55cb86857ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 -- 172.21.15.27:0/49737239 shutdown_connections 2024-04-25T13:42:32.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 -- 172.21.15.27:0/49737239 wait complete. 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.895+0000 7f25ff18e740 1 -- start start 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25ff18e740 1 Processor -- start 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86481550 0x55cb86879fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86489730 0x55cb8687a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25ff18e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8687b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25ff18e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f25f01a7cd0 con 0x55cb86481550 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25ff18e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25f01a9140 con 0x55cb864aa410 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25ff18e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f25f01a83a0 con 0x55cb86489730 2024-04-25T13:42:32.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25ff18e740 10 client.0 _ll_register_callbacks cb 0x7f25ec0e28e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25f587d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86489730 0x55cb8687a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25f587d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86489730 0x55cb8687a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44764/0 (socket says 172.21.15.27:44764) 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.896+0000 7f25f587d640 1 -- 172.21.15.27:0/3600937485 learned_addr learned my addr 172.21.15.27:0/3600937485 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f687f640 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8687b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f687f640 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86481550 0x55cb86879fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f587d640 1 -- 172.21.15.27:0/3600937485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 msgr2=0x55cb8687b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f587d640 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8687b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f587d640 1 -- 172.21.15.27:0/3600937485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86481550 msgr2=0x55cb86879fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f587d640 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86481550 0x55cb86879fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f587d640 1 -- 172.21.15.27:0/3600937485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb8687b9d0 con 0x55cb86489730 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f687f640 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86481550 0x55cb86879fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.897+0000 7f25f587d640 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86489730 0x55cb8687a4e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f25e0031440 tx=0x7f25e0004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e886aece227f98b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:32.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3600937485 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0055060 con 0x55cb86489730 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3600937485 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f25e0003bb0 con 0x55cb86489730 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 1 -- 172.21.15.27:0/3600937485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25e00469f0 con 0x55cb86489730 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3600937485 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f25e004c040 con 0x55cb86489730 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 10 client.25396 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 1 -- 172.21.15.27:0/3600937485 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55cb8687ccc0 con 0x55cb86489730 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3600937485 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25e00469f0 con 0x55cb86489730 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3600937485 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f25e00317f0 con 0x55cb86489730 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 10 client.25396 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 10 client.25396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 20 client.25396 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cb8643e630 0x55cb8685b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 4 client.25396 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:42:32.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.898+0000 7f25ff18e740 1 -- 172.21.15.27:0/3600937485 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cb867b8580 con 0x55cb8643e630 2024-04-25T13:42:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.899+0000 7f25f687f640 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cb8643e630 0x55cb8685b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.900+0000 7f25f687f640 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cb8643e630 0x55cb8685b5c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45400204c58a306e server_cookie=43804c25aa0e7f2d in_seq=0 out_seq=0 2024-04-25T13:42:32.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.900+0000 7f25d9ffb640 10 client.25396 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:32.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.902+0000 7f25d9ffb640 1 -- 172.21.15.27:0/3600937485 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cb867b8580 con 0x55cb8643e630 2024-04-25T13:42:32.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.902+0000 7f25d9ffb640 10 client.25396 handle_command_reply: tid=0 2024-04-25T13:42:32.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.902+0000 7f25ff18e740 1 client.25396 shutdown 2024-04-25T13:42:32.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.903+0000 7f25ff18e740 1 -- 172.21.15.27:0/3600937485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86489730 msgr2=0x55cb8687a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.903+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86489730 0x55cb8687a4e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f25e0031440 tx=0x7f25e0004530 comp rx=0 tx=0).stop 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.903+0000 7f25ff18e740 1 -- 172.21.15.27:0/3600937485 shutdown_connections 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.903+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cb8643e630 0x55cb8685b5c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.903+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cb864aa410 0x55cb8687b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.903+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cb86489730 0x55cb8687a4e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.903+0000 7f25ff18e740 1 --2- 172.21.15.27:0/3600937485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cb86481550 0x55cb86879fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.903+0000 7f25ff18e740 1 -- 172.21.15.27:0/3600937485 >> 172.21.15.27:0/3600937485 conn(0x55cb8649e560 msgr2=0x55cb86853aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.904+0000 7f25d9ffb640 0 client.25396 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.904+0000 7f25ff18e740 1 -- 172.21.15.27:0/3600937485 shutdown_connections 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.904+0000 7f25ff18e740 1 -- 172.21.15.27:0/3600937485 wait complete. 2024-04-25T13:42:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.904+0000 7f25ff18e740 20 client.25396 trim_cache size 0 max 16384 2024-04-25T13:42:32.906 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:42:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:42:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "239d93fb-c355-4026-8529-a19d91846125", 2024-04-25T13:42:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:42:32.906 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:42:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.904+0000 7f25f82e2640 1 -- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f25dc0afaf0 msgr2=0x7f25dc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.904+0000 7f25f82e2640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f25dc0afaf0 0x7f25dc0b2000 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55cb86850c90 tx=0x7f25ec0e2a10 comp rx=0 tx=0).stop 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.904+0000 7f25f82e2640 1 -- 172.21.15.27:0/1966995112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f0103ae0 msgr2=0x7f25f019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.904+0000 7f25f82e2640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f0103ae0 0x7f25f019a220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f25f01033b0 tx=0x7f25ec00ee50 comp rx=0 tx=0).stop 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.905+0000 7f25f82e2640 1 -- 172.21.15.27:0/1966995112 shutdown_connections 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.905+0000 7f25f82e2640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f25dc0afaf0 0x7f25dc0b2000 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.905+0000 7f25f82e2640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25f0109a20 0x7f25f01a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.905+0000 7f25f82e2640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25f0104d20 0x7f25f019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.905+0000 7f25f82e2640 1 --2- 172.21.15.27:0/1966995112 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25f0103ae0 0x7f25f019a220 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.905+0000 7f25f82e2640 1 -- 172.21.15.27:0/1966995112 >> 172.21.15.27:0/1966995112 conn(0x7f25f00fecd0 msgr2=0x7f25f0105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.905+0000 7f25f82e2640 1 -- 172.21.15.27:0/1966995112 shutdown_connections 2024-04-25T13:42:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:32.905+0000 7f25f82e2640 1 -- 172.21.15.27:0/1966995112 wait complete. 2024-04-25T13:42:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 85.0 s, 3388.6 tps, lat 29.476 ms stddev 30.761 2024-04-25T13:42:32.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:42:32.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:42:33.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.093+0000 7f1388851640 1 -- 172.21.15.27:0/3534506596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384109a20 msgr2=0x7f138410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.093+0000 7f1388851640 1 --2- 172.21.15.27:0/3534506596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384109a20 0x7f138410be40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f137c009900 tx=0x7f137c02a410 comp rx=0 tx=0).stop 2024-04-25T13:42:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.094+0000 7f1388851640 1 -- 172.21.15.27:0/3534506596 shutdown_connections 2024-04-25T13:42:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.094+0000 7f1388851640 1 --2- 172.21.15.27:0/3534506596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384109a20 0x7f138410be40 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.094+0000 7f1388851640 1 --2- 172.21.15.27:0/3534506596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1384104d20 0x7f1384105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.094+0000 7f1388851640 1 --2- 172.21.15.27:0/3534506596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1384103ae0 0x7f13841047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.094+0000 7f1388851640 1 -- 172.21.15.27:0/3534506596 >> 172.21.15.27:0/3534506596 conn(0x7f13840fed10 msgr2=0x7f1384101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.094+0000 7f1388851640 1 -- 172.21.15.27:0/3534506596 shutdown_connections 2024-04-25T13:42:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.094+0000 7f1388851640 1 -- 172.21.15.27:0/3534506596 wait complete. 2024-04-25T13:42:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.095+0000 7f1388851640 1 Processor -- start 2024-04-25T13:42:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.095+0000 7f1388851640 1 -- start start 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.095+0000 7f1388851640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384103ae0 0x7f138419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.095+0000 7f1388851640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1384104d20 0x7f138419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.095+0000 7f1388851640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1384109a20 0x7f13841a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.095+0000 7f1388851640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f138410f5f0 con 0x7f1384103ae0 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.095+0000 7f1388851640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f138410f470 con 0x7f1384104d20 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.095+0000 7f1388851640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f138410f770 con 0x7f1384109a20 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384103ae0 0x7f138419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f1382d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1384109a20 0x7f13841a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384103ae0 0x7f138419a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46656/0 (socket says 172.21.15.27:46656) 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 -- 172.21.15.27:0/389478345 learned_addr learned my addr 172.21.15.27:0/389478345 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f1381d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1384104d20 0x7f138419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 -- 172.21.15.27:0/389478345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1384109a20 msgr2=0x7f13841a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1384109a20 0x7f13841a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 -- 172.21.15.27:0/389478345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1384104d20 msgr2=0x7f138419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1384104d20 0x7f138419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 -- 172.21.15.27:0/389478345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13841a1cd0 con 0x7f1384103ae0 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f1381d9b640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1384104d20 0x7f138419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f1382d9d640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1384109a20 0x7f13841a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f138259c640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384103ae0 0x7f138419a190 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f137400eb90 tx=0x7f137400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f13737fe640 1 -- 172.21.15.27:0/389478345 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374003b70 con 0x7f1384103ae0 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f13737fe640 1 -- 172.21.15.27:0/389478345 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f137400ee90 con 0x7f1384103ae0 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f1388851640 1 -- 172.21.15.27:0/389478345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13841a1fc0 con 0x7f1384103ae0 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f13737fe640 1 -- 172.21.15.27:0/389478345 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13740197b0 con 0x7f1384103ae0 2024-04-25T13:42:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.096+0000 7f1388851640 1 -- 172.21.15.27:0/389478345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13841a2420 con 0x7f1384103ae0 2024-04-25T13:42:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.100+0000 7f138f7e1740 1 Processor -- start 2024-04-25T13:42:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.100+0000 7f138f7e1740 1 -- start start 2024-04-25T13:42:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.101+0000 7f13737fe640 1 -- 172.21.15.27:0/389478345 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1374004030 con 0x7f1384103ae0 2024-04-25T13:42:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.101+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5df390 0x56247d980a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.101+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d980fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.101+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d981c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.101+0000 7f138f7e1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56247d8e5d80 con 0x56247d5df390 2024-04-25T13:42:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.101+0000 7f138f7e1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56247d89f8b0 con 0x56247d5be6b0 2024-04-25T13:42:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.101+0000 7f138f7e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56247d8e1d40 con 0x56247d5b64d0 2024-04-25T13:42:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1382d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5df390 0x56247d980a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1382d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5df390 0x56247d980a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46666/0 (socket says 172.21.15.27:46666) 2024-04-25T13:42:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1382d9d640 1 -- 172.21.15.27:0/4250109790 learned_addr learned my addr 172.21.15.27:0/4250109790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1381d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d981c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f138259c640 1 --2- 172.21.15.27:0/4250109790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d980fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f13737fe640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f136c0afaf0 0x7f136c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1381d9b640 1 -- 172.21.15.27:0/4250109790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 msgr2=0x56247d980fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1381d9b640 1 --2- 172.21.15.27:0/4250109790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d980fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1381d9b640 1 -- 172.21.15.27:0/4250109790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5df390 msgr2=0x56247d980a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1381d9b640 1 --2- 172.21.15.27:0/4250109790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5df390 0x56247d980a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.102+0000 7f1381d9b640 1 -- 172.21.15.27:0/4250109790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56247d870a40 con 0x56247d5be6b0 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.103+0000 7f13737fe640 1 -- 172.21.15.27:0/389478345 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f13740df470 con 0x7f1384103ae0 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.103+0000 7f1382d9d640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f136c0afaf0 0x7f136c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.103+0000 7f1381d9b640 1 --2- 172.21.15.27:0/4250109790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d981c70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1378009f20 tx=0x7f1378013c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f43f9ba3271b5ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.103+0000 7f1382d9d640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f136c0afaf0 0x7f136c0b2000 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f137c032fd0 tx=0x7f137c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.103+0000 7f1370ff9640 1 -- 172.21.15.27:0/4250109790 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137800ea50 con 0x56247d5be6b0 2024-04-25T13:42:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f1370ff9640 1 -- 172.21.15.27:0/4250109790 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1378013e50 con 0x56247d5be6b0 2024-04-25T13:42:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f1370ff9640 1 -- 172.21.15.27:0/4250109790 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378003b60 con 0x56247d5be6b0 2024-04-25T13:42:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f138f7e1740 1 -- 172.21.15.27:0/4250109790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 msgr2=0x56247d981c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f138f7e1740 1 --2- 172.21.15.27:0/4250109790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d981c70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1378009f20 tx=0x7f1378013c60 comp rx=0 tx=0).stop 2024-04-25T13:42:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f138f7e1740 1 -- 172.21.15.27:0/4250109790 shutdown_connections 2024-04-25T13:42:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f138f7e1740 1 --2- 172.21.15.27:0/4250109790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d981c70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f138f7e1740 1 --2- 172.21.15.27:0/4250109790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d980fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f138f7e1740 1 --2- 172.21.15.27:0/4250109790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5df390 0x56247d980a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.104+0000 7f138f7e1740 1 -- 172.21.15.27:0/4250109790 >> 172.21.15.27:0/4250109790 conn(0x56247d5d34e0 msgr2=0x56247d97c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.105+0000 7f138f7e1740 1 -- 172.21.15.27:0/4250109790 shutdown_connections 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.105+0000 7f138f7e1740 1 -- 172.21.15.27:0/4250109790 wait complete. 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.105+0000 7f138f7e1740 1 -- start start 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.105+0000 7f138f7e1740 1 Processor -- start 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.105+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d98a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.105+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 0x56247d98ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.105+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5df390 0x56247d991d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f138f7e1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56247d982780 con 0x56247d5be6b0 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f138f7e1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56247d86cd10 con 0x56247d5df390 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f138f7e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56247d982900 con 0x56247d5b64d0 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f138f7e1740 10 client.0 _ll_register_callbacks cb 0x56247d899210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1382d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5df390 0x56247d991d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1382d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5df390 0x56247d991d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56882/0 (socket says 172.21.15.27:56882) 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1382d9d640 1 -- 172.21.15.27:0/1715509541 learned_addr learned my addr 172.21.15.27:0/1715509541 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1381d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 0x56247d98ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f138259c640 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d98a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1382d9d640 1 -- 172.21.15.27:0/1715509541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 msgr2=0x56247d98a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1382d9d640 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d98a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1382d9d640 1 -- 172.21.15.27:0/1715509541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 msgr2=0x56247d98ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1382d9d640 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 0x56247d98ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.106+0000 7f1382d9d640 1 -- 172.21.15.27:0/1715509541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56247d6c17d0 con 0x56247d5df390 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.107+0000 7f1382d9d640 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5df390 0x56247d991d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f137c009230 tx=0x7f137c02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b980d6ee8d45f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.107+0000 7f1369ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.107+0000 7f1369ffb640 1 -- 172.21.15.27:0/1715509541 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c035040 con 0x56247d5df390 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.107+0000 7f1369ffb640 1 -- 172.21.15.27:0/1715509541 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f137c0048c0 con 0x56247d5df390 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.107+0000 7f138f7e1740 1 -- 172.21.15.27:0/1715509541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1378013e50 con 0x56247d5df390 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.107+0000 7f1369ffb640 1 -- 172.21.15.27:0/1715509541 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f137c058070 con 0x56247d5df390 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.107+0000 7f138f7e1740 10 client.25400 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:33.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.107+0000 7f138f7e1740 1 -- 172.21.15.27:0/1715509541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56247d5f73f0 con 0x56247d5df390 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f1369ffb640 1 -- 172.21.15.27:0/1715509541 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1378013e50 con 0x56247d5df390 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f1369ffb640 1 -- 172.21.15.27:0/1715509541 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f137c051c30 con 0x56247d5df390 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f138f7e1740 10 client.25400 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f138f7e1740 10 client.25400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f138f7e1740 20 client.25400 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f138f7e1740 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56247d59df60 0x56247d9982c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f138f7e1740 4 client.25400 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f138f7e1740 1 -- 172.21.15.27:0/1715509541 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56247d8f3990 con 0x56247d59df60 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.108+0000 7f138259c640 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56247d59df60 0x56247d9982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.109+0000 7f1369ffb640 10 client.25400 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:33.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.109+0000 7f138259c640 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56247d59df60 0x56247d9982c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82f4c56c9bc9e8bc server_cookie=1e28ad2c086034f4 in_seq=0 out_seq=0 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f1369ffb640 1 -- 172.21.15.27:0/1715509541 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56247d8f3990 con 0x56247d59df60 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f1369ffb640 10 client.25400 handle_command_reply: tid=0 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 client.25400 shutdown 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 -- 172.21.15.27:0/1715509541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5df390 msgr2=0x56247d991d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5df390 0x56247d991d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f137c009230 tx=0x7f137c02a410 comp rx=0 tx=0).stop 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 -- 172.21.15.27:0/1715509541 shutdown_connections 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56247d59df60 0x56247d9982c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5df390 0x56247d991d40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f1369ffb640 0 client.25400 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 0x56247d98ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 --2- 172.21.15.27:0/1715509541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d98a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 -- 172.21.15.27:0/1715509541 >> 172.21.15.27:0/1715509541 conn(0x56247d5d34e0 msgr2=0x56247d97b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 -- 172.21.15.27:0/1715509541 shutdown_connections 2024-04-25T13:42:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 1 -- 172.21.15.27:0/1715509541 wait complete. 2024-04-25T13:42:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.110+0000 7f138f7e1740 20 client.25400 trim_cache size 0 max 16384 2024-04-25T13:42:33.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.138+0000 7f138f7e1740 1 Processor -- start 2024-04-25T13:42:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.138+0000 7f138f7e1740 1 -- start start 2024-04-25T13:42:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.138+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d59df60 0x56247d988f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d9894b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d59df60 0x56247d988f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d59df60 0x56247d988f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46680/0 (socket says 172.21.15.27:46680) 2024-04-25T13:42:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138259c640 1 -- 172.21.15.27:0/975058787 learned_addr learned my addr 172.21.15.27:0/975058787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138f7e1740 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d99ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138f7e1740 1 -- 172.21.15.27:0/975058787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f13841a6f30 con 0x56247d59df60 2024-04-25T13:42:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138f7e1740 1 -- 172.21.15.27:0/975058787 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13841a97b0 con 0x56247d5be6b0 2024-04-25T13:42:33.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138f7e1740 1 -- 172.21.15.27:0/975058787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f13841a75a0 con 0x56247d5b64d0 2024-04-25T13:42:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f1382d9d640 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d99ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138259c640 1 -- 172.21.15.27:0/975058787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 msgr2=0x56247d9894b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:42:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138259c640 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d9894b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138259c640 1 -- 172.21.15.27:0/975058787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 msgr2=0x56247d99ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.139+0000 7f138259c640 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d99ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.140+0000 7f138259c640 1 -- 172.21.15.27:0/975058787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56247d5b70f0 con 0x56247d59df60 2024-04-25T13:42:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.140+0000 7f1382d9d640 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d99ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:42:33.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.140+0000 7f138259c640 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d59df60 0x56247d988f70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f13740049d0 tx=0x7f137400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca82dfd982706f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.140+0000 7f136bfff640 1 -- 172.21.15.27:0/975058787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13740a7090 con 0x56247d59df60 2024-04-25T13:42:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.140+0000 7f136bfff640 1 -- 172.21.15.27:0/975058787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13740b00c0 con 0x56247d59df60 2024-04-25T13:42:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.140+0000 7f136bfff640 1 -- 172.21.15.27:0/975058787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13740b0780 con 0x56247d59df60 2024-04-25T13:42:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 -- 172.21.15.27:0/975058787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d59df60 msgr2=0x56247d988f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d59df60 0x56247d988f70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f13740049d0 tx=0x7f137400cf90 comp rx=0 tx=0).stop 2024-04-25T13:42:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 -- 172.21.15.27:0/975058787 shutdown_connections 2024-04-25T13:42:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5be6b0 0x56247d99ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d5b64d0 0x56247d9894b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 --2- 172.21.15.27:0/975058787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d59df60 0x56247d988f70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 -- 172.21.15.27:0/975058787 >> 172.21.15.27:0/975058787 conn(0x56247d5d34e0 msgr2=0x56247d9855c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 -- 172.21.15.27:0/975058787 shutdown_connections 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 -- 172.21.15.27:0/975058787 wait complete. 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 -- start start 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.141+0000 7f138f7e1740 1 Processor -- start 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d59df60 0x56247d98e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5b64d0 0x56247d98e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138f7e1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 0x56247d9fe620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138f7e1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f13841a7c70 con 0x56247d5be6b0 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138f7e1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13841a90e0 con 0x56247d5b64d0 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138f7e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f13841a8340 con 0x56247d59df60 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138f7e1740 10 client.0 _ll_register_callbacks cb 0x7f137c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d59df60 0x56247d98e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d59df60 0x56247d98e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44808/0 (socket says 172.21.15.27:44808) 2024-04-25T13:42:33.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f138259c640 1 -- 172.21.15.27:0/712446533 learned_addr learned my addr 172.21.15.27:0/712446533 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f1382d9d640 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 0x56247d9fe620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.142+0000 7f1381d9b640 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5b64d0 0x56247d98e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f138259c640 1 -- 172.21.15.27:0/712446533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5b64d0 msgr2=0x56247d98e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f138259c640 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5b64d0 0x56247d98e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f138259c640 1 -- 172.21.15.27:0/712446533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 msgr2=0x56247d9fe620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f138259c640 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 0x56247d9fe620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f138259c640 1 -- 172.21.15.27:0/712446533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56247d8a64b0 con 0x56247d59df60 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f138259c640 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d59df60 0x56247d98e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f137400cba0 tx=0x7f1374004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfd11ed0353b0fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f1369ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f1369ffb640 1 -- 172.21.15.27:0/712446533 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13740a8020 con 0x56247d59df60 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f1369ffb640 1 -- 172.21.15.27:0/712446533 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f13740df0f0 con 0x56247d59df60 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.143+0000 7f138f7e1740 1 -- 172.21.15.27:0/712446533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56247d8ed800 con 0x56247d59df60 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.144+0000 7f1369ffb640 1 -- 172.21.15.27:0/712446533 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f13740b0780 con 0x56247d59df60 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.144+0000 7f138f7e1740 10 client.25411 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.144+0000 7f138f7e1740 1 -- 172.21.15.27:0/712446533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56247d9ff7f0 con 0x56247d59df60 2024-04-25T13:42:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.146+0000 7f1369ffb640 1 -- 172.21.15.27:0/712446533 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56247d8ed800 con 0x56247d59df60 2024-04-25T13:42:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.146+0000 7f1369ffb640 1 -- 172.21.15.27:0/712446533 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f13740c8bd0 con 0x56247d59df60 2024-04-25T13:42:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.147+0000 7f138f7e1740 10 client.25411 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.147+0000 7f138f7e1740 10 client.25411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.147+0000 7f138f7e1740 20 client.25411 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.147+0000 7f138f7e1740 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56247d5df390 0x56247da0a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.147+0000 7f138f7e1740 4 client.25411 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:42:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.147+0000 7f138f7e1740 1 -- 172.21.15.27:0/712446533 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56247d8684a0 con 0x56247d5df390 2024-04-25T13:42:33.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.147+0000 7f1381d9b640 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56247d5df390 0x56247da0a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.148+0000 7f1381d9b640 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56247d5df390 0x56247da0a7d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d31eb643be44137 server_cookie=e6c9cfe0fbb9f8e8 in_seq=0 out_seq=0 2024-04-25T13:42:33.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.149+0000 7f1369ffb640 10 client.25411 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.150+0000 7f1369ffb640 1 -- 172.21.15.27:0/712446533 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56247d8684a0 con 0x56247d5df390 2024-04-25T13:42:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.150+0000 7f1369ffb640 10 client.25411 handle_command_reply: tid=0 2024-04-25T13:42:33.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.151+0000 7f138f7e1740 1 client.25411 shutdown 2024-04-25T13:42:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.151+0000 7f138f7e1740 1 -- 172.21.15.27:0/712446533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d59df60 msgr2=0x56247d98e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.151+0000 7f138f7e1740 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d59df60 0x56247d98e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f137400cba0 tx=0x7f1374004760 comp rx=0 tx=0).stop 2024-04-25T13:42:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.151+0000 7f138f7e1740 1 -- 172.21.15.27:0/712446533 shutdown_connections 2024-04-25T13:42:33.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.152+0000 7f138f7e1740 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56247d5df390 0x56247da0a7d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.152+0000 7f138f7e1740 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56247d5be6b0 0x56247d9fe620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.152+0000 7f138f7e1740 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56247d5b64d0 0x56247d98e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.152+0000 7f138f7e1740 1 --2- 172.21.15.27:0/712446533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56247d59df60 0x56247d98e090 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.152+0000 7f138f7e1740 1 -- 172.21.15.27:0/712446533 >> 172.21.15.27:0/712446533 conn(0x56247d5d34e0 msgr2=0x56247d988f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.152+0000 7f1369ffb640 0 client.25411 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:33.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.152+0000 7f138f7e1740 1 -- 172.21.15.27:0/712446533 shutdown_connections 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.153+0000 7f138f7e1740 1 -- 172.21.15.27:0/712446533 wait complete. 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.153+0000 7f138f7e1740 20 client.25411 trim_cache size 0 max 16384 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "239d93fb-c355-4026-8529-a19d91846125": { 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "239d93fb-c355-4026-8529-a19d91846125", 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:42:33.153 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:42:33.154 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:42:33.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 -- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f136c0afaf0 msgr2=0x7f136c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f136c0afaf0 0x7f136c0b2000 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f137c032fd0 tx=0x7f137c009260 comp rx=0 tx=0).stop 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 -- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384103ae0 msgr2=0x7f138419a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384103ae0 0x7f138419a190 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f137400eb90 tx=0x7f137400c6a0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 -- 172.21.15.27:0/389478345 shutdown_connections 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f136c0afaf0 0x7f136c0b2000 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1384109a20 0x7f13841a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1384104d20 0x7f138419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 --2- 172.21.15.27:0/389478345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384103ae0 0x7f138419a190 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 -- 172.21.15.27:0/389478345 >> 172.21.15.27:0/389478345 conn(0x7f13840fed10 msgr2=0x7f1384105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 -- 172.21.15.27:0/389478345 shutdown_connections 2024-04-25T13:42:33.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:42:33.154+0000 7f1388851640 1 -- 172.21.15.27:0/389478345 wait complete. 2024-04-25T13:42:33.170 INFO:tasks.cephfs.filesystem:scrub status for tag:239d93fb-c355-4026-8529-a19d91846125 - {'path': '/', 'tag': '239d93fb-c355-4026-8529-a19d91846125', 'options': 'recursive,force'} 2024-04-25T13:42:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:34 smithi027 ceph-mon[29140]: pgmap v399: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 MiB/s wr, 812 op/s 2024-04-25T13:42:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:34 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:42:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:34 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:34 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:42:34.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:34 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:34.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:34 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:42:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:34 smithi122 ceph-mon[32843]: pgmap v399: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 MiB/s wr, 812 op/s 2024-04-25T13:42:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:34 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:42:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:34 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:34 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:42:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:34 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:34 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:42:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:34 smithi045 ceph-mon[32731]: pgmap v399: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 MiB/s wr, 812 op/s 2024-04-25T13:42:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:34 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:42:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:34 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:34 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:42:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:34 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:42:34.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:34 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:42:36.403 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:36 smithi027 ceph-mon[29140]: pgmap v400: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.4 MiB/s wr, 693 op/s 2024-04-25T13:42:36.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:36 smithi122 ceph-mon[32843]: pgmap v400: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.4 MiB/s wr, 693 op/s 2024-04-25T13:42:36.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:36 smithi045 ceph-mon[32731]: pgmap v400: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.4 MiB/s wr, 693 op/s 2024-04-25T13:42:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 90.0 s, 3660.8 tps, lat 27.253 ms stddev 28.699 2024-04-25T13:42:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:38 smithi027 ceph-mon[29140]: pgmap v401: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 10 MiB/s wr, 985 op/s 2024-04-25T13:42:38.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:38 smithi122 ceph-mon[32843]: pgmap v401: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 10 MiB/s wr, 985 op/s 2024-04-25T13:42:38.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:38 smithi045 ceph-mon[32731]: pgmap v401: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 10 MiB/s wr, 985 op/s 2024-04-25T13:42:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:40 smithi027 ceph-mon[29140]: pgmap v402: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.7 MiB/s wr, 735 op/s 2024-04-25T13:42:40.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:40 smithi122 ceph-mon[32843]: pgmap v402: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.7 MiB/s wr, 735 op/s 2024-04-25T13:42:40.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:40 smithi045 ceph-mon[32731]: pgmap v402: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.7 MiB/s wr, 735 op/s 2024-04-25T13:42:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.044+0000 7fc921155640 1 -- 172.21.15.27:0/2631683931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c110f70 msgr2=0x7fc91c113480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.044+0000 7fc921155640 1 --2- 172.21.15.27:0/2631683931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c110f70 0x7fc91c113480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc914009ca0 tx=0x7fc91402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.044+0000 7fc921155640 1 -- 172.21.15.27:0/2631683931 shutdown_connections 2024-04-25T13:42:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.044+0000 7fc921155640 1 --2- 172.21.15.27:0/2631683931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c110f70 0x7fc91c113480 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.044+0000 7fc921155640 1 --2- 172.21.15.27:0/2631683931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c10e400 0x7fc91c1108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.044+0000 7fc921155640 1 --2- 172.21.15.27:0/2631683931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c10b9f0 0x7fc91c10de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.044+0000 7fc921155640 1 -- 172.21.15.27:0/2631683931 >> 172.21.15.27:0/2631683931 conn(0x7fc91c070540 msgr2=0x7fc91c0729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.044+0000 7fc921155640 1 -- 172.21.15.27:0/2631683931 shutdown_connections 2024-04-25T13:42:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.045+0000 7fc921155640 1 -- 172.21.15.27:0/2631683931 wait complete. 2024-04-25T13:42:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.045+0000 7fc921155640 1 Processor -- start 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc921155640 1 -- start start 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc921155640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c10b9f0 0x7fc91c1a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc921155640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c10e400 0x7fc91c1a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc921155640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c110f70 0x7fc91c1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc921155640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc91c115d10 con 0x7fc91c10b9f0 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc921155640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc91c115b90 con 0x7fc91c110f70 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc921155640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc91c115e90 con 0x7fc91c10e400 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc91a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c10e400 0x7fc91c1a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc91b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c110f70 0x7fc91c1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc91b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c110f70 0x7fc91c1a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45024/0 (socket says 172.21.15.27:45024) 2024-04-25T13:42:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc91a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c10e400 0x7fc91c1a42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47682/0 (socket says 172.21.15.27:47682) 2024-04-25T13:42:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.046+0000 7fc91b59e640 1 -- 172.21.15.27:0/664666248 learned_addr learned my addr 172.21.15.27:0/664666248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.047+0000 7fc91b59e640 1 -- 172.21.15.27:0/664666248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c10e400 msgr2=0x7fc91c1a42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.047+0000 7fc91b59e640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c10e400 0x7fc91c1a42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.047+0000 7fc91b59e640 1 -- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c10b9f0 msgr2=0x7fc91c1a3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.047+0000 7fc91b59e640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c10b9f0 0x7fc91c1a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.047+0000 7fc91b59e640 1 -- 172.21.15.27:0/664666248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91c1a10e0 con 0x7fc91c110f70 2024-04-25T13:42:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.047+0000 7fc91b59e640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c110f70 0x7fc91c1a08c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc914004af0 tx=0x7fc914031700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.047+0000 7fc903fff640 1 -- 172.21.15.27:0/664666248 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914003b70 con 0x7fc91c110f70 2024-04-25T13:42:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.048+0000 7fc903fff640 1 -- 172.21.15.27:0/664666248 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc914003d10 con 0x7fc91c110f70 2024-04-25T13:42:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.048+0000 7fc903fff640 1 -- 172.21.15.27:0/664666248 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91402fc00 con 0x7fc91c110f70 2024-04-25T13:42:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.048+0000 7fc921155640 1 -- 172.21.15.27:0/664666248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc91c1a1370 con 0x7fc91c110f70 2024-04-25T13:42:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.048+0000 7fc921155640 1 -- 172.21.15.27:0/664666248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc91c1b1420 con 0x7fc91c110f70 2024-04-25T13:42:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.050+0000 7fc903fff640 1 -- 172.21.15.27:0/664666248 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc9140305c0 con 0x7fc91c110f70 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc928005740 1 Processor -- start 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc928005740 1 -- start start 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc903fff640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc9040afb60 0x7fc9040b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc903fff640 1 -- 172.21.15.27:0/664666248 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc914100220 con 0x7fc91c110f70 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc91a59c640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc9040afb60 0x7fc9040b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc928005740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d864970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc928005740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d864ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc928005740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc928005740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56279d7ca460 con 0x56279d4c41b0 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc928005740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56279d784590 con 0x56279d49b2f0 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc928005740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56279d7c6420 con 0x56279d4a34d0 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc91b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc91b59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47690/0 (socket says 172.21.15.27:47690) 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc91b59e640 1 -- 172.21.15.27:0/2680636316 learned_addr learned my addr 172.21.15.27:0/2680636316 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.051+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d864970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.052+0000 7fc91a59c640 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d864ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.052+0000 7fc91b59e640 1 -- 172.21.15.27:0/2680636316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 msgr2=0x56279d864ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.052+0000 7fc91b59e640 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d864ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.052+0000 7fc91b59e640 1 -- 172.21.15.27:0/2680636316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 msgr2=0x56279d864970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.052+0000 7fc91b59e640 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d864970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.052+0000 7fc91b59e640 1 -- 172.21.15.27:0/2680636316 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56279d7910d0 con 0x56279d4a34d0 2024-04-25T13:42:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.052+0000 7fc91b59e640 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86bdf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc9140c67d0 tx=0x7fc9140c8d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc9cf39d960bb875 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.053+0000 7fc9017fa640 1 -- 172.21.15.27:0/2680636316 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9140cfe60 con 0x56279d4a34d0 2024-04-25T13:42:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.053+0000 7fc9017fa640 1 -- 172.21.15.27:0/2680636316 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc914036070 con 0x56279d4a34d0 2024-04-25T13:42:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.053+0000 7fc9017fa640 1 -- 172.21.15.27:0/2680636316 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9140c57f0 con 0x56279d4a34d0 2024-04-25T13:42:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 -- 172.21.15.27:0/2680636316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 msgr2=0x56279d86bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86bdf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc9140c67d0 tx=0x7fc9140c8d60 comp rx=0 tx=0).stop 2024-04-25T13:42:41.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc91a59c640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc9040afb60 0x7fc9040b2070 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc91c1a1960 tx=0x7fc9100113e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 -- 172.21.15.27:0/2680636316 shutdown_connections 2024-04-25T13:42:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86bdf0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d864ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 --2- 172.21.15.27:0/2680636316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d864970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 -- 172.21.15.27:0/2680636316 >> 172.21.15.27:0/2680636316 conn(0x56279d4b8300 msgr2=0x56279d860300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:41.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 -- 172.21.15.27:0/2680636316 shutdown_connections 2024-04-25T13:42:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 -- 172.21.15.27:0/2680636316 wait complete. 2024-04-25T13:42:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.054+0000 7fc928005740 1 -- start start 2024-04-25T13:42:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc928005740 1 Processor -- start 2024-04-25T13:42:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc928005740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d86e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc928005740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc928005740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d8756f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc928005740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56279d7c6420 con 0x56279d4c41b0 2024-04-25T13:42:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc928005740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56279d751b60 con 0x56279d49b2f0 2024-04-25T13:42:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc928005740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56279d86c600 con 0x56279d4a34d0 2024-04-25T13:42:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc928005740 10 client.0 _ll_register_callbacks cb 0x56279d7762b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d8756f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d8756f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59092/0 (socket says 172.21.15.27:59092) 2024-04-25T13:42:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91b59e640 1 -- 172.21.15.27:0/1562672873 learned_addr learned my addr 172.21.15.27:0/1562672873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91a59c640 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d86e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91b59e640 1 -- 172.21.15.27:0/1562672873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 msgr2=0x56279d86e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91b59e640 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91b59e640 1 -- 172.21.15.27:0/1562672873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 msgr2=0x56279d86e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91b59e640 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d86e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.055+0000 7fc91b59e640 1 -- 172.21.15.27:0/1562672873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56279d4dc210 con 0x56279d4c41b0 2024-04-25T13:42:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc91b59e640 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d8756f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc91402f550 tx=0x7fc91402ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ddac6e0f51ddd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc8feff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:42:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc8feff5640 1 -- 172.21.15.27:0/1562672873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9140cfe60 con 0x56279d4c41b0 2024-04-25T13:42:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc8feff5640 1 -- 172.21.15.27:0/1562672873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc9140c9850 con 0x56279d4c41b0 2024-04-25T13:42:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 1 -- 172.21.15.27:0/1562672873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc914036070 con 0x56279d4c41b0 2024-04-25T13:42:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc8feff5640 1 -- 172.21.15.27:0/1562672873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc9140e9070 con 0x56279d4c41b0 2024-04-25T13:42:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 10 client.15684 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 1 -- 172.21.15.27:0/1562672873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56279d5a8620 con 0x56279d4c41b0 2024-04-25T13:42:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc8feff5640 1 -- 172.21.15.27:0/1562672873 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc914036070 con 0x56279d4c41b0 2024-04-25T13:42:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc8feff5640 1 -- 172.21.15.27:0/1562672873 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc9140c7540 con 0x56279d4c41b0 2024-04-25T13:42:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 10 client.15684 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 10 client.15684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 20 client.15684 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56279d482d80 0x56279d87bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 4 client.15684 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.056+0000 7fc928005740 1 -- 172.21.15.27:0/1562672873 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56279d7d8070 con 0x56279d482d80 2024-04-25T13:42:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.057+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56279d482d80 0x56279d87bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.058+0000 7fc8feff5640 10 client.15684 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.058+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56279d482d80 0x56279d87bed0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9ca5c3521b18844 server_cookie=74e8207b13ba14b7 in_seq=0 out_seq=0 2024-04-25T13:42:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.058+0000 7fc8feff5640 1 -- 172.21.15.27:0/1562672873 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56279d7d8070 con 0x56279d482d80 2024-04-25T13:42:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.058+0000 7fc8feff5640 10 client.15684 handle_command_reply: tid=0 2024-04-25T13:42:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.058+0000 7fc928005740 1 client.15684 shutdown 2024-04-25T13:42:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.058+0000 7fc928005740 1 -- 172.21.15.27:0/1562672873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 msgr2=0x56279d8756f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.058+0000 7fc928005740 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d8756f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc91402f550 tx=0x7fc91402ffd0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 1 -- 172.21.15.27:0/1562672873 shutdown_connections 2024-04-25T13:42:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56279d482d80 0x56279d87bed0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc8feff5640 0 client.15684 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4c41b0 0x56279d8756f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4a34d0 0x56279d86e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 1 --2- 172.21.15.27:0/1562672873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d86e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 1 -- 172.21.15.27:0/1562672873 >> 172.21.15.27:0/1562672873 conn(0x56279d4b8300 msgr2=0x56279d863210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 1 -- 172.21.15.27:0/1562672873 shutdown_connections 2024-04-25T13:42:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 1 -- 172.21.15.27:0/1562672873 wait complete. 2024-04-25T13:42:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.059+0000 7fc928005740 20 client.15684 trim_cache size 0 max 16384 2024-04-25T13:42:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.085+0000 7fc928005740 1 Processor -- start 2024-04-25T13:42:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.090+0000 7fc928005740 1 -- start start 2024-04-25T13:42:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.091+0000 7fc928005740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d863100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.091+0000 7fc928005740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d49b2f0 0x56279d875610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.091+0000 7fc928005740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d4a34d0 0x56279d863640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.091+0000 7fc928005740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc90800c380 con 0x56279d49b2f0 2024-04-25T13:42:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.091+0000 7fc928005740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc90800ec00 con 0x56279d4a34d0 2024-04-25T13:42:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.091+0000 7fc928005740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc90800c9f0 con 0x56279d4b8300 2024-04-25T13:42:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.092+0000 7fc91ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d863100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.092+0000 7fc91ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d863100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47704/0 (socket says 172.21.15.27:47704) 2024-04-25T13:42:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.093+0000 7fc91ad9d640 1 -- 172.21.15.27:0/2480282009 learned_addr learned my addr 172.21.15.27:0/2480282009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.093+0000 7fc91b59e640 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d4a34d0 0x56279d863640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.095+0000 7fc91a59c640 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d49b2f0 0x56279d875610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.095+0000 7fc91ad9d640 1 -- 172.21.15.27:0/2480282009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d4a34d0 msgr2=0x56279d863640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.095+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d4a34d0 0x56279d863640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.095+0000 7fc91ad9d640 1 -- 172.21.15.27:0/2480282009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d49b2f0 msgr2=0x56279d875610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.095+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d49b2f0 0x56279d875610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.095+0000 7fc91ad9d640 1 -- 172.21.15.27:0/2480282009 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56279d85f830 con 0x56279d4b8300 2024-04-25T13:42:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.096+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d863100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc91c10a400 tx=0x7fc90c00bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b31a7f4054e2f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.102+0000 7fc900ff9640 1 -- 172.21.15.27:0/2480282009 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90c003c20 con 0x56279d4b8300 2024-04-25T13:42:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.103+0000 7fc900ff9640 1 -- 172.21.15.27:0/2480282009 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc90c021030 con 0x56279d4b8300 2024-04-25T13:42:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.103+0000 7fc900ff9640 1 -- 172.21.15.27:0/2480282009 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90c0176c0 con 0x56279d4b8300 2024-04-25T13:42:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.106+0000 7fc928005740 1 -- 172.21.15.27:0/2480282009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 msgr2=0x56279d863100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.106+0000 7fc928005740 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d863100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc91c10a400 tx=0x7fc90c00bf90 comp rx=0 tx=0).stop 2024-04-25T13:42:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.108+0000 7fc928005740 1 -- 172.21.15.27:0/2480282009 shutdown_connections 2024-04-25T13:42:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.108+0000 7fc928005740 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d4a34d0 0x56279d863640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.108+0000 7fc928005740 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d49b2f0 0x56279d875610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.108+0000 7fc928005740 1 --2- 172.21.15.27:0/2480282009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d863100 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.108+0000 7fc928005740 1 -- 172.21.15.27:0/2480282009 >> 172.21.15.27:0/2480282009 conn(0x56279d868330 msgr2=0x56279d868a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.111+0000 7fc928005740 1 -- 172.21.15.27:0/2480282009 shutdown_connections 2024-04-25T13:42:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.111+0000 7fc928005740 1 -- 172.21.15.27:0/2480282009 wait complete. 2024-04-25T13:42:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.111+0000 7fc928005740 1 -- start start 2024-04-25T13:42:41.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.111+0000 7fc928005740 1 Processor -- start 2024-04-25T13:42:41.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc928005740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d873350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc928005740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4a34d0 0x56279d873890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc91ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d873350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc91ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d873350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45066/0 (socket says 172.21.15.27:45066) 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc91ad9d640 1 -- 172.21.15.27:0/977758226 learned_addr learned my addr 172.21.15.27:0/977758226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc928005740 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d874750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc90800d0c0 con 0x56279d4a34d0 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc90800e530 con 0x56279d49b2f0 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc90800d790 con 0x56279d4b8300 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc91b59e640 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d874750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc928005740 10 client.0 _ll_register_callbacks cb 0x7fc90c00b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.112+0000 7fc91a59c640 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4a34d0 0x56279d873890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.113+0000 7fc91ad9d640 1 -- 172.21.15.27:0/977758226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 msgr2=0x56279d874750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.113+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d874750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.113+0000 7fc91ad9d640 1 -- 172.21.15.27:0/977758226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4a34d0 msgr2=0x56279d873890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.113+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4a34d0 0x56279d873890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.113+0000 7fc91ad9d640 1 -- 172.21.15.27:0/977758226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56279d874d10 con 0x56279d49b2f0 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.113+0000 7fc91b59e640 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d874750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.114+0000 7fc91ad9d640 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d873350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc90c0043f0 tx=0x7fc90c003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8939730810ae3c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.114+0000 7fc8feff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.115+0000 7fc8feff5640 1 -- 172.21.15.27:0/977758226 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90c01d540 con 0x56279d49b2f0 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.115+0000 7fc8feff5640 1 -- 172.21.15.27:0/977758226 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc90c01d6e0 con 0x56279d49b2f0 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.115+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc90c021030 con 0x56279d49b2f0 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.115+0000 7fc8feff5640 1 -- 172.21.15.27:0/977758226 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc90c01d8e0 con 0x56279d49b2f0 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.115+0000 7fc928005740 10 client.25418 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.115+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56279d8e42b0 con 0x56279d49b2f0 2024-04-25T13:42:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.117+0000 7fc8feff5640 1 -- 172.21.15.27:0/977758226 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc90c021030 con 0x56279d49b2f0 2024-04-25T13:42:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.117+0000 7fc8feff5640 1 -- 172.21.15.27:0/977758226 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc90c0265b0 con 0x56279d49b2f0 2024-04-25T13:42:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.118+0000 7fc928005740 10 client.25418 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.118+0000 7fc928005740 10 client.25418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.118+0000 7fc928005740 20 client.25418 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.118+0000 7fc928005740 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56279d4c41b0 0x56279d8e6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.118+0000 7fc928005740 4 client.25418 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:42:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.118+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56279d7d1ee0 con 0x56279d4c41b0 2024-04-25T13:42:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.119+0000 7fc91a59c640 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56279d4c41b0 0x56279d8e6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.121+0000 7fc91a59c640 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56279d4c41b0 0x56279d8e6c30 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a84055e58ea9e52 server_cookie=d9686fee497ad5f0 in_seq=0 out_seq=0 2024-04-25T13:42:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.121+0000 7fc8feff5640 10 client.25418 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.139+0000 7fc8feff5640 1 -- 172.21.15.27:0/977758226 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56279d7d1ee0 con 0x56279d4c41b0 2024-04-25T13:42:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.139+0000 7fc8feff5640 10 client.25418 handle_command_reply: tid=0 2024-04-25T13:42:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.139+0000 7fc928005740 1 client.25418 shutdown 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 msgr2=0x56279d873350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d873350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc90c0043f0 tx=0x7fc90c003cd0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 shutdown_connections 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56279d4c41b0 0x56279d8e6c30 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56279d4b8300 0x56279d874750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56279d4a34d0 0x56279d873890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 --2- 172.21.15.27:0/977758226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56279d49b2f0 0x56279d873350 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 >> 172.21.15.27:0/977758226 conn(0x56279d868330 msgr2=0x56279d863100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc8feff5640 0 client.25418 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 shutdown_connections 2024-04-25T13:42:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 1 -- 172.21.15.27:0/977758226 wait complete. 2024-04-25T13:42:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.140+0000 7fc928005740 20 client.25418 trim_cache size 0 max 16384 2024-04-25T13:42:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:42:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:42:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:42:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:42:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 -- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc9040afb60 msgr2=0x7fc9040b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc9040afb60 0x7fc9040b2070 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc91c1a1960 tx=0x7fc9100113e0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 -- 172.21.15.27:0/664666248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c110f70 msgr2=0x7fc91c1a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c110f70 0x7fc91c1a08c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc914004af0 tx=0x7fc914031700 comp rx=0 tx=0).stop 2024-04-25T13:42:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 -- 172.21.15.27:0/664666248 shutdown_connections 2024-04-25T13:42:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc9040afb60 0x7fc9040b2070 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc91c110f70 0x7fc91c1a08c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc91c10e400 0x7fc91c1a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 --2- 172.21.15.27:0/664666248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc91c10b9f0 0x7fc91c1a3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 -- 172.21.15.27:0/664666248 >> 172.21.15.27:0/664666248 conn(0x7fc91c070540 msgr2=0x7fc91c10c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 -- 172.21.15.27:0/664666248 shutdown_connections 2024-04-25T13:42:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:41.141+0000 7fc921155640 1 -- 172.21.15.27:0/664666248 wait complete. 2024-04-25T13:42:42.380 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:42 smithi045 ceph-mon[32731]: pgmap v403: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 10 MiB/s wr, 873 op/s 2024-04-25T13:42:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:42 smithi027 ceph-mon[29140]: pgmap v403: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 10 MiB/s wr, 873 op/s 2024-04-25T13:42:42.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:42 smithi122 ceph-mon[32843]: pgmap v403: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 10 MiB/s wr, 873 op/s 2024-04-25T13:42:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 95.0 s, 3558.8 tps, lat 28.171 ms stddev 30.046 2024-04-25T13:42:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:44 smithi027 ceph-mon[29140]: pgmap v404: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 10 MiB/s wr, 898 op/s 2024-04-25T13:42:44.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:44 smithi122 ceph-mon[32843]: pgmap v404: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 10 MiB/s wr, 898 op/s 2024-04-25T13:42:44.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:44 smithi045 ceph-mon[32731]: pgmap v404: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 10 MiB/s wr, 898 op/s 2024-04-25T13:42:46.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:46 smithi027 ceph-mon[29140]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 8.6 MiB/s wr, 797 op/s 2024-04-25T13:42:46.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:46 smithi122 ceph-mon[32843]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 8.6 MiB/s wr, 797 op/s 2024-04-25T13:42:46.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:46 smithi045 ceph-mon[32731]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 8.6 MiB/s wr, 797 op/s 2024-04-25T13:42:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 100.0 s, 3754.2 tps, lat 26.548 ms stddev 27.667 2024-04-25T13:42:48.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:48 smithi027 ceph-mon[29140]: pgmap v406: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 983 op/s 2024-04-25T13:42:48.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:48 smithi122 ceph-mon[32843]: pgmap v406: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 983 op/s 2024-04-25T13:42:48.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:48 smithi045 ceph-mon[32731]: pgmap v406: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 983 op/s 2024-04-25T13:42:50.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:50 smithi027 ceph-mon[29140]: pgmap v407: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.5 MiB/s wr, 679 op/s 2024-04-25T13:42:50.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:50 smithi122 ceph-mon[32843]: pgmap v407: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.5 MiB/s wr, 679 op/s 2024-04-25T13:42:50.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:50 smithi045 ceph-mon[32731]: pgmap v407: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.5 MiB/s wr, 679 op/s 2024-04-25T13:42:52.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:52 smithi027 ceph-mon[29140]: pgmap v408: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 MiB/s wr, 870 op/s 2024-04-25T13:42:52.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:52 smithi122 ceph-mon[32843]: pgmap v408: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 MiB/s wr, 870 op/s 2024-04-25T13:42:52.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:52 smithi045 ceph-mon[32731]: pgmap v408: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 MiB/s wr, 870 op/s 2024-04-25T13:42:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 105.0 s, 3317.4 tps, lat 30.258 ms stddev 32.931 2024-04-25T13:42:54.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.292+0000 7f6a3c584640 1 -- 172.21.15.27:0/490457474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 msgr2=0x7f6a34102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.292+0000 7f6a3c584640 1 --2- 172.21.15.27:0/490457474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 0x7f6a34102cd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009900 tx=0x7f6a3002a410 comp rx=0 tx=0).stop 2024-04-25T13:42:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.292+0000 7f6a3c584640 1 -- 172.21.15.27:0/490457474 shutdown_connections 2024-04-25T13:42:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.292+0000 7f6a3c584640 1 --2- 172.21.15.27:0/490457474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a34107970 0x7f6a34107d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.292+0000 7f6a3c584640 1 --2- 172.21.15.27:0/490457474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 0x7f6a34102cd0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.292+0000 7f6a3c584640 1 --2- 172.21.15.27:0/490457474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a34101dc0 0x7f6a34102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.292+0000 7f6a3c584640 1 -- 172.21.15.27:0/490457474 >> 172.21.15.27:0/490457474 conn(0x7f6a340fd390 msgr2=0x7f6a340ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:54.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.293+0000 7f6a3c584640 1 -- 172.21.15.27:0/490457474 shutdown_connections 2024-04-25T13:42:54.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.293+0000 7f6a3c584640 1 -- 172.21.15.27:0/490457474 wait complete. 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.294+0000 7f6a3c584640 1 Processor -- start 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.294+0000 7f6a3c584640 1 -- start start 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.294+0000 7f6a3c584640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a34101dc0 0x7f6a341a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.294+0000 7f6a3c584640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 0x7f6a341a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3c584640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a34107970 0x7f6a341a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3c584640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a34115c40 con 0x7f6a34102800 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3c584640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a34115ac0 con 0x7f6a34107970 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3c584640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6a34115dc0 con 0x7f6a34101dc0 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 0x7f6a341a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 0x7f6a341a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33790/0 (socket says 172.21.15.27:33790) 2024-04-25T13:42:54.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3a320640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a34101dc0 0x7f6a341a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a39b1f640 1 -- 172.21.15.27:0/1952334959 learned_addr learned my addr 172.21.15.27:0/1952334959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:54.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a34107970 0x7f6a341a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3a320640 1 -- 172.21.15.27:0/1952334959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a34107970 msgr2=0x7f6a341a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3a320640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a34107970 0x7f6a341a7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3a320640 1 -- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 msgr2=0x7f6a341a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3a320640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 0x7f6a341a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3a320640 1 -- 172.21.15.27:0/1952334959 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a341a8340 con 0x7f6a34101dc0 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.295+0000 7f6a3a320640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a34101dc0 0x7f6a341a0800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a2400e9f0 tx=0x7f6a2400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.296+0000 7f6a2b7fe640 1 -- 172.21.15.27:0/1952334959 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2400ccc0 con 0x7f6a34101dc0 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.296+0000 7f6a2b7fe640 1 -- 172.21.15.27:0/1952334959 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a24005020 con 0x7f6a34101dc0 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.296+0000 7f6a2b7fe640 1 -- 172.21.15.27:0/1952334959 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24004a00 con 0x7f6a34101dc0 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.297+0000 7f6a3c584640 1 -- 172.21.15.27:0/1952334959 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a341a8630 con 0x7f6a34101dc0 2024-04-25T13:42:54.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.297+0000 7f6a3c584640 1 -- 172.21.15.27:0/1952334959 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a341a8b20 con 0x7f6a34101dc0 2024-04-25T13:42:54.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.300+0000 7f6a2b7fe640 1 -- 172.21.15.27:0/1952334959 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6a24004ba0 con 0x7f6a34101dc0 2024-04-25T13:42:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.301+0000 7f6a43435740 1 Processor -- start 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a43435740 1 -- start start 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a2b7fe640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a200afc30 0x7f6a200b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a43435740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d1249e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a43435740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120d52f0 0x555d1249eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a43435740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120dd4d0 0x555d124a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a200afc30 0x7f6a200b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a43435740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555d12404460 con 0x555d120d52f0 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a43435740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555d123be590 con 0x555d120dd4d0 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a43435740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555d12400420 con 0x555d120fe1b0 2024-04-25T13:42:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120d52f0 0x555d1249eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120d52f0 0x555d1249eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33798/0 (socket says 172.21.15.27:33798) 2024-04-25T13:42:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.302+0000 7f6a39b1f640 1 -- 172.21.15.27:0/4039903245 learned_addr learned my addr 172.21.15.27:0/4039903245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a3a320640 1 --2- 172.21.15.27:0/4039903245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d1249e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a39b1f640 1 -- 172.21.15.27:0/4039903245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 msgr2=0x555d1249e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a39b1f640 1 --2- 172.21.15.27:0/4039903245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d1249e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a39b1f640 1 -- 172.21.15.27:0/4039903245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120dd4d0 msgr2=0x555d124a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a39b1f640 1 --2- 172.21.15.27:0/4039903245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120dd4d0 0x555d124a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a39b1f640 1 -- 172.21.15.27:0/4039903245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d123cb0d0 con 0x555d120d52f0 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a39b1f640 1 --2- 172.21.15.27:0/4039903245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120d52f0 0x555d1249eef0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6a30000c00 tx=0x7f6a30031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241a80efffa4bdb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a2b7fe640 1 -- 172.21.15.27:0/1952334959 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6a240deae0 con 0x7f6a34101dc0 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.303+0000 7f6a28bfe640 1 -- 172.21.15.27:0/4039903245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30035030 con 0x555d120d52f0 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.304+0000 7f6a28bfe640 1 -- 172.21.15.27:0/4039903245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a30040070 con 0x555d120d52f0 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.304+0000 7f6a28bfe640 1 -- 172.21.15.27:0/4039903245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30003a10 con 0x555d120d52f0 2024-04-25T13:42:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.305+0000 7f6a43435740 1 -- 172.21.15.27:0/4039903245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120d52f0 msgr2=0x555d1249eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.305+0000 7f6a43435740 1 --2- 172.21.15.27:0/4039903245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120d52f0 0x555d1249eef0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6a30000c00 tx=0x7f6a30031b10 comp rx=0 tx=0).stop 2024-04-25T13:42:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.305+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a200afc30 0x7f6a200b2140 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c005fd0 tx=0x7f6a2c009b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.305+0000 7f6a43435740 1 -- 172.21.15.27:0/4039903245 shutdown_connections 2024-04-25T13:42:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.305+0000 7f6a43435740 1 --2- 172.21.15.27:0/4039903245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120dd4d0 0x555d124a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.305+0000 7f6a43435740 1 --2- 172.21.15.27:0/4039903245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120d52f0 0x555d1249eef0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.305+0000 7f6a43435740 1 --2- 172.21.15.27:0/4039903245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d1249e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.305+0000 7f6a43435740 1 -- 172.21.15.27:0/4039903245 >> 172.21.15.27:0/4039903245 conn(0x555d120f2300 msgr2=0x555d1249a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.306+0000 7f6a43435740 1 -- 172.21.15.27:0/4039903245 shutdown_connections 2024-04-25T13:42:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.306+0000 7f6a43435740 1 -- 172.21.15.27:0/4039903245 wait complete. 2024-04-25T13:42:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.306+0000 7f6a43435740 1 -- start start 2024-04-25T13:42:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a43435740 1 Processor -- start 2024-04-25T13:42:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a43435740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a43435740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a43435740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a3a320640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33814/0 (socket says 172.21.15.27:33814) 2024-04-25T13:42:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a39b1f640 1 -- 172.21.15.27:0/2260363913 learned_addr learned my addr 172.21.15.27:0/2260363913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555d124a67e0 con 0x555d120dd4d0 2024-04-25T13:42:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555d1238bb60 con 0x555d120d52f0 2024-04-25T13:42:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555d124a6960 con 0x555d120fe1b0 2024-04-25T13:42:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.307+0000 7f6a43435740 10 client.0 _ll_register_callbacks cb 0x555d123b02b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a3ab21640 1 -- 172.21.15.27:0/2260363913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 msgr2=0x555d124a85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a3ab21640 1 -- 172.21.15.27:0/2260363913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 msgr2=0x555d124a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a3ab21640 1 -- 172.21.15.27:0/2260363913 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d12116210 con 0x555d120fe1b0 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124afa90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c010b60 tx=0x7f6a2c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be3560d84bff9861 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a1dffb640 1 -- 172.21.15.27:0/2260363913 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c027060 con 0x555d120fe1b0 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a1dffb640 1 -- 172.21.15.27:0/2260363913 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6a2c00eb40 con 0x555d120fe1b0 2024-04-25T13:42:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.308+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a30040070 con 0x555d120fe1b0 2024-04-25T13:42:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.309+0000 7f6a1dffb640 1 -- 172.21.15.27:0/2260363913 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6a2c017020 con 0x555d120fe1b0 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.310+0000 7f6a43435740 10 client.25441 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.310+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555d121e2620 con 0x555d120fe1b0 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.310+0000 7f6a1dffb640 1 -- 172.21.15.27:0/2260363913 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a30040070 con 0x555d120fe1b0 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.311+0000 7f6a1dffb640 1 -- 172.21.15.27:0/2260363913 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6a2c022430 con 0x555d120fe1b0 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.311+0000 7f6a43435740 10 client.25441 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.311+0000 7f6a43435740 10 client.25441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.311+0000 7f6a43435740 20 client.25441 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.311+0000 7f6a43435740 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d120bcd80 0x555d124b6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.311+0000 7f6a43435740 4 client.25441 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.311+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d12412070 con 0x555d120bcd80 2024-04-25T13:42:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.311+0000 7f6a3a320640 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d120bcd80 0x555d124b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.312+0000 7f6a3a320640 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d120bcd80 0x555d124b6080 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae1aa3b9f57e418b server_cookie=1e7f574b7a7b3f42 in_seq=0 out_seq=0 2024-04-25T13:42:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.312+0000 7f6a1dffb640 10 client.25441 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.312+0000 7f6a1dffb640 1 -- 172.21.15.27:0/2260363913 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555d12412070 con 0x555d120bcd80 2024-04-25T13:42:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.312+0000 7f6a1dffb640 10 client.25441 handle_command_reply: tid=0 2024-04-25T13:42:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.312+0000 7f6a43435740 1 client.25441 shutdown 2024-04-25T13:42:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 msgr2=0x555d124afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124afa90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c010b60 tx=0x7f6a2c004ab0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 shutdown_connections 2024-04-25T13:42:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d120bcd80 0x555d124b6080 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124afa90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 --2- 172.21.15.27:0/2260363913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 >> 172.21.15.27:0/2260363913 conn(0x555d120f2300 msgr2=0x555d1249d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a1dffb640 0 client.25441 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 shutdown_connections 2024-04-25T13:42:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 1 -- 172.21.15.27:0/2260363913 wait complete. 2024-04-25T13:42:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.313+0000 7f6a43435740 20 client.25441 trim_cache size 0 max 16384 2024-04-25T13:42:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.337+0000 7f6a43435740 1 Processor -- start 2024-04-25T13:42:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.337+0000 7f6a43435740 1 -- start start 2024-04-25T13:42:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a43435740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a43435740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a43435740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a43435740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341af730 con 0x555d120dd4d0 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a43435740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341b1fb0 con 0x555d120d52f0 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a43435740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341afda0 con 0x555d120fe1b0 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a3a320640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a3a320640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48362/0 (socket says 172.21.15.27:48362) 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a3a320640 1 -- 172.21.15.27:0/1059944441 learned_addr learned my addr 172.21.15.27:0/1059944441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a39b1f640 1 -- 172.21.15.27:0/1059944441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 msgr2=0x555d124af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a39b1f640 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a39b1f640 1 -- 172.21.15.27:0/1059944441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 msgr2=0x555d124a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a39b1f640 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a39b1f640 1 -- 172.21.15.27:0/1059944441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d124afef0 con 0x555d120dd4d0 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a3a320640 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:42:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.338+0000 7f6a39b1f640 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a3010 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6a30004610 tx=0x7f6a300045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e5645402a955b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a1ffff640 1 -- 172.21.15.27:0/1059944441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30035070 con 0x555d120dd4d0 2024-04-25T13:42:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a1ffff640 1 -- 172.21.15.27:0/1059944441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a30040040 con 0x555d120dd4d0 2024-04-25T13:42:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a1ffff640 1 -- 172.21.15.27:0/1059944441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a300456d0 con 0x555d120dd4d0 2024-04-25T13:42:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a43435740 1 -- 172.21.15.27:0/1059944441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 msgr2=0x555d124a3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a43435740 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a3010 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6a30004610 tx=0x7f6a300045a0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a43435740 1 -- 172.21.15.27:0/1059944441 shutdown_connections 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a43435740 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a43435740 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124a3010 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.339+0000 7f6a43435740 1 --2- 172.21.15.27:0/1059944441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a43435740 1 -- 172.21.15.27:0/1059944441 >> 172.21.15.27:0/1059944441 conn(0x555d120f2300 msgr2=0x555d124a6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a43435740 1 -- 172.21.15.27:0/1059944441 shutdown_connections 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a43435740 1 -- 172.21.15.27:0/1059944441 wait complete. 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a43435740 1 -- start start 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a43435740 1 Processor -- start 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a43435740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a43435740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124ad340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a43435740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124ae270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a3a320640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.340+0000 7f6a3a320640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48370/0 (socket says 172.21.15.27:48370) 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a39b1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124ad340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33836/0 (socket says 172.21.15.27:33836) 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a3a320640 1 -- 172.21.15.27:0/1157452552 learned_addr learned my addr 172.21.15.27:0/1157452552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341b0470 con 0x555d120dd4d0 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341b18e0 con 0x555d120d52f0 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341b0b40 con 0x555d120fe1b0 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a43435740 10 client.0 _ll_register_callbacks cb 0x7f6a2c009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a3ab21640 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a3a320640 1 -- 172.21.15.27:0/1157452552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 msgr2=0x555d124ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a3a320640 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a3a320640 1 -- 172.21.15.27:0/1157452552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 msgr2=0x555d124ad340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a3a320640 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124ad340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.341+0000 7f6a3a320640 1 -- 172.21.15.27:0/1157452552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d124ae830 con 0x555d120d52f0 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.342+0000 7f6a3a320640 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124ace00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a240dc580 tx=0x7f6a240096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eca76993136e8260 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.342+0000 7f6a1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.342+0000 7f6a1dffb640 1 -- 172.21.15.27:0/1157452552 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2401c040 con 0x555d120d52f0 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.342+0000 7f6a1dffb640 1 -- 172.21.15.27:0/1157452552 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6a240dc6c0 con 0x555d120d52f0 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.342+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a30040040 con 0x555d120d52f0 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.342+0000 7f6a1dffb640 1 -- 172.21.15.27:0/1157452552 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6a240ab340 con 0x555d120d52f0 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.342+0000 7f6a43435740 10 client.25433 fetch_fsmap learned FSMap version 19 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.342+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555d124afb90 con 0x555d120d52f0 2024-04-25T13:42:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a1dffb640 1 -- 172.21.15.27:0/1157452552 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a30040040 con 0x555d120d52f0 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a1dffb640 1 -- 172.21.15.27:0/1157452552 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6a2408e250 con 0x555d120d52f0 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a43435740 10 client.25433 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a43435740 10 client.25433 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a43435740 20 client.25433 populate_metadata read hostname 'smithi027' 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a43435740 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d120921c0 0x555d12520b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a43435740 4 client.25433 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d1240bee0 con 0x555d120921c0 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.343+0000 7f6a39b1f640 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d120921c0 0x555d12520b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.344+0000 7f6a1dffb640 10 client.25433 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.344+0000 7f6a39b1f640 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d120921c0 0x555d12520b60 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cba5866291c6d70 server_cookie=66f7541fd38c2177 in_seq=0 out_seq=0 2024-04-25T13:42:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a1dffb640 1 -- 172.21.15.27:0/1157452552 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d1240bee0 con 0x555d120921c0 2024-04-25T13:42:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a1dffb640 10 client.25433 handle_command_reply: tid=0 2024-04-25T13:42:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 client.25433 shutdown 2024-04-25T13:42:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 msgr2=0x555d124ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124ace00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a240dc580 tx=0x7f6a240096e0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 shutdown_connections 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d120921c0 0x555d12520b60 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a1dffb640 0 client.25433 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d120fe1b0 0x555d124ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d120dd4d0 0x555d124ad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 --2- 172.21.15.27:0/1157452552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d120d52f0 0x555d124ace00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 >> 172.21.15.27:0/1157452552 conn(0x555d120f2300 msgr2=0x555d124a6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 shutdown_connections 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 1 -- 172.21.15.27:0/1157452552 wait complete. 2024-04-25T13:42:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.358+0000 7f6a43435740 20 client.25433 trim_cache size 0 max 16384 2024-04-25T13:42:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:42:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:42:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:42:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 -- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a200afc30 msgr2=0x7f6a200b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a200afc30 0x7f6a200b2140 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c005fd0 tx=0x7f6a2c009b10 comp rx=0 tx=0).stop 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 -- 172.21.15.27:0/1952334959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a34101dc0 msgr2=0x7f6a341a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a34101dc0 0x7f6a341a0800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a2400e9f0 tx=0x7f6a2400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 -- 172.21.15.27:0/1952334959 shutdown_connections 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6a200afc30 0x7f6a200b2140 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a34107970 0x7f6a341a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a34102800 0x7f6a341a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 --2- 172.21.15.27:0/1952334959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a34101dc0 0x7f6a341a0800 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 -- 172.21.15.27:0/1952334959 >> 172.21.15.27:0/1952334959 conn(0x7f6a340fd390 msgr2=0x7f6a34103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 -- 172.21.15.27:0/1952334959 shutdown_connections 2024-04-25T13:42:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:42:54.359+0000 7f6a3c584640 1 -- 172.21.15.27:0/1952334959 wait complete. 2024-04-25T13:42:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:54 smithi027 ceph-mon[29140]: pgmap v409: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-25T13:42:54.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:54 smithi122 ceph-mon[32843]: pgmap v409: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-25T13:42:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:54 smithi045 ceph-mon[32731]: pgmap v409: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-25T13:42:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:56 smithi027 ceph-mon[29140]: pgmap v410: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.7 MiB/s wr, 673 op/s 2024-04-25T13:42:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:56 smithi122 ceph-mon[32843]: pgmap v410: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.7 MiB/s wr, 673 op/s 2024-04-25T13:42:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:56 smithi045 ceph-mon[32731]: pgmap v410: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.7 MiB/s wr, 673 op/s 2024-04-25T13:42:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 110.0 s, 3512.6 tps, lat 28.381 ms stddev 30.233 2024-04-25T13:42:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:42:58 smithi122 ceph-mon[32843]: pgmap v411: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 864 op/s 2024-04-25T13:42:58.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:42:58 smithi045 ceph-mon[32731]: pgmap v411: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 864 op/s 2024-04-25T13:42:58.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:42:58 smithi027 ceph-mon[29140]: pgmap v411: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 864 op/s 2024-04-25T13:43:00.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:00 smithi122 ceph-mon[32843]: pgmap v412: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-25T13:43:00.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:00 smithi045 ceph-mon[32731]: pgmap v412: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-25T13:43:00.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:00 smithi027 ceph-mon[29140]: pgmap v412: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-25T13:43:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:02 smithi122 ceph-mon[32843]: pgmap v413: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.8 MiB/s wr, 828 op/s 2024-04-25T13:43:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:02 smithi045 ceph-mon[32731]: pgmap v413: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.8 MiB/s wr, 828 op/s 2024-04-25T13:43:02.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:02 smithi027 ceph-mon[29140]: pgmap v413: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.8 MiB/s wr, 828 op/s 2024-04-25T13:43:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 115.0 s, 3309.2 tps, lat 30.311 ms stddev 32.565 2024-04-25T13:43:03.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:43:03.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:43:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.348+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/3579434341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010fef0 msgr2=0x7ff2d0110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.348+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/3579434341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010fef0 0x7ff2d0110300 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009860 tx=0x7ff2c002a340 comp rx=0 tx=0).stop 2024-04-25T13:43:03.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.353+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/3579434341 shutdown_connections 2024-04-25T13:43:03.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.353+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/3579434341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010fef0 0x7ff2d0110300 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.353+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/3579434341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2d010b1d0 0x7ff2d010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.353+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/3579434341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d010a350 0x7ff2d010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.353+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/3579434341 >> 172.21.15.27:0/3579434341 conn(0x7ff2d0070400 msgr2=0x7ff2d0072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:03.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.353+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/3579434341 shutdown_connections 2024-04-25T13:43:03.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.354+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/3579434341 wait complete. 2024-04-25T13:43:03.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.355+0000 7ff2d4d0a640 1 Processor -- start 2024-04-25T13:43:03.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.355+0000 7ff2d4d0a640 1 -- start start 2024-04-25T13:43:03.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.355+0000 7ff2d4d0a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2d010a350 0x7ff2d01a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.355+0000 7ff2d4d0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010b1d0 0x7ff2d01a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.356+0000 7ff2d4d0a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d010fef0 0x7ff2d01a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.356+0000 7ff2d4d0a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0113a50 con 0x7ff2d010b1d0 2024-04-25T13:43:03.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.356+0000 7ff2d4d0a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01138d0 con 0x7ff2d010fef0 2024-04-25T13:43:03.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.356+0000 7ff2d4d0a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0113bd0 con 0x7ff2d010a350 2024-04-25T13:43:03.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.356+0000 7ff2ceffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010b1d0 0x7ff2d01a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.356+0000 7ff2ceffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010b1d0 0x7ff2d01a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33854/0 (socket says 172.21.15.27:33854) 2024-04-25T13:43:03.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.356+0000 7ff2ceffd640 1 -- 172.21.15.27:0/2342218717 learned_addr learned my addr 172.21.15.27:0/2342218717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:03.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.357+0000 7ff2ceffd640 1 -- 172.21.15.27:0/2342218717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2d010a350 msgr2=0x7ff2d01a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.357+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2d010a350 0x7ff2d01a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.357+0000 7ff2ceffd640 1 -- 172.21.15.27:0/2342218717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d010fef0 msgr2=0x7ff2d01a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.357+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d010fef0 0x7ff2d01a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.357+0000 7ff2ceffd640 1 -- 172.21.15.27:0/2342218717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d01a8100 con 0x7ff2d010b1d0 2024-04-25T13:43:03.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.357+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010b1d0 0x7ff2d01a0b00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff2c401ea20 tx=0x7ff2c401eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:03.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.358+0000 7ff2ccff9640 1 -- 172.21.15.27:0/2342218717 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4028070 con 0x7ff2d010b1d0 2024-04-25T13:43:03.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.358+0000 7ff2ccff9640 1 -- 172.21.15.27:0/2342218717 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c401abc0 con 0x7ff2d010b1d0 2024-04-25T13:43:03.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.358+0000 7ff2ccff9640 1 -- 172.21.15.27:0/2342218717 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4020630 con 0x7ff2d010b1d0 2024-04-25T13:43:03.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.359+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/2342218717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d01a83f0 con 0x7ff2d010b1d0 2024-04-25T13:43:03.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.360+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/2342218717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d01a88f0 con 0x7ff2d010b1d0 2024-04-25T13:43:03.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.368+0000 7ff2dce19740 1 Processor -- start 2024-04-25T13:43:03.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.370+0000 7ff2ccff9640 1 -- 172.21.15.27:0/2342218717 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff2c4021470 con 0x7ff2d010b1d0 2024-04-25T13:43:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.371+0000 7ff2ccff9640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2b80afd00 0x7ff2b80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.372+0000 7ff2ccff9640 1 -- 172.21.15.27:0/2342218717 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff2c40ef6c0 con 0x7ff2d010b1d0 2024-04-25T13:43:03.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.372+0000 7ff2dce19740 1 -- start start 2024-04-25T13:43:03.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.373+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 0x55b019288c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.373+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ebe4d0 0x55b0192891f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.373+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ec66b0 0x55b019289e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.373+0000 7ff2dce19740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b0191edd80 con 0x55b018ee7390 2024-04-25T13:43:03.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.373+0000 7ff2dce19740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b0191a78b0 con 0x55b018ebe4d0 2024-04-25T13:43:03.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.373+0000 7ff2dce19740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b0191e9d40 con 0x55b018ec66b0 2024-04-25T13:43:03.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2cffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ec66b0 0x55b019289e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2cffff640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2b80afd00 0x7ff2b80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2ceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ebe4d0 0x55b0192891f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2ceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ebe4d0 0x55b0192891f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48392/0 (socket says 172.21.15.27:48392) 2024-04-25T13:43:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2ceffd640 1 -- 172.21.15.27:0/1339918547 learned_addr learned my addr 172.21.15.27:0/1339918547 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2ceffd640 1 -- 172.21.15.27:0/1339918547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ec66b0 msgr2=0x55b019289e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:03.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/1339918547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ec66b0 0x55b019289e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2ceffd640 1 -- 172.21.15.27:0/1339918547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 msgr2=0x55b019288c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/1339918547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 0x55b019288c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.375+0000 7ff2ceffd640 1 -- 172.21.15.27:0/1339918547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b019178a40 con 0x55b018ebe4d0 2024-04-25T13:43:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.376+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/1339918547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ebe4d0 0x55b0192891f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40b3c70 tx=0x7ff2c40c04d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440b62857eb6593 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:03.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.376+0000 7ff2cffff640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2b80afd00 0x7ff2b80b2210 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00304b0 tx=0x7ff2c00303b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.377+0000 7ff2b5ffb640 1 -- 172.21.15.27:0/1339918547 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40b7bf0 con 0x55b018ebe4d0 2024-04-25T13:43:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.377+0000 7ff2b5ffb640 1 -- 172.21.15.27:0/1339918547 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c40bbfa0 con 0x55b018ebe4d0 2024-04-25T13:43:03.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.377+0000 7ff2b5ffb640 1 -- 172.21.15.27:0/1339918547 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4021760 con 0x55b018ebe4d0 2024-04-25T13:43:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 -- 172.21.15.27:0/1339918547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ebe4d0 msgr2=0x55b0192891f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1339918547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ebe4d0 0x55b0192891f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40b3c70 tx=0x7ff2c40c04d0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 -- 172.21.15.27:0/1339918547 shutdown_connections 2024-04-25T13:43:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1339918547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ec66b0 0x55b019289e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1339918547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ebe4d0 0x55b0192891f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1339918547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 0x55b019288c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 -- 172.21.15.27:0/1339918547 >> 172.21.15.27:0/1339918547 conn(0x55b018edb4e0 msgr2=0x55b019284480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:03.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 -- 172.21.15.27:0/1339918547 shutdown_connections 2024-04-25T13:43:03.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.378+0000 7ff2dce19740 1 -- 172.21.15.27:0/1339918547 wait complete. 2024-04-25T13:43:03.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.379+0000 7ff2dce19740 1 -- start start 2024-04-25T13:43:03.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.379+0000 7ff2dce19740 1 Processor -- start 2024-04-25T13:43:03.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.380+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019292bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.380+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019293110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.380+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b01929a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.380+0000 7ff2dce19740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b01928a930 con 0x55b018ec66b0 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.380+0000 7ff2dce19740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b019174d10 con 0x55b018ee7390 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.380+0000 7ff2dce19740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b01928aab0 con 0x55b018ebe4d0 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.380+0000 7ff2dce19740 10 client.0 _ll_register_callbacks cb 0x55b0191a1210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.381+0000 7ff2cf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019292bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.381+0000 7ff2cf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019292bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42616/0 (socket says 172.21.15.27:42616) 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.381+0000 7ff2cf7fe640 1 -- 172.21.15.27:0/2334220107 learned_addr learned my addr 172.21.15.27:0/2334220107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:03.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.381+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019293110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.381+0000 7ff2cffff640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b01929a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2cf7fe640 1 -- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 msgr2=0x55b01929a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2cf7fe640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b01929a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2cf7fe640 1 -- 172.21.15.27:0/2334220107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 msgr2=0x55b019293110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2cf7fe640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019293110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2cf7fe640 1 -- 172.21.15.27:0/2334220107 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b018fc97d0 con 0x55b018ebe4d0 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2cffff640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b01929a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019293110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2cf7fe640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019292bd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009fd0 tx=0x7ff2c800ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c24abc806b7cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.382+0000 7ff2aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2aeffd640 1 -- 172.21.15.27:0/2334220107 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8004d20 con 0x55b018ebe4d0 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 1 -- 172.21.15.27:0/2334220107 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2c40bbfa0 con 0x55b018ebe4d0 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2aeffd640 1 -- 172.21.15.27:0/2334220107 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff2c800c9c0 con 0x55b018ebe4d0 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2aeffd640 1 -- 172.21.15.27:0/2334220107 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff2c800ee80 con 0x55b018ebe4d0 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 10 client.25456 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 1 -- 172.21.15.27:0/2334220107 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b018eff3f0 con 0x55b018ebe4d0 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2aeffd640 1 -- 172.21.15.27:0/2334220107 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c40bbfa0 con 0x55b018ebe4d0 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2aeffd640 1 -- 172.21.15.27:0/2334220107 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff2c8010430 con 0x55b018ebe4d0 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 10 client.25456 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 10 client.25456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 20 client.25456 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b018ea5f60 0x55b0192a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 4 client.25456 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.383+0000 7ff2dce19740 1 -- 172.21.15.27:0/2334220107 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0191fb990 con 0x55b018ea5f60 2024-04-25T13:43:03.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.384+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b018ea5f60 0x55b0192a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.384+0000 7ff2aeffd640 10 client.25456 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:03.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.384+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b018ea5f60 0x55b0192a0640 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13bb08956975519b server_cookie=b01a6a3d6c02f01b in_seq=0 out_seq=0 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2aeffd640 1 -- 172.21.15.27:0/2334220107 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b0191fb990 con 0x55b018ea5f60 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2aeffd640 10 client.25456 handle_command_reply: tid=0 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 client.25456 shutdown 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 -- 172.21.15.27:0/2334220107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 msgr2=0x55b019292bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019292bd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009fd0 tx=0x7ff2c800ec30 comp rx=0 tx=0).stop 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 -- 172.21.15.27:0/2334220107 shutdown_connections 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b018ea5f60 0x55b0192a0640 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b01929a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019293110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2aeffd640 0 client.25456 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 --2- 172.21.15.27:0/2334220107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019292bd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 -- 172.21.15.27:0/2334220107 >> 172.21.15.27:0/2334220107 conn(0x55b018edb4e0 msgr2=0x55b019287390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 -- 172.21.15.27:0/2334220107 shutdown_connections 2024-04-25T13:43:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.385+0000 7ff2dce19740 1 -- 172.21.15.27:0/2334220107 wait complete. 2024-04-25T13:43:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.386+0000 7ff2dce19740 20 client.25456 trim_cache size 0 max 16384 2024-04-25T13:43:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2dce19740 1 Processor -- start 2024-04-25T13:43:03.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2dce19740 1 -- start start 2024-04-25T13:43:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019284520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ec66b0 0x55b019291820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 0x55b019299f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2dce19740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01af660 con 0x55b018ee7390 2024-04-25T13:43:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2dce19740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b1ee0 con 0x55b018ec66b0 2024-04-25T13:43:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2dce19740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01afcd0 con 0x55b018ebe4d0 2024-04-25T13:43:03.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.412+0000 7ff2ceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ec66b0 0x55b019291820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.413+0000 7ff2ceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ec66b0 0x55b019291820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48414/0 (socket says 172.21.15.27:48414) 2024-04-25T13:43:03.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.414+0000 7ff2ceffd640 1 -- 172.21.15.27:0/3248389625 learned_addr learned my addr 172.21.15.27:0/3248389625 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.414+0000 7ff2ceffd640 1 -- 172.21.15.27:0/3248389625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 msgr2=0x55b019284520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.414+0000 7ff2cf7fe640 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019284520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.414+0000 7ff2cffff640 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 0x55b019299f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.414+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019284520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.414+0000 7ff2ceffd640 1 -- 172.21.15.27:0/3248389625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 msgr2=0x55b019299f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.414+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 0x55b019299f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.414+0000 7ff2ceffd640 1 -- 172.21.15.27:0/3248389625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b018ebf0f0 con 0x55b018ec66b0 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.415+0000 7ff2cf7fe640 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019284520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.415+0000 7ff2cffff640 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 0x55b019299f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.415+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ec66b0 0x55b019291820 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40b7b80 tx=0x7ff2c40b3b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4672c2be16d4d3c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.415+0000 7ff2b57fa640 1 -- 172.21.15.27:0/3248389625 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40cc070 con 0x55b018ec66b0 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.415+0000 7ff2b57fa640 1 -- 172.21.15.27:0/3248389625 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c40c4030 con 0x55b018ec66b0 2024-04-25T13:43:03.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.415+0000 7ff2b57fa640 1 -- 172.21.15.27:0/3248389625 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c40b7140 con 0x55b018ec66b0 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 -- 172.21.15.27:0/3248389625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ec66b0 msgr2=0x55b019291820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ec66b0 0x55b019291820 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff2c40b7b80 tx=0x7ff2c40b3b30 comp rx=0 tx=0).stop 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 -- 172.21.15.27:0/3248389625 shutdown_connections 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ee7390 0x55b019299f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ec66b0 0x55b019291820 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 --2- 172.21.15.27:0/3248389625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b019284520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 -- 172.21.15.27:0/3248389625 >> 172.21.15.27:0/3248389625 conn(0x55b018edb4e0 msgr2=0x55b019291110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 -- 172.21.15.27:0/3248389625 shutdown_connections 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 -- 172.21.15.27:0/3248389625 wait complete. 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.416+0000 7ff2dce19740 1 -- start start 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2dce19740 1 Processor -- start 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b01929a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019297060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2dce19740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b0192975a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2dce19740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b03a0 con 0x55b018ec66b0 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2dce19740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b1810 con 0x55b018ee7390 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2dce19740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b0a70 con 0x55b018ebe4d0 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2dce19740 10 client.0 _ll_register_callbacks cb 0x7ff2c80098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2ceffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019297060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2cf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b01929a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2ceffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019297060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33882/0 (socket says 172.21.15.27:33882) 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b0192975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2cf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b01929a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42638/0 (socket says 172.21.15.27:42638) 2024-04-25T13:43:03.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2ceffd640 1 -- 172.21.15.27:0/1935891269 learned_addr learned my addr 172.21.15.27:0/1935891269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2cf7fe640 1 -- 172.21.15.27:0/1935891269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 msgr2=0x55b0192975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2cf7fe640 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b0192975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2cf7fe640 1 -- 172.21.15.27:0/1935891269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 msgr2=0x55b019297060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.417+0000 7ff2cf7fe640 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019297060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.418+0000 7ff2cf7fe640 1 -- 172.21.15.27:0/1935891269 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0191ae4b0 con 0x55b018ebe4d0 2024-04-25T13:43:03.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.418+0000 7ff2cffff640 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b0192975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.418+0000 7ff2cf7fe640 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b01929a4e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8015bb0 tx=0x7ff2c80092b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55022eb1b087f340 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.418+0000 7ff2aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.418+0000 7ff2aeffd640 1 -- 172.21.15.27:0/1935891269 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8028040 con 0x55b018ebe4d0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.418+0000 7ff2aeffd640 1 -- 172.21.15.27:0/1935891269 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff2c80152d0 con 0x55b018ebe4d0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.419+0000 7ff2dce19740 1 -- 172.21.15.27:0/1935891269 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2c40c4030 con 0x55b018ebe4d0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.419+0000 7ff2aeffd640 1 -- 172.21.15.27:0/1935891269 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff2c8016530 con 0x55b018ebe4d0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2dce19740 10 client.25459 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2dce19740 1 -- 172.21.15.27:0/1935891269 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b019307620 con 0x55b018ebe4d0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2aeffd640 1 -- 172.21.15.27:0/1935891269 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c40c4030 con 0x55b018ebe4d0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2aeffd640 1 -- 172.21.15.27:0/1935891269 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff2c8010430 con 0x55b018ebe4d0 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2dce19740 10 client.25459 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2dce19740 10 client.25459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2dce19740 20 client.25459 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b018e7b3a0 0x55b019309f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2dce19740 4 client.25459 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:43:03.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.420+0000 7ff2dce19740 1 -- 172.21.15.27:0/1935891269 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b0191f5800 con 0x55b018e7b3a0 2024-04-25T13:43:03.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.422+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b018e7b3a0 0x55b019309f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.423+0000 7ff2aeffd640 10 client.25459 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:03.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.423+0000 7ff2ceffd640 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b018e7b3a0 0x55b019309f80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4492422caa5c55c server_cookie=fab16c9141dc2d3 in_seq=0 out_seq=0 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.423+0000 7ff2aeffd640 1 -- 172.21.15.27:0/1935891269 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b0191f5800 con 0x55b018e7b3a0 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.423+0000 7ff2aeffd640 10 client.25459 handle_command_reply: tid=0 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.423+0000 7ff2dce19740 1 client.25459 shutdown 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 -- 172.21.15.27:0/1935891269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 msgr2=0x55b01929a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b01929a4e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8015bb0 tx=0x7ff2c80092b0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 -- 172.21.15.27:0/1935891269 shutdown_connections 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b018e7b3a0 0x55b019309f80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b018ee7390 0x55b0192975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b018ec66b0 0x55b019297060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 --2- 172.21.15.27:0/1935891269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b018ebe4d0 0x55b01929a4e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2aeffd640 0 client.25459 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 -- 172.21.15.27:0/1935891269 >> 172.21.15.27:0/1935891269 conn(0x55b018edb4e0 msgr2=0x55b0192918a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 -- 172.21.15.27:0/1935891269 shutdown_connections 2024-04-25T13:43:03.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 1 -- 172.21.15.27:0/1935891269 wait complete. 2024-04-25T13:43:03.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2dce19740 20 client.25459 trim_cache size 0 max 16384 2024-04-25T13:43:03.425 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:43:03.425 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:43:03.425 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:43:03.425 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:43:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2b80afd00 msgr2=0x7ff2b80b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2b80afd00 0x7ff2b80b2210 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00304b0 tx=0x7ff2c00303b0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010b1d0 msgr2=0x7ff2d01a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010b1d0 0x7ff2d01a0b00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff2c401ea20 tx=0x7ff2c401eee0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/2342218717 shutdown_connections 2024-04-25T13:43:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2b80afd00 0x7ff2b80b2210 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d010fef0 0x7ff2d01a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d010b1d0 0x7ff2d01a0b00 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 --2- 172.21.15.27:0/2342218717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2d010a350 0x7ff2d01a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.424+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/2342218717 >> 172.21.15.27:0/2342218717 conn(0x7ff2d0070400 msgr2=0x7ff2d010dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.425+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/2342218717 shutdown_connections 2024-04-25T13:43:03.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.425+0000 7ff2d4d0a640 1 -- 172.21.15.27:0/2342218717 wait complete. 2024-04-25T13:43:03.437 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:43:03.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:43:03.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.598+0000 7f9cb1138640 1 -- 172.21.15.27:0/603598486 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca402f310 con 0x7f9cac10a110 2024-04-25T13:43:03.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 -- 172.21.15.27:0/603598486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac10a110 msgr2=0x7f9cac10ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 --2- 172.21.15.27:0/603598486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac10a110 0x7f9cac10ae10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4009ca0 tx=0x7f9ca402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 -- 172.21.15.27:0/603598486 shutdown_connections 2024-04-25T13:43:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 --2- 172.21.15.27:0/603598486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cac110050 0x7f9cac112470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 --2- 172.21.15.27:0/603598486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cac10b350 0x7f9cac10b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 --2- 172.21.15.27:0/603598486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac10a110 0x7f9cac10ae10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 -- 172.21.15.27:0/603598486 >> 172.21.15.27:0/603598486 conn(0x7f9cac078de0 msgr2=0x7f9cac1079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 -- 172.21.15.27:0/603598486 shutdown_connections 2024-04-25T13:43:03.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.599+0000 7f9cb439e640 1 -- 172.21.15.27:0/603598486 wait complete. 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb439e640 1 Processor -- start 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb439e640 1 -- start start 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb439e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cac10a110 0x7f9cac1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb439e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cac10b350 0x7f9cac1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb439e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac110050 0x7f9cac1a7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb439e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac115c50 con 0x7f9cac10a110 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb439e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac115ad0 con 0x7f9cac10b350 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb439e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac115dd0 con 0x7f9cac110050 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb293b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac110050 0x7f9cac1a7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb213a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cac10a110 0x7f9cac1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb293b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac110050 0x7f9cac1a7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42644/0 (socket says 172.21.15.27:42644) 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb293b640 1 -- 172.21.15.27:0/2483316642 learned_addr learned my addr 172.21.15.27:0/2483316642 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:03.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb1939640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cac10b350 0x7f9cac1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb293b640 1 -- 172.21.15.27:0/2483316642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cac10b350 msgr2=0x7f9cac1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb293b640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cac10b350 0x7f9cac1a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb293b640 1 -- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cac10a110 msgr2=0x7f9cac1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb293b640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cac10a110 0x7f9cac1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.601+0000 7f9cb293b640 1 -- 172.21.15.27:0/2483316642 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cac1a8390 con 0x7f9cac110050 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.602+0000 7f9cb293b640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac110050 0x7f9cac1a7d20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800e970 tx=0x7f9ca800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.602+0000 7f9ca37fe640 1 -- 172.21.15.27:0/2483316642 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca800ccc0 con 0x7f9cac110050 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.602+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cac1a8680 con 0x7f9cac110050 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.602+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cac1a8bd0 con 0x7f9cac110050 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.602+0000 7f9ca37fe640 1 -- 172.21.15.27:0/2483316642 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca80054d0 con 0x7f9cac110050 2024-04-25T13:43:03.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.602+0000 7f9ca37fe640 1 -- 172.21.15.27:0/2483316642 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8010430 con 0x7f9cac110050 2024-04-25T13:43:03.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.604+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cac10ee60 con 0x7f9cac110050 2024-04-25T13:43:03.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.606+0000 7f9ca37fe640 1 -- 172.21.15.27:0/2483316642 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9ca80046c0 con 0x7f9cac110050 2024-04-25T13:43:03.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.607+0000 7f9ca37fe640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c980afb30 0x7f9c980b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.608+0000 7f9ca37fe640 1 -- 172.21.15.27:0/2483316642 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9ca80df220 con 0x7f9cac110050 2024-04-25T13:43:03.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.608+0000 7f9cb213a640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c980afb30 0x7f9c980b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:03.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.609+0000 7f9cb213a640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c980afb30 0x7f9c980b2040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4000f80 tx=0x7f9ca4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:03.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.610+0000 7f9ca37fe640 1 -- 172.21.15.27:0/2483316642 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ca80a7e30 con 0x7f9cac110050 2024-04-25T13:43:03.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.972+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cac0637a0 con 0x7f9cac110050 2024-04-25T13:43:03.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.973+0000 7f9ca37fe640 1 -- 172.21.15.27:0/2483316642 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f9ca80a7730 con 0x7f9cac110050 2024-04-25T13:43:03.973 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:43:03.974 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:43:03.974 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:43:03.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.976+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c980afb30 msgr2=0x7f9c980b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.976+0000 7f9cb439e640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c980afb30 0x7f9c980b2040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4000f80 tx=0x7f9ca4005fd0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.976+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac110050 msgr2=0x7f9cac1a7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:03.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.976+0000 7f9cb439e640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac110050 0x7f9cac1a7d20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800e970 tx=0x7f9ca800ee30 comp rx=0 tx=0).stop 2024-04-25T13:43:03.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.977+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 shutdown_connections 2024-04-25T13:43:03.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.977+0000 7f9cb439e640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c980afb30 0x7f9c980b2040 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.977+0000 7f9cb439e640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9cac110050 0x7f9cac1a7d20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.977+0000 7f9cb439e640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9cac10b350 0x7f9cac1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.977+0000 7f9cb439e640 1 --2- 172.21.15.27:0/2483316642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cac10a110 0x7f9cac1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:03.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.977+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 >> 172.21.15.27:0/2483316642 conn(0x7f9cac078de0 msgr2=0x7f9cac10c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:03.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.977+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 shutdown_connections 2024-04-25T13:43:03.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:03.977+0000 7f9cb439e640 1 -- 172.21.15.27:0/2483316642 wait complete. 2024-04-25T13:43:03.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:43:04.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.152+0000 7fba38bf8640 1 -- 172.21.15.27:0/63230033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 msgr2=0x7fba341033b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.152+0000 7fba38bf8640 1 --2- 172.21.15.27:0/63230033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 0x7fba341033b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fba24009900 tx=0x7fba2402a410 comp rx=0 tx=0).stop 2024-04-25T13:43:04.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.152+0000 7fba38bf8640 1 -- 172.21.15.27:0/63230033 shutdown_connections 2024-04-25T13:43:04.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.152+0000 7fba38bf8640 1 --2- 172.21.15.27:0/63230033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba34104460 0x7fba3410aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.152+0000 7fba38bf8640 1 --2- 172.21.15.27:0/63230033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba34103980 0x7fba34103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.152+0000 7fba38bf8640 1 --2- 172.21.15.27:0/63230033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 0x7fba341033b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.152+0000 7fba38bf8640 1 -- 172.21.15.27:0/63230033 >> 172.21.15.27:0/63230033 conn(0x7fba340fd340 msgr2=0x7fba340ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:04.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.153+0000 7fba38bf8640 1 -- 172.21.15.27:0/63230033 shutdown_connections 2024-04-25T13:43:04.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.153+0000 7fba38bf8640 1 -- 172.21.15.27:0/63230033 wait complete. 2024-04-25T13:43:04.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.154+0000 7fba38bf8640 1 Processor -- start 2024-04-25T13:43:04.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.154+0000 7fba38bf8640 1 -- start start 2024-04-25T13:43:04.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.154+0000 7fba38bf8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 0x7fba3419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba38bf8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba34103980 0x7fba3419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba38bf8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba34104460 0x7fba341a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba38bf8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba3410e510 con 0x7fba34103980 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba38bf8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba3410e390 con 0x7fba34104460 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba38bf8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fba3410e690 con 0x7fba34102fa0 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 0x7fba3419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 0x7fba3419a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39452/0 (socket says 172.21.15.27:39452) 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 -- 172.21.15.27:0/3606357284 learned_addr learned my addr 172.21.15.27:0/3606357284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba31d9b640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba34103980 0x7fba3419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba32d9d640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba34104460 0x7fba341a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 -- 172.21.15.27:0/3606357284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba34104460 msgr2=0x7fba341a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba34104460 0x7fba341a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 -- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba34103980 msgr2=0x7fba3419a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba34103980 0x7fba3419a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 -- 172.21.15.27:0/3606357284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba341a1e00 con 0x7fba34102fa0 2024-04-25T13:43:04.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.155+0000 7fba3259c640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 0x7fba3419a2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fba24006010 tx=0x7fba24004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.156+0000 7fba237fe640 1 -- 172.21.15.27:0/3606357284 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2403f030 con 0x7fba34102fa0 2024-04-25T13:43:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.156+0000 7fba237fe640 1 -- 172.21.15.27:0/3606357284 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba24003aa0 con 0x7fba34102fa0 2024-04-25T13:43:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.156+0000 7fba38bf8640 1 -- 172.21.15.27:0/3606357284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba341a2090 con 0x7fba34102fa0 2024-04-25T13:43:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.156+0000 7fba237fe640 1 -- 172.21.15.27:0/3606357284 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24035070 con 0x7fba34102fa0 2024-04-25T13:43:04.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.156+0000 7fba38bf8640 1 -- 172.21.15.27:0/3606357284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba341a2580 con 0x7fba34102fa0 2024-04-25T13:43:04.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.160+0000 7fba3fb89740 1 Processor -- start 2024-04-25T13:43:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.160+0000 7fba237fe640 1 -- 172.21.15.27:0/3606357284 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fba24042030 con 0x7fba34102fa0 2024-04-25T13:43:04.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.160+0000 7fba3fb89740 1 -- start start 2024-04-25T13:43:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.161+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 0x56262b9dc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.161+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9dcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.161+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9dda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.161+0000 7fba3fb89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56262b940d80 con 0x56262b63a390 2024-04-25T13:43:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.161+0000 7fba31d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9dda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.161+0000 7fba31d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9dda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39460/0 (socket says 172.21.15.27:39460) 2024-04-25T13:43:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.161+0000 7fba31d9b640 1 -- 172.21.15.27:0/3626331204 learned_addr learned my addr 172.21.15.27:0/3626331204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:04.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.161+0000 7fba32d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 0x56262b9dc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.162+0000 7fba3fb89740 1 -- 172.21.15.27:0/3626331204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56262b8faa20 con 0x56262b6114d0 2024-04-25T13:43:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.162+0000 7fba3fb89740 1 -- 172.21.15.27:0/3626331204 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56262b93cd40 con 0x56262b6196b0 2024-04-25T13:43:04.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.162+0000 7fba3259c640 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9dcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.162+0000 7fba31d9b640 1 -- 172.21.15.27:0/3626331204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 msgr2=0x56262b9dcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba31d9b640 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9dcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba31d9b640 1 -- 172.21.15.27:0/3626331204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 msgr2=0x56262b9dc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba31d9b640 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 0x56262b9dc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba31d9b640 1 -- 172.21.15.27:0/3626331204 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56262b905c30 con 0x56262b6196b0 2024-04-25T13:43:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba237fe640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba1c0afb70 0x7fba1c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba3259c640 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9dcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba31d9b640 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9dda20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fba28004650 tx=0x7fba2800bba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=148ef0211632a347 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba237fe640 1 -- 172.21.15.27:0/3606357284 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fba240ff100 con 0x7fba34102fa0 2024-04-25T13:43:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.163+0000 7fba32d9d640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba1c0afb70 0x7fba1c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.165+0000 7fba32d9d640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba1c0afb70 0x7fba1c0b2080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x56262b9dd860 tx=0x7fba2c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:04.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.166+0000 7fba20ff9640 1 -- 172.21.15.27:0/3626331204 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28003bd0 con 0x56262b6196b0 2024-04-25T13:43:04.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.166+0000 7fba20ff9640 1 -- 172.21.15.27:0/3626331204 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba28015070 con 0x56262b6196b0 2024-04-25T13:43:04.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.166+0000 7fba20ff9640 1 -- 172.21.15.27:0/3626331204 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28004d80 con 0x56262b6196b0 2024-04-25T13:43:04.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.167+0000 7fba3fb89740 1 -- 172.21.15.27:0/3626331204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 msgr2=0x56262b9dda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.167+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9dda20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fba28004650 tx=0x7fba2800bba0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.168+0000 7fba3fb89740 1 -- 172.21.15.27:0/3626331204 shutdown_connections 2024-04-25T13:43:04.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.168+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9dda20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.168+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9dcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.168+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3626331204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 0x56262b9dc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.168+0000 7fba3fb89740 1 -- 172.21.15.27:0/3626331204 >> 172.21.15.27:0/3626331204 conn(0x56262b62e4e0 msgr2=0x56262b9d8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:04.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.168+0000 7fba3fb89740 1 -- 172.21.15.27:0/3626331204 shutdown_connections 2024-04-25T13:43:04.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.168+0000 7fba3fb89740 1 -- 172.21.15.27:0/3626331204 wait complete. 2024-04-25T13:43:04.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.169+0000 7fba3fb89740 1 -- start start 2024-04-25T13:43:04.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.169+0000 7fba3fb89740 1 Processor -- start 2024-04-25T13:43:04.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b6114d0 0x56262b9ea770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9eacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b63a390 0x56262b9f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3fb89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56262b9de530 con 0x56262b6114d0 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3fb89740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56262b8c7dc0 con 0x56262b63a390 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3fb89740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56262b9de6b0 con 0x56262b6196b0 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b6114d0 0x56262b9ea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b6114d0 0x56262b9ea770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36338/0 (socket says 172.21.15.27:36338) 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3259c640 1 -- 172.21.15.27:0/3783654616 learned_addr learned my addr 172.21.15.27:0/3783654616 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3fb89740 10 client.0 _ll_register_callbacks cb 0x56262b8e3600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba31d9b640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9eacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba32d9d640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b63a390 0x56262b9f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3259c640 1 -- 172.21.15.27:0/3783654616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 msgr2=0x56262b9eacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:04.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3259c640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9eacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.170+0000 7fba3259c640 1 -- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b63a390 msgr2=0x56262b9f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:04.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.171+0000 7fba3259c640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b63a390 0x56262b9f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.171+0000 7fba3259c640 1 -- 172.21.15.27:0/3783654616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56262b6523f0 con 0x56262b6114d0 2024-04-25T13:43:04.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.171+0000 7fba31d9b640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9eacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:04.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.171+0000 7fba32d9d640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b63a390 0x56262b9f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:04.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.171+0000 7fba3259c640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b6114d0 0x56262b9ea770 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba240c3630 tx=0x7fba2403b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcc143cf75c8294 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:04.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.171+0000 7fba19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:43:04.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.172+0000 7fba19ffb640 1 -- 172.21.15.27:0/3783654616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24049070 con 0x56262b6114d0 2024-04-25T13:43:04.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.172+0000 7fba19ffb640 1 -- 172.21.15.27:0/3783654616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fba240c3840 con 0x56262b6114d0 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.172+0000 7fba3fb89740 1 -- 172.21.15.27:0/3783654616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba28015070 con 0x56262b6114d0 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.173+0000 7fba19ffb640 1 -- 172.21.15.27:0/3783654616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fba240c70c0 con 0x56262b6114d0 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.173+0000 7fba3fb89740 10 client.15738 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.173+0000 7fba3fb89740 1 -- 172.21.15.27:0/3783654616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56262b71c7d0 con 0x56262b6114d0 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.173+0000 7fba19ffb640 1 -- 172.21.15.27:0/3783654616 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba28015070 con 0x56262b6114d0 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.174+0000 7fba19ffb640 1 -- 172.21.15.27:0/3783654616 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fba240cece0 con 0x56262b6114d0 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.174+0000 7fba3fb89740 10 client.15738 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.174+0000 7fba3fb89740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.174+0000 7fba3fb89740 20 client.15738 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:04.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.174+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56262b5f8f60 0x56262b9f83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.174+0000 7fba3fb89740 4 client.15738 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.174+0000 7fba3fb89740 1 -- 172.21.15.27:0/3783654616 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56262b94e990 con 0x56262b5f8f60 2024-04-25T13:43:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.174+0000 7fba31d9b640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56262b5f8f60 0x56262b9f83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.175+0000 7fba19ffb640 10 client.15738 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.175+0000 7fba31d9b640 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56262b5f8f60 0x56262b9f83d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17717f895b1293f2 server_cookie=e136ffa0beb8bf0b in_seq=0 out_seq=0 2024-04-25T13:43:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.176+0000 7fba19ffb640 1 -- 172.21.15.27:0/3783654616 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56262b94e990 con 0x56262b5f8f60 2024-04-25T13:43:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.176+0000 7fba19ffb640 10 client.15738 handle_command_reply: tid=0 2024-04-25T13:43:04.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.176+0000 7fba3fb89740 1 client.15738 shutdown 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.176+0000 7fba3fb89740 1 -- 172.21.15.27:0/3783654616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b6114d0 msgr2=0x56262b9ea770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.176+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b6114d0 0x56262b9ea770 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba240c3630 tx=0x7fba2403b6b0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 1 -- 172.21.15.27:0/3783654616 shutdown_connections 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56262b5f8f60 0x56262b9f83d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b63a390 0x56262b9f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9eacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 1 --2- 172.21.15.27:0/3783654616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b6114d0 0x56262b9ea770 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 1 -- 172.21.15.27:0/3783654616 >> 172.21.15.27:0/3783654616 conn(0x56262b62e4e0 msgr2=0x56262b9d7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:04.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba19ffb640 0 client.15738 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:04.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 1 -- 172.21.15.27:0/3783654616 shutdown_connections 2024-04-25T13:43:04.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 1 -- 172.21.15.27:0/3783654616 wait complete. 2024-04-25T13:43:04.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.177+0000 7fba3fb89740 20 client.15738 trim_cache size 0 max 16384 2024-04-25T13:43:04.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3fb89740 1 Processor -- start 2024-04-25T13:43:04.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3fb89740 1 -- start start 2024-04-25T13:43:04.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9d7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 0x56262b9d7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3fb89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba341a6ff0 con 0x56262b63a390 2024-04-25T13:43:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3fb89740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba341a9870 con 0x56262b6114d0 2024-04-25T13:43:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3fb89740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fba341a7660 con 0x56262b6196b0 2024-04-25T13:43:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba31d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba32d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 0x56262b9d7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba31d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39480/0 (socket says 172.21.15.27:39480) 2024-04-25T13:43:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba31d9b640 1 -- 172.21.15.27:0/1950274787 learned_addr learned my addr 172.21.15.27:0/1950274787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.200+0000 7fba3259c640 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9d7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba31d9b640 1 -- 172.21.15.27:0/1950274787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 msgr2=0x56262b9d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba31d9b640 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9d7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba31d9b640 1 -- 172.21.15.27:0/1950274787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 msgr2=0x56262b9d7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba31d9b640 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 0x56262b9d7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba31d9b640 1 -- 172.21.15.27:0/1950274787 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56262b9e6080 con 0x56262b6196b0 2024-04-25T13:43:04.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba3259c640 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9d7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:04.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba31d9b640 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9f1ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c010 tx=0x7fba2800b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1a53d13a80809ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:04.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba1bfff640 1 -- 172.21.15.27:0/1950274787 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28003bd0 con 0x56262b6196b0 2024-04-25T13:43:04.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba1bfff640 1 -- 172.21.15.27:0/1950274787 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba28015040 con 0x56262b6196b0 2024-04-25T13:43:04.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.201+0000 7fba1bfff640 1 -- 172.21.15.27:0/1950274787 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2802aab0 con 0x56262b6196b0 2024-04-25T13:43:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.202+0000 7fba3fb89740 1 -- 172.21.15.27:0/1950274787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 msgr2=0x56262b9f1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.202+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9f1ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c010 tx=0x7fba2800b420 comp rx=0 tx=0).stop 2024-04-25T13:43:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.203+0000 7fba3fb89740 1 -- 172.21.15.27:0/1950274787 shutdown_connections 2024-04-25T13:43:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.203+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b63a390 0x56262b9d7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.203+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b6196b0 0x56262b9f1ba0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c010 tx=0x7fba2800b420 comp rx=0 tx=0).stop 2024-04-25T13:43:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.203+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1950274787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9d7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.203+0000 7fba3fb89740 1 -- 172.21.15.27:0/1950274787 >> 172.21.15.27:0/1950274787 conn(0x56262b62e4e0 msgr2=0x56262b9d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.204+0000 7fba3fb89740 1 -- 172.21.15.27:0/1950274787 shutdown_connections 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.204+0000 7fba3fb89740 1 -- 172.21.15.27:0/1950274787 wait complete. 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.204+0000 7fba3fb89740 1 -- start start 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3fb89740 1 Processor -- start 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9ec8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b63a390 0x56262b9ece10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3fb89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b5ce3a0 0x56262ba5e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3fb89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba341a7d30 con 0x56262b5ce3a0 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3fb89740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba341a91a0 con 0x56262b6114d0 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3fb89740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fba341a8400 con 0x56262b63a390 2024-04-25T13:43:04.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3fb89740 10 client.0 _ll_register_callbacks cb 0x7fba24005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba32d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b5ce3a0 0x56262ba5e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba32d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b5ce3a0 0x56262ba5e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36364/0 (socket says 172.21.15.27:36364) 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba32d9d640 1 -- 172.21.15.27:0/1860354454 learned_addr learned my addr 172.21.15.27:0/1860354454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba31d9b640 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b63a390 0x56262b9ece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.205+0000 7fba3259c640 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9ec8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba32d9d640 1 -- 172.21.15.27:0/1860354454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b63a390 msgr2=0x56262b9ece10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba32d9d640 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b63a390 0x56262b9ece10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba32d9d640 1 -- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 msgr2=0x56262b9ec8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba32d9d640 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9ec8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba32d9d640 1 -- 172.21.15.27:0/1860354454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56262b9f1de0 con 0x56262b5ce3a0 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba3259c640 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9ec8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba32d9d640 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b5ce3a0 0x56262ba5e770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fba2c01e970 tx=0x7fba2c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c60f1f70fdc321 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba19ffb640 1 -- 172.21.15.27:0/1860354454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c01db20 con 0x56262b5ce3a0 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.206+0000 7fba19ffb640 1 -- 172.21.15.27:0/1860354454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fba2c01dcc0 con 0x56262b5ce3a0 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.207+0000 7fba3fb89740 1 -- 172.21.15.27:0/1860354454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba28015040 con 0x56262b5ce3a0 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.207+0000 7fba19ffb640 1 -- 172.21.15.27:0/1860354454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fba2c035460 con 0x56262b5ce3a0 2024-04-25T13:43:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.207+0000 7fba3fb89740 10 client.15744 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.207+0000 7fba3fb89740 1 -- 172.21.15.27:0/1860354454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56262ba5fd00 con 0x56262b5ce3a0 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.208+0000 7fba19ffb640 1 -- 172.21.15.27:0/1860354454 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba2c02e070 con 0x56262b5ce3a0 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.208+0000 7fba19ffb640 1 -- 172.21.15.27:0/1860354454 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fba2c0116e0 con 0x56262b5ce3a0 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.208+0000 7fba3fb89740 10 client.15744 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.208+0000 7fba3fb89740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.208+0000 7fba3fb89740 20 client.15744 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.208+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56262ba60710 0x56262ba62b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.208+0000 7fba3fb89740 4 client.15744 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.208+0000 7fba3fb89740 1 -- 172.21.15.27:0/1860354454 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fba28015040 con 0x56262ba60710 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.209+0000 7fba3259c640 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56262ba60710 0x56262ba62b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.209+0000 7fba19ffb640 10 client.15744 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.209+0000 7fba3259c640 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56262ba60710 0x56262ba62b30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4edbf9b5dc5fa819 server_cookie=c538a82e8073c3bb in_seq=0 out_seq=0 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.209+0000 7fba19ffb640 1 -- 172.21.15.27:0/1860354454 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fba28015040 con 0x56262ba60710 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.209+0000 7fba19ffb640 10 client.15744 handle_command_reply: tid=0 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.209+0000 7fba3fb89740 1 client.15744 shutdown 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 -- 172.21.15.27:0/1860354454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b5ce3a0 msgr2=0x56262ba5e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b5ce3a0 0x56262ba5e770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fba2c01e970 tx=0x7fba2c004ab0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 -- 172.21.15.27:0/1860354454 shutdown_connections 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56262ba60710 0x56262ba62b30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56262b5ce3a0 0x56262ba5e770 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56262b63a390 0x56262b9ece10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 --2- 172.21.15.27:0/1860354454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56262b6114d0 0x56262b9ec8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 -- 172.21.15.27:0/1860354454 >> 172.21.15.27:0/1860354454 conn(0x56262b62e4e0 msgr2=0x56262b9ea340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba19ffb640 0 client.15744 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 -- 172.21.15.27:0/1860354454 shutdown_connections 2024-04-25T13:43:04.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 1 -- 172.21.15.27:0/1860354454 wait complete. 2024-04-25T13:43:04.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.210+0000 7fba3fb89740 20 client.15744 trim_cache size 0 max 16384 2024-04-25T13:43:04.217 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:43:04.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 -- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba1c0afb70 msgr2=0x7fba1c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba1c0afb70 0x7fba1c0b2080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x56262b9dd860 tx=0x7fba2c009500 comp rx=0 tx=0).stop 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 -- 172.21.15.27:0/3606357284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 msgr2=0x7fba3419a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 0x7fba3419a2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fba24006010 tx=0x7fba24004270 comp rx=0 tx=0).stop 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 -- 172.21.15.27:0/3606357284 shutdown_connections 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fba1c0afb70 0x7fba1c0b2080 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba34104460 0x7fba341a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba34103980 0x7fba3419a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 --2- 172.21.15.27:0/3606357284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba34102fa0 0x7fba3419a2c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.211+0000 7fba38bf8640 1 -- 172.21.15.27:0/3606357284 >> 172.21.15.27:0/3606357284 conn(0x7fba340fd340 msgr2=0x7fba340fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.212+0000 7fba38bf8640 1 -- 172.21.15.27:0/3606357284 shutdown_connections 2024-04-25T13:43:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:04.212+0000 7fba38bf8640 1 -- 172.21.15.27:0/3606357284 wait complete. 2024-04-25T13:43:04.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:04 smithi027 ceph-mon[29140]: pgmap v414: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.2 MiB/s wr, 718 op/s 2024-04-25T13:43:04.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:04 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2483316642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:43:04.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:04 smithi122 ceph-mon[32843]: pgmap v414: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.2 MiB/s wr, 718 op/s 2024-04-25T13:43:04.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:04 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2483316642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:43:04.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:04 smithi045 ceph-mon[32731]: pgmap v414: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.2 MiB/s wr, 718 op/s 2024-04-25T13:43:04.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:04 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2483316642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:43:05.231 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:43:05.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:43:05.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:43:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.402+0000 7fce0789f640 1 -- 172.21.15.27:0/1985389622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 msgr2=0x7fce00104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.402+0000 7fce0789f640 1 --2- 172.21.15.27:0/1985389622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 0x7fce00104030 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009900 tx=0x7fcdf002a410 comp rx=0 tx=0).stop 2024-04-25T13:43:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.402+0000 7fce0789f640 1 -- 172.21.15.27:0/1985389622 shutdown_connections 2024-04-25T13:43:05.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.402+0000 7fce0789f640 1 --2- 172.21.15.27:0/1985389622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce00108920 0x7fce0010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.402+0000 7fce0789f640 1 --2- 172.21.15.27:0/1985389622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 0x7fce00104030 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.402+0000 7fce0789f640 1 --2- 172.21.15.27:0/1985389622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce001029e0 0x7fce001036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.402+0000 7fce0789f640 1 -- 172.21.15.27:0/1985389622 >> 172.21.15.27:0/1985389622 conn(0x7fce000fdbd0 msgr2=0x7fce00100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.403+0000 7fce0789f640 1 -- 172.21.15.27:0/1985389622 shutdown_connections 2024-04-25T13:43:05.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.403+0000 7fce0789f640 1 -- 172.21.15.27:0/1985389622 wait complete. 2024-04-25T13:43:05.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.404+0000 7fce0789f640 1 Processor -- start 2024-04-25T13:43:05.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.404+0000 7fce0789f640 1 -- start start 2024-04-25T13:43:05.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0789f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce001029e0 0x7fce0019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0789f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 0x7fce0019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0789f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce00108920 0x7fce001a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0789f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fce0010e560 con 0x7fce00103c20 2024-04-25T13:43:05.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0789f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce0010e3e0 con 0x7fce00108920 2024-04-25T13:43:05.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0789f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fce0010e6e0 con 0x7fce001029e0 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce04e3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 0x7fce0019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0563b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce001029e0 0x7fce0019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0563b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce001029e0 0x7fce0019c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39504/0 (socket says 172.21.15.27:39504) 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce04e3a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 0x7fce0019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36384/0 (socket says 172.21.15.27:36384) 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0563b640 1 -- 172.21.15.27:0/3752017435 learned_addr learned my addr 172.21.15.27:0/3752017435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce05e3c640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce00108920 0x7fce001a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0563b640 1 -- 172.21.15.27:0/3752017435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce00108920 msgr2=0x7fce001a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0563b640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce00108920 0x7fce001a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0563b640 1 -- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 msgr2=0x7fce0019cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0563b640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 0x7fce0019cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.405+0000 7fce0563b640 1 -- 172.21.15.27:0/3752017435 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce001a4290 con 0x7fce001029e0 2024-04-25T13:43:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.406+0000 7fce0563b640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce001029e0 0x7fce0019c750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00e9f0 tx=0x7fcdfc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.406+0000 7fcdf67fc640 1 -- 172.21.15.27:0/3752017435 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc00ccc0 con 0x7fce001029e0 2024-04-25T13:43:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.406+0000 7fce0789f640 1 -- 172.21.15.27:0/3752017435 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce001a4580 con 0x7fce001029e0 2024-04-25T13:43:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.406+0000 7fcdf67fc640 1 -- 172.21.15.27:0/3752017435 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdfc0035f0 con 0x7fce001029e0 2024-04-25T13:43:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.406+0000 7fce0789f640 1 -- 172.21.15.27:0/3752017435 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce001a4af0 con 0x7fce001029e0 2024-04-25T13:43:05.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.406+0000 7fcdf67fc640 1 -- 172.21.15.27:0/3752017435 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc00ce60 con 0x7fce001029e0 2024-04-25T13:43:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fce0e82d740 1 Processor -- start 2024-04-25T13:43:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fce0e82d740 1 -- start start 2024-04-25T13:43:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 0x5613bf52f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fcdf67fc640 1 -- 172.21.15.27:0/3752017435 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcdfc010690 con 0x7fce001029e0 2024-04-25T13:43:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf52fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf536920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fce0e82d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5613bf494b00 con 0x5613bf165550 2024-04-25T13:43:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fce0e82d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5613bf430920 con 0x5613bf18e410 2024-04-25T13:43:05.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.410+0000 7fce0e82d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5613bf490ac0 con 0x5613bf16d730 2024-04-25T13:43:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce04e3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf536920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce04e3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf536920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39516/0 (socket says 172.21.15.27:39516) 2024-04-25T13:43:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce04e3a640 1 -- 172.21.15.27:0/4154347238 learned_addr learned my addr 172.21.15.27:0/4154347238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce0563b640 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf52fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce05e3c640 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 0x5613bf52f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce04e3a640 1 -- 172.21.15.27:0/4154347238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 msgr2=0x5613bf52f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce04e3a640 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 0x5613bf52f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce04e3a640 1 -- 172.21.15.27:0/4154347238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 msgr2=0x5613bf52fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce04e3a640 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf52fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.411+0000 7fce04e3a640 1 -- 172.21.15.27:0/4154347238 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613bf26e7e0 con 0x5613bf16d730 2024-04-25T13:43:05.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.412+0000 7fcdf67fc640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcdec0afbb0 0x7fcdec0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.412+0000 7fce05e3c640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcdec0afbb0 0x7fcdec0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.412+0000 7fcdf67fc640 1 -- 172.21.15.27:0/3752017435 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcdfc0def40 con 0x7fce001029e0 2024-04-25T13:43:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.412+0000 7fce04e3a640 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf536920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009230 tx=0x7fcdf0005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe027cc632957b91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.413+0000 7fcdeb7fe640 1 -- 172.21.15.27:0/4154347238 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0040070 con 0x5613bf16d730 2024-04-25T13:43:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.413+0000 7fcdeb7fe640 1 -- 172.21.15.27:0/4154347238 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdf0003aa0 con 0x5613bf16d730 2024-04-25T13:43:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.413+0000 7fcdeb7fe640 1 -- 172.21.15.27:0/4154347238 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0035040 con 0x5613bf16d730 2024-04-25T13:43:05.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.413+0000 7fce05e3c640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcdec0afbb0 0x7fcdec0b20c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5613bf530670 tx=0x7fcdf8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 -- 172.21.15.27:0/4154347238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 msgr2=0x5613bf536920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf536920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009230 tx=0x7fcdf0005fd0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 -- 172.21.15.27:0/4154347238 shutdown_connections 2024-04-25T13:43:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf536920 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf52fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 --2- 172.21.15.27:0/4154347238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 0x5613bf52f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 -- 172.21.15.27:0/4154347238 >> 172.21.15.27:0/4154347238 conn(0x5613bf182560 msgr2=0x5613bf52ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 -- 172.21.15.27:0/4154347238 shutdown_connections 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 -- 172.21.15.27:0/4154347238 wait complete. 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 -- start start 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.414+0000 7fce0e82d740 1 Processor -- start 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf541ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf541ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 0x5613bf548f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce0e82d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5613bf53fb20 con 0x5613bf165550 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce0e82d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5613bf42e5d0 con 0x5613bf18e410 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce0e82d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5613bf53fca0 con 0x5613bf16d730 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce0e82d740 10 client.0 _ll_register_callbacks cb 0x5613bf41c080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce04e3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf541ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce0563b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf541ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce04e3a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf541ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39526/0 (socket says 172.21.15.27:39526) 2024-04-25T13:43:05.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce04e3a640 1 -- 172.21.15.27:0/1378327507 learned_addr learned my addr 172.21.15.27:0/1378327507 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.415+0000 7fce05e3c640 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 0x5613bf548f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fce04e3a640 1 -- 172.21.15.27:0/1378327507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 msgr2=0x5613bf548f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fce04e3a640 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 0x5613bf548f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fce04e3a640 1 -- 172.21.15.27:0/1378327507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 msgr2=0x5613bf541ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fce04e3a640 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf541ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fce04e3a640 1 -- 172.21.15.27:0/1378327507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613bf2cef70 con 0x5613bf16d730 2024-04-25T13:43:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fce0563b640 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf541ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:05.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fce04e3a640 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf541ff0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcdf002a450 tx=0x7fcdf0003e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=849d5cefdc0dbf4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fcde8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fcde8ff9640 1 -- 172.21.15.27:0/1378327507 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0035040 con 0x5613bf16d730 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fcde8ff9640 1 -- 172.21.15.27:0/1378327507 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcdf00318b0 con 0x5613bf16d730 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.416+0000 7fce0e82d740 1 -- 172.21.15.27:0/1378327507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdf0003aa0 con 0x5613bf16d730 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.417+0000 7fcde8ff9640 1 -- 172.21.15.27:0/1378327507 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcdf0040040 con 0x5613bf16d730 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.417+0000 7fce0e82d740 10 client.25495 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.417+0000 7fce0e82d740 1 -- 172.21.15.27:0/1378327507 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5613bf166170 con 0x5613bf16d730 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.418+0000 7fcde8ff9640 1 -- 172.21.15.27:0/1378327507 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdf0003aa0 con 0x5613bf16d730 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.418+0000 7fcde8ff9640 1 -- 172.21.15.27:0/1378327507 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcdf0045ac0 con 0x5613bf16d730 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.418+0000 7fce0e82d740 10 client.25495 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.418+0000 7fce0e82d740 10 client.25495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.418+0000 7fce0e82d740 20 client.25495 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.418+0000 7fce0e82d740 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5613bf14d1f0 0x5613bf54f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.418+0000 7fce0e82d740 4 client.25495 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:05.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.418+0000 7fce0e82d740 1 -- 172.21.15.27:0/1378327507 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613bf4a2710 con 0x5613bf14d1f0 2024-04-25T13:43:05.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.419+0000 7fce0563b640 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5613bf14d1f0 0x5613bf54f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.419+0000 7fcde8ff9640 10 client.25495 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:05.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.419+0000 7fce0563b640 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5613bf14d1f0 0x5613bf54f010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e1893cd46be4905 server_cookie=dfdf2d1b6a9b9293 in_seq=0 out_seq=0 2024-04-25T13:43:05.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.420+0000 7fcde8ff9640 1 -- 172.21.15.27:0/1378327507 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5613bf4a2710 con 0x5613bf14d1f0 2024-04-25T13:43:05.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.420+0000 7fcde8ff9640 10 client.25495 handle_command_reply: tid=0 2024-04-25T13:43:05.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.420+0000 7fce0e82d740 1 client.25495 shutdown 2024-04-25T13:43:05.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.420+0000 7fce0e82d740 1 -- 172.21.15.27:0/1378327507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 msgr2=0x5613bf541ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.420+0000 7fce0e82d740 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf541ff0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcdf002a450 tx=0x7fcdf0003e80 comp rx=0 tx=0).stop 2024-04-25T13:43:05.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 1 -- 172.21.15.27:0/1378327507 shutdown_connections 2024-04-25T13:43:05.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5613bf14d1f0 0x5613bf54f010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf18e410 0x5613bf548f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf541ff0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 1 --2- 172.21.15.27:0/1378327507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf165550 0x5613bf541ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 1 -- 172.21.15.27:0/1378327507 >> 172.21.15.27:0/1378327507 conn(0x5613bf182560 msgr2=0x5613bf529ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fcde8ff9640 0 client.25495 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:05.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 1 -- 172.21.15.27:0/1378327507 shutdown_connections 2024-04-25T13:43:05.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 1 -- 172.21.15.27:0/1378327507 wait complete. 2024-04-25T13:43:05.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.421+0000 7fce0e82d740 20 client.25495 trim_cache size 0 max 16384 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.451+0000 7fce0e82d740 1 Processor -- start 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.451+0000 7fce0e82d740 1 -- start start 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.451+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf165550 0x5613bf540780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.451+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf5535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.451+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf553ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.451+0000 7fce0e82d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fce001a6fe0 con 0x5613bf18e410 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.451+0000 7fce0e82d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce001a9840 con 0x5613bf165550 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.451+0000 7fce0e82d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fce001a7630 con 0x5613bf16d730 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce05e3c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf553ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce05e3c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf553ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36420/0 (socket says 172.21.15.27:36420) 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce05e3c640 1 -- 172.21.15.27:0/3137384907 learned_addr learned my addr 172.21.15.27:0/3137384907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce04e3a640 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf5535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce0563b640 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf165550 0x5613bf540780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce05e3c640 1 -- 172.21.15.27:0/3137384907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 msgr2=0x5613bf5535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce05e3c640 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf5535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce05e3c640 1 -- 172.21.15.27:0/3137384907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf165550 msgr2=0x5613bf540780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce05e3c640 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf165550 0x5613bf540780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.452+0000 7fce05e3c640 1 -- 172.21.15.27:0/3137384907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613bf53fe10 con 0x5613bf18e410 2024-04-25T13:43:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.453+0000 7fce05e3c640 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf553ae0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8003eb0 tx=0x7fcdf801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684296f1b67ee97f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.453+0000 7fcdeaffd640 1 -- 172.21.15.27:0/3137384907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf801f4f0 con 0x5613bf18e410 2024-04-25T13:43:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.453+0000 7fcdeaffd640 1 -- 172.21.15.27:0/3137384907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdf801fb40 con 0x5613bf18e410 2024-04-25T13:43:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.453+0000 7fcdeaffd640 1 -- 172.21.15.27:0/3137384907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf802ac00 con 0x5613bf18e410 2024-04-25T13:43:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 -- 172.21.15.27:0/3137384907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 msgr2=0x5613bf553ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf553ae0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8003eb0 tx=0x7fcdf801b000 comp rx=0 tx=0).stop 2024-04-25T13:43:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 -- 172.21.15.27:0/3137384907 shutdown_connections 2024-04-25T13:43:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf553ae0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf16d730 0x5613bf5535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 --2- 172.21.15.27:0/3137384907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf165550 0x5613bf540780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 -- 172.21.15.27:0/3137384907 >> 172.21.15.27:0/3137384907 conn(0x5613bf182560 msgr2=0x5613bf540070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 -- 172.21.15.27:0/3137384907 shutdown_connections 2024-04-25T13:43:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 -- 172.21.15.27:0/3137384907 wait complete. 2024-04-25T13:43:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 -- start start 2024-04-25T13:43:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.455+0000 7fce0e82d740 1 Processor -- start 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf165550 0x5613bf546220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf16d730 0x5613bf546760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0e82d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf547690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0e82d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fce001a7d00 con 0x5613bf18e410 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0e82d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce001a9170 con 0x5613bf16d730 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0e82d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fce001a83d0 con 0x5613bf165550 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0e82d740 10 client.0 _ll_register_callbacks cb 0x7fcdf0004360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0563b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf165550 0x5613bf546220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce05e3c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf547690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0563b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf165550 0x5613bf546220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39548/0 (socket says 172.21.15.27:39548) 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0563b640 1 -- 172.21.15.27:0/2655907693 learned_addr learned my addr 172.21.15.27:0/2655907693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce04e3a640 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf16d730 0x5613bf546760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0563b640 1 -- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf16d730 msgr2=0x5613bf546760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0563b640 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf16d730 0x5613bf546760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0563b640 1 -- 172.21.15.27:0/2655907693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 msgr2=0x5613bf547690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0563b640 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf547690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce0563b640 1 -- 172.21.15.27:0/2655907693 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613bf534180 con 0x5613bf165550 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce04e3a640 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf16d730 0x5613bf546760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.456+0000 7fce05e3c640 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf547690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0563b640 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf165550 0x5613bf546220 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00e9c0 tx=0x7fcdfc003e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5c5fc297b477bda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fcde8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fcde8ff9640 1 -- 172.21.15.27:0/2655907693 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0b0250 con 0x5613bf165550 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 1 -- 172.21.15.27:0/2655907693 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdf801fb40 con 0x5613bf165550 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fcde8ff9640 1 -- 172.21.15.27:0/2655907693 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcdfc003aa0 con 0x5613bf165550 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fcde8ff9640 1 -- 172.21.15.27:0/2655907693 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcdfc003ca0 con 0x5613bf165550 2024-04-25T13:43:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 10 client.25501 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 1 -- 172.21.15.27:0/2655907693 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5613bf5b74a0 con 0x5613bf165550 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fcde8ff9640 1 -- 172.21.15.27:0/2655907693 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdf801fb40 con 0x5613bf165550 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fcde8ff9640 1 -- 172.21.15.27:0/2655907693 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcdfc0c1ce0 con 0x5613bf165550 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 10 client.25501 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 10 client.25501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 20 client.25501 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5613bf122630 0x5613bf5b9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 4 client.25501 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.457+0000 7fce0e82d740 1 -- 172.21.15.27:0/2655907693 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5613bf49c580 con 0x5613bf122630 2024-04-25T13:43:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.458+0000 7fce04e3a640 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5613bf122630 0x5613bf5b9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.458+0000 7fcde8ff9640 10 client.25501 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.459+0000 7fce04e3a640 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5613bf122630 0x5613bf5b9e40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5568fb92e9164fb3 server_cookie=d1e337c171398a37 in_seq=0 out_seq=0 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fcde8ff9640 1 -- 172.21.15.27:0/2655907693 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5613bf49c580 con 0x5613bf122630 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fcde8ff9640 10 client.25501 handle_command_reply: tid=0 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 client.25501 shutdown 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 -- 172.21.15.27:0/2655907693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf165550 msgr2=0x5613bf546220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf165550 0x5613bf546220 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00e9c0 tx=0x7fcdfc003e70 comp rx=0 tx=0).stop 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 -- 172.21.15.27:0/2655907693 shutdown_connections 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5613bf122630 0x5613bf5b9e40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5613bf18e410 0x5613bf547690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5613bf16d730 0x5613bf546760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 --2- 172.21.15.27:0/2655907693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5613bf165550 0x5613bf546220 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00e9c0 tx=0x7fcdfc003e70 comp rx=0 tx=0).stop 2024-04-25T13:43:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 -- 172.21.15.27:0/2655907693 >> 172.21.15.27:0/2655907693 conn(0x5613bf182560 msgr2=0x5613bf52dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fcde8ff9640 0 client.25501 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.460+0000 7fce0e82d740 1 -- 172.21.15.27:0/2655907693 shutdown_connections 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0e82d740 1 -- 172.21.15.27:0/2655907693 wait complete. 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0e82d740 20 client.25501 trim_cache size 0 max 16384 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "4754a0aa-3aff-4237-bb0b-57e8ba4f5eff", 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:43:05.464 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:43:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 -- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcdec0afbb0 msgr2=0x7fcdec0b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcdec0afbb0 0x7fcdec0b20c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5613bf530670 tx=0x7fcdf8009500 comp rx=0 tx=0).stop 2024-04-25T13:43:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 -- 172.21.15.27:0/3752017435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce001029e0 msgr2=0x7fce0019c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce001029e0 0x7fce0019c750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00e9f0 tx=0x7fcdfc00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 -- 172.21.15.27:0/3752017435 shutdown_connections 2024-04-25T13:43:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcdec0afbb0 0x7fcdec0b20c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce00108920 0x7fce001a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce00103c20 0x7fce0019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 --2- 172.21.15.27:0/3752017435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce001029e0 0x7fce0019c750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 -- 172.21.15.27:0/3752017435 >> 172.21.15.27:0/3752017435 conn(0x7fce000fdbd0 msgr2=0x7fce000ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 -- 172.21.15.27:0/3752017435 shutdown_connections 2024-04-25T13:43:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.461+0000 7fce0789f640 1 -- 172.21.15.27:0/3752017435 wait complete. 2024-04-25T13:43:05.478 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:43:05.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:43:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 -- 172.21.15.27:0/2375790655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524109a20 msgr2=0x7f952410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 --2- 172.21.15.27:0/2375790655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524109a20 0x7f952410be40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f951c009be0 tx=0x7f951c02a6c0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 -- 172.21.15.27:0/2375790655 shutdown_connections 2024-04-25T13:43:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 --2- 172.21.15.27:0/2375790655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524109a20 0x7f952410be40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 --2- 172.21.15.27:0/2375790655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9524104d20 0x7f9524105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 --2- 172.21.15.27:0/2375790655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9524103ae0 0x7f95241047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 -- 172.21.15.27:0/2375790655 >> 172.21.15.27:0/2375790655 conn(0x7f95240fecd0 msgr2=0x7f9524101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 -- 172.21.15.27:0/2375790655 shutdown_connections 2024-04-25T13:43:05.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.656+0000 7f952b06e640 1 -- 172.21.15.27:0/2375790655 wait complete. 2024-04-25T13:43:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952b06e640 1 Processor -- start 2024-04-25T13:43:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952b06e640 1 -- start start 2024-04-25T13:43:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952b06e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9524103ae0 0x7f9524198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952b06e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524104d20 0x7f9524198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952b06e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9524109a20 0x7f9524195050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952b06e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f952410f5f0 con 0x7f9524103ae0 2024-04-25T13:43:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952b06e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f952410f470 con 0x7f9524104d20 2024-04-25T13:43:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952b06e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f952410f770 con 0x7f9524109a20 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f9528e0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9524103ae0 0x7f9524198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f9528e0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9524103ae0 0x7f9524198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36440/0 (socket says 172.21.15.27:36440) 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f9528e0a640 1 -- 172.21.15.27:0/3781180803 learned_addr learned my addr 172.21.15.27:0/3781180803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f951bfff640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524104d20 0x7f9524198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.657+0000 7f952960b640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9524109a20 0x7f9524195050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f951bfff640 1 -- 172.21.15.27:0/3781180803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9524109a20 msgr2=0x7f9524195050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f951bfff640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9524109a20 0x7f9524195050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f951bfff640 1 -- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9524103ae0 msgr2=0x7f9524198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f951bfff640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9524103ae0 0x7f9524198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f951bfff640 1 -- 172.21.15.27:0/3781180803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9524195870 con 0x7f9524104d20 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f9528e0a640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9524103ae0 0x7f9524198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f951bfff640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524104d20 0x7f9524198a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f951400e9f0 tx=0x7f951400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f9519ffb640 1 -- 172.21.15.27:0/3781180803 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951400ccc0 con 0x7f9524104d20 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f9519ffb640 1 -- 172.21.15.27:0/3781180803 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95140108e0 con 0x7f9524104d20 2024-04-25T13:43:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f9519ffb640 1 -- 172.21.15.27:0/3781180803 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514018840 con 0x7f9524104d20 2024-04-25T13:43:05.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.658+0000 7f952b06e640 1 -- 172.21.15.27:0/3781180803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9524195b60 con 0x7f9524104d20 2024-04-25T13:43:05.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.659+0000 7f952b06e640 1 -- 172.21.15.27:0/3781180803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95241a8b80 con 0x7f9524104d20 2024-04-25T13:43:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.663+0000 7f9519ffb640 1 -- 172.21.15.27:0/3781180803 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f95140189e0 con 0x7f9524104d20 2024-04-25T13:43:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.663+0000 7f9532001740 1 Processor -- start 2024-04-25T13:43:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9532001740 1 -- start start 2024-04-25T13:43:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9532001740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ef390 0x55f563c90a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9532001740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 0x55f563c90fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9532001740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563c91c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9532001740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f563bf5d80 con 0x55f5638ef390 2024-04-25T13:43:05.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9532001740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f563baf8b0 con 0x55f5638c64d0 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9532001740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f563bf1d40 con 0x55f5638ce6b0 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f952960b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ef390 0x55f563c90a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f952960b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ef390 0x55f563c90a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36450/0 (socket says 172.21.15.27:36450) 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f952960b640 1 -- 172.21.15.27:0/1527394420 learned_addr learned my addr 172.21.15.27:0/1527394420 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9528e0a640 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 0x55f563c90fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f951bfff640 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563c91c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f9519ffb640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f950c0afbf0 0x7f950c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f952960b640 1 -- 172.21.15.27:0/1527394420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 msgr2=0x55f563c91c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f952960b640 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563c91c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f952960b640 1 -- 172.21.15.27:0/1527394420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 msgr2=0x55f563c90fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f952960b640 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 0x55f563c90fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.664+0000 7f952960b640 1 -- 172.21.15.27:0/1527394420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f563b80a40 con 0x55f5638ef390 2024-04-25T13:43:05.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.665+0000 7f9528e0a640 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 0x55f563c90fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.665+0000 7f951bfff640 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563c91c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.665+0000 7f9519ffb640 1 -- 172.21.15.27:0/3781180803 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f95140dfdb0 con 0x7f9524104d20 2024-04-25T13:43:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.665+0000 7f952960b640 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ef390 0x55f563c90a60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f951c009be0 tx=0x7f951c0093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e0e5e928ad43b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.665+0000 7f952960b640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f950c0afbf0 0x7f950c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.665+0000 7f950affd640 1 -- 172.21.15.27:0/1527394420 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c041030 con 0x55f5638ef390 2024-04-25T13:43:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.665+0000 7f950affd640 1 -- 172.21.15.27:0/1527394420 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f951c030660 con 0x55f5638ef390 2024-04-25T13:43:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.665+0000 7f950affd640 1 -- 172.21.15.27:0/1527394420 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c036040 con 0x55f5638ef390 2024-04-25T13:43:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.666+0000 7f9532001740 1 -- 172.21.15.27:0/1527394420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ef390 msgr2=0x55f563c90a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.666+0000 7f9532001740 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ef390 0x55f563c90a60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f951c009be0 tx=0x7f951c0093a0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.666+0000 7f952960b640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f950c0afbf0 0x7f950c0b2100 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f951c005cc0 tx=0x7f951c004630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.666+0000 7f9532001740 1 -- 172.21.15.27:0/1527394420 shutdown_connections 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.666+0000 7f9532001740 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563c91c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.666+0000 7f9532001740 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 0x55f563c90fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.666+0000 7f9532001740 1 --2- 172.21.15.27:0/1527394420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ef390 0x55f563c90a60 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.667+0000 7f9532001740 1 -- 172.21.15.27:0/1527394420 >> 172.21.15.27:0/1527394420 conn(0x55f5638e34e0 msgr2=0x55f563c8c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.667+0000 7f9532001740 1 -- 172.21.15.27:0/1527394420 shutdown_connections 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.667+0000 7f9532001740 1 -- 172.21.15.27:0/1527394420 wait complete. 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.667+0000 7f9532001740 1 -- start start 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.667+0000 7f9532001740 1 Processor -- start 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9532001740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638c64d0 0x55f563c9a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9532001740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ce6b0 0x55f563c9aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9532001740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ef390 0x55f563ca1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9532001740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f563c92720 con 0x55f5638ce6b0 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9528e0a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638c64d0 0x55f563c9a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9532001740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f563b7cd10 con 0x55f5638ef390 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9532001740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f563c928a0 con 0x55f5638c64d0 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9528e0a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638c64d0 0x55f563c9a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39574/0 (socket says 172.21.15.27:39574) 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9528e0a640 1 -- 172.21.15.27:0/2335153501 learned_addr learned my addr 172.21.15.27:0/2335153501 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9532001740 10 client.0 _ll_register_callbacks cb 0x55f563ba9210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f951bfff640 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ce6b0 0x55f563c9aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f952960b640 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ef390 0x55f563ca1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9528e0a640 1 -- 172.21.15.27:0/2335153501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ef390 msgr2=0x55f563ca1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.668+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ef390 0x55f563ca1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.669+0000 7f9528e0a640 1 -- 172.21.15.27:0/2335153501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ce6b0 msgr2=0x55f563c9aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.669+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ce6b0 0x55f563c9aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.669+0000 7f9528e0a640 1 -- 172.21.15.27:0/2335153501 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5639d17d0 con 0x55f5638c64d0 2024-04-25T13:43:05.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.669+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638c64d0 0x55f563c9a960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9520012d20 tx=0x7f952000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c22727f528c74376 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.669+0000 7f9503fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:05.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.669+0000 7f9503fff640 1 -- 172.21.15.27:0/2335153501 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952000fe20 con 0x55f5638c64d0 2024-04-25T13:43:05.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.669+0000 7f9503fff640 1 -- 172.21.15.27:0/2335153501 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f95200047a0 con 0x55f5638c64d0 2024-04-25T13:43:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.670+0000 7f9532001740 1 -- 172.21.15.27:0/2335153501 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f951c030660 con 0x55f5638c64d0 2024-04-25T13:43:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.671+0000 7f9503fff640 1 -- 172.21.15.27:0/2335153501 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f95200049a0 con 0x55f5638c64d0 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.671+0000 7f9532001740 10 client.25513 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.671+0000 7f9532001740 1 -- 172.21.15.27:0/2335153501 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f5639073f0 con 0x55f5638c64d0 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.671+0000 7f9503fff640 1 -- 172.21.15.27:0/2335153501 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9520005020 con 0x55f5638c64d0 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.671+0000 7f9503fff640 1 -- 172.21.15.27:0/2335153501 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f952001c590 con 0x55f5638c64d0 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.671+0000 7f9532001740 10 client.25513 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.672+0000 7f9532001740 10 client.25513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.672+0000 7f9532001740 20 client.25513 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.672+0000 7f9532001740 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5638adf60 0x55f563ca83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.672+0000 7f9532001740 4 client.25513 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.672+0000 7f9532001740 1 -- 172.21.15.27:0/2335153501 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f951c030660 con 0x55f5638adf60 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.672+0000 7f951bfff640 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5638adf60 0x55f563ca83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.673+0000 7f9503fff640 10 client.25513 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:05.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.673+0000 7f951bfff640 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5638adf60 0x55f563ca83e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af6c1576043c573d server_cookie=9911f3cdeef54c1a in_seq=0 out_seq=0 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9503fff640 1 -- 172.21.15.27:0/2335153501 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f951c030660 con 0x55f5638adf60 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9503fff640 10 client.25513 handle_command_reply: tid=0 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 client.25513 shutdown 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 -- 172.21.15.27:0/2335153501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638c64d0 msgr2=0x55f563c9a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638c64d0 0x55f563c9a960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9520012d20 tx=0x7f952000f790 comp rx=0 tx=0).stop 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 -- 172.21.15.27:0/2335153501 shutdown_connections 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5638adf60 0x55f563ca83e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ef390 0x55f563ca1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9503fff640 0 client.25513 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:05.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638ce6b0 0x55f563c9aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 --2- 172.21.15.27:0/2335153501 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638c64d0 0x55f563c9a960 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 -- 172.21.15.27:0/2335153501 >> 172.21.15.27:0/2335153501 conn(0x55f5638e34e0 msgr2=0x55f563c8b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 -- 172.21.15.27:0/2335153501 shutdown_connections 2024-04-25T13:43:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 1 -- 172.21.15.27:0/2335153501 wait complete. 2024-04-25T13:43:05.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.674+0000 7f9532001740 20 client.25513 trim_cache size 0 max 16384 2024-04-25T13:43:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.704+0000 7f9532001740 1 Processor -- start 2024-04-25T13:43:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.704+0000 7f9532001740 1 -- start start 2024-04-25T13:43:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.705+0000 7f9532001740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638adf60 0x55f563c8c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.705+0000 7f9532001740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 0x55f563c991a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.705+0000 7f9532001740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563ca1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.705+0000 7f9532001740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f951000c460 con 0x55f5638adf60 2024-04-25T13:43:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.705+0000 7f9532001740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f951000ece0 con 0x55f5638c64d0 2024-04-25T13:43:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.705+0000 7f9532001740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f951000cad0 con 0x55f5638ce6b0 2024-04-25T13:43:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.706+0000 7f9528e0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638adf60 0x55f563c8c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.706+0000 7f9528e0a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638adf60 0x55f563c8c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36472/0 (socket says 172.21.15.27:36472) 2024-04-25T13:43:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.706+0000 7f9528e0a640 1 -- 172.21.15.27:0/2547573389 learned_addr learned my addr 172.21.15.27:0/2547573389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.706+0000 7f952960b640 1 --2- 172.21.15.27:0/2547573389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563ca1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.707+0000 7f9528e0a640 1 -- 172.21.15.27:0/2547573389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 msgr2=0x55f563ca1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.707+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2547573389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563ca1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.707+0000 7f9528e0a640 1 -- 172.21.15.27:0/2547573389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 msgr2=0x55f563c991a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:43:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.707+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2547573389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 0x55f563c991a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.707+0000 7f9528e0a640 1 -- 172.21.15.27:0/2547573389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5638c70f0 con 0x55f5638adf60 2024-04-25T13:43:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.707+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2547573389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638adf60 0x55f563c8c2a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9520009860 tx=0x7f9520012360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390115bd8870ff0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.714+0000 7f950a7fc640 1 -- 172.21.15.27:0/2547573389 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520004720 con 0x55f5638adf60 2024-04-25T13:43:05.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.714+0000 7f950a7fc640 1 -- 172.21.15.27:0/2547573389 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f952001cb50 con 0x55f5638adf60 2024-04-25T13:43:05.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.714+0000 7f950a7fc640 1 -- 172.21.15.27:0/2547573389 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520005280 con 0x55f5638adf60 2024-04-25T13:43:05.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.717+0000 7f9532001740 1 -- 172.21.15.27:0/2547573389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638adf60 msgr2=0x55f563c8c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.717+0000 7f9532001740 1 --2- 172.21.15.27:0/2547573389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638adf60 0x55f563c8c2a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9520009860 tx=0x7f9520012360 comp rx=0 tx=0).stop 2024-04-25T13:43:05.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.719+0000 7f9532001740 1 -- 172.21.15.27:0/2547573389 shutdown_connections 2024-04-25T13:43:05.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.719+0000 7f9532001740 1 --2- 172.21.15.27:0/2547573389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638ce6b0 0x55f563ca1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.719+0000 7f9532001740 1 --2- 172.21.15.27:0/2547573389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638c64d0 0x55f563c991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.719+0000 7f9532001740 1 --2- 172.21.15.27:0/2547573389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638adf60 0x55f563c8c2a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.719+0000 7f9532001740 1 -- 172.21.15.27:0/2547573389 >> 172.21.15.27:0/2547573389 conn(0x55f5638e34e0 msgr2=0x55f563c8b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.721+0000 7f9532001740 1 -- 172.21.15.27:0/2547573389 shutdown_connections 2024-04-25T13:43:05.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.726+0000 7f9532001740 1 -- 172.21.15.27:0/2547573389 wait complete. 2024-04-25T13:43:05.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.726+0000 7f9532001740 1 -- start start 2024-04-25T13:43:05.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.727+0000 7f9532001740 1 Processor -- start 2024-04-25T13:43:05.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.729+0000 7f9532001740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638adf60 0x55f563c9f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.729+0000 7f9532001740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638c64d0 0x55f563c9fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.729+0000 7f9532001740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ce6b0 0x55f563ca0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.729+0000 7f9532001740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f951000d1a0 con 0x55f5638c64d0 2024-04-25T13:43:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.729+0000 7f9532001740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f951000e610 con 0x55f5638ce6b0 2024-04-25T13:43:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.729+0000 7f9532001740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f951000d870 con 0x55f5638adf60 2024-04-25T13:43:05.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.729+0000 7f9532001740 10 client.0 _ll_register_callbacks cb 0x7f9520009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:05.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.731+0000 7f951bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638c64d0 0x55f563c9fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.731+0000 7f951bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638c64d0 0x55f563c9fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36484/0 (socket says 172.21.15.27:36484) 2024-04-25T13:43:05.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.731+0000 7f951bfff640 1 -- 172.21.15.27:0/2408618653 learned_addr learned my addr 172.21.15.27:0/2408618653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.732+0000 7f952960b640 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ce6b0 0x55f563ca0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.732+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638adf60 0x55f563c9f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.733+0000 7f9528e0a640 1 -- 172.21.15.27:0/2408618653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ce6b0 msgr2=0x55f563ca0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.733+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ce6b0 0x55f563ca0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.733+0000 7f9528e0a640 1 -- 172.21.15.27:0/2408618653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638c64d0 msgr2=0x55f563c9fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:05.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.733+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638c64d0 0x55f563c9fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.733+0000 7f9528e0a640 1 -- 172.21.15.27:0/2408618653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f563bb64b0 con 0x55f5638adf60 2024-04-25T13:43:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.733+0000 7f9528e0a640 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638adf60 0x55f563c9f760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9520012d20 tx=0x7f952001cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a4f800cc9841496 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:05.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.733+0000 7f9503fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.734+0000 7f9503fff640 1 -- 172.21.15.27:0/2408618653 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952001c470 con 0x55f5638adf60 2024-04-25T13:43:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.734+0000 7f9532001740 1 -- 172.21.15.27:0/2408618653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f952001cb50 con 0x55f5638adf60 2024-04-25T13:43:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.734+0000 7f9503fff640 1 -- 172.21.15.27:0/2408618653 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9520004b40 con 0x55f5638adf60 2024-04-25T13:43:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.735+0000 7f9503fff640 1 -- 172.21.15.27:0/2408618653 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9520004d60 con 0x55f5638adf60 2024-04-25T13:43:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.735+0000 7f9532001740 10 client.25519 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:05.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.735+0000 7f9532001740 1 -- 172.21.15.27:0/2408618653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f563ca2250 con 0x55f5638adf60 2024-04-25T13:43:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.737+0000 7f9503fff640 1 -- 172.21.15.27:0/2408618653 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f952001cb50 con 0x55f5638adf60 2024-04-25T13:43:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.737+0000 7f9503fff640 1 -- 172.21.15.27:0/2408618653 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9520039020 con 0x55f5638adf60 2024-04-25T13:43:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.737+0000 7f9532001740 10 client.25519 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.737+0000 7f9532001740 10 client.25519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.737+0000 7f9532001740 20 client.25519 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.737+0000 7f9532001740 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5638ef390 0x55f563d237b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.737+0000 7f9532001740 4 client.25519 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:43:05.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.737+0000 7f9532001740 1 -- 172.21.15.27:0/2408618653 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f563c03990 con 0x55f5638ef390 2024-04-25T13:43:05.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.740+0000 7f951bfff640 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5638ef390 0x55f563d237b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.742+0000 7f951bfff640 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5638ef390 0x55f563d237b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3371c8aa189647c server_cookie=de740bcaaf97be5a in_seq=0 out_seq=0 2024-04-25T13:43:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.742+0000 7f9503fff640 10 client.25519 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.742+0000 7f9503fff640 1 -- 172.21.15.27:0/2408618653 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f563c03990 con 0x55f5638ef390 2024-04-25T13:43:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.742+0000 7f9503fff640 10 client.25519 handle_command_reply: tid=0 2024-04-25T13:43:05.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.742+0000 7f9532001740 1 client.25519 shutdown 2024-04-25T13:43:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 -- 172.21.15.27:0/2408618653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638adf60 msgr2=0x55f563c9f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638adf60 0x55f563c9f760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9520012d20 tx=0x7f952001cd70 comp rx=0 tx=0).stop 2024-04-25T13:43:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 -- 172.21.15.27:0/2408618653 shutdown_connections 2024-04-25T13:43:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5638ef390 0x55f563d237b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5638ce6b0 0x55f563ca0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5638c64d0 0x55f563c9fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 --2- 172.21.15.27:0/2408618653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5638adf60 0x55f563c9f760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 -- 172.21.15.27:0/2408618653 >> 172.21.15.27:0/2408618653 conn(0x55f5638e34e0 msgr2=0x55f563c8b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9503fff640 0 client.25519 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 -- 172.21.15.27:0/2408618653 shutdown_connections 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.743+0000 7f9532001740 1 -- 172.21.15.27:0/2408618653 wait complete. 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.744+0000 7f9532001740 20 client.25519 trim_cache size 0 max 16384 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:43:05.744 INFO:teuthology.orchestra.run.smithi027.stdout: "4754a0aa-3aff-4237-bb0b-57e8ba4f5eff": { 2024-04-25T13:43:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:43:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "4754a0aa-3aff-4237-bb0b-57e8ba4f5eff", 2024-04-25T13:43:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:43:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:43:05.745 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:43:05.745 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:43:05.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.744+0000 7f952b06e640 1 -- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f950c0afbf0 msgr2=0x7f950c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.744+0000 7f952b06e640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f950c0afbf0 0x7f950c0b2100 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f951c005cc0 tx=0x7f951c004630 comp rx=0 tx=0).stop 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.744+0000 7f952b06e640 1 -- 172.21.15.27:0/3781180803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524104d20 msgr2=0x7f9524198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.744+0000 7f952b06e640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524104d20 0x7f9524198a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f951400e9f0 tx=0x7f951400edf0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.744+0000 7f952b06e640 1 -- 172.21.15.27:0/3781180803 shutdown_connections 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.744+0000 7f952b06e640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f950c0afbf0 0x7f950c0b2100 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.745+0000 7f952b06e640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9524109a20 0x7f9524195050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.745+0000 7f952b06e640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9524104d20 0x7f9524198a40 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f951400e9f0 tx=0x7f951400edf0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.745+0000 7f952b06e640 1 --2- 172.21.15.27:0/3781180803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9524103ae0 0x7f9524198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.745+0000 7f952b06e640 1 -- 172.21.15.27:0/3781180803 >> 172.21.15.27:0/3781180803 conn(0x7f95240fecd0 msgr2=0x7f9524105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.745+0000 7f952b06e640 1 -- 172.21.15.27:0/3781180803 shutdown_connections 2024-04-25T13:43:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:05.745+0000 7f952b06e640 1 -- 172.21.15.27:0/3781180803 wait complete. 2024-04-25T13:43:05.762 INFO:tasks.cephfs.filesystem:scrub status for tag:4754a0aa-3aff-4237-bb0b-57e8ba4f5eff - {'path': '/', 'tag': '4754a0aa-3aff-4237-bb0b-57e8ba4f5eff', 'options': 'recursive,force'} 2024-04-25T13:43:06.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:06 smithi122 ceph-mon[32843]: pgmap v415: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 7.5 MiB/s wr, 639 op/s 2024-04-25T13:43:06.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:06 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:43:06.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:06 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:06.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:06 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:43:06.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:06 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:06.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:06 smithi045 ceph-mon[32731]: pgmap v415: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 7.5 MiB/s wr, 639 op/s 2024-04-25T13:43:06.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:06 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:43:06.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:06 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:06.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:06 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:43:06.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:06 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:06.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:06 smithi027 ceph-mon[29140]: pgmap v415: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 7.5 MiB/s wr, 639 op/s 2024-04-25T13:43:06.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:06 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:43:06.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:06 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:06.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:06 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:43:06.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:06 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.502+0000 7f93e7906640 1 -- 172.21.15.27:0/166904455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 msgr2=0x7f93e00751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.502 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.502+0000 7f93e7906640 1 --2- 172.21.15.27:0/166904455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 0x7f93e00751e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f93d8009ca0 tx=0x7f93d802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.502 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.503+0000 7f93e7906640 1 -- 172.21.15.27:0/166904455 shutdown_connections 2024-04-25T13:43:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.503+0000 7f93e7906640 1 --2- 172.21.15.27:0/166904455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 0x7f93e00751e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.503+0000 7f93e7906640 1 --2- 172.21.15.27:0/166904455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e0103c50 0x7f93e0104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.503+0000 7f93e7906640 1 --2- 172.21.15.27:0/166904455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e0102a10 0x7f93e0103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.503+0000 7f93e7906640 1 -- 172.21.15.27:0/166904455 >> 172.21.15.27:0/166904455 conn(0x7f93e00fdc00 msgr2=0x7f93e01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.503+0000 7f93e7906640 1 -- 172.21.15.27:0/166904455 shutdown_connections 2024-04-25T13:43:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.504+0000 7f93e7906640 1 -- 172.21.15.27:0/166904455 wait complete. 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e7906640 1 Processor -- start 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e7906640 1 -- start start 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e7906640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 0x7f93e019c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e7906640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e0102a10 0x7f93e019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e7906640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e0103c50 0x7f93e01a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e7906640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93e0111850 con 0x7f93e0103c50 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e7906640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01116d0 con 0x7f93e0102a10 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e7906640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01119d0 con 0x7f93e0074dd0 2024-04-25T13:43:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e56a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 0x7f93e019c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e5ea3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e0103c50 0x7f93e01a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e56a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 0x7f93e019c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39612/0 (socket says 172.21.15.27:39612) 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.505+0000 7f93e56a2640 1 -- 172.21.15.27:0/797443021 learned_addr learned my addr 172.21.15.27:0/797443021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e4ea1640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e0102a10 0x7f93e019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e56a2640 1 -- 172.21.15.27:0/797443021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e0102a10 msgr2=0x7f93e019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e56a2640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e0102a10 0x7f93e019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e56a2640 1 -- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e0103c50 msgr2=0x7f93e01a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e56a2640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e0103c50 0x7f93e01a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e56a2640 1 -- 172.21.15.27:0/797443021 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e01a3f40 con 0x7f93e0074dd0 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e56a2640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 0x7f93e019c400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00e9f0 tx=0x7f93dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93d67fc640 1 -- 172.21.15.27:0/797443021 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc00ccc0 con 0x7f93e0074dd0 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93d67fc640 1 -- 172.21.15.27:0/797443021 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93dc0035f0 con 0x7f93e0074dd0 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e7906640 1 -- 172.21.15.27:0/797443021 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e01a4230 con 0x7f93e0074dd0 2024-04-25T13:43:07.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93d67fc640 1 -- 172.21.15.27:0/797443021 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc00ce60 con 0x7f93e0074dd0 2024-04-25T13:43:07.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.506+0000 7f93e7906640 1 -- 172.21.15.27:0/797443021 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e01a4780 con 0x7f93e0074dd0 2024-04-25T13:43:07.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.510+0000 7f93ee7bb740 1 Processor -- start 2024-04-25T13:43:07.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.510+0000 7f93ee7bb740 1 -- start start 2024-04-25T13:43:07.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae846b1b0 0x555ae8814180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 0x555ae8814700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae881b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555ae8771460 con 0x555ae84422f0 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555ae872b590 con 0x555ae844a4d0 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555ae876d420 con 0x555ae846b1b0 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93e5ea3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae846b1b0 0x555ae8814180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93e4ea1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae881b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93e4ea1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae881b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:42716/0 (socket says 172.21.15.27:42716) 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93e4ea1640 1 -- 172.21.15.27:0/777550890 learned_addr learned my addr 172.21.15.27:0/777550890 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.511+0000 7f93d67fc640 1 -- 172.21.15.27:0/797443021 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f93dc003aa0 con 0x7f93e0074dd0 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.512+0000 7f93e4ea1640 1 -- 172.21.15.27:0/777550890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae846b1b0 msgr2=0x555ae8814180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.512+0000 7f93e4ea1640 1 --2- 172.21.15.27:0/777550890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae846b1b0 0x555ae8814180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.512+0000 7f93e4ea1640 1 -- 172.21.15.27:0/777550890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 msgr2=0x555ae8814700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.512+0000 7f93e4ea1640 1 --2- 172.21.15.27:0/777550890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 0x555ae8814700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.512+0000 7f93e4ea1640 1 -- 172.21.15.27:0/777550890 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ae87380d0 con 0x555ae844a4d0 2024-04-25T13:43:07.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.512+0000 7f93e4ea1640 1 --2- 172.21.15.27:0/777550890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae881b600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009c20 tx=0x7f93d00117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5d462d03df91011 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:07.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.513+0000 7f93d67fc640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93cc0afb60 0x7f93cc0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.513+0000 7f93cb7fe640 1 -- 172.21.15.27:0/777550890 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d0012450 con 0x555ae844a4d0 2024-04-25T13:43:07.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.513+0000 7f93cb7fe640 1 -- 172.21.15.27:0/777550890 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93d0012aa0 con 0x555ae844a4d0 2024-04-25T13:43:07.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.513+0000 7f93cb7fe640 1 -- 172.21.15.27:0/777550890 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d000fa70 con 0x555ae844a4d0 2024-04-25T13:43:07.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.513+0000 7f93d67fc640 1 -- 172.21.15.27:0/797443021 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f93dc0defd0 con 0x7f93e0074dd0 2024-04-25T13:43:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.514+0000 7f93ee7bb740 1 -- 172.21.15.27:0/777550890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 msgr2=0x555ae881b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.514+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/777550890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae881b600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009c20 tx=0x7f93d00117b0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93e5ea3640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93cc0afb60 0x7f93cc0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93ee7bb740 1 -- 172.21.15.27:0/777550890 shutdown_connections 2024-04-25T13:43:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/777550890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae881b600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/777550890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 0x555ae8814700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/777550890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae846b1b0 0x555ae8814180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93ee7bb740 1 -- 172.21.15.27:0/777550890 >> 172.21.15.27:0/777550890 conn(0x555ae845f300 msgr2=0x555ae8807300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93e5ea3640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93cc0afb60 0x7f93cc0b2070 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x555ae8815350 tx=0x7f93d8009380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93ee7bb740 1 -- 172.21.15.27:0/777550890 shutdown_connections 2024-04-25T13:43:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.515+0000 7f93ee7bb740 1 -- 172.21.15.27:0/777550890 wait complete. 2024-04-25T13:43:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.516+0000 7f93ee7bb740 1 -- start start 2024-04-25T13:43:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.516+0000 7f93ee7bb740 1 Processor -- start 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae84422f0 0x555ae880d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae880de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae846b1b0 0x555ae8812dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555ae880b9c0 con 0x555ae846b1b0 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555ae86f8b60 con 0x555ae844a4d0 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555ae880bb40 con 0x555ae84422f0 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93ee7bb740 10 client.0 _ll_register_callbacks cb 0x555ae871d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93e56a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae84422f0 0x555ae880d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93e56a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae84422f0 0x555ae880d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39618/0 (socket says 172.21.15.27:39618) 2024-04-25T13:43:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.517+0000 7f93e56a2640 1 -- 172.21.15.27:0/2015661764 learned_addr learned my addr 172.21.15.27:0/2015661764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93e56a2640 1 -- 172.21.15.27:0/2015661764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 msgr2=0x555ae880de80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93e56a2640 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae880de80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93e56a2640 1 -- 172.21.15.27:0/2015661764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae846b1b0 msgr2=0x555ae8812dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93e56a2640 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae846b1b0 0x555ae8812dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93e56a2640 1 -- 172.21.15.27:0/2015661764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ae8483210 con 0x555ae84422f0 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93e56a2640 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae84422f0 0x555ae880d940 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0a3500 tx=0x7f93dc0a6cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a01d1787ab75d518 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93c8ff9640 1 -- 172.21.15.27:0/2015661764 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc0ab830 con 0x555ae84422f0 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93c8ff9640 1 -- 172.21.15.27:0/2015661764 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f93dc0afe50 con 0x555ae84422f0 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.518+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2015661764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93d0012aa0 con 0x555ae84422f0 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.519+0000 7f93c8ff9640 1 -- 172.21.15.27:0/2015661764 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f93dc0b02e0 con 0x555ae84422f0 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.519+0000 7f93ee7bb740 10 client.25531 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.519+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2015661764 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555ae854f620 con 0x555ae84422f0 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93c8ff9640 1 -- 172.21.15.27:0/2015661764 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93d0012aa0 con 0x555ae84422f0 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93c8ff9640 1 -- 172.21.15.27:0/2015661764 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f93dc0a2f50 con 0x555ae84422f0 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93ee7bb740 10 client.25531 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93ee7bb740 10 client.25531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93ee7bb740 20 client.25531 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555ae8429d80 0x555ae8834490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93ee7bb740 4 client.25531 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2015661764 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ae877f070 con 0x555ae8429d80 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.520+0000 7f93e4ea1640 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555ae8429d80 0x555ae8834490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.521+0000 7f93c8ff9640 10 client.25531 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:07.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.521+0000 7f93e4ea1640 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555ae8429d80 0x555ae8834490 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4299b1bde69f6ae server_cookie=e2140274170dcb91 in_seq=0 out_seq=0 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93c8ff9640 1 -- 172.21.15.27:0/2015661764 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555ae877f070 con 0x555ae8429d80 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93c8ff9640 10 client.25531 handle_command_reply: tid=0 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93ee7bb740 1 client.25531 shutdown 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2015661764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae84422f0 msgr2=0x555ae880d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae84422f0 0x555ae880d940 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0a3500 tx=0x7f93dc0a6cb0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2015661764 shutdown_connections 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555ae8429d80 0x555ae8834490 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93c8ff9640 0 client.25531 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae846b1b0 0x555ae8812dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae844a4d0 0x555ae880de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.522+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2015661764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae84422f0 0x555ae880d940 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.523+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2015661764 >> 172.21.15.27:0/2015661764 conn(0x555ae845f300 msgr2=0x555ae8806250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.523+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2015661764 shutdown_connections 2024-04-25T13:43:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.523+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2015661764 wait complete. 2024-04-25T13:43:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.523+0000 7f93ee7bb740 20 client.25531 trim_cache size 0 max 16384 2024-04-25T13:43:07.551 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.552+0000 7f93ee7bb740 1 Processor -- start 2024-04-25T13:43:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.552+0000 7f93ee7bb740 1 -- start start 2024-04-25T13:43:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.552+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae8429d80 0x555ae880bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.552+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae84422f0 0x555ae880c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.552+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 0x555ae880d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.552+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01a6fa0 con 0x555ae8429d80 2024-04-25T13:43:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.552+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01a9820 con 0x555ae84422f0 2024-04-25T13:43:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.552+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01a7610 con 0x555ae844a4d0 2024-04-25T13:43:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.553+0000 7f93e56a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae8429d80 0x555ae880bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.553+0000 7f93e56a2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae8429d80 0x555ae880bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36528/0 (socket says 172.21.15.27:36528) 2024-04-25T13:43:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.553+0000 7f93e56a2640 1 -- 172.21.15.27:0/2073004836 learned_addr learned my addr 172.21.15.27:0/2073004836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:07.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.553+0000 7f93e4ea1640 1 --2- 172.21.15.27:0/2073004836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae84422f0 0x555ae880c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.553+0000 7f93e56a2640 1 -- 172.21.15.27:0/2073004836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 msgr2=0x555ae880d510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:43:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.553+0000 7f93e56a2640 1 --2- 172.21.15.27:0/2073004836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 0x555ae880d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.553+0000 7f93e56a2640 1 -- 172.21.15.27:0/2073004836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae84422f0 msgr2=0x555ae880c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.554+0000 7f93e56a2640 1 --2- 172.21.15.27:0/2073004836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae84422f0 0x555ae880c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.554+0000 7f93e56a2640 1 -- 172.21.15.27:0/2073004836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ae880ea80 con 0x555ae8429d80 2024-04-25T13:43:07.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.554+0000 7f93e56a2640 1 --2- 172.21.15.27:0/2073004836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae8429d80 0x555ae880bf10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00e9c0 tx=0x7f93dc0a6d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a4797c99b88575 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.555+0000 7f93caffd640 1 -- 172.21.15.27:0/2073004836 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc0ab830 con 0x555ae8429d80 2024-04-25T13:43:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.555+0000 7f93caffd640 1 -- 172.21.15.27:0/2073004836 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93dc0042c0 con 0x555ae8429d80 2024-04-25T13:43:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.555+0000 7f93caffd640 1 -- 172.21.15.27:0/2073004836 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc0afe50 con 0x555ae8429d80 2024-04-25T13:43:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.556+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2073004836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae8429d80 msgr2=0x555ae880bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.556+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2073004836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae8429d80 0x555ae880bf10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00e9c0 tx=0x7f93dc0a6d20 comp rx=0 tx=0).stop 2024-04-25T13:43:07.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.556+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2073004836 shutdown_connections 2024-04-25T13:43:07.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.556+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2073004836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 0x555ae880d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.556+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2073004836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae84422f0 0x555ae880c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.556+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/2073004836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae8429d80 0x555ae880bf10 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.556+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2073004836 >> 172.21.15.27:0/2073004836 conn(0x555ae845f300 msgr2=0x555ae8818420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.557+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2073004836 shutdown_connections 2024-04-25T13:43:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.557+0000 7f93ee7bb740 1 -- 172.21.15.27:0/2073004836 wait complete. 2024-04-25T13:43:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.558+0000 7f93ee7bb740 1 -- start start 2024-04-25T13:43:07.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.558+0000 7f93ee7bb740 1 Processor -- start 2024-04-25T13:43:07.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:07 smithi122 ceph-mon[32843]: pgmap v416: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 10 MiB/s wr, 942 op/s 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae8429d80 0x555ae88431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 0x555ae8843730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93ee7bb740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 0x555ae8844660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01a7ce0 con 0x555ae84422f0 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01a9150 con 0x555ae8429d80 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93ee7bb740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01a83b0 con 0x555ae844a4d0 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93ee7bb740 10 client.0 _ll_register_callbacks cb 0x7f93dc0045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93e56a2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae8429d80 0x555ae88431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93e56a2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae8429d80 0x555ae88431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:42732/0 (socket says 172.21.15.27:42732) 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93e56a2640 1 -- 172.21.15.27:0/3615422270 learned_addr learned my addr 172.21.15.27:0/3615422270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.559+0000 7f93e5ea3640 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 0x555ae8844660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93e4ea1640 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 0x555ae8843730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93e5ea3640 1 -- 172.21.15.27:0/3615422270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae8429d80 msgr2=0x555ae88431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93e5ea3640 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae8429d80 0x555ae88431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93e5ea3640 1 -- 172.21.15.27:0/3615422270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 msgr2=0x555ae8843730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93e5ea3640 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 0x555ae8843730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93e5ea3640 1 -- 172.21.15.27:0/3615422270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ae8844c20 con 0x555ae844a4d0 2024-04-25T13:43:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93e5ea3640 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 0x555ae8844660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f93d8031b50 tx=0x7f93d8009410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be74625c9bfcd66f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93c8ff9640 1 -- 172.21.15.27:0/3615422270 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d8004250 con 0x555ae844a4d0 2024-04-25T13:43:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93c8ff9640 1 -- 172.21.15.27:0/3615422270 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f93d80043f0 con 0x555ae844a4d0 2024-04-25T13:43:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.560+0000 7f93ee7bb740 1 -- 172.21.15.27:0/3615422270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93dc0042c0 con 0x555ae844a4d0 2024-04-25T13:43:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.561+0000 7f93c8ff9640 1 -- 172.21.15.27:0/3615422270 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f93d80045f0 con 0x555ae844a4d0 2024-04-25T13:43:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.561+0000 7f93ee7bb740 10 client.25534 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.561+0000 7f93ee7bb740 1 -- 172.21.15.27:0/3615422270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555ae8845f00 con 0x555ae844a4d0 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.561+0000 7f93c8ff9640 1 -- 172.21.15.27:0/3615422270 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93dc0042c0 con 0x555ae844a4d0 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.562+0000 7f93c8ff9640 1 -- 172.21.15.27:0/3615422270 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f93d8051070 con 0x555ae844a4d0 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.562+0000 7f93ee7bb740 10 client.25534 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.562+0000 7f93ee7bb740 10 client.25534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.562+0000 7f93ee7bb740 20 client.25534 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.562+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555ae846b1b0 0x555ae889ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.562+0000 7f93ee7bb740 4 client.25534 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.562+0000 7f93ee7bb740 1 -- 172.21.15.27:0/3615422270 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ae8778ee0 con 0x555ae846b1b0 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.562+0000 7f93e56a2640 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555ae846b1b0 0x555ae889ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.563+0000 7f93c8ff9640 10 client.25534 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:07.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.563+0000 7f93e56a2640 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555ae846b1b0 0x555ae889ebd0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a25ee3c77e23ca6 server_cookie=944da6bb1ac646b8 in_seq=0 out_seq=0 2024-04-25T13:43:07.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:07 smithi045 ceph-mon[32731]: pgmap v416: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 10 MiB/s wr, 942 op/s 2024-04-25T13:43:07.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93c8ff9640 1 -- 172.21.15.27:0/3615422270 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555ae8778ee0 con 0x555ae846b1b0 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93c8ff9640 10 client.25534 handle_command_reply: tid=0 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 client.25534 shutdown 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 -- 172.21.15.27:0/3615422270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 msgr2=0x555ae8844660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 0x555ae8844660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f93d8031b50 tx=0x7f93d8009410 comp rx=0 tx=0).stop 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 -- 172.21.15.27:0/3615422270 shutdown_connections 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555ae846b1b0 0x555ae889ebd0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555ae844a4d0 0x555ae8844660 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555ae84422f0 0x555ae8843730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 --2- 172.21.15.27:0/3615422270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555ae8429d80 0x555ae88431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93ee7bb740 1 -- 172.21.15.27:0/3615422270 >> 172.21.15.27:0/3615422270 conn(0x555ae845f300 msgr2=0x555ae880bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.583+0000 7f93c8ff9640 0 client.25534 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.584+0000 7f93ee7bb740 1 -- 172.21.15.27:0/3615422270 shutdown_connections 2024-04-25T13:43:07.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.584+0000 7f93ee7bb740 1 -- 172.21.15.27:0/3615422270 wait complete. 2024-04-25T13:43:07.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.584+0000 7f93ee7bb740 20 client.25534 trim_cache size 0 max 16384 2024-04-25T13:43:07.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:43:07.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:43:07.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:43:07.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 -- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93cc0afb60 msgr2=0x7f93cc0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93cc0afb60 0x7f93cc0b2070 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x555ae8815350 tx=0x7f93d8009380 comp rx=0 tx=0).stop 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 -- 172.21.15.27:0/797443021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 msgr2=0x7f93e019c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 0x7f93e019c400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00e9f0 tx=0x7f93dc00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 -- 172.21.15.27:0/797443021 shutdown_connections 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93cc0afb60 0x7f93cc0b2070 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93e0103c50 0x7f93e01a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e0102a10 0x7f93e019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.585+0000 7f93e7906640 1 --2- 172.21.15.27:0/797443021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e0074dd0 0x7f93e019c400 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.586+0000 7f93e7906640 1 -- 172.21.15.27:0/797443021 >> 172.21.15.27:0/797443021 conn(0x7f93e00fdc00 msgr2=0x7f93e0108b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.586+0000 7f93e7906640 1 -- 172.21.15.27:0/797443021 shutdown_connections 2024-04-25T13:43:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:07.586+0000 7f93e7906640 1 -- 172.21.15.27:0/797443021 wait complete. 2024-04-25T13:43:07.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:07 smithi027 ceph-mon[29140]: pgmap v416: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 10 MiB/s wr, 942 op/s 2024-04-25T13:43:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 120.0 s, 2973.0 tps, lat 33.533 ms stddev 34.751 2024-04-25T13:43:09.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:09 smithi122 ceph-mon[32843]: pgmap v417: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 7.8 MiB/s wr, 706 op/s 2024-04-25T13:43:09.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:09 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:43:09.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:09 smithi045 ceph-mon[32731]: pgmap v417: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 7.8 MiB/s wr, 706 op/s 2024-04-25T13:43:09.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:09 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:43:09.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:09 smithi027 ceph-mon[29140]: pgmap v417: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 7.8 MiB/s wr, 706 op/s 2024-04-25T13:43:09.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:09 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:43:11.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:11 smithi122 ceph-mon[32843]: pgmap v418: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.6 MiB/s wr, 813 op/s 2024-04-25T13:43:11.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:11 smithi045 ceph-mon[32731]: pgmap v418: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.6 MiB/s wr, 813 op/s 2024-04-25T13:43:11.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:11 smithi027 ceph-mon[29140]: pgmap v418: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.6 MiB/s wr, 813 op/s 2024-04-25T13:43:12.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:12 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:43:12.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:12 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:43:12.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:12 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:43:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 125.0 s, 3466.6 tps, lat 28.758 ms stddev 35.431 2024-04-25T13:43:13.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:13 smithi122 ceph-mon[32843]: pgmap v419: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.7 MiB/s wr, 842 op/s 2024-04-25T13:43:13.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:43:13.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:43:13.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:43:13.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:13 smithi045 ceph-mon[32731]: pgmap v419: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.7 MiB/s wr, 842 op/s 2024-04-25T13:43:13.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:43:13.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:43:13.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:43:13.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:13 smithi027 ceph-mon[29140]: pgmap v419: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.7 MiB/s wr, 842 op/s 2024-04-25T13:43:13.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:43:13.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:43:13.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:43:15.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:15 smithi122 ceph-mon[32843]: pgmap v420: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 9.1 MiB/s wr, 792 op/s 2024-04-25T13:43:15.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:15 smithi045 ceph-mon[32731]: pgmap v420: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 9.1 MiB/s wr, 792 op/s 2024-04-25T13:43:15.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:15 smithi027 ceph-mon[29140]: pgmap v420: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 9.1 MiB/s wr, 792 op/s 2024-04-25T13:43:17.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:17 smithi122 ceph-mon[32843]: pgmap v421: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T13:43:17.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:17 smithi045 ceph-mon[32731]: pgmap v421: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T13:43:17.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:17 smithi027 ceph-mon[29140]: pgmap v421: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T13:43:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 130.0 s, 3797.4 tps, lat 26.501 ms stddev 29.440 2024-04-25T13:43:19.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:19 smithi122 ceph-mon[32843]: pgmap v422: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 9.8 MiB/s wr, 690 op/s 2024-04-25T13:43:19.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:19 smithi045 ceph-mon[32731]: pgmap v422: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 9.8 MiB/s wr, 690 op/s 2024-04-25T13:43:19.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:19 smithi027 ceph-mon[29140]: pgmap v422: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 9.8 MiB/s wr, 690 op/s 2024-04-25T13:43:20.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.728+0000 7fb856701640 1 -- 172.21.15.27:0/2343433947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850107970 msgr2=0x7fb850107d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.728+0000 7fb856701640 1 --2- 172.21.15.27:0/2343433947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850107970 0x7fb850107d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb848009be0 tx=0x7fb84802a6c0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.728+0000 7fb856701640 1 -- 172.21.15.27:0/2343433947 shutdown_connections 2024-04-25T13:43:20.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.728+0000 7fb856701640 1 --2- 172.21.15.27:0/2343433947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850107970 0x7fb850107d80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.728+0000 7fb856701640 1 --2- 172.21.15.27:0/2343433947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850102800 0x7fb850102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.728+0000 7fb856701640 1 --2- 172.21.15.27:0/2343433947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb850101dc0 0x7fb850102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.728+0000 7fb856701640 1 -- 172.21.15.27:0/2343433947 >> 172.21.15.27:0/2343433947 conn(0x7fb8500fd390 msgr2=0x7fb8500ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:20.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.728+0000 7fb856701640 1 -- 172.21.15.27:0/2343433947 shutdown_connections 2024-04-25T13:43:20.730 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.729+0000 7fb856701640 1 -- 172.21.15.27:0/2343433947 wait complete. 2024-04-25T13:43:20.730 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.729+0000 7fb856701640 1 Processor -- start 2024-04-25T13:43:20.730 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.729+0000 7fb856701640 1 -- start start 2024-04-25T13:43:20.730 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb856701640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850101dc0 0x7fb8501a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.730 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb856701640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850102800 0x7fb8501a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb856701640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb850107970 0x7fb8501a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb856701640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb850115c40 con 0x7fb850107970 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb856701640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb850115ac0 con 0x7fb850101dc0 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb856701640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb850115dc0 con 0x7fb850102800 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb84ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850101dc0 0x7fb8501a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb84ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850101dc0 0x7fb8501a0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52214/0 (socket says 172.21.15.27:52214) 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb84ffff640 1 -- 172.21.15.27:0/2693062348 learned_addr learned my addr 172.21.15.27:0/2693062348 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.730+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850102800 0x7fb8501a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.731+0000 7fb854c9e640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb850107970 0x7fb8501a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.731+0000 7fb854c9e640 1 -- 172.21.15.27:0/2693062348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850102800 msgr2=0x7fb8501a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.731+0000 7fb854c9e640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850102800 0x7fb8501a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.731+0000 7fb854c9e640 1 -- 172.21.15.27:0/2693062348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850101dc0 msgr2=0x7fb8501a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.731+0000 7fb854c9e640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850101dc0 0x7fb8501a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.731+0000 7fb854c9e640 1 -- 172.21.15.27:0/2693062348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8501a8270 con 0x7fb850107970 2024-04-25T13:43:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.731+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850102800 0x7fb8501a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:20.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.732+0000 7fb854c9e640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb850107970 0x7fb8501a7c00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb848009b30 tx=0x7fb848030e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:20.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.733+0000 7fb84d7fa640 1 -- 172.21.15.27:0/2693062348 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb848041020 con 0x7fb850107970 2024-04-25T13:43:20.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.733+0000 7fb856701640 1 -- 172.21.15.27:0/2693062348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8501a8500 con 0x7fb850107970 2024-04-25T13:43:20.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.733+0000 7fb84d7fa640 1 -- 172.21.15.27:0/2693062348 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8480318f0 con 0x7fb850107970 2024-04-25T13:43:20.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.733+0000 7fb856701640 1 -- 172.21.15.27:0/2693062348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8501a89c0 con 0x7fb850107970 2024-04-25T13:43:20.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.733+0000 7fb84d7fa640 1 -- 172.21.15.27:0/2693062348 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb848036040 con 0x7fb850107970 2024-04-25T13:43:20.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.736+0000 7fb85d5b0740 1 Processor -- start 2024-04-25T13:43:20.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb84d7fa640 1 -- 172.21.15.27:0/2693062348 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb848044020 con 0x7fb850107970 2024-04-25T13:43:20.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb85d5b0740 1 -- start start 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b481b0 0x556a68ee8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b274d0 0x556a68eefdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b481b0 0x556a68ee8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b481b0 0x556a68ee8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33350/0 (socket says 172.21.15.27:33350) 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb84d7fa640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8380afae0 0x7fb8380b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb84f7fe640 1 -- 172.21.15.27:0/187265059 learned_addr learned my addr 172.21.15.27:0/187265059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.737+0000 7fb85d5b0740 1 -- 172.21.15.27:0/187265059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556a68e4e460 con 0x556a68b274d0 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb85d5b0740 1 -- 172.21.15.27:0/187265059 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556a68e08590 con 0x556a68b1f2f0 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb85d5b0740 1 -- 172.21.15.27:0/187265059 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556a68e4a420 con 0x556a68b481b0 2024-04-25T13:43:20.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb84d7fa640 1 -- 172.21.15.27:0/2693062348 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb8480ff5f0 con 0x7fb850107970 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb84ffff640 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b274d0 0x556a68eefdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb84f7fe640 1 -- 172.21.15.27:0/187265059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 msgr2=0x556a68ee8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb84f7fe640 1 -- 172.21.15.27:0/187265059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b274d0 msgr2=0x556a68eefdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b274d0 0x556a68eefdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb84f7fe640 1 -- 172.21.15.27:0/187265059 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a68e150d0 con 0x556a68b481b0 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.738+0000 7fb854c9e640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8380afae0 0x7fb8380b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.739+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b481b0 0x556a68ee8970 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb844012bb0 tx=0x7fb84400bba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab876d2fe207dd03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.739+0000 7fb8367fc640 1 -- 172.21.15.27:0/187265059 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb844012ca0 con 0x556a68b481b0 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.739+0000 7fb8367fc640 1 -- 172.21.15.27:0/187265059 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb844012e40 con 0x556a68b481b0 2024-04-25T13:43:20.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.739+0000 7fb8367fc640 1 -- 172.21.15.27:0/187265059 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb844003b50 con 0x556a68b481b0 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 -- 172.21.15.27:0/187265059 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b481b0 msgr2=0x556a68ee8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b481b0 0x556a68ee8970 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb844012bb0 tx=0x7fb84400bba0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb854c9e640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8380afae0 0x7fb8380b1ff0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb84802abe0 tx=0x7fb848009280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 -- 172.21.15.27:0/187265059 shutdown_connections 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b274d0 0x556a68eefdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/187265059 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b481b0 0x556a68ee8970 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 -- 172.21.15.27:0/187265059 >> 172.21.15.27:0/187265059 conn(0x556a68b3c300 msgr2=0x556a68ee4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 -- 172.21.15.27:0/187265059 shutdown_connections 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 -- 172.21.15.27:0/187265059 wait complete. 2024-04-25T13:43:20.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.741+0000 7fb85d5b0740 1 -- start start 2024-04-25T13:43:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.742+0000 7fb85d5b0740 1 Processor -- start 2024-04-25T13:43:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.742+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ef26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.742+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.742+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.742+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556a68ef07e0 con 0x556a68b481b0 2024-04-25T13:43:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.742+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556a68dd5b60 con 0x556a68b1f2f0 2024-04-25T13:43:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.742+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556a68ef0960 con 0x556a68b274d0 2024-04-25T13:43:20.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.742+0000 7fb85d5b0740 10 client.0 _ll_register_callbacks cb 0x556a68dfa2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45048/0 (socket says 172.21.15.27:45048) 2024-04-25T13:43:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 -- 172.21.15.27:0/2904153248 learned_addr learned my addr 172.21.15.27:0/2904153248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ef26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 -- 172.21.15.27:0/2904153248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 msgr2=0x556a68ef2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.745 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 -- 172.21.15.27:0/2904153248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 msgr2=0x556a68ef26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ef26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.743+0000 7fb84f7fe640 1 -- 172.21.15.27:0/2904153248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a68b60210 con 0x556a68b481b0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.744+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef9b40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb844010f60 tx=0x7fb84401b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562a401d87070002 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.744+0000 7fb82f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.744+0000 7fb82f7fe640 1 -- 172.21.15.27:0/2904153248 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8440206a0 con 0x556a68b481b0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.744+0000 7fb82f7fe640 1 -- 172.21.15.27:0/2904153248 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb844020840 con 0x556a68b481b0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.744+0000 7fb85d5b0740 1 -- 172.21.15.27:0/2904153248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556a68e5c070 con 0x556a68b481b0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.744+0000 7fb82f7fe640 1 -- 172.21.15.27:0/2904153248 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb844020a40 con 0x556a68b481b0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb85d5b0740 10 client.15804 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb85d5b0740 1 -- 172.21.15.27:0/2904153248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556a68c2c620 con 0x556a68b481b0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb82f7fe640 1 -- 172.21.15.27:0/2904153248 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556a68e5c070 con 0x556a68b481b0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb82f7fe640 1 -- 172.21.15.27:0/2904153248 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb84402d9f0 con 0x556a68b481b0 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb85d5b0740 10 client.15804 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:20.746 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb85d5b0740 10 client.15804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb85d5b0740 20 client.15804 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a68b06d80 0x556a68efff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb85d5b0740 4 client.15804 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.745+0000 7fb85d5b0740 1 -- 172.21.15.27:0/2904153248 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a68e55ee0 con 0x556a68b06d80 2024-04-25T13:43:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.746+0000 7fb84ffff640 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a68b06d80 0x556a68efff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.746+0000 7fb84ffff640 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a68b06d80 0x556a68efff70 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=879894f2c6557c3a server_cookie=7416e1cec42bf229 in_seq=0 out_seq=0 2024-04-25T13:43:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.746+0000 7fb82f7fe640 10 client.15804 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb82f7fe640 1 -- 172.21.15.27:0/2904153248 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556a68e55ee0 con 0x556a68b06d80 2024-04-25T13:43:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb82f7fe640 10 client.15804 handle_command_reply: tid=0 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb85d5b0740 1 client.15804 shutdown 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb85d5b0740 1 -- 172.21.15.27:0/2904153248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 msgr2=0x556a68ef9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef9b40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb844010f60 tx=0x7fb84401b040 comp rx=0 tx=0).stop 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb85d5b0740 1 -- 172.21.15.27:0/2904153248 shutdown_connections 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a68b06d80 0x556a68efff70 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef9b40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.748+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/2904153248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ef26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.749+0000 7fb85d5b0740 1 -- 172.21.15.27:0/2904153248 >> 172.21.15.27:0/2904153248 conn(0x556a68b3c300 msgr2=0x556a68ee3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.749+0000 7fb82f7fe640 0 client.15804 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.749+0000 7fb85d5b0740 1 -- 172.21.15.27:0/2904153248 shutdown_connections 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.749+0000 7fb85d5b0740 1 -- 172.21.15.27:0/2904153248 wait complete. 2024-04-25T13:43:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.749+0000 7fb85d5b0740 20 client.15804 trim_cache size 0 max 16384 2024-04-25T13:43:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.776+0000 7fb85d5b0740 1 Processor -- start 2024-04-25T13:43:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.776+0000 7fb85d5b0740 1 -- start start 2024-04-25T13:43:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.776+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.776+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.776+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.776+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb8501af6e0 con 0x556a68b481b0 2024-04-25T13:43:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.776+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8501b1f60 con 0x556a68b1f2f0 2024-04-25T13:43:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.776+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb8501afd50 con 0x556a68b274d0 2024-04-25T13:43:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45064/0 (socket says 172.21.15.27:45064) 2024-04-25T13:43:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 -- 172.21.15.27:0/1917711240 learned_addr learned my addr 172.21.15.27:0/1917711240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84ffff640 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 -- 172.21.15.27:0/1917711240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 msgr2=0x556a68ef9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 -- 172.21.15.27:0/1917711240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 msgr2=0x556a68ef0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.777+0000 7fb84f7fe640 1 -- 172.21.15.27:0/1917711240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a68eed580 con 0x556a68b1f2f0 2024-04-25T13:43:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.778+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee42b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb844005fd0 tx=0x7fb84400b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc4327d4c8ce50df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.778+0000 7fb835ffb640 1 -- 172.21.15.27:0/1917711240 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb844040070 con 0x556a68b1f2f0 2024-04-25T13:43:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.778+0000 7fb835ffb640 1 -- 172.21.15.27:0/1917711240 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb844010c50 con 0x556a68b1f2f0 2024-04-25T13:43:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.778+0000 7fb835ffb640 1 -- 172.21.15.27:0/1917711240 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84402c9c0 con 0x556a68b1f2f0 2024-04-25T13:43:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.780+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1917711240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 msgr2=0x556a68ee42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.780+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee42b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb844005fd0 tx=0x7fb84400b390 comp rx=0 tx=0).stop 2024-04-25T13:43:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.780+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1917711240 shutdown_connections 2024-04-25T13:43:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.780+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b481b0 0x556a68ef0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.780+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.780+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1917711240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b1f2f0 0x556a68ee42b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.780+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1917711240 >> 172.21.15.27:0/1917711240 conn(0x556a68b3c300 msgr2=0x556a68ee3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.781+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1917711240 shutdown_connections 2024-04-25T13:43:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.781+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1917711240 wait complete. 2024-04-25T13:43:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 1 -- start start 2024-04-25T13:43:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 1 Processor -- start 2024-04-25T13:43:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b1f2f0 0x556a68ef6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b481b0 0x556a68f662d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb8501b0420 con 0x556a68b1f2f0 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8501b1890 con 0x556a68b481b0 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb8501b0af0 con 0x556a68b274d0 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb85d5b0740 10 client.0 _ll_register_callbacks cb 0x7fb844012ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b1f2f0 0x556a68ef6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b1f2f0 0x556a68ef6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45072/0 (socket says 172.21.15.27:45072) 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb84f7fe640 1 -- 172.21.15.27:0/1631432087 learned_addr learned my addr 172.21.15.27:0/1631432087 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.782+0000 7fb84ffff640 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb84ffff640 1 -- 172.21.15.27:0/1631432087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b481b0 msgr2=0x556a68f662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb84ffff640 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b481b0 0x556a68f662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb84ffff640 1 -- 172.21.15.27:0/1631432087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b1f2f0 msgr2=0x556a68ef6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb84ffff640 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b1f2f0 0x556a68ef6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb84ffff640 1 -- 172.21.15.27:0/1631432087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a68ef9fa0 con 0x556a68b274d0 2024-04-25T13:43:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b1f2f0 0x556a68ef6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb84ffff640 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef7040 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x556a68f006a0 tx=0x7fb840012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61827fadf7105cea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb82f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb82f7fe640 1 -- 172.21.15.27:0/1631432087 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84001b490 con 0x556a68b274d0 2024-04-25T13:43:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb82f7fe640 1 -- 172.21.15.27:0/1631432087 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb84001b630 con 0x556a68b274d0 2024-04-25T13:43:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.783+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1631432087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb844010c50 con 0x556a68b274d0 2024-04-25T13:43:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.784+0000 7fb82f7fe640 1 -- 172.21.15.27:0/1631432087 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb84001b830 con 0x556a68b274d0 2024-04-25T13:43:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.784+0000 7fb85d5b0740 10 client.25549 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.784+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1631432087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556a68f67a00 con 0x556a68b274d0 2024-04-25T13:43:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.784+0000 7fb82f7fe640 1 -- 172.21.15.27:0/1631432087 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb844010c50 con 0x556a68b274d0 2024-04-25T13:43:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.784+0000 7fb82f7fe640 1 -- 172.21.15.27:0/1631432087 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb840011640 con 0x556a68b274d0 2024-04-25T13:43:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.786+0000 7fb85d5b0740 10 client.25549 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.786+0000 7fb85d5b0740 10 client.25549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.786+0000 7fb85d5b0740 20 client.25549 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.786+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a68adc1c0 0x556a68f6a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.786+0000 7fb85d5b0740 4 client.25549 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:43:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.786+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1631432087 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556a68dd15f0 con 0x556a68adc1c0 2024-04-25T13:43:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.786+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a68adc1c0 0x556a68f6a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.787+0000 7fb82f7fe640 10 client.25549 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.787+0000 7fb84f7fe640 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a68adc1c0 0x556a68f6a3f0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18b4b12d9ae5daa5 server_cookie=9b986b68f908327f in_seq=0 out_seq=0 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.852+0000 7fb82f7fe640 1 -- 172.21.15.27:0/1631432087 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556a68dd15f0 con 0x556a68adc1c0 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.852+0000 7fb82f7fe640 10 client.25549 handle_command_reply: tid=0 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.852+0000 7fb85d5b0740 1 client.25549 shutdown 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1631432087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 msgr2=0x556a68ef7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef7040 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x556a68f006a0 tx=0x7fb840012820 comp rx=0 tx=0).stop 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1631432087 shutdown_connections 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a68adc1c0 0x556a68f6a3f0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a68b481b0 0x556a68f662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a68b274d0 0x556a68ef7040 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 --2- 172.21.15.27:0/1631432087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a68b1f2f0 0x556a68ef6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb82f7fe640 0 client.25549 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1631432087 >> 172.21.15.27:0/1631432087 conn(0x556a68b3c300 msgr2=0x556a68ef0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1631432087 shutdown_connections 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 1 -- 172.21.15.27:0/1631432087 wait complete. 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb85d5b0740 20 client.25549 trim_cache size 0 max 16384 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:43:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:43:20.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 -- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8380afae0 msgr2=0x7fb8380b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8380afae0 0x7fb8380b1ff0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb84802abe0 tx=0x7fb848009280 comp rx=0 tx=0).stop 2024-04-25T13:43:20.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 -- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb850107970 msgr2=0x7fb8501a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:20.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb850107970 0x7fb8501a7c00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb848009b30 tx=0x7fb848030e00 comp rx=0 tx=0).stop 2024-04-25T13:43:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 -- 172.21.15.27:0/2693062348 shutdown_connections 2024-04-25T13:43:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb8380afae0 0x7fb8380b1ff0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb850107970 0x7fb8501a7c00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850102800 0x7fb8501a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 --2- 172.21.15.27:0/2693062348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850101dc0 0x7fb8501a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.853+0000 7fb856701640 1 -- 172.21.15.27:0/2693062348 >> 172.21.15.27:0/2693062348 conn(0x7fb8500fd390 msgr2=0x7fb850103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.854+0000 7fb856701640 1 -- 172.21.15.27:0/2693062348 shutdown_connections 2024-04-25T13:43:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:20.854+0000 7fb856701640 1 -- 172.21.15.27:0/2693062348 wait complete. 2024-04-25T13:43:21.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:21 smithi122 ceph-mon[32843]: pgmap v423: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 11 MiB/s wr, 803 op/s 2024-04-25T13:43:21.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:21 smithi045 ceph-mon[32731]: pgmap v423: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 11 MiB/s wr, 803 op/s 2024-04-25T13:43:21.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:21 smithi027 ceph-mon[29140]: pgmap v423: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 11 MiB/s wr, 803 op/s 2024-04-25T13:43:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 135.0 s, 3421.2 tps, lat 28.955 ms stddev 32.496 2024-04-25T13:43:23.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:23 smithi122 ceph-mon[32843]: pgmap v424: 97 pgs: 97 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 840 op/s 2024-04-25T13:43:23.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:23 smithi045 ceph-mon[32731]: pgmap v424: 97 pgs: 97 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 840 op/s 2024-04-25T13:43:23.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:23 smithi027 ceph-mon[29140]: pgmap v424: 97 pgs: 97 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 840 op/s 2024-04-25T13:43:25.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:25 smithi122 ceph-mon[32843]: pgmap v425: 97 pgs: 97 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 675 op/s 2024-04-25T13:43:25.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:25 smithi045 ceph-mon[32731]: pgmap v425: 97 pgs: 97 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 675 op/s 2024-04-25T13:43:25.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:25 smithi027 ceph-mon[29140]: pgmap v425: 97 pgs: 97 active+clean; 1.2 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 675 op/s 2024-04-25T13:43:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:27 smithi122 ceph-mon[32843]: pgmap v426: 97 pgs: 97 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 893 op/s 2024-04-25T13:43:27.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:27 smithi045 ceph-mon[32731]: pgmap v426: 97 pgs: 97 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 893 op/s 2024-04-25T13:43:27.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:27 smithi027 ceph-mon[29140]: pgmap v426: 97 pgs: 97 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 893 op/s 2024-04-25T13:43:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 140.0 s, 3167.0 tps, lat 31.784 ms stddev 35.316 2024-04-25T13:43:29.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:29 smithi122 ceph-mon[32843]: pgmap v427: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.9 MiB/s wr, 667 op/s 2024-04-25T13:43:29.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:29 smithi045 ceph-mon[32731]: pgmap v427: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.9 MiB/s wr, 667 op/s 2024-04-25T13:43:29.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:29 smithi027 ceph-mon[29140]: pgmap v427: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.9 MiB/s wr, 667 op/s 2024-04-25T13:43:31.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:31 smithi122 ceph-mon[32843]: pgmap v428: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8.9 MiB/s wr, 772 op/s 2024-04-25T13:43:31.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:31 smithi045 ceph-mon[32731]: pgmap v428: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8.9 MiB/s wr, 772 op/s 2024-04-25T13:43:31.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:31 smithi027 ceph-mon[29140]: pgmap v428: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8.9 MiB/s wr, 772 op/s 2024-04-25T13:43:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 145.0 s, 3135.0 tps, lat 31.761 ms stddev 32.657 2024-04-25T13:43:33.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:33 smithi122 ceph-mon[32843]: pgmap v429: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.6 MiB/s wr, 777 op/s 2024-04-25T13:43:33.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:33 smithi045 ceph-mon[32731]: pgmap v429: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.6 MiB/s wr, 777 op/s 2024-04-25T13:43:33.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:33 smithi027 ceph-mon[29140]: pgmap v429: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.6 MiB/s wr, 777 op/s 2024-04-25T13:43:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 -- 172.21.15.27:0/2478941829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54a4108980 msgr2=0x7f54a410ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 --2- 172.21.15.27:0/2478941829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54a4108980 0x7f54a410ada0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f549c009be0 tx=0x7f549c02a6c0 comp rx=0 tx=0).stop 2024-04-25T13:43:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 -- 172.21.15.27:0/2478941829 shutdown_connections 2024-04-25T13:43:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 --2- 172.21.15.27:0/2478941829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54a4108980 0x7f54a410ada0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 --2- 172.21.15.27:0/2478941829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54a4103c80 0x7f54a4104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 --2- 172.21.15.27:0/2478941829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54a4102a40 0x7f54a4103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 -- 172.21.15.27:0/2478941829 >> 172.21.15.27:0/2478941829 conn(0x7f54a40fdc30 msgr2=0x7f54a41000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:33.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 -- 172.21.15.27:0/2478941829 shutdown_connections 2024-04-25T13:43:33.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.994+0000 7f54ac682640 1 -- 172.21.15.27:0/2478941829 wait complete. 2024-04-25T13:43:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.995+0000 7f54ac682640 1 Processor -- start 2024-04-25T13:43:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.995+0000 7f54ac682640 1 -- start start 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.995+0000 7f54ac682640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54a4102a40 0x7f54a41a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.995+0000 7f54ac682640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54a4103c80 0x7f54a41a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.995+0000 7f54ac682640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54a4108980 0x7f54a41a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.995+0000 7f54ac682640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f54a4115c70 con 0x7f54a4108980 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.995+0000 7f54aa41e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54a4102a40 0x7f54a41a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.995+0000 7f54aa41e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54a4102a40 0x7f54a41a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40216/0 (socket says 172.21.15.27:40216) 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.996+0000 7f54aa41e640 1 -- 172.21.15.27:0/4065356894 learned_addr learned my addr 172.21.15.27:0/4065356894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.996+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f54a4115af0 con 0x7f54a4103c80 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.996+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f54a4115df0 con 0x7f54a4102a40 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.996+0000 7f54aac1f640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54a4108980 0x7f54a41a7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.996+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54a4103c80 0x7f54a41a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.997+0000 7f54aa41e640 1 -- 172.21.15.27:0/4065356894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54a4103c80 msgr2=0x7f54a41a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.997+0000 7f54aa41e640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54a4103c80 0x7f54a41a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.997+0000 7f54aa41e640 1 -- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54a4108980 msgr2=0x7f54a41a7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.997+0000 7f54aa41e640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54a4108980 0x7f54a41a7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.997+0000 7f54aa41e640 1 -- 172.21.15.27:0/4065356894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a41a8300 con 0x7f54a4102a40 2024-04-25T13:43:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.997+0000 7f54aa41e640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54a4102a40 0x7f54a41a07c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f54a000e980 tx=0x7f54a000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.998+0000 7f549b7fe640 1 -- 172.21.15.27:0/4065356894 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a000cd80 con 0x7f54a4102a40 2024-04-25T13:43:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.998+0000 7f549b7fe640 1 -- 172.21.15.27:0/4065356894 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54a00108e0 con 0x7f54a4102a40 2024-04-25T13:43:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.998+0000 7f549b7fe640 1 -- 172.21.15.27:0/4065356894 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a000cd80 con 0x7f54a4102a40 2024-04-25T13:43:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.998+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a41a85f0 con 0x7f54a4102a40 2024-04-25T13:43:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:33.998+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a41a8af0 con 0x7f54a4102a40 2024-04-25T13:43:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.001+0000 7f54b360d740 1 Processor -- start 2024-04-25T13:43:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.001+0000 7f549b7fe640 1 -- 172.21.15.27:0/4065356894 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f54a001f020 con 0x7f54a4102a40 2024-04-25T13:43:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54b360d740 1 -- start start 2024-04-25T13:43:34.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54b360d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce195970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54b360d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54b360d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 0x5608ce19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54b360d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5608ce0fb460 con 0x5608cddcc2f0 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54b360d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5608ce0b5590 con 0x5608cddd44d0 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40230/0 (socket says 172.21.15.27:40230) 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54aac1f640 1 -- 172.21.15.27:0/4088196074 learned_addr learned my addr 172.21.15.27:0/4088196074 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.002+0000 7f54b360d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5608ce0f7420 con 0x5608cddf51b0 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f54aa41e640 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 0x5608ce19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f54aac1f640 1 -- 172.21.15.27:0/4088196074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 msgr2=0x5608ce19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f54aac1f640 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 0x5608ce19cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f54aac1f640 1 -- 172.21.15.27:0/4088196074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 msgr2=0x5608ce195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f54aac1f640 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f54aac1f640 1 -- 172.21.15.27:0/4088196074 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608ce0c20d0 con 0x5608cddf51b0 2024-04-25T13:43:34.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f54aa41e640 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:43:34.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f549b7fe640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f54900afcd0 0x7f54900b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.003+0000 7f549b7fe640 1 -- 172.21.15.27:0/4065356894 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f54a00de900 con 0x7f54a4102a40 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.004+0000 7f54aac1f640 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce195970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f549c009be0 tx=0x7f549c0303a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=348efcf55cc778ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.004+0000 7f54aa41e640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f54900afcd0 0x7f54900b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.004+0000 7f5498bfe640 1 -- 172.21.15.27:0/4088196074 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c040030 con 0x5608cddf51b0 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.004+0000 7f5498bfe640 1 -- 172.21.15.27:0/4088196074 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549c030590 con 0x5608cddf51b0 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.004+0000 7f5498bfe640 1 -- 172.21.15.27:0/4088196074 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c036040 con 0x5608cddf51b0 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.004+0000 7f54aa41e640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f54900afcd0 0x7f54900b21e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x5608ce196d50 tx=0x7f54a0010c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 -- 172.21.15.27:0/4088196074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 msgr2=0x5608ce195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce195970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f549c009be0 tx=0x7f549c0303a0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 -- 172.21.15.27:0/4088196074 shutdown_connections 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 0x5608ce19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 --2- 172.21.15.27:0/4088196074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce195970 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 -- 172.21.15.27:0/4088196074 >> 172.21.15.27:0/4088196074 conn(0x5608cdde9300 msgr2=0x5608ce191300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 -- 172.21.15.27:0/4088196074 shutdown_connections 2024-04-25T13:43:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.006+0000 7f54b360d740 1 -- 172.21.15.27:0/4088196074 wait complete. 2024-04-25T13:43:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.007+0000 7f54b360d740 1 -- start start 2024-04-25T13:43:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.007+0000 7f54b360d740 1 Processor -- start 2024-04-25T13:43:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54b360d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddcc2f0 0x5608ce19f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54b360d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 0x5608ce19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddcc2f0 0x5608ce19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddcc2f0 0x5608ce19f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43730/0 (socket says 172.21.15.27:43730) 2024-04-25T13:43:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54aac1f640 1 -- 172.21.15.27:0/815144763 learned_addr learned my addr 172.21.15.27:0/815144763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54b360d740 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce1a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 0x5608ce19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5608ce19d830 con 0x5608cddd44d0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5608ce082b60 con 0x5608cddcc2f0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.008+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5608ce19d9b0 con 0x5608cddf51b0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.009+0000 7f54b360d740 10 client.0 _ll_register_callbacks cb 0x5608ce0a72b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.009+0000 7f54aac1f640 1 -- 172.21.15.27:0/815144763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 msgr2=0x5608ce1a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.009+0000 7f54aac1f640 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce1a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.009+0000 7f54aac1f640 1 -- 172.21.15.27:0/815144763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 msgr2=0x5608ce19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.009+0000 7f54aac1f640 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 0x5608ce19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.009+0000 7f54aac1f640 1 -- 172.21.15.27:0/815144763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608cde0d210 con 0x5608cddcc2f0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.009+0000 7f54aac1f640 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddcc2f0 0x5608ce19f650 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f549c034040 tx=0x7f549c009300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c268a16d3cdcc869 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.009+0000 7f548dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f548dffb640 1 -- 172.21.15.27:0/815144763 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c036050 con 0x5608cddcc2f0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f548dffb640 1 -- 172.21.15.27:0/815144763 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f549c030a70 con 0x5608cddcc2f0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f549c030590 con 0x5608cddcc2f0 2024-04-25T13:43:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f548dffb640 1 -- 172.21.15.27:0/815144763 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f549c040040 con 0x5608cddcc2f0 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f54b360d740 10 client.25538 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5608cded9620 con 0x5608cddcc2f0 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f548dffb640 1 -- 172.21.15.27:0/815144763 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549c030c70 con 0x5608cddcc2f0 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f548dffb640 1 -- 172.21.15.27:0/815144763 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f549c031440 con 0x5608cddcc2f0 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.010+0000 7f54b360d740 10 client.25538 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.011+0000 7f54b360d740 10 client.25538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.011+0000 7f54b360d740 20 client.25538 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.011+0000 7f54b360d740 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5608cddb3d80 0x5608ce1aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.011+0000 7f54b360d740 4 client.25538 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.011+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f549c030590 con 0x5608cddb3d80 2024-04-25T13:43:34.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.011+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5608cddb3d80 0x5608ce1aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.012+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5608cddb3d80 0x5608ce1aceb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f353bb65af1ce791 server_cookie=fc810b8986388763 in_seq=0 out_seq=0 2024-04-25T13:43:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.012+0000 7f548dffb640 10 client.25538 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f548dffb640 1 -- 172.21.15.27:0/815144763 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f549c030590 con 0x5608cddb3d80 2024-04-25T13:43:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f548dffb640 10 client.25538 handle_command_reply: tid=0 2024-04-25T13:43:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 client.25538 shutdown 2024-04-25T13:43:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddcc2f0 msgr2=0x5608ce19f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddcc2f0 0x5608ce19f650 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f549c034040 tx=0x7f549c009300 comp rx=0 tx=0).stop 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 shutdown_connections 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5608cddb3d80 0x5608ce1aceb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f548dffb640 0 client.25538 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddf51b0 0x5608ce1a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 0x5608ce19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 --2- 172.21.15.27:0/815144763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddcc2f0 0x5608ce19f650 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.013+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 >> 172.21.15.27:0/815144763 conn(0x5608cdde9300 msgr2=0x5608ce190250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.014+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 shutdown_connections 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.014+0000 7f54b360d740 1 -- 172.21.15.27:0/815144763 wait complete. 2024-04-25T13:43:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.014+0000 7f54b360d740 20 client.25538 trim_cache size 0 max 16384 2024-04-25T13:43:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.042+0000 7f54b360d740 1 Processor -- start 2024-04-25T13:43:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.042+0000 7f54b360d740 1 -- start start 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54b360d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddb3d80 0x5608ce190ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54b360d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce1910e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54b360d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 0x5608ce19f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54b360d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f54a41af870 con 0x5608cddcc2f0 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54b360d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f54a41b20f0 con 0x5608cddd44d0 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54b360d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f54a41afee0 con 0x5608cddb3d80 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddb3d80 0x5608ce190ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 0x5608ce19f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddb3d80 0x5608ce190ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40240/0 (socket says 172.21.15.27:40240) 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 -- 172.21.15.27:0/3230207097 learned_addr learned my addr 172.21.15.27:0/3230207097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/3230207097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce1910e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.044 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 -- 172.21.15.27:0/3230207097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 msgr2=0x5608ce19f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:34.044 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 --2- 172.21.15.27:0/3230207097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 0x5608ce19f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.044 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 -- 172.21.15.27:0/3230207097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 msgr2=0x5608ce1910e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:34.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 --2- 172.21.15.27:0/3230207097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce1910e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.043+0000 7f54aac1f640 1 -- 172.21.15.27:0/3230207097 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608ce1a06c0 con 0x5608cddb3d80 2024-04-25T13:43:34.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.044+0000 7f54aac1f640 1 --2- 172.21.15.27:0/3230207097 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddb3d80 0x5608ce190ba0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f549c031890 tx=0x7f549c000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=264bd5b9604a1d0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:34.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.044+0000 7f548ffff640 1 -- 172.21.15.27:0/3230207097 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c036030 con 0x5608cddb3d80 2024-04-25T13:43:34.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.044+0000 7f548ffff640 1 -- 172.21.15.27:0/3230207097 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549c05e070 con 0x5608cddb3d80 2024-04-25T13:43:34.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.044+0000 7f548ffff640 1 -- 172.21.15.27:0/3230207097 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c058680 con 0x5608cddb3d80 2024-04-25T13:43:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.045+0000 7f54b360d740 1 -- 172.21.15.27:0/3230207097 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddb3d80 msgr2=0x5608ce190ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.045+0000 7f54b360d740 1 --2- 172.21.15.27:0/3230207097 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddb3d80 0x5608ce190ba0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f549c031890 tx=0x7f549c000f40 comp rx=0 tx=0).stop 2024-04-25T13:43:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.045+0000 7f54b360d740 1 -- 172.21.15.27:0/3230207097 shutdown_connections 2024-04-25T13:43:34.047 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.046+0000 7f54b360d740 1 --2- 172.21.15.27:0/3230207097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddd44d0 0x5608ce19f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.046+0000 7f54b360d740 1 --2- 172.21.15.27:0/3230207097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddcc2f0 0x5608ce1910e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.046+0000 7f54b360d740 1 --2- 172.21.15.27:0/3230207097 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddb3d80 0x5608ce190ba0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.046+0000 7f54b360d740 1 -- 172.21.15.27:0/3230207097 >> 172.21.15.27:0/3230207097 conn(0x5608cdde9300 msgr2=0x5608ce198d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.046+0000 7f54b360d740 1 -- 172.21.15.27:0/3230207097 shutdown_connections 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.046+0000 7f54b360d740 1 -- 172.21.15.27:0/3230207097 wait complete. 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.046+0000 7f54b360d740 1 -- start start 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.046+0000 7f54b360d740 1 Processor -- start 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54b360d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddb3d80 0x5608ce1a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54b360d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddcc2f0 0x5608ce1a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54b360d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 0x5608ce1a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54b360d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f54a41b05b0 con 0x5608cddd44d0 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54b360d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f54a41b1a20 con 0x5608cddb3d80 2024-04-25T13:43:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54b360d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f54a41b0c80 con 0x5608cddcc2f0 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54b360d740 10 client.0 _ll_register_callbacks cb 0x7f5494005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddb3d80 0x5608ce1a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54a9c1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddcc2f0 0x5608ce1a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54aac1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddb3d80 0x5608ce1a38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43736/0 (socket says 172.21.15.27:43736) 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54a9c1d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddcc2f0 0x5608ce1a3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40250/0 (socket says 172.21.15.27:40250) 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54aac1f640 1 -- 172.21.15.27:0/2042875686 learned_addr learned my addr 172.21.15.27:0/2042875686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54aac1f640 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 0x5608ce1a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54a9c1d640 1 -- 172.21.15.27:0/2042875686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddb3d80 msgr2=0x5608ce1a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddb3d80 0x5608ce1a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54a9c1d640 1 -- 172.21.15.27:0/2042875686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 msgr2=0x5608ce1a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 0x5608ce1a4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.047+0000 7f54a9c1d640 1 -- 172.21.15.27:0/2042875686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608ce1a5320 con 0x5608cddcc2f0 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.048+0000 7f54a9c1d640 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddcc2f0 0x5608ce1a3e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x5608ce1ad4d0 tx=0x7f5494020720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=230949d16175f0a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.048+0000 7f548dffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.048+0000 7f548dffb640 1 -- 172.21.15.27:0/2042875686 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494004220 con 0x5608cddcc2f0 2024-04-25T13:43:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.048+0000 7f548dffb640 1 -- 172.21.15.27:0/2042875686 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f54940043c0 con 0x5608cddcc2f0 2024-04-25T13:43:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.048+0000 7f54b360d740 1 -- 172.21.15.27:0/2042875686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f549c05e070 con 0x5608cddcc2f0 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.049+0000 7f548dffb640 1 -- 172.21.15.27:0/2042875686 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f5494011850 con 0x5608cddcc2f0 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.049+0000 7f54b360d740 10 client.25564 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.049+0000 7f54b360d740 1 -- 172.21.15.27:0/2042875686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5608ce2150e0 con 0x5608cddcc2f0 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f548dffb640 1 -- 172.21.15.27:0/2042875686 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5494024070 con 0x5608cddcc2f0 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f548dffb640 1 -- 172.21.15.27:0/2042875686 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f549401f480 con 0x5608cddcc2f0 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f54b360d740 10 client.25564 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f54b360d740 10 client.25564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f54b360d740 20 client.25564 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f54b360d740 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5608cddf51b0 0x5608ce2179b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f54b360d740 4 client.25564 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f54b360d740 1 -- 172.21.15.27:0/2042875686 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f549c05e070 con 0x5608cddf51b0 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.050+0000 7f54aac1f640 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5608cddf51b0 0x5608ce2179b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.051+0000 7f548dffb640 10 client.25564 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.051+0000 7f54aac1f640 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5608cddf51b0 0x5608ce2179b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7a7cb5bd3945ab4 server_cookie=df2afe55029678f9 in_seq=0 out_seq=0 2024-04-25T13:43:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f548dffb640 1 -- 172.21.15.27:0/2042875686 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f549c05e070 con 0x5608cddf51b0 2024-04-25T13:43:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f548dffb640 10 client.25564 handle_command_reply: tid=0 2024-04-25T13:43:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 client.25564 shutdown 2024-04-25T13:43:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 -- 172.21.15.27:0/2042875686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddcc2f0 msgr2=0x5608ce1a3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddcc2f0 0x5608ce1a3e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x5608ce1ad4d0 tx=0x7f5494020720 comp rx=0 tx=0).stop 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 -- 172.21.15.27:0/2042875686 shutdown_connections 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5608cddf51b0 0x5608ce2179b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608cddd44d0 0x5608ce1a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f548dffb640 0 client.25564 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5608cddcc2f0 0x5608ce1a3e30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 --2- 172.21.15.27:0/2042875686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608cddb3d80 0x5608ce1a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.068+0000 7f54b360d740 1 -- 172.21.15.27:0/2042875686 >> 172.21.15.27:0/2042875686 conn(0x5608cdde9300 msgr2=0x5608ce190ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54b360d740 1 -- 172.21.15.27:0/2042875686 shutdown_connections 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54b360d740 1 -- 172.21.15.27:0/2042875686 wait complete. 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54b360d740 20 client.25564 trim_cache size 0 max 16384 2024-04-25T13:43:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:43:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:43:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:43:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:43:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f54900afcd0 msgr2=0x7f54900b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f54900afcd0 0x7f54900b21e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x5608ce196d50 tx=0x7f54a0010c70 comp rx=0 tx=0).stop 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54a4102a40 msgr2=0x7f54a41a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54a4102a40 0x7f54a41a07c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f54a000e980 tx=0x7f54a000ee40 comp rx=0 tx=0).stop 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 shutdown_connections 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f54900afcd0 0x7f54900b21e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54a4108980 0x7f54a41a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54a4103c80 0x7f54a41a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 --2- 172.21.15.27:0/4065356894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54a4102a40 0x7f54a41a07c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.069+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 >> 172.21.15.27:0/4065356894 conn(0x7f54a40fdc30 msgr2=0x7f54a4104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.070+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 shutdown_connections 2024-04-25T13:43:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:34.070+0000 7f54ac682640 1 -- 172.21.15.27:0/4065356894 wait complete. 2024-04-25T13:43:35.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:43:35.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:43:35.791 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:35 smithi027 ceph-mon[29140]: pgmap v430: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 7.4 MiB/s wr, 665 op/s 2024-04-25T13:43:35.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:35 smithi122 ceph-mon[32843]: pgmap v430: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 7.4 MiB/s wr, 665 op/s 2024-04-25T13:43:35.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:35 smithi045 ceph-mon[32731]: pgmap v430: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 7.4 MiB/s wr, 665 op/s 2024-04-25T13:43:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 -- 172.21.15.27:0/2612533315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c108920 msgr2=0x7f905c10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 --2- 172.21.15.27:0/2612533315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c108920 0x7f905c10ad40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9054009ca0 tx=0x7f905402a8b0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 -- 172.21.15.27:0/2612533315 shutdown_connections 2024-04-25T13:43:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 --2- 172.21.15.27:0/2612533315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c108920 0x7f905c10ad40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 --2- 172.21.15.27:0/2612533315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f905c103c20 0x7f905c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 --2- 172.21.15.27:0/2612533315 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f905c1029e0 0x7f905c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 -- 172.21.15.27:0/2612533315 >> 172.21.15.27:0/2612533315 conn(0x7f905c0fdbd0 msgr2=0x7f905c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 -- 172.21.15.27:0/2612533315 shutdown_connections 2024-04-25T13:43:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.937+0000 7f90635d6640 1 -- 172.21.15.27:0/2612533315 wait complete. 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.938+0000 7f90635d6640 1 Processor -- start 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f90635d6640 1 -- start start 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f90635d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c1029e0 0x7f905c06a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f90635d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f905c103c20 0x7f905c1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f90635d6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f905c108920 0x7f905c1a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f90635d6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f905c10e560 con 0x7f905c1029e0 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f90635d6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f905c10e3e0 con 0x7f905c103c20 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f90635d6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f905c10e6e0 con 0x7f905c108920 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c1029e0 0x7f905c06a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061b73640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f905c108920 0x7f905c1a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9060b71640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f905c103c20 0x7f905c1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061372640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c1029e0 0x7f905c06a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47548/0 (socket says 172.21.15.27:47548) 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061372640 1 -- 172.21.15.27:0/604526884 learned_addr learned my addr 172.21.15.27:0/604526884 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061372640 1 -- 172.21.15.27:0/604526884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f905c108920 msgr2=0x7f905c1a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061372640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f905c108920 0x7f905c1a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061372640 1 -- 172.21.15.27:0/604526884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f905c103c20 msgr2=0x7f905c1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061372640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f905c103c20 0x7f905c1a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.939+0000 7f9061372640 1 -- 172.21.15.27:0/604526884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c1a4270 con 0x7f905c1029e0 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.940+0000 7f9061372640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c1029e0 0x7f905c06a900 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f905800e9f0 tx=0x7f905800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.940+0000 7f90527fc640 1 -- 172.21.15.27:0/604526884 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905800cc40 con 0x7f905c1029e0 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.940+0000 7f90635d6640 1 -- 172.21.15.27:0/604526884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c1a4560 con 0x7f905c1029e0 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.940+0000 7f90527fc640 1 -- 172.21.15.27:0/604526884 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f905800cde0 con 0x7f905c1029e0 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.940+0000 7f90527fc640 1 -- 172.21.15.27:0/604526884 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058010430 con 0x7f905c1029e0 2024-04-25T13:43:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.940+0000 7f90635d6640 1 -- 172.21.15.27:0/604526884 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c1a4b80 con 0x7f905c1029e0 2024-04-25T13:43:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.944+0000 7f906a485740 1 Processor -- start 2024-04-25T13:43:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.945+0000 7f906a485740 1 -- start start 2024-04-25T13:43:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.945+0000 7f90527fc640 1 -- 172.21.15.27:0/604526884 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9058003ef0 con 0x7f905c1029e0 2024-04-25T13:43:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.945+0000 7f906a485740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fe2390 0x563ac638c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.945+0000 7f906a485740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac638c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.945+0000 7f906a485740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac638d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.945+0000 7f906a485740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563ac62e8d80 con 0x563ac5fc16b0 2024-04-25T13:43:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.945+0000 7f906a485740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563ac62a28b0 con 0x563ac5fe2390 2024-04-25T13:43:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f906a485740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563ac62e4d40 con 0x563ac5fb94d0 2024-04-25T13:43:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061b73640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fe2390 0x563ac638c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061b73640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fe2390 0x563ac638c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43750/0 (socket says 172.21.15.27:43750) 2024-04-25T13:43:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061b73640 1 -- 172.21.15.27:0/2472925711 learned_addr learned my addr 172.21.15.27:0/2472925711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061372640 1 --2- 172.21.15.27:0/2472925711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac638c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061b73640 1 -- 172.21.15.27:0/2472925711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 msgr2=0x563ac638c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061b73640 1 --2- 172.21.15.27:0/2472925711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac638c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061b73640 1 -- 172.21.15.27:0/2472925711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 msgr2=0x563ac638d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061b73640 1 --2- 172.21.15.27:0/2472925711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac638d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.946+0000 7f9061b73640 1 -- 172.21.15.27:0/2472925711 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac6273a40 con 0x563ac5fe2390 2024-04-25T13:43:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.947+0000 7f90527fc640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90480afc30 0x7f90480b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.947+0000 7f9061b73640 1 --2- 172.21.15.27:0/2472925711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fe2390 0x563ac638c270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9054009ca0 tx=0x7f9054030f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a3f74888ef9e588 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:35.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.947+0000 7f90477fe640 1 -- 172.21.15.27:0/2472925711 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054030900 con 0x563ac5fe2390 2024-04-25T13:43:35.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.947+0000 7f9061b73640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90480afc30 0x7f90480b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.948+0000 7f90477fe640 1 -- 172.21.15.27:0/2472925711 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9054036030 con 0x563ac5fe2390 2024-04-25T13:43:35.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.948+0000 7f90477fe640 1 -- 172.21.15.27:0/2472925711 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90540046e0 con 0x563ac5fe2390 2024-04-25T13:43:35.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.948+0000 7f90527fc640 1 -- 172.21.15.27:0/604526884 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f90580def00 con 0x7f905c1029e0 2024-04-25T13:43:35.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.949+0000 7f906a485740 1 -- 172.21.15.27:0/2472925711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fe2390 msgr2=0x563ac638c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.949+0000 7f9061b73640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90480afc30 0x7f90480b2140 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f90540049d0 tx=0x7f9054045440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:35.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.949+0000 7f906a485740 1 --2- 172.21.15.27:0/2472925711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fe2390 0x563ac638c270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9054009ca0 tx=0x7f9054030f00 comp rx=0 tx=0).stop 2024-04-25T13:43:35.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.949+0000 7f906a485740 1 -- 172.21.15.27:0/2472925711 shutdown_connections 2024-04-25T13:43:35.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.950+0000 7f906a485740 1 --2- 172.21.15.27:0/2472925711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac638d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.950+0000 7f906a485740 1 --2- 172.21.15.27:0/2472925711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac638c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.950+0000 7f906a485740 1 --2- 172.21.15.27:0/2472925711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fe2390 0x563ac638c270 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.950+0000 7f906a485740 1 -- 172.21.15.27:0/2472925711 >> 172.21.15.27:0/2472925711 conn(0x563ac5fd64e0 msgr2=0x563ac6387a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:35.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.950+0000 7f906a485740 1 -- 172.21.15.27:0/2472925711 shutdown_connections 2024-04-25T13:43:35.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.950+0000 7f906a485740 1 -- 172.21.15.27:0/2472925711 wait complete. 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.950+0000 7f906a485740 1 -- start start 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.950+0000 7f906a485740 1 Processor -- start 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f906a485740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fb94d0 0x563ac6395fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f906a485740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fc16b0 0x563ac6396500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f906a485740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fe2390 0x563ac639d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f906a485740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563ac638df30 con 0x563ac5fb94d0 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f906a485740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563ac626fd10 con 0x563ac5fc16b0 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f906a485740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563ac638e0b0 con 0x563ac5fe2390 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f906a485740 10 client.0 _ll_register_callbacks cb 0x563ac629c210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:35.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9060b71640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fc16b0 0x563ac6396500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9060b71640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fc16b0 0x563ac6396500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43766/0 (socket says 172.21.15.27:43766) 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9060b71640 1 -- 172.21.15.27:0/2819384211 learned_addr learned my addr 172.21.15.27:0/2819384211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9061372640 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fb94d0 0x563ac6395fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9061b73640 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fe2390 0x563ac639d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9060b71640 1 -- 172.21.15.27:0/2819384211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fe2390 msgr2=0x563ac639d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9060b71640 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fe2390 0x563ac639d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9060b71640 1 -- 172.21.15.27:0/2819384211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fb94d0 msgr2=0x563ac6395fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.951+0000 7f9060b71640 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fb94d0 0x563ac6395fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f9060b71640 1 -- 172.21.15.27:0/2819384211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac60c47d0 con 0x563ac5fc16b0 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f9060b71640 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fc16b0 0x563ac6396500 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e480 tx=0x7f904c00e880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=112ea216d46cf2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f9044ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f9044ff9640 1 -- 172.21.15.27:0/2819384211 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c017070 con 0x563ac5fc16b0 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f9044ff9640 1 -- 172.21.15.27:0/2819384211 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f904c013680 con 0x563ac5fc16b0 2024-04-25T13:43:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f906a485740 1 -- 172.21.15.27:0/2819384211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9054036030 con 0x563ac5fc16b0 2024-04-25T13:43:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f9044ff9640 1 -- 172.21.15.27:0/2819384211 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f904c023050 con 0x563ac5fc16b0 2024-04-25T13:43:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f906a485740 10 client.25553 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f906a485740 1 -- 172.21.15.27:0/2819384211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563ac5ffa3f0 con 0x563ac5fc16b0 2024-04-25T13:43:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.952+0000 7f9044ff9640 1 -- 172.21.15.27:0/2819384211 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9054036030 con 0x563ac5fc16b0 2024-04-25T13:43:35.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.953+0000 7f9044ff9640 1 -- 172.21.15.27:0/2819384211 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f904c012880 con 0x563ac5fc16b0 2024-04-25T13:43:35.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.953+0000 7f906a485740 10 client.25553 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:35.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.953+0000 7f906a485740 10 client.25553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:35.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.953+0000 7f906a485740 20 client.25553 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:35.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.953+0000 7f906a485740 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563ac5fa0f60 0x563ac63a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.953+0000 7f906a485740 4 client.25553 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.953+0000 7f906a485740 1 -- 172.21.15.27:0/2819384211 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ac62f6990 con 0x563ac5fa0f60 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.953+0000 7f9061372640 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563ac5fa0f60 0x563ac63a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.954+0000 7f9044ff9640 10 client.25553 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.954+0000 7f9061372640 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563ac5fa0f60 0x563ac63a39d0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90d39493a232cd3c server_cookie=c94c8ddbdcbbf491 in_seq=0 out_seq=0 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f9044ff9640 1 -- 172.21.15.27:0/2819384211 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563ac62f6990 con 0x563ac5fa0f60 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f9044ff9640 10 client.25553 handle_command_reply: tid=0 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 client.25553 shutdown 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 -- 172.21.15.27:0/2819384211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fc16b0 msgr2=0x563ac6396500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fc16b0 0x563ac6396500 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e480 tx=0x7f904c00e880 comp rx=0 tx=0).stop 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 -- 172.21.15.27:0/2819384211 shutdown_connections 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563ac5fa0f60 0x563ac63a39d0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fe2390 0x563ac639d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f9044ff9640 0 client.25553 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fc16b0 0x563ac6396500 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 --2- 172.21.15.27:0/2819384211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fb94d0 0x563ac6395fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 -- 172.21.15.27:0/2819384211 >> 172.21.15.27:0/2819384211 conn(0x563ac5fd64e0 msgr2=0x563ac637e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 -- 172.21.15.27:0/2819384211 shutdown_connections 2024-04-25T13:43:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 1 -- 172.21.15.27:0/2819384211 wait complete. 2024-04-25T13:43:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.955+0000 7f906a485740 20 client.25553 trim_cache size 0 max 16384 2024-04-25T13:43:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f906a485740 1 Processor -- start 2024-04-25T13:43:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f906a485740 1 -- start start 2024-04-25T13:43:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f906a485740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac6395250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f906a485740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f906a485740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac6390d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f906a485740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1a6fc0 con 0x563ac5fc16b0 2024-04-25T13:43:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f906a485740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1a9840 con 0x563ac5fa0f60 2024-04-25T13:43:35.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f906a485740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1a7630 con 0x563ac5fb94d0 2024-04-25T13:43:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f9060b71640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f9060b71640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40288/0 (socket says 172.21.15.27:40288) 2024-04-25T13:43:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f9060b71640 1 -- 172.21.15.27:0/4250940257 learned_addr learned my addr 172.21.15.27:0/4250940257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.983+0000 7f9061372640 1 --2- 172.21.15.27:0/4250940257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac6395250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.984+0000 7f9060b71640 1 -- 172.21.15.27:0/4250940257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 msgr2=0x563ac6395250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.984+0000 7f9060b71640 1 --2- 172.21.15.27:0/4250940257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac6395250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.984+0000 7f9060b71640 1 -- 172.21.15.27:0/4250940257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 msgr2=0x563ac6390d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.984+0000 7f9060b71640 1 --2- 172.21.15.27:0/4250940257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac6390d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.984+0000 7f9060b71640 1 -- 172.21.15.27:0/4250940257 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac63966a0 con 0x563ac5fb94d0 2024-04-25T13:43:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.984+0000 7f9060b71640 1 --2- 172.21.15.27:0/4250940257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639d380 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f904c004810 tx=0x7f904c009110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b92c7e114b6230b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.985+0000 7f9046ffd640 1 -- 172.21.15.27:0/4250940257 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c02c070 con 0x563ac5fb94d0 2024-04-25T13:43:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.985+0000 7f9046ffd640 1 -- 172.21.15.27:0/4250940257 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f904c020030 con 0x563ac5fb94d0 2024-04-25T13:43:35.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.985+0000 7f9046ffd640 1 -- 172.21.15.27:0/4250940257 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c017030 con 0x563ac5fb94d0 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 -- 172.21.15.27:0/4250940257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 msgr2=0x563ac639d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 --2- 172.21.15.27:0/4250940257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639d380 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f904c004810 tx=0x7f904c009110 comp rx=0 tx=0).stop 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 -- 172.21.15.27:0/4250940257 shutdown_connections 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 --2- 172.21.15.27:0/4250940257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac6390d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 --2- 172.21.15.27:0/4250940257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639d380 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 --2- 172.21.15.27:0/4250940257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac6395250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 -- 172.21.15.27:0/4250940257 >> 172.21.15.27:0/4250940257 conn(0x563ac5fd64e0 msgr2=0x563ac638c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 -- 172.21.15.27:0/4250940257 shutdown_connections 2024-04-25T13:43:35.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 -- 172.21.15.27:0/4250940257 wait complete. 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 -- start start 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.986+0000 7f906a485740 1 Processor -- start 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f906a485740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac639a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f906a485740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f906a485740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac639bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f906a485740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1a7d00 con 0x563ac5fc16b0 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f906a485740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1a9170 con 0x563ac5fa0f60 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f906a485740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1a83d0 con 0x563ac5fb94d0 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f906a485740 10 client.0 _ll_register_callbacks cb 0x7f904c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9061372640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac639a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9061b73640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac639bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9060b71640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9061372640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac639a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43788/0 (socket says 172.21.15.27:43788) 2024-04-25T13:43:35.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9061b73640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac639bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47582/0 (socket says 172.21.15.27:47582) 2024-04-25T13:43:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9061372640 1 -- 172.21.15.27:0/452397093 learned_addr learned my addr 172.21.15.27:0/452397093 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9060b71640 1 -- 172.21.15.27:0/452397093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 msgr2=0x563ac639a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9060b71640 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac639a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9060b71640 1 -- 172.21.15.27:0/452397093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 msgr2=0x563ac639bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9060b71640 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac639bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.987+0000 7f9060b71640 1 -- 172.21.15.27:0/452397093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac5fba0f0 con 0x563ac5fb94d0 2024-04-25T13:43:35.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.988+0000 7f9060b71640 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639acf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f904c012dc0 tx=0x7f904c00ee10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca849793e3e7ed01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.989+0000 7f9044ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.989+0000 7f9044ff9640 1 -- 172.21.15.27:0/452397093 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c017070 con 0x563ac5fb94d0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.989+0000 7f9044ff9640 1 -- 172.21.15.27:0/452397093 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f904c0059c0 con 0x563ac5fb94d0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.989+0000 7f906a485740 1 -- 172.21.15.27:0/452397093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f904c020030 con 0x563ac5fb94d0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.989+0000 7f9044ff9640 1 -- 172.21.15.27:0/452397093 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f904c038070 con 0x563ac5fb94d0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f906a485740 10 client.25582 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f906a485740 1 -- 172.21.15.27:0/452397093 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563ac62a94b0 con 0x563ac5fb94d0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f9044ff9640 1 -- 172.21.15.27:0/452397093 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f904c020030 con 0x563ac5fb94d0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f9044ff9640 1 -- 172.21.15.27:0/452397093 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f904c005f90 con 0x563ac5fb94d0 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f906a485740 10 client.25582 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:35.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f906a485740 10 client.25582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f906a485740 20 client.25582 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f906a485740 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563ac5fe2390 0x563ac640de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f906a485740 4 client.25582 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f906a485740 1 -- 172.21.15.27:0/452397093 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ac62f0800 con 0x563ac5fe2390 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.990+0000 7f9061372640 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563ac5fe2390 0x563ac640de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f9044ff9640 10 client.25582 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f9061372640 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563ac5fe2390 0x563ac640de10 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e75deb7970a8fd15 server_cookie=b528b5643299a391 in_seq=0 out_seq=0 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f9044ff9640 1 -- 172.21.15.27:0/452397093 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563ac62f0800 con 0x563ac5fe2390 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f9044ff9640 10 client.25582 handle_command_reply: tid=0 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 client.25582 shutdown 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 -- 172.21.15.27:0/452397093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 msgr2=0x563ac639acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639acf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f904c012dc0 tx=0x7f904c00ee10 comp rx=0 tx=0).stop 2024-04-25T13:43:35.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 -- 172.21.15.27:0/452397093 shutdown_connections 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563ac5fe2390 0x563ac640de10 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563ac5fc16b0 0x563ac639bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563ac5fb94d0 0x563ac639acf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f9044ff9640 0 client.25582 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 --2- 172.21.15.27:0/452397093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ac5fa0f60 0x563ac639a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 -- 172.21.15.27:0/452397093 >> 172.21.15.27:0/452397093 conn(0x563ac5fd64e0 msgr2=0x563ac6390d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 -- 172.21.15.27:0/452397093 shutdown_connections 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 1 -- 172.21.15.27:0/452397093 wait complete. 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.991+0000 7f906a485740 20 client.25582 trim_cache size 0 max 16384 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:43:35.995 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:43:35.996 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:43:35.996 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.992+0000 7f90635d6640 1 -- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90480afc30 msgr2=0x7f90480b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.992+0000 7f90635d6640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90480afc30 0x7f90480b2140 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f90540049d0 tx=0x7f9054045440 comp rx=0 tx=0).stop 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.992+0000 7f90635d6640 1 -- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c1029e0 msgr2=0x7f905c06a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.992+0000 7f90635d6640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c1029e0 0x7f905c06a900 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f905800e9f0 tx=0x7f905800edf0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.993+0000 7f90635d6640 1 -- 172.21.15.27:0/604526884 shutdown_connections 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.993+0000 7f90635d6640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90480afc30 0x7f90480b2140 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.993+0000 7f90635d6640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f905c108920 0x7f905c1a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.993+0000 7f90635d6640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f905c103c20 0x7f905c1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.993+0000 7f90635d6640 1 --2- 172.21.15.27:0/604526884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f905c1029e0 0x7f905c06a900 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.993+0000 7f90635d6640 1 -- 172.21.15.27:0/604526884 >> 172.21.15.27:0/604526884 conn(0x7f905c0fdbd0 msgr2=0x7f905c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:35.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.993+0000 7f90635d6640 1 -- 172.21.15.27:0/604526884 shutdown_connections 2024-04-25T13:43:35.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:35.993+0000 7f90635d6640 1 -- 172.21.15.27:0/604526884 wait complete. 2024-04-25T13:43:36.007 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:43:36.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:43:36.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.210+0000 7f63b4100640 1 -- 172.21.15.27:0/115811886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac106f40 msgr2=0x7f63ac107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.210+0000 7f63b4100640 1 --2- 172.21.15.27:0/115811886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac106f40 0x7f63ac107350 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f639c009900 tx=0x7f639c02a410 comp rx=0 tx=0).stop 2024-04-25T13:43:36.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.210+0000 7f63b4100640 1 -- 172.21.15.27:0/115811886 shutdown_connections 2024-04-25T13:43:36.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.210+0000 7f63b4100640 1 --2- 172.21.15.27:0/115811886 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ac10bc40 0x7f63ac10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.210+0000 7f63b4100640 1 --2- 172.21.15.27:0/115811886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac106f40 0x7f63ac107350 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.210+0000 7f63b4100640 1 --2- 172.21.15.27:0/115811886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ac105d00 0x7f63ac106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.210+0000 7f63b4100640 1 -- 172.21.15.27:0/115811886 >> 172.21.15.27:0/115811886 conn(0x7f63ac100f10 msgr2=0x7f63ac1033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:36.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.211+0000 7f63b4100640 1 -- 172.21.15.27:0/115811886 shutdown_connections 2024-04-25T13:43:36.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.211+0000 7f63b4100640 1 -- 172.21.15.27:0/115811886 wait complete. 2024-04-25T13:43:36.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.211+0000 7f63b4100640 1 Processor -- start 2024-04-25T13:43:36.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.211+0000 7f63b4100640 1 -- start start 2024-04-25T13:43:36.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b4100640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ac105d00 0x7f63ac19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b4100640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ac106f40 0x7f63ac19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b4100640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac10bc40 0x7f63ac1a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ac105d00 0x7f63ac19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b4100640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111800 con 0x7f63ac10bc40 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b4100640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111680 con 0x7f63ac105d00 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ac105d00 0x7f63ac19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43814/0 (socket says 172.21.15.27:43814) 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 -- 172.21.15.27:0/3462120031 learned_addr learned my addr 172.21.15.27:0/3462120031 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b4100640 1 -- 172.21.15.27:0/3462120031 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111980 con 0x7f63ac106f40 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b169b640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ac106f40 0x7f63ac19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b269d640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac10bc40 0x7f63ac1a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 -- 172.21.15.27:0/3462120031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ac106f40 msgr2=0x7f63ac19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ac106f40 0x7f63ac19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 -- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac10bc40 msgr2=0x7f63ac1a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac10bc40 0x7f63ac1a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 -- 172.21.15.27:0/3462120031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac1a3d70 con 0x7f63ac105d00 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b269d640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac10bc40 0x7f63ac1a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.212+0000 7f63b1e9c640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ac105d00 0x7f63ac19c280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f63a800e9f0 tx=0x7f63a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.213+0000 7f63a2ffd640 1 -- 172.21.15.27:0/3462120031 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a800ccc0 con 0x7f63ac105d00 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.213+0000 7f63a2ffd640 1 -- 172.21.15.27:0/3462120031 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a80035f0 con 0x7f63ac105d00 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.213+0000 7f63b4100640 1 -- 172.21.15.27:0/3462120031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ac1a4060 con 0x7f63ac105d00 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.213+0000 7f63a2ffd640 1 -- 172.21.15.27:0/3462120031 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a800ce60 con 0x7f63ac105d00 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.213+0000 7f63b4100640 1 -- 172.21.15.27:0/3462120031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ac1a45b0 con 0x7f63ac105d00 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.214+0000 7f63a0ff9640 1 -- 172.21.15.27:0/3462120031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6394005290 con 0x7f63ac105d00 2024-04-25T13:43:36.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.216+0000 7f63a2ffd640 1 -- 172.21.15.27:0/3462120031 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f63a8003aa0 con 0x7f63ac105d00 2024-04-25T13:43:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.217+0000 7f63a2ffd640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63980afaf0 0x7f63980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.218+0000 7f63a2ffd640 1 -- 172.21.15.27:0/3462120031 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f63a80df120 con 0x7f63ac105d00 2024-04-25T13:43:36.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.218+0000 7f63b169b640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63980afaf0 0x7f63980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.218+0000 7f63b169b640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63980afaf0 0x7f63980b2000 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f639c004770 tx=0x7f639c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.220+0000 7f63a2ffd640 1 -- 172.21.15.27:0/3462120031 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63a80a7cd0 con 0x7f63ac105d00 2024-04-25T13:43:36.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.592+0000 7f63a0ff9640 1 -- 172.21.15.27:0/3462120031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6394005580 con 0x7f63ac105d00 2024-04-25T13:43:36.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.593+0000 7f63a2ffd640 1 -- 172.21.15.27:0/3462120031 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f63a80a75d0 con 0x7f63ac105d00 2024-04-25T13:43:36.594 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:43:36.594 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:43:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.596+0000 7f63a0ff9640 1 -- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63980afaf0 msgr2=0x7f63980b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63980afaf0 0x7f63980b2000 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f639c004770 tx=0x7f639c000bc0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 -- 172.21.15.27:0/3462120031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ac105d00 msgr2=0x7f63ac19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ac105d00 0x7f63ac19c280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f63a800e9f0 tx=0x7f63a800eeb0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 -- 172.21.15.27:0/3462120031 shutdown_connections 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63980afaf0 0x7f63980b2000 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ac10bc40 0x7f63ac1a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ac106f40 0x7f63ac19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 --2- 172.21.15.27:0/3462120031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ac105d00 0x7f63ac19c280 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 -- 172.21.15.27:0/3462120031 >> 172.21.15.27:0/3462120031 conn(0x7f63ac100f10 msgr2=0x7f63ac103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 -- 172.21.15.27:0/3462120031 shutdown_connections 2024-04-25T13:43:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.597+0000 7f63a0ff9640 1 -- 172.21.15.27:0/3462120031 wait complete. 2024-04-25T13:43:36.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:43:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.772+0000 7f1a93d4d640 1 -- 172.21.15.27:0/418059356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a8c103c20 msgr2=0x7f1a8c104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.772+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/418059356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a8c103c20 0x7f1a8c104030 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009ca0 tx=0x7f1a8402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.772+0000 7f1a93d4d640 1 -- 172.21.15.27:0/418059356 shutdown_connections 2024-04-25T13:43:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.772+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/418059356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8c108920 0x7f1a8c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.772+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/418059356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a8c103c20 0x7f1a8c104030 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.772+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/418059356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c1029e0 0x7f1a8c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.773+0000 7f1a93d4d640 1 -- 172.21.15.27:0/418059356 >> 172.21.15.27:0/418059356 conn(0x7f1a8c0fdbd0 msgr2=0x7f1a8c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:36.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.773+0000 7f1a93d4d640 1 -- 172.21.15.27:0/418059356 shutdown_connections 2024-04-25T13:43:36.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.773+0000 7f1a93d4d640 1 -- 172.21.15.27:0/418059356 wait complete. 2024-04-25T13:43:36.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.774+0000 7f1a93d4d640 1 Processor -- start 2024-04-25T13:43:36.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.774+0000 7f1a93d4d640 1 -- start start 2024-04-25T13:43:36.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a93d4d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8c1029e0 0x7f1a8c06a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a93d4d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a8c103c20 0x7f1a8c073cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a93d4d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c108920 0x7f1a8c074200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a93d4d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10e500 con 0x7f1a8c103c20 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a93d4d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10e380 con 0x7f1a8c1029e0 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a93d4d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10e680 con 0x7f1a8c108920 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a91ae9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8c1029e0 0x7f1a8c06a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a91ae9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8c1029e0 0x7f1a8c06a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43828/0 (socket says 172.21.15.27:43828) 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a91ae9640 1 -- 172.21.15.27:0/3455304386 learned_addr learned my addr 172.21.15.27:0/3455304386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a912e8640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a8c103c20 0x7f1a8c073cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a922ea640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c108920 0x7f1a8c074200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a91ae9640 1 -- 172.21.15.27:0/3455304386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c108920 msgr2=0x7f1a8c074200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a91ae9640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c108920 0x7f1a8c074200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a91ae9640 1 -- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a8c103c20 msgr2=0x7f1a8c073cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a91ae9640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a8c103c20 0x7f1a8c073cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.775+0000 7f1a91ae9640 1 -- 172.21.15.27:0/3455304386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8c074930 con 0x7f1a8c1029e0 2024-04-25T13:43:36.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.776+0000 7f1a91ae9640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8c1029e0 0x7f1a8c06a800 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800e9f0 tx=0x7f1a8800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:36.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.776+0000 7f1a82ffd640 1 -- 172.21.15.27:0/3455304386 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8800ccc0 con 0x7f1a8c1029e0 2024-04-25T13:43:36.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.776+0000 7f1a93d4d640 1 -- 172.21.15.27:0/3455304386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8c1aac20 con 0x7f1a8c1029e0 2024-04-25T13:43:36.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.776+0000 7f1a82ffd640 1 -- 172.21.15.27:0/3455304386 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a880035f0 con 0x7f1a8c1029e0 2024-04-25T13:43:36.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.776+0000 7f1a82ffd640 1 -- 172.21.15.27:0/3455304386 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8800ce60 con 0x7f1a8c1029e0 2024-04-25T13:43:36.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.776+0000 7f1a93d4d640 1 -- 172.21.15.27:0/3455304386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8c1ab090 con 0x7f1a8c1029e0 2024-04-25T13:43:36.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a9ac03740 1 Processor -- start 2024-04-25T13:43:36.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a9ac03740 1 -- start start 2024-04-25T13:43:36.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a82ffd640 1 -- 172.21.15.27:0/3455304386 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1a88010690 con 0x7f1a8c1029e0 2024-04-25T13:43:36.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56356150e390 0x5635618bb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618bdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635614ed6b0 0x5635618c0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563561814d80 con 0x5635614ed6b0 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5635617cea20 con 0x56356150e390 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563561810d40 con 0x5635614e54d0 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a922ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56356150e390 0x5635618bb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.781+0000 7f1a922ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56356150e390 0x5635618bb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43842/0 (socket says 172.21.15.27:43842) 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a922ea640 1 -- 172.21.15.27:0/1944365272 learned_addr learned my addr 172.21.15.27:0/1944365272 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a91ae9640 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618bdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a912e8640 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635614ed6b0 0x5635618c0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a922ea640 1 -- 172.21.15.27:0/1944365272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 msgr2=0x5635618bdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618bdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a922ea640 1 -- 172.21.15.27:0/1944365272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635614ed6b0 msgr2=0x5635618c0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635614ed6b0 0x5635618c0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a922ea640 1 -- 172.21.15.27:0/1944365272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635617d9c30 con 0x56356150e390 2024-04-25T13:43:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a912e8640 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635614ed6b0 0x5635618c0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a82ffd640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a780afaf0 0x7f1a780b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56356150e390 0x5635618bb6e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009ca0 tx=0x7f1a7c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=373895e76b11875c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a91ae9640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a780afaf0 0x7f1a780b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a77fff640 1 -- 172.21.15.27:0/1944365272 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c01e070 con 0x56356150e390 2024-04-25T13:43:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a77fff640 1 -- 172.21.15.27:0/1944365272 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a7c009e50 con 0x56356150e390 2024-04-25T13:43:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a77fff640 1 -- 172.21.15.27:0/1944365272 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c00ea50 con 0x56356150e390 2024-04-25T13:43:36.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.782+0000 7f1a82ffd640 1 -- 172.21.15.27:0/3455304386 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a880def50 con 0x7f1a8c1029e0 2024-04-25T13:43:36.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.783+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1944365272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56356150e390 msgr2=0x5635618bb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a91ae9640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a780afaf0 0x7f1a780b2000 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800e9c0 tx=0x7f1a88090040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56356150e390 0x5635618bb6e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009ca0 tx=0x7f1a7c011810 comp rx=0 tx=0).stop 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1944365272 shutdown_connections 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635614ed6b0 0x5635618c0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618bdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1944365272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56356150e390 0x5635618bb6e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1944365272 >> 172.21.15.27:0/1944365272 conn(0x5635615024e0 msgr2=0x5635618abfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1944365272 shutdown_connections 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.784+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1944365272 wait complete. 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.785+0000 7f1a9ac03740 1 -- start start 2024-04-25T13:43:36.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.785+0000 7f1a9ac03740 1 Processor -- start 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.785+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614e54d0 0x5635618c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614ed6b0 0x5635618c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5635618c0f60 con 0x56356150e390 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614e54d0 0x5635618c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56356179bdc0 con 0x5635614e54d0 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5635618c10e0 con 0x5635614ed6b0 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614e54d0 0x5635618c2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43846/0 (socket says 172.21.15.27:43846) 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 -- 172.21.15.27:0/2783881167 learned_addr learned my addr 172.21.15.27:0/2783881167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a912e8640 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614ed6b0 0x5635618c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a9ac03740 10 client.0 _ll_register_callbacks cb 0x5635617b7600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:36.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 -- 172.21.15.27:0/2783881167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614ed6b0 msgr2=0x5635618c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:36.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614ed6b0 0x5635618c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 -- 172.21.15.27:0/2783881167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 msgr2=0x5635618ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:36.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a922ea640 1 -- 172.21.15.27:0/2783881167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635615263f0 con 0x5635614e54d0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.786+0000 7f1a912e8640 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614ed6b0 0x5635618c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.787+0000 7f1a922ea640 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614e54d0 0x5635618c2d40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00e4f0 tx=0x7f1a7c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51677075f6743e35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.787+0000 7f1a757fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.787+0000 7f1a757fa640 1 -- 172.21.15.27:0/2783881167 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c01e070 con 0x5635614e54d0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.787+0000 7f1a9ac03740 1 -- 172.21.15.27:0/2783881167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a7c009e50 con 0x5635614e54d0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.787+0000 7f1a757fa640 1 -- 172.21.15.27:0/2783881167 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a7c004030 con 0x5635614e54d0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a757fa640 1 -- 172.21.15.27:0/2783881167 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1a7c00ee50 con 0x5635614e54d0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a9ac03740 10 client.25577 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a9ac03740 1 -- 172.21.15.27:0/2783881167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5635615f07d0 con 0x5635614e54d0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a757fa640 1 -- 172.21.15.27:0/2783881167 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a7c009e50 con 0x5635614e54d0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a757fa640 1 -- 172.21.15.27:0/2783881167 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1a7c024030 con 0x5635614e54d0 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a9ac03740 10 client.25577 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a9ac03740 10 client.25577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:36.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a9ac03740 20 client.25577 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:36.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635614ccf60 0x5635618d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a9ac03740 4 client.25577 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:36.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.788+0000 7f1a9ac03740 1 -- 172.21.15.27:0/2783881167 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563561822990 con 0x5635614ccf60 2024-04-25T13:43:36.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.789+0000 7f1a912e8640 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635614ccf60 0x5635618d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.789+0000 7f1a757fa640 10 client.25577 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:36.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.789+0000 7f1a912e8640 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635614ccf60 0x5635618d0700 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68870f2ad3ced53f server_cookie=ff9f8788867ce5f in_seq=0 out_seq=0 2024-04-25T13:43:36.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a757fa640 1 -- 172.21.15.27:0/2783881167 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563561822990 con 0x5635614ccf60 2024-04-25T13:43:36.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a757fa640 10 client.25577 handle_command_reply: tid=0 2024-04-25T13:43:36.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 client.25577 shutdown 2024-04-25T13:43:36.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 -- 172.21.15.27:0/2783881167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614e54d0 msgr2=0x5635618c2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614e54d0 0x5635618c2d40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00e4f0 tx=0x7f1a7c005850 comp rx=0 tx=0).stop 2024-04-25T13:43:36.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 -- 172.21.15.27:0/2783881167 shutdown_connections 2024-04-25T13:43:36.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635614ccf60 0x5635618d0700 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614ed6b0 0x5635618c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/2783881167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614e54d0 0x5635618c2d40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a757fa640 0 client.25577 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:36.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 -- 172.21.15.27:0/2783881167 >> 172.21.15.27:0/2783881167 conn(0x5635615024e0 msgr2=0x5635618aa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:36.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 -- 172.21.15.27:0/2783881167 shutdown_connections 2024-04-25T13:43:36.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 1 -- 172.21.15.27:0/2783881167 wait complete. 2024-04-25T13:43:36.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.790+0000 7f1a9ac03740 20 client.25577 trim_cache size 0 max 16384 2024-04-25T13:43:36.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.809+0000 7f1a9ac03740 1 Processor -- start 2024-04-25T13:43:36.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.809+0000 7f1a9ac03740 1 -- start start 2024-04-25T13:43:36.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.809+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618ab960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.809+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618abea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.809+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618be640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.809+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1af7f0 con 0x56356150e390 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.809+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1b2070 con 0x5635614ed6b0 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.809+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1afe60 con 0x5635614e54d0 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a912e8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a922ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618be640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a912e8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618abea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43858/0 (socket says 172.21.15.27:43858) 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a912e8640 1 -- 172.21.15.27:0/1786838591 learned_addr learned my addr 172.21.15.27:0/1786838591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1786838591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a922ea640 1 -- 172.21.15.27:0/1786838591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 msgr2=0x5635618ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1786838591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618ab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a922ea640 1 -- 172.21.15.27:0/1786838591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 msgr2=0x5635618abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1786838591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a922ea640 1 -- 172.21.15.27:0/1786838591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635614cd190 con 0x56356150e390 2024-04-25T13:43:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1786838591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618be640 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c003640 tx=0x7f1a7c01b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83390a529ec7857e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a777fe640 1 -- 172.21.15.27:0/1786838591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c039070 con 0x56356150e390 2024-04-25T13:43:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a777fe640 1 -- 172.21.15.27:0/1786838591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a7c01f040 con 0x56356150e390 2024-04-25T13:43:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.810+0000 7f1a777fe640 1 -- 172.21.15.27:0/1786838591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c01e070 con 0x56356150e390 2024-04-25T13:43:36.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.811+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1786838591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 msgr2=0x5635618be640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.812+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1786838591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618be640 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c003640 tx=0x7f1a7c01b040 comp rx=0 tx=0).stop 2024-04-25T13:43:36.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.812+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1786838591 shutdown_connections 2024-04-25T13:43:36.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.812+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1786838591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618be640 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.812+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1786838591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.812+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1786838591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.812+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1786838591 >> 172.21.15.27:0/1786838591 conn(0x5635615024e0 msgr2=0x5635618c2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.812+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1786838591 shutdown_connections 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.813+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1786838591 wait complete. 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.813+0000 7f1a9ac03740 1 -- start start 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.813+0000 7f1a9ac03740 1 Processor -- start 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.813+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.813+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618c5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a9ac03740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1b0530 con 0x56356150e390 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1b19a0 con 0x5635614ed6b0 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a9ac03740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1b0c00 con 0x5635614e54d0 2024-04-25T13:43:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a9ac03740 10 client.0 _ll_register_callbacks cb 0x7f1a7c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:36.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a922ea640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43864/0 (socket says 172.21.15.27:43864) 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 -- 172.21.15.27:0/1614614699 learned_addr learned my addr 172.21.15.27:0/1614614699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 -- 172.21.15.27:0/1614614699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 msgr2=0x5635618c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 -- 172.21.15.27:0/1614614699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 msgr2=0x5635618c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 -- 172.21.15.27:0/1614614699 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635618c8550 con 0x5635614ed6b0 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a912e8640 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618c5050 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009620 tx=0x7f1a84031930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c16275b448b478e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a757fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a757fa640 1 -- 172.21.15.27:0/1614614699 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8404e070 con 0x5635614ed6b0 2024-04-25T13:43:36.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1614614699 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a7c01f040 con 0x5635614ed6b0 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.814+0000 7f1a757fa640 1 -- 172.21.15.27:0/1614614699 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a84003d80 con 0x5635614ed6b0 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a757fa640 1 -- 172.21.15.27:0/1614614699 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1a84031d30 con 0x5635614ed6b0 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a9ac03740 10 client.25583 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1614614699 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5635618c9840 con 0x5635614ed6b0 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a757fa640 1 -- 172.21.15.27:0/1614614699 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a7c01f040 con 0x5635614ed6b0 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a757fa640 1 -- 172.21.15.27:0/1614614699 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1a84030990 con 0x5635614ed6b0 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a9ac03740 10 client.25583 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a9ac03740 10 client.25583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a9ac03740 20 client.25583 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635614a23a0 0x56356193aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a9ac03740 4 client.25583 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1614614699 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56356181c800 con 0x5635614a23a0 2024-04-25T13:43:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.815+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635614a23a0 0x56356193aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.816+0000 7f1a922ea640 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635614a23a0 0x56356193aef0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a1cd068cafa55ca server_cookie=d3da2d4ee243efa2 in_seq=0 out_seq=0 2024-04-25T13:43:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.816+0000 7f1a757fa640 10 client.25583 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.816+0000 7f1a757fa640 1 -- 172.21.15.27:0/1614614699 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56356181c800 con 0x5635614a23a0 2024-04-25T13:43:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.816+0000 7f1a757fa640 10 client.25583 handle_command_reply: tid=0 2024-04-25T13:43:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.816+0000 7f1a9ac03740 1 client.25583 shutdown 2024-04-25T13:43:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1614614699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 msgr2=0x5635618c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618c5050 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009620 tx=0x7f1a84031930 comp rx=0 tx=0).stop 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1614614699 shutdown_connections 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635614a23a0 0x56356193aef0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56356150e390 0x5635618c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635614ed6b0 0x5635618c5050 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a9ac03740 1 --2- 172.21.15.27:0/1614614699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635614e54d0 0x5635618c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1614614699 >> 172.21.15.27:0/1614614699 conn(0x5635615024e0 msgr2=0x5635618b84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.817+0000 7f1a757fa640 0 client.25583 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.818+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1614614699 shutdown_connections 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.818+0000 7f1a9ac03740 1 -- 172.21.15.27:0/1614614699 wait complete. 2024-04-25T13:43:36.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a9ac03740 20 client.25583 trim_cache size 0 max 16384 2024-04-25T13:43:36.823 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:43:36.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 -- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a780afaf0 msgr2=0x7f1a780b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a780afaf0 0x7f1a780b2000 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800e9c0 tx=0x7f1a88090040 comp rx=0 tx=0).stop 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 -- 172.21.15.27:0/3455304386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8c1029e0 msgr2=0x7f1a8c06a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8c1029e0 0x7f1a8c06a800 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800e9f0 tx=0x7f1a8800eeb0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 -- 172.21.15.27:0/3455304386 shutdown_connections 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a780afaf0 0x7f1a780b2000 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c108920 0x7f1a8c074200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a8c103c20 0x7f1a8c073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 --2- 172.21.15.27:0/3455304386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a8c1029e0 0x7f1a8c06a800 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.819+0000 7f1a93d4d640 1 -- 172.21.15.27:0/3455304386 >> 172.21.15.27:0/3455304386 conn(0x7f1a8c0fdbd0 msgr2=0x7f1a8c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.820+0000 7f1a93d4d640 1 -- 172.21.15.27:0/3455304386 shutdown_connections 2024-04-25T13:43:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:36.820+0000 7f1a93d4d640 1 -- 172.21.15.27:0/3455304386 wait complete. 2024-04-25T13:43:37.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:37 smithi122 ceph-mon[32843]: pgmap v431: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 9.6 MiB/s wr, 844 op/s 2024-04-25T13:43:37.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:37 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3462120031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:43:37.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:37 smithi045 ceph-mon[32731]: pgmap v431: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 9.6 MiB/s wr, 844 op/s 2024-04-25T13:43:37.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:37 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3462120031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:43:37.833 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:43:37.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:43:37.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:43:37.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:37 smithi027 ceph-mon[29140]: pgmap v431: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 9.6 MiB/s wr, 844 op/s 2024-04-25T13:43:37.862 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:37 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3462120031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:43:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 150.0 s, 3579.8 tps, lat 28.080 ms stddev 30.764 2024-04-25T13:43:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:37.999+0000 7fef78830640 1 -- 172.21.15.27:0/1558310083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef74118a90 msgr2=0x7fef7411aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:37.999+0000 7fef78830640 1 --2- 172.21.15.27:0/1558310083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef74118a90 0x7fef7411aeb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fef68009900 tx=0x7fef6802a420 comp rx=0 tx=0).stop 2024-04-25T13:43:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.000+0000 7fef78830640 1 -- 172.21.15.27:0/1558310083 shutdown_connections 2024-04-25T13:43:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.000+0000 7fef78830640 1 --2- 172.21.15.27:0/1558310083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef74118a90 0x7fef7411aeb0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.000+0000 7fef78830640 1 --2- 172.21.15.27:0/1558310083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef7406d1b0 0x7fef74118550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.000+0000 7fef78830640 1 --2- 172.21.15.27:0/1558310083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7406c860 0x7fef7406cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.000+0000 7fef78830640 1 -- 172.21.15.27:0/1558310083 >> 172.21.15.27:0/1558310083 conn(0x7fef74072700 msgr2=0x7fef74109ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.001+0000 7fef78830640 1 -- 172.21.15.27:0/1558310083 shutdown_connections 2024-04-25T13:43:38.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.002+0000 7fef78830640 1 -- 172.21.15.27:0/1558310083 wait complete. 2024-04-25T13:43:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.003+0000 7fef78830640 1 Processor -- start 2024-04-25T13:43:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.003+0000 7fef78830640 1 -- start start 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.003+0000 7fef78830640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7406c860 0x7fef741a9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.003+0000 7fef78830640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef7406d1b0 0x7fef741a9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.003+0000 7fef78830640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef74118a90 0x7fef741b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.003+0000 7fef78830640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fef7411e490 con 0x7fef7406d1b0 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.003+0000 7fef78830640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef7411e310 con 0x7fef7406c860 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.003+0000 7fef78830640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fef7411e610 con 0x7fef74118a90 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.004+0000 7fef7259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7406c860 0x7fef741a9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.004+0000 7fef7259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7406c860 0x7fef741a9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43896/0 (socket says 172.21.15.27:43896) 2024-04-25T13:43:38.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.004+0000 7fef7259c640 1 -- 172.21.15.27:0/1790514585 learned_addr learned my addr 172.21.15.27:0/1790514585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.005+0000 7fef71d9b640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef7406d1b0 0x7fef741a9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.005+0000 7fef72d9d640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef74118a90 0x7fef741b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.006+0000 7fef72d9d640 1 -- 172.21.15.27:0/1790514585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7406c860 msgr2=0x7fef741a9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.006+0000 7fef72d9d640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7406c860 0x7fef741a9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.006+0000 7fef72d9d640 1 -- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef7406d1b0 msgr2=0x7fef741a9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.006+0000 7fef72d9d640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef7406d1b0 0x7fef741a9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.006+0000 7fef72d9d640 1 -- 172.21.15.27:0/1790514585 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef741b0c40 con 0x7fef74118a90 2024-04-25T13:43:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.007+0000 7fef72d9d640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef74118a90 0x7fef741b05d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef68009850 tx=0x7fef68003b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.008+0000 7fef637fe640 1 -- 172.21.15.27:0/1790514585 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6803f070 con 0x7fef74118a90 2024-04-25T13:43:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.009+0000 7fef637fe640 1 -- 172.21.15.27:0/1790514585 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef68031c50 con 0x7fef74118a90 2024-04-25T13:43:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.009+0000 7fef637fe640 1 -- 172.21.15.27:0/1790514585 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68035030 con 0x7fef74118a90 2024-04-25T13:43:38.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.009+0000 7fef78830640 1 -- 172.21.15.27:0/1790514585 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef741b0ed0 con 0x7fef74118a90 2024-04-25T13:43:38.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.009+0000 7fef78830640 1 -- 172.21.15.27:0/1790514585 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef741b1360 con 0x7fef74118a90 2024-04-25T13:43:38.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.012+0000 7fef7f7b6740 1 Processor -- start 2024-04-25T13:43:38.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.012+0000 7fef637fe640 1 -- 172.21.15.27:0/1790514585 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fef68042070 con 0x7fef74118a90 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7f7b6740 1 -- start start 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59cb3410 0x55dd5a0544a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c8a550 0x55dd5a054a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a05b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dd59fb9b00 con 0x55dd59c92730 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55dd59f55920 con 0x55dd59cb3410 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55dd59fb5ac0 con 0x55dd59c8a550 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c8a550 0x55dd5a054a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c8a550 0x55dd5a054a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40394/0 (socket says 172.21.15.27:40394) 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef7259c640 1 -- 172.21.15.27:0/2501533294 learned_addr learned my addr 172.21.15.27:0/2501533294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59cb3410 0x55dd5a0544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a05b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.013+0000 7fef637fe640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef5c0afc80 0x7fef5c0b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef7259c640 1 -- 172.21.15.27:0/2501533294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59cb3410 msgr2=0x55dd5a0544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef7259c640 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59cb3410 0x55dd5a0544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef7259c640 1 -- 172.21.15.27:0/2501533294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 msgr2=0x55dd5a05b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef7259c640 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a05b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef7259c640 1 -- 172.21.15.27:0/2501533294 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd59d937e0 con 0x55dd59c8a550 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a05b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef72d9d640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef5c0afc80 0x7fef5c0b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef637fe640 1 -- 172.21.15.27:0/1790514585 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fef680ff2a0 con 0x7fef74118a90 2024-04-25T13:43:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.014+0000 7fef7259c640 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c8a550 0x55dd5a054a20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fef64009af0 tx=0x7fef6400e1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90a759b073479e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.015+0000 7fef72d9d640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef5c0afc80 0x7fef5c0b2190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fef68088c40 tx=0x7fef6808dcc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.015+0000 7fef60ff9640 1 -- 172.21.15.27:0/2501533294 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64017070 con 0x55dd59c8a550 2024-04-25T13:43:38.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.015+0000 7fef60ff9640 1 -- 172.21.15.27:0/2501533294 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6400ec10 con 0x55dd59c8a550 2024-04-25T13:43:38.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.015+0000 7fef60ff9640 1 -- 172.21.15.27:0/2501533294 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64012630 con 0x55dd59c8a550 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2501533294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c8a550 msgr2=0x55dd5a054a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c8a550 0x55dd5a054a20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fef64009af0 tx=0x7fef6400e1d0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2501533294 shutdown_connections 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a05b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c8a550 0x55dd5a054a20 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fef64009af0 tx=0x7fef6400e1d0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2501533294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59cb3410 0x55dd5a0544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2501533294 >> 172.21.15.27:0/2501533294 conn(0x55dd59ca7560 msgr2=0x55dd5a04fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2501533294 shutdown_connections 2024-04-25T13:43:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.017+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2501533294 wait complete. 2024-04-25T13:43:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 1 -- start start 2024-04-25T13:43:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 1 Processor -- start 2024-04-25T13:43:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c92730 0x55dd5a05e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 0x55dd5a05e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c721f0 0x55dd5a065950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dd5a05c370 con 0x55dd59cb3410 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55dd59f535d0 con 0x55dd59c92730 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55dd5a05c4f0 con 0x55dd59c721f0 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7f7b6740 10 client.0 _ll_register_callbacks cb 0x55dd59f41080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef71d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c92730 0x55dd5a05e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef71d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c92730 0x55dd5a05e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43924/0 (socket says 172.21.15.27:43924) 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef71d9b640 1 -- 172.21.15.27:0/2802971954 learned_addr learned my addr 172.21.15.27:0/2802971954 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c721f0 0x55dd5a065950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef7259c640 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 0x55dd5a05e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.018+0000 7fef71d9b640 1 -- 172.21.15.27:0/2802971954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c721f0 msgr2=0x55dd5a065950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c721f0 0x55dd5a065950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef71d9b640 1 -- 172.21.15.27:0/2802971954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 msgr2=0x55dd5a05e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 0x55dd5a05e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef71d9b640 1 -- 172.21.15.27:0/2802971954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd59df3f70 con 0x55dd59c92730 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c92730 0x55dd5a05e4b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fef6c013fc0 tx=0x7fef6c009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf014fb3e22cf725 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef59ffb640 1 -- 172.21.15.27:0/2802971954 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c013250 con 0x55dd59c92730 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef59ffb640 1 -- 172.21.15.27:0/2802971954 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fef6c004b20 con 0x55dd59c92730 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2802971954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6400ec10 con 0x55dd59c92730 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef59ffb640 1 -- 172.21.15.27:0/2802971954 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fef6c004d20 con 0x55dd59c92730 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef7f7b6740 10 client.25592 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2802971954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55dd59c8b170 con 0x55dd59c92730 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.019+0000 7fef59ffb640 1 -- 172.21.15.27:0/2802971954 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6400ec10 con 0x55dd59c92730 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef59ffb640 1 -- 172.21.15.27:0/2802971954 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fef6c00f040 con 0x55dd59c92730 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef7f7b6740 10 client.25592 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef7f7b6740 10 client.25592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:38.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef7f7b6740 20 client.25592 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dd59c47630 0x55dd5a06bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef7f7b6740 4 client.25592 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2802971954 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd59fc7710 con 0x55dd59c47630 2024-04-25T13:43:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef7259c640 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dd59c47630 0x55dd5a06bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef59ffb640 10 client.25592 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.020+0000 7fef7259c640 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dd59c47630 0x55dd5a06bef0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af05b751847872c7 server_cookie=b1847e301ce1cd27 in_seq=0 out_seq=0 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef59ffb640 1 -- 172.21.15.27:0/2802971954 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dd59fc7710 con 0x55dd59c47630 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef59ffb640 10 client.25592 handle_command_reply: tid=0 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef7f7b6740 1 client.25592 shutdown 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2802971954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c92730 msgr2=0x55dd5a05e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c92730 0x55dd5a05e4b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fef6c013fc0 tx=0x7fef6c009ee0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2802971954 shutdown_connections 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dd59c47630 0x55dd5a06bef0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c721f0 0x55dd5a065950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 0x55dd5a05e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2802971954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c92730 0x55dd5a05e4b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.021+0000 7fef59ffb640 0 client.25592 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.022+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2802971954 >> 172.21.15.27:0/2802971954 conn(0x55dd59ca7560 msgr2=0x55dd5a04eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.022+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2802971954 shutdown_connections 2024-04-25T13:43:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.022+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2802971954 wait complete. 2024-04-25T13:43:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.022+0000 7fef7f7b6740 20 client.25592 trim_cache size 0 max 16384 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.055+0000 7fef7f7b6740 1 Processor -- start 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.055+0000 7fef7f7b6740 1 -- start start 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.055+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a05e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c92730 0x55dd5a065910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 0x55dd5a065e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fef741b80f0 con 0x55dd59cb3410 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef741ba970 con 0x55dd59c721f0 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fef741b8760 con 0x55dd59c92730 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef7259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c92730 0x55dd5a065910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef7259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c92730 0x55dd5a065910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40418/0 (socket says 172.21.15.27:40418) 2024-04-25T13:43:38.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef7259c640 1 -- 172.21.15.27:0/2349676351 learned_addr learned my addr 172.21.15.27:0/2349676351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a05e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 0x55dd5a065e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef71d9b640 1 -- 172.21.15.27:0/2349676351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c92730 msgr2=0x55dd5a065910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c92730 0x55dd5a065910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef71d9b640 1 -- 172.21.15.27:0/2349676351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 msgr2=0x55dd5a065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 0x55dd5a065e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.056+0000 7fef71d9b640 1 -- 172.21.15.27:0/2349676351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd5a05c8c0 con 0x55dd59c721f0 2024-04-25T13:43:38.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.057+0000 7fef7259c640 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c92730 0x55dd5a065910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:38.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.057+0000 7fef71d9b640 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a05e790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fef6c011010 tx=0x7fef6c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2ccdb2a8361a36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.057+0000 7fef5bfff640 1 -- 172.21.15.27:0/2349676351 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c013310 con 0x55dd59c721f0 2024-04-25T13:43:38.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.057+0000 7fef5bfff640 1 -- 172.21.15.27:0/2349676351 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6c004b00 con 0x55dd59c721f0 2024-04-25T13:43:38.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.057+0000 7fef5bfff640 1 -- 172.21.15.27:0/2349676351 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c021c10 con 0x55dd59c721f0 2024-04-25T13:43:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2349676351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 msgr2=0x55dd5a05e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a05e790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fef6c011010 tx=0x7fef6c000f40 comp rx=0 tx=0).stop 2024-04-25T13:43:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2349676351 shutdown_connections 2024-04-25T13:43:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59cb3410 0x55dd5a065e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59c92730 0x55dd5a065910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/2349676351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a05e790 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2349676351 >> 172.21.15.27:0/2349676351 conn(0x55dd59ca7560 msgr2=0x55dd5a05e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2349676351 shutdown_connections 2024-04-25T13:43:38.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 -- 172.21.15.27:0/2349676351 wait complete. 2024-04-25T13:43:38.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 -- start start 2024-04-25T13:43:38.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.059+0000 7fef7f7b6740 1 Processor -- start 2024-04-25T13:43:38.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.060+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a063920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.060+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a063e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.060+0000 7fef7f7b6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59cb3410 0x55dd5a064d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.060+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fef741b8e30 con 0x55dd59c92730 2024-04-25T13:43:38.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.060+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef741ba2a0 con 0x55dd59c721f0 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.060+0000 7fef7f7b6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fef741b9500 con 0x55dd59cb3410 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.060+0000 7fef7f7b6740 10 client.0 _ll_register_callbacks cb 0x7fef6c0103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a063e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a063e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47700/0 (socket says 172.21.15.27:47700) 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 -- 172.21.15.27:0/4165573762 learned_addr learned my addr 172.21.15.27:0/4165573762 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef71d9b640 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59cb3410 0x55dd5a064d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef71d9b640 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a063920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 -- 172.21.15.27:0/4165573762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59cb3410 msgr2=0x55dd5a064d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59cb3410 0x55dd5a064d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 -- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 msgr2=0x55dd5a063920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a063920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 -- 172.21.15.27:0/4165573762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd5a065350 con 0x55dd59c92730 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef71d9b640 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a063920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:43:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef7259c640 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a063e60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fef64009af0 tx=0x7fef6400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e9a94dd130514d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.061+0000 7fef59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:43:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.062+0000 7fef59ffb640 1 -- 172.21.15.27:0/4165573762 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64017040 con 0x55dd59c92730 2024-04-25T13:43:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.062+0000 7fef59ffb640 1 -- 172.21.15.27:0/4165573762 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fef64013670 con 0x55dd59c92730 2024-04-25T13:43:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.062+0000 7fef7f7b6740 1 -- 172.21.15.27:0/4165573762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6c004b00 con 0x55dd59c92730 2024-04-25T13:43:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.062+0000 7fef59ffb640 1 -- 172.21.15.27:0/4165573762 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fef6403a070 con 0x55dd59c92730 2024-04-25T13:43:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.062+0000 7fef7f7b6740 10 client.15879 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.062+0000 7fef7f7b6740 1 -- 172.21.15.27:0/4165573762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55dd5a0d43e0 con 0x55dd59c92730 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.063+0000 7fef59ffb640 1 -- 172.21.15.27:0/4165573762 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6c004b00 con 0x55dd59c92730 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.063+0000 7fef59ffb640 1 -- 172.21.15.27:0/4165573762 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fef64026040 con 0x55dd59c92730 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.063+0000 7fef7f7b6740 10 client.15879 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.063+0000 7fef7f7b6740 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.063+0000 7fef7f7b6740 20 client.15879 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.063+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dd5a0d4a00 0x55dd5a0d6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.063+0000 7fef7f7b6740 4 client.15879 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:43:38.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.063+0000 7fef7f7b6740 1 -- 172.21.15.27:0/4165573762 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dd59fc1580 con 0x55dd5a0d4a00 2024-04-25T13:43:38.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.064+0000 7fef71d9b640 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dd5a0d4a00 0x55dd5a0d6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.065+0000 7fef71d9b640 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dd5a0d4a00 0x55dd5a0d6e20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8acf8806529abda9 server_cookie=f20698a4d6085917 in_seq=0 out_seq=0 2024-04-25T13:43:38.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.065+0000 7fef59ffb640 10 client.15879 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:38.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.066+0000 7fef59ffb640 1 -- 172.21.15.27:0/4165573762 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dd59fc1580 con 0x55dd5a0d4a00 2024-04-25T13:43:38.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.066+0000 7fef59ffb640 10 client.15879 handle_command_reply: tid=0 2024-04-25T13:43:38.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 client.15879 shutdown 2024-04-25T13:43:38.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 -- 172.21.15.27:0/4165573762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 msgr2=0x55dd5a063e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a063e60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fef64009af0 tx=0x7fef6400ee40 comp rx=0 tx=0).stop 2024-04-25T13:43:38.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 -- 172.21.15.27:0/4165573762 shutdown_connections 2024-04-25T13:43:38.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dd5a0d4a00 0x55dd5a0d6e20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dd59cb3410 0x55dd5a064d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dd59c92730 0x55dd5a063e60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 --2- 172.21.15.27:0/4165573762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd59c721f0 0x55dd5a063920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef59ffb640 0 client.15879 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 -- 172.21.15.27:0/4165573762 >> 172.21.15.27:0/4165573762 conn(0x55dd59ca7560 msgr2=0x55dd5a05e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 -- 172.21.15.27:0/4165573762 shutdown_connections 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.067+0000 7fef7f7b6740 1 -- 172.21.15.27:0/4165573762 wait complete. 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef7f7b6740 20 client.15879 trim_cache size 0 max 16384 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:43:38.069 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "d790558d-d685-461c-b84e-b8c52788c2e3", 2024-04-25T13:43:38.070 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:43:38.070 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 -- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef5c0afc80 msgr2=0x7fef5c0b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef5c0afc80 0x7fef5c0b2190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fef68088c40 tx=0x7fef6808dcc0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 -- 172.21.15.27:0/1790514585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef74118a90 msgr2=0x7fef741b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef74118a90 0x7fef741b05d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef68009850 tx=0x7fef68003b00 comp rx=0 tx=0).stop 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 -- 172.21.15.27:0/1790514585 shutdown_connections 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fef5c0afc80 0x7fef5c0b2190 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef74118a90 0x7fef741b05d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef7406d1b0 0x7fef741a9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 --2- 172.21.15.27:0/1790514585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef7406c860 0x7fef741a9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 -- 172.21.15.27:0/1790514585 >> 172.21.15.27:0/1790514585 conn(0x7fef74072700 msgr2=0x7fef7410b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 -- 172.21.15.27:0/1790514585 shutdown_connections 2024-04-25T13:43:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.068+0000 7fef78830640 1 -- 172.21.15.27:0/1790514585 wait complete. 2024-04-25T13:43:38.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:43:38.084 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:43:38.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.304+0000 7f9ca15e0640 1 -- 172.21.15.27:0/3916666047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c9c10b330 msgr2=0x7f9c9c10b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.304+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/3916666047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c9c10b330 0x7f9c9c10b740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009860 tx=0x7f9c8c02a340 comp rx=0 tx=0).stop 2024-04-25T13:43:38.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.305+0000 7f9ca15e0640 1 -- 172.21.15.27:0/3916666047 shutdown_connections 2024-04-25T13:43:38.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.305+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/3916666047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9c110030 0x7f9c9c112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.305+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/3916666047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c9c10b330 0x7f9c9c10b740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.305+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/3916666047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c9c10a3d0 0x7f9c9c10adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.305+0000 7f9ca15e0640 1 -- 172.21.15.27:0/3916666047 >> 172.21.15.27:0/3916666047 conn(0x7f9c9c078de0 msgr2=0x7f9c9c07b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.305+0000 7f9ca15e0640 1 -- 172.21.15.27:0/3916666047 shutdown_connections 2024-04-25T13:43:38.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.305+0000 7f9ca15e0640 1 -- 172.21.15.27:0/3916666047 wait complete. 2024-04-25T13:43:38.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.307+0000 7f9ca15e0640 1 Processor -- start 2024-04-25T13:43:38.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.307+0000 7f9ca15e0640 1 -- start start 2024-04-25T13:43:38.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.307+0000 7f9ca15e0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c9c10a3d0 0x7f9c9c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.307+0000 7f9ca15e0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c9c10b330 0x7f9c9c1a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.307+0000 7f9ca15e0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9c110030 0x7f9c9c1a7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.307+0000 7f9ca15e0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c115c10 con 0x7f9c9c110030 2024-04-25T13:43:38.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.307+0000 7f9ca15e0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c115a90 con 0x7f9c9c10b330 2024-04-25T13:43:38.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.307+0000 7f9ca15e0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c115d90 con 0x7f9c9c10a3d0 2024-04-25T13:43:38.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c9c10b330 0x7f9c9c1a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9c110030 0x7f9c9c1a7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c9c10b330 0x7f9c9c1a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43952/0 (socket says 172.21.15.27:43952) 2024-04-25T13:43:38.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2581179856 learned_addr learned my addr 172.21.15.27:0/2581179856 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2581179856 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c9c10a3d0 msgr2=0x7f9c9c1a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c9c10a3d0 0x7f9c9c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9c110030 msgr2=0x7f9c9c1a7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9c110030 0x7f9c9c1a7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2581179856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9c1a8380 con 0x7f9c9c10b330 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.308+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c9c10b330 0x7f9c9c1a0d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009980 tx=0x7f9c8c032dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.309+0000 7f9c83fff640 1 -- 172.21.15.27:0/2581179856 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c040020 con 0x7f9c9c10b330 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.309+0000 7f9c83fff640 1 -- 172.21.15.27:0/2581179856 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c8c030dd0 con 0x7f9c9c10b330 2024-04-25T13:43:38.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.309+0000 7f9c83fff640 1 -- 172.21.15.27:0/2581179856 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c037040 con 0x7f9c9c10b330 2024-04-25T13:43:38.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.309+0000 7f9ca15e0640 1 -- 172.21.15.27:0/2581179856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c9c1a8610 con 0x7f9c9c10b330 2024-04-25T13:43:38.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.309+0000 7f9ca15e0640 1 -- 172.21.15.27:0/2581179856 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9c1a8ab0 con 0x7f9c9c10b330 2024-04-25T13:43:38.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.312+0000 7f9ca8568740 1 Processor -- start 2024-04-25T13:43:38.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.313+0000 7f9ca8568740 1 -- start start 2024-04-25T13:43:38.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.313+0000 7f9c83fff640 1 -- 172.21.15.27:0/2581179856 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9c8c043020 con 0x7f9c9c10b330 2024-04-25T13:43:38.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.313+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd40390 0x562acc0e1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.313+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd174d0 0x562acc0e1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.313+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0e2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.313+0000 7f9ca8568740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562acc046d80 con 0x562acbd1f6b0 2024-04-25T13:43:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.313+0000 7f9ca8568740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562acc0008b0 con 0x562acbd174d0 2024-04-25T13:43:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.313+0000 7f9ca8568740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562acc042d40 con 0x562acbd40390 2024-04-25T13:43:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0e2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0e2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47730/0 (socket says 172.21.15.27:47730) 2024-04-25T13:43:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/3830668556 learned_addr learned my addr 172.21.15.27:0/3830668556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9b7fe640 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd40390 0x562acc0e1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/3830668556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd40390 msgr2=0x562acc0e1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9affd640 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd174d0 0x562acc0e1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd40390 0x562acc0e1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/3830668556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd174d0 msgr2=0x562acc0e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd174d0 0x562acc0e1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/3830668556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562acbfd1a40 con 0x562acbd1f6b0 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c83fff640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c840afcc0 0x7f9c840b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.314+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0e2c70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c104a90 tx=0x7f9c8c104b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3d12f25659ee2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.315+0000 7f9c9b7fe640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c840afcc0 0x7f9c840b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.315+0000 7f9c83fff640 1 -- 172.21.15.27:0/2581179856 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9c8c0fe9f0 con 0x7f9c9c10b330 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.315+0000 7f9c817fa640 1 -- 172.21.15.27:0/3830668556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0c2de0 con 0x562acbd1f6b0 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.315+0000 7f9c817fa640 1 -- 172.21.15.27:0/3830668556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c8c0c6a30 con 0x562acbd1f6b0 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9c817fa640 1 -- 172.21.15.27:0/3830668556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0c6690 con 0x562acbd1f6b0 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9c9b7fe640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c840afcc0 0x7f9c840b21d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x562acc0e2ab0 tx=0x7f9c94010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 -- 172.21.15.27:0/3830668556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 msgr2=0x562acc0e2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0e2c70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c104a90 tx=0x7f9c8c104b80 comp rx=0 tx=0).stop 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 -- 172.21.15.27:0/3830668556 shutdown_connections 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0e2c70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd174d0 0x562acc0e1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 --2- 172.21.15.27:0/3830668556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd40390 0x562acc0e1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 -- 172.21.15.27:0/3830668556 >> 172.21.15.27:0/3830668556 conn(0x562acbd344e0 msgr2=0x562acc0dd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 -- 172.21.15.27:0/3830668556 shutdown_connections 2024-04-25T13:43:38.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.316+0000 7f9ca8568740 1 -- 172.21.15.27:0/3830668556 wait complete. 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.317+0000 7f9ca8568740 1 -- start start 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.317+0000 7f9ca8568740 1 Processor -- start 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd174d0 0x562acc0eb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd1f6b0 0x562acc0ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 0x562acc0f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9ca8568740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562acc046d80 con 0x562acbd174d0 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9ca8568740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562acbfcdd10 con 0x562acbd40390 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9ca8568740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562acc0e3540 con 0x562acbd1f6b0 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd1f6b0 0x562acc0ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd1f6b0 0x562acc0ebb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40474/0 (socket says 172.21.15.27:40474) 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2602635760 learned_addr learned my addr 172.21.15.27:0/2602635760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.318+0000 7f9ca8568740 10 client.0 _ll_register_callbacks cb 0x562acbffa210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:38.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c9affd640 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd174d0 0x562acc0eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c9b7fe640 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 0x562acc0f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2602635760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 msgr2=0x562acc0f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 0x562acc0f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2602635760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd174d0 msgr2=0x562acc0eb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd174d0 0x562acc0eb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2602635760 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562acbe227d0 con 0x562acbd1f6b0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd1f6b0 0x562acc0ebb40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c035040 tx=0x7f9c8c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55725422b41b995d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c7eff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.319+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2602635760 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0c6870 con 0x562acbd1f6b0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2602635760 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9c8c004160 con 0x562acbd1f6b0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 1 -- 172.21.15.27:0/2602635760 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c8c0c6a30 con 0x562acbd1f6b0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2602635760 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9c8c0e7020 con 0x562acbd1f6b0 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 10 client.25633 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 1 -- 172.21.15.27:0/2602635760 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562acbd583f0 con 0x562acbd1f6b0 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2602635760 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c8c0c6a30 con 0x562acbd1f6b0 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2602635760 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9c8c04d060 con 0x562acbd1f6b0 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 10 client.25633 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 10 client.25633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 20 client.25633 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562acbcfef60 0x562acc0f8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 4 client.25633 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.320+0000 7f9ca8568740 1 -- 172.21.15.27:0/2602635760 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562acc054990 con 0x562acbcfef60 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.321+0000 7f9c9affd640 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562acbcfef60 0x562acc0f8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.321+0000 7f9c7eff5640 10 client.25633 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.321+0000 7f9c9affd640 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562acbcfef60 0x562acc0f8c50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32770911800d4a9f server_cookie=220f6ab13c0e1066 in_seq=0 out_seq=0 2024-04-25T13:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2602635760 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562acc054990 con 0x562acbcfef60 2024-04-25T13:43:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9c7eff5640 10 client.25633 handle_command_reply: tid=0 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 client.25633 shutdown 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 -- 172.21.15.27:0/2602635760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd1f6b0 msgr2=0x562acc0ebb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd1f6b0 0x562acc0ebb40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c035040 tx=0x7f9c8c005f90 comp rx=0 tx=0).stop 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 -- 172.21.15.27:0/2602635760 shutdown_connections 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562acbcfef60 0x562acc0f8c50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 0x562acc0f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd1f6b0 0x562acc0ebb40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2602635760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd174d0 0x562acc0eb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9c7eff5640 0 client.25633 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 -- 172.21.15.27:0/2602635760 >> 172.21.15.27:0/2602635760 conn(0x562acbd344e0 msgr2=0x562acc0dc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 -- 172.21.15.27:0/2602635760 shutdown_connections 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.322+0000 7f9ca8568740 1 -- 172.21.15.27:0/2602635760 wait complete. 2024-04-25T13:43:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.323+0000 7f9ca8568740 20 client.25633 trim_cache size 0 max 16384 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.349+0000 7f9ca8568740 1 Processor -- start 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.349+0000 7f9ca8568740 1 -- start start 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.349+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0e9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.349+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd1f6b0 0x562acc0ea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.349+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd40390 0x562acc0f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.349+0000 7f9ca8568740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c1af7b0 con 0x562acbd40390 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.349+0000 7f9ca8568740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c1b2030 con 0x562acbd1f6b0 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.349+0000 7f9ca8568740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c1afe20 con 0x562acbd174d0 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd40390 0x562acc0f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd40390 0x562acc0f29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47738/0 (socket says 172.21.15.27:47738) 2024-04-25T13:43:38.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0e9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9b7fe640 1 -- 172.21.15.27:0/3129921435 learned_addr learned my addr 172.21.15.27:0/3129921435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/3129921435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd1f6b0 0x562acc0ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9affd640 1 -- 172.21.15.27:0/3129921435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd1f6b0 msgr2=0x562acc0ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9affd640 1 --2- 172.21.15.27:0/3129921435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd1f6b0 0x562acc0ea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9affd640 1 -- 172.21.15.27:0/3129921435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd40390 msgr2=0x562acc0f29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9affd640 1 --2- 172.21.15.27:0/3129921435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd40390 0x562acc0f29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.350+0000 7f9c9affd640 1 -- 172.21.15.27:0/3129921435 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562acbd180f0 con 0x562acbd174d0 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.351+0000 7f9c9affd640 1 --2- 172.21.15.27:0/3129921435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0e9c20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x562acc0f9380 tx=0x7f9c900095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2277593489264d54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.351+0000 7f9c80ff9640 1 -- 172.21.15.27:0/3129921435 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9001f030 con 0x562acbd174d0 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.352+0000 7f9c80ff9640 1 -- 172.21.15.27:0/3129921435 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c90009cf0 con 0x562acbd174d0 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.352+0000 7f9c80ff9640 1 -- 172.21.15.27:0/3129921435 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9001bac0 con 0x562acbd174d0 2024-04-25T13:43:38.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.353+0000 7f9ca8568740 1 -- 172.21.15.27:0/3129921435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 msgr2=0x562acc0e9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.353+0000 7f9ca8568740 1 --2- 172.21.15.27:0/3129921435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0e9c20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x562acc0f9380 tx=0x7f9c900095e0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.353+0000 7f9ca8568740 1 -- 172.21.15.27:0/3129921435 shutdown_connections 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.353+0000 7f9ca8568740 1 --2- 172.21.15.27:0/3129921435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd40390 0x562acc0f29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.353+0000 7f9ca8568740 1 --2- 172.21.15.27:0/3129921435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd1f6b0 0x562acc0ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.353+0000 7f9ca8568740 1 --2- 172.21.15.27:0/3129921435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0e9c20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.353+0000 7f9ca8568740 1 -- 172.21.15.27:0/3129921435 >> 172.21.15.27:0/3129921435 conn(0x562acbd344e0 msgr2=0x562acc0e64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.353+0000 7f9ca8568740 1 -- 172.21.15.27:0/3129921435 shutdown_connections 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.354+0000 7f9ca8568740 1 -- 172.21.15.27:0/3129921435 wait complete. 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.354+0000 7f9ca8568740 1 -- start start 2024-04-25T13:43:38.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.354+0000 7f9ca8568740 1 Processor -- start 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.354+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0f03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.354+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0f0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.354+0000 7f9ca8568740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 0x562acc0f1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.354+0000 7f9ca8568740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c1b04f0 con 0x562acbd1f6b0 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9ca8568740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c1b1960 con 0x562acbd40390 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9ca8568740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c1b0bc0 con 0x562acbd174d0 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9ca8568740 10 client.0 _ll_register_callbacks cb 0x7f9c94009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0f0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47752/0 (socket says 172.21.15.27:47752) 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2156580220 learned_addr learned my addr 172.21.15.27:0/2156580220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9affd640 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0f03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9b7fe640 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 0x562acc0f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2156580220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 msgr2=0x562acc0f03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0f03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2156580220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 msgr2=0x562acc0f1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 0x562acc0f1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 -- 172.21.15.27:0/2156580220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562acc0074b0 con 0x562acbd1f6b0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c9a7fc640 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0f0910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c0cfd80 tx=0x7f9c8c004390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecac7ab8acddeae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c7eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2156580220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c0c6cf0 con 0x562acbd1f6b0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2156580220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9c8c104040 con 0x562acbd1f6b0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.355+0000 7f9ca8568740 1 -- 172.21.15.27:0/2156580220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c90009cf0 con 0x562acbd1f6b0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2156580220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9c8c0d4050 con 0x562acbd1f6b0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9ca8568740 10 client.15897 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9ca8568740 1 -- 172.21.15.27:0/2156580220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562acc0074b0 con 0x562acbd1f6b0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2156580220 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c90009cf0 con 0x562acbd1f6b0 2024-04-25T13:43:38.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2156580220 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9c8c0cf8f0 con 0x562acbd1f6b0 2024-04-25T13:43:38.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9ca8568740 10 client.15897 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:38.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9ca8568740 10 client.15897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:38.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9ca8568740 20 client.15897 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:38.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562acbcd43a0 0x562acc162fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:38.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9ca8568740 4 client.15897 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:43:38.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9ca8568740 1 -- 172.21.15.27:0/2156580220 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562acc04e800 con 0x562acbcd43a0 2024-04-25T13:43:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.356+0000 7f9c9affd640 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562acbcd43a0 0x562acc162fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.357+0000 7f9c7eff5640 10 client.15897 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.357+0000 7f9c9affd640 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562acbcd43a0 0x562acc162fd0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c80234ca71aa3faf server_cookie=99bb78ebdf142174 in_seq=0 out_seq=0 2024-04-25T13:43:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.357+0000 7f9c7eff5640 1 -- 172.21.15.27:0/2156580220 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562acc04e800 con 0x562acbcd43a0 2024-04-25T13:43:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.357+0000 7f9c7eff5640 10 client.15897 handle_command_reply: tid=0 2024-04-25T13:43:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.357+0000 7f9ca8568740 1 client.15897 shutdown 2024-04-25T13:43:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 -- 172.21.15.27:0/2156580220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 msgr2=0x562acc0f0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0f0910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c0cfd80 tx=0x7f9c8c004390 comp rx=0 tx=0).stop 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 -- 172.21.15.27:0/2156580220 shutdown_connections 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562acbcd43a0 0x562acc162fd0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562acbd40390 0x562acc0f1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562acbd1f6b0 0x562acc0f0910 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 --2- 172.21.15.27:0/2156580220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562acbd174d0 0x562acc0f03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 -- 172.21.15.27:0/2156580220 >> 172.21.15.27:0/2156580220 conn(0x562acbd344e0 msgr2=0x562acc0e64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9c7eff5640 0 client.15897 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.358+0000 7f9ca8568740 1 -- 172.21.15.27:0/2156580220 shutdown_connections 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca8568740 1 -- 172.21.15.27:0/2156580220 wait complete. 2024-04-25T13:43:38.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca8568740 20 client.15897 trim_cache size 0 max 16384 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout: "d790558d-d685-461c-b84e-b8c52788c2e3": { 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "d790558d-d685-461c-b84e-b8c52788c2e3", 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:43:38.365 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:43:38.366 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:43:38.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 -- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c840afcc0 msgr2=0x7f9c840b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c840afcc0 0x7f9c840b21d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x562acc0e2ab0 tx=0x7f9c94010040 comp rx=0 tx=0).stop 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 -- 172.21.15.27:0/2581179856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c9c10b330 msgr2=0x7f9c9c1a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c9c10b330 0x7f9c9c1a0d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009980 tx=0x7f9c8c032dc0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 -- 172.21.15.27:0/2581179856 shutdown_connections 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c840afcc0 0x7f9c840b21d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c9c110030 0x7f9c9c1a7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c9c10b330 0x7f9c9c1a0d80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 --2- 172.21.15.27:0/2581179856 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c9c10a3d0 0x7f9c9c1a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 -- 172.21.15.27:0/2581179856 >> 172.21.15.27:0/2581179856 conn(0x7f9c9c078de0 msgr2=0x7f9c9c10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 -- 172.21.15.27:0/2581179856 shutdown_connections 2024-04-25T13:43:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:43:38.359+0000 7f9ca15e0640 1 -- 172.21.15.27:0/2581179856 wait complete. 2024-04-25T13:43:38.377 INFO:tasks.cephfs.filesystem:scrub status for tag:d790558d-d685-461c-b84e-b8c52788c2e3 - {'path': '/', 'tag': 'd790558d-d685-461c-b84e-b8c52788c2e3', 'options': 'recursive,force'} 2024-04-25T13:43:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:39 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:43:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:39 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:39 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:43:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:39 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:39 smithi122 ceph-mon[32843]: pgmap v432: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-25T13:43:39.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:39 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:43:39.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:39 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:43:39.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:39 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:39.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:39 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:43:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:39 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:39 smithi045 ceph-mon[32731]: pgmap v432: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-25T13:43:39.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:39 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:43:39.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:39 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:43:39.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:39 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:39.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:39 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:43:39.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:39 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:43:39.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:39 smithi027 ceph-mon[29140]: pgmap v432: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.8 MiB/s wr, 652 op/s 2024-04-25T13:43:39.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:39 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:43:41.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:41 smithi122 ceph-mon[32843]: pgmap v433: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.5 MiB/s wr, 823 op/s 2024-04-25T13:43:41.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:41 smithi045 ceph-mon[32731]: pgmap v433: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.5 MiB/s wr, 823 op/s 2024-04-25T13:43:41.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:41 smithi027 ceph-mon[29140]: pgmap v433: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.5 MiB/s wr, 823 op/s 2024-04-25T13:43:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 155.0 s, 3461.7 tps, lat 28.821 ms stddev 29.169 2024-04-25T13:43:43.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:43 smithi122 ceph-mon[32843]: pgmap v434: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 9.3 MiB/s wr, 854 op/s 2024-04-25T13:43:43.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:43 smithi045 ceph-mon[32731]: pgmap v434: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 9.3 MiB/s wr, 854 op/s 2024-04-25T13:43:43.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:43 smithi027 ceph-mon[29140]: pgmap v434: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 9.3 MiB/s wr, 854 op/s 2024-04-25T13:43:45.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:45 smithi122 ceph-mon[32843]: pgmap v435: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-25T13:43:45.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:45 smithi045 ceph-mon[32731]: pgmap v435: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-25T13:43:45.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:45 smithi027 ceph-mon[29140]: pgmap v435: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.8 MiB/s wr, 777 op/s 2024-04-25T13:43:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.213+0000 7fb46af0d640 1 -- 172.21.15.27:0/3268602964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb464104d60 msgr2=0x7fb464107180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.213+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3268602964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb464104d60 0x7fb464107180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb454009900 tx=0x7fb45402a400 comp rx=0 tx=0).stop 2024-04-25T13:43:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.216+0000 7fb46af0d640 1 -- 172.21.15.27:0/3268602964 shutdown_connections 2024-04-25T13:43:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.216+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3268602964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4641076c0 0x7fb464107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.216+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3268602964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb464104d60 0x7fb464107180 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.216+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3268602964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464102400 0x7fb464104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.216+0000 7fb46af0d640 1 -- 172.21.15.27:0/3268602964 >> 172.21.15.27:0/3268602964 conn(0x7fb4640fdc10 msgr2=0x7fb4641000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.217+0000 7fb46af0d640 1 -- 172.21.15.27:0/3268602964 shutdown_connections 2024-04-25T13:43:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.217+0000 7fb46af0d640 1 -- 172.21.15.27:0/3268602964 wait complete. 2024-04-25T13:43:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.218+0000 7fb46af0d640 1 Processor -- start 2024-04-25T13:43:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.218+0000 7fb46af0d640 1 -- start start 2024-04-25T13:43:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb46af0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb464102400 0x7fb4641a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb46af0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb464104d60 0x7fb4641a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb46af0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4641076c0 0x7fb4641a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb468ca9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb464102400 0x7fb4641a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb468ca9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb464102400 0x7fb4641a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33926/0 (socket says 172.21.15.27:33926) 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb468ca9640 1 -- 172.21.15.27:0/3165486740 learned_addr learned my addr 172.21.15.27:0/3165486740 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb464115cf0 con 0x7fb464104d60 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb463fff640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb464104d60 0x7fb4641a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb464115b70 con 0x7fb4641076c0 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb464115e70 con 0x7fb464102400 2024-04-25T13:43:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb4694aa640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4641076c0 0x7fb4641a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb468ca9640 1 -- 172.21.15.27:0/3165486740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4641076c0 msgr2=0x7fb4641a7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb468ca9640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4641076c0 0x7fb4641a7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb468ca9640 1 -- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb464104d60 msgr2=0x7fb4641a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb468ca9640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb464104d60 0x7fb4641a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.219+0000 7fb468ca9640 1 -- 172.21.15.27:0/3165486740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4641a81e0 con 0x7fb464102400 2024-04-25T13:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.220+0000 7fb468ca9640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb464102400 0x7fb4641a06a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00b9c0 tx=0x7fb45c00be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.220+0000 7fb461ffb640 1 -- 172.21.15.27:0/3165486740 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c019070 con 0x7fb464102400 2024-04-25T13:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.220+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4641a84d0 con 0x7fb464102400 2024-04-25T13:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.220+0000 7fb461ffb640 1 -- 172.21.15.27:0/3165486740 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb45c00cba0 con 0x7fb464102400 2024-04-25T13:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.220+0000 7fb461ffb640 1 -- 172.21.15.27:0/3165486740 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c012580 con 0x7fb464102400 2024-04-25T13:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.220+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4641a8af0 con 0x7fb464102400 2024-04-25T13:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.223+0000 7fb461ffb640 1 -- 172.21.15.27:0/3165486740 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb45c01c070 con 0x7fb464102400 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb461ffb640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb44c0afb30 0x7fb44c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb471dc3740 1 Processor -- start 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb471dc3740 1 -- start start 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb463fff640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb44c0afb30 0x7fb44c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb461ffb640 1 -- 172.21.15.27:0/3165486740 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb45c0de1d0 con 0x7fb464102400 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c45a1b0 0x55b10c7fa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c7faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4394d0 0x55b10c801df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb471dc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b10c760460 con 0x55b10c4394d0 2024-04-25T13:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb471dc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b10c71a590 con 0x55b10c45a1b0 2024-04-25T13:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb471dc3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b10c75c420 con 0x55b10c4312f0 2024-04-25T13:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.224+0000 7fb463fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c7faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb4694aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4394d0 0x55b10c801df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb463fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c7faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33930/0 (socket says 172.21.15.27:33930) 2024-04-25T13:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb468ca9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c45a1b0 0x55b10c7fa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb4694aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4394d0 0x55b10c801df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44220/0 (socket says 172.21.15.27:44220) 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb463fff640 1 -- 172.21.15.27:0/3112757873 learned_addr learned my addr 172.21.15.27:0/3112757873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb468ca9640 1 -- 172.21.15.27:0/3112757873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 msgr2=0x55b10c7faef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb468ca9640 1 --2- 172.21.15.27:0/3112757873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c7faef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb468ca9640 1 -- 172.21.15.27:0/3112757873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4394d0 msgr2=0x55b10c801df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb468ca9640 1 --2- 172.21.15.27:0/3112757873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4394d0 0x55b10c801df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb468ca9640 1 -- 172.21.15.27:0/3112757873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b10c7270d0 con 0x55b10c45a1b0 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb463fff640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb44c0afb30 0x7fb44c0b2040 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb4540096c0 tx=0x7fb454004610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb468ca9640 1 --2- 172.21.15.27:0/3112757873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c45a1b0 0x55b10c7fa970 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0dbca0 tx=0x7fb45c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f151d94223d95cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb44affd640 1 -- 172.21.15.27:0/3112757873 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c0af4e0 con 0x55b10c45a1b0 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb44affd640 1 -- 172.21.15.27:0/3112757873 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb45c00cd50 con 0x55b10c45a1b0 2024-04-25T13:43:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.225+0000 7fb44affd640 1 -- 172.21.15.27:0/3112757873 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c08de30 con 0x55b10c45a1b0 2024-04-25T13:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.226+0000 7fb471dc3740 1 -- 172.21.15.27:0/3112757873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c45a1b0 msgr2=0x55b10c7fa970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.226+0000 7fb471dc3740 1 --2- 172.21.15.27:0/3112757873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c45a1b0 0x55b10c7fa970 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0dbca0 tx=0x7fb45c004a40 comp rx=0 tx=0).stop 2024-04-25T13:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.227+0000 7fb471dc3740 1 -- 172.21.15.27:0/3112757873 shutdown_connections 2024-04-25T13:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.227+0000 7fb471dc3740 1 --2- 172.21.15.27:0/3112757873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4394d0 0x55b10c801df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.227+0000 7fb471dc3740 1 --2- 172.21.15.27:0/3112757873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c7faef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.227+0000 7fb471dc3740 1 --2- 172.21.15.27:0/3112757873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c45a1b0 0x55b10c7fa970 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.227+0000 7fb471dc3740 1 -- 172.21.15.27:0/3112757873 >> 172.21.15.27:0/3112757873 conn(0x55b10c44e300 msgr2=0x55b10c7f6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.228+0000 7fb471dc3740 1 -- 172.21.15.27:0/3112757873 shutdown_connections 2024-04-25T13:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.228+0000 7fb471dc3740 1 -- 172.21.15.27:0/3112757873 wait complete. 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.229+0000 7fb471dc3740 1 -- start start 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.229+0000 7fb471dc3740 1 Processor -- start 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c8047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c804d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c80bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c8047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb471dc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b10c802840 con 0x55b10c45a1b0 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb471dc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b10c6e7b60 con 0x55b10c4394d0 2024-04-25T13:43:47.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c8047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33942/0 (socket says 172.21.15.27:33942) 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 -- 172.21.15.27:0/2120077673 learned_addr learned my addr 172.21.15.27:0/2120077673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b10c8029c0 con 0x55b10c4312f0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb4694aa640 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c80bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb471dc3740 10 client.0 _ll_register_callbacks cb 0x55b10c70c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb463fff640 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c804d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 -- 172.21.15.27:0/2120077673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 msgr2=0x55b10c804d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c804d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 -- 172.21.15.27:0/2120077673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 msgr2=0x55b10c80bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c80bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 -- 172.21.15.27:0/2120077673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b10c472210 con 0x55b10c4312f0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb468ca9640 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c8047c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0a68f0 tx=0x7fb45c0af300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997c76ae0eb7c147 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb443fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb443fff640 1 -- 172.21.15.27:0/2120077673 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c08de20 con 0x55b10c4312f0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb443fff640 1 -- 172.21.15.27:0/2120077673 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb45c0a2d20 con 0x55b10c4312f0 2024-04-25T13:43:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.230+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb45c00cd50 con 0x55b10c4312f0 2024-04-25T13:43:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb443fff640 1 -- 172.21.15.27:0/2120077673 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb45c0c7050 con 0x55b10c4312f0 2024-04-25T13:43:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb471dc3740 10 client.25651 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b10c53e620 con 0x55b10c4312f0 2024-04-25T13:43:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb443fff640 1 -- 172.21.15.27:0/2120077673 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb45c00cd50 con 0x55b10c4312f0 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb443fff640 1 -- 172.21.15.27:0/2120077673 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb45c0bbbb0 con 0x55b10c4312f0 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb471dc3740 10 client.25651 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb471dc3740 10 client.25651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb471dc3740 20 client.25651 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb471dc3740 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b10c418d80 0x55b10c811e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb471dc3740 4 client.25651 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.231+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b10c76e070 con 0x55b10c418d80 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.232+0000 7fb463fff640 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b10c418d80 0x55b10c811e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.232+0000 7fb443fff640 10 client.25651 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.232+0000 7fb463fff640 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b10c418d80 0x55b10c811e10 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32185ed67393e430 server_cookie=7c882153a8696644 in_seq=0 out_seq=0 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb443fff640 1 -- 172.21.15.27:0/2120077673 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b10c76e070 con 0x55b10c418d80 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb443fff640 10 client.25651 handle_command_reply: tid=0 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 client.25651 shutdown 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 msgr2=0x55b10c8047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c8047c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0a68f0 tx=0x7fb45c0af300 comp rx=0 tx=0).stop 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 shutdown_connections 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b10c418d80 0x55b10c811e10 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c80bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c804d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb443fff640 0 client.25651 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 --2- 172.21.15.27:0/2120077673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c8047c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 >> 172.21.15.27:0/2120077673 conn(0x55b10c44e300 msgr2=0x55b10c7f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 shutdown_connections 2024-04-25T13:43:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 1 -- 172.21.15.27:0/2120077673 wait complete. 2024-04-25T13:43:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.233+0000 7fb471dc3740 20 client.25651 trim_cache size 0 max 16384 2024-04-25T13:43:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.256+0000 7fb471dc3740 1 Processor -- start 2024-04-25T13:43:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.258+0000 7fb471dc3740 1 -- start start 2024-04-25T13:43:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.258+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4312f0 0x55b10c8173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.258+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c8178f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c45a1b0 0x55b10c802d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb471dc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4641af6b0 con 0x55b10c4312f0 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb471dc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb4641b1f30 con 0x55b10c4394d0 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb471dc3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb4641afd20 con 0x55b10c45a1b0 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb463fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c8178f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb463fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c8178f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40390/0 (socket says 172.21.15.27:40390) 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb463fff640 1 -- 172.21.15.27:0/4120688983 learned_addr learned my addr 172.21.15.27:0/4120688983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb4694aa640 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c45a1b0 0x55b10c802d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb468ca9640 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4312f0 0x55b10c8173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb4694aa640 1 -- 172.21.15.27:0/4120688983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 msgr2=0x55b10c8178f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb4694aa640 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c8178f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb4694aa640 1 -- 172.21.15.27:0/4120688983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4312f0 msgr2=0x55b10c8173b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb4694aa640 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4312f0 0x55b10c8173b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb4694aa640 1 -- 172.21.15.27:0/4120688983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b10c801ff0 con 0x55b10c45a1b0 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.259+0000 7fb4694aa640 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c45a1b0 0x55b10c802d60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55b10c7f7090 tx=0x7fb458012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41912a673caa40a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.260+0000 7fb44a7fc640 1 -- 172.21.15.27:0/4120688983 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45800be30 con 0x55b10c45a1b0 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.260+0000 7fb44a7fc640 1 -- 172.21.15.27:0/4120688983 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4580054d0 con 0x55b10c45a1b0 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.260+0000 7fb44a7fc640 1 -- 172.21.15.27:0/4120688983 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45801b670 con 0x55b10c45a1b0 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 -- 172.21.15.27:0/4120688983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c45a1b0 msgr2=0x55b10c802d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c45a1b0 0x55b10c802d60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55b10c7f7090 tx=0x7fb458012820 comp rx=0 tx=0).stop 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 -- 172.21.15.27:0/4120688983 shutdown_connections 2024-04-25T13:43:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c45a1b0 0x55b10c802d60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c8178f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 --2- 172.21.15.27:0/4120688983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c4312f0 0x55b10c8173b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 -- 172.21.15.27:0/4120688983 >> 172.21.15.27:0/4120688983 conn(0x55b10c44e300 msgr2=0x55b10c7fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 -- 172.21.15.27:0/4120688983 shutdown_connections 2024-04-25T13:43:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 -- 172.21.15.27:0/4120688983 wait complete. 2024-04-25T13:43:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 -- start start 2024-04-25T13:43:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.261+0000 7fb471dc3740 1 Processor -- start 2024-04-25T13:43:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c80a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c80a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb471dc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c877f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb471dc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4641b03f0 con 0x55b10c45a1b0 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb471dc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb4641b1860 con 0x55b10c4394d0 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb471dc3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb4641b0ac0 con 0x55b10c4312f0 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb471dc3740 10 client.0 _ll_register_callbacks cb 0x7fb458005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c877f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c877f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44240/0 (socket says 172.21.15.27:44240) 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 -- 172.21.15.27:0/393838497 learned_addr learned my addr 172.21.15.27:0/393838497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb463fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c80a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb468ca9640 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c80a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 -- 172.21.15.27:0/393838497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 msgr2=0x55b10c80a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c80a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 -- 172.21.15.27:0/393838497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 msgr2=0x55b10c80a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c80a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 -- 172.21.15.27:0/393838497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b10c878490 con 0x55b10c45a1b0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb4694aa640 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c877f50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb45801d040 tx=0x7fb458005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af5a07c5c227392 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.262+0000 7fb443fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb443fff640 1 -- 172.21.15.27:0/393838497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb458029020 con 0x55b10c45a1b0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb443fff640 1 -- 172.21.15.27:0/393838497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb4580048d0 con 0x55b10c45a1b0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 1 -- 172.21.15.27:0/393838497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4580054d0 con 0x55b10c45a1b0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb443fff640 1 -- 172.21.15.27:0/393838497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb458005020 con 0x55b10c45a1b0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 10 client.15915 fetch_fsmap learned FSMap version 19 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 1 -- 172.21.15.27:0/393838497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b10c8795d0 con 0x55b10c45a1b0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb443fff640 1 -- 172.21.15.27:0/393838497 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4580054d0 con 0x55b10c45a1b0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb443fff640 1 -- 172.21.15.27:0/393838497 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb45801b410 con 0x55b10c45a1b0 2024-04-25T13:43:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 10 client.15915 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:43:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:43:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 20 client.15915 populate_metadata read hostname 'smithi027' 2024-04-25T13:43:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b10c3ee1c0 0x55b10c87bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:43:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 4 client.15915 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:43:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.263+0000 7fb471dc3740 1 -- 172.21.15.27:0/393838497 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b10c767ee0 con 0x55b10c3ee1c0 2024-04-25T13:43:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.264+0000 7fb468ca9640 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b10c3ee1c0 0x55b10c87bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:43:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.265+0000 7fb468ca9640 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b10c3ee1c0 0x55b10c87bf60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b91d61a48b78321f server_cookie=e8b8ac16c57502da in_seq=0 out_seq=0 2024-04-25T13:43:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.265+0000 7fb443fff640 10 client.15915 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.284+0000 7fb443fff640 1 -- 172.21.15.27:0/393838497 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b10c767ee0 con 0x55b10c3ee1c0 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.284+0000 7fb443fff640 10 client.15915 handle_command_reply: tid=0 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.284+0000 7fb471dc3740 1 client.15915 shutdown 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 -- 172.21.15.27:0/393838497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 msgr2=0x55b10c877f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c877f50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb45801d040 tx=0x7fb458005af0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 -- 172.21.15.27:0/393838497 shutdown_connections 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b10c3ee1c0 0x55b10c87bf60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b10c45a1b0 0x55b10c877f50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b10c4394d0 0x55b10c80a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 --2- 172.21.15.27:0/393838497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b10c4312f0 0x55b10c80a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb443fff640 0 client.15915 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:43:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 -- 172.21.15.27:0/393838497 >> 172.21.15.27:0/393838497 conn(0x55b10c44e300 msgr2=0x55b10c802d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 -- 172.21.15.27:0/393838497 shutdown_connections 2024-04-25T13:43:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 1 -- 172.21.15.27:0/393838497 wait complete. 2024-04-25T13:43:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.285+0000 7fb471dc3740 20 client.15915 trim_cache size 0 max 16384 2024-04-25T13:43:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:43:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:43:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:43:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.286+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb44c0afb30 msgr2=0x7fb44c0b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.286+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb44c0afb30 0x7fb44c0b2040 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb4540096c0 tx=0x7fb454004610 comp rx=0 tx=0).stop 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.286+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb464102400 msgr2=0x7fb4641a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.286+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb464102400 0x7fb4641a06a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00b9c0 tx=0x7fb45c00be80 comp rx=0 tx=0).stop 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.287+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 shutdown_connections 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.287+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb44c0afb30 0x7fb44c0b2040 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.287+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4641076c0 0x7fb4641a7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.287+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb464104d60 0x7fb4641a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.287+0000 7fb46af0d640 1 --2- 172.21.15.27:0/3165486740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb464102400 0x7fb4641a06a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.287+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 >> 172.21.15.27:0/3165486740 conn(0x7fb4640fdc10 msgr2=0x7fb464103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.287+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 shutdown_connections 2024-04-25T13:43:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:43:47.287+0000 7fb46af0d640 1 -- 172.21.15.27:0/3165486740 wait complete. 2024-04-25T13:43:47.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:47 smithi122 ceph-mon[32843]: pgmap v436: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 12 MiB/s wr, 957 op/s 2024-04-25T13:43:47.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:47 smithi045 ceph-mon[32731]: pgmap v436: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 12 MiB/s wr, 957 op/s 2024-04-25T13:43:47.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:47 smithi027 ceph-mon[29140]: pgmap v436: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 12 MiB/s wr, 957 op/s 2024-04-25T13:43:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 160.0 s, 2977.2 tps, lat 33.678 ms stddev 35.019 2024-04-25T13:43:49.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:49 smithi122 ceph-mon[32843]: pgmap v437: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 9.4 MiB/s wr, 754 op/s 2024-04-25T13:43:49.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:49 smithi045 ceph-mon[32731]: pgmap v437: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 9.4 MiB/s wr, 754 op/s 2024-04-25T13:43:49.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:49 smithi027 ceph-mon[29140]: pgmap v437: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 9.4 MiB/s wr, 754 op/s 2024-04-25T13:43:51.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:51 smithi122 ceph-mon[32843]: pgmap v438: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 10 MiB/s wr, 829 op/s 2024-04-25T13:43:51.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:51 smithi045 ceph-mon[32731]: pgmap v438: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 10 MiB/s wr, 829 op/s 2024-04-25T13:43:51.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:51 smithi027 ceph-mon[29140]: pgmap v438: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 10 MiB/s wr, 829 op/s 2024-04-25T13:43:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 165.0 s, 2730.6 tps, lat 36.545 ms stddev 40.083 2024-04-25T13:43:53.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:53 smithi045 ceph-mon[32731]: pgmap v439: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.3 MiB/s wr, 766 op/s 2024-04-25T13:43:53.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:53 smithi027 ceph-mon[29140]: pgmap v439: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.3 MiB/s wr, 766 op/s 2024-04-25T13:43:54.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:53 smithi122 ceph-mon[32843]: pgmap v439: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.3 MiB/s wr, 766 op/s 2024-04-25T13:43:55.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:55 smithi045 ceph-mon[32731]: pgmap v440: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 696 op/s 2024-04-25T13:43:55.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:55 smithi027 ceph-mon[29140]: pgmap v440: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 696 op/s 2024-04-25T13:43:56.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:55 smithi122 ceph-mon[32843]: pgmap v440: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 696 op/s 2024-04-25T13:43:57.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:57 smithi045 ceph-mon[32731]: pgmap v441: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 844 op/s 2024-04-25T13:43:57.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:57 smithi027 ceph-mon[29140]: pgmap v441: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 844 op/s 2024-04-25T13:43:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 170.0 s, 3019.2 tps, lat 33.107 ms stddev 38.772 2024-04-25T13:43:58.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:57 smithi122 ceph-mon[32843]: pgmap v441: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 844 op/s 2024-04-25T13:43:59.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:43:59 smithi045 ceph-mon[32731]: pgmap v442: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 MiB/s wr, 697 op/s 2024-04-25T13:43:59.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:43:59 smithi027 ceph-mon[29140]: pgmap v442: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 MiB/s wr, 697 op/s 2024-04-25T13:44:00.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:43:59 smithi122 ceph-mon[32843]: pgmap v442: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 MiB/s wr, 697 op/s 2024-04-25T13:44:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 -- 172.21.15.27:0/4206425348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4103b40 msgr2=0x7f26c4104840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 --2- 172.21.15.27:0/4206425348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4103b40 0x7f26c4104840 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009900 tx=0x7f26b402a3f0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 -- 172.21.15.27:0/4206425348 shutdown_connections 2024-04-25T13:44:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 --2- 172.21.15.27:0/4206425348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4109a80 0x7f26c410bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 --2- 172.21.15.27:0/4206425348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26c4104d80 0x7f26c4105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 --2- 172.21.15.27:0/4206425348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4103b40 0x7f26c4104840 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 -- 172.21.15.27:0/4206425348 >> 172.21.15.27:0/4206425348 conn(0x7f26c40fed30 msgr2=0x7f26c41011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 -- 172.21.15.27:0/4206425348 shutdown_connections 2024-04-25T13:44:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.437+0000 7f26ca215640 1 -- 172.21.15.27:0/4206425348 wait complete. 2024-04-25T13:44:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26ca215640 1 Processor -- start 2024-04-25T13:44:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26ca215640 1 -- start start 2024-04-25T13:44:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26ca215640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4103b40 0x7f26c41a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26ca215640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4104d80 0x7f26c41a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26ca215640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26c4109a80 0x7f26c41a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26ca215640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4076090 con 0x7f26c4104d80 2024-04-25T13:44:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26ca215640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4075f10 con 0x7f26c4109a80 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26ca215640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4076210 con 0x7f26c4103b40 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.438+0000 7f26c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4104d80 0x7f26c41a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4103b40 0x7f26c41a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4104d80 0x7f26c41a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60206/0 (socket says 172.21.15.27:60206) 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c2ffd640 1 -- 172.21.15.27:0/92285908 learned_addr learned my addr 172.21.15.27:0/92285908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c3fff640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26c4109a80 0x7f26c41a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c3fff640 1 -- 172.21.15.27:0/92285908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4103b40 msgr2=0x7f26c41a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c3fff640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4103b40 0x7f26c41a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c3fff640 1 -- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4104d80 msgr2=0x7f26c41a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c3fff640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4104d80 0x7f26c41a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c3fff640 1 -- 172.21.15.27:0/92285908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c41a8310 con 0x7f26c4109a80 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4104d80 0x7f26c41a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c37fe640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4103b40 0x7f26c41a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c3fff640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26c4109a80 0x7f26c41a7d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00cab0 tx=0x7f26bc00cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.439+0000 7f26c0ff9640 1 -- 172.21.15.27:0/92285908 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc00ae20 con 0x7f26c4109a80 2024-04-25T13:44:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.440+0000 7f26c0ff9640 1 -- 172.21.15.27:0/92285908 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26bc00f900 con 0x7f26c4109a80 2024-04-25T13:44:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.440+0000 7f26c0ff9640 1 -- 172.21.15.27:0/92285908 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc0189c0 con 0x7f26c4109a80 2024-04-25T13:44:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.440+0000 7f26ca215640 1 -- 172.21.15.27:0/92285908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c41a8600 con 0x7f26c4109a80 2024-04-25T13:44:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.440+0000 7f26ca215640 1 -- 172.21.15.27:0/92285908 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c41a8aa0 con 0x7f26c4109a80 2024-04-25T13:44:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.443+0000 7f26c0ff9640 1 -- 172.21.15.27:0/92285908 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f26bc00fab0 con 0x7f26c4109a80 2024-04-25T13:44:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.444+0000 7f26d11a7740 1 Processor -- start 2024-04-25T13:44:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.444+0000 7f26d11a7740 1 -- start start 2024-04-25T13:44:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d941b0 0x5604ba134970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d734d0 0x5604ba13bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26d11a7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5604ba09a460 con 0x5604b9d734d0 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26d11a7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5604ba054590 con 0x5604b9d941b0 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26d11a7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5604ba096420 con 0x5604b9d6b2f0 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c0ff9640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f26ac0afbe0 0x7f26ac0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d941b0 0x5604ba134970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d941b0 0x5604ba134970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39384/0 (socket says 172.21.15.27:39384) 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba134ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c2ffd640 1 -- 172.21.15.27:0/2798951442 learned_addr learned my addr 172.21.15.27:0/2798951442 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/2798951442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d734d0 0x5604ba13bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c0ff9640 1 -- 172.21.15.27:0/92285908 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f26bc0deed0 con 0x7f26c4109a80 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c37fe640 1 -- 172.21.15.27:0/2798951442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d941b0 msgr2=0x5604ba134970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c37fe640 1 --2- 172.21.15.27:0/2798951442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d941b0 0x5604ba134970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c37fe640 1 -- 172.21.15.27:0/2798951442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d734d0 msgr2=0x5604ba13bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c37fe640 1 --2- 172.21.15.27:0/2798951442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d734d0 0x5604ba13bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.445+0000 7f26c37fe640 1 -- 172.21.15.27:0/2798951442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ba0610d0 con 0x5604b9d6b2f0 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.446+0000 7f26c37fe640 1 --2- 172.21.15.27:0/2798951442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba134ef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009230 tx=0x7f26b40045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9f5f3331c8f79ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.446+0000 7f26c3fff640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f26ac0afbe0 0x7f26ac0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.446+0000 7f26a9ffb640 1 -- 172.21.15.27:0/2798951442 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4035030 con 0x5604b9d6b2f0 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.446+0000 7f26a9ffb640 1 -- 172.21.15.27:0/2798951442 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b4051070 con 0x5604b9d6b2f0 2024-04-25T13:44:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.446+0000 7f26a9ffb640 1 -- 172.21.15.27:0/2798951442 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4031e10 con 0x5604b9d6b2f0 2024-04-25T13:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.447+0000 7f26d11a7740 1 -- 172.21.15.27:0/2798951442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 msgr2=0x5604ba134ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.447+0000 7f26d11a7740 1 --2- 172.21.15.27:0/2798951442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba134ef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009230 tx=0x7f26b40045d0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.447+0000 7f26c3fff640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f26ac0afbe0 0x7f26ac0b20f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0a45f0 tx=0x7f26bc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.447+0000 7f26d11a7740 1 -- 172.21.15.27:0/2798951442 shutdown_connections 2024-04-25T13:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.447+0000 7f26d11a7740 1 --2- 172.21.15.27:0/2798951442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d734d0 0x5604ba13bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.447+0000 7f26d11a7740 1 --2- 172.21.15.27:0/2798951442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba134ef0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.447+0000 7f26d11a7740 1 --2- 172.21.15.27:0/2798951442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d941b0 0x5604ba134970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.447+0000 7f26d11a7740 1 -- 172.21.15.27:0/2798951442 >> 172.21.15.27:0/2798951442 conn(0x5604b9d88300 msgr2=0x5604ba130300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.448+0000 7f26d11a7740 1 -- 172.21.15.27:0/2798951442 shutdown_connections 2024-04-25T13:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.448+0000 7f26d11a7740 1 -- 172.21.15.27:0/2798951442 wait complete. 2024-04-25T13:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.448+0000 7f26d11a7740 1 -- start start 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.449+0000 7f26d11a7740 1 Processor -- start 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.449+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d6b2f0 0x5604ba13e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.449+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d734d0 0x5604ba13eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.449+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d941b0 0x5604ba145a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.449+0000 7f26d11a7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5604ba13c7e0 con 0x5604b9d6b2f0 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.449+0000 7f26d11a7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5604ba021b60 con 0x5604b9d734d0 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.449+0000 7f26d11a7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5604ba13c960 con 0x5604b9d941b0 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.449+0000 7f26d11a7740 10 client.0 _ll_register_callbacks cb 0x5604ba0462b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d734d0 0x5604ba13eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d734d0 0x5604ba13eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39386/0 (socket says 172.21.15.27:39386) 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c37fe640 1 -- 172.21.15.27:0/1828422558 learned_addr learned my addr 172.21.15.27:0/1828422558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d6b2f0 0x5604ba13e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d941b0 0x5604ba145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c37fe640 1 -- 172.21.15.27:0/1828422558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d941b0 msgr2=0x5604ba145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c37fe640 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d941b0 0x5604ba145a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c37fe640 1 -- 172.21.15.27:0/1828422558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d6b2f0 msgr2=0x5604ba13e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c37fe640 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d6b2f0 0x5604ba13e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c37fe640 1 -- 172.21.15.27:0/1828422558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604b9dac210 con 0x5604b9d734d0 2024-04-25T13:44:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.450+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d941b0 0x5604ba145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.451+0000 7f26c37fe640 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d734d0 0x5604ba13eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f26b40092a0 tx=0x7f26b40317d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fedffc9c4b11344 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.451+0000 7f26a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.451+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1828422558 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4035070 con 0x5604b9d734d0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.451+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1828422558 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f26b4003a30 con 0x5604b9d734d0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.451+0000 7f26d11a7740 1 -- 172.21.15.27:0/1828422558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26b4051070 con 0x5604b9d734d0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.451+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1828422558 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f26b4003c30 con 0x5604b9d734d0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.451+0000 7f26d11a7740 10 client.25640 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.451+0000 7f26d11a7740 1 -- 172.21.15.27:0/1828422558 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5604b9e78620 con 0x5604b9d734d0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1828422558 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b4051070 con 0x5604b9d734d0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1828422558 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f26b405e020 con 0x5604b9d734d0 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26d11a7740 10 client.25640 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26d11a7740 10 client.25640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26d11a7740 20 client.25640 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604b9d52d80 0x5604ba14bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26d11a7740 4 client.25640 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26d11a7740 1 -- 172.21.15.27:0/1828422558 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604ba0a8070 con 0x5604b9d52d80 2024-04-25T13:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.452+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604b9d52d80 0x5604ba14bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.453+0000 7f26a2ffd640 10 client.25640 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.453+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604b9d52d80 0x5604ba14bee0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d526917459ade07 server_cookie=d009271451a82f4f in_seq=0 out_seq=0 2024-04-25T13:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.454+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1828422558 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5604ba0a8070 con 0x5604b9d52d80 2024-04-25T13:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.454+0000 7f26a2ffd640 10 client.25640 handle_command_reply: tid=0 2024-04-25T13:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.454+0000 7f26d11a7740 1 client.25640 shutdown 2024-04-25T13:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 -- 172.21.15.27:0/1828422558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d734d0 msgr2=0x5604ba13eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d734d0 0x5604ba13eb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f26b40092a0 tx=0x7f26b40317d0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 -- 172.21.15.27:0/1828422558 shutdown_connections 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604b9d52d80 0x5604ba14bee0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d941b0 0x5604ba145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d734d0 0x5604ba13eb30 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f26b40092a0 tx=0x7f26b40317d0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1828422558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d6b2f0 0x5604ba13e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 -- 172.21.15.27:0/1828422558 >> 172.21.15.27:0/1828422558 conn(0x5604b9d88300 msgr2=0x5604ba133210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26a2ffd640 0 client.25640 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 -- 172.21.15.27:0/1828422558 shutdown_connections 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 1 -- 172.21.15.27:0/1828422558 wait complete. 2024-04-25T13:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.455+0000 7f26d11a7740 20 client.25640 trim_cache size 0 max 16384 2024-04-25T13:44:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.480+0000 7f26d11a7740 1 Processor -- start 2024-04-25T13:44:00.480 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26d11a7740 1 -- start start 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba12fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba145a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba130190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26d11a7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4119180 con 0x5604b9d941b0 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26d11a7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41a97a0 con 0x5604b9d281c0 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26d11a7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41197f0 con 0x5604b9d6b2f0 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26c2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba12fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba145a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60228/0 (socket says 172.21.15.27:60228) 2024-04-25T13:44:00.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.481+0000 7f26c37fe640 1 -- 172.21.15.27:0/3821320163 learned_addr learned my addr 172.21.15.27:0/3821320163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba130190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26c37fe640 1 -- 172.21.15.27:0/3821320163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 msgr2=0x5604ba12fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26c37fe640 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba12fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26c37fe640 1 -- 172.21.15.27:0/3821320163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 msgr2=0x5604ba130190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26c37fe640 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba130190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26c37fe640 1 -- 172.21.15.27:0/3821320163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ba135660 con 0x5604b9d941b0 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba130190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26c37fe640 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba145a50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f26b402a9e0 tx=0x7f26b4009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8430f4f745f3f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26a97fa640 1 -- 172.21.15.27:0/3821320163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4035040 con 0x5604b9d941b0 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26a97fa640 1 -- 172.21.15.27:0/3821320163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b4051070 con 0x5604b9d941b0 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.482+0000 7f26a97fa640 1 -- 172.21.15.27:0/3821320163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4004a80 con 0x5604b9d941b0 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 -- 172.21.15.27:0/3821320163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 msgr2=0x5604ba145a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba145a50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f26b402a9e0 tx=0x7f26b4009230 comp rx=0 tx=0).stop 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 -- 172.21.15.27:0/3821320163 shutdown_connections 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba130190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba145a50 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 --2- 172.21.15.27:0/3821320163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba12fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 -- 172.21.15.27:0/3821320163 >> 172.21.15.27:0/3821320163 conn(0x5604b9d88300 msgr2=0x5604ba137b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 -- 172.21.15.27:0/3821320163 shutdown_connections 2024-04-25T13:44:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.483+0000 7f26d11a7740 1 -- 172.21.15.27:0/3821320163 wait complete. 2024-04-25T13:44:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.484+0000 7f26d11a7740 1 -- start start 2024-04-25T13:44:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.484+0000 7f26d11a7740 1 Processor -- start 2024-04-25T13:44:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.484+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba143480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba1439c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39400/0 (socket says 172.21.15.27:39400) 2024-04-25T13:44:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26d11a7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba144880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 -- 172.21.15.27:0/1947183605 learned_addr learned my addr 172.21.15.27:0/1947183605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4119ec0 con 0x5604b9d941b0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41a90d0 con 0x5604b9d281c0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f26c411a590 con 0x5604b9d6b2f0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26d11a7740 10 client.0 _ll_register_callbacks cb 0x7f26b8011780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c37fe640 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba1439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 -- 172.21.15.27:0/1947183605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 msgr2=0x5604ba1439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba1439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 -- 172.21.15.27:0/1947183605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 msgr2=0x5604ba144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba144880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.485+0000 7f26c2ffd640 1 -- 172.21.15.27:0/1947183605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604ba139500 con 0x5604b9d281c0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.486+0000 7f26c2ffd640 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba143480 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f26b8025010 tx=0x7f26b4009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af363dc76756298 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.486+0000 7f26a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.486+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1947183605 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b8023650 con 0x5604b9d281c0 2024-04-25T13:44:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.486+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26b4051070 con 0x5604b9d281c0 2024-04-25T13:44:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.486+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1947183605 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f26b80237f0 con 0x5604b9d281c0 2024-04-25T13:44:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.486+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1947183605 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f26b80239f0 con 0x5604b9d281c0 2024-04-25T13:44:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.486+0000 7f26d11a7740 10 client.25646 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.486+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5604ba1b4150 con 0x5604b9d281c0 2024-04-25T13:44:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.487+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1947183605 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b4051070 con 0x5604b9d281c0 2024-04-25T13:44:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.487+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1947183605 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f26b802c030 con 0x5604b9d281c0 2024-04-25T13:44:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.487+0000 7f26d11a7740 10 client.25646 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.487+0000 7f26d11a7740 10 client.25646 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.487+0000 7f26d11a7740 20 client.25646 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.487+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604ba1b4ba0 0x5604ba1b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.487+0000 7f26d11a7740 4 client.25646 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:44:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.487+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5604ba0a1ee0 con 0x5604ba1b4ba0 2024-04-25T13:44:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.488+0000 7f26c37fe640 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604ba1b4ba0 0x5604ba1b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.488+0000 7f26a2ffd640 10 client.25646 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.489+0000 7f26c37fe640 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604ba1b4ba0 0x5604ba1b6fc0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f426cd8aee6957c1 server_cookie=5e7329f1bbc25d6f in_seq=0 out_seq=0 2024-04-25T13:44:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.511+0000 7f26a2ffd640 1 -- 172.21.15.27:0/1947183605 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5604ba0a1ee0 con 0x5604ba1b4ba0 2024-04-25T13:44:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.511+0000 7f26a2ffd640 10 client.25646 handle_command_reply: tid=0 2024-04-25T13:44:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.511+0000 7f26d11a7740 1 client.25646 shutdown 2024-04-25T13:44:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 msgr2=0x5604ba143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba143480 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f26b8025010 tx=0x7f26b4009640 comp rx=0 tx=0).stop 2024-04-25T13:44:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 shutdown_connections 2024-04-25T13:44:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604ba1b4ba0 0x5604ba1b6fc0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604b9d941b0 0x5604ba144880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26a2ffd640 0 client.25646 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604b9d6b2f0 0x5604ba1439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 --2- 172.21.15.27:0/1947183605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604b9d281c0 0x5604ba143480 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 >> 172.21.15.27:0/1947183605 conn(0x5604b9d88300 msgr2=0x5604ba133100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 shutdown_connections 2024-04-25T13:44:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 1 -- 172.21.15.27:0/1947183605 wait complete. 2024-04-25T13:44:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.512+0000 7f26d11a7740 20 client.25646 trim_cache size 0 max 16384 2024-04-25T13:44:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:44:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:44:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:44:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:44:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 -- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f26ac0afbe0 msgr2=0x7f26ac0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f26ac0afbe0 0x7f26ac0b20f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f26bc0a45f0 tx=0x7f26bc004a40 comp rx=0 tx=0).stop 2024-04-25T13:44:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 -- 172.21.15.27:0/92285908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26c4109a80 msgr2=0x7f26c41a7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26c4109a80 0x7f26c41a7d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00cab0 tx=0x7f26bc00cf70 comp rx=0 tx=0).stop 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 -- 172.21.15.27:0/92285908 shutdown_connections 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f26ac0afbe0 0x7f26ac0b20f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26c4109a80 0x7f26c41a7d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c4104d80 0x7f26c41a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 --2- 172.21.15.27:0/92285908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26c4103b40 0x7f26c41a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.513+0000 7f26ca215640 1 -- 172.21.15.27:0/92285908 >> 172.21.15.27:0/92285908 conn(0x7f26c40fed30 msgr2=0x7f26c4101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.514+0000 7f26ca215640 1 -- 172.21.15.27:0/92285908 shutdown_connections 2024-04-25T13:44:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:00.515+0000 7f26ca215640 1 -- 172.21.15.27:0/92285908 wait complete. 2024-04-25T13:44:01.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:01 smithi045 ceph-mon[32731]: pgmap v443: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 823 op/s 2024-04-25T13:44:01.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:01 smithi027 ceph-mon[29140]: pgmap v443: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 823 op/s 2024-04-25T13:44:02.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:01 smithi122 ceph-mon[32843]: pgmap v443: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 823 op/s 2024-04-25T13:44:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 175.0 s, 3376.6 tps, lat 29.649 ms stddev 34.898 2024-04-25T13:44:03.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:03 smithi045 ceph-mon[32731]: pgmap v444: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 31 MiB/s wr, 875 op/s 2024-04-25T13:44:03.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:03 smithi027 ceph-mon[29140]: pgmap v444: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 31 MiB/s wr, 875 op/s 2024-04-25T13:44:04.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:03 smithi122 ceph-mon[32843]: pgmap v444: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 31 MiB/s wr, 875 op/s 2024-04-25T13:44:05.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:05 smithi045 ceph-mon[32731]: pgmap v445: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 MiB/s wr, 825 op/s 2024-04-25T13:44:05.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:05 smithi027 ceph-mon[29140]: pgmap v445: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 MiB/s wr, 825 op/s 2024-04-25T13:44:06.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:05 smithi122 ceph-mon[32843]: pgmap v445: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 MiB/s wr, 825 op/s 2024-04-25T13:44:07.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:07 smithi027 ceph-mon[29140]: pgmap v446: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 46 MiB/s wr, 984 op/s 2024-04-25T13:44:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 180.0 s, 3767.0 tps, lat 26.551 ms stddev 32.572 2024-04-25T13:44:08.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:07 smithi122 ceph-mon[32843]: pgmap v446: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 46 MiB/s wr, 984 op/s 2024-04-25T13:44:08.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:07 smithi045 ceph-mon[32731]: pgmap v446: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 46 MiB/s wr, 984 op/s 2024-04-25T13:44:08.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:44:08.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:44:08.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.534+0000 7fa529f08640 1 -- 172.21.15.27:0/3863496933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5241029e0 msgr2=0x7fa5241036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.534+0000 7fa529f08640 1 --2- 172.21.15.27:0/3863496933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5241029e0 0x7fa5241036e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009ca0 tx=0x7fa51c02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.534+0000 7fa529f08640 1 -- 172.21.15.27:0/3863496933 shutdown_connections 2024-04-25T13:44:08.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.535+0000 7fa529f08640 1 --2- 172.21.15.27:0/3863496933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa524108920 0x7fa52410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.535+0000 7fa529f08640 1 --2- 172.21.15.27:0/3863496933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa524103c20 0x7fa524104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.535+0000 7fa529f08640 1 --2- 172.21.15.27:0/3863496933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5241029e0 0x7fa5241036e0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009ca0 tx=0x7fa51c02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.535+0000 7fa529f08640 1 -- 172.21.15.27:0/3863496933 >> 172.21.15.27:0/3863496933 conn(0x7fa5240fdbd0 msgr2=0x7fa524100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:08.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.536+0000 7fa529f08640 1 -- 172.21.15.27:0/3863496933 shutdown_connections 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.536+0000 7fa529f08640 1 -- 172.21.15.27:0/3863496933 wait complete. 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa529f08640 1 Processor -- start 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa529f08640 1 -- start start 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa529f08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa524103c20 0x7fa52406a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa529f08640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa524108920 0x7fa52419f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa529f08640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52419f8d0 0x7fa5241a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa529f08640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa52410e4e0 con 0x7fa524103c20 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa529f08640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa52410e360 con 0x7fa52419f8d0 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa529f08640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa52410e660 con 0x7fa524108920 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa5237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa524103c20 0x7fa52406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa524108920 0x7fa52419f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa523fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52419f8d0 0x7fa5241a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa524108920 0x7fa52419f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:53412/0 (socket says 172.21.15.27:53412) 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa522ffd640 1 -- 172.21.15.27:0/1796512700 learned_addr learned my addr 172.21.15.27:0/1796512700 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.537+0000 7fa5237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa524103c20 0x7fa52406a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45636/0 (socket says 172.21.15.27:45636) 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa522ffd640 1 -- 172.21.15.27:0/1796512700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52419f8d0 msgr2=0x7fa5241a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa522ffd640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52419f8d0 0x7fa5241a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa522ffd640 1 -- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa524103c20 msgr2=0x7fa52406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa522ffd640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa524103c20 0x7fa52406a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa522ffd640 1 -- 172.21.15.27:0/1796512700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5241a2600 con 0x7fa524108920 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa522ffd640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa524108920 0x7fa52419f340 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa51400ea60 tx=0x7fa51400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:08.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa520ff9640 1 -- 172.21.15.27:0/1796512700 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51400cde0 con 0x7fa524108920 2024-04-25T13:44:08.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa520ff9640 1 -- 172.21.15.27:0/1796512700 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5140035f0 con 0x7fa524108920 2024-04-25T13:44:08.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa520ff9640 1 -- 172.21.15.27:0/1796512700 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514010690 con 0x7fa524108920 2024-04-25T13:44:08.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa529f08640 1 -- 172.21.15.27:0/1796512700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5241a28f0 con 0x7fa524108920 2024-04-25T13:44:08.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.538+0000 7fa529f08640 1 -- 172.21.15.27:0/1796512700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5241a2dc0 con 0x7fa524108920 2024-04-25T13:44:08.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa520ff9640 1 -- 172.21.15.27:0/1796512700 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa5140037a0 con 0x7fa524108920 2024-04-25T13:44:08.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa530db7740 1 Processor -- start 2024-04-25T13:44:08.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa530db7740 1 -- start start 2024-04-25T13:44:08.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa530db7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42effaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa530db7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42effafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa530db7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42effbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa530db7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f42ef5fd80 con 0x55f42ec304d0 2024-04-25T13:44:08.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa530db7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f42ef198b0 con 0x55f42ec386b0 2024-04-25T13:44:08.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa530db7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f42ef5bd40 con 0x55f42ec59390 2024-04-25T13:44:08.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.542+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42effbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.543+0000 7fa523fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42effaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.543+0000 7fa523fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42effaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:53428/0 (socket says 172.21.15.27:53428) 2024-04-25T13:44:08.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.543+0000 7fa523fff640 1 -- 172.21.15.27:0/2613839770 learned_addr learned my addr 172.21.15.27:0/2613839770 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.543+0000 7fa5237fe640 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42effafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.543+0000 7fa520ff9640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa50c0afb70 0x7fa50c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.544+0000 7fa5237fe640 1 -- 172.21.15.27:0/2613839770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 msgr2=0x55f42effaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.544+0000 7fa5237fe640 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42effaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.544+0000 7fa5237fe640 1 -- 172.21.15.27:0/2613839770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 msgr2=0x55f42effbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.544+0000 7fa523fff640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa50c0afb70 0x7fa50c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.544+0000 7fa5237fe640 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42effbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.544+0000 7fa5237fe640 1 -- 172.21.15.27:0/2613839770 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f42eeeaa40 con 0x55f42ec304d0 2024-04-25T13:44:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.544+0000 7fa522ffd640 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42effbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:44:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.544+0000 7fa520ff9640 1 -- 172.21.15.27:0/1796512700 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5140deec0 con 0x7fa524108920 2024-04-25T13:44:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.545+0000 7fa5237fe640 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42effafe0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa524102080 tx=0x7fa51c02f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddba51c4efb6e5df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.545+0000 7fa509ffb640 1 -- 172.21.15.27:0/2613839770 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c031bf0 con 0x55f42ec304d0 2024-04-25T13:44:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.545+0000 7fa509ffb640 1 -- 172.21.15.27:0/2613839770 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c036070 con 0x55f42ec304d0 2024-04-25T13:44:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.545+0000 7fa509ffb640 1 -- 172.21.15.27:0/2613839770 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0042a0 con 0x55f42ec304d0 2024-04-25T13:44:08.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 -- 172.21.15.27:0/2613839770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 msgr2=0x55f42effafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42effafe0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa524102080 tx=0x7fa51c02f310 comp rx=0 tx=0).stop 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa523fff640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa50c0afb70 0x7fa50c0b2080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa518009fd0 tx=0x7fa518004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 -- 172.21.15.27:0/2613839770 shutdown_connections 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42effbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42effafe0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 --2- 172.21.15.27:0/2613839770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42effaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 -- 172.21.15.27:0/2613839770 >> 172.21.15.27:0/2613839770 conn(0x55f42ec4d4e0 msgr2=0x55f42eff6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 -- 172.21.15.27:0/2613839770 shutdown_connections 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 -- 172.21.15.27:0/2613839770 wait complete. 2024-04-25T13:44:08.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.546+0000 7fa530db7740 1 -- start start 2024-04-25T13:44:08.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa530db7740 1 Processor -- start 2024-04-25T13:44:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa530db7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f004710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa530db7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42f004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa530db7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42f005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa530db7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f42effc5a0 con 0x55f42ec386b0 2024-04-25T13:44:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa530db7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f42eee6d10 con 0x55f42ec59390 2024-04-25T13:44:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa530db7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f42effc720 con 0x55f42ec304d0 2024-04-25T13:44:08.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa530db7740 10 client.0 _ll_register_callbacks cb 0x55f42ef13210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42f004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42f004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45650/0 (socket says 172.21.15.27:45650) 2024-04-25T13:44:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa522ffd640 1 -- 172.21.15.27:0/45072087 learned_addr learned my addr 172.21.15.27:0/45072087 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa5237fe640 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f004710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.547+0000 7fa523fff640 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42f005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.548+0000 7fa5237fe640 1 -- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 msgr2=0x55f42f005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.548+0000 7fa5237fe640 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42f005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.548+0000 7fa5237fe640 1 -- 172.21.15.27:0/45072087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 msgr2=0x55f42f004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.548+0000 7fa5237fe640 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42f004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.548+0000 7fa5237fe640 1 -- 172.21.15.27:0/45072087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f42ed3b7d0 con 0x55f42ec304d0 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.548+0000 7fa523fff640 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42f005b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.548+0000 7fa5237fe640 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f004710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa51c005e30 tx=0x7fa51c02b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=567e8b6941eca851 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.548+0000 7fa502ffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.549+0000 7fa502ffd640 1 -- 172.21.15.27:0/45072087 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c031650 con 0x55f42ec304d0 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.549+0000 7fa530db7740 1 -- 172.21.15.27:0/45072087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa51c036070 con 0x55f42ec304d0 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.549+0000 7fa502ffd640 1 -- 172.21.15.27:0/45072087 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa51c003f60 con 0x55f42ec304d0 2024-04-25T13:44:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.549+0000 7fa502ffd640 1 -- 172.21.15.27:0/45072087 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa51c0317f0 con 0x55f42ec304d0 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.549+0000 7fa530db7740 10 client.25678 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.549+0000 7fa530db7740 1 -- 172.21.15.27:0/45072087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f42ec713f0 con 0x55f42ec304d0 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.549+0000 7fa502ffd640 1 -- 172.21.15.27:0/45072087 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c036070 con 0x55f42ec304d0 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa502ffd640 1 -- 172.21.15.27:0/45072087 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa51c043020 con 0x55f42ec304d0 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa530db7740 10 client.25678 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa530db7740 10 client.25678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa530db7740 20 client.25678 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa530db7740 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f42ec17f60 0x55f42f01a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa530db7740 4 client.25678 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa530db7740 1 -- 172.21.15.27:0/45072087 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f42ef6d990 con 0x55f42ec17f60 2024-04-25T13:44:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa522ffd640 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f42ec17f60 0x55f42f01a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa502ffd640 10 client.25678 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.550+0000 7fa522ffd640 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f42ec17f60 0x55f42f01a750 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1fbc1d006f6de01 server_cookie=b4986776b10293d6 in_seq=0 out_seq=0 2024-04-25T13:44:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.551+0000 7fa502ffd640 1 -- 172.21.15.27:0/45072087 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f42ef6d990 con 0x55f42ec17f60 2024-04-25T13:44:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.551+0000 7fa502ffd640 10 client.25678 handle_command_reply: tid=0 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.551+0000 7fa530db7740 1 client.25678 shutdown 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 -- 172.21.15.27:0/45072087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 msgr2=0x55f42f004710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f004710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa51c005e30 tx=0x7fa51c02b040 comp rx=0 tx=0).stop 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 -- 172.21.15.27:0/45072087 shutdown_connections 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f42ec17f60 0x55f42f01a750 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42f005b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42f004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 --2- 172.21.15.27:0/45072087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f004710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa502ffd640 0 client.25678 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 -- 172.21.15.27:0/45072087 >> 172.21.15.27:0/45072087 conn(0x55f42ec4d4e0 msgr2=0x55f42eff9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 -- 172.21.15.27:0/45072087 shutdown_connections 2024-04-25T13:44:08.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 1 -- 172.21.15.27:0/45072087 wait complete. 2024-04-25T13:44:08.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.553+0000 7fa530db7740 20 client.25678 trim_cache size 0 max 16384 2024-04-25T13:44:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.579+0000 7fa530db7740 1 Processor -- start 2024-04-25T13:44:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.579+0000 7fa530db7740 1 -- start start 2024-04-25T13:44:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa530db7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f0061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa530db7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42eff5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa530db7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42eff6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa530db7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241a7530 con 0x55f42ec386b0 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa530db7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241a9d90 con 0x55f42ec59390 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa530db7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241a7b80 con 0x55f42ec304d0 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42eff5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa5237fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f0061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42eff5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45666/0 (socket says 172.21.15.27:45666) 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa522ffd640 1 -- 172.21.15.27:0/2236842504 learned_addr learned my addr 172.21.15.27:0/2236842504 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa523fff640 1 --2- 172.21.15.27:0/2236842504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42eff6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa5237fe640 1 -- 172.21.15.27:0/2236842504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 msgr2=0x55f42eff6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa5237fe640 1 --2- 172.21.15.27:0/2236842504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42eff6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa5237fe640 1 -- 172.21.15.27:0/2236842504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 msgr2=0x55f42eff5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:08.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa5237fe640 1 --2- 172.21.15.27:0/2236842504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42eff5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.580+0000 7fa5237fe640 1 -- 172.21.15.27:0/2236842504 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f42ec310f0 con 0x55f42ec304d0 2024-04-25T13:44:08.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.581+0000 7fa5237fe640 1 --2- 172.21.15.27:0/2236842504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f0061b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa51c031a90 tx=0x7fa51c02f310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f0bc73da05dca6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:08.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.583+0000 7fa5097fa640 1 -- 172.21.15.27:0/2236842504 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c031650 con 0x55f42ec304d0 2024-04-25T13:44:08.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.583+0000 7fa5097fa640 1 -- 172.21.15.27:0/2236842504 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c036040 con 0x55f42ec304d0 2024-04-25T13:44:08.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.583+0000 7fa5097fa640 1 -- 172.21.15.27:0/2236842504 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c02fb20 con 0x55f42ec304d0 2024-04-25T13:44:08.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 -- 172.21.15.27:0/2236842504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 msgr2=0x55f42f0061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 --2- 172.21.15.27:0/2236842504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f0061b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa51c031a90 tx=0x7fa51c02f310 comp rx=0 tx=0).stop 2024-04-25T13:44:08.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 -- 172.21.15.27:0/2236842504 shutdown_connections 2024-04-25T13:44:08.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 --2- 172.21.15.27:0/2236842504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec59390 0x55f42eff6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 --2- 172.21.15.27:0/2236842504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec386b0 0x55f42eff5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 --2- 172.21.15.27:0/2236842504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec304d0 0x55f42f0061b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 -- 172.21.15.27:0/2236842504 >> 172.21.15.27:0/2236842504 conn(0x55f42ec4d4e0 msgr2=0x55f42f005aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 -- 172.21.15.27:0/2236842504 shutdown_connections 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.585+0000 7fa530db7740 1 -- 172.21.15.27:0/2236842504 wait complete. 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 1 -- start start 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 1 Processor -- start 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42f0124d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42f012a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42f080af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241a8250 con 0x55f42ec304d0 2024-04-25T13:44:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241a96c0 con 0x55f42ec386b0 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa5241a8920 con 0x55f42ec59390 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa530db7740 10 client.0 _ll_register_callbacks cb 0x7fa5180097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.586+0000 7fa5237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42f0124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa522ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42f012a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa523fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42f080af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa5237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42f0124d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45674/0 (socket says 172.21.15.27:45674) 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa5237fe640 1 -- 172.21.15.27:0/398563645 learned_addr learned my addr 172.21.15.27:0/398563645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa522ffd640 1 -- 172.21.15.27:0/398563645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 msgr2=0x55f42f080af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa522ffd640 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42f080af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa522ffd640 1 -- 172.21.15.27:0/398563645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 msgr2=0x55f42f0124d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa522ffd640 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42f0124d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa522ffd640 1 -- 172.21.15.27:0/398563645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f42ef204b0 con 0x55f42ec386b0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa522ffd640 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42f012a10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c00 tx=0x7fa5140a7260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26670044cee9cbfd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa502ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa502ffd640 1 -- 172.21.15.27:0/398563645 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5140b01d0 con 0x55f42ec386b0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa502ffd640 1 -- 172.21.15.27:0/398563645 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa514004000 con 0x55f42ec386b0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa530db7740 1 -- 172.21.15.27:0/398563645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa51c036040 con 0x55f42ec386b0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.587+0000 7fa502ffd640 1 -- 172.21.15.27:0/398563645 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa5140c8070 con 0x55f42ec386b0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa530db7740 10 client.25661 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa530db7740 1 -- 172.21.15.27:0/398563645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f42f082100 con 0x55f42ec386b0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa502ffd640 1 -- 172.21.15.27:0/398563645 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c036040 con 0x55f42ec386b0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa502ffd640 1 -- 172.21.15.27:0/398563645 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa5140ab270 con 0x55f42ec386b0 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa530db7740 10 client.25661 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa530db7740 10 client.25661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:08.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa530db7740 20 client.25661 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:08.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa530db7740 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f42ebed3a0 0x55f42f084af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa530db7740 4 client.25661 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:44:08.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa530db7740 1 -- 172.21.15.27:0/398563645 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f42ef67800 con 0x55f42ebed3a0 2024-04-25T13:44:08.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.588+0000 7fa5237fe640 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f42ebed3a0 0x55f42f084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.589+0000 7fa502ffd640 10 client.25661 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.590+0000 7fa5237fe640 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f42ebed3a0 0x55f42f084af0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccb0afea951b3bbc server_cookie=5c18cea050e940f6 in_seq=0 out_seq=0 2024-04-25T13:44:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.590+0000 7fa502ffd640 1 -- 172.21.15.27:0/398563645 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f42ef67800 con 0x55f42ebed3a0 2024-04-25T13:44:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.590+0000 7fa502ffd640 10 client.25661 handle_command_reply: tid=0 2024-04-25T13:44:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.590+0000 7fa530db7740 1 client.25661 shutdown 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 -- 172.21.15.27:0/398563645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 msgr2=0x55f42f012a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42f012a10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c00 tx=0x7fa5140a7260 comp rx=0 tx=0).stop 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 -- 172.21.15.27:0/398563645 shutdown_connections 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f42ebed3a0 0x55f42f084af0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f42ec59390 0x55f42f080af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f42ec386b0 0x55f42f012a10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 --2- 172.21.15.27:0/398563645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f42ec304d0 0x55f42f0124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 -- 172.21.15.27:0/398563645 >> 172.21.15.27:0/398563645 conn(0x55f42ec4d4e0 msgr2=0x55f42eff5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa502ffd640 0 client.25661 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 -- 172.21.15.27:0/398563645 shutdown_connections 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 1 -- 172.21.15.27:0/398563645 wait complete. 2024-04-25T13:44:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.591+0000 7fa530db7740 20 client.25661 trim_cache size 0 max 16384 2024-04-25T13:44:08.595 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:44:08.595 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:44:08.595 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:44:08.595 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 -- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa50c0afb70 msgr2=0x7fa50c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa50c0afb70 0x7fa50c0b2080 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa518009fd0 tx=0x7fa518004730 comp rx=0 tx=0).stop 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 -- 172.21.15.27:0/1796512700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa524108920 msgr2=0x7fa52419f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa524108920 0x7fa52419f340 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa51400ea60 tx=0x7fa51400ef20 comp rx=0 tx=0).stop 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 -- 172.21.15.27:0/1796512700 shutdown_connections 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa50c0afb70 0x7fa50c0b2080 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52419f8d0 0x7fa5241a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa524108920 0x7fa52419f340 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 --2- 172.21.15.27:0/1796512700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa524103c20 0x7fa52406a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 -- 172.21.15.27:0/1796512700 >> 172.21.15.27:0/1796512700 conn(0x7fa5240fdbd0 msgr2=0x7fa524104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.592+0000 7fa529f08640 1 -- 172.21.15.27:0/1796512700 shutdown_connections 2024-04-25T13:44:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.593+0000 7fa529f08640 1 -- 172.21.15.27:0/1796512700 wait complete. 2024-04-25T13:44:08.609 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:44:08.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:44:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 -- 172.21.15.27:0/3482976733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430106f10 msgr2=0x7f0430109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 --2- 172.21.15.27:0/3482976733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430106f10 0x7f0430109330 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0424009900 tx=0x7f042402a400 comp rx=0 tx=0).stop 2024-04-25T13:44:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 -- 172.21.15.27:0/3482976733 shutdown_connections 2024-04-25T13:44:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 --2- 172.21.15.27:0/3482976733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0430109870 0x7f043010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 --2- 172.21.15.27:0/3482976733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430106f10 0x7f0430109330 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 --2- 172.21.15.27:0/3482976733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04301045b0 0x7f04301069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 -- 172.21.15.27:0/3482976733 >> 172.21.15.27:0/3482976733 conn(0x7f04300fdbd0 msgr2=0x7f0430100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 -- 172.21.15.27:0/3482976733 shutdown_connections 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.768+0000 7f0435fb2640 1 -- 172.21.15.27:0/3482976733 wait complete. 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.769+0000 7f0435fb2640 1 Processor -- start 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f0435fb2640 1 -- start start 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f0435fb2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04301045b0 0x7f043019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f0435fb2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0430106f10 0x7f043019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f0435fb2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430109870 0x7f04301a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f042f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04301045b0 0x7f043019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f042f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04301045b0 0x7f043019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45686/0 (socket says 172.21.15.27:45686) 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f042f7fe640 1 -- 172.21.15.27:0/2056848483 learned_addr learned my addr 172.21.15.27:0/2056848483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0430111860 con 0x7f04301045b0 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04301116e0 con 0x7f0430106f10 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f04301119e0 con 0x7f0430109870 2024-04-25T13:44:08.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f042effd640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0430106f10 0x7f043019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.770+0000 7f042ffff640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430109870 0x7f04301a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042effd640 1 -- 172.21.15.27:0/2056848483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430109870 msgr2=0x7f04301a3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042effd640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430109870 0x7f04301a3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042effd640 1 -- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04301045b0 msgr2=0x7f043019c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042effd640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04301045b0 0x7f043019c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042effd640 1 -- 172.21.15.27:0/2056848483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04301a3f80 con 0x7f0430106f10 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042f7fe640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04301045b0 0x7f043019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042ffff640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430109870 0x7f04301a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042effd640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0430106f10 0x7f043019c980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0424006010 tx=0x7f0424004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042cff9640 1 -- 172.21.15.27:0/2056848483 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042403f070 con 0x7f0430106f10 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042cff9640 1 -- 172.21.15.27:0/2056848483 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0424003ca0 con 0x7f0430106f10 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f042cff9640 1 -- 172.21.15.27:0/2056848483 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424035070 con 0x7f0430106f10 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04301a4210 con 0x7f0430106f10 2024-04-25T13:44:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.771+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04301a47d0 con 0x7f0430106f10 2024-04-25T13:44:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.772+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04300ff6f0 con 0x7f0430106f10 2024-04-25T13:44:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.774+0000 7f042cff9640 1 -- 172.21.15.27:0/2056848483 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f042403b4f0 con 0x7f0430106f10 2024-04-25T13:44:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.776+0000 7f042cff9640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04180afbf0 0x7f04180b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.776+0000 7f042cff9640 1 -- 172.21.15.27:0/2056848483 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f04240ff690 con 0x7f0430106f10 2024-04-25T13:44:08.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.776+0000 7f042f7fe640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04180afbf0 0x7f04180b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:08.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.776+0000 7f042f7fe640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04180afbf0 0x7f04180b2100 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f04200058c0 tx=0x7f0420005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:08.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:08.778+0000 7f042cff9640 1 -- 172.21.15.27:0/2056848483 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04240c7bf0 con 0x7f0430106f10 2024-04-25T13:44:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.187+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f043002db50 con 0x7f0430106f10 2024-04-25T13:44:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.188+0000 7f042cff9640 1 -- 172.21.15.27:0/2056848483 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f04240d02c0 con 0x7f0430106f10 2024-04-25T13:44:09.189 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:44:09.189 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:44:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:44:09.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.192+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04180afbf0 msgr2=0x7f04180b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.192+0000 7f0435fb2640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04180afbf0 0x7f04180b2100 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f04200058c0 tx=0x7f0420005f90 comp rx=0 tx=0).stop 2024-04-25T13:44:09.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.192+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0430106f10 msgr2=0x7f043019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.192+0000 7f0435fb2640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0430106f10 0x7f043019c980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0424006010 tx=0x7f0424004270 comp rx=0 tx=0).stop 2024-04-25T13:44:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.193+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 shutdown_connections 2024-04-25T13:44:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.193+0000 7f0435fb2640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f04180afbf0 0x7f04180b2100 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.193+0000 7f0435fb2640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0430109870 0x7f04301a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.193+0000 7f0435fb2640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0430106f10 0x7f043019c980 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.193+0000 7f0435fb2640 1 --2- 172.21.15.27:0/2056848483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04301045b0 0x7f043019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.193+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 >> 172.21.15.27:0/2056848483 conn(0x7f04300fdbd0 msgr2=0x7f0430105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.193+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 shutdown_connections 2024-04-25T13:44:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.193+0000 7f0435fb2640 1 -- 172.21.15.27:0/2056848483 wait complete. 2024-04-25T13:44:09.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:44:09.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.397+0000 7f14227fc640 1 -- 172.21.15.27:0/3547195488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14140318a0 con 0x7f142410a420 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.398+0000 7f142a164640 1 -- 172.21.15.27:0/3547195488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f142410a420 msgr2=0x7f142410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.398+0000 7f142a164640 1 --2- 172.21.15.27:0/3547195488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f142410a420 0x7f142410add0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1414009900 tx=0x7f141402a410 comp rx=0 tx=0).stop 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.398+0000 7f142a164640 1 -- 172.21.15.27:0/3547195488 shutdown_connections 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.398+0000 7f142a164640 1 --2- 172.21.15.27:0/3547195488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1424110030 0x7f1424110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.398+0000 7f142a164640 1 --2- 172.21.15.27:0/3547195488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f142410b310 0x7f142410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.398+0000 7f142a164640 1 --2- 172.21.15.27:0/3547195488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f142410a420 0x7f142410add0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.398+0000 7f142a164640 1 -- 172.21.15.27:0/3547195488 >> 172.21.15.27:0/3547195488 conn(0x7f14240704e0 msgr2=0x7f1424072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.399+0000 7f142a164640 1 -- 172.21.15.27:0/3547195488 shutdown_connections 2024-04-25T13:44:09.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.399+0000 7f142a164640 1 -- 172.21.15.27:0/3547195488 wait complete. 2024-04-25T13:44:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f142a164640 1 Processor -- start 2024-04-25T13:44:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f142a164640 1 -- start start 2024-04-25T13:44:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f142a164640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f142410a420 0x7f14241a9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f142a164640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142410b310 0x7f14241a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f142a164640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1424110030 0x7f14241b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f142a164640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f142411e520 con 0x7f1424110030 2024-04-25T13:44:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f142a164640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f142411e3a0 con 0x7f142410b310 2024-04-25T13:44:09.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f142a164640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f142411e6a0 con 0x7f142410a420 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f1423fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1424110030 0x7f14241b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f1423fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1424110030 0x7f14241b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45704/0 (socket says 172.21.15.27:45704) 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f14237fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f142410a420 0x7f14241a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f1423fff640 1 -- 172.21.15.27:0/665227870 learned_addr learned my addr 172.21.15.27:0/665227870 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.400+0000 7f1422ffd640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142410b310 0x7f14241a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1423fff640 1 -- 172.21.15.27:0/665227870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f142410a420 msgr2=0x7f14241a9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1423fff640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f142410a420 0x7f14241a9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1423fff640 1 -- 172.21.15.27:0/665227870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142410b310 msgr2=0x7f14241a9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1423fff640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142410b310 0x7f14241a9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1423fff640 1 -- 172.21.15.27:0/665227870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1424111c90 con 0x7f1424110030 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f14237fe640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f142410a420 0x7f14241a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1423fff640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1424110030 0x7f14241b0600 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f140c009fd0 tx=0x7f140c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1420ff9640 1 -- 172.21.15.27:0/665227870 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c011830 con 0x7f1424110030 2024-04-25T13:44:09.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1420ff9640 1 -- 172.21.15.27:0/665227870 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f140c016070 con 0x7f1424110030 2024-04-25T13:44:09.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f142a164640 1 -- 172.21.15.27:0/665227870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1424111f80 con 0x7f1424110030 2024-04-25T13:44:09.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f142a164640 1 -- 172.21.15.27:0/665227870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1424112550 con 0x7f1424110030 2024-04-25T13:44:09.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.401+0000 7f1420ff9640 1 -- 172.21.15.27:0/665227870 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c010de0 con 0x7f1424110030 2024-04-25T13:44:09.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.405+0000 7f1431015740 1 Processor -- start 2024-04-25T13:44:09.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.405+0000 7f1420ff9640 1 -- 172.21.15.27:0/665227870 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f140c01b4a0 con 0x7f1424110030 2024-04-25T13:44:09.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.405+0000 7f1431015740 1 -- start start 2024-04-25T13:44:09.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f1431015740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 0x561ef4b0e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f1431015740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b0ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f1431015740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 0x561ef4b0fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f1431015740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561ef4a72d80 con 0x561ef47434d0 2024-04-25T13:44:09.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f1431015740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561ef4a2ca20 con 0x561ef476c390 2024-04-25T13:44:09.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f1431015740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561ef4a6ed40 con 0x561ef474b6b0 2024-04-25T13:44:09.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f14237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b0ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f14237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b0ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45716/0 (socket says 172.21.15.27:45716) 2024-04-25T13:44:09.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f14237fe640 1 -- 172.21.15.27:0/1339565790 learned_addr learned my addr 172.21.15.27:0/1339565790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:09.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f14237fe640 1 -- 172.21.15.27:0/1339565790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 msgr2=0x561ef4b0fa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f14237fe640 1 --2- 172.21.15.27:0/1339565790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 0x561ef4b0fa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f14237fe640 1 -- 172.21.15.27:0/1339565790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 msgr2=0x561ef4b0e810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f14237fe640 1 --2- 172.21.15.27:0/1339565790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 0x561ef4b0e810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.406+0000 7f14237fe640 1 -- 172.21.15.27:0/1339565790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ef4a37c30 con 0x561ef47434d0 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.407+0000 7f1420ff9640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14100afc30 0x7f14100b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.407+0000 7f14237fe640 1 --2- 172.21.15.27:0/1339565790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b0ed90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1414005710 tx=0x7f1414004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f609486bdf810e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.407+0000 7f1420ff9640 1 -- 172.21.15.27:0/665227870 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f140c0de110 con 0x7f1424110030 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.407+0000 7f1419ffb640 1 -- 172.21.15.27:0/1339565790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1414040070 con 0x561ef47434d0 2024-04-25T13:44:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.408+0000 7f1422ffd640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14100afc30 0x7f14100b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.409+0000 7f1419ffb640 1 -- 172.21.15.27:0/1339565790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1414003e50 con 0x561ef47434d0 2024-04-25T13:44:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.410+0000 7f1419ffb640 1 -- 172.21.15.27:0/1339565790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1414035040 con 0x561ef47434d0 2024-04-25T13:44:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.410+0000 7f1431015740 1 -- 172.21.15.27:0/1339565790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 msgr2=0x561ef4b0ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.410+0000 7f1431015740 1 --2- 172.21.15.27:0/1339565790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b0ed90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1414005710 tx=0x7f1414004610 comp rx=0 tx=0).stop 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.410+0000 7f1422ffd640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14100afc30 0x7f14100b2140 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f141c0099a0 tx=0x7f141c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.410+0000 7f1431015740 1 -- 172.21.15.27:0/1339565790 shutdown_connections 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.410+0000 7f1431015740 1 --2- 172.21.15.27:0/1339565790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 0x561ef4b0fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.410+0000 7f1431015740 1 --2- 172.21.15.27:0/1339565790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b0ed90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.410+0000 7f1431015740 1 --2- 172.21.15.27:0/1339565790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 0x561ef4b0e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.411+0000 7f1431015740 1 -- 172.21.15.27:0/1339565790 >> 172.21.15.27:0/1339565790 conn(0x561ef47604e0 msgr2=0x561ef4b0a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.411+0000 7f1431015740 1 -- 172.21.15.27:0/1339565790 shutdown_connections 2024-04-25T13:44:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.411+0000 7f1431015740 1 -- 172.21.15.27:0/1339565790 wait complete. 2024-04-25T13:44:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.411+0000 7f1431015740 1 -- start start 2024-04-25T13:44:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.411+0000 7f1431015740 1 Processor -- start 2024-04-25T13:44:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1431015740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b184f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1431015740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 0x561ef4b18a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1431015740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 0x561ef4b1f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1431015740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561ef4b10570 con 0x561ef47434d0 2024-04-25T13:44:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1431015740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561ef49f9dc0 con 0x561ef476c390 2024-04-25T13:44:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1431015740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561ef4b106f0 con 0x561ef474b6b0 2024-04-25T13:44:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1431015740 10 client.0 _ll_register_callbacks cb 0x561ef4a15600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f14237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b184f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f14237fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b184f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45720/0 (socket says 172.21.15.27:45720) 2024-04-25T13:44:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f14237fe640 1 -- 172.21.15.27:0/131433252 learned_addr learned my addr 172.21.15.27:0/131433252 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1422ffd640 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 0x561ef4b18a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.413+0000 7f1423fff640 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 0x561ef4b1f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.414+0000 7f14237fe640 1 -- 172.21.15.27:0/131433252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 msgr2=0x561ef4b18a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.414+0000 7f14237fe640 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 0x561ef4b18a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.414+0000 7f14237fe640 1 -- 172.21.15.27:0/131433252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 msgr2=0x561ef4b1f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.414+0000 7f14237fe640 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 0x561ef4b1f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.414+0000 7f14237fe640 1 -- 172.21.15.27:0/131433252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ef47843f0 con 0x561ef47434d0 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.414+0000 7f1422ffd640 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 0x561ef4b18a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.414+0000 7f14237fe640 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b184f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f141402aa00 tx=0x7f14140317d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93972c5d059fef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.414+0000 7f1402ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.415+0000 7f1402ffd640 1 -- 172.21.15.27:0/131433252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141404b070 con 0x561ef47434d0 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.415+0000 7f1402ffd640 1 -- 172.21.15.27:0/131433252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1414003bb0 con 0x561ef47434d0 2024-04-25T13:44:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.415+0000 7f1431015740 1 -- 172.21.15.27:0/131433252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1414003e50 con 0x561ef47434d0 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.415+0000 7f1402ffd640 1 -- 172.21.15.27:0/131433252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1414057070 con 0x561ef47434d0 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.415+0000 7f1431015740 10 client.15966 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.415+0000 7f1431015740 1 -- 172.21.15.27:0/131433252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x561ef484e7d0 con 0x561ef47434d0 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.416+0000 7f1402ffd640 1 -- 172.21.15.27:0/131433252 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1414003e50 con 0x561ef47434d0 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.416+0000 7f1402ffd640 1 -- 172.21.15.27:0/131433252 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1414045460 con 0x561ef47434d0 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.416+0000 7f1431015740 10 client.15966 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.416+0000 7f1431015740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.416+0000 7f1431015740 20 client.15966 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.416+0000 7f1431015740 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561ef472af60 0x561ef4b263a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.416+0000 7f1431015740 4 client.15966 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.416+0000 7f1431015740 1 -- 172.21.15.27:0/131433252 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ef4a80990 con 0x561ef472af60 2024-04-25T13:44:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.417+0000 7f1422ffd640 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561ef472af60 0x561ef4b263a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.417+0000 7f1402ffd640 10 client.15966 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.418+0000 7f1422ffd640 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561ef472af60 0x561ef4b263a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d04ce2f10e5e52d1 server_cookie=279c988c9554aa0b in_seq=0 out_seq=0 2024-04-25T13:44:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.418+0000 7f1402ffd640 1 -- 172.21.15.27:0/131433252 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561ef4a80990 con 0x561ef472af60 2024-04-25T13:44:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.418+0000 7f1402ffd640 10 client.15966 handle_command_reply: tid=0 2024-04-25T13:44:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.418+0000 7f1431015740 1 client.15966 shutdown 2024-04-25T13:44:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1431015740 1 -- 172.21.15.27:0/131433252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 msgr2=0x561ef4b184f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1431015740 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b184f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f141402aa00 tx=0x7f14140317d0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1431015740 1 -- 172.21.15.27:0/131433252 shutdown_connections 2024-04-25T13:44:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1431015740 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561ef472af60 0x561ef4b263a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1431015740 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef476c390 0x561ef4b1f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1431015740 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef474b6b0 0x561ef4b18a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1431015740 1 --2- 172.21.15.27:0/131433252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef47434d0 0x561ef4b184f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1431015740 1 -- 172.21.15.27:0/131433252 >> 172.21.15.27:0/131433252 conn(0x561ef47604e0 msgr2=0x561ef4b0cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.419+0000 7f1402ffd640 0 client.15966 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.420+0000 7f1431015740 1 -- 172.21.15.27:0/131433252 shutdown_connections 2024-04-25T13:44:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.420+0000 7f1431015740 1 -- 172.21.15.27:0/131433252 wait complete. 2024-04-25T13:44:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.420+0000 7f1431015740 20 client.15966 trim_cache size 0 max 16384 2024-04-25T13:44:09.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1431015740 1 Processor -- start 2024-04-25T13:44:09.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1431015740 1 -- start start 2024-04-25T13:44:09.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1431015740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b16db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1431015740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b099e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1431015740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b09f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1431015740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f140400c360 con 0x561ef474b6b0 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1431015740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f140400ebe0 con 0x561ef47434d0 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1431015740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f140400c9d0 con 0x561ef476c390 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1422ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1422ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b099e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45728/0 (socket says 172.21.15.27:45728) 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1422ffd640 1 -- 172.21.15.27:0/243516365 learned_addr learned my addr 172.21.15.27:0/243516365 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f1423fff640 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b09f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.448+0000 7f14237fe640 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b16db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.449+0000 7f1423fff640 1 -- 172.21.15.27:0/243516365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 msgr2=0x561ef4b16db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.449+0000 7f1423fff640 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b16db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.449+0000 7f1423fff640 1 -- 172.21.15.27:0/243516365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 msgr2=0x561ef4b099e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.449+0000 7f1423fff640 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b099e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.449+0000 7f1423fff640 1 -- 172.21.15.27:0/243516365 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ef4b13b80 con 0x561ef476c390 2024-04-25T13:44:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.449+0000 7f1423fff640 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b09f20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f140c006fd0 tx=0x7f140c0dbab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c75a107eb680328 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.450+0000 7f14197fa640 1 -- 172.21.15.27:0/243516365 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c0a65c0 con 0x561ef476c390 2024-04-25T13:44:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.450+0000 7f14197fa640 1 -- 172.21.15.27:0/243516365 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f140c0aa970 con 0x561ef476c390 2024-04-25T13:44:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.450+0000 7f14197fa640 1 -- 172.21.15.27:0/243516365 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c0a1f30 con 0x561ef476c390 2024-04-25T13:44:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.451+0000 7f1431015740 1 -- 172.21.15.27:0/243516365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 msgr2=0x561ef4b09f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.451+0000 7f1431015740 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b09f20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f140c006fd0 tx=0x7f140c0dbab0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.451+0000 7f1431015740 1 -- 172.21.15.27:0/243516365 shutdown_connections 2024-04-25T13:44:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.451+0000 7f1431015740 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b09f20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.451+0000 7f1431015740 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.451+0000 7f1431015740 1 --2- 172.21.15.27:0/243516365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b16db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.451+0000 7f1431015740 1 -- 172.21.15.27:0/243516365 >> 172.21.15.27:0/243516365 conn(0x561ef47604e0 msgr2=0x561ef4b0e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.452+0000 7f1431015740 1 -- 172.21.15.27:0/243516365 shutdown_connections 2024-04-25T13:44:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.452+0000 7f1431015740 1 -- 172.21.15.27:0/243516365 wait complete. 2024-04-25T13:44:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.452+0000 7f1431015740 1 -- start start 2024-04-25T13:44:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.453+0000 7f1431015740 1 Processor -- start 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.453+0000 7f1431015740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b1eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.453+0000 7f1431015740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b1ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f14237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b1eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f1431015740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b1bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f1431015740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f140400d0a0 con 0x561ef474b6b0 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f1431015740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f140400e510 con 0x561ef47434d0 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f14237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b1eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38988/0 (socket says 172.21.15.27:38988) 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f1431015740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f140400d770 con 0x561ef476c390 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f14237fe640 1 -- 172.21.15.27:0/4084420226 learned_addr learned my addr 172.21.15.27:0/4084420226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f1422ffd640 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b1ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f1431015740 10 client.0 _ll_register_callbacks cb 0x7f14140035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f1423fff640 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b1bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f14237fe640 1 -- 172.21.15.27:0/4084420226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 msgr2=0x561ef4b1bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f14237fe640 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b1bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f14237fe640 1 -- 172.21.15.27:0/4084420226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 msgr2=0x561ef4b1ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f14237fe640 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b1ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.454+0000 7f14237fe640 1 -- 172.21.15.27:0/4084420226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ef4b8c700 con 0x561ef47434d0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.455+0000 7f14237fe640 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b1eec0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1414005710 tx=0x7f1414009300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8759b0300542244 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.455+0000 7f1402ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.455+0000 7f1402ffd640 1 -- 172.21.15.27:0/4084420226 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1414035040 con 0x561ef47434d0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.455+0000 7f1402ffd640 1 -- 172.21.15.27:0/4084420226 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1414045cb0 con 0x561ef47434d0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.455+0000 7f1431015740 1 -- 172.21.15.27:0/4084420226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f140c0aa970 con 0x561ef47434d0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.456+0000 7f1402ffd640 1 -- 172.21.15.27:0/4084420226 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1414040040 con 0x561ef47434d0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.456+0000 7f1431015740 10 client.25679 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.456+0000 7f1431015740 1 -- 172.21.15.27:0/4084420226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x561ef4b8d860 con 0x561ef47434d0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.456+0000 7f1402ffd640 1 -- 172.21.15.27:0/4084420226 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1414060020 con 0x561ef47434d0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.457+0000 7f1402ffd640 1 -- 172.21.15.27:0/4084420226 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1414050460 con 0x561ef47434d0 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.457+0000 7f1431015740 10 client.25679 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.457+0000 7f1431015740 10 client.25679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:09.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.457+0000 7f1431015740 20 client.25679 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.457+0000 7f1431015740 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561ef47003a0 0x561ef4b901d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.457+0000 7f1431015740 4 client.25679 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:44:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.457+0000 7f1431015740 1 -- 172.21.15.27:0/4084420226 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f140c0aa970 con 0x561ef47003a0 2024-04-25T13:44:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.457+0000 7f1422ffd640 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561ef47003a0 0x561ef4b901d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:09.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.458+0000 7f1422ffd640 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561ef47003a0 0x561ef4b901d0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb6f0676fe6990d9 server_cookie=c5512bb512d79c35 in_seq=0 out_seq=0 2024-04-25T13:44:09.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.458+0000 7f1402ffd640 10 client.25679 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:09.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.458+0000 7f1402ffd640 1 -- 172.21.15.27:0/4084420226 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f140c0aa970 con 0x561ef47003a0 2024-04-25T13:44:09.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.458+0000 7f1402ffd640 10 client.25679 handle_command_reply: tid=0 2024-04-25T13:44:09.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.458+0000 7f1431015740 1 client.25679 shutdown 2024-04-25T13:44:09.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.458+0000 7f1431015740 1 -- 172.21.15.27:0/4084420226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 msgr2=0x561ef4b1eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.458+0000 7f1431015740 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b1eec0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1414005710 tx=0x7f1414009300 comp rx=0 tx=0).stop 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 1 -- 172.21.15.27:0/4084420226 shutdown_connections 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561ef47003a0 0x561ef4b901d0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ef476c390 0x561ef4b1bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ef474b6b0 0x561ef4b1ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 1 --2- 172.21.15.27:0/4084420226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561ef47434d0 0x561ef4b1eec0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 1 -- 172.21.15.27:0/4084420226 >> 172.21.15.27:0/4084420226 conn(0x561ef47604e0 msgr2=0x561ef4b099e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1402ffd640 0 client.25679 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 1 -- 172.21.15.27:0/4084420226 shutdown_connections 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 1 -- 172.21.15.27:0/4084420226 wait complete. 2024-04-25T13:44:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.459+0000 7f1431015740 20 client.25679 trim_cache size 0 max 16384 2024-04-25T13:44:09.461 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:44:09.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 -- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14100afc30 msgr2=0x7f14100b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14100afc30 0x7f14100b2140 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f141c0099a0 tx=0x7f141c009500 comp rx=0 tx=0).stop 2024-04-25T13:44:09.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 -- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1424110030 msgr2=0x7f14241b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:09.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1424110030 0x7f14241b0600 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f140c009fd0 tx=0x7f140c00b8e0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 -- 172.21.15.27:0/665227870 shutdown_connections 2024-04-25T13:44:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14100afc30 0x7f14100b2140 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1424110030 0x7f14241b0600 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f142410b310 0x7f14241a9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 --2- 172.21.15.27:0/665227870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f142410a420 0x7f14241a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.461+0000 7f142a164640 1 -- 172.21.15.27:0/665227870 >> 172.21.15.27:0/665227870 conn(0x7f14240704e0 msgr2=0x7f142410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.462+0000 7f142a164640 1 -- 172.21.15.27:0/665227870 shutdown_connections 2024-04-25T13:44:09.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:09.462+0000 7f142a164640 1 -- 172.21.15.27:0/665227870 wait complete. 2024-04-25T13:44:09.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:09 smithi027 ceph-mon[29140]: pgmap v447: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 40 MiB/s wr, 781 op/s 2024-04-25T13:44:09.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:09 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2056848483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:44:10.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:09 smithi122 ceph-mon[32843]: pgmap v447: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 40 MiB/s wr, 781 op/s 2024-04-25T13:44:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:09 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2056848483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:44:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:09 smithi045 ceph-mon[32731]: pgmap v447: 97 pgs: 97 active+clean; 1.4 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 40 MiB/s wr, 781 op/s 2024-04-25T13:44:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:09 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2056848483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:44:10.490 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:44:10.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:44:10.490 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:44:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.675+0000 7f275357a640 1 -- 172.21.15.27:0/1649105842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 msgr2=0x7f274c1069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.675+0000 7f275357a640 1 --2- 172.21.15.27:0/1649105842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 0x7f274c1069f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2744009be0 tx=0x7f274402a6c0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.675+0000 7f275357a640 1 -- 172.21.15.27:0/1649105842 shutdown_connections 2024-04-25T13:44:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.675+0000 7f275357a640 1 --2- 172.21.15.27:0/1649105842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f274c10bc30 0x7f274c10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.675+0000 7f275357a640 1 --2- 172.21.15.27:0/1649105842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f274c106f30 0x7f274c107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.675+0000 7f275357a640 1 --2- 172.21.15.27:0/1649105842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 0x7f274c1069f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.675+0000 7f275357a640 1 -- 172.21.15.27:0/1649105842 >> 172.21.15.27:0/1649105842 conn(0x7f274c100f00 msgr2=0x7f274c103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.676+0000 7f275357a640 1 -- 172.21.15.27:0/1649105842 shutdown_connections 2024-04-25T13:44:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.676+0000 7f275357a640 1 -- 172.21.15.27:0/1649105842 wait complete. 2024-04-25T13:44:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f275357a640 1 Processor -- start 2024-04-25T13:44:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f275357a640 1 -- start start 2024-04-25T13:44:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f275357a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 0x7f274c06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f275357a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f274c106f30 0x7f274c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f275357a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f274c10bc30 0x7f274c073e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f275357a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f274c111820 con 0x7f274c105cf0 2024-04-25T13:44:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f275357a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1116a0 con 0x7f274c106f30 2024-04-25T13:44:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f275357a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1119a0 con 0x7f274c10bc30 2024-04-25T13:44:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f2751b17640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f274c10bc30 0x7f274c073e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f2751b17640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f274c10bc30 0x7f274c073e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:53546/0 (socket says 172.21.15.27:53546) 2024-04-25T13:44:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f2751316640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 0x7f274c06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.677+0000 7f2751b17640 1 -- 172.21.15.27:0/3889749154 learned_addr learned my addr 172.21.15.27:0/3889749154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.678+0000 7f2751316640 1 -- 172.21.15.27:0/3889749154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f274c10bc30 msgr2=0x7f274c073e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.678+0000 7f2751316640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f274c10bc30 0x7f274c073e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.678+0000 7f2751316640 1 -- 172.21.15.27:0/3889749154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f274c106f30 msgr2=0x7f274c06ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.678+0000 7f2751316640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f274c106f30 0x7f274c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.678+0000 7f2751316640 1 -- 172.21.15.27:0/3889749154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274c0743d0 con 0x7f274c105cf0 2024-04-25T13:44:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.679+0000 7f2751316640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 0x7f274c06c940 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2744034040 tx=0x7f2744030d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.679+0000 7f274a7fc640 1 -- 172.21.15.27:0/3889749154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744041020 con 0x7f274c105cf0 2024-04-25T13:44:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.680+0000 7f275357a640 1 -- 172.21.15.27:0/3889749154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274c074660 con 0x7f274c105cf0 2024-04-25T13:44:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.680+0000 7f275357a640 1 -- 172.21.15.27:0/3889749154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274c074b70 con 0x7f274c105cf0 2024-04-25T13:44:10.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.680+0000 7f274a7fc640 1 -- 172.21.15.27:0/3889749154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2744030e80 con 0x7f274c105cf0 2024-04-25T13:44:10.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.680+0000 7f274a7fc640 1 -- 172.21.15.27:0/3889749154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744036040 con 0x7f274c105cf0 2024-04-25T13:44:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.684+0000 7f275a41f740 1 Processor -- start 2024-04-25T13:44:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.684+0000 7f275a41f740 1 -- start start 2024-04-25T13:44:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.684+0000 7f275a41f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d43cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.684+0000 7f275a41f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588971550 0x55f588d44230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.684+0000 7f275a41f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588979730 0x55f588d4b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.684+0000 7f275a41f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f588ca0b00 con 0x55f588979730 2024-04-25T13:44:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.684+0000 7f275a41f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f588c3c920 con 0x55f58899a410 2024-04-25T13:44:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.684+0000 7f275a41f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f588c9cac0 con 0x55f588971550 2024-04-25T13:44:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2750b15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588979730 0x55f588d4b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2750b15640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588979730 0x55f588d4b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45754/0 (socket says 172.21.15.27:45754) 2024-04-25T13:44:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2750b15640 1 -- 172.21.15.27:0/4066260841 learned_addr learned my addr 172.21.15.27:0/4066260841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2751b17640 1 --2- 172.21.15.27:0/4066260841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d43cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2750b15640 1 -- 172.21.15.27:0/4066260841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588971550 msgr2=0x55f588d44230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:44:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2750b15640 1 --2- 172.21.15.27:0/4066260841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588971550 0x55f588d44230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2750b15640 1 -- 172.21.15.27:0/4066260841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 msgr2=0x55f588d43cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2750b15640 1 --2- 172.21.15.27:0/4066260841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d43cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f2750b15640 1 -- 172.21.15.27:0/4066260841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f588a7a7e0 con 0x55f588979730 2024-04-25T13:44:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.685+0000 7f274a7fc640 1 -- 172.21.15.27:0/3889749154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2744044020 con 0x7f274c105cf0 2024-04-25T13:44:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.686+0000 7f2750b15640 1 --2- 172.21.15.27:0/4066260841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588979730 0x55f588d4b130 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2734009b20 tx=0x7f273400e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b151dee0292fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.687+0000 7f27437fe640 1 -- 172.21.15.27:0/4066260841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273400ed80 con 0x55f588979730 2024-04-25T13:44:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.687+0000 7f27437fe640 1 -- 172.21.15.27:0/4066260841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27340035f0 con 0x55f588979730 2024-04-25T13:44:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.687+0000 7f27437fe640 1 -- 172.21.15.27:0/4066260841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734010690 con 0x55f588979730 2024-04-25T13:44:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.687+0000 7f274a7fc640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f27380afbb0 0x7f27380b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.687+0000 7f274a7fc640 1 -- 172.21.15.27:0/3889749154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f27440ff2d0 con 0x7f274c105cf0 2024-04-25T13:44:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.688+0000 7f275a41f740 1 -- 172.21.15.27:0/4066260841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588979730 msgr2=0x55f588d4b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.689+0000 7f275a41f740 1 --2- 172.21.15.27:0/4066260841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588979730 0x55f588d4b130 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2734009b20 tx=0x7f273400e350 comp rx=0 tx=0).stop 2024-04-25T13:44:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.689+0000 7f2751b17640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f27380afbb0 0x7f27380b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.689+0000 7f275a41f740 1 -- 172.21.15.27:0/4066260841 shutdown_connections 2024-04-25T13:44:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.689+0000 7f275a41f740 1 --2- 172.21.15.27:0/4066260841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588979730 0x55f588d4b130 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.689+0000 7f275a41f740 1 --2- 172.21.15.27:0/4066260841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588971550 0x55f588d44230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.689+0000 7f275a41f740 1 --2- 172.21.15.27:0/4066260841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d43cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.689+0000 7f275a41f740 1 -- 172.21.15.27:0/4066260841 >> 172.21.15.27:0/4066260841 conn(0x55f58898e560 msgr2=0x55f588d36e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.689+0000 7f275a41f740 1 -- 172.21.15.27:0/4066260841 shutdown_connections 2024-04-25T13:44:10.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.690+0000 7f275a41f740 1 -- 172.21.15.27:0/4066260841 wait complete. 2024-04-25T13:44:10.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.690+0000 7f2751b17640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f27380afbb0 0x7f27380b20c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55f588d44e80 tx=0x7f273c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.690+0000 7f275a41f740 1 -- start start 2024-04-25T13:44:10.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.690+0000 7f275a41f740 1 Processor -- start 2024-04-25T13:44:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.691+0000 7f275a41f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.691+0000 7f275a41f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d3d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.691+0000 7f275a41f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d42780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.691+0000 7f275a41f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f588d3b4f0 con 0x55f588971550 2024-04-25T13:44:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.691+0000 7f275a41f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f588c3a5d0 con 0x55f58899a410 2024-04-25T13:44:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.691+0000 7f275a41f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f588d3b670 con 0x55f588979730 2024-04-25T13:44:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.691+0000 7f275a41f740 10 client.0 _ll_register_callbacks cb 0x55f588c28080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751316640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751316640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45758/0 (socket says 172.21.15.27:45758) 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751316640 1 -- 172.21.15.27:0/1724787191 learned_addr learned my addr 172.21.15.27:0/1724787191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2750b15640 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d3d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751b17640 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d42780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751316640 1 -- 172.21.15.27:0/1724787191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 msgr2=0x55f588d3d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751316640 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d3d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751316640 1 -- 172.21.15.27:0/1724787191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 msgr2=0x55f588d42780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751316640 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d42780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.692+0000 7f2751316640 1 -- 172.21.15.27:0/1724787191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f588adaf70 con 0x55f588971550 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.693+0000 7f2751316640 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d2f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2744004af0 tx=0x7f27440c7610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd92013722faca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.693+0000 7f2740ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:44:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.693+0000 7f2740ff9640 1 -- 172.21.15.27:0/1724787191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27440e1060 con 0x55f588971550 2024-04-25T13:44:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.693+0000 7f2740ff9640 1 -- 172.21.15.27:0/1724787191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2744003b70 con 0x55f588971550 2024-04-25T13:44:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.693+0000 7f275a41f740 1 -- 172.21.15.27:0/1724787191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27340035f0 con 0x55f588971550 2024-04-25T13:44:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.693+0000 7f2740ff9640 1 -- 172.21.15.27:0/1724787191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f27440c7e80 con 0x55f588971550 2024-04-25T13:44:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.694+0000 7f275a41f740 10 client.15984 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.694+0000 7f275a41f740 1 -- 172.21.15.27:0/1724787191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f588972170 con 0x55f588971550 2024-04-25T13:44:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.694+0000 7f2740ff9640 1 -- 172.21.15.27:0/1724787191 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27340035f0 con 0x55f588971550 2024-04-25T13:44:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.695+0000 7f2740ff9640 1 -- 172.21.15.27:0/1724787191 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f27440dcbd0 con 0x55f588971550 2024-04-25T13:44:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.695+0000 7f275a41f740 10 client.15984 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.695+0000 7f275a41f740 10 client.15984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.695+0000 7f275a41f740 20 client.15984 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.695+0000 7f275a41f740 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5889591f0 0x55f588d63a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.695+0000 7f275a41f740 4 client.15984 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.695+0000 7f275a41f740 1 -- 172.21.15.27:0/1724787191 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f588cae710 con 0x55f5889591f0 2024-04-25T13:44:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.695+0000 7f2750b15640 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5889591f0 0x55f588d63a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.697+0000 7f2750b15640 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5889591f0 0x55f588d63a50 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba15116291ff3a09 server_cookie=e23d0defa3cb4fee in_seq=0 out_seq=0 2024-04-25T13:44:10.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.697+0000 7f2740ff9640 10 client.15984 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.698+0000 7f2740ff9640 1 -- 172.21.15.27:0/1724787191 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f588cae710 con 0x55f5889591f0 2024-04-25T13:44:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.698+0000 7f2740ff9640 10 client.15984 handle_command_reply: tid=0 2024-04-25T13:44:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.698+0000 7f275a41f740 1 client.15984 shutdown 2024-04-25T13:44:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.698+0000 7f275a41f740 1 -- 172.21.15.27:0/1724787191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 msgr2=0x55f588d3d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.698+0000 7f275a41f740 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d2f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2744004af0 tx=0x7f27440c7610 comp rx=0 tx=0).stop 2024-04-25T13:44:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.699+0000 7f275a41f740 1 -- 172.21.15.27:0/1724787191 shutdown_connections 2024-04-25T13:44:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.699+0000 7f275a41f740 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5889591f0 0x55f588d63a50 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.700+0000 7f275a41f740 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d42780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.700+0000 7f275a41f740 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d3d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.700+0000 7f275a41f740 1 --2- 172.21.15.27:0/1724787191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d2f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.700+0000 7f275a41f740 1 -- 172.21.15.27:0/1724787191 >> 172.21.15.27:0/1724787191 conn(0x55f58898e560 msgr2=0x55f588d35ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.700+0000 7f2740ff9640 0 client.15984 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.701+0000 7f275a41f740 1 -- 172.21.15.27:0/1724787191 shutdown_connections 2024-04-25T13:44:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.701+0000 7f275a41f740 1 -- 172.21.15.27:0/1724787191 wait complete. 2024-04-25T13:44:10.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.701+0000 7f275a41f740 20 client.15984 trim_cache size 0 max 16384 2024-04-25T13:44:10.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.739+0000 7f275a41f740 1 Processor -- start 2024-04-25T13:44:10.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.741+0000 7f275a41f740 1 -- start start 2024-04-25T13:44:10.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.742+0000 7f275a41f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.742+0000 7f275a41f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d354c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.743+0000 7f275a41f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d35a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.743+0000 7f275a41f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00c340 con 0x55f588971550 2024-04-25T13:44:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.743+0000 7f275a41f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00ebc0 con 0x55f58899a410 2024-04-25T13:44:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.743+0000 7f275a41f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00c9b0 con 0x55f588979730 2024-04-25T13:44:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.743+0000 7f2751316640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.743+0000 7f2751316640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45768/0 (socket says 172.21.15.27:45768) 2024-04-25T13:44:10.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.743+0000 7f2751316640 1 -- 172.21.15.27:0/1373937235 learned_addr learned my addr 172.21.15.27:0/1373937235 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:10.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.744+0000 7f2750b15640 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.744+0000 7f2751b17640 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d35a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.745+0000 7f2751316640 1 -- 172.21.15.27:0/1373937235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 msgr2=0x55f588d354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.745+0000 7f2751316640 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d354c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.745+0000 7f2751316640 1 -- 172.21.15.27:0/1373937235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 msgr2=0x55f588d35a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.745+0000 7f2751316640 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d35a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.745+0000 7f2751316640 1 -- 172.21.15.27:0/1373937235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f588d487a0 con 0x55f588971550 2024-04-25T13:44:10.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.745+0000 7f2751316640 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d5d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f27440c49f0 tx=0x7f274402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a9c2fa19dc03e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.746+0000 7f2742ffd640 1 -- 172.21.15.27:0/1373937235 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27440cbb30 con 0x55f588971550 2024-04-25T13:44:10.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.746+0000 7f2742ffd640 1 -- 172.21.15.27:0/1373937235 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27440d05e0 con 0x55f588971550 2024-04-25T13:44:10.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.746+0000 7f2742ffd640 1 -- 172.21.15.27:0/1373937235 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274402fc30 con 0x55f588971550 2024-04-25T13:44:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.747+0000 7f275a41f740 1 -- 172.21.15.27:0/1373937235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 msgr2=0x55f588d3d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.747+0000 7f275a41f740 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d5d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f27440c49f0 tx=0x7f274402a6c0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.747+0000 7f275a41f740 1 -- 172.21.15.27:0/1373937235 shutdown_connections 2024-04-25T13:44:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.747+0000 7f275a41f740 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d35a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.747+0000 7f275a41f740 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d354c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.747+0000 7f275a41f740 1 --2- 172.21.15.27:0/1373937235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d3d5d0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.747+0000 7f275a41f740 1 -- 172.21.15.27:0/1373937235 >> 172.21.15.27:0/1373937235 conn(0x55f58898e560 msgr2=0x55f588d3cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.748+0000 7f275a41f740 1 -- 172.21.15.27:0/1373937235 shutdown_connections 2024-04-25T13:44:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.748+0000 7f275a41f740 1 -- 172.21.15.27:0/1373937235 wait complete. 2024-04-25T13:44:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.749+0000 7f275a41f740 1 -- start start 2024-04-25T13:44:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.749+0000 7f275a41f740 1 Processor -- start 2024-04-25T13:44:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f275a41f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d72f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f275a41f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d734d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f275a41f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d74400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f275a41f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00d080 con 0x55f588971550 2024-04-25T13:44:10.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f275a41f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00e4f0 con 0x55f58899a410 2024-04-25T13:44:10.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f2750b15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f2750b15640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d734d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:53568/0 (socket says 172.21.15.27:53568) 2024-04-25T13:44:10.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f2750b15640 1 -- 172.21.15.27:0/3471874273 learned_addr learned my addr 172.21.15.27:0/3471874273 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:10.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.750+0000 7f2751b17640 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d74400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.751+0000 7f2751316640 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d72f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.751+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00d750 con 0x55f588979730 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.751+0000 7f275a41f740 10 client.0 _ll_register_callbacks cb 0x7f27440c2f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.751+0000 7f2750b15640 1 -- 172.21.15.27:0/3471874273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 msgr2=0x55f588d74400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.751+0000 7f2750b15640 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d74400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.751+0000 7f2750b15640 1 -- 172.21.15.27:0/3471874273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 msgr2=0x55f588d72f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.751+0000 7f2750b15640 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d72f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.751+0000 7f2750b15640 1 -- 172.21.15.27:0/3471874273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f588d749c0 con 0x55f588979730 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.752+0000 7f2750b15640 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d734d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f273400eba0 tx=0x7f2734004230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b3dd41b036c0ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.752+0000 7f2740ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.752+0000 7f2740ff9640 1 -- 172.21.15.27:0/3471874273 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273401d040 con 0x55f588979730 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.752+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27440d05e0 con 0x55f588979730 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.752+0000 7f2740ff9640 1 -- 172.21.15.27:0/3471874273 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2734005050 con 0x55f588979730 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.752+0000 7f2740ff9640 1 -- 172.21.15.27:0/3471874273 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2734004520 con 0x55f588979730 2024-04-25T13:44:10.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.752+0000 7f275a41f740 10 client.25717 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:10.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.752+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f588dcbff0 con 0x55f588979730 2024-04-25T13:44:10.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.753+0000 7f2740ff9640 1 -- 172.21.15.27:0/3471874273 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27440d05e0 con 0x55f588979730 2024-04-25T13:44:10.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.753+0000 7f2740ff9640 1 -- 172.21.15.27:0/3471874273 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f273402e070 con 0x55f588979730 2024-04-25T13:44:10.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.754+0000 7f275a41f740 10 client.25717 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:10.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.754+0000 7f275a41f740 10 client.25717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:10.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.754+0000 7f275a41f740 20 client.25717 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.754+0000 7f275a41f740 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f58892e630 0x55f588dce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.754+0000 7f275a41f740 4 client.25717 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:44:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.754+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f588ca8580 con 0x55f58892e630 2024-04-25T13:44:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.754+0000 7f2751316640 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f58892e630 0x55f588dce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.755+0000 7f2740ff9640 10 client.25717 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.755+0000 7f2751316640 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f58892e630 0x55f588dce8d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb09cf5295f13aa2 server_cookie=d036dfeac47560ff in_seq=0 out_seq=0 2024-04-25T13:44:10.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.756+0000 7f2740ff9640 1 -- 172.21.15.27:0/3471874273 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f588ca8580 con 0x55f58892e630 2024-04-25T13:44:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.756+0000 7f2740ff9640 10 client.25717 handle_command_reply: tid=0 2024-04-25T13:44:10.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.756+0000 7f275a41f740 1 client.25717 shutdown 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.756+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 msgr2=0x55f588d734d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.757+0000 7f275a41f740 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d734d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f273400eba0 tx=0x7f2734004230 comp rx=0 tx=0).stop 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.757+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 shutdown_connections 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.757+0000 7f275a41f740 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f58892e630 0x55f588dce8d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.757+0000 7f275a41f740 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f58899a410 0x55f588d74400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.757+0000 7f2740ff9640 0 client.25717 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.757+0000 7f275a41f740 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f588979730 0x55f588d734d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.757+0000 7f275a41f740 1 --2- 172.21.15.27:0/3471874273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f588971550 0x55f588d72f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.757+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 >> 172.21.15.27:0/3471874273 conn(0x55f58898e560 msgr2=0x55f588d39c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.758+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 shutdown_connections 2024-04-25T13:44:10.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.758+0000 7f275a41f740 1 -- 172.21.15.27:0/3471874273 wait complete. 2024-04-25T13:44:10.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.758+0000 7f275a41f740 20 client.25717 trim_cache size 0 max 16384 2024-04-25T13:44:10.758 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:44:10.759 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:44:10.759 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "11ffcb64-dba1-4cd0-84bd-1a76f03f1117", 2024-04-25T13:44:10.759 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:44:10.759 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:44:10.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 -- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f27380afbb0 msgr2=0x7f27380b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f27380afbb0 0x7f27380b20c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55f588d44e80 tx=0x7f273c00a300 comp rx=0 tx=0).stop 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 -- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 msgr2=0x7f274c06c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 0x7f274c06c940 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2744034040 tx=0x7f2744030d10 comp rx=0 tx=0).stop 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 -- 172.21.15.27:0/3889749154 shutdown_connections 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f27380afbb0 0x7f27380b20c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f274c10bc30 0x7f274c073e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f274c106f30 0x7f274c06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 --2- 172.21.15.27:0/3889749154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f274c105cf0 0x7f274c06c940 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.760+0000 7f275357a640 1 -- 172.21.15.27:0/3889749154 >> 172.21.15.27:0/3889749154 conn(0x7f274c100f00 msgr2=0x7f274c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.761+0000 7f275357a640 1 -- 172.21.15.27:0/3889749154 shutdown_connections 2024-04-25T13:44:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.761+0000 7f275357a640 1 -- 172.21.15.27:0/3889749154 wait complete. 2024-04-25T13:44:10.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:44:10.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:44:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.969+0000 7f90be900640 1 -- 172.21.15.27:0/1124389515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90b8104d30 msgr2=0x7f90b8107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.969+0000 7f90be900640 1 --2- 172.21.15.27:0/1124389515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90b8104d30 0x7f90b8107150 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009900 tx=0x7f90a802a420 comp rx=0 tx=0).stop 2024-04-25T13:44:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.973+0000 7f90be900640 1 -- 172.21.15.27:0/1124389515 shutdown_connections 2024-04-25T13:44:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.973+0000 7f90be900640 1 --2- 172.21.15.27:0/1124389515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90b8107690 0x7f90b8107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.973+0000 7f90be900640 1 --2- 172.21.15.27:0/1124389515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90b8104d30 0x7f90b8107150 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.973+0000 7f90be900640 1 --2- 172.21.15.27:0/1124389515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b81023d0 0x7f90b81047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.973+0000 7f90be900640 1 -- 172.21.15.27:0/1124389515 >> 172.21.15.27:0/1124389515 conn(0x7f90b80fdbe0 msgr2=0x7f90b8100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.974+0000 7f90be900640 1 -- 172.21.15.27:0/1124389515 shutdown_connections 2024-04-25T13:44:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.977+0000 7f90be900640 1 -- 172.21.15.27:0/1124389515 wait complete. 2024-04-25T13:44:10.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.978+0000 7f90be900640 1 Processor -- start 2024-04-25T13:44:10.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.978+0000 7f90be900640 1 -- start start 2024-04-25T13:44:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.979+0000 7f90be900640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b81023d0 0x7f90b81a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.979+0000 7f90be900640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90b8104d30 0x7f90b81a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.979+0000 7f90be900640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90b8107690 0x7f90b81a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.979+0000 7f90b77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90b8104d30 0x7f90b81a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.979+0000 7f90b77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90b8104d30 0x7f90b81a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:53590/0 (socket says 172.21.15.27:53590) 2024-04-25T13:44:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.979+0000 7f90b77fe640 1 -- 172.21.15.27:0/377565510 learned_addr learned my addr 172.21.15.27:0/377565510 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.979+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f90b8115cc0 con 0x7f90b8107690 2024-04-25T13:44:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.979+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f90b8115b40 con 0x7f90b81023d0 2024-04-25T13:44:10.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.980+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f90b8115e40 con 0x7f90b8104d30 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.980+0000 7f90b7fff640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b81023d0 0x7f90b81a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.980+0000 7f90bce9d640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90b8107690 0x7f90b81a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.980+0000 7f90b77fe640 1 -- 172.21.15.27:0/377565510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b81023d0 msgr2=0x7f90b81a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.980+0000 7f90b77fe640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b81023d0 0x7f90b81a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.980+0000 7f90b77fe640 1 -- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90b8107690 msgr2=0x7f90b81a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.980+0000 7f90b77fe640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90b8107690 0x7f90b81a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.980+0000 7f90b77fe640 1 -- 172.21.15.27:0/377565510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b81a8410 con 0x7f90b8104d30 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.981+0000 7f90b77fe640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90b8104d30 0x7f90b81a0e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f90a8031870 tx=0x7f90a8030bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.981+0000 7f90b57fa640 1 -- 172.21.15.27:0/377565510 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8040070 con 0x7f90b8104d30 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.981+0000 7f90b57fa640 1 -- 172.21.15.27:0/377565510 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90a803cb20 con 0x7f90b8104d30 2024-04-25T13:44:10.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.981+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b81a86a0 con 0x7f90b8104d30 2024-04-25T13:44:10.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.981+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b81a8be0 con 0x7f90b8104d30 2024-04-25T13:44:10.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.981+0000 7f90b57fa640 1 -- 172.21.15.27:0/377565510 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8035030 con 0x7f90b8104d30 2024-04-25T13:44:10.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.984+0000 7f90b57fa640 1 -- 172.21.15.27:0/377565510 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f90a803baf0 con 0x7f90b8104d30 2024-04-25T13:44:10.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.985+0000 7f90c57b1740 1 Processor -- start 2024-04-25T13:44:10.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.985+0000 7f90c57b1740 1 -- start start 2024-04-25T13:44:10.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.985+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 0x564da9619a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.985+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9619fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.985+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da961ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.985+0000 7f90c57b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564da957ed80 con 0x564da92576b0 2024-04-25T13:44:10.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.985+0000 7f90c57b1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564da95388b0 con 0x564da9278390 2024-04-25T13:44:10.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.985+0000 7f90c57b1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564da957ad40 con 0x564da924f4d0 2024-04-25T13:44:10.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9619fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9619fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:53604/0 (socket says 172.21.15.27:53604) 2024-04-25T13:44:10.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90bce9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 0x564da9619a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b7fff640 1 -- 172.21.15.27:0/2745877081 learned_addr learned my addr 172.21.15.27:0/2745877081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:10.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b77fe640 1 --2- 172.21.15.27:0/2745877081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da961ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b7fff640 1 -- 172.21.15.27:0/2745877081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 msgr2=0x564da9619a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b7fff640 1 --2- 172.21.15.27:0/2745877081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 0x564da9619a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b7fff640 1 -- 172.21.15.27:0/2745877081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 msgr2=0x564da961ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b7fff640 1 --2- 172.21.15.27:0/2745877081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da961ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b7fff640 1 -- 172.21.15.27:0/2745877081 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564da9509a40 con 0x564da924f4d0 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.986+0000 7f90b57fa640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90a00afba0 0x7f90a00b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.987+0000 7f90b7fff640 1 --2- 172.21.15.27:0/2745877081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9619fe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009f50 tx=0x7f90b000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d821e71dd32ebf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.987+0000 7f90bce9d640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90a00afba0 0x7f90a00b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.987+0000 7f90b57fa640 1 -- 172.21.15.27:0/377565510 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f90a80ffbe0 con 0x7f90b8104d30 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.987+0000 7f909e7fc640 1 -- 172.21.15.27:0/2745877081 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b001a070 con 0x564da924f4d0 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.987+0000 7f909e7fc640 1 -- 172.21.15.27:0/2745877081 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90b00158c0 con 0x564da924f4d0 2024-04-25T13:44:10.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.987+0000 7f909e7fc640 1 -- 172.21.15.27:0/2745877081 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b000a820 con 0x564da924f4d0 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.988+0000 7f90c57b1740 1 -- 172.21.15.27:0/2745877081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 msgr2=0x564da9619fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.988+0000 7f90c57b1740 1 --2- 172.21.15.27:0/2745877081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9619fe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009f50 tx=0x7f90b000c8b0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.988+0000 7f90bce9d640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90a00afba0 0x7f90a00b20b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00a490 tx=0x7f90ac00a420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.988+0000 7f90c57b1740 1 -- 172.21.15.27:0/2745877081 shutdown_connections 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.988+0000 7f90c57b1740 1 --2- 172.21.15.27:0/2745877081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da961ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.988+0000 7f90c57b1740 1 --2- 172.21.15.27:0/2745877081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9619fe0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.988+0000 7f90c57b1740 1 --2- 172.21.15.27:0/2745877081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 0x564da9619a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.988+0000 7f90c57b1740 1 -- 172.21.15.27:0/2745877081 >> 172.21.15.27:0/2745877081 conn(0x564da926c4e0 msgr2=0x564da9615270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.989+0000 7f90c57b1740 1 -- 172.21.15.27:0/2745877081 shutdown_connections 2024-04-25T13:44:10.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.989+0000 7f90c57b1740 1 -- 172.21.15.27:0/2745877081 wait complete. 2024-04-25T13:44:10.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.990+0000 7f90c57b1740 1 -- start start 2024-04-25T13:44:10.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.990+0000 7f90c57b1740 1 Processor -- start 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.990+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da96349d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.990+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da9634f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.990+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.990+0000 7f90c57b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564da961b7d0 con 0x564da924f4d0 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.990+0000 7f90c57b1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564da9505d10 con 0x564da92576b0 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.990+0000 7f90c57b1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564da961b950 con 0x564da9278390 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90c57b1740 10 client.0 _ll_register_callbacks cb 0x564da9532210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da96349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90b77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da9634f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90bce9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90b77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da9634f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39072/0 (socket says 172.21.15.27:39072) 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90b77fe640 1 -- 172.21.15.27:0/4216219980 learned_addr learned my addr 172.21.15.27:0/4216219980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da96349d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45816/0 (socket says 172.21.15.27:45816) 2024-04-25T13:44:10.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90bce9d640 1 -- 172.21.15.27:0/4216219980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 msgr2=0x564da9634f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90bce9d640 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da9634f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90bce9d640 1 -- 172.21.15.27:0/4216219980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 msgr2=0x564da96349d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90bce9d640 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da96349d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90bce9d640 1 -- 172.21.15.27:0/4216219980 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564da935a7d0 con 0x564da9278390 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90b77fe640 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da9634f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90bce9d640 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963be70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0058c0 tx=0x7f90ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7439bb4e238ace02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.991+0000 7f90977fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.992+0000 7f90977fe640 1 -- 172.21.15.27:0/4216219980 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac02a060 con 0x564da9278390 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.992+0000 7f90977fe640 1 -- 172.21.15.27:0/4216219980 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f90ac01b460 con 0x564da9278390 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.992+0000 7f90c57b1740 1 -- 172.21.15.27:0/4216219980 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90b00158c0 con 0x564da9278390 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.992+0000 7f90977fe640 1 -- 172.21.15.27:0/4216219980 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f90ac010ea0 con 0x564da9278390 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.992+0000 7f90c57b1740 10 client.25726 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:10.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.992+0000 7f90c57b1740 1 -- 172.21.15.27:0/4216219980 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564da92903f0 con 0x564da9278390 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.992+0000 7f90977fe640 1 -- 172.21.15.27:0/4216219980 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90b00158c0 con 0x564da9278390 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.992+0000 7f90977fe640 1 -- 172.21.15.27:0/4216219980 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f90ac019040 con 0x564da9278390 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.993+0000 7f90c57b1740 10 client.25726 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.993+0000 7f90c57b1740 10 client.25726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.993+0000 7f90c57b1740 20 client.25726 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.993+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564da9236f60 0x564da96424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.993+0000 7f90c57b1740 4 client.25726 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.993+0000 7f90c57b1740 1 -- 172.21.15.27:0/4216219980 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564da958c990 con 0x564da9236f60 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.993+0000 7f90b7fff640 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564da9236f60 0x564da96424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.994+0000 7f90977fe640 10 client.25726 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.994+0000 7f90b7fff640 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564da9236f60 0x564da96424b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bcd2f4ba883fe8d server_cookie=edd020363eab069 in_seq=0 out_seq=0 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.994+0000 7f90977fe640 1 -- 172.21.15.27:0/4216219980 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564da958c990 con 0x564da9236f60 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.994+0000 7f90977fe640 10 client.25726 handle_command_reply: tid=0 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 client.25726 shutdown 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 -- 172.21.15.27:0/4216219980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 msgr2=0x564da963be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963be70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f90ac0058c0 tx=0x7f90ac0046c0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 -- 172.21.15.27:0/4216219980 shutdown_connections 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564da9236f60 0x564da96424b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963be70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da9634f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90977fe640 0 client.25726 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4216219980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da96349d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 -- 172.21.15.27:0/4216219980 >> 172.21.15.27:0/4216219980 conn(0x564da926c4e0 msgr2=0x564da9618180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 -- 172.21.15.27:0/4216219980 shutdown_connections 2024-04-25T13:44:10.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 1 -- 172.21.15.27:0/4216219980 wait complete. 2024-04-25T13:44:10.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:10.995+0000 7f90c57b1740 20 client.25726 trim_cache size 0 max 16384 2024-04-25T13:44:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.025+0000 7f90c57b1740 1 Processor -- start 2024-04-25T13:44:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.025+0000 7f90c57b1740 1 -- start start 2024-04-25T13:44:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.025+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da9633700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.025+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da963bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90c57b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f90a400c3d0 con 0x564da924f4d0 2024-04-25T13:44:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90c57b1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f90a400ec50 con 0x564da92576b0 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90c57b1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f90a400ca40 con 0x564da9278390 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da9633700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da9633700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45828/0 (socket says 172.21.15.27:45828) 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da963bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b7fff640 1 -- 172.21.15.27:0/2359991583 learned_addr learned my addr 172.21.15.27:0/2359991583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90bce9d640 1 --2- 172.21.15.27:0/2359991583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b77fe640 1 -- 172.21.15.27:0/2359991583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 msgr2=0x564da963c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b77fe640 1 --2- 172.21.15.27:0/2359991583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b77fe640 1 -- 172.21.15.27:0/2359991583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 msgr2=0x564da9633700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b77fe640 1 --2- 172.21.15.27:0/2359991583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da9633700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.026+0000 7f90b77fe640 1 -- 172.21.15.27:0/2359991583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564da92500f0 con 0x564da92576b0 2024-04-25T13:44:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.027+0000 7f90b77fe640 1 --2- 172.21.15.27:0/2359991583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da963bd90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f90a8006010 tx=0x7f90a8030f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ba6d975df829c20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.027+0000 7f909dffb640 1 -- 172.21.15.27:0/2359991583 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8049070 con 0x564da92576b0 2024-04-25T13:44:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.027+0000 7f909dffb640 1 -- 172.21.15.27:0/2359991583 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90a80c8790 con 0x564da92576b0 2024-04-25T13:44:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.027+0000 7f909dffb640 1 -- 172.21.15.27:0/2359991583 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a80315a0 con 0x564da92576b0 2024-04-25T13:44:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.028+0000 7f90c57b1740 1 -- 172.21.15.27:0/2359991583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 msgr2=0x564da963bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.028+0000 7f90c57b1740 1 --2- 172.21.15.27:0/2359991583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da963bd90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f90a8006010 tx=0x7f90a8030f70 comp rx=0 tx=0).stop 2024-04-25T13:44:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.028+0000 7f90c57b1740 1 -- 172.21.15.27:0/2359991583 shutdown_connections 2024-04-25T13:44:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.028+0000 7f90c57b1740 1 --2- 172.21.15.27:0/2359991583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da9278390 0x564da963c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.028+0000 7f90c57b1740 1 --2- 172.21.15.27:0/2359991583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da92576b0 0x564da963bd90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.028+0000 7f90c57b1740 1 --2- 172.21.15.27:0/2359991583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da924f4d0 0x564da9633700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.028+0000 7f90c57b1740 1 -- 172.21.15.27:0/2359991583 >> 172.21.15.27:0/2359991583 conn(0x564da926c4e0 msgr2=0x564da9632ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.028+0000 7f90c57b1740 1 -- 172.21.15.27:0/2359991583 shutdown_connections 2024-04-25T13:44:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.029+0000 7f90c57b1740 1 -- 172.21.15.27:0/2359991583 wait complete. 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.029+0000 7f90c57b1740 1 -- start start 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90c57b1740 1 Processor -- start 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9639470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da96399b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90c57b1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 0x564da96a8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90c57b1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f90a400d110 con 0x564da92576b0 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90c57b1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f90a400e580 con 0x564da9278390 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90c57b1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f90a400d7e0 con 0x564da924f4d0 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90c57b1740 10 client.0 _ll_register_callbacks cb 0x7f90ac00a880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.030+0000 7f90b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da96399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9639470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da96399b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45844/0 (socket says 172.21.15.27:45844) 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90b77fe640 1 -- 172.21.15.27:0/4115973606 learned_addr learned my addr 172.21.15.27:0/4115973606 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90bce9d640 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 0x564da96a8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90b7fff640 1 -- 172.21.15.27:0/4115973606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 msgr2=0x564da96a8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90b7fff640 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 0x564da96a8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90b7fff640 1 -- 172.21.15.27:0/4115973606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 msgr2=0x564da96399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90b7fff640 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da96399b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.031+0000 7f90b7fff640 1 -- 172.21.15.27:0/4115973606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564da953f4b0 con 0x564da924f4d0 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.032+0000 7f90b7fff640 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9639470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009f20 tx=0x7f90b0015410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18c100006aca9cb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.032+0000 7f90977fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.032+0000 7f90977fe640 1 -- 172.21.15.27:0/4115973606 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b002b020 con 0x564da924f4d0 2024-04-25T13:44:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.032+0000 7f90c57b1740 1 -- 172.21.15.27:0/4115973606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90a80c8790 con 0x564da924f4d0 2024-04-25T13:44:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.032+0000 7f90977fe640 1 -- 172.21.15.27:0/4115973606 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f90b0015710 con 0x564da924f4d0 2024-04-25T13:44:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90977fe640 1 -- 172.21.15.27:0/4115973606 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f90b001a040 con 0x564da924f4d0 2024-04-25T13:44:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90c57b1740 10 client.25732 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90c57b1740 1 -- 172.21.15.27:0/4115973606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564da96a9c30 con 0x564da924f4d0 2024-04-25T13:44:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90977fe640 1 -- 172.21.15.27:0/4115973606 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90a80c8790 con 0x564da924f4d0 2024-04-25T13:44:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90977fe640 1 -- 172.21.15.27:0/4115973606 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f90b0026900 con 0x564da924f4d0 2024-04-25T13:44:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90c57b1740 10 client.25732 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90c57b1740 10 client.25732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90c57b1740 20 client.25732 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564da920c3a0 0x564da96ac3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90c57b1740 4 client.25732 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.033+0000 7f90c57b1740 1 -- 172.21.15.27:0/4115973606 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564da9586800 con 0x564da920c3a0 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.034+0000 7f90b77fe640 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564da920c3a0 0x564da96ac3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.035+0000 7f90b77fe640 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564da920c3a0 0x564da96ac3e0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5f702d0ffdb91d9 server_cookie=12c87af184dd733a in_seq=0 out_seq=0 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.035+0000 7f90977fe640 10 client.25732 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.035+0000 7f90977fe640 1 -- 172.21.15.27:0/4115973606 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564da9586800 con 0x564da920c3a0 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.035+0000 7f90977fe640 10 client.25732 handle_command_reply: tid=0 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.035+0000 7f90c57b1740 1 client.25732 shutdown 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.035+0000 7f90c57b1740 1 -- 172.21.15.27:0/4115973606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 msgr2=0x564da9639470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.035+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9639470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009f20 tx=0x7f90b0015410 comp rx=0 tx=0).stop 2024-04-25T13:44:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 1 -- 172.21.15.27:0/4115973606 shutdown_connections 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564da920c3a0 0x564da96ac3e0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90977fe640 0 client.25732 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564da9278390 0x564da96a8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564da92576b0 0x564da96399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 1 --2- 172.21.15.27:0/4115973606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564da924f4d0 0x564da9639470 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 1 -- 172.21.15.27:0/4115973606 >> 172.21.15.27:0/4115973606 conn(0x564da926c4e0 msgr2=0x564da9614c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 1 -- 172.21.15.27:0/4115973606 shutdown_connections 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 1 -- 172.21.15.27:0/4115973606 wait complete. 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.036+0000 7f90c57b1740 20 client.25732 trim_cache size 0 max 16384 2024-04-25T13:44:11.038 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "11ffcb64-dba1-4cd0-84bd-1a76f03f1117": { 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "11ffcb64-dba1-4cd0-84bd-1a76f03f1117", 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:44:11.039 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:44:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90a00afba0 msgr2=0x7f90a00b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90a00afba0 0x7f90a00b20b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00a490 tx=0x7f90ac00a420 comp rx=0 tx=0).stop 2024-04-25T13:44:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90b8104d30 msgr2=0x7f90b81a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90b8104d30 0x7f90b81a0e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f90a8031870 tx=0x7f90a8030bc0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 shutdown_connections 2024-04-25T13:44:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90a00afba0 0x7f90a00b20b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90b8107690 0x7f90b81a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f90b8104d30 0x7f90b81a0e10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 --2- 172.21.15.27:0/377565510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90b81023d0 0x7f90b81a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.037+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 >> 172.21.15.27:0/377565510 conn(0x7f90b80fdbe0 msgr2=0x7f90b8102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.039+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 shutdown_connections 2024-04-25T13:44:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:11.039+0000 7f90be900640 1 -- 172.21.15.27:0/377565510 wait complete. 2024-04-25T13:44:11.056 INFO:tasks.cephfs.filesystem:scrub status for tag:11ffcb64-dba1-4cd0-84bd-1a76f03f1117 - {'path': '/', 'tag': '11ffcb64-dba1-4cd0-84bd-1a76f03f1117', 'options': 'recursive,force'} 2024-04-25T13:44:11.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:11 smithi122 ceph-mon[32843]: pgmap v448: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 45 MiB/s wr, 844 op/s 2024-04-25T13:44:11.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:11 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:44:11.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:11 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:11.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:11 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:44:11.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:11 smithi027 ceph-mon[29140]: pgmap v448: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 45 MiB/s wr, 844 op/s 2024-04-25T13:44:11.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:11 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:44:11.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:11 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:11.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:11 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:44:12.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:11 smithi045 ceph-mon[32731]: pgmap v448: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 45 MiB/s wr, 844 op/s 2024-04-25T13:44:12.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:11 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:44:12.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:11 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:12.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:11 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:44:12.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:12 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 185.0 s, 3753.8 tps, lat 26.616 ms stddev 33.054 2024-04-25T13:44:13.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:12 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:13.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:12 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.686+0000 7ff44d1a8640 1 -- 172.21.15.27:0/2546601235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff448110480 msgr2=0x7ff4481128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.686+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/2546601235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff448110480 0x7ff4481128a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff438005fd0 tx=0x7ff43802b320 comp rx=0 tx=0).stop 2024-04-25T13:44:13.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.687+0000 7ff44d1a8640 1 -- 172.21.15.27:0/2546601235 shutdown_connections 2024-04-25T13:44:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.687+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/2546601235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff448110480 0x7ff4481128a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.687+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/2546601235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44810b780 0x7ff44810bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.687+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/2546601235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff44810a540 0x7ff44810b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.687+0000 7ff44d1a8640 1 -- 172.21.15.27:0/2546601235 >> 172.21.15.27:0/2546601235 conn(0x7ff448070960 msgr2=0x7ff448072e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.687+0000 7ff44d1a8640 1 -- 172.21.15.27:0/2546601235 shutdown_connections 2024-04-25T13:44:13.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.687+0000 7ff44d1a8640 1 -- 172.21.15.27:0/2546601235 wait complete. 2024-04-25T13:44:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff44d1a8640 1 Processor -- start 2024-04-25T13:44:13.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff44d1a8640 1 -- start start 2024-04-25T13:44:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff44d1a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff44810a540 0x7ff4481a8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff44d1a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44810b780 0x7ff4481a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff44d1a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff448110480 0x7ff4481b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff44d1a8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff448115c20 con 0x7ff44810b780 2024-04-25T13:44:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff44d1a8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff448115aa0 con 0x7ff44810a540 2024-04-25T13:44:13.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff44d1a8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff448115da0 con 0x7ff448110480 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.689+0000 7ff446d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff44810a540 0x7ff4481a8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.690+0000 7ff446d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff44810a540 0x7ff4481a8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:46446/0 (socket says 172.21.15.27:46446) 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.690+0000 7ff446d9d640 1 -- 172.21.15.27:0/99304262 learned_addr learned my addr 172.21.15.27:0/99304262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.690+0000 7ff44759e640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff448110480 0x7ff4481b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.690+0000 7ff44759e640 1 -- 172.21.15.27:0/99304262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff44810a540 msgr2=0x7ff4481a8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.690+0000 7ff44759e640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff44810a540 0x7ff4481a8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.690+0000 7ff44759e640 1 -- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44810b780 msgr2=0x7ff4481a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.690+0000 7ff44759e640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44810b780 0x7ff4481a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.690+0000 7ff44759e640 1 -- 172.21.15.27:0/99304262 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4481b0ac0 con 0x7ff448110480 2024-04-25T13:44:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.691+0000 7ff44759e640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff448110480 0x7ff4481b0450 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff43802b840 tx=0x7ff438008c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.691+0000 7ff43ffff640 1 -- 172.21.15.27:0/99304262 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438041020 con 0x7ff448110480 2024-04-25T13:44:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.691+0000 7ff43ffff640 1 -- 172.21.15.27:0/99304262 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff438033e50 con 0x7ff448110480 2024-04-25T13:44:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.691+0000 7ff44d1a8640 1 -- 172.21.15.27:0/99304262 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4481b0d50 con 0x7ff448110480 2024-04-25T13:44:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.691+0000 7ff44d1a8640 1 -- 172.21.15.27:0/99304262 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4481b1260 con 0x7ff448110480 2024-04-25T13:44:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.691+0000 7ff43ffff640 1 -- 172.21.15.27:0/99304262 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4380045e0 con 0x7ff448110480 2024-04-25T13:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.696+0000 7ff454051740 1 Processor -- start 2024-04-25T13:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.696+0000 7ff454051740 1 -- start start 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.696+0000 7ff43ffff640 1 -- 172.21.15.27:0/99304262 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff438044020 con 0x7ff448110480 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.696+0000 7ff454051740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c267131b0 0x559c26ab3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff454051740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ab3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff454051740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 0x559c26abadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff454051740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559c26a19460 con 0x559c266f24d0 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff454051740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559c269d3590 con 0x559c267131b0 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff454051740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559c26a15420 con 0x559c266ea2f0 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff44659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c267131b0 0x559c26ab3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff44659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c267131b0 0x559c26ab3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:46450/0 (socket says 172.21.15.27:46450) 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff44659c640 1 -- 172.21.15.27:0/921626409 learned_addr learned my addr 172.21.15.27:0/921626409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.697+0000 7ff446d9d640 1 --2- 172.21.15.27:0/921626409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ab3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.698+0000 7ff446d9d640 1 -- 172.21.15.27:0/921626409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c267131b0 msgr2=0x559c26ab3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.698+0000 7ff446d9d640 1 --2- 172.21.15.27:0/921626409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c267131b0 0x559c26ab3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.698+0000 7ff446d9d640 1 -- 172.21.15.27:0/921626409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 msgr2=0x559c26abadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.698+0000 7ff446d9d640 1 --2- 172.21.15.27:0/921626409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 0x559c26abadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.698+0000 7ff446d9d640 1 -- 172.21.15.27:0/921626409 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c269e00d0 con 0x559c266ea2f0 2024-04-25T13:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.698+0000 7ff446d9d640 1 --2- 172.21.15.27:0/921626409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ab3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff440009c40 tx=0x7ff440011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62ffc78e810f487c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.699+0000 7ff43ffff640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4340afbe0 0x7ff4340b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.699+0000 7ff43d7fa640 1 -- 172.21.15.27:0/921626409 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440004230 con 0x559c266ea2f0 2024-04-25T13:44:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.699+0000 7ff43ffff640 1 -- 172.21.15.27:0/99304262 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff4380ffab0 con 0x7ff448110480 2024-04-25T13:44:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.699+0000 7ff43d7fa640 1 -- 172.21.15.27:0/921626409 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4400043d0 con 0x559c266ea2f0 2024-04-25T13:44:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.699+0000 7ff43d7fa640 1 -- 172.21.15.27:0/921626409 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440004dd0 con 0x559c266ea2f0 2024-04-25T13:44:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.700+0000 7ff44659c640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4340afbe0 0x7ff4340b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.701+0000 7ff454051740 1 -- 172.21.15.27:0/921626409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 msgr2=0x559c26ab3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.701+0000 7ff454051740 1 --2- 172.21.15.27:0/921626409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ab3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff440009c40 tx=0x7ff440011810 comp rx=0 tx=0).stop 2024-04-25T13:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.701+0000 7ff44659c640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4340afbe0 0x7ff4340b20f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x559c26ab4b40 tx=0x7ff43000d520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.707+0000 7ff454051740 1 -- 172.21.15.27:0/921626409 shutdown_connections 2024-04-25T13:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.707+0000 7ff454051740 1 --2- 172.21.15.27:0/921626409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 0x559c26abadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.707+0000 7ff454051740 1 --2- 172.21.15.27:0/921626409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ab3ef0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.707+0000 7ff454051740 1 --2- 172.21.15.27:0/921626409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c267131b0 0x559c26ab3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.707+0000 7ff454051740 1 -- 172.21.15.27:0/921626409 >> 172.21.15.27:0/921626409 conn(0x559c26707300 msgr2=0x559c26aaf300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.709+0000 7ff454051740 1 -- 172.21.15.27:0/921626409 shutdown_connections 2024-04-25T13:44:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.713+0000 7ff454051740 1 -- 172.21.15.27:0/921626409 wait complete. 2024-04-25T13:44:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.713+0000 7ff454051740 1 -- start start 2024-04-25T13:44:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.713+0000 7ff454051740 1 Processor -- start 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.717+0000 7ff454051740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266ea2f0 0x559c26abd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.717+0000 7ff454051740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266f24d0 0x559c26abdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.717+0000 7ff454051740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c267131b0 0x559c26ac4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.717+0000 7ff454051740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559c26abb840 con 0x559c266ea2f0 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.717+0000 7ff454051740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559c269a0b60 con 0x559c266f24d0 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.717+0000 7ff454051740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559c26abb9c0 con 0x559c267131b0 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.717+0000 7ff454051740 10 client.0 _ll_register_callbacks cb 0x559c269c52b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff446d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266ea2f0 0x559c26abd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff446d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266ea2f0 0x559c26abd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50726/0 (socket says 172.21.15.27:50726) 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff446d9d640 1 -- 172.21.15.27:0/916243693 learned_addr learned my addr 172.21.15.27:0/916243693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff44659c640 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266f24d0 0x559c26abdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff446d9d640 1 -- 172.21.15.27:0/916243693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c267131b0 msgr2=0x559c26ac4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff446d9d640 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c267131b0 0x559c26ac4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff446d9d640 1 -- 172.21.15.27:0/916243693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266f24d0 msgr2=0x559c26abdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff446d9d640 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266f24d0 0x559c26abdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.718+0000 7ff446d9d640 1 -- 172.21.15.27:0/916243693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c2672b210 con 0x559c266ea2f0 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.719+0000 7ff446d9d640 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266ea2f0 0x559c26abd6b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff440005c80 tx=0x7ff440019b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8a423569da3ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.719+0000 7ff42e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.719+0000 7ff42e7fc640 1 -- 172.21.15.27:0/916243693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440028020 con 0x559c266ea2f0 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.719+0000 7ff454051740 1 -- 172.21.15.27:0/916243693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4400043d0 con 0x559c266ea2f0 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.719+0000 7ff42e7fc640 1 -- 172.21.15.27:0/916243693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff440024420 con 0x559c266ea2f0 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.720+0000 7ff42e7fc640 1 -- 172.21.15.27:0/916243693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff4400046c0 con 0x559c266ea2f0 2024-04-25T13:44:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.720+0000 7ff454051740 10 client.16014 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.720+0000 7ff454051740 1 -- 172.21.15.27:0/916243693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559c267f7620 con 0x559c266ea2f0 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.720+0000 7ff42e7fc640 1 -- 172.21.15.27:0/916243693 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4400043d0 con 0x559c266ea2f0 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.720+0000 7ff42e7fc640 1 -- 172.21.15.27:0/916243693 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff440004ca0 con 0x559c266ea2f0 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.721+0000 7ff454051740 10 client.16014 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.721+0000 7ff454051740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.721+0000 7ff454051740 20 client.16014 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.721+0000 7ff454051740 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559c266d1d80 0x559c26aed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.721+0000 7ff454051740 4 client.16014 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.721+0000 7ff454051740 1 -- 172.21.15.27:0/916243693 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c26a27070 con 0x559c266d1d80 2024-04-25T13:44:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.721+0000 7ff44659c640 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559c266d1d80 0x559c26aed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.724+0000 7ff44659c640 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559c266d1d80 0x559c26aed7e0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=389da84f8ab412da server_cookie=fba1f53c9d55c11 in_seq=0 out_seq=0 2024-04-25T13:44:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.725+0000 7ff42e7fc640 10 client.16014 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.728+0000 7ff42e7fc640 1 -- 172.21.15.27:0/916243693 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559c26a27070 con 0x559c266d1d80 2024-04-25T13:44:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.728+0000 7ff42e7fc640 10 client.16014 handle_command_reply: tid=0 2024-04-25T13:44:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.728+0000 7ff454051740 1 client.16014 shutdown 2024-04-25T13:44:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.728+0000 7ff454051740 1 -- 172.21.15.27:0/916243693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266ea2f0 msgr2=0x559c26abd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.728+0000 7ff454051740 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266ea2f0 0x559c26abd6b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff440005c80 tx=0x7ff440019b40 comp rx=0 tx=0).stop 2024-04-25T13:44:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.735+0000 7ff454051740 1 -- 172.21.15.27:0/916243693 shutdown_connections 2024-04-25T13:44:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.735+0000 7ff454051740 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559c266d1d80 0x559c26aed7e0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.735+0000 7ff454051740 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c267131b0 0x559c26ac4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.735+0000 7ff454051740 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266f24d0 0x559c26abdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.735+0000 7ff454051740 1 --2- 172.21.15.27:0/916243693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266ea2f0 0x559c26abd6b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.735+0000 7ff454051740 1 -- 172.21.15.27:0/916243693 >> 172.21.15.27:0/916243693 conn(0x559c26707300 msgr2=0x559c26ab2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.735+0000 7ff42e7fc640 0 client.16014 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.736+0000 7ff454051740 1 -- 172.21.15.27:0/916243693 shutdown_connections 2024-04-25T13:44:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.737+0000 7ff454051740 1 -- 172.21.15.27:0/916243693 wait complete. 2024-04-25T13:44:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.737+0000 7ff454051740 20 client.16014 trim_cache size 0 max 16384 2024-04-25T13:44:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff454051740 1 Processor -- start 2024-04-25T13:44:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff454051740 1 -- start start 2024-04-25T13:44:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff454051740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c26707300 0x559c26ab2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff454051740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ac4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff454051740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 0x559c26aaec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff454051740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff42800c4a0 con 0x559c266f24d0 2024-04-25T13:44:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff454051740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff42800ed20 con 0x559c26707300 2024-04-25T13:44:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff44659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ac4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff44759e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 0x559c26aaec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff44659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ac4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42928/0 (socket says 172.21.15.27:42928) 2024-04-25T13:44:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff44659c640 1 -- 172.21.15.27:0/2320153144 learned_addr learned my addr 172.21.15.27:0/2320153144 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.767+0000 7ff446d9d640 1 --2- 172.21.15.27:0/2320153144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c26707300 0x559c26ab2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.768+0000 7ff454051740 1 -- 172.21.15.27:0/2320153144 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff42800cb10 con 0x559c266ea2f0 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.768+0000 7ff44659c640 1 -- 172.21.15.27:0/2320153144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c26707300 msgr2=0x559c26ab2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.768+0000 7ff44659c640 1 --2- 172.21.15.27:0/2320153144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c26707300 0x559c26ab2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.768+0000 7ff44659c640 1 -- 172.21.15.27:0/2320153144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 msgr2=0x559c26aaec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.768+0000 7ff44659c640 1 --2- 172.21.15.27:0/2320153144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 0x559c26aaec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.768+0000 7ff44659c640 1 -- 172.21.15.27:0/2320153144 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c26ab84d0 con 0x559c266ea2f0 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.768+0000 7ff44659c640 1 --2- 172.21.15.27:0/2320153144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ac4a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff430010c30 tx=0x7ff430013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8c890d82c1bc4e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.769+0000 7ff43cff9640 1 -- 172.21.15.27:0/2320153144 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430017070 con 0x559c266ea2f0 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.769+0000 7ff43cff9640 1 -- 172.21.15.27:0/2320153144 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff43002e070 con 0x559c266ea2f0 2024-04-25T13:44:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.769+0000 7ff43cff9640 1 -- 172.21.15.27:0/2320153144 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43001fab0 con 0x559c266ea2f0 2024-04-25T13:44:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.770+0000 7ff454051740 1 -- 172.21.15.27:0/2320153144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 msgr2=0x559c26ac4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.770+0000 7ff454051740 1 --2- 172.21.15.27:0/2320153144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ac4a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff430010c30 tx=0x7ff430013000 comp rx=0 tx=0).stop 2024-04-25T13:44:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.772+0000 7ff454051740 1 -- 172.21.15.27:0/2320153144 shutdown_connections 2024-04-25T13:44:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.772+0000 7ff454051740 1 --2- 172.21.15.27:0/2320153144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c266f24d0 0x559c26aaec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.772+0000 7ff454051740 1 --2- 172.21.15.27:0/2320153144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266ea2f0 0x559c26ac4a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.772+0000 7ff454051740 1 --2- 172.21.15.27:0/2320153144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c26707300 0x559c26ab2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.772+0000 7ff454051740 1 -- 172.21.15.27:0/2320153144 >> 172.21.15.27:0/2320153144 conn(0x559c26ab7350 msgr2=0x559c26ab2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.772+0000 7ff454051740 1 -- 172.21.15.27:0/2320153144 shutdown_connections 2024-04-25T13:44:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.772+0000 7ff454051740 1 -- 172.21.15.27:0/2320153144 wait complete. 2024-04-25T13:44:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.773+0000 7ff454051740 1 -- start start 2024-04-25T13:44:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.773+0000 7ff454051740 1 Processor -- start 2024-04-25T13:44:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.773+0000 7ff454051740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266ea2f0 0x559c26ac1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff454051740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266f24d0 0x559c26ac1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff454051740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c26707300 0x559c26b53b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff454051740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff42800d1e0 con 0x559c26707300 2024-04-25T13:44:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff44659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266f24d0 0x559c26ac1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff44659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266f24d0 0x559c26ac1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42938/0 (socket says 172.21.15.27:42938) 2024-04-25T13:44:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff44659c640 1 -- 172.21.15.27:0/864853254 learned_addr learned my addr 172.21.15.27:0/864853254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff42800e650 con 0x559c266ea2f0 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff42800d8b0 con 0x559c266f24d0 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff446d9d640 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266ea2f0 0x559c26ac1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff454051740 10 client.0 _ll_register_callbacks cb 0x7ff43000d3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.774+0000 7ff44759e640 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c26707300 0x559c26b53b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.775+0000 7ff44659c640 1 -- 172.21.15.27:0/864853254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266ea2f0 msgr2=0x559c26ac1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.775+0000 7ff44659c640 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266ea2f0 0x559c26ac1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.775+0000 7ff44659c640 1 -- 172.21.15.27:0/864853254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c26707300 msgr2=0x559c26b53b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.775+0000 7ff44659c640 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c26707300 0x559c26b53b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.775+0000 7ff44659c640 1 -- 172.21.15.27:0/864853254 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c26ac4f40 con 0x559c266f24d0 2024-04-25T13:44:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.776+0000 7ff44659c640 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266f24d0 0x559c26ac1da0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff430028650 tx=0x7ff430028680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cad6c4da08d3d591 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.776+0000 7ff42e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.777+0000 7ff42e7fc640 1 -- 172.21.15.27:0/864853254 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43001f9b0 con 0x559c266f24d0 2024-04-25T13:44:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.777+0000 7ff42e7fc640 1 -- 172.21.15.27:0/864853254 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff43001fb50 con 0x559c266f24d0 2024-04-25T13:44:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.777+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff43002e070 con 0x559c266f24d0 2024-04-25T13:44:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.777+0000 7ff42e7fc640 1 -- 172.21.15.27:0/864853254 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff430011e40 con 0x559c266f24d0 2024-04-25T13:44:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.777+0000 7ff454051740 10 client.25747 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.777+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559c26b54fd0 con 0x559c266f24d0 2024-04-25T13:44:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.778+0000 7ff42e7fc640 1 -- 172.21.15.27:0/864853254 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff430015070 con 0x559c266f24d0 2024-04-25T13:44:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.778+0000 7ff42e7fc640 1 -- 172.21.15.27:0/864853254 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff43001a030 con 0x559c266f24d0 2024-04-25T13:44:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.778+0000 7ff454051740 10 client.25747 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.778+0000 7ff454051740 10 client.25747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.778+0000 7ff454051740 20 client.25747 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.778+0000 7ff454051740 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559c267131b0 0x559c26b57980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.778+0000 7ff454051740 4 client.25747 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:44:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.778+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff43002e070 con 0x559c267131b0 2024-04-25T13:44:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.779+0000 7ff446d9d640 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559c267131b0 0x559c26b57980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.780+0000 7ff446d9d640 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559c267131b0 0x559c26b57980 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0965dc15d74954b server_cookie=8276ccd5812ac795 in_seq=0 out_seq=0 2024-04-25T13:44:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.780+0000 7ff42e7fc640 10 client.25747 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.808+0000 7ff42e7fc640 1 -- 172.21.15.27:0/864853254 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff43002e070 con 0x559c267131b0 2024-04-25T13:44:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.809+0000 7ff42e7fc640 10 client.25747 handle_command_reply: tid=0 2024-04-25T13:44:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.809+0000 7ff454051740 1 client.25747 shutdown 2024-04-25T13:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266f24d0 msgr2=0x559c26ac1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266f24d0 0x559c26ac1da0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff430028650 tx=0x7ff430028680 comp rx=0 tx=0).stop 2024-04-25T13:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 shutdown_connections 2024-04-25T13:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559c267131b0 0x559c26b57980 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559c26707300 0x559c26b53b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559c266f24d0 0x559c26ac1da0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 --2- 172.21.15.27:0/864853254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c266ea2f0 0x559c26ac1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 >> 172.21.15.27:0/864853254 conn(0x559c26ab7350 msgr2=0x559c26abcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff42e7fc640 0 client.25747 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 shutdown_connections 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 1 -- 172.21.15.27:0/864853254 wait complete. 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.811+0000 7ff454051740 20 client.25747 trim_cache size 0 max 16384 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:44:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 -- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4340afbe0 msgr2=0x7ff4340b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4340afbe0 0x7ff4340b20f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x559c26ab4b40 tx=0x7ff43000d520 comp rx=0 tx=0).stop 2024-04-25T13:44:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 -- 172.21.15.27:0/99304262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff448110480 msgr2=0x7ff4481b0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:13.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff448110480 0x7ff4481b0450 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff43802b840 tx=0x7ff438008c90 comp rx=0 tx=0).stop 2024-04-25T13:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 -- 172.21.15.27:0/99304262 shutdown_connections 2024-04-25T13:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4340afbe0 0x7ff4340b20f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff448110480 0x7ff4481b0450 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff44810b780 0x7ff4481a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 --2- 172.21.15.27:0/99304262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff44810a540 0x7ff4481a8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.813+0000 7ff44d1a8640 1 -- 172.21.15.27:0/99304262 >> 172.21.15.27:0/99304262 conn(0x7ff448070960 msgr2=0x7ff44810c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.814+0000 7ff44d1a8640 1 -- 172.21.15.27:0/99304262 shutdown_connections 2024-04-25T13:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:13.814+0000 7ff44d1a8640 1 -- 172.21.15.27:0/99304262 wait complete. 2024-04-25T13:44:13.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:13 smithi027 ceph-mon[29140]: pgmap v449: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 48 MiB/s wr, 914 op/s 2024-04-25T13:44:13.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:13 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:44:13.978 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:13 smithi045 ceph-mon[32731]: pgmap v449: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 48 MiB/s wr, 914 op/s 2024-04-25T13:44:13.978 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:13 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:44:14.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:13 smithi122 ceph-mon[32843]: pgmap v449: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 48 MiB/s wr, 914 op/s 2024-04-25T13:44:14.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:13 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:44:14.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:14 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:44:14.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:14 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:44:14.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:14 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:44:14.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:14 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:44:15.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:14 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:44:15.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:44:15.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:44:15.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:14 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:44:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:14 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:44:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:14 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:44:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:14 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:44:15.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:14 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:44:15.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:15 smithi027 ceph-mon[29140]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 35 MiB/s wr, 777 op/s 2024-04-25T13:44:16.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:15 smithi122 ceph-mon[32843]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 35 MiB/s wr, 777 op/s 2024-04-25T13:44:16.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:15 smithi045 ceph-mon[32731]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 35 MiB/s wr, 777 op/s 2024-04-25T13:44:17.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:17 smithi027 ceph-mon[29140]: pgmap v451: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 63 MiB/s wr, 1.57k op/s 2024-04-25T13:44:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 190.0 s, 3150.4 tps, lat 31.824 ms stddev 56.030 2024-04-25T13:44:18.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:17 smithi122 ceph-mon[32843]: pgmap v451: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 63 MiB/s wr, 1.57k op/s 2024-04-25T13:44:18.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:17 smithi045 ceph-mon[32731]: pgmap v451: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 63 MiB/s wr, 1.57k op/s 2024-04-25T13:44:19.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:19 smithi027 ceph-mon[29140]: pgmap v452: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 54 MiB/s wr, 1.43k op/s 2024-04-25T13:44:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:19 smithi122 ceph-mon[32843]: pgmap v452: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 54 MiB/s wr, 1.43k op/s 2024-04-25T13:44:20.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:19 smithi045 ceph-mon[32731]: pgmap v452: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 54 MiB/s wr, 1.43k op/s 2024-04-25T13:44:21.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:21 smithi027 ceph-mon[29140]: pgmap v453: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 60 MiB/s wr, 1.66k op/s 2024-04-25T13:44:22.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:21 smithi122 ceph-mon[32843]: pgmap v453: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 60 MiB/s wr, 1.66k op/s 2024-04-25T13:44:22.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:21 smithi045 ceph-mon[32731]: pgmap v453: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 60 MiB/s wr, 1.66k op/s 2024-04-25T13:44:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 195.0 s, 4406.4 tps, lat 22.623 ms stddev 25.345 2024-04-25T13:44:23.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:23 smithi027 ceph-mon[29140]: pgmap v454: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 56 MiB/s wr, 1.67k op/s 2024-04-25T13:44:24.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:23 smithi122 ceph-mon[32843]: pgmap v454: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 56 MiB/s wr, 1.67k op/s 2024-04-25T13:44:24.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:23 smithi045 ceph-mon[32731]: pgmap v454: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 56 MiB/s wr, 1.67k op/s 2024-04-25T13:44:25.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:25 smithi027 ceph-mon[29140]: pgmap v455: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 51 MiB/s wr, 1.59k op/s 2024-04-25T13:44:26.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:25 smithi122 ceph-mon[32843]: pgmap v455: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 51 MiB/s wr, 1.59k op/s 2024-04-25T13:44:26.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:25 smithi045 ceph-mon[32731]: pgmap v455: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 51 MiB/s wr, 1.59k op/s 2024-04-25T13:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 -- 172.21.15.27:0/1385452698 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 msgr2=0x7f5054103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 --2- 172.21.15.27:0/1385452698 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 0x7f5054103720 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5050009860 tx=0x7f505002a340 comp rx=0 tx=0).stop 2024-04-25T13:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 -- 172.21.15.27:0/1385452698 shutdown_connections 2024-04-25T13:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 --2- 172.21.15.27:0/1385452698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5054108960 0x7f505410ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 --2- 172.21.15.27:0/1385452698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5054103c60 0x7f5054104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 --2- 172.21.15.27:0/1385452698 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 0x7f5054103720 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 -- 172.21.15.27:0/1385452698 >> 172.21.15.27:0/1385452698 conn(0x7f50540fdc10 msgr2=0x7f50541000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 -- 172.21.15.27:0/1385452698 shutdown_connections 2024-04-25T13:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.976+0000 7f505c3eb640 1 -- 172.21.15.27:0/1385452698 wait complete. 2024-04-25T13:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.977+0000 7f505c3eb640 1 Processor -- start 2024-04-25T13:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.977+0000 7f505c3eb640 1 -- start start 2024-04-25T13:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.977+0000 7f505c3eb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 0x7f5054198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505c3eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5054103c60 0x7f5054198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505c3eb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5054108960 0x7f505419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505c3eb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f505410d400 con 0x7f5054108960 2024-04-25T13:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505c3eb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f505410d280 con 0x7f5054103c60 2024-04-25T13:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505c3eb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f505410d580 con 0x7f5054102a20 2024-04-25T13:44:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 0x7f5054198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f5059986640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5054103c60 0x7f5054198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a988640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5054108960 0x7f505419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 0x7f5054198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:60048/0 (socket says 172.21.15.27:60048) 2024-04-25T13:44:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 -- 172.21.15.27:0/3226943686 learned_addr learned my addr 172.21.15.27:0/3226943686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:26.982 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 -- 172.21.15.27:0/3226943686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5054103c60 msgr2=0x7f5054198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5054103c60 0x7f5054198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 -- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5054108960 msgr2=0x7f505419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5054108960 0x7f505419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 -- 172.21.15.27:0/3226943686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505419fb70 con 0x7f5054102a20 2024-04-25T13:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.978+0000 7f505a187640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 0x7f5054198030 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f50500304b0 tx=0x7f5050032d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.979+0000 7f504b7fe640 1 -- 172.21.15.27:0/3226943686 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050037040 con 0x7f5054102a20 2024-04-25T13:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.979+0000 7f504b7fe640 1 -- 172.21.15.27:0/3226943686 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5050030c10 con 0x7f5054102a20 2024-04-25T13:44:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.979+0000 7f504b7fe640 1 -- 172.21.15.27:0/3226943686 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050003850 con 0x7f5054102a20 2024-04-25T13:44:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.979+0000 7f505c3eb640 1 -- 172.21.15.27:0/3226943686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505419fe00 con 0x7f5054102a20 2024-04-25T13:44:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.980+0000 7f505c3eb640 1 -- 172.21.15.27:0/3226943686 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50541a01b0 con 0x7f5054102a20 2024-04-25T13:44:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.983+0000 7f504b7fe640 1 -- 172.21.15.27:0/3226943686 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5050043020 con 0x7f5054102a20 2024-04-25T13:44:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.983+0000 7f5063370740 1 Processor -- start 2024-04-25T13:44:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.984+0000 7f5063370740 1 -- start start 2024-04-25T13:44:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.984+0000 7f5063370740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e5f1b0 0x5636ea20a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.984+0000 7f5063370740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea20cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.984+0000 7f5063370740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e3e4d0 0x5636ea20f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e5f1b0 0x5636ea20a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e5f1b0 0x5636ea20a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57442/0 (socket says 172.21.15.27:57442) 2024-04-25T13:44:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f5063370740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5636ea165460 con 0x5636e9e3e4d0 2024-04-25T13:44:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f5063370740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5636ea11f590 con 0x5636e9e5f1b0 2024-04-25T13:44:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f5063370740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5636ea161420 con 0x5636e9e362f0 2024-04-25T13:44:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a187640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea20cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 -- 172.21.15.27:0/2746359519 learned_addr learned my addr 172.21.15.27:0/2746359519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f504b7fe640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f50400afb60 0x7f50400b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 -- 172.21.15.27:0/2746359519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 msgr2=0x5636ea20cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 --2- 172.21.15.27:0/2746359519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea20cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f5059986640 1 --2- 172.21.15.27:0/2746359519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e3e4d0 0x5636ea20f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 -- 172.21.15.27:0/2746359519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e3e4d0 msgr2=0x5636ea20f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 --2- 172.21.15.27:0/2746359519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e3e4d0 0x5636ea20f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 -- 172.21.15.27:0/2746359519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636ea12c0d0 con 0x5636e9e5f1b0 2024-04-25T13:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f50400afb60 0x7f50400b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f504b7fe640 1 -- 172.21.15.27:0/3226943686 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f50500fdfd0 con 0x7f5054102a20 2024-04-25T13:44:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.985+0000 7f505a988640 1 --2- 172.21.15.27:0/2746359519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e5f1b0 0x5636ea20a7d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f504c009990 tx=0x7f504c00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a4ef151604fc8e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.986+0000 7f5048bfe640 1 -- 172.21.15.27:0/2746359519 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c01c040 con 0x5636e9e5f1b0 2024-04-25T13:44:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.986+0000 7f5048bfe640 1 -- 172.21.15.27:0/2746359519 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f504c023b00 con 0x5636e9e5f1b0 2024-04-25T13:44:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.986+0000 7f5048bfe640 1 -- 172.21.15.27:0/2746359519 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c018850 con 0x5636e9e5f1b0 2024-04-25T13:44:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f505a988640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f50400afb60 0x7f50400b2070 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f504c022000 tx=0x7f504c023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 -- 172.21.15.27:0/2746359519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e5f1b0 msgr2=0x5636ea20a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:27.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 --2- 172.21.15.27:0/2746359519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e5f1b0 0x5636ea20a7d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f504c009990 tx=0x7f504c00c760 comp rx=0 tx=0).stop 2024-04-25T13:44:27.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 -- 172.21.15.27:0/2746359519 shutdown_connections 2024-04-25T13:44:27.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 --2- 172.21.15.27:0/2746359519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e3e4d0 0x5636ea20f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 --2- 172.21.15.27:0/2746359519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea20cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 --2- 172.21.15.27:0/2746359519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e5f1b0 0x5636ea20a7d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 -- 172.21.15.27:0/2746359519 >> 172.21.15.27:0/2746359519 conn(0x5636e9e53300 msgr2=0x5636ea1fb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 -- 172.21.15.27:0/2746359519 shutdown_connections 2024-04-25T13:44:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.987+0000 7f5063370740 1 -- 172.21.15.27:0/2746359519 wait complete. 2024-04-25T13:44:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.988+0000 7f5063370740 1 -- start start 2024-04-25T13:44:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.988+0000 7f5063370740 1 Processor -- start 2024-04-25T13:44:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f5063370740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e362f0 0x5636ea211e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f5063370740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e3e4d0 0x5636ea2123d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f5063370740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 0x5636ea219330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f5063370740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5636ea2100d0 con 0x5636e9e5f1b0 2024-04-25T13:44:27.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f5063370740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5636ea0ecb60 con 0x5636e9e362f0 2024-04-25T13:44:27.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f5063370740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5636ea210250 con 0x5636e9e3e4d0 2024-04-25T13:44:27.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f5063370740 10 client.0 _ll_register_callbacks cb 0x5636ea1112b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a187640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e362f0 0x5636ea211e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a187640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e362f0 0x5636ea211e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57446/0 (socket says 172.21.15.27:57446) 2024-04-25T13:44:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a187640 1 -- 172.21.15.27:0/2421730897 learned_addr learned my addr 172.21.15.27:0/2421730897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a988640 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 0x5636ea219330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f5059986640 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e3e4d0 0x5636ea2123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a187640 1 -- 172.21.15.27:0/2421730897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e3e4d0 msgr2=0x5636ea2123d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a187640 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e3e4d0 0x5636ea2123d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a187640 1 -- 172.21.15.27:0/2421730897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 msgr2=0x5636ea219330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a187640 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 0x5636ea219330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.989+0000 7f505a187640 1 -- 172.21.15.27:0/2421730897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e9e77210 con 0x5636e9e362f0 2024-04-25T13:44:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f5059986640 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e3e4d0 0x5636ea2123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:44:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f505a187640 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e362f0 0x5636ea211e90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f505002a930 tx=0x7f50500045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=120be4a9cd736b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f503dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f503dffb640 1 -- 172.21.15.27:0/2421730897 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50500e1e60 con 0x5636e9e362f0 2024-04-25T13:44:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f503dffb640 1 -- 172.21.15.27:0/2421730897 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f50500cf0c0 con 0x5636e9e362f0 2024-04-25T13:44:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f5063370740 1 -- 172.21.15.27:0/2421730897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f504c023b00 con 0x5636e9e362f0 2024-04-25T13:44:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f503dffb640 1 -- 172.21.15.27:0/2421730897 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f50500c5e30 con 0x5636e9e362f0 2024-04-25T13:44:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f5063370740 10 client.25736 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f5063370740 1 -- 172.21.15.27:0/2421730897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5636e9f43620 con 0x5636e9e362f0 2024-04-25T13:44:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.990+0000 7f503dffb640 1 -- 172.21.15.27:0/2421730897 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50500c5ff0 con 0x5636e9e362f0 2024-04-25T13:44:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f503dffb640 1 -- 172.21.15.27:0/2421730897 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f5050100080 con 0x5636e9e362f0 2024-04-25T13:44:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f5063370740 10 client.25736 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f5063370740 10 client.25736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f5063370740 20 client.25736 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f5063370740 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5636e9e1dd80 0x5636ea21fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f5063370740 4 client.25736 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f5063370740 1 -- 172.21.15.27:0/2421730897 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f504c023b00 con 0x5636e9e1dd80 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f5059986640 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5636e9e1dd80 0x5636ea21fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f5059986640 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5636e9e1dd80 0x5636ea21fb30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9e044035da42ef4 server_cookie=9a08bf9580934b42 in_seq=0 out_seq=0 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.991+0000 7f503dffb640 10 client.25736 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:27.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f503dffb640 1 -- 172.21.15.27:0/2421730897 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f504c023b00 con 0x5636e9e1dd80 2024-04-25T13:44:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f503dffb640 10 client.25736 handle_command_reply: tid=0 2024-04-25T13:44:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 client.25736 shutdown 2024-04-25T13:44:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 -- 172.21.15.27:0/2421730897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e362f0 msgr2=0x5636ea211e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e362f0 0x5636ea211e90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f505002a930 tx=0x7f50500045e0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 -- 172.21.15.27:0/2421730897 shutdown_connections 2024-04-25T13:44:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5636e9e1dd80 0x5636ea21fb30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 0x5636ea219330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e3e4d0 0x5636ea2123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 --2- 172.21.15.27:0/2421730897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e362f0 0x5636ea211e90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 -- 172.21.15.27:0/2421730897 >> 172.21.15.27:0/2421730897 conn(0x5636e9e53300 msgr2=0x5636ea1fa250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f503dffb640 0 client.25736 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 -- 172.21.15.27:0/2421730897 shutdown_connections 2024-04-25T13:44:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.994+0000 7f5063370740 1 -- 172.21.15.27:0/2421730897 wait complete. 2024-04-25T13:44:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:26.995+0000 7f5063370740 20 client.25736 trim_cache size 0 max 16384 2024-04-25T13:44:27.017 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.017+0000 7f5063370740 1 Processor -- start 2024-04-25T13:44:27.017 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.017+0000 7f5063370740 1 -- start start 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.017+0000 7f5063370740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e362f0 0x5636ea209fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.017+0000 7f5063370740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea212480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f5063370740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e5f1b0 0x5636ea1f9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f5063370740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f50541a6fa0 con 0x5636e9e362f0 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f5063370740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f50541a9820 con 0x5636e9e3e4d0 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f5063370740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f50541a7610 con 0x5636e9e5f1b0 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a187640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e362f0 0x5636ea209fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a187640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e362f0 0x5636ea209fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52146/0 (socket says 172.21.15.27:52146) 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a187640 1 -- 172.21.15.27:0/2450920096 learned_addr learned my addr 172.21.15.27:0/2450920096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a988640 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e5f1b0 0x5636ea1f9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f5059986640 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea212480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a988640 1 -- 172.21.15.27:0/2450920096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 msgr2=0x5636ea212480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:27.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a988640 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea212480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a988640 1 -- 172.21.15.27:0/2450920096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e362f0 msgr2=0x5636ea209fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:27.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a988640 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e362f0 0x5636ea209fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.018+0000 7f505a988640 1 -- 172.21.15.27:0/2450920096 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e9e1dfb0 con 0x5636e9e5f1b0 2024-04-25T13:44:27.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.019+0000 7f5059986640 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea212480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:44:27.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.019+0000 7f505a988640 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e5f1b0 0x5636ea1f9f30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f504c00cc60 tx=0x7f504c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e167e3b551912aaa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.019+0000 7f503ffff640 1 -- 172.21.15.27:0/2450920096 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c030020 con 0x5636e9e5f1b0 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.019+0000 7f503ffff640 1 -- 172.21.15.27:0/2450920096 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f504c018810 con 0x5636e9e5f1b0 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.019+0000 7f503ffff640 1 -- 172.21.15.27:0/2450920096 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c01c040 con 0x5636e9e5f1b0 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.020+0000 7f5063370740 1 -- 172.21.15.27:0/2450920096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e5f1b0 msgr2=0x5636ea1f9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.020+0000 7f5063370740 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e5f1b0 0x5636ea1f9f30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f504c00cc60 tx=0x7f504c014000 comp rx=0 tx=0).stop 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 -- 172.21.15.27:0/2450920096 shutdown_connections 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e5f1b0 0x5636ea1f9f30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea212480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 --2- 172.21.15.27:0/2450920096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e362f0 0x5636ea209fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 -- 172.21.15.27:0/2450920096 >> 172.21.15.27:0/2450920096 conn(0x5636e9e53300 msgr2=0x5636ea1f9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 -- 172.21.15.27:0/2450920096 shutdown_connections 2024-04-25T13:44:27.022 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 -- 172.21.15.27:0/2450920096 wait complete. 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 -- start start 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.021+0000 7f5063370740 1 Processor -- start 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5063370740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea215df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5063370740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea216330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5063370740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 0x5636ea217260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5063370740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f50541a7ce0 con 0x5636e9e5f1b0 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5063370740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f50541a9150 con 0x5636e9e3e4d0 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5063370740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f50541a83b0 con 0x5636e9e362f0 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5063370740 10 client.0 _ll_register_callbacks cb 0x7f5050005e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5059986640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea216330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f505a187640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea215df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5059986640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea216330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57460/0 (socket says 172.21.15.27:57460) 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5059986640 1 -- 172.21.15.27:0/3270884998 learned_addr learned my addr 172.21.15.27:0/3270884998 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f505a187640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea215df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:60098/0 (socket says 172.21.15.27:60098) 2024-04-25T13:44:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f505a988640 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 0x5636ea217260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5059986640 1 -- 172.21.15.27:0/3270884998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 msgr2=0x5636ea215df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.022+0000 7f5059986640 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea215df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.023+0000 7f5059986640 1 -- 172.21.15.27:0/3270884998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 msgr2=0x5636ea217260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.023+0000 7f5059986640 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 0x5636ea217260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.023+0000 7f5059986640 1 -- 172.21.15.27:0/3270884998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636ea217820 con 0x5636e9e3e4d0 2024-04-25T13:44:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.023+0000 7f5059986640 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea216330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5044009530 tx=0x7f5044011860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2db31aa55c691684 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.023+0000 7f503dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.023+0000 7f503dffb640 1 -- 172.21.15.27:0/3270884998 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504401d8f0 con 0x5636e9e3e4d0 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.023+0000 7f5063370740 1 -- 172.21.15.27:0/3270884998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f504c018810 con 0x5636e9e3e4d0 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.023+0000 7f503dffb640 1 -- 172.21.15.27:0/3270884998 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5044005020 con 0x5636e9e3e4d0 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f503dffb640 1 -- 172.21.15.27:0/3270884998 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f504402c050 con 0x5636e9e3e4d0 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f5063370740 10 client.25742 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f5063370740 1 -- 172.21.15.27:0/3270884998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5636ea218bb0 con 0x5636e9e3e4d0 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f503dffb640 1 -- 172.21.15.27:0/3270884998 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f504c018810 con 0x5636e9e3e4d0 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f503dffb640 1 -- 172.21.15.27:0/3270884998 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f5044010430 con 0x5636e9e3e4d0 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f5063370740 10 client.25742 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f5063370740 10 client.25742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f5063370740 20 client.25742 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f5063370740 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5636e9df31c0 0x5636ea28a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f5063370740 4 client.25742 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.024+0000 7f5063370740 1 -- 172.21.15.27:0/3270884998 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5636ea173070 con 0x5636e9df31c0 2024-04-25T13:44:27.026 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.025+0000 7f505a187640 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5636e9df31c0 0x5636ea28a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:27.027 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.025+0000 7f505a187640 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5636e9df31c0 0x5636ea28a590 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6e7be4d356217d5 server_cookie=3ce7cce5b8dbae42 in_seq=0 out_seq=0 2024-04-25T13:44:27.027 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.025+0000 7f503dffb640 10 client.25742 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f503dffb640 1 -- 172.21.15.27:0/3270884998 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5636ea173070 con 0x5636e9df31c0 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f503dffb640 10 client.25742 handle_command_reply: tid=0 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 client.25742 shutdown 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 -- 172.21.15.27:0/3270884998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 msgr2=0x5636ea216330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea216330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5044009530 tx=0x7f5044011860 comp rx=0 tx=0).stop 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 -- 172.21.15.27:0/3270884998 shutdown_connections 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5636e9df31c0 0x5636ea28a590 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5636e9e5f1b0 0x5636ea217260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636e9e3e4d0 0x5636ea216330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 --2- 172.21.15.27:0/3270884998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5636e9e362f0 0x5636ea215df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 -- 172.21.15.27:0/3270884998 >> 172.21.15.27:0/3270884998 conn(0x5636e9e53300 msgr2=0x5636ea1fa250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f503dffb640 0 client.25742 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 -- 172.21.15.27:0/3270884998 shutdown_connections 2024-04-25T13:44:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.064+0000 7f5063370740 1 -- 172.21.15.27:0/3270884998 wait complete. 2024-04-25T13:44:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f5063370740 20 client.25742 trim_cache size 0 max 16384 2024-04-25T13:44:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:44:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:44:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:44:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 -- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f50400afb60 msgr2=0x7f50400b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f50400afb60 0x7f50400b2070 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f504c022000 tx=0x7f504c023000 comp rx=0 tx=0).stop 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 -- 172.21.15.27:0/3226943686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 msgr2=0x7f5054198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 0x7f5054198030 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f50500304b0 tx=0x7f5050032d20 comp rx=0 tx=0).stop 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 -- 172.21.15.27:0/3226943686 shutdown_connections 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f50400afb60 0x7f50400b2070 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5054108960 0x7f505419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5054103c60 0x7f5054198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 --2- 172.21.15.27:0/3226943686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5054102a20 0x7f5054198030 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 -- 172.21.15.27:0/3226943686 >> 172.21.15.27:0/3226943686 conn(0x7f50540fdc10 msgr2=0x7f5054104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 -- 172.21.15.27:0/3226943686 shutdown_connections 2024-04-25T13:44:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:27.065+0000 7f505c3eb640 1 -- 172.21.15.27:0/3226943686 wait complete. 2024-04-25T13:44:27.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:27 smithi027 ceph-mon[29140]: pgmap v456: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 52 MiB/s wr, 1.71k op/s 2024-04-25T13:44:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 200.0 s, 3564.4 tps, lat 28.080 ms stddev 30.161 2024-04-25T13:44:28.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:27 smithi122 ceph-mon[32843]: pgmap v456: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 52 MiB/s wr, 1.71k op/s 2024-04-25T13:44:28.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:27 smithi045 ceph-mon[32731]: pgmap v456: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 52 MiB/s wr, 1.71k op/s 2024-04-25T13:44:29.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:29 smithi027 ceph-mon[29140]: pgmap v457: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 20 MiB/s wr, 866 op/s 2024-04-25T13:44:30.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:29 smithi122 ceph-mon[32843]: pgmap v457: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 20 MiB/s wr, 866 op/s 2024-04-25T13:44:30.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:29 smithi045 ceph-mon[32731]: pgmap v457: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 20 MiB/s wr, 866 op/s 2024-04-25T13:44:31.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:31 smithi027 ceph-mon[29140]: pgmap v458: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 19 MiB/s wr, 937 op/s 2024-04-25T13:44:32.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:31 smithi122 ceph-mon[32843]: pgmap v458: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 19 MiB/s wr, 937 op/s 2024-04-25T13:44:32.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:31 smithi045 ceph-mon[32731]: pgmap v458: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 19 MiB/s wr, 937 op/s 2024-04-25T13:44:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 205.0 s, 3258.4 tps, lat 30.607 ms stddev 32.581 2024-04-25T13:44:33.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:33 smithi027 ceph-mon[29140]: pgmap v459: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 840 op/s 2024-04-25T13:44:34.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:33 smithi122 ceph-mon[32843]: pgmap v459: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 840 op/s 2024-04-25T13:44:34.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:33 smithi045 ceph-mon[32731]: pgmap v459: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 840 op/s 2024-04-25T13:44:35.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:35 smithi027 ceph-mon[29140]: pgmap v460: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 751 op/s 2024-04-25T13:44:36.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:35 smithi122 ceph-mon[32843]: pgmap v460: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 751 op/s 2024-04-25T13:44:36.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:35 smithi045 ceph-mon[32731]: pgmap v460: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 751 op/s 2024-04-25T13:44:37.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:37 smithi027 ceph-mon[29140]: pgmap v461: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 11 MiB/s wr, 832 op/s 2024-04-25T13:44:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 210.0 s, 3280.4 tps, lat 30.526 ms stddev 36.946 2024-04-25T13:44:38.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:37 smithi122 ceph-mon[32843]: pgmap v461: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 11 MiB/s wr, 832 op/s 2024-04-25T13:44:38.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:37 smithi045 ceph-mon[32731]: pgmap v461: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 11 MiB/s wr, 832 op/s 2024-04-25T13:44:39.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:39 smithi027 ceph-mon[29140]: pgmap v462: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.6 MiB/s wr, 659 op/s 2024-04-25T13:44:40.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:39 smithi122 ceph-mon[32843]: pgmap v462: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.6 MiB/s wr, 659 op/s 2024-04-25T13:44:40.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:39 smithi045 ceph-mon[32731]: pgmap v462: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.6 MiB/s wr, 659 op/s 2024-04-25T13:44:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.225+0000 7f1828908640 1 -- 172.21.15.27:0/2170389719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182410bc90 msgr2=0x7f182410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.225+0000 7f1828908640 1 --2- 172.21.15.27:0/2170389719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182410bc90 0x7f182410e0b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1818009900 tx=0x7f181802a3f0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.225+0000 7f1828908640 1 -- 172.21.15.27:0/2170389719 shutdown_connections 2024-04-25T13:44:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.225+0000 7f1828908640 1 --2- 172.21.15.27:0/2170389719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182410bc90 0x7f182410e0b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.225+0000 7f1828908640 1 --2- 172.21.15.27:0/2170389719 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1824106f90 0x7f18241073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.225+0000 7f1828908640 1 --2- 172.21.15.27:0/2170389719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1824105d50 0x7f1824106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.225+0000 7f1828908640 1 -- 172.21.15.27:0/2170389719 >> 172.21.15.27:0/2170389719 conn(0x7f1824100f40 msgr2=0x7f18241033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:40.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.226+0000 7f1828908640 1 -- 172.21.15.27:0/2170389719 shutdown_connections 2024-04-25T13:44:40.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.226+0000 7f1828908640 1 -- 172.21.15.27:0/2170389719 wait complete. 2024-04-25T13:44:40.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.227+0000 7f1828908640 1 Processor -- start 2024-04-25T13:44:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f1828908640 1 -- start start 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f1828908640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1824105d50 0x7f182407b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f1828908640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1824106f90 0x7f182407be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f1828908640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182410bc90 0x7f1824078470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f1828908640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1824111880 con 0x7f182410bc90 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f1828908640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1824111700 con 0x7f1824106f90 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f1828908640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1824111a00 con 0x7f1824105d50 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f182259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1824105d50 0x7f182407b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f182259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1824105d50 0x7f182407b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37186/0 (socket says 172.21.15.27:37186) 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f182259c640 1 -- 172.21.15.27:0/3488684635 learned_addr learned my addr 172.21.15.27:0/3488684635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.228+0000 7f1821d9b640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1824106f90 0x7f182407be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.229+0000 7f182259c640 1 -- 172.21.15.27:0/3488684635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1824106f90 msgr2=0x7f182407be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.229+0000 7f182259c640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1824106f90 0x7f182407be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.229+0000 7f182259c640 1 -- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182410bc90 msgr2=0x7f1824078470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.229+0000 7f182259c640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182410bc90 0x7f1824078470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.229+0000 7f182259c640 1 -- 172.21.15.27:0/3488684635 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1824078c90 con 0x7f1824105d50 2024-04-25T13:44:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.229+0000 7f182259c640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1824105d50 0x7f182407b920 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f181c00ea50 tx=0x7f181c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.230+0000 7f18137fe640 1 -- 172.21.15.27:0/3488684635 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c00cdc0 con 0x7f1824105d50 2024-04-25T13:44:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.230+0000 7f18137fe640 1 -- 172.21.15.27:0/3488684635 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181c011920 con 0x7f1824105d50 2024-04-25T13:44:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.230+0000 7f18137fe640 1 -- 172.21.15.27:0/3488684635 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c01aa10 con 0x7f1824105d50 2024-04-25T13:44:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.230+0000 7f1828908640 1 -- 172.21.15.27:0/3488684635 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1824078f80 con 0x7f1824105d50 2024-04-25T13:44:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.230+0000 7f1828908640 1 -- 172.21.15.27:0/3488684635 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f182419c2e0 con 0x7f1824105d50 2024-04-25T13:44:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.234+0000 7f18137fe640 1 -- 172.21.15.27:0/3488684635 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f181c01abb0 con 0x7f1824105d50 2024-04-25T13:44:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.234+0000 7f182f7b6740 1 Processor -- start 2024-04-25T13:44:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182f7b6740 1 -- start start 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892331b0 0x55ef895d3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895d3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892124d0 0x55ef895dadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182f7b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ef89539460 con 0x55ef892331b0 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182f7b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ef894f3590 con 0x55ef8920a2f0 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182f7b6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ef89535420 con 0x55ef892124d0 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895d3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895d3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43336/0 (socket says 172.21.15.27:43336) 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f182259c640 1 -- 172.21.15.27:0/3609366084 learned_addr learned my addr 172.21.15.27:0/3609366084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.235+0000 7f1822d9d640 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892331b0 0x55ef895d3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f1821d9b640 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892124d0 0x55ef895dadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f182259c640 1 -- 172.21.15.27:0/3609366084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892124d0 msgr2=0x55ef895dadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:40.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f182259c640 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892124d0 0x55ef895dadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f182259c640 1 -- 172.21.15.27:0/3609366084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892331b0 msgr2=0x55ef895d3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:40.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f182259c640 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892331b0 0x55ef895d3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f182259c640 1 -- 172.21.15.27:0/3609366084 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef895000d0 con 0x55ef8920a2f0 2024-04-25T13:44:40.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f1822d9d640 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892331b0 0x55ef895d3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:44:40.237 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f182259c640 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895d3ef0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f181c0e7160 tx=0x7f181c0e7620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=998e1f331d0ec871 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.236+0000 7f18137fe640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f180c0afcf0 0x7f180c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.237+0000 7f1810ff9640 1 -- 172.21.15.27:0/3609366084 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c0f2070 con 0x55ef8920a2f0 2024-04-25T13:44:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.237+0000 7f1810ff9640 1 -- 172.21.15.27:0/3609366084 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181c0e7e50 con 0x55ef8920a2f0 2024-04-25T13:44:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.237+0000 7f1810ff9640 1 -- 172.21.15.27:0/3609366084 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c0ed660 con 0x55ef8920a2f0 2024-04-25T13:44:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.237+0000 7f18137fe640 1 -- 172.21.15.27:0/3488684635 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f181c0df060 con 0x7f1824105d50 2024-04-25T13:44:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.237+0000 7f1822d9d640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f180c0afcf0 0x7f180c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 -- 172.21.15.27:0/3609366084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 msgr2=0x55ef895d3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895d3ef0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f181c0e7160 tx=0x7f181c0e7620 comp rx=0 tx=0).stop 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f1822d9d640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f180c0afcf0 0x7f180c0b2200 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1818031500 tx=0x7f1818031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 -- 172.21.15.27:0/3609366084 shutdown_connections 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892124d0 0x55ef895dadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895d3ef0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3609366084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892331b0 0x55ef895d3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 -- 172.21.15.27:0/3609366084 >> 172.21.15.27:0/3609366084 conn(0x55ef89227300 msgr2=0x55ef895cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 -- 172.21.15.27:0/3609366084 shutdown_connections 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.238+0000 7f182f7b6740 1 -- 172.21.15.27:0/3609366084 wait complete. 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.239+0000 7f182f7b6740 1 -- start start 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.239+0000 7f182f7b6740 1 Processor -- start 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.239+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef8920a2f0 0x55ef895dd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895ddbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892331b0 0x55ef895e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f182f7b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ef895db840 con 0x55ef8920a2f0 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f1821d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895ddbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f1822d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892331b0 0x55ef895e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f1821d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895ddbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43350/0 (socket says 172.21.15.27:43350) 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f1821d9b640 1 -- 172.21.15.27:0/3192375178 learned_addr learned my addr 172.21.15.27:0/3192375178 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f1822d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892331b0 0x55ef895e4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37204/0 (socket says 172.21.15.27:37204) 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ef894c0b60 con 0x55ef892124d0 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ef895db9c0 con 0x55ef892331b0 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f182f7b6740 10 client.0 _ll_register_callbacks cb 0x55ef894e52b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.240+0000 7f182259c640 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef8920a2f0 0x55ef895dd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1822d9d640 1 -- 172.21.15.27:0/3192375178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 msgr2=0x55ef895ddbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1822d9d640 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895ddbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1822d9d640 1 -- 172.21.15.27:0/3192375178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef8920a2f0 msgr2=0x55ef895dd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1822d9d640 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef8920a2f0 0x55ef895dd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1822d9d640 1 -- 172.21.15.27:0/3192375178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef8924b210 con 0x55ef892331b0 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1821d9b640 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895ddbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1822d9d640 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892331b0 0x55ef895e4b40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1818009850 tx=0x7f1818005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=989b1d207cc074f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1809ffb640 1 -- 172.21.15.27:0/3192375178 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818042070 con 0x55ef892331b0 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.241+0000 7f1809ffb640 1 -- 172.21.15.27:0/3192375178 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f18180037f0 con 0x55ef892331b0 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.242+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f181c0e7e50 con 0x55ef892331b0 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.242+0000 7f1809ffb640 1 -- 172.21.15.27:0/3192375178 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1818035040 con 0x55ef892331b0 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.242+0000 7f182f7b6740 10 client.25777 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.242+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ef89317620 con 0x55ef892331b0 2024-04-25T13:44:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.242+0000 7f1809ffb640 1 -- 172.21.15.27:0/3192375178 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181c0e7e50 con 0x55ef892331b0 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.243+0000 7f1809ffb640 1 -- 172.21.15.27:0/3192375178 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f181803a040 con 0x55ef892331b0 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.243+0000 7f182f7b6740 10 client.25777 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.243+0000 7f182f7b6740 10 client.25777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.243+0000 7f182f7b6740 20 client.25777 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.243+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ef891f1d80 0x55ef895eb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.243+0000 7f182f7b6740 4 client.25777 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.243+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef89547070 con 0x55ef891f1d80 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.243+0000 7f182259c640 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ef891f1d80 0x55ef895eb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.244+0000 7f182259c640 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ef891f1d80 0x55ef895eb3c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=688d28c8f4ae5897 server_cookie=a54e231d4490b422 in_seq=0 out_seq=0 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.244+0000 7f1809ffb640 10 client.25777 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.245+0000 7f1809ffb640 1 -- 172.21.15.27:0/3192375178 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ef89547070 con 0x55ef891f1d80 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.245+0000 7f1809ffb640 10 client.25777 handle_command_reply: tid=0 2024-04-25T13:44:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.245+0000 7f182f7b6740 1 client.25777 shutdown 2024-04-25T13:44:40.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892331b0 msgr2=0x55ef895e4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892331b0 0x55ef895e4b40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1818009850 tx=0x7f1818005fd0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 shutdown_connections 2024-04-25T13:44:40.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ef891f1d80 0x55ef895eb3c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef892331b0 0x55ef895e4b40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f1809ffb640 0 client.25777 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:40.249 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895ddbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.249 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3192375178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef8920a2f0 0x55ef895dd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.249 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.246+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 >> 172.21.15.27:0/3192375178 conn(0x55ef89227300 msgr2=0x55ef895d2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:40.249 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.247+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 shutdown_connections 2024-04-25T13:44:40.249 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.247+0000 7f182f7b6740 1 -- 172.21.15.27:0/3192375178 wait complete. 2024-04-25T13:44:40.249 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.247+0000 7f182f7b6740 20 client.25777 trim_cache size 0 max 16384 2024-04-25T13:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.270+0000 7f182f7b6740 1 Processor -- start 2024-04-25T13:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.270+0000 7f182f7b6740 1 -- start start 2024-04-25T13:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef891f1d80 0x55ef895d7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef8920a2f0 0x55ef895d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f182f7b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f18241a2e00 con 0x55ef891f1d80 2024-04-25T13:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f182f7b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f18241ba6c0 con 0x55ef892124d0 2024-04-25T13:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f182f7b6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f18241a3470 con 0x55ef8920a2f0 2024-04-25T13:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1822d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f182259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef891f1d80 0x55ef895d7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1822d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895e4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43352/0 (socket says 172.21.15.27:43352) 2024-04-25T13:44:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1822d9d640 1 -- 172.21.15.27:0/1349603296 learned_addr learned my addr 172.21.15.27:0/1349603296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f182259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef891f1d80 0x55ef895d7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44858/0 (socket says 172.21.15.27:44858) 2024-04-25T13:44:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1821d9b640 1 --2- 172.21.15.27:0/1349603296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef8920a2f0 0x55ef895d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1822d9d640 1 -- 172.21.15.27:0/1349603296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef8920a2f0 msgr2=0x55ef895d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1822d9d640 1 --2- 172.21.15.27:0/1349603296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef8920a2f0 0x55ef895d7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1822d9d640 1 -- 172.21.15.27:0/1349603296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef891f1d80 msgr2=0x55ef895d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1822d9d640 1 --2- 172.21.15.27:0/1349603296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef891f1d80 0x55ef895d7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.271+0000 7f1822d9d640 1 -- 172.21.15.27:0/1349603296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef895e4fa0 con 0x55ef892124d0 2024-04-25T13:44:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.272+0000 7f1822d9d640 1 --2- 172.21.15.27:0/1349603296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895e4a60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1818032a30 tx=0x7f181802a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48d014c806c9ab19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.273+0000 7f180bfff640 1 -- 172.21.15.27:0/1349603296 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818063070 con 0x55ef892124d0 2024-04-25T13:44:40.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.273+0000 7f180bfff640 1 -- 172.21.15.27:0/1349603296 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181804e070 con 0x55ef892124d0 2024-04-25T13:44:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.273+0000 7f180bfff640 1 -- 172.21.15.27:0/1349603296 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818042040 con 0x55ef892124d0 2024-04-25T13:44:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 -- 172.21.15.27:0/1349603296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 msgr2=0x55ef895e4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 --2- 172.21.15.27:0/1349603296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895e4a60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1818032a30 tx=0x7f181802a910 comp rx=0 tx=0).stop 2024-04-25T13:44:40.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 -- 172.21.15.27:0/1349603296 shutdown_connections 2024-04-25T13:44:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 --2- 172.21.15.27:0/1349603296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef892124d0 0x55ef895e4a60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 --2- 172.21.15.27:0/1349603296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef8920a2f0 0x55ef895d7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 --2- 172.21.15.27:0/1349603296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef891f1d80 0x55ef895d7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 -- 172.21.15.27:0/1349603296 >> 172.21.15.27:0/1349603296 conn(0x55ef89227300 msgr2=0x55ef895dbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 -- 172.21.15.27:0/1349603296 shutdown_connections 2024-04-25T13:44:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.274+0000 7f182f7b6740 1 -- 172.21.15.27:0/1349603296 wait complete. 2024-04-25T13:44:40.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.275+0000 7f182f7b6740 1 -- start start 2024-04-25T13:44:40.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.275+0000 7f182f7b6740 1 Processor -- start 2024-04-25T13:44:40.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.275+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef891f1d80 0x55ef895e16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895e1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f182f7b6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892124d0 0x55ef89651760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f182f7b6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f18241a3b40 con 0x55ef892124d0 2024-04-25T13:44:40.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1821d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895e1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1821d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895e1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43366/0 (socket says 172.21.15.27:43366) 2024-04-25T13:44:40.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f182259c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef891f1d80 0x55ef895e16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1821d9b640 1 -- 172.21.15.27:0/3807087378 learned_addr learned my addr 172.21.15.27:0/3807087378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:40.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f18241b9ff0 con 0x55ef8920a2f0 2024-04-25T13:44:40.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f18241a4210 con 0x55ef891f1d80 2024-04-25T13:44:40.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1822d9d640 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892124d0 0x55ef89651760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f182f7b6740 10 client.0 _ll_register_callbacks cb 0x7f1814005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:40.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1821d9b640 1 -- 172.21.15.27:0/3807087378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef891f1d80 msgr2=0x55ef895e16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:40.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1821d9b640 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef891f1d80 0x55ef895e16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1821d9b640 1 -- 172.21.15.27:0/3807087378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892124d0 msgr2=0x55ef89651760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:40.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1821d9b640 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892124d0 0x55ef89651760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f1821d9b640 1 -- 172.21.15.27:0/3807087378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef895e4ba0 con 0x55ef8920a2f0 2024-04-25T13:44:40.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.276+0000 7f182259c640 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef891f1d80 0x55ef895e16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:40.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.277+0000 7f1821d9b640 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895e1c30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1814012ca0 tx=0x7f181400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f49fc74e79309540 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:40.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.277+0000 7f1809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:40.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.277+0000 7f1809ffb640 1 -- 172.21.15.27:0/3807087378 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181401e440 con 0x55ef8920a2f0 2024-04-25T13:44:40.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.277+0000 7f1809ffb640 1 -- 172.21.15.27:0/3807087378 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f181401e5e0 con 0x55ef8920a2f0 2024-04-25T13:44:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.277+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f181804e070 con 0x55ef8920a2f0 2024-04-25T13:44:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.278+0000 7f1809ffb640 1 -- 172.21.15.27:0/3807087378 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f181401e7e0 con 0x55ef8920a2f0 2024-04-25T13:44:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.278+0000 7f182f7b6740 10 client.25760 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.278+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ef89652bf0 con 0x55ef8920a2f0 2024-04-25T13:44:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f1809ffb640 1 -- 172.21.15.27:0/3807087378 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f181401ee30 con 0x55ef8920a2f0 2024-04-25T13:44:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f1809ffb640 1 -- 172.21.15.27:0/3807087378 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1814026030 con 0x55ef8920a2f0 2024-04-25T13:44:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f182f7b6740 10 client.25760 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f182f7b6740 10 client.25760 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f182f7b6740 20 client.25760 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ef892331b0 0x55ef89655640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f182f7b6740 4 client.25760 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:44:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f181804e070 con 0x55ef892331b0 2024-04-25T13:44:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.279+0000 7f182259c640 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ef892331b0 0x55ef89655640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.280+0000 7f1809ffb640 10 client.25760 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.280+0000 7f182259c640 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ef892331b0 0x55ef89655640 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=649f03be74e34006 server_cookie=5854c23f67e8060f in_seq=0 out_seq=0 2024-04-25T13:44:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.301+0000 7f1809ffb640 1 -- 172.21.15.27:0/3807087378 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f181804e070 con 0x55ef892331b0 2024-04-25T13:44:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f1809ffb640 10 client.25760 handle_command_reply: tid=0 2024-04-25T13:44:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 client.25760 shutdown 2024-04-25T13:44:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 msgr2=0x55ef895e1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895e1c30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1814012ca0 tx=0x7f181400f790 comp rx=0 tx=0).stop 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 shutdown_connections 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ef892331b0 0x55ef89655640 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ef892124d0 0x55ef89651760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef8920a2f0 0x55ef895e1c30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 --2- 172.21.15.27:0/3807087378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ef891f1d80 0x55ef895e16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 >> 172.21.15.27:0/3807087378 conn(0x55ef89227300 msgr2=0x55ef895d7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f1809ffb640 0 client.25760 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 shutdown_connections 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 1 -- 172.21.15.27:0/3807087378 wait complete. 2024-04-25T13:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.302+0000 7f182f7b6740 20 client.25760 trim_cache size 0 max 16384 2024-04-25T13:44:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:44:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:44:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:44:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 -- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f180c0afcf0 msgr2=0x7f180c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f180c0afcf0 0x7f180c0b2200 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1818031500 tx=0x7f1818031400 comp rx=0 tx=0).stop 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 -- 172.21.15.27:0/3488684635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1824105d50 msgr2=0x7f182407b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1824105d50 0x7f182407b920 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f181c00ea50 tx=0x7f181c00ef10 comp rx=0 tx=0).stop 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 -- 172.21.15.27:0/3488684635 shutdown_connections 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f180c0afcf0 0x7f180c0b2200 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182410bc90 0x7f1824078470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1824106f90 0x7f182407be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 --2- 172.21.15.27:0/3488684635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1824105d50 0x7f182407b920 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 -- 172.21.15.27:0/3488684635 >> 172.21.15.27:0/3488684635 conn(0x7f1824100f40 msgr2=0x7f1824107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.303+0000 7f1828908640 1 -- 172.21.15.27:0/3488684635 shutdown_connections 2024-04-25T13:44:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:40.304+0000 7f1828908640 1 -- 172.21.15.27:0/3488684635 wait complete. 2024-04-25T13:44:41.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:44:41.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 -- 172.21.15.27:0/1435862293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010a3d0 msgr2=0x7f999010adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 --2- 172.21.15.27:0/1435862293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010a3d0 0x7f999010adf0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f998c009860 tx=0x7f998c02a340 comp rx=0 tx=0).stop 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 -- 172.21.15.27:0/1435862293 shutdown_connections 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 --2- 172.21.15.27:0/1435862293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9990110030 0x7f9990112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 --2- 172.21.15.27:0/1435862293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f999010b330 0x7f999010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 --2- 172.21.15.27:0/1435862293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010a3d0 0x7f999010adf0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 -- 172.21.15.27:0/1435862293 >> 172.21.15.27:0/1435862293 conn(0x7f9990078de0 msgr2=0x7f999007b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 -- 172.21.15.27:0/1435862293 shutdown_connections 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.237+0000 7f9997ad1640 1 -- 172.21.15.27:0/1435862293 wait complete. 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f9997ad1640 1 Processor -- start 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f9997ad1640 1 -- start start 2024-04-25T13:44:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f9997ad1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f999010a3d0 0x7f99901a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f9997ad1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010b330 0x7f99901a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f9997ad1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9990110030 0x7f99901a7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f9997ad1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9990115c10 con 0x7f999010b330 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f9997ad1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9990115a90 con 0x7f9990110030 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f9997ad1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9990115d90 con 0x7f999010a3d0 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999506c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010b330 0x7f99901a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999506c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010b330 0x7f99901a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44892/0 (socket says 172.21.15.27:44892) 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999506c640 1 -- 172.21.15.27:0/2038626119 learned_addr learned my addr 172.21.15.27:0/2038626119 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999586d640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f999010a3d0 0x7f99901a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999606e640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9990110030 0x7f99901a7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999506c640 1 -- 172.21.15.27:0/2038626119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f999010a3d0 msgr2=0x7f99901a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999506c640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f999010a3d0 0x7f99901a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999506c640 1 -- 172.21.15.27:0/2038626119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9990110030 msgr2=0x7f99901a7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999506c640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9990110030 0x7f99901a7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999506c640 1 -- 172.21.15.27:0/2038626119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99901a8360 con 0x7f999010b330 2024-04-25T13:44:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999606e640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9990110030 0x7f99901a7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:44:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.238+0000 7f999586d640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f999010a3d0 0x7f99901a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.239+0000 7f999506c640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010b330 0x7f99901a0d60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f998000e9f0 tx=0x7f998000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:41.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.240+0000 7f9986ffd640 1 -- 172.21.15.27:0/2038626119 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998000ccc0 con 0x7f999010b330 2024-04-25T13:44:41.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.241+0000 7f9997ad1640 1 -- 172.21.15.27:0/2038626119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99901a8650 con 0x7f999010b330 2024-04-25T13:44:41.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.241+0000 7f9997ad1640 1 -- 172.21.15.27:0/2038626119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99901a8ba0 con 0x7f999010b330 2024-04-25T13:44:41.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.241+0000 7f9986ffd640 1 -- 172.21.15.27:0/2038626119 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99800054d0 con 0x7f999010b330 2024-04-25T13:44:41.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.241+0000 7f9986ffd640 1 -- 172.21.15.27:0/2038626119 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9980010430 con 0x7f999010b330 2024-04-25T13:44:41.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.246+0000 7f9986ffd640 1 -- 172.21.15.27:0/2038626119 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f99800106d0 con 0x7f999010b330 2024-04-25T13:44:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.247+0000 7f999e97d740 1 Processor -- start 2024-04-25T13:44:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.248+0000 7f9986ffd640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f997c0afcc0 0x7f997c0b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.248+0000 7f9986ffd640 1 -- 172.21.15.27:0/2038626119 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f99800046c0 con 0x7f999010b330 2024-04-25T13:44:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.250+0000 7f999e97d740 1 -- start start 2024-04-25T13:44:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.251+0000 7f999e97d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb6c390 0x55e27bf0da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.251+0000 7f999e97d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb434d0 0x55e27bf0dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.251+0000 7f999e97d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 0x55e27bf0ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.251+0000 7f999e97d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e27be72d80 con 0x55e27bb6c390 2024-04-25T13:44:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.251+0000 7f999e97d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e27be2c8b0 con 0x55e27bb4b6b0 2024-04-25T13:44:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.251+0000 7f999e97d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e27be6ed40 con 0x55e27bb434d0 2024-04-25T13:44:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.252+0000 7f999586d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb6c390 0x55e27bf0da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.252+0000 7f999586d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb6c390 0x55e27bf0da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44894/0 (socket says 172.21.15.27:44894) 2024-04-25T13:44:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.252+0000 7f999586d640 1 -- 172.21.15.27:0/4046960590 learned_addr learned my addr 172.21.15.27:0/4046960590 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.253+0000 7f999586d640 1 -- 172.21.15.27:0/4046960590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb434d0 msgr2=0x55e27bf0dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:44:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.253+0000 7f999586d640 1 --2- 172.21.15.27:0/4046960590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb434d0 0x55e27bf0dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.253+0000 7f999586d640 1 -- 172.21.15.27:0/4046960590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 msgr2=0x55e27bf0ec70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:44:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.253+0000 7f999586d640 1 --2- 172.21.15.27:0/4046960590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 0x55e27bf0ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.253+0000 7f999586d640 1 -- 172.21.15.27:0/4046960590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e27bdfda40 con 0x55e27bb6c390 2024-04-25T13:44:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.254+0000 7f999586d640 1 --2- 172.21.15.27:0/4046960590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb6c390 0x55e27bf0da60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f998c02a930 tx=0x7f998c005cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b09db615c1cac12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:41.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.255+0000 7f997bfff640 1 -- 172.21.15.27:0/4046960590 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c037040 con 0x55e27bb6c390 2024-04-25T13:44:41.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.255+0000 7f997bfff640 1 -- 172.21.15.27:0/4046960590 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f998c030e10 con 0x55e27bb6c390 2024-04-25T13:44:41.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.255+0000 7f997bfff640 1 -- 172.21.15.27:0/4046960590 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c032b50 con 0x55e27bb6c390 2024-04-25T13:44:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.256+0000 7f999606e640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f997c0afcc0 0x7f997c0b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.257+0000 7f999e97d740 1 -- 172.21.15.27:0/4046960590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb6c390 msgr2=0x55e27bf0da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.257+0000 7f999e97d740 1 --2- 172.21.15.27:0/4046960590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb6c390 0x55e27bf0da60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f998c02a930 tx=0x7f998c005cd0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.261+0000 7f999e97d740 1 -- 172.21.15.27:0/4046960590 shutdown_connections 2024-04-25T13:44:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.261+0000 7f999e97d740 1 --2- 172.21.15.27:0/4046960590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 0x55e27bf0ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.261+0000 7f999e97d740 1 --2- 172.21.15.27:0/4046960590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb434d0 0x55e27bf0dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.261+0000 7f999e97d740 1 --2- 172.21.15.27:0/4046960590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb6c390 0x55e27bf0da60 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.261+0000 7f999e97d740 1 -- 172.21.15.27:0/4046960590 >> 172.21.15.27:0/4046960590 conn(0x55e27bb604e0 msgr2=0x55e27bf09270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:41.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.266+0000 7f999606e640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f997c0afcc0 0x7f997c0b21d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9988005fd0 tx=0x7f9988009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:41.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.270+0000 7f999e97d740 1 -- 172.21.15.27:0/4046960590 shutdown_connections 2024-04-25T13:44:41.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.271+0000 7f999e97d740 1 -- 172.21.15.27:0/4046960590 wait complete. 2024-04-25T13:44:41.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.271+0000 7f999e97d740 1 -- start start 2024-04-25T13:44:41.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.272+0000 7f999e97d740 1 Processor -- start 2024-04-25T13:44:41.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.272+0000 7f999e97d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb434d0 0x55e27bf17930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.272+0000 7f999e97d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb4b6b0 0x55e27bf17e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.273+0000 7f999e97d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.273+0000 7f999e97d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e27bf0f720 con 0x55e27bb4b6b0 2024-04-25T13:44:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.273+0000 7f999e97d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e27bdf9d10 con 0x55e27bb6c390 2024-04-25T13:44:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.273+0000 7f999e97d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e27bf0f8a0 con 0x55e27bb434d0 2024-04-25T13:44:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.273+0000 7f999e97d740 10 client.0 _ll_register_callbacks cb 0x55e27be26210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.274+0000 7f999606e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.274+0000 7f999606e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43388/0 (socket says 172.21.15.27:43388) 2024-04-25T13:44:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.274+0000 7f999606e640 1 -- 172.21.15.27:0/442152332 learned_addr learned my addr 172.21.15.27:0/442152332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:41.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.275+0000 7f999506c640 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb4b6b0 0x55e27bf17e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.275+0000 7f999606e640 1 -- 172.21.15.27:0/442152332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb434d0 msgr2=0x55e27bf17930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:41.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.275+0000 7f999606e640 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb434d0 0x55e27bf17930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.275+0000 7f999606e640 1 -- 172.21.15.27:0/442152332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb4b6b0 msgr2=0x55e27bf17e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:41.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.275+0000 7f999606e640 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb4b6b0 0x55e27bf17e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.275+0000 7f999606e640 1 -- 172.21.15.27:0/442152332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e27bc4e7d0 con 0x55e27bb6c390 2024-04-25T13:44:41.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.276+0000 7f999606e640 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1edd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9988011c90 tx=0x7f9988004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69458cfd05422e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:41.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.276+0000 7f99797fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:41.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.278+0000 7f99797fa640 1 -- 172.21.15.27:0/442152332 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998801e620 con 0x55e27bb6c390 2024-04-25T13:44:41.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.278+0000 7f99797fa640 1 -- 172.21.15.27:0/442152332 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f998801e7c0 con 0x55e27bb6c390 2024-04-25T13:44:41.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.278+0000 7f999e97d740 1 -- 172.21.15.27:0/442152332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f998c030e10 con 0x55e27bb6c390 2024-04-25T13:44:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.284+0000 7f99797fa640 1 -- 172.21.15.27:0/442152332 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f998801e9c0 con 0x55e27bb6c390 2024-04-25T13:44:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.285+0000 7f999e97d740 10 client.25769 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.285+0000 7f999e97d740 1 -- 172.21.15.27:0/442152332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e27bb843f0 con 0x55e27bb6c390 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.285+0000 7f99797fa640 1 -- 172.21.15.27:0/442152332 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f998c030e10 con 0x55e27bb6c390 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.286+0000 7f99797fa640 1 -- 172.21.15.27:0/442152332 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9988030b00 con 0x55e27bb6c390 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.286+0000 7f999e97d740 10 client.25769 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.286+0000 7f999e97d740 10 client.25769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.286+0000 7f999e97d740 20 client.25769 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.286+0000 7f999e97d740 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27bb2af60 0x55e27bf253c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.286+0000 7f999e97d740 4 client.25769 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.286+0000 7f999e97d740 1 -- 172.21.15.27:0/442152332 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e27be80990 con 0x55e27bb2af60 2024-04-25T13:44:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.287+0000 7f999586d640 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27bb2af60 0x55e27bf253c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.288+0000 7f999586d640 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27bb2af60 0x55e27bf253c0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1afe32d8e1801b11 server_cookie=81ed7da815f59b61 in_seq=0 out_seq=0 2024-04-25T13:44:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.288+0000 7f99797fa640 10 client.25769 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.289+0000 7f99797fa640 1 -- 172.21.15.27:0/442152332 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e27be80990 con 0x55e27bb2af60 2024-04-25T13:44:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.289+0000 7f99797fa640 10 client.25769 handle_command_reply: tid=0 2024-04-25T13:44:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.291+0000 7f999e97d740 1 client.25769 shutdown 2024-04-25T13:44:41.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.295+0000 7f999e97d740 1 -- 172.21.15.27:0/442152332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 msgr2=0x55e27bf1edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.295+0000 7f999e97d740 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1edd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9988011c90 tx=0x7f9988004ab0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.295+0000 7f999e97d740 1 -- 172.21.15.27:0/442152332 shutdown_connections 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.295+0000 7f999e97d740 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27bb2af60 0x55e27bf253c0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.295+0000 7f999e97d740 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1edd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.295+0000 7f999e97d740 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb4b6b0 0x55e27bf17e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.295+0000 7f999e97d740 1 --2- 172.21.15.27:0/442152332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb434d0 0x55e27bf17930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.295+0000 7f999e97d740 1 -- 172.21.15.27:0/442152332 >> 172.21.15.27:0/442152332 conn(0x55e27bb604e0 msgr2=0x55e27bf083f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.296+0000 7f99797fa640 0 client.25769 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.296+0000 7f999e97d740 1 -- 172.21.15.27:0/442152332 shutdown_connections 2024-04-25T13:44:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.296+0000 7f999e97d740 1 -- 172.21.15.27:0/442152332 wait complete. 2024-04-25T13:44:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.298+0000 7f999e97d740 20 client.25769 trim_cache size 0 max 16384 2024-04-25T13:44:41.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.326+0000 7f999e97d740 1 Processor -- start 2024-04-25T13:44:41.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999e97d740 1 -- start start 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999e97d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf15f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999e97d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 0x55e27bf164b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999e97d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb6c390 0x55e27bf1ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999e97d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f997400c460 con 0x55e27bb434d0 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999e97d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f997400ece0 con 0x55e27bb4b6b0 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999e97d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f997400cad0 con 0x55e27bb6c390 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999586d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf15f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999586d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf15f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44910/0 (socket says 172.21.15.27:44910) 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999586d640 1 -- 172.21.15.27:0/3177625986 learned_addr learned my addr 172.21.15.27:0/3177625986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999506c640 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 0x55e27bf164b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.327+0000 7f999606e640 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb6c390 0x55e27bf1ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f999606e640 1 -- 172.21.15.27:0/3177625986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 msgr2=0x55e27bf164b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f999606e640 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 0x55e27bf164b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f999606e640 1 -- 172.21.15.27:0/3177625986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 msgr2=0x55e27bf15f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:41.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f999606e640 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf15f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f999606e640 1 -- 172.21.15.27:0/3177625986 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e27bb440f0 con 0x55e27bb6c390 2024-04-25T13:44:41.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f999586d640 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf15f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:41.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f999606e640 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb6c390 0x55e27bf1ecf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9988030f50 tx=0x7f9988009dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dd7501c2e3dd921 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f997b7fe640 1 -- 172.21.15.27:0/3177625986 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988035040 con 0x55e27bb6c390 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f997b7fe640 1 -- 172.21.15.27:0/3177625986 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f998801ed60 con 0x55e27bb6c390 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.328+0000 7f997b7fe640 1 -- 172.21.15.27:0/3177625986 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988030a30 con 0x55e27bb6c390 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 -- 172.21.15.27:0/3177625986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb6c390 msgr2=0x55e27bf1ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb6c390 0x55e27bf1ecf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9988030f50 tx=0x7f9988009dd0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 -- 172.21.15.27:0/3177625986 shutdown_connections 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb6c390 0x55e27bf1ecf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb4b6b0 0x55e27bf164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 --2- 172.21.15.27:0/3177625986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf15f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 -- 172.21.15.27:0/3177625986 >> 172.21.15.27:0/3177625986 conn(0x55e27bb604e0 msgr2=0x55e27bf12660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 -- 172.21.15.27:0/3177625986 shutdown_connections 2024-04-25T13:44:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 -- 172.21.15.27:0/3177625986 wait complete. 2024-04-25T13:44:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 -- start start 2024-04-25T13:44:41.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.330+0000 7f999e97d740 1 Processor -- start 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999e97d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf1f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999e97d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb4b6b0 0x55e27bf1c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999e97d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999e97d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f997400d1a0 con 0x55e27bb434d0 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999e97d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f997400e610 con 0x55e27bb6c390 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999e97d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f997400d870 con 0x55e27bb4b6b0 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999e97d740 10 client.0 _ll_register_callbacks cb 0x7f9988009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999506c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb4b6b0 0x55e27bf1c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999506c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb4b6b0 0x55e27bf1c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37290/0 (socket says 172.21.15.27:37290) 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999506c640 1 -- 172.21.15.27:0/4031159655 learned_addr learned my addr 172.21.15.27:0/4031159655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.331+0000 7f999606e640 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f999586d640 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf1f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f999506c640 1 -- 172.21.15.27:0/4031159655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 msgr2=0x55e27bf1c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f999506c640 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f999506c640 1 -- 172.21.15.27:0/4031159655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 msgr2=0x55e27bf1f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f999506c640 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf1f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f999506c640 1 -- 172.21.15.27:0/4031159655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e27be334b0 con 0x55e27bb4b6b0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f999506c640 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb4b6b0 0x55e27bf1c010 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99800a7fb0 tx=0x7f99800a8850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c660189e046353a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f99797fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f99797fa640 1 -- 172.21.15.27:0/4031159655 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99800a89b0 con 0x55e27bb4b6b0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f99797fa640 1 -- 172.21.15.27:0/4031159655 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9980005290 con 0x55e27bb4b6b0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.332+0000 7f999e97d740 1 -- 172.21.15.27:0/4031159655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f998801ed60 con 0x55e27bb4b6b0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f99797fa640 1 -- 172.21.15.27:0/4031159655 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f99800b9070 con 0x55e27bb4b6b0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999e97d740 10 client.25795 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999e97d740 1 -- 172.21.15.27:0/4031159655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e27bf8c820 con 0x55e27bb4b6b0 2024-04-25T13:44:41.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f99797fa640 1 -- 172.21.15.27:0/4031159655 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f998801ed60 con 0x55e27bb4b6b0 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f99797fa640 1 -- 172.21.15.27:0/4031159655 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f99800c2ba0 con 0x55e27bb4b6b0 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999e97d740 10 client.25795 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999e97d740 10 client.25795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999e97d740 20 client.25795 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999e97d740 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27bb003a0 0x55e27bf8f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999e97d740 4 client.25795 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999e97d740 1 -- 172.21.15.27:0/4031159655 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e27bf0e3e0 con 0x55e27bb003a0 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.333+0000 7f999586d640 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27bb003a0 0x55e27bf8f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.334+0000 7f99797fa640 10 client.25795 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.334+0000 7f999586d640 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27bb003a0 0x55e27bf8f210 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97e2b054e7c4dee9 server_cookie=a07b8b851c1e1540 in_seq=0 out_seq=0 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.334+0000 7f99797fa640 1 -- 172.21.15.27:0/4031159655 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e27bf0e3e0 con 0x55e27bb003a0 2024-04-25T13:44:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.335+0000 7f99797fa640 10 client.25795 handle_command_reply: tid=0 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.335+0000 7f999e97d740 1 client.25795 shutdown 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.335+0000 7f999e97d740 1 -- 172.21.15.27:0/4031159655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb4b6b0 msgr2=0x55e27bf1c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.335+0000 7f999e97d740 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb4b6b0 0x55e27bf1c010 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99800a7fb0 tx=0x7f99800a8850 comp rx=0 tx=0).stop 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 1 -- 172.21.15.27:0/4031159655 shutdown_connections 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27bb003a0 0x55e27bf8f210 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27bb6c390 0x55e27bf1c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27bb4b6b0 0x55e27bf1c010 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 1 --2- 172.21.15.27:0/4031159655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27bb434d0 0x55e27bf1f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 1 -- 172.21.15.27:0/4031159655 >> 172.21.15.27:0/4031159655 conn(0x55e27bb604e0 msgr2=0x55e27bf0c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f99797fa640 0 client.25795 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 1 -- 172.21.15.27:0/4031159655 shutdown_connections 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 1 -- 172.21.15.27:0/4031159655 wait complete. 2024-04-25T13:44:41.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.336+0000 7f999e97d740 20 client.25795 trim_cache size 0 max 16384 2024-04-25T13:44:41.338 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:44:41.338 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:44:41.338 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:44:41.338 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 -- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f997c0afcc0 msgr2=0x7f997c0b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f997c0afcc0 0x7f997c0b21d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9988005fd0 tx=0x7f9988009500 comp rx=0 tx=0).stop 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 -- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010b330 msgr2=0x7f99901a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010b330 0x7f99901a0d60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f998000e9f0 tx=0x7f998000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 -- 172.21.15.27:0/2038626119 shutdown_connections 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f997c0afcc0 0x7f997c0b21d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9990110030 0x7f99901a7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999010b330 0x7f99901a0d60 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 --2- 172.21.15.27:0/2038626119 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f999010a3d0 0x7f99901a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 -- 172.21.15.27:0/2038626119 >> 172.21.15.27:0/2038626119 conn(0x7f9990078de0 msgr2=0x7f999010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:41.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 -- 172.21.15.27:0/2038626119 shutdown_connections 2024-04-25T13:44:41.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.337+0000 7f9997ad1640 1 -- 172.21.15.27:0/2038626119 wait complete. 2024-04-25T13:44:41.352 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:44:41.353 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:44:41.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.520+0000 7f11cd74d640 1 -- 172.21.15.27:0/98351428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c8107670 msgr2=0x7f11c8107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.520+0000 7f11cd74d640 1 --2- 172.21.15.27:0/98351428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c8107670 0x7f11c8107a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009ca0 tx=0x7f11c002a8d0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.520+0000 7f11cd74d640 1 -- 172.21.15.27:0/98351428 shutdown_connections 2024-04-25T13:44:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.520+0000 7f11cd74d640 1 --2- 172.21.15.27:0/98351428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c8107670 0x7f11c8107a80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.520+0000 7f11cd74d640 1 --2- 172.21.15.27:0/98351428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8104d10 0x7f11c8107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.520+0000 7f11cd74d640 1 --2- 172.21.15.27:0/98351428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c81023b0 0x7f11c81047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.520+0000 7f11cd74d640 1 -- 172.21.15.27:0/98351428 >> 172.21.15.27:0/98351428 conn(0x7f11c80fdba0 msgr2=0x7f11c8100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.521+0000 7f11cd74d640 1 -- 172.21.15.27:0/98351428 shutdown_connections 2024-04-25T13:44:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.521+0000 7f11cd74d640 1 -- 172.21.15.27:0/98351428 wait complete. 2024-04-25T13:44:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.521+0000 7f11cd74d640 1 Processor -- start 2024-04-25T13:44:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.522+0000 7f11cd74d640 1 -- start start 2024-04-25T13:44:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.522+0000 7f11cd74d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c81023b0 0x7f11c810daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.522+0000 7f11cd74d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8104d10 0x7f11c810e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.522+0000 7f11cd74d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8107670 0x7f11c810ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.522+0000 7f11cd74d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f11c8115cd0 con 0x7f11c8104d10 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.522+0000 7f11cd74d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f11c8115b50 con 0x7f11c8107670 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.522+0000 7f11cd74d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f11c8115e50 con 0x7f11c81023b0 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8104d10 0x7f11c810e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c81023b0 0x7f11c810daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8104d10 0x7f11c810e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44940/0 (socket says 172.21.15.27:44940) 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c67fc640 1 -- 172.21.15.27:0/4002495288 learned_addr learned my addr 172.21.15.27:0/4002495288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c77fe640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8107670 0x7f11c810ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c67fc640 1 -- 172.21.15.27:0/4002495288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c81023b0 msgr2=0x7f11c810daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c67fc640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c81023b0 0x7f11c810daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c67fc640 1 -- 172.21.15.27:0/4002495288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8107670 msgr2=0x7f11c810ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c67fc640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8107670 0x7f11c810ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.523+0000 7f11c67fc640 1 -- 172.21.15.27:0/4002495288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c810f600 con 0x7f11c8104d10 2024-04-25T13:44:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.524+0000 7f11c67fc640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8104d10 0x7f11c810e030 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00b4a0 tx=0x7f11bc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.524+0000 7f11c6ffd640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c81023b0 0x7f11c810daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:44:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.524+0000 7f11affff640 1 -- 172.21.15.27:0/4002495288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc014070 con 0x7f11c8104d10 2024-04-25T13:44:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.524+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c810f8f0 con 0x7f11c8104d10 2024-04-25T13:44:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.524+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c810ff10 con 0x7f11c8104d10 2024-04-25T13:44:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.524+0000 7f11affff640 1 -- 172.21.15.27:0/4002495288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11bc010ac0 con 0x7f11c8104d10 2024-04-25T13:44:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.524+0000 7f11affff640 1 -- 172.21.15.27:0/4002495288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc017c40 con 0x7f11c8104d10 2024-04-25T13:44:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.525+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b40052c0 con 0x7f11c8104d10 2024-04-25T13:44:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.527+0000 7f11affff640 1 -- 172.21.15.27:0/4002495288 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f11bc017070 con 0x7f11c8104d10 2024-04-25T13:44:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.528+0000 7f11affff640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11b00afcf0 0x7f11b00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.528+0000 7f11affff640 1 -- 172.21.15.27:0/4002495288 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f11bc0179e0 con 0x7f11c8104d10 2024-04-25T13:44:41.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.528+0000 7f11c6ffd640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11b00afcf0 0x7f11b00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:41.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.529+0000 7f11c6ffd640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11b00afcf0 0x7f11b00b2200 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f11b8006fd0 tx=0x7f11b8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.532+0000 7f11affff640 1 -- 172.21.15.27:0/4002495288 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f11bc0a7140 con 0x7f11c8104d10 2024-04-25T13:44:41.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.892+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11b40055b0 con 0x7f11c8104d10 2024-04-25T13:44:41.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.893+0000 7f11affff640 1 -- 172.21.15.27:0/4002495288 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f11bc0af1c0 con 0x7f11c8104d10 2024-04-25T13:44:41.893 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:44:41.894 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:44:41.894 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:44:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11b00afcf0 msgr2=0x7f11b00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11b00afcf0 0x7f11b00b2200 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f11b8006fd0 tx=0x7f11b8008040 comp rx=0 tx=0).stop 2024-04-25T13:44:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8104d10 msgr2=0x7f11c810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8104d10 0x7f11c810e030 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00b4a0 tx=0x7f11bc00b960 comp rx=0 tx=0).stop 2024-04-25T13:44:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 shutdown_connections 2024-04-25T13:44:41.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f11b00afcf0 0x7f11b00b2200 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8107670 0x7f11c810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c8104d10 0x7f11c810e030 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 --2- 172.21.15.27:0/4002495288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f11c81023b0 0x7f11c810daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:41.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 >> 172.21.15.27:0/4002495288 conn(0x7f11c80fdba0 msgr2=0x7f11c81031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:41.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 shutdown_connections 2024-04-25T13:44:41.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:41.897+0000 7f11cd74d640 1 -- 172.21.15.27:0/4002495288 wait complete. 2024-04-25T13:44:41.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:41 smithi027 ceph-mon[29140]: pgmap v463: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.7 MiB/s wr, 751 op/s 2024-04-25T13:44:41.915 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:44:42.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:41 smithi122 ceph-mon[32843]: pgmap v463: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.7 MiB/s wr, 751 op/s 2024-04-25T13:44:42.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:41 smithi045 ceph-mon[32731]: pgmap v463: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.7 MiB/s wr, 751 op/s 2024-04-25T13:44:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.091+0000 7f0cbbed8640 1 -- 172.21.15.27:0/723985968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb4103c30 msgr2=0x7f0cb4104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.091+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/723985968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb4103c30 0x7f0cb4104040 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009900 tx=0x7f0ca402a410 comp rx=0 tx=0).stop 2024-04-25T13:44:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.092+0000 7f0cbbed8640 1 -- 172.21.15.27:0/723985968 shutdown_connections 2024-04-25T13:44:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.092+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/723985968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb4108930 0x7f0cb410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.092+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/723985968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb4103c30 0x7f0cb4104040 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.092+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/723985968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cb41029f0 0x7f0cb41036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.092+0000 7f0cbbed8640 1 -- 172.21.15.27:0/723985968 >> 172.21.15.27:0/723985968 conn(0x7f0cb40fdc00 msgr2=0x7f0cb4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.092+0000 7f0cbbed8640 1 -- 172.21.15.27:0/723985968 shutdown_connections 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.092+0000 7f0cbbed8640 1 -- 172.21.15.27:0/723985968 wait complete. 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cbbed8640 1 Processor -- start 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cbbed8640 1 -- start start 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cbbed8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb41029f0 0x7f0cb419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cbbed8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cb4103c30 0x7f0cb419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cbbed8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb4108930 0x7f0cb41a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cbbed8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb410e510 con 0x7f0cb41029f0 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cbbed8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb410e390 con 0x7f0cb4108930 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cbbed8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb410e690 con 0x7f0cb4103c30 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cb9c74640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb41029f0 0x7f0cb419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cb9c74640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb41029f0 0x7f0cb419a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44956/0 (socket says 172.21.15.27:44956) 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cba475640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb4108930 0x7f0cb41a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cb9c74640 1 -- 172.21.15.27:0/1846229694 learned_addr learned my addr 172.21.15.27:0/1846229694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.093+0000 7f0cb9473640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cb4103c30 0x7f0cb419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.094+0000 7f0cb9c74640 1 -- 172.21.15.27:0/1846229694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cb4103c30 msgr2=0x7f0cb419a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.094+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cb4103c30 0x7f0cb419a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.094+0000 7f0cb9c74640 1 -- 172.21.15.27:0/1846229694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb4108930 msgr2=0x7f0cb41a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.094+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb4108930 0x7f0cb41a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.094+0000 7f0cb9c74640 1 -- 172.21.15.27:0/1846229694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb41a1b80 con 0x7f0cb41029f0 2024-04-25T13:44:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.094+0000 7f0cba475640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb4108930 0x7f0cb41a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.095+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb41029f0 0x7f0cb419a040 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0005d00 tx=0x7f0cb000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.095+0000 7f0caaffd640 1 -- 172.21.15.27:0/1846229694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0017070 con 0x7f0cb41029f0 2024-04-25T13:44:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.095+0000 7f0caaffd640 1 -- 172.21.15.27:0/1846229694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cb0009c30 con 0x7f0cb41029f0 2024-04-25T13:44:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.095+0000 7f0caaffd640 1 -- 172.21.15.27:0/1846229694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb000f620 con 0x7f0cb41029f0 2024-04-25T13:44:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.095+0000 7f0cbbed8640 1 -- 172.21.15.27:0/1846229694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb41a1e70 con 0x7f0cb41029f0 2024-04-25T13:44:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.095+0000 7f0cbbed8640 1 -- 172.21.15.27:0/1846229694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb41a2280 con 0x7f0cb41029f0 2024-04-25T13:44:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.099+0000 7f0cc2d83740 1 Processor -- start 2024-04-25T13:44:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.099+0000 7f0cc2d83740 1 -- start start 2024-04-25T13:44:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.099+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae241f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.099+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 0x557ae2421cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.099+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20516b0 0x557ae2424440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.099+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557ae2378d80 con 0x557ae20494d0 2024-04-25T13:44:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557ae2332a20 con 0x557ae20516b0 2024-04-25T13:44:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557ae2374d40 con 0x557ae2072390 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cb9473640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20516b0 0x557ae2424440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cba475640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae241f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cb9473640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20516b0 0x557ae2424440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43480/0 (socket says 172.21.15.27:43480) 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cb9473640 1 -- 172.21.15.27:0/1744761279 learned_addr learned my addr 172.21.15.27:0/1744761279 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cb9473640 1 -- 172.21.15.27:0/1744761279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 msgr2=0x557ae241f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cb9473640 1 --2- 172.21.15.27:0/1744761279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae241f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cb9473640 1 -- 172.21.15.27:0/1744761279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 msgr2=0x557ae2421cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cb9473640 1 --2- 172.21.15.27:0/1744761279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 0x557ae2421cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0cb9473640 1 -- 172.21.15.27:0/1744761279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ae233dc30 con 0x557ae20516b0 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.100+0000 7f0caaffd640 1 -- 172.21.15.27:0/1846229694 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0cb001a070 con 0x7f0cb41029f0 2024-04-25T13:44:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.101+0000 7f0cb9473640 1 --2- 172.21.15.27:0/1744761279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20516b0 0x557ae2424440 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009900 tx=0x7f0ca4031b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480e22c10fe01f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.101+0000 7f0caaffd640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ca00afb70 0x7f0ca00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.101+0000 7f0cba475640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ca00afb70 0x7f0ca00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.102+0000 7f0caaffd640 1 -- 172.21.15.27:0/1846229694 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0cb00de500 con 0x7f0cb41029f0 2024-04-25T13:44:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.102+0000 7f0c9ffff640 1 -- 172.21.15.27:0/1744761279 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4040070 con 0x557ae20516b0 2024-04-25T13:44:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.102+0000 7f0cba475640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ca00afb70 0x7f0ca00b2080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009fd0 tx=0x7f0cac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.102+0000 7f0c9ffff640 1 -- 172.21.15.27:0/1744761279 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca4031e40 con 0x557ae20516b0 2024-04-25T13:44:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.102+0000 7f0c9ffff640 1 -- 172.21.15.27:0/1744761279 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4035040 con 0x557ae20516b0 2024-04-25T13:44:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.103+0000 7f0cc2d83740 1 -- 172.21.15.27:0/1744761279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20516b0 msgr2=0x557ae2424440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.103+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/1744761279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20516b0 0x557ae2424440 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009900 tx=0x7f0ca4031b90 comp rx=0 tx=0).stop 2024-04-25T13:44:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.103+0000 7f0cc2d83740 1 -- 172.21.15.27:0/1744761279 shutdown_connections 2024-04-25T13:44:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.103+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/1744761279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20516b0 0x557ae2424440 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.103+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/1744761279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 0x557ae2421cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.103+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/1744761279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae241f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.103+0000 7f0cc2d83740 1 -- 172.21.15.27:0/1744761279 >> 172.21.15.27:0/1744761279 conn(0x557ae20664e0 msgr2=0x557ae240ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.104+0000 7f0cc2d83740 1 -- 172.21.15.27:0/1744761279 shutdown_connections 2024-04-25T13:44:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.104+0000 7f0cc2d83740 1 -- 172.21.15.27:0/1744761279 wait complete. 2024-04-25T13:44:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.104+0000 7f0cc2d83740 1 -- start start 2024-04-25T13:44:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.104+0000 7f0cc2d83740 1 Processor -- start 2024-04-25T13:44:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.104+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 0x557ae2426d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20516b0 0x557ae24272d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae242e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cb9c74640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 0x557ae2426d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cb9c74640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 0x557ae2426d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44962/0 (socket says 172.21.15.27:44962) 2024-04-25T13:44:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557ae2424f00 con 0x557ae20494d0 2024-04-25T13:44:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557ae22ffdc0 con 0x557ae2072390 2024-04-25T13:44:42.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557ae2425080 con 0x557ae20516b0 2024-04-25T13:44:42.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cb9c74640 1 -- 172.21.15.27:0/130040270 learned_addr learned my addr 172.21.15.27:0/130040270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:42.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.105+0000 7f0cba475640 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae242e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.106+0000 7f0cc2d83740 10 client.0 _ll_register_callbacks cb 0x557ae231b600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:42.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.106+0000 7f0cba475640 1 -- 172.21.15.27:0/130040270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20516b0 msgr2=0x557ae24272d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:44:42.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.106+0000 7f0cba475640 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20516b0 0x557ae24272d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.106+0000 7f0cba475640 1 -- 172.21.15.27:0/130040270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 msgr2=0x557ae2426d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.106+0000 7f0cba475640 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 0x557ae2426d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.106+0000 7f0cba475640 1 -- 172.21.15.27:0/130040270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ae208a3f0 con 0x557ae2072390 2024-04-25T13:44:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.106+0000 7f0cba475640 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae242e230 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0cac005fd0 tx=0x7f0cac004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb71f71200637542 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.107+0000 7f0c9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.107+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/130040270 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00fe60 con 0x557ae2072390 2024-04-25T13:44:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.107+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/130040270 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0cac003ca0 con 0x557ae2072390 2024-04-25T13:44:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.107+0000 7f0cc2d83740 1 -- 172.21.15.27:0/130040270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557ae2386990 con 0x557ae2072390 2024-04-25T13:44:42.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.107+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/130040270 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0cac003ea0 con 0x557ae2072390 2024-04-25T13:44:42.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.107+0000 7f0cc2d83740 10 client.25787 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:42.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0cc2d83740 1 -- 172.21.15.27:0/130040270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557ae21547d0 con 0x557ae2072390 2024-04-25T13:44:42.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/130040270 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x557ae2386990 con 0x557ae2072390 2024-04-25T13:44:42.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/130040270 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f0cac039430 con 0x557ae2072390 2024-04-25T13:44:42.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0cc2d83740 10 client.25787 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:42.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0cc2d83740 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:42.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0cc2d83740 20 client.25787 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:42.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ae2030f60 0x557ae2418bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0cc2d83740 4 client.25787 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:42.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.108+0000 7f0cc2d83740 1 -- 172.21.15.27:0/130040270 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ae2380800 con 0x557ae2030f60 2024-04-25T13:44:42.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.109+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ae2030f60 0x557ae2418bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.109+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ae2030f60 0x557ae2418bb0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d62e6eb7ac179560 server_cookie=a2ab451b29e17eda in_seq=0 out_seq=0 2024-04-25T13:44:42.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.109+0000 7f0c9d7fa640 10 client.25787 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:42.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/130040270 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557ae2380800 con 0x557ae2030f60 2024-04-25T13:44:42.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0c9d7fa640 10 client.25787 handle_command_reply: tid=0 2024-04-25T13:44:42.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 client.25787 shutdown 2024-04-25T13:44:42.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 -- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 msgr2=0x557ae242e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae242e230 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0cac005fd0 tx=0x7f0cac004ab0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 -- 172.21.15.27:0/130040270 shutdown_connections 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ae2030f60 0x557ae2418bb0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae242e230 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20516b0 0x557ae24272d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/130040270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20494d0 0x557ae2426d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0c9d7fa640 0 client.25787 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 -- 172.21.15.27:0/130040270 >> 172.21.15.27:0/130040270 conn(0x557ae20664e0 msgr2=0x557ae240e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 -- 172.21.15.27:0/130040270 shutdown_connections 2024-04-25T13:44:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 1 -- 172.21.15.27:0/130040270 wait complete. 2024-04-25T13:44:42.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.110+0000 7f0cc2d83740 20 client.25787 trim_cache size 0 max 16384 2024-04-25T13:44:42.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.139+0000 7f0cc2d83740 1 Processor -- start 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.143+0000 7f0cc2d83740 1 -- start start 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.143+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20494d0 0x557ae2425b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.143+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae2420c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.143+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae2422640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.143+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb41a6e40 con 0x557ae20516b0 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.143+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb41a96c0 con 0x557ae2072390 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.143+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb41a74b0 con 0x557ae20494d0 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.144+0000 7f0cb9c74640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20494d0 0x557ae2425b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.144+0000 7f0cb9473640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae2420c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.144+0000 7f0cb9473640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae2420c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44978/0 (socket says 172.21.15.27:44978) 2024-04-25T13:44:42.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.144+0000 7f0cb9473640 1 -- 172.21.15.27:0/437290305 learned_addr learned my addr 172.21.15.27:0/437290305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:42.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.147+0000 7f0cba475640 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae2422640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.149+0000 7f0cba475640 1 -- 172.21.15.27:0/437290305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20494d0 msgr2=0x557ae2425b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:42.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.149+0000 7f0cba475640 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20494d0 0x557ae2425b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.149+0000 7f0cba475640 1 -- 172.21.15.27:0/437290305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 msgr2=0x557ae2420c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:42.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.149+0000 7f0cba475640 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae2420c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.149+0000 7f0cba475640 1 -- 172.21.15.27:0/437290305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ae241f740 con 0x557ae2072390 2024-04-25T13:44:42.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.149+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20494d0 0x557ae2425b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:42.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.149+0000 7f0cb9473640 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae2420c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:42.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.150+0000 7f0cba475640 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae2422640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0398d0 tx=0x7f0cac012400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147afc5a5aaf9852 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:42.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.152+0000 7f0c9f7fe640 1 -- 172.21.15.27:0/437290305 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac003ea0 con 0x557ae2072390 2024-04-25T13:44:42.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.152+0000 7f0c9f7fe640 1 -- 172.21.15.27:0/437290305 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cac039430 con 0x557ae2072390 2024-04-25T13:44:42.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.152+0000 7f0c9f7fe640 1 -- 172.21.15.27:0/437290305 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac011910 con 0x557ae2072390 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 -- 172.21.15.27:0/437290305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 msgr2=0x557ae2422640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae2422640 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0398d0 tx=0x7f0cac012400 comp rx=0 tx=0).stop 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 -- 172.21.15.27:0/437290305 shutdown_connections 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae2072390 0x557ae2422640 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae2420c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/437290305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae20494d0 0x557ae2425b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 -- 172.21.15.27:0/437290305 >> 172.21.15.27:0/437290305 conn(0x557ae20664e0 msgr2=0x557ae2425480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 -- 172.21.15.27:0/437290305 shutdown_connections 2024-04-25T13:44:42.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 -- 172.21.15.27:0/437290305 wait complete. 2024-04-25T13:44:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 -- start start 2024-04-25T13:44:42.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.156+0000 7f0cc2d83740 1 Processor -- start 2024-04-25T13:44:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20494d0 0x557ae242b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae242b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cc2d83740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae242c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb41a7b80 con 0x557ae20516b0 2024-04-25T13:44:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb41a8ff0 con 0x557ae20494d0 2024-04-25T13:44:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cc2d83740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb41a8250 con 0x557ae2072390 2024-04-25T13:44:42.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cc2d83740 10 client.0 _ll_register_callbacks cb 0x7f0cac0126f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:42.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cba475640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae242c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cba475640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae242c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37370/0 (socket says 172.21.15.27:37370) 2024-04-25T13:44:42.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.157+0000 7f0cba475640 1 -- 172.21.15.27:0/3954485 learned_addr learned my addr 172.21.15.27:0/3954485 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:42.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.158+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20494d0 0x557ae242b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.158+0000 7f0cb9473640 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae242b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0cb9c74640 1 -- 172.21.15.27:0/3954485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 msgr2=0x557ae242c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:42.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae242c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0cb9c74640 1 -- 172.21.15.27:0/3954485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 msgr2=0x557ae242b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae242b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0cb9c74640 1 -- 172.21.15.27:0/3954485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ae242cd40 con 0x557ae20494d0 2024-04-25T13:44:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0cb9473640 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae242b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:42.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0cb9c74640 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20494d0 0x557ae242b310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00a6a50 tx=0x7f0cb0010c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=160cc1a6e6c9094 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0c9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/3954485 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0021040 con 0x557ae20494d0 2024-04-25T13:44:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/3954485 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0cb00a30b0 con 0x557ae20494d0 2024-04-25T13:44:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.159+0000 7f0cc2d83740 1 -- 172.21.15.27:0/3954485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cac039430 con 0x557ae20494d0 2024-04-25T13:44:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.160+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/3954485 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0cb00a6ac0 con 0x557ae20494d0 2024-04-25T13:44:42.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.160+0000 7f0cc2d83740 10 client.25793 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:42.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.160+0000 7f0cc2d83740 1 -- 172.21.15.27:0/3954485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557ae2493b50 con 0x557ae20494d0 2024-04-25T13:44:42.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.161+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/3954485 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cb00aadc0 con 0x557ae20494d0 2024-04-25T13:44:42.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.161+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/3954485 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f0cb00c90b0 con 0x557ae20494d0 2024-04-25T13:44:42.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.161+0000 7f0cc2d83740 10 client.25793 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:42.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.161+0000 7f0cc2d83740 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:42.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.161+0000 7f0cc2d83740 20 client.25793 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:42.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.161+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ae20063a0 0x557ae2496470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:42.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.161+0000 7f0cc2d83740 4 client.25793 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:44:42.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.161+0000 7f0cc2d83740 1 -- 172.21.15.27:0/3954485 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0cac039430 con 0x557ae20063a0 2024-04-25T13:44:42.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.162+0000 7f0cb9473640 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ae20063a0 0x557ae2496470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:42.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.162+0000 7f0c9d7fa640 10 client.25793 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:42.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.163+0000 7f0cb9473640 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ae20063a0 0x557ae2496470 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26204ad40ba0c7b0 server_cookie=d6fb6e36ce176de8 in_seq=0 out_seq=0 2024-04-25T13:44:42.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0c9d7fa640 1 -- 172.21.15.27:0/3954485 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0cac039430 con 0x557ae20063a0 2024-04-25T13:44:42.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0c9d7fa640 10 client.25793 handle_command_reply: tid=0 2024-04-25T13:44:42.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 client.25793 shutdown 2024-04-25T13:44:42.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 -- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20494d0 msgr2=0x557ae242b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20494d0 0x557ae242b310 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00a6a50 tx=0x7f0cb0010c00 comp rx=0 tx=0).stop 2024-04-25T13:44:42.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 -- 172.21.15.27:0/3954485 shutdown_connections 2024-04-25T13:44:42.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ae20063a0 0x557ae2496470 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ae2072390 0x557ae242c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ae20516b0 0x557ae242b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 --2- 172.21.15.27:0/3954485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ae20494d0 0x557ae242b310 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 -- 172.21.15.27:0/3954485 >> 172.21.15.27:0/3954485 conn(0x557ae20664e0 msgr2=0x557ae241fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:42.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0c9d7fa640 0 client.25793 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:42.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 -- 172.21.15.27:0/3954485 shutdown_connections 2024-04-25T13:44:42.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 1 -- 172.21.15.27:0/3954485 wait complete. 2024-04-25T13:44:42.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.164+0000 7f0cc2d83740 20 client.25793 trim_cache size 0 max 16384 2024-04-25T13:44:42.171 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 -- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ca00afb70 msgr2=0x7f0ca00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ca00afb70 0x7f0ca00b2080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009fd0 tx=0x7f0cac009500 comp rx=0 tx=0).stop 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 -- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb41029f0 msgr2=0x7f0cb419a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb41029f0 0x7f0cb419a040 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0005d00 tx=0x7f0cb000dde0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 -- 172.21.15.27:0/1846229694 shutdown_connections 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0ca00afb70 0x7f0ca00b2080 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb4108930 0x7f0cb41a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0cb4103c30 0x7f0cb419a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 --2- 172.21.15.27:0/1846229694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb41029f0 0x7f0cb419a040 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 -- 172.21.15.27:0/1846229694 >> 172.21.15.27:0/1846229694 conn(0x7f0cb40fdc00 msgr2=0x7f0cb40ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 -- 172.21.15.27:0/1846229694 shutdown_connections 2024-04-25T13:44:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:42.165+0000 7f0cbbed8640 1 -- 172.21.15.27:0/1846229694 wait complete. 2024-04-25T13:44:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 215.0 s, 3963.8 tps, lat 25.243 ms stddev 28.083 2024-04-25T13:44:43.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:42 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/4002495288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:44:43.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:42 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/4002495288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:44:43.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:42 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/4002495288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:44:43.183 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:44:43.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:44:43.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:44:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 -- 172.21.15.27:0/3656005903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 msgr2=0x7f082c10b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 --2- 172.21.15.27:0/3656005903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 0x7f082c10b720 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f081c009860 tx=0x7f081c02a340 comp rx=0 tx=0).stop 2024-04-25T13:44:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 -- 172.21.15.27:0/3656005903 shutdown_connections 2024-04-25T13:44:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 --2- 172.21.15.27:0/3656005903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c110030 0x7f082c110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 --2- 172.21.15.27:0/3656005903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 0x7f082c10b720 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 --2- 172.21.15.27:0/3656005903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c10a420 0x7f082c10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 -- 172.21.15.27:0/3656005903 >> 172.21.15.27:0/3656005903 conn(0x7f082c0704e0 msgr2=0x7f082c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 -- 172.21.15.27:0/3656005903 shutdown_connections 2024-04-25T13:44:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.350+0000 7f08344aa640 1 -- 172.21.15.27:0/3656005903 wait complete. 2024-04-25T13:44:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.352+0000 7f08344aa640 1 Processor -- start 2024-04-25T13:44:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.352+0000 7f08344aa640 1 -- start start 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f08344aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c10a420 0x7f082c1a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f08344aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 0x7f082c1a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f08344aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c110030 0x7f082c1b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f08344aa640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f082c11e490 con 0x7f082c110030 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f08344aa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f082c11e310 con 0x7f082c10b310 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f08344aa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f082c11e610 con 0x7f082c10a420 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0831a45640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 0x7f082c1a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0832a47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c110030 0x7f082c1b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0831a45640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 0x7f082c1a95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43542/0 (socket says 172.21.15.27:43542) 2024-04-25T13:44:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0831a45640 1 -- 172.21.15.27:0/2182793498 learned_addr learned my addr 172.21.15.27:0/2182793498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0832246640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c10a420 0x7f082c1a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0831a45640 1 -- 172.21.15.27:0/2182793498 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c10a420 msgr2=0x7f082c1a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0831a45640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c10a420 0x7f082c1a9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0831a45640 1 -- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c110030 msgr2=0x7f082c1b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0831a45640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c110030 0x7f082c1b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.353+0000 7f0831a45640 1 -- 172.21.15.27:0/2182793498 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082c1b0bb0 con 0x7f082c10b310 2024-04-25T13:44:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.354+0000 7f0831a45640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 0x7f082c1a95b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f081c009980 tx=0x7f081c032dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.354+0000 7f08237fe640 1 -- 172.21.15.27:0/2182793498 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c040020 con 0x7f082c10b310 2024-04-25T13:44:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.354+0000 7f08237fe640 1 -- 172.21.15.27:0/2182793498 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f081c030dd0 con 0x7f082c10b310 2024-04-25T13:44:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.354+0000 7f08237fe640 1 -- 172.21.15.27:0/2182793498 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c037040 con 0x7f082c10b310 2024-04-25T13:44:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.354+0000 7f08344aa640 1 -- 172.21.15.27:0/2182793498 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082c1b0e40 con 0x7f082c10b310 2024-04-25T13:44:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.354+0000 7f08344aa640 1 -- 172.21.15.27:0/2182793498 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082c1b11f0 con 0x7f082c10b310 2024-04-25T13:44:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f083b435740 1 Processor -- start 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f083b435740 1 -- start start 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f08237fe640 1 -- 172.21.15.27:0/2182793498 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f081c043020 con 0x7f082c10b310 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f083b435740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x5630434144a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f083b435740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043414a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f083b435740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304341b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f083b435740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563043379b00 con 0x563043052730 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f083b435740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563043315920 con 0x56304304a550 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f083b435740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563043375ac0 con 0x563043073410 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f0832246640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043414a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f0832246640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043414a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43558/0 (socket says 172.21.15.27:43558) 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.358+0000 7f0832246640 1 -- 172.21.15.27:0/3887367578 learned_addr learned my addr 172.21.15.27:0/3887367578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.359+0000 7f0832a47640 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x5630434144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.359+0000 7f0831a45640 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304341b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.359+0000 7f0832a47640 1 -- 172.21.15.27:0/3887367578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 msgr2=0x563043414a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.359+0000 7f0832a47640 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043414a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.359+0000 7f0832a47640 1 -- 172.21.15.27:0/3887367578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 msgr2=0x56304341b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.359+0000 7f0832a47640 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304341b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.359+0000 7f0832a47640 1 -- 172.21.15.27:0/3887367578 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630431537e0 con 0x563043073410 2024-04-25T13:44:43.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.359+0000 7f0832a47640 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x5630434144a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f082400e450 tx=0x7f08240095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d282f956df770092 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.360+0000 7f0820bfe640 1 -- 172.21.15.27:0/3887367578 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824004b20 con 0x563043073410 2024-04-25T13:44:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.360+0000 7f0820bfe640 1 -- 172.21.15.27:0/3887367578 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0824004cc0 con 0x563043073410 2024-04-25T13:44:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.360+0000 7f0820bfe640 1 -- 172.21.15.27:0/3887367578 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08240055e0 con 0x563043073410 2024-04-25T13:44:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f08237fe640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f08180afcc0 0x7f08180b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f083b435740 1 -- 172.21.15.27:0/3887367578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 msgr2=0x5630434144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f083b435740 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x5630434144a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f082400e450 tx=0x7f08240095c0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f0832246640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f08180afcc0 0x7f08180b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f08237fe640 1 -- 172.21.15.27:0/2182793498 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f081c0fec40 con 0x7f082c10b310 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f083b435740 1 -- 172.21.15.27:0/3887367578 shutdown_connections 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f083b435740 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304341b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f083b435740 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043414a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f083b435740 1 --2- 172.21.15.27:0/3887367578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x5630434144a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.361+0000 7f083b435740 1 -- 172.21.15.27:0/3887367578 >> 172.21.15.27:0/3887367578 conn(0x563043067560 msgr2=0x56304340fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.362+0000 7f083b435740 1 -- 172.21.15.27:0/3887367578 shutdown_connections 2024-04-25T13:44:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.363+0000 7f083b435740 1 -- 172.21.15.27:0/3887367578 wait complete. 2024-04-25T13:44:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.363+0000 7f0832246640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f08180afcc0 0x7f08180b21d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x563043410bc0 tx=0x7f0828005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.363+0000 7f083b435740 1 -- start start 2024-04-25T13:44:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.363+0000 7f083b435740 1 Processor -- start 2024-04-25T13:44:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f083b435740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56304304a550 0x56304342f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f083b435740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304342f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f083b435740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563043073410 0x563043436850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f083b435740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56304342d380 con 0x563043052730 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f083b435740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5630433135d0 con 0x563043073410 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f083b435740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56304342d500 con 0x56304304a550 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f083b435740 10 client.0 _ll_register_callbacks cb 0x563043301080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304342f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304342f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45026/0 (socket says 172.21.15.27:45026) 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 -- 172.21.15.27:0/4195232615 learned_addr learned my addr 172.21.15.27:0/4195232615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0832246640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56304304a550 0x56304342f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0832a47640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563043073410 0x563043436850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 -- 172.21.15.27:0/4195232615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56304304a550 msgr2=0x56304342f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56304304a550 0x56304342f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 -- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563043073410 msgr2=0x563043436850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563043073410 0x563043436850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 -- 172.21.15.27:0/4195232615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630431b3f70 con 0x563043052730 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0832246640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56304304a550 0x56304342f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0832a47640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563043073410 0x563043436850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0831a45640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304342f8f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f081c035040 tx=0x7f081c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c7057a2d5890f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.364+0000 7f0815ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.365+0000 7f0815ffb640 1 -- 172.21.15.27:0/4195232615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c0c6a40 con 0x563043052730 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.365+0000 7f0815ffb640 1 -- 172.21.15.27:0/4195232615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f081c0cb150 con 0x563043052730 2024-04-25T13:44:43.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.365+0000 7f083b435740 1 -- 172.21.15.27:0/4195232615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0824004cc0 con 0x563043052730 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.365+0000 7f0815ffb640 1 -- 172.21.15.27:0/4195232615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f081c0c6860 con 0x563043052730 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.365+0000 7f083b435740 10 client.16098 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.365+0000 7f083b435740 1 -- 172.21.15.27:0/4195232615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56304304b170 con 0x563043052730 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.366+0000 7f0815ffb640 1 -- 172.21.15.27:0/4195232615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0824004cc0 con 0x563043052730 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.366+0000 7f0815ffb640 1 -- 172.21.15.27:0/4195232615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f081c0fcbd0 con 0x563043052730 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.366+0000 7f083b435740 10 client.16098 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.366+0000 7f083b435740 10 client.16098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.366+0000 7f083b435740 20 client.16098 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.366+0000 7f083b435740 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630430321f0 0x56304343cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.366+0000 7f083b435740 4 client.16098 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:43.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.366+0000 7f083b435740 1 -- 172.21.15.27:0/4195232615 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563043387710 con 0x5630430321f0 2024-04-25T13:44:43.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.367+0000 7f0832246640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630430321f0 0x56304343cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.367+0000 7f0815ffb640 10 client.16098 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:43.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.367+0000 7f0832246640 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630430321f0 0x56304343cd10 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa552fd4d41c4cd5 server_cookie=9885d14b90ca8a51 in_seq=0 out_seq=0 2024-04-25T13:44:43.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f0815ffb640 1 -- 172.21.15.27:0/4195232615 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563043387710 con 0x5630430321f0 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f0815ffb640 10 client.16098 handle_command_reply: tid=0 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f083b435740 1 client.16098 shutdown 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f083b435740 1 -- 172.21.15.27:0/4195232615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 msgr2=0x56304342f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f083b435740 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304342f8f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f081c035040 tx=0x7f081c005f90 comp rx=0 tx=0).stop 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f083b435740 1 -- 172.21.15.27:0/4195232615 shutdown_connections 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f083b435740 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630430321f0 0x56304343cd10 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f083b435740 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563043073410 0x563043436850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f083b435740 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043052730 0x56304342f8f0 secure :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f081c035040 tx=0x7f081c005f90 comp rx=0 tx=0).stop 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.369+0000 7f0815ffb640 0 client.16098 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.368+0000 7f083b435740 1 --2- 172.21.15.27:0/4195232615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56304304a550 0x56304342f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.369+0000 7f083b435740 1 -- 172.21.15.27:0/4195232615 >> 172.21.15.27:0/4195232615 conn(0x563043067560 msgr2=0x56304340eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.369+0000 7f083b435740 1 -- 172.21.15.27:0/4195232615 shutdown_connections 2024-04-25T13:44:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.369+0000 7f083b435740 1 -- 172.21.15.27:0/4195232615 wait complete. 2024-04-25T13:44:43.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.369+0000 7f083b435740 20 client.16098 trim_cache size 0 max 16384 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.395+0000 7f083b435740 1 Processor -- start 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.395+0000 7f083b435740 1 -- start start 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.395+0000 7f083b435740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043418e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.395+0000 7f083b435740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043073410 0x56304340ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f083b435740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043007630 0x563043436810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f083b435740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f082c118630 con 0x563043073410 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f083b435740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f082c1b1f70 con 0x56304304a550 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f083b435740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f082c118ca0 con 0x563043007630 2024-04-25T13:44:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0831a45640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043073410 0x56304340ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0832246640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043418e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0831a45640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043073410 0x56304340ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45040/0 (socket says 172.21.15.27:45040) 2024-04-25T13:44:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0831a45640 1 -- 172.21.15.27:0/920257398 learned_addr learned my addr 172.21.15.27:0/920257398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0832a47640 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043007630 0x563043436810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0832a47640 1 -- 172.21.15.27:0/920257398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 msgr2=0x563043418e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0832a47640 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043418e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0832a47640 1 -- 172.21.15.27:0/920257398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043073410 msgr2=0x56304340ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0832a47640 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043073410 0x56304340ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.396+0000 7f0832a47640 1 -- 172.21.15.27:0/920257398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630434156a0 con 0x563043007630 2024-04-25T13:44:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.397+0000 7f0832246640 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043418e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.397+0000 7f0832a47640 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043007630 0x563043436810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0824005130 tx=0x7f0824004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c75fb8919d47e48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.397+0000 7f0817fff640 1 -- 172.21.15.27:0/920257398 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824004ba0 con 0x563043007630 2024-04-25T13:44:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.397+0000 7f0817fff640 1 -- 172.21.15.27:0/920257398 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0824003e20 con 0x563043007630 2024-04-25T13:44:43.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.397+0000 7f0817fff640 1 -- 172.21.15.27:0/920257398 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082400dcd0 con 0x563043007630 2024-04-25T13:44:43.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 -- 172.21.15.27:0/920257398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043007630 msgr2=0x563043436810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043007630 0x563043436810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0824005130 tx=0x7f0824004ab0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 -- 172.21.15.27:0/920257398 shutdown_connections 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043007630 0x563043436810 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043073410 0x56304340ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 --2- 172.21.15.27:0/920257398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043418e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 -- 172.21.15.27:0/920257398 >> 172.21.15.27:0/920257398 conn(0x563043067560 msgr2=0x56304340e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 -- 172.21.15.27:0/920257398 shutdown_connections 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.398+0000 7f083b435740 1 -- 172.21.15.27:0/920257398 wait complete. 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 1 -- start start 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 1 Processor -- start 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043007630 0x563043434af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043435030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x563043435f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f082c119370 con 0x563043007630 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f082c1b18a0 con 0x56304304a550 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f082c119a40 con 0x563043073410 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f083b435740 10 client.0 _ll_register_callbacks cb 0x7f081c102130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f0832246640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043007630 0x563043434af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f0832246640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043007630 0x563043434af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45048/0 (socket says 172.21.15.27:45048) 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f0832246640 1 -- 172.21.15.27:0/3068573040 learned_addr learned my addr 172.21.15.27:0/3068573040 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f0831a45640 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043435030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.399+0000 7f0832a47640 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x563043435f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0832246640 1 -- 172.21.15.27:0/3068573040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 msgr2=0x563043435f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0832246640 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x563043435f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0832246640 1 -- 172.21.15.27:0/3068573040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 msgr2=0x563043435030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0832246640 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043435030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0832246640 1 -- 172.21.15.27:0/3068573040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563043436520 con 0x563043007630 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0832246640 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043007630 0x563043434af0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0828004450 tx=0x7f08280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8244993cc26cadb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0815ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0815ffb640 1 -- 172.21.15.27:0/3068573040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082802b560 con 0x563043007630 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0815ffb640 1 -- 172.21.15.27:0/3068573040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f082802b700 con 0x563043007630 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f083b435740 1 -- 172.21.15.27:0/3068573040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0824003e20 con 0x563043007630 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0815ffb640 1 -- 172.21.15.27:0/3068573040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f082803a070 con 0x563043007630 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f083b435740 10 client.16104 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f083b435740 1 -- 172.21.15.27:0/3068573040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5630434a5410 con 0x563043007630 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.400+0000 7f0815ffb640 1 -- 172.21.15.27:0/3068573040 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f082802b900 con 0x563043007630 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f0815ffb640 1 -- 172.21.15.27:0/3068573040 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f082801f490 con 0x563043007630 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f083b435740 10 client.16104 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f083b435740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f083b435740 20 client.16104 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:43.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f083b435740 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630434a5da0 0x5630434a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f083b435740 4 client.16104 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:44:43.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f083b435740 1 -- 172.21.15.27:0/3068573040 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f0824003e20 con 0x5630434a5da0 2024-04-25T13:44:43.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f0831a45640 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630434a5da0 0x5630434a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f0815ffb640 10 client.16104 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:43.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.401+0000 7f0831a45640 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630434a5da0 0x5630434a81c0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d7fc6bc49a2fc6f server_cookie=a4c7ba35d607ea2d in_seq=0 out_seq=0 2024-04-25T13:44:43.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f0815ffb640 1 -- 172.21.15.27:0/3068573040 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f0824003e20 con 0x5630434a5da0 2024-04-25T13:44:43.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f0815ffb640 10 client.16104 handle_command_reply: tid=0 2024-04-25T13:44:43.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 client.16104 shutdown 2024-04-25T13:44:43.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 -- 172.21.15.27:0/3068573040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043007630 msgr2=0x563043434af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043007630 0x563043434af0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0828004450 tx=0x7f08280046c0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 -- 172.21.15.27:0/3068573040 shutdown_connections 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630434a5da0 0x5630434a81c0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563043073410 0x563043435f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56304304a550 0x563043435030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 --2- 172.21.15.27:0/3068573040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563043007630 0x563043434af0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f083b435740 1 -- 172.21.15.27:0/3068573040 >> 172.21.15.27:0/3068573040 conn(0x563043067560 msgr2=0x56304340e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.403+0000 7f0815ffb640 0 client.16104 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f083b435740 1 -- 172.21.15.27:0/3068573040 shutdown_connections 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f083b435740 1 -- 172.21.15.27:0/3068573040 wait complete. 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f083b435740 20 client.16104 trim_cache size 0 max 16384 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:44:43.407 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:44:43.408 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "29e8da23-a196-4204-ab60-c13a56fa4b7e", 2024-04-25T13:44:43.408 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:44:43.408 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:44:43.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 -- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f08180afcc0 msgr2=0x7f08180b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f08180afcc0 0x7f08180b21d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x563043410bc0 tx=0x7f0828005f90 comp rx=0 tx=0).stop 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 -- 172.21.15.27:0/2182793498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 msgr2=0x7f082c1a95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 0x7f082c1a95b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f081c009980 tx=0x7f081c032dc0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 -- 172.21.15.27:0/2182793498 shutdown_connections 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f08180afcc0 0x7f08180b21d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c110030 0x7f082c1b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f082c10b310 0x7f082c1a95b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 --2- 172.21.15.27:0/2182793498 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c10a420 0x7f082c1a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 -- 172.21.15.27:0/2182793498 >> 172.21.15.27:0/2182793498 conn(0x7f082c0704e0 msgr2=0x7f082c10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 -- 172.21.15.27:0/2182793498 shutdown_connections 2024-04-25T13:44:43.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.404+0000 7f08344aa640 1 -- 172.21.15.27:0/2182793498 wait complete. 2024-04-25T13:44:43.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:44:43.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:44:43.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.588+0000 7f2177b0d640 1 -- 172.21.15.27:0/504103202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2170108930 msgr2=0x7f217010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.588+0000 7f2177b0d640 1 --2- 172.21.15.27:0/504103202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2170108930 0x7f217010ad50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2168009ca0 tx=0x7f216802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.589+0000 7f2177b0d640 1 -- 172.21.15.27:0/504103202 shutdown_connections 2024-04-25T13:44:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.589+0000 7f2177b0d640 1 --2- 172.21.15.27:0/504103202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2170108930 0x7f217010ad50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.589+0000 7f2177b0d640 1 --2- 172.21.15.27:0/504103202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2170103c30 0x7f2170104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.589+0000 7f2177b0d640 1 --2- 172.21.15.27:0/504103202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21701029f0 0x7f21701036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.589+0000 7f2177b0d640 1 -- 172.21.15.27:0/504103202 >> 172.21.15.27:0/504103202 conn(0x7f21700fdc00 msgr2=0x7f2170100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.589+0000 7f2177b0d640 1 -- 172.21.15.27:0/504103202 shutdown_connections 2024-04-25T13:44:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.589+0000 7f2177b0d640 1 -- 172.21.15.27:0/504103202 wait complete. 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f2177b0d640 1 Processor -- start 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f2177b0d640 1 -- start start 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f2177b0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21701029f0 0x7f2170197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f2177b0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2170103c30 0x7f2170197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f2177b0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2170108930 0x7f2170193f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f2177b0d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f217010e520 con 0x7f21701029f0 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f2177b0d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f217010e3a0 con 0x7f2170108930 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f2177b0d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f217010e6a0 con 0x7f2170103c30 2024-04-25T13:44:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f21750a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2170103c30 0x7f2170197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f21750a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2170103c30 0x7f2170197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37458/0 (socket says 172.21.15.27:37458) 2024-04-25T13:44:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.590+0000 7f21750a8640 1 -- 172.21.15.27:0/3936608082 learned_addr learned my addr 172.21.15.27:0/3936608082 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.591+0000 7f21760aa640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2170108930 0x7f2170193f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.591+0000 7f21758a9640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21701029f0 0x7f2170197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.591+0000 7f21760aa640 1 -- 172.21.15.27:0/3936608082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2170103c30 msgr2=0x7f2170197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.591+0000 7f21760aa640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2170103c30 0x7f2170197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.591+0000 7f21760aa640 1 -- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21701029f0 msgr2=0x7f2170197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.591+0000 7f21760aa640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21701029f0 0x7f2170197430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.591+0000 7f21760aa640 1 -- 172.21.15.27:0/3936608082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21701947a0 con 0x7f2170108930 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.591+0000 7f21760aa640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2170108930 0x7f2170193f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2168033040 tx=0x7f2168031680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.592+0000 7f2166ffd640 1 -- 172.21.15.27:0/3936608082 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168003b70 con 0x7f2170108930 2024-04-25T13:44:43.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.592+0000 7f2177b0d640 1 -- 172.21.15.27:0/3936608082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2170194a30 con 0x7f2170108930 2024-04-25T13:44:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.592+0000 7f2166ffd640 1 -- 172.21.15.27:0/3936608082 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2168003d10 con 0x7f2170108930 2024-04-25T13:44:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.592+0000 7f2166ffd640 1 -- 172.21.15.27:0/3936608082 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216802f970 con 0x7f2170108930 2024-04-25T13:44:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.592+0000 7f2177b0d640 1 -- 172.21.15.27:0/3936608082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21701a8b10 con 0x7f2170108930 2024-04-25T13:44:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.596+0000 7f217e9c2740 1 Processor -- start 2024-04-25T13:44:43.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.597+0000 7f2166ffd640 1 -- 172.21.15.27:0/3936608082 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f216802fb10 con 0x7f2170108930 2024-04-25T13:44:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.597+0000 7f217e9c2740 1 -- start start 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f50799aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d04d0 0x55f50799afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 0x55f50799bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f217e9c2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f5078ffd80 con 0x55f5075d86b0 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f217e9c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f5078b98b0 con 0x55f5075d04d0 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f217e9c2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f5078fbd40 con 0x55f5075f9390 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21750a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f50799aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21750a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f50799aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37474/0 (socket says 172.21.15.27:37474) 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21758a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d04d0 0x55f50799afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21750a8640 1 -- 172.21.15.27:0/2040185745 learned_addr learned my addr 172.21.15.27:0/2040185745 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21750a8640 1 --2- 172.21.15.27:0/2040185745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 0x55f50799bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21758a9640 1 -- 172.21.15.27:0/2040185745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 msgr2=0x55f50799aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21758a9640 1 --2- 172.21.15.27:0/2040185745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f50799aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21758a9640 1 -- 172.21.15.27:0/2040185745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 msgr2=0x55f50799bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21758a9640 1 --2- 172.21.15.27:0/2040185745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 0x55f50799bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.598+0000 7f21758a9640 1 -- 172.21.15.27:0/2040185745 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f50788aa40 con 0x55f5075d04d0 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.599+0000 7f2166ffd640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f215c0afbb0 0x7f215c0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.599+0000 7f21758a9640 1 --2- 172.21.15.27:0/2040185745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d04d0 0x55f50799afe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f21701022c0 tx=0x7f216c0113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3b20b658d5007f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.599+0000 7f21750a8640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f215c0afbb0 0x7f215c0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.599+0000 7f215bfff640 1 -- 172.21.15.27:0/2040185745 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c011d60 con 0x55f5075d04d0 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.599+0000 7f2166ffd640 1 -- 172.21.15.27:0/3936608082 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f21680fff90 con 0x7f2170108930 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.599+0000 7f215bfff640 1 -- 172.21.15.27:0/2040185745 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f216c012900 con 0x55f5075d04d0 2024-04-25T13:44:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.599+0000 7f215bfff640 1 -- 172.21.15.27:0/2040185745 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c00f8b0 con 0x55f5075d04d0 2024-04-25T13:44:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f21750a8640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f215c0afbb0 0x7f215c0b20c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2160009fd0 tx=0x7f2160005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 -- 172.21.15.27:0/2040185745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d04d0 msgr2=0x55f50799afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 --2- 172.21.15.27:0/2040185745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d04d0 0x55f50799afe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f21701022c0 tx=0x7f216c0113e0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 -- 172.21.15.27:0/2040185745 shutdown_connections 2024-04-25T13:44:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 --2- 172.21.15.27:0/2040185745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 0x55f50799bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 --2- 172.21.15.27:0/2040185745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d04d0 0x55f50799afe0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 --2- 172.21.15.27:0/2040185745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f50799aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 -- 172.21.15.27:0/2040185745 >> 172.21.15.27:0/2040185745 conn(0x55f5075ed4e0 msgr2=0x55f507996270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 -- 172.21.15.27:0/2040185745 shutdown_connections 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 -- 172.21.15.27:0/2040185745 wait complete. 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.600+0000 7f217e9c2740 1 -- start start 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f217e9c2740 1 Processor -- start 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d86b0 0x55f5079a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f5079abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f21758a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f21758a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45092/0 (socket says 172.21.15.27:45092) 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f21758a9640 1 -- 172.21.15.27:0/3505521551 learned_addr learned my addr 172.21.15.27:0/3505521551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f50799c780 con 0x55f5075d04d0 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f507886d10 con 0x55f5075d86b0 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f50799c900 con 0x55f5075f9390 2024-04-25T13:44:43.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f217e9c2740 10 client.0 _ll_register_callbacks cb 0x55f5078b3210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f21760aa640 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f5079abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.601+0000 7f21750a8640 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d86b0 0x55f5079a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.602+0000 7f21760aa640 1 -- 172.21.15.27:0/3505521551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d86b0 msgr2=0x55f5079a4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.602+0000 7f21760aa640 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d86b0 0x55f5079a4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.602+0000 7f21760aa640 1 -- 172.21.15.27:0/3505521551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 msgr2=0x55f5079a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.602+0000 7f21760aa640 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.602+0000 7f21760aa640 1 -- 172.21.15.27:0/3505521551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5076db7d0 con 0x55f5075f9390 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.602+0000 7f21760aa640 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f5079abdb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f216802aeb0 tx=0x7f21680c7bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7192dab2fa5e8d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.602+0000 7f21758a9640 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.602+0000 7f21597fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f21597fa640 1 -- 172.21.15.27:0/3505521551 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21680c8b40 con 0x55f5075f9390 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f216c012900 con 0x55f5075f9390 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f21597fa640 1 -- 172.21.15.27:0/3505521551 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f21680c46d0 con 0x55f5075f9390 2024-04-25T13:44:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f21597fa640 1 -- 172.21.15.27:0/3505521551 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f21680c8440 con 0x55f5075f9390 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f217e9c2740 10 client.25843 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f5076113f0 con 0x55f5075f9390 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f21597fa640 1 -- 172.21.15.27:0/3505521551 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f216c012900 con 0x55f5075f9390 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f21597fa640 1 -- 172.21.15.27:0/3505521551 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f21680d0e10 con 0x55f5075f9390 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.603+0000 7f217e9c2740 10 client.25843 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.604+0000 7f217e9c2740 10 client.25843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.604+0000 7f217e9c2740 20 client.25843 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.604+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5075b7f60 0x55f5079b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.604+0000 7f217e9c2740 4 client.25843 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.604+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f50790d990 con 0x55f5075b7f60 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.604+0000 7f21758a9640 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5075b7f60 0x55f5079b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.604+0000 7f21597fa640 10 client.25843 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:43.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.604+0000 7f21758a9640 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5075b7f60 0x55f5079b22e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b337cbdfdb7753c6 server_cookie=b657474edebad210 in_seq=0 out_seq=0 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f21597fa640 1 -- 172.21.15.27:0/3505521551 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f50790d990 con 0x55f5075b7f60 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f21597fa640 10 client.25843 handle_command_reply: tid=0 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 client.25843 shutdown 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 msgr2=0x55f5079abdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f5079abdb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f216802aeb0 tx=0x7f21680c7bb0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 shutdown_connections 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5075b7f60 0x55f5079b22e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075f9390 0x55f5079abdb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075d86b0 0x55f5079a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f21597fa640 0 client.25843 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3505521551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 >> 172.21.15.27:0/3505521551 conn(0x55f5075ed4e0 msgr2=0x55f507999180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 shutdown_connections 2024-04-25T13:44:43.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.605+0000 7f217e9c2740 1 -- 172.21.15.27:0/3505521551 wait complete. 2024-04-25T13:44:43.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.606+0000 7f217e9c2740 20 client.25843 trim_cache size 0 max 16384 2024-04-25T13:44:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f217e9c2740 1 Processor -- start 2024-04-25T13:44:43.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f217e9c2740 1 -- start start 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d04d0 0x55f507996320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 0x55f5079a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f217e9c2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f21701af680 con 0x55f5075d86b0 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f217e9c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f21701b1f00 con 0x55f5075f9390 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f217e9c2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f21701afcf0 con 0x55f5075d04d0 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f21750a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 0x55f5079a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f21758a9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d04d0 0x55f507996320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f21760aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f21758a9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d04d0 0x55f507996320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37484/0 (socket says 172.21.15.27:37484) 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.630+0000 7f21758a9640 1 -- 172.21.15.27:0/3905184750 learned_addr learned my addr 172.21.15.27:0/3905184750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f21758a9640 1 -- 172.21.15.27:0/3905184750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 msgr2=0x55f5079a34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f21758a9640 1 --2- 172.21.15.27:0/3905184750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079a34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f21758a9640 1 -- 172.21.15.27:0/3905184750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 msgr2=0x55f5079a2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f21758a9640 1 --2- 172.21.15.27:0/3905184750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 0x55f5079a2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f21758a9640 1 -- 172.21.15.27:0/3905184750 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5075d10f0 con 0x55f5075d04d0 2024-04-25T13:44:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f21760aa640 1 --2- 172.21.15.27:0/3905184750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079a34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f21758a9640 1 --2- 172.21.15.27:0/3905184750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d04d0 0x55f507996320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f216c009fd0 tx=0x7f216c011f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5cd3fa574e8cd9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f215b7fe640 1 -- 172.21.15.27:0/3905184750 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c011ca0 con 0x55f5075d04d0 2024-04-25T13:44:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f215b7fe640 1 -- 172.21.15.27:0/3905184750 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f216c030070 con 0x55f5075d04d0 2024-04-25T13:44:43.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.631+0000 7f215b7fe640 1 -- 172.21.15.27:0/3905184750 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c003bb0 con 0x55f5075d04d0 2024-04-25T13:44:43.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.632+0000 7f217e9c2740 1 -- 172.21.15.27:0/3905184750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d04d0 msgr2=0x55f507996320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.633+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3905184750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d04d0 0x55f507996320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f216c009fd0 tx=0x7f216c011f20 comp rx=0 tx=0).stop 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.633+0000 7f217e9c2740 1 -- 172.21.15.27:0/3905184750 shutdown_connections 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.633+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3905184750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079a34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.633+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3905184750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d86b0 0x55f5079a2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.633+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3905184750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d04d0 0x55f507996320 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.633+0000 7f217e9c2740 1 -- 172.21.15.27:0/3905184750 >> 172.21.15.27:0/3905184750 conn(0x55f5075ed4e0 msgr2=0x55f507995c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.633+0000 7f217e9c2740 1 -- 172.21.15.27:0/3905184750 shutdown_connections 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.633+0000 7f217e9c2740 1 -- 172.21.15.27:0/3905184750 wait complete. 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 1 -- start start 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 1 Processor -- start 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d86b0 0x55f5079a9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f21701b03c0 con 0x55f5075d04d0 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f21701b1830 con 0x55f5075f9390 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f21701b0a90 con 0x55f5075d86b0 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f217e9c2740 10 client.0 _ll_register_callbacks cb 0x7f2168003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f21758a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f21758a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45098/0 (socket says 172.21.15.27:45098) 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f21760aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.634+0000 7f21758a9640 1 -- 172.21.15.27:0/3110470845 learned_addr learned my addr 172.21.15.27:0/3110470845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.635+0000 7f21750a8640 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d86b0 0x55f5079a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.635+0000 7f21760aa640 1 -- 172.21.15.27:0/3110470845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d86b0 msgr2=0x55f5079a9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.635+0000 7f21760aa640 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d86b0 0x55f5079a9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.635+0000 7f21760aa640 1 -- 172.21.15.27:0/3110470845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 msgr2=0x55f5079a8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.635+0000 7f21760aa640 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.635+0000 7f21760aa640 1 -- 172.21.15.27:0/3110470845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5078c04b0 con 0x55f5075f9390 2024-04-25T13:44:43.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f21760aa640 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079aa140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f21680c8140 tx=0x7f21680095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55b4e41dd706a92f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f21597fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:44:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f21597fa640 1 -- 172.21.15.27:0/3110470845 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21680c8b40 con 0x55f5075f9390 2024-04-25T13:44:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f21597fa640 1 -- 172.21.15.27:0/3110470845 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f21680c46d0 con 0x55f5075f9390 2024-04-25T13:44:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f217e9c2740 1 -- 172.21.15.27:0/3110470845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f216c030070 con 0x55f5075f9390 2024-04-25T13:44:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f21597fa640 1 -- 172.21.15.27:0/3110470845 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f21680cc7f0 con 0x55f5075f9390 2024-04-25T13:44:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f217e9c2740 10 client.25829 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f217e9c2740 1 -- 172.21.15.27:0/3110470845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f507a1a690 con 0x55f5075f9390 2024-04-25T13:44:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f21597fa640 1 -- 172.21.15.27:0/3110470845 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f216c030070 con 0x55f5075f9390 2024-04-25T13:44:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f21597fa640 1 -- 172.21.15.27:0/3110470845 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f21680c3db0 con 0x55f5075f9390 2024-04-25T13:44:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f217e9c2740 10 client.25829 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f217e9c2740 10 client.25829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:43.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.636+0000 7f217e9c2740 20 client.25829 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f50758d3a0 0x55f507a1ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f217e9c2740 4 client.25829 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f217e9c2740 1 -- 172.21.15.27:0/3110470845 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f507907800 con 0x55f50758d3a0 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f21758a9640 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f50758d3a0 0x55f507a1ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f21597fa640 10 client.25829 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f21758a9640 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f50758d3a0 0x55f507a1ce60 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81ca11b3eb7a490f server_cookie=5a0355d9b9e6c1f2 in_seq=0 out_seq=0 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f21597fa640 1 -- 172.21.15.27:0/3110470845 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f507907800 con 0x55f50758d3a0 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f21597fa640 10 client.25829 handle_command_reply: tid=0 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.637+0000 7f217e9c2740 1 client.25829 shutdown 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 -- 172.21.15.27:0/3110470845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 msgr2=0x55f5079aa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079aa140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f21680c8140 tx=0x7f21680095b0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 -- 172.21.15.27:0/3110470845 shutdown_connections 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f50758d3a0 0x55f507a1ce60 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5075f9390 0x55f5079aa140 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5075d86b0 0x55f5079a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 --2- 172.21.15.27:0/3110470845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5075d04d0 0x55f5079a8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 -- 172.21.15.27:0/3110470845 >> 172.21.15.27:0/3110470845 conn(0x55f5075ed4e0 msgr2=0x55f5079a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f21597fa640 0 client.25829 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 -- 172.21.15.27:0/3110470845 shutdown_connections 2024-04-25T13:44:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.638+0000 7f217e9c2740 1 -- 172.21.15.27:0/3110470845 wait complete. 2024-04-25T13:44:43.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.639+0000 7f217e9c2740 20 client.25829 trim_cache size 0 max 16384 2024-04-25T13:44:43.642 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: "29e8da23-a196-4204-ab60-c13a56fa4b7e": { 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "29e8da23-a196-4204-ab60-c13a56fa4b7e", 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:44:43.643 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 -- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f215c0afbb0 msgr2=0x7f215c0b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f215c0afbb0 0x7f215c0b20c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2160009fd0 tx=0x7f2160005f90 comp rx=0 tx=0).stop 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 -- 172.21.15.27:0/3936608082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2170108930 msgr2=0x7f2170193f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2170108930 0x7f2170193f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2168033040 tx=0x7f2168031680 comp rx=0 tx=0).stop 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 -- 172.21.15.27:0/3936608082 shutdown_connections 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f215c0afbb0 0x7f215c0b20c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2170108930 0x7f2170193f80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2170103c30 0x7f2170197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 --2- 172.21.15.27:0/3936608082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21701029f0 0x7f2170197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:43.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 -- 172.21.15.27:0/3936608082 >> 172.21.15.27:0/3936608082 conn(0x7f21700fdc00 msgr2=0x7f2170100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:43.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.640+0000 7f2177b0d640 1 -- 172.21.15.27:0/3936608082 shutdown_connections 2024-04-25T13:44:43.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:44:43.641+0000 7f2177b0d640 1 -- 172.21.15.27:0/3936608082 wait complete. 2024-04-25T13:44:43.655 INFO:tasks.cephfs.filesystem:scrub status for tag:29e8da23-a196-4204-ab60-c13a56fa4b7e - {'path': '/', 'tag': '29e8da23-a196-4204-ab60-c13a56fa4b7e', 'options': 'recursive,force'} 2024-04-25T13:44:43.978 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:43 smithi045 ceph-mon[32731]: pgmap v464: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 MiB/s wr, 786 op/s 2024-04-25T13:44:44.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:43 smithi122 ceph-mon[32843]: pgmap v464: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 MiB/s wr, 786 op/s 2024-04-25T13:44:44.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:43 smithi027 ceph-mon[29140]: pgmap v464: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.8 MiB/s wr, 786 op/s 2024-04-25T13:44:45.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:44 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:44:45.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:44 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:45.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:44 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:44:45.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:44 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:45.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:44 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:44:45.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:44 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:44:45.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:44 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:45.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:44 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:44:45.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:44 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:45.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:44 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:44:45.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:44 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:44:45.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:44 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:45.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:44 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:44:45.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:44 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:44:45.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:44 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:44:46.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:45 smithi122 ceph-mon[32843]: pgmap v465: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.7 MiB/s wr, 717 op/s 2024-04-25T13:44:46.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:45 smithi045 ceph-mon[32731]: pgmap v465: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.7 MiB/s wr, 717 op/s 2024-04-25T13:44:46.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:45 smithi027 ceph-mon[29140]: pgmap v465: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.7 MiB/s wr, 717 op/s 2024-04-25T13:44:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 220.0 s, 2817.6 tps, lat 35.441 ms stddev 41.987 2024-04-25T13:44:48.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:47 smithi122 ceph-mon[32843]: pgmap v466: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-25T13:44:48.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:47 smithi045 ceph-mon[32731]: pgmap v466: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-25T13:44:48.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:47 smithi027 ceph-mon[29140]: pgmap v466: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-25T13:44:50.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:49 smithi122 ceph-mon[32843]: pgmap v467: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.5 MiB/s wr, 770 op/s 2024-04-25T13:44:50.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:49 smithi045 ceph-mon[32731]: pgmap v467: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.5 MiB/s wr, 770 op/s 2024-04-25T13:44:50.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:49 smithi027 ceph-mon[29140]: pgmap v467: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.5 MiB/s wr, 770 op/s 2024-04-25T13:44:52.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:51 smithi122 ceph-mon[32843]: pgmap v468: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.3 MiB/s wr, 861 op/s 2024-04-25T13:44:52.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:51 smithi045 ceph-mon[32731]: pgmap v468: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.3 MiB/s wr, 861 op/s 2024-04-25T13:44:52.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:51 smithi027 ceph-mon[29140]: pgmap v468: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.3 MiB/s wr, 861 op/s 2024-04-25T13:44:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 225.0 s, 4172.4 tps, lat 23.992 ms stddev 26.191 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.437+0000 7f88fdb08640 1 -- 172.21.15.27:0/412667206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f810bd00 msgr2=0x7f88f810e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.437+0000 7f88fdb08640 1 --2- 172.21.15.27:0/412667206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f810bd00 0x7f88f810e120 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009ca0 tx=0x7f88f002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.437+0000 7f88fdb08640 1 -- 172.21.15.27:0/412667206 shutdown_connections 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.437+0000 7f88fdb08640 1 --2- 172.21.15.27:0/412667206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f810bd00 0x7f88f810e120 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.437+0000 7f88fdb08640 1 --2- 172.21.15.27:0/412667206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88f8105ac0 0x7f88f8105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.437+0000 7f88fdb08640 1 --2- 172.21.15.27:0/412667206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f8107400 0x7f88f8105580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.437+0000 7f88fdb08640 1 -- 172.21.15.27:0/412667206 >> 172.21.15.27:0/412667206 conn(0x7f88f8100f90 msgr2=0x7f88f8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.438+0000 7f88fdb08640 1 -- 172.21.15.27:0/412667206 shutdown_connections 2024-04-25T13:44:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.438+0000 7f88fdb08640 1 -- 172.21.15.27:0/412667206 wait complete. 2024-04-25T13:44:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.439+0000 7f88fdb08640 1 Processor -- start 2024-04-25T13:44:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88fdb08640 1 -- start start 2024-04-25T13:44:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88fdb08640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88f8105ac0 0x7f88f81a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88fdb08640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f8107400 0x7f88f81a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88fdb08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f810bd00 0x7f88f81a7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88fdb08640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81119d0 con 0x7f88f810bd00 2024-04-25T13:44:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88f6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f8107400 0x7f88f81a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88f8105ac0 0x7f88f81a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88f6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f8107400 0x7f88f81a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33290/0 (socket says 172.21.15.27:33290) 2024-04-25T13:44:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88f6ffd640 1 -- 172.21.15.27:0/2944044471 learned_addr learned my addr 172.21.15.27:0/2944044471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88f7fff640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f810bd00 0x7f88f81a7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f88f8111850 con 0x7f88f8105ac0 2024-04-25T13:44:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.440+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f88f8111b50 con 0x7f88f8107400 2024-04-25T13:44:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.441+0000 7f88f6ffd640 1 -- 172.21.15.27:0/2944044471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88f8105ac0 msgr2=0x7f88f81a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.441+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88f8105ac0 0x7f88f81a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.441+0000 7f88f6ffd640 1 -- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f810bd00 msgr2=0x7f88f81a7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.441+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f810bd00 0x7f88f81a7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.441+0000 7f88f6ffd640 1 -- 172.21.15.27:0/2944044471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f81a8440 con 0x7f88f8107400 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.442+0000 7f88f77fe640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88f8105ac0 0x7f88f81a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.442+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f8107400 0x7f88f81a0e40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00e9f0 tx=0x7f88ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.442+0000 7f88f4ff9640 1 -- 172.21.15.27:0/2944044471 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec00cd80 con 0x7f88f8107400 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.442+0000 7f88f4ff9640 1 -- 172.21.15.27:0/2944044471 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88ec0108e0 con 0x7f88f8107400 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.442+0000 7f88f4ff9640 1 -- 172.21.15.27:0/2944044471 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec018980 con 0x7f88f8107400 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.442+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f81a8730 con 0x7f88f8107400 2024-04-25T13:44:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.442+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f81a8b70 con 0x7f88f8107400 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f89049b7740 1 Processor -- start 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f89049b7740 1 -- start start 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f88f4ff9640 1 -- 172.21.15.27:0/2944044471 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f88ec010430 con 0x7f88f8107400 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f89049b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9240970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f89049b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd9240ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f89049b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd9247df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f89049b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ddd91a6460 con 0x55ddd8e7f4d0 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f89049b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ddd9160590 con 0x55ddd8e772f0 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f89049b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ddd91a2420 con 0x55ddd8ea01b0 2024-04-25T13:44:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd9240ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f88f6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd9247df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd9240ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:59158/0 (socket says 172.21.15.27:59158) 2024-04-25T13:44:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.447+0000 7f88f77fe640 1 -- 172.21.15.27:0/513430375 learned_addr learned my addr 172.21.15.27:0/513430375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.448+0000 7f88f7fff640 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9240970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.448+0000 7f88f4ff9640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88e00afb60 0x7f88e00b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.448+0000 7f88f7fff640 1 -- 172.21.15.27:0/513430375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 msgr2=0x55ddd9240ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.448+0000 7f88f7fff640 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd9240ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88f7fff640 1 -- 172.21.15.27:0/513430375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 msgr2=0x55ddd9247df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88f7fff640 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd9247df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88f7fff640 1 -- 172.21.15.27:0/513430375 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd916d0d0 con 0x55ddd8ea01b0 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88f4ff9640 1 -- 172.21.15.27:0/2944044471 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f88ec0deef0 con 0x7f88f8107400 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88f7fff640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88e00afb60 0x7f88e00b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88f7fff640 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9240970 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f88f002ade0 tx=0x7f88f00098d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a102fd8d12a3c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd9247df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88ddffb640 1 -- 172.21.15.27:0/513430375 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f0043040 con 0x55ddd8ea01b0 2024-04-25T13:44:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88ddffb640 1 -- 172.21.15.27:0/513430375 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88f0036040 con 0x55ddd8ea01b0 2024-04-25T13:44:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.449+0000 7f88ddffb640 1 -- 172.21.15.27:0/513430375 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f0004610 con 0x55ddd8ea01b0 2024-04-25T13:44:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 -- 172.21.15.27:0/513430375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 msgr2=0x55ddd9240970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9240970 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f88f002ade0 tx=0x7f88f00098d0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f88f7fff640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88e00afb60 0x7f88e00b2070 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f88f0031980 tx=0x7f88f00047b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 -- 172.21.15.27:0/513430375 shutdown_connections 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd9247df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd9240ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 --2- 172.21.15.27:0/513430375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9240970 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 -- 172.21.15.27:0/513430375 >> 172.21.15.27:0/513430375 conn(0x55ddd8e94300 msgr2=0x55ddd923c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 -- 172.21.15.27:0/513430375 shutdown_connections 2024-04-25T13:44:53.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.451+0000 7f89049b7740 1 -- 172.21.15.27:0/513430375 wait complete. 2024-04-25T13:44:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.452+0000 7f89049b7740 1 -- start start 2024-04-25T13:44:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.452+0000 7f89049b7740 1 Processor -- start 2024-04-25T13:44:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.452+0000 7f89049b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd924a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f89049b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f89049b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9251a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f89049b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ddd9248840 con 0x55ddd8e7f4d0 2024-04-25T13:44:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f89049b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ddd912db60 con 0x55ddd8e772f0 2024-04-25T13:44:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f89049b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ddd92489c0 con 0x55ddd8ea01b0 2024-04-25T13:44:53.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f89049b7740 10 client.0 _ll_register_callbacks cb 0x55ddd91522b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd924a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd924a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:59160/0 (socket says 172.21.15.27:59160) 2024-04-25T13:44:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f77fe640 1 -- 172.21.15.27:0/795725403 learned_addr learned my addr 172.21.15.27:0/795725403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f7fff640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9251a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f7fff640 1 -- 172.21.15.27:0/795725403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 msgr2=0x55ddd924a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f7fff640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd924a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f7fff640 1 -- 172.21.15.27:0/795725403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 msgr2=0x55ddd924ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f7fff640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f7fff640 1 -- 172.21.15.27:0/795725403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddd8eb8210 con 0x55ddd8ea01b0 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.453+0000 7f88f77fe640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd924a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.454+0000 7f88f7fff640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9251a90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f88f0004af0 tx=0x7f88f0039000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=143288b1328eae2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.454+0000 7f88d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.454+0000 7f88d6ffd640 1 -- 172.21.15.27:0/795725403 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f0062060 con 0x55ddd8ea01b0 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.454+0000 7f88d6ffd640 1 -- 172.21.15.27:0/795725403 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f88f0004d40 con 0x55ddd8ea01b0 2024-04-25T13:44:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.454+0000 7f89049b7740 1 -- 172.21.15.27:0/795725403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88f0036040 con 0x55ddd8ea01b0 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.455+0000 7f88d6ffd640 1 -- 172.21.15.27:0/795725403 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f88f0058050 con 0x55ddd8ea01b0 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f89049b7740 10 client.25861 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f89049b7740 1 -- 172.21.15.27:0/795725403 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ddd8f84620 con 0x55ddd8ea01b0 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f88d6ffd640 1 -- 172.21.15.27:0/795725403 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88f0036040 con 0x55ddd8ea01b0 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f88d6ffd640 1 -- 172.21.15.27:0/795725403 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f88f0003970 con 0x55ddd8ea01b0 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f89049b7740 10 client.25861 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f89049b7740 10 client.25861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f89049b7740 20 client.25861 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f89049b7740 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ddd8e5ed80 0x55ddd9257ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f89049b7740 4 client.25861 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.456+0000 7f89049b7740 1 -- 172.21.15.27:0/795725403 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddd91b4070 con 0x55ddd8e5ed80 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.457+0000 7f88f77fe640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ddd8e5ed80 0x55ddd9257ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.457+0000 7f88f77fe640 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ddd8e5ed80 0x55ddd9257ee0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=548aee1107fce13b server_cookie=6ecb423009a16bc4 in_seq=0 out_seq=0 2024-04-25T13:44:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.457+0000 7f88d6ffd640 10 client.25861 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:53.458 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.458+0000 7f88d6ffd640 1 -- 172.21.15.27:0/795725403 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ddd91b4070 con 0x55ddd8e5ed80 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.458+0000 7f88d6ffd640 10 client.25861 handle_command_reply: tid=0 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.458+0000 7f89049b7740 1 client.25861 shutdown 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 -- 172.21.15.27:0/795725403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 msgr2=0x55ddd9251a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9251a90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f88f0004af0 tx=0x7f88f0039000 comp rx=0 tx=0).stop 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 -- 172.21.15.27:0/795725403 shutdown_connections 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ddd8e5ed80 0x55ddd9257ee0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd9251a90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 --2- 172.21.15.27:0/795725403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e772f0 0x55ddd924a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 -- 172.21.15.27:0/795725403 >> 172.21.15.27:0/795725403 conn(0x55ddd8e94300 msgr2=0x55ddd923f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f88d6ffd640 0 client.25861 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 -- 172.21.15.27:0/795725403 shutdown_connections 2024-04-25T13:44:53.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 1 -- 172.21.15.27:0/795725403 wait complete. 2024-04-25T13:44:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.459+0000 7f89049b7740 20 client.25861 trim_cache size 0 max 16384 2024-04-25T13:44:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f89049b7740 1 Processor -- start 2024-04-25T13:44:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f89049b7740 1 -- start start 2024-04-25T13:44:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f89049b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd923bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f89049b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 0x55ddd92519b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f89049b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd923c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f89049b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81af820 con 0x55ddd8e772f0 2024-04-25T13:44:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f89049b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81b20a0 con 0x55ddd8e7f4d0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f89049b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81afe90 con 0x55ddd8ea01b0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd923bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd923bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60718/0 (socket says 172.21.15.27:60718) 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f88f77fe640 1 -- 172.21.15.27:0/4201625712 learned_addr learned my addr 172.21.15.27:0/4201625712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f88f7fff640 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd923c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.486+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 0x55ddd92519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88f77fe640 1 -- 172.21.15.27:0/4201625712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 msgr2=0x55ddd923c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88f77fe640 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd923c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88f77fe640 1 -- 172.21.15.27:0/4201625712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 msgr2=0x55ddd92519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88f77fe640 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 0x55ddd92519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88f77fe640 1 -- 172.21.15.27:0/4201625712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd92481b0 con 0x55ddd8e772f0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88f77fe640 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd923bcc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f88e80126b0 tx=0x7f88e8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163b17ef2a54d230 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88dd7fa640 1 -- 172.21.15.27:0/4201625712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8012e60 con 0x55ddd8e772f0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88dd7fa640 1 -- 172.21.15.27:0/4201625712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88e8027070 con 0x55ddd8e772f0 2024-04-25T13:44:53.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.487+0000 7f88dd7fa640 1 -- 172.21.15.27:0/4201625712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8003850 con 0x55ddd8e772f0 2024-04-25T13:44:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 -- 172.21.15.27:0/4201625712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 msgr2=0x55ddd923bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd923bcc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f88e80126b0 tx=0x7f88e8005f90 comp rx=0 tx=0).stop 2024-04-25T13:44:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 -- 172.21.15.27:0/4201625712 shutdown_connections 2024-04-25T13:44:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd923c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 0x55ddd92519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 --2- 172.21.15.27:0/4201625712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd923bcc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 -- 172.21.15.27:0/4201625712 >> 172.21.15.27:0/4201625712 conn(0x55ddd8e94300 msgr2=0x55ddd9244a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 -- 172.21.15.27:0/4201625712 shutdown_connections 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.488+0000 7f89049b7740 1 -- 172.21.15.27:0/4201625712 wait complete. 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.489+0000 7f89049b7740 1 -- start start 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.489+0000 7f89049b7740 1 Processor -- start 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.489+0000 7f89049b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd924ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.489+0000 7f89049b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.489+0000 7f89049b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd92501b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.489+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd924ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.489+0000 7f88f77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd924ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60726/0 (socket says 172.21.15.27:60726) 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.489+0000 7f89049b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81b0560 con 0x55ddd8e772f0 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f77fe640 1 -- 172.21.15.27:0/1271052455 learned_addr learned my addr 172.21.15.27:0/1271052455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81b19d0 con 0x55ddd8e7f4d0 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f88f81b0c30 con 0x55ddd8ea01b0 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f89049b7740 10 client.0 _ll_register_callbacks cb 0x7f88ec0a3ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:44:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f7fff640 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd92501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f77fe640 1 -- 172.21.15.27:0/1271052455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 msgr2=0x55ddd92501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f77fe640 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd92501b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f77fe640 1 -- 172.21.15.27:0/1271052455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 msgr2=0x55ddd924f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f77fe640 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f77fe640 1 -- 172.21.15.27:0/1271052455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddd9251ef0 con 0x55ddd8e772f0 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88f77fe640 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd924ed40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f88e8012610 tx=0x7f88e80128d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50b317aaf915784 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:44:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88d6ffd640 1 -- 172.21.15.27:0/1271052455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8012e60 con 0x55ddd8e772f0 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f88d6ffd640 1 -- 172.21.15.27:0/1271052455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f88e8011b10 con 0x55ddd8e772f0 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.490+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88e8027070 con 0x55ddd8e772f0 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f88d6ffd640 1 -- 172.21.15.27:0/1271052455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f88e8011d10 con 0x55ddd8e772f0 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f89049b7740 10 client.16137 fetch_fsmap learned FSMap version 19 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ddd9251810 con 0x55ddd8e772f0 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f88d6ffd640 1 -- 172.21.15.27:0/1271052455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88e8027070 con 0x55ddd8e772f0 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f88d6ffd640 1 -- 172.21.15.27:0/1271052455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f88e8021320 con 0x55ddd8e772f0 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f89049b7740 10 client.16137 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f89049b7740 10 client.16137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f89049b7740 20 client.16137 populate_metadata read hostname 'smithi027' 2024-04-25T13:44:53.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f89049b7740 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ddd8e341c0 0x55ddd92c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:44:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f89049b7740 4 client.16137 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:44:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ddd91adee0 con 0x55ddd8e341c0 2024-04-25T13:44:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.491+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ddd8e341c0 0x55ddd92c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:44:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.492+0000 7f88d6ffd640 10 client.16137 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:53.496 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.492+0000 7f88f6ffd640 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ddd8e341c0 0x55ddd92c24c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d9cbc3de3a2d602 server_cookie=d095f8a4099931da in_seq=0 out_seq=0 2024-04-25T13:44:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.571+0000 7f88d6ffd640 1 -- 172.21.15.27:0/1271052455 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ddd91adee0 con 0x55ddd8e341c0 2024-04-25T13:44:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f88d6ffd640 10 client.16137 handle_command_reply: tid=0 2024-04-25T13:44:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 client.16137 shutdown 2024-04-25T13:44:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 msgr2=0x55ddd924ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd924ed40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f88e8012610 tx=0x7f88e80128d0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 shutdown_connections 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ddd8e341c0 0x55ddd92c24c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f88d6ffd640 0 client.16137 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ddd8ea01b0 0x55ddd92501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ddd8e7f4d0 0x55ddd924f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 --2- 172.21.15.27:0/1271052455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ddd8e772f0 0x55ddd924ed40 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 >> 172.21.15.27:0/1271052455 conn(0x55ddd8e94300 msgr2=0x55ddd923f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 shutdown_connections 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.572+0000 7f89049b7740 1 -- 172.21.15.27:0/1271052455 wait complete. 2024-04-25T13:44:53.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f89049b7740 20 client.16137 trim_cache size 0 max 16384 2024-04-25T13:44:53.574 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:44:53.574 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:44:53.574 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:44:53.574 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88e00afb60 msgr2=0x7f88e00b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88e00afb60 0x7f88e00b2070 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f88f0031980 tx=0x7f88f00047b0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f8107400 msgr2=0x7f88f81a0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f8107400 0x7f88f81a0e40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00e9f0 tx=0x7f88ec00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 shutdown_connections 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88e00afb60 0x7f88e00b2070 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88f810bd00 0x7f88f81a7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88f8107400 0x7f88f81a0e40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 --2- 172.21.15.27:0/2944044471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88f8105ac0 0x7f88f81a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.573+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 >> 172.21.15.27:0/2944044471 conn(0x7f88f8100f90 msgr2=0x7f88f8107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.574+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 shutdown_connections 2024-04-25T13:44:53.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:44:53.574+0000 7f88fdb08640 1 -- 172.21.15.27:0/2944044471 wait complete. 2024-04-25T13:44:54.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:53 smithi122 ceph-mon[32843]: pgmap v469: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.9 MiB/s wr, 928 op/s 2024-04-25T13:44:54.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:53 smithi045 ceph-mon[32731]: pgmap v469: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.9 MiB/s wr, 928 op/s 2024-04-25T13:44:54.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:53 smithi027 ceph-mon[29140]: pgmap v469: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.9 MiB/s wr, 928 op/s 2024-04-25T13:44:56.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:55 smithi122 ceph-mon[32843]: pgmap v470: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.7 MiB/s wr, 883 op/s 2024-04-25T13:44:56.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:55 smithi045 ceph-mon[32731]: pgmap v470: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.7 MiB/s wr, 883 op/s 2024-04-25T13:44:56.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:55 smithi027 ceph-mon[29140]: pgmap v470: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.7 MiB/s wr, 883 op/s 2024-04-25T13:44:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 230.0 s, 3274.8 tps, lat 30.527 ms stddev 36.015 2024-04-25T13:44:58.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:57 smithi122 ceph-mon[32843]: pgmap v471: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 11 MiB/s wr, 994 op/s 2024-04-25T13:44:58.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:57 smithi045 ceph-mon[32731]: pgmap v471: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 11 MiB/s wr, 994 op/s 2024-04-25T13:44:58.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:57 smithi027 ceph-mon[29140]: pgmap v471: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 11 MiB/s wr, 994 op/s 2024-04-25T13:45:00.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:44:59 smithi122 ceph-mon[32843]: pgmap v472: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 727 op/s 2024-04-25T13:45:00.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:44:59 smithi045 ceph-mon[32731]: pgmap v472: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 727 op/s 2024-04-25T13:45:00.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:44:59 smithi027 ceph-mon[29140]: pgmap v472: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 727 op/s 2024-04-25T13:45:02.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:01 smithi122 ceph-mon[32843]: pgmap v473: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 9.5 MiB/s wr, 827 op/s 2024-04-25T13:45:02.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:01 smithi045 ceph-mon[32731]: pgmap v473: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 9.5 MiB/s wr, 827 op/s 2024-04-25T13:45:02.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:01 smithi027 ceph-mon[29140]: pgmap v473: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 9.5 MiB/s wr, 827 op/s 2024-04-25T13:45:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 235.0 s, 3452.8 tps, lat 28.911 ms stddev 32.035 2024-04-25T13:45:04.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:03 smithi122 ceph-mon[32843]: pgmap v474: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 896 op/s 2024-04-25T13:45:04.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:03 smithi045 ceph-mon[32731]: pgmap v474: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 896 op/s 2024-04-25T13:45:04.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:03 smithi027 ceph-mon[29140]: pgmap v474: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 896 op/s 2024-04-25T13:45:06.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:05 smithi122 ceph-mon[32843]: pgmap v475: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9.4 MiB/s wr, 783 op/s 2024-04-25T13:45:06.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:05 smithi045 ceph-mon[32731]: pgmap v475: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9.4 MiB/s wr, 783 op/s 2024-04-25T13:45:06.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:05 smithi027 ceph-mon[29140]: pgmap v475: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 9.4 MiB/s wr, 783 op/s 2024-04-25T13:45:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 -- 172.21.15.27:0/583325149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d84108100 msgr2=0x7f1d8410a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 --2- 172.21.15.27:0/583325149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d84108100 0x7f1d8410a520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c009ca0 tx=0x7f1d7c02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 -- 172.21.15.27:0/583325149 shutdown_connections 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 --2- 172.21.15.27:0/583325149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d84108100 0x7f1d8410a520 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 --2- 172.21.15.27:0/583325149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d84103400 0x7f1d84103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 --2- 172.21.15.27:0/583325149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d841021c0 0x7f1d84102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 -- 172.21.15.27:0/583325149 >> 172.21.15.27:0/583325149 conn(0x7f1d840fd390 msgr2=0x7f1d840ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 -- 172.21.15.27:0/583325149 shutdown_connections 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.724+0000 7f1d8b484640 1 -- 172.21.15.27:0/583325149 wait complete. 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d8b484640 1 Processor -- start 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d8b484640 1 -- start start 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d8b484640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d841021c0 0x7f1d84109d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d8b484640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d84103400 0x7f1d8419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d8b484640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d84108100 0x7f1d841a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d8b484640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8410dce0 con 0x7f1d841021c0 2024-04-25T13:45:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d8b484640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8410db60 con 0x7f1d84108100 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d8b484640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8410de60 con 0x7f1d84103400 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d89220640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d841021c0 0x7f1d84109d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d89220640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d841021c0 0x7f1d84109d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36414/0 (socket says 172.21.15.27:36414) 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d89220640 1 -- 172.21.15.27:0/1285392537 learned_addr learned my addr 172.21.15.27:0/1285392537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.725+0000 7f1d89a21640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d84108100 0x7f1d841a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d84103400 0x7f1d8419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d89220640 1 -- 172.21.15.27:0/1285392537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d84103400 msgr2=0x7f1d8419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d89220640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d84103400 0x7f1d8419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d89220640 1 -- 172.21.15.27:0/1285392537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d84108100 msgr2=0x7f1d841a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d89220640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d84108100 0x7f1d841a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d89220640 1 -- 172.21.15.27:0/1285392537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d841a1ae0 con 0x7f1d841021c0 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d89220640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d841021c0 0x7f1d84109d40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1d80005fd0 tx=0x7f1d8000dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d7a7fc640 1 -- 172.21.15.27:0/1285392537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80018070 con 0x7f1d841021c0 2024-04-25T13:45:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d7a7fc640 1 -- 172.21.15.27:0/1285392537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d80004220 con 0x7f1d841021c0 2024-04-25T13:45:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d7a7fc640 1 -- 172.21.15.27:0/1285392537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80012e40 con 0x7f1d841021c0 2024-04-25T13:45:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d8b484640 1 -- 172.21.15.27:0/1285392537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d84198ba0 con 0x7f1d841021c0 2024-04-25T13:45:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.726+0000 7f1d8b484640 1 -- 172.21.15.27:0/1285392537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d841990f0 con 0x7f1d841021c0 2024-04-25T13:45:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.730+0000 7f1d9240b740 1 Processor -- start 2024-04-25T13:45:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d9240b740 1 -- start start 2024-04-25T13:45:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d87180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d87700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d8e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d9240b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fd68ce4460 con 0x55fd689b52f0 2024-04-25T13:45:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d9240b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55fd68c9e590 con 0x55fd689bd4d0 2024-04-25T13:45:06.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d9240b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55fd68ce0420 con 0x55fd689de1b0 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d89a21640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d87180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d89a21640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d87180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59708/0 (socket says 172.21.15.27:59708) 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.731+0000 7f1d89a21640 1 -- 172.21.15.27:0/1434014101 learned_addr learned my addr 172.21.15.27:0/1434014101 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.732+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/1434014101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d8e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.732+0000 7f1d89a21640 1 -- 172.21.15.27:0/1434014101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 msgr2=0x55fd68d8e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.732+0000 7f1d89a21640 1 --2- 172.21.15.27:0/1434014101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d8e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.732+0000 7f1d89a21640 1 -- 172.21.15.27:0/1434014101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 msgr2=0x55fd68d87700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.732+0000 7f1d89a21640 1 --2- 172.21.15.27:0/1434014101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d87700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.732+0000 7f1d89a21640 1 -- 172.21.15.27:0/1434014101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd68cab0d0 con 0x55fd689de1b0 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.732+0000 7f1d7a7fc640 1 -- 172.21.15.27:0/1285392537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1d800099b0 con 0x7f1d841021c0 2024-04-25T13:45:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.732+0000 7f1d89a21640 1 --2- 172.21.15.27:0/1434014101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d87180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0098b0 tx=0x7f1d7c0098e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2a996cb4d322bb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.733+0000 7f1d6f7fe640 1 -- 172.21.15.27:0/1434014101 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c031650 con 0x55fd689de1b0 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.733+0000 7f1d6f7fe640 1 -- 172.21.15.27:0/1434014101 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d7c036030 con 0x55fd689de1b0 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.733+0000 7f1d6f7fe640 1 -- 172.21.15.27:0/1434014101 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0045a0 con 0x55fd689de1b0 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.733+0000 7f1d7a7fc640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d700afc30 0x7f1d700b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.733+0000 7f1d7a7fc640 1 -- 172.21.15.27:0/1285392537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1d800de200 con 0x7f1d841021c0 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.733+0000 7f1d89220640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d700afc30 0x7f1d700b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 -- 172.21.15.27:0/1434014101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 msgr2=0x55fd68d87180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1434014101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d87180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0098b0 tx=0x7f1d7c0098e0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 -- 172.21.15.27:0/1434014101 shutdown_connections 2024-04-25T13:45:06.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1434014101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d8e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1434014101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d87700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1434014101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d87180 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 -- 172.21.15.27:0/1434014101 >> 172.21.15.27:0/1434014101 conn(0x55fd689d2300 msgr2=0x55fd68d82b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 -- 172.21.15.27:0/1434014101 shutdown_connections 2024-04-25T13:45:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 -- 172.21.15.27:0/1434014101 wait complete. 2024-04-25T13:45:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d89220640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d700afc30 0x7f1d700b2140 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1d800a6640 tx=0x7f1d8000b660 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 -- start start 2024-04-25T13:45:06.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.734+0000 7f1d9240b740 1 Processor -- start 2024-04-25T13:45:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d90d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d91240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d981a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d9240b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fd68d8ee70 con 0x55fd689b52f0 2024-04-25T13:45:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d9240b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55fd68c6bb60 con 0x55fd689bd4d0 2024-04-25T13:45:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d9240b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55fd68d8eff0 con 0x55fd689de1b0 2024-04-25T13:45:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d9240b740 10 client.0 _ll_register_callbacks cb 0x55fd68c902b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d89a21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d90d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d88a1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d91240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d89a21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d90d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36430/0 (socket says 172.21.15.27:36430) 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d88a1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d91240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:46184/0 (socket says 172.21.15.27:46184) 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d89a21640 1 -- 172.21.15.27:0/4285210702 learned_addr learned my addr 172.21.15.27:0/4285210702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d89a21640 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d88a1f640 1 -- 172.21.15.27:0/4285210702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 msgr2=0x55fd68d981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d88a1f640 1 -- 172.21.15.27:0/4285210702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 msgr2=0x55fd68d90d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d90d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d88a1f640 1 -- 172.21.15.27:0/4285210702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd689f6210 con 0x55fd689bd4d0 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d91240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d841096e0 tx=0x7f1d74013b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=895256b7542f0e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.735+0000 7f1d6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.736+0000 7f1d6cff9640 1 -- 172.21.15.27:0/4285210702 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74004460 con 0x55fd689bd4d0 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.736+0000 7f1d6cff9640 1 -- 172.21.15.27:0/4285210702 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1d740119c0 con 0x55fd689bd4d0 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.736+0000 7f1d9240b740 1 -- 172.21.15.27:0/4285210702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d7c036030 con 0x55fd689bd4d0 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.736+0000 7f1d6cff9640 1 -- 172.21.15.27:0/4285210702 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1d74011bc0 con 0x55fd689bd4d0 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.736+0000 7f1d9240b740 10 client.25856 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.736+0000 7f1d9240b740 1 -- 172.21.15.27:0/4285210702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55fd68ac2620 con 0x55fd689bd4d0 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.736+0000 7f1d6cff9640 1 -- 172.21.15.27:0/4285210702 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d7c036030 con 0x55fd689bd4d0 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.736+0000 7f1d6cff9640 1 -- 172.21.15.27:0/4285210702 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1d7401a070 con 0x55fd689bd4d0 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.737+0000 7f1d9240b740 10 client.25856 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.737+0000 7f1d9240b740 10 client.25856 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.737+0000 7f1d9240b740 20 client.25856 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.737+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fd6899cd80 0x55fd68d9eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.737+0000 7f1d9240b740 4 client.25856 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.737+0000 7f1d9240b740 1 -- 172.21.15.27:0/4285210702 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd68cf2070 con 0x55fd6899cd80 2024-04-25T13:45:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.737+0000 7f1d89a21640 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fd6899cd80 0x55fd68d9eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.738+0000 7f1d89a21640 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fd6899cd80 0x55fd68d9eaa0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=196815b31b317c1a server_cookie=909c1dd13064f14a in_seq=0 out_seq=0 2024-04-25T13:45:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.738+0000 7f1d6cff9640 10 client.25856 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d6cff9640 1 -- 172.21.15.27:0/4285210702 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fd68cf2070 con 0x55fd6899cd80 2024-04-25T13:45:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d6cff9640 10 client.25856 handle_command_reply: tid=0 2024-04-25T13:45:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 client.25856 shutdown 2024-04-25T13:45:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 -- 172.21.15.27:0/4285210702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 msgr2=0x55fd68d91240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d91240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d841096e0 tx=0x7f1d74013b70 comp rx=0 tx=0).stop 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 -- 172.21.15.27:0/4285210702 shutdown_connections 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fd6899cd80 0x55fd68d9eaa0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689de1b0 0x55fd68d981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689bd4d0 0x55fd68d91240 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4285210702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d90d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d6cff9640 0 client.25856 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 -- 172.21.15.27:0/4285210702 >> 172.21.15.27:0/4285210702 conn(0x55fd689d2300 msgr2=0x55fd68d79250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.739+0000 7f1d9240b740 1 -- 172.21.15.27:0/4285210702 shutdown_connections 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.740+0000 7f1d9240b740 1 -- 172.21.15.27:0/4285210702 wait complete. 2024-04-25T13:45:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.740+0000 7f1d9240b740 20 client.25856 trim_cache size 0 max 16384 2024-04-25T13:45:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.763+0000 7f1d9240b740 1 Processor -- start 2024-04-25T13:45:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.763+0000 7f1d9240b740 1 -- start start 2024-04-25T13:45:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.763+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689b52f0 0x55fd68d8b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689bd4d0 0x55fd68d8b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 0x55fd68d980c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d88a1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689bd4d0 0x55fd68d8b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d9240b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1d841a6d60 con 0x55fd689bd4d0 2024-04-25T13:45:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d88a1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689bd4d0 0x55fd68d8b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36442/0 (socket says 172.21.15.27:36442) 2024-04-25T13:45:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d9240b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1d841a95e0 con 0x55fd689de1b0 2024-04-25T13:45:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d88a1f640 1 -- 172.21.15.27:0/4061962216 learned_addr learned my addr 172.21.15.27:0/4061962216 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d9240b740 1 -- 172.21.15.27:0/4061962216 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1d841a73d0 con 0x55fd689b52f0 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d88a1f640 1 -- 172.21.15.27:0/4061962216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689b52f0 msgr2=0x55fd68d8b360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/4061962216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689b52f0 0x55fd68d8b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d88a1f640 1 -- 172.21.15.27:0/4061962216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 msgr2=0x55fd68d980c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.764+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/4061962216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 0x55fd68d980c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.765+0000 7f1d88a1f640 1 -- 172.21.15.27:0/4061962216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd68d98600 con 0x55fd689bd4d0 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.765+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/4061962216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689bd4d0 0x55fd68d8b8a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1d740097e0 tx=0x7f1d740117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329ef03e76e224ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.765+0000 7f1d6effd640 1 -- 172.21.15.27:0/4061962216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7401c820 con 0x55fd689bd4d0 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.765+0000 7f1d6effd640 1 -- 172.21.15.27:0/4061962216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d7401c9c0 con 0x55fd689bd4d0 2024-04-25T13:45:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.765+0000 7f1d6effd640 1 -- 172.21.15.27:0/4061962216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74003db0 con 0x55fd689bd4d0 2024-04-25T13:45:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.766+0000 7f1d9240b740 1 -- 172.21.15.27:0/4061962216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689bd4d0 msgr2=0x55fd68d8b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.766+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4061962216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689bd4d0 0x55fd68d8b8a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1d740097e0 tx=0x7f1d740117b0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.766+0000 7f1d9240b740 1 -- 172.21.15.27:0/4061962216 shutdown_connections 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.766+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4061962216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 0x55fd68d980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.766+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4061962216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689bd4d0 0x55fd68d8b8a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.766+0000 7f1d9240b740 1 --2- 172.21.15.27:0/4061962216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689b52f0 0x55fd68d8b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.766+0000 7f1d9240b740 1 -- 172.21.15.27:0/4061962216 >> 172.21.15.27:0/4061962216 conn(0x55fd689d2300 msgr2=0x55fd68d8f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.767+0000 7f1d9240b740 1 -- 172.21.15.27:0/4061962216 shutdown_connections 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.767+0000 7f1d9240b740 1 -- 172.21.15.27:0/4061962216 wait complete. 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.767+0000 7f1d9240b740 1 -- start start 2024-04-25T13:45:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.767+0000 7f1d9240b740 1 Processor -- start 2024-04-25T13:45:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.767+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d95700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.767+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689bd4d0 0x55fd68d95c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d9240b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 0x55fd68d96ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d9240b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1d841a7aa0 con 0x55fd689b52f0 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d9240b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1d841a8f10 con 0x55fd689de1b0 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d9240b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1d841a8170 con 0x55fd689bd4d0 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d89a21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d95700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d9240b740 10 client.0 _ll_register_callbacks cb 0x7f1d74009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d89a21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d95700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36446/0 (socket says 172.21.15.27:36446) 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d89a21640 1 -- 172.21.15.27:0/1927296322 learned_addr learned my addr 172.21.15.27:0/1927296322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689bd4d0 0x55fd68d95c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d89a21640 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 0x55fd68d96ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d88a1f640 1 -- 172.21.15.27:0/1927296322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 msgr2=0x55fd68d96ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 0x55fd68d96ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d88a1f640 1 -- 172.21.15.27:0/1927296322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 msgr2=0x55fd68d95700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d95700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.768+0000 7f1d88a1f640 1 -- 172.21.15.27:0/1927296322 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd68d97070 con 0x55fd689bd4d0 2024-04-25T13:45:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.769+0000 7f1d88a1f640 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689bd4d0 0x55fd68d95c40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1d7401c6d0 tx=0x7f1d74005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=704f7836217e1d0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.769+0000 7f1d6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d6cff9640 1 -- 172.21.15.27:0/1927296322 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74005510 con 0x55fd689bd4d0 2024-04-25T13:45:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d6cff9640 1 -- 172.21.15.27:0/1927296322 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1d74003b30 con 0x55fd689bd4d0 2024-04-25T13:45:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d9240b740 1 -- 172.21.15.27:0/1927296322 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d7401c9c0 con 0x55fd689bd4d0 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d6cff9640 1 -- 172.21.15.27:0/1927296322 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1d740056b0 con 0x55fd689bd4d0 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d9240b740 10 client.25882 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d9240b740 1 -- 172.21.15.27:0/1927296322 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55fd68d98300 con 0x55fd689bd4d0 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d6cff9640 1 -- 172.21.15.27:0/1927296322 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d7401c9c0 con 0x55fd689bd4d0 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d6cff9640 1 -- 172.21.15.27:0/1927296322 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1d74040030 con 0x55fd689bd4d0 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d9240b740 10 client.25882 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d9240b740 10 client.25882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d9240b740 20 client.25882 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.770+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fd689721c0 0x55fd68e094d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.771+0000 7f1d9240b740 4 client.25882 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:45:06.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.771+0000 7f1d9240b740 1 -- 172.21.15.27:0/1927296322 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd68cebee0 con 0x55fd689721c0 2024-04-25T13:45:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.771+0000 7f1d89a21640 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fd689721c0 0x55fd68e094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.771+0000 7f1d6cff9640 10 client.25882 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:06.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.771+0000 7f1d89a21640 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fd689721c0 0x55fd68e094d0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6557eda8ca099c67 server_cookie=288c27f8f0f8a5de in_seq=0 out_seq=0 2024-04-25T13:45:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d6cff9640 1 -- 172.21.15.27:0/1927296322 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fd68cebee0 con 0x55fd689721c0 2024-04-25T13:45:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d6cff9640 10 client.25882 handle_command_reply: tid=0 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 client.25882 shutdown 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 -- 172.21.15.27:0/1927296322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689bd4d0 msgr2=0x55fd68d95c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689bd4d0 0x55fd68d95c40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1d7401c6d0 tx=0x7f1d74005f80 comp rx=0 tx=0).stop 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 -- 172.21.15.27:0/1927296322 shutdown_connections 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fd689721c0 0x55fd68e094d0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd689de1b0 0x55fd68d96ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d6cff9640 0 client.25882 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fd689bd4d0 0x55fd68d95c40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 --2- 172.21.15.27:0/1927296322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fd689b52f0 0x55fd68d95700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 -- 172.21.15.27:0/1927296322 >> 172.21.15.27:0/1927296322 conn(0x55fd689d2300 msgr2=0x55fd68d8f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.788+0000 7f1d9240b740 1 -- 172.21.15.27:0/1927296322 shutdown_connections 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d9240b740 1 -- 172.21.15.27:0/1927296322 wait complete. 2024-04-25T13:45:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d9240b740 20 client.25882 trim_cache size 0 max 16384 2024-04-25T13:45:06.790 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:45:06.790 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:45:06.790 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:45:06.790 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 -- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d700afc30 msgr2=0x7f1d700b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d700afc30 0x7f1d700b2140 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1d800a6640 tx=0x7f1d8000b660 comp rx=0 tx=0).stop 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 -- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d841021c0 msgr2=0x7f1d84109d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d841021c0 0x7f1d84109d40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1d80005fd0 tx=0x7f1d8000dd20 comp rx=0 tx=0).stop 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 -- 172.21.15.27:0/1285392537 shutdown_connections 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1d700afc30 0x7f1d700b2140 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d84108100 0x7f1d841a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d84103400 0x7f1d8419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 --2- 172.21.15.27:0/1285392537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d841021c0 0x7f1d84109d40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 -- 172.21.15.27:0/1285392537 >> 172.21.15.27:0/1285392537 conn(0x7f1d840fd390 msgr2=0x7f1d841040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 -- 172.21.15.27:0/1285392537 shutdown_connections 2024-04-25T13:45:06.791 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:06.789+0000 7f1d8b484640 1 -- 172.21.15.27:0/1285392537 wait complete. 2024-04-25T13:45:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 240.0 s, 3321.8 tps, lat 30.103 ms stddev 33.038 2024-04-25T13:45:08.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:07 smithi122 ceph-mon[32843]: pgmap v476: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.3 MiB/s wr, 837 op/s 2024-04-25T13:45:08.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:07 smithi045 ceph-mon[32731]: pgmap v476: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.3 MiB/s wr, 837 op/s 2024-04-25T13:45:08.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:07 smithi027 ceph-mon[29140]: pgmap v476: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.3 MiB/s wr, 837 op/s 2024-04-25T13:45:10.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:09 smithi122 ceph-mon[32843]: pgmap v477: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 654 op/s 2024-04-25T13:45:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:09 smithi045 ceph-mon[32731]: pgmap v477: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 654 op/s 2024-04-25T13:45:10.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:09 smithi027 ceph-mon[29140]: pgmap v477: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 654 op/s 2024-04-25T13:45:12.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:11 smithi122 ceph-mon[32843]: pgmap v478: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 756 op/s 2024-04-25T13:45:12.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:11 smithi045 ceph-mon[32731]: pgmap v478: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 756 op/s 2024-04-25T13:45:12.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:11 smithi027 ceph-mon[29140]: pgmap v478: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 756 op/s 2024-04-25T13:45:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 245.0 s, 3600.4 tps, lat 27.840 ms stddev 31.323 2024-04-25T13:45:13.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:45:13.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:45:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3322485920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7010a3d0 msgr2=0x7f1e7010adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3322485920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7010a3d0 0x7f1e7010adf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009860 tx=0x7f1e6c02a340 comp rx=0 tx=0).stop 2024-04-25T13:45:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3322485920 shutdown_connections 2024-04-25T13:45:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3322485920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e70110030 0x7f1e70112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3322485920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e7010b330 0x7f1e7010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3322485920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e7010a3d0 0x7f1e7010adf0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3322485920 >> 172.21.15.27:0/3322485920 conn(0x7f1e70078de0 msgr2=0x7f1e7007b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3322485920 shutdown_connections 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.831+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3322485920 wait complete. 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.832+0000 7f1e77b6c640 1 Processor -- start 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.832+0000 7f1e77b6c640 1 -- start start 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.832+0000 7f1e77b6c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7010a3d0 0x7f1e701a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.832+0000 7f1e77b6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e7010b330 0x7f1e701a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.832+0000 7f1e77b6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e70110030 0x7f1e701a7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.832+0000 7f1e77b6c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e70115be0 con 0x7f1e7010a3d0 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.832+0000 7f1e77b6c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1e70115a60 con 0x7f1e70110030 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.832+0000 7f1e77b6c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1e70115d60 con 0x7f1e7010b330 2024-04-25T13:45:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7010a3d0 0x7f1e701a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7010a3d0 0x7f1e701a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45520/0 (socket says 172.21.15.27:45520) 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 -- 172.21.15.27:0/3320058309 learned_addr learned my addr 172.21.15.27:0/3320058309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75107640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e7010b330 0x7f1e701a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e76109640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e70110030 0x7f1e701a7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 -- 172.21.15.27:0/3320058309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e7010b330 msgr2=0x7f1e701a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e7010b330 0x7f1e701a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 -- 172.21.15.27:0/3320058309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e70110030 msgr2=0x7f1e701a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e70110030 0x7f1e701a7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 -- 172.21.15.27:0/3320058309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e701a83b0 con 0x7f1e7010a3d0 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75107640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e7010b330 0x7f1e701a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.833+0000 7f1e75908640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7010a3d0 0x7f1e701a0870 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0304b0 tx=0x7f1e6c032d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.835+0000 7f1e66ffd640 1 -- 172.21.15.27:0/3320058309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c037040 con 0x7f1e7010a3d0 2024-04-25T13:45:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.836+0000 7f1e66ffd640 1 -- 172.21.15.27:0/3320058309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e6c030c10 con 0x7f1e7010a3d0 2024-04-25T13:45:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.836+0000 7f1e66ffd640 1 -- 172.21.15.27:0/3320058309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c003850 con 0x7f1e7010a3d0 2024-04-25T13:45:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.836+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3320058309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e701a8640 con 0x7f1e7010a3d0 2024-04-25T13:45:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.836+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3320058309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e701a8a80 con 0x7f1e7010a3d0 2024-04-25T13:45:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.839+0000 7f1e7ea16740 1 Processor -- start 2024-04-25T13:45:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.839+0000 7f1e66ffd640 1 -- 172.21.15.27:0/3320058309 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1e6c043020 con 0x7f1e7010a3d0 2024-04-25T13:45:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.840+0000 7f1e7ea16740 1 -- start start 2024-04-25T13:45:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.840+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e0807390 0x55b4e0ba8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.840+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0ba8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.840+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0ba9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b4e0b0dd80 con 0x55b4e07de4d0 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b4e0ac78b0 con 0x55b4e07e66b0 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b4e0b09d40 con 0x55b4e0807390 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e76109640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e0807390 0x55b4e0ba8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e76109640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e0807390 0x55b4e0ba8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:35344/0 (socket says 172.21.15.27:35344) 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e76109640 1 -- 172.21.15.27:0/139220939 learned_addr learned my addr 172.21.15.27:0/139220939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e75908640 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0ba8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e75107640 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0ba9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.841+0000 7f1e66ffd640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1e5c0afc40 0x7f1e5c0b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.842+0000 7f1e76109640 1 -- 172.21.15.27:0/139220939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 msgr2=0x55b4e0ba9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.842+0000 7f1e76109640 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0ba9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.842+0000 7f1e76109640 1 -- 172.21.15.27:0/139220939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 msgr2=0x55b4e0ba8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.842+0000 7f1e76109640 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0ba8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.842+0000 7f1e76109640 1 -- 172.21.15.27:0/139220939 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4e0a98a40 con 0x55b4e0807390 2024-04-25T13:45:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.842+0000 7f1e66ffd640 1 -- 172.21.15.27:0/3320058309 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1e6c0fdf40 con 0x7f1e7010a3d0 2024-04-25T13:45:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.842+0000 7f1e76109640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1e5c0afc40 0x7f1e5c0b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.843+0000 7f1e75107640 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0ba9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:45:13.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.843+0000 7f1e76109640 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e0807390 0x55b4e0ba8a60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800e520 tx=0x7f1e6800e9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3591b09e097ea1eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:13.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.843+0000 7f1e5bfff640 1 -- 172.21.15.27:0/139220939 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6801b480 con 0x55b4e0807390 2024-04-25T13:45:13.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.843+0000 7f1e5bfff640 1 -- 172.21.15.27:0/139220939 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e6801bb50 con 0x55b4e0807390 2024-04-25T13:45:13.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.843+0000 7f1e5bfff640 1 -- 172.21.15.27:0/139220939 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6801ac30 con 0x55b4e0807390 2024-04-25T13:45:13.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 -- 172.21.15.27:0/139220939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e0807390 msgr2=0x55b4e0ba8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e0807390 0x55b4e0ba8a60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800e520 tx=0x7f1e6800e9e0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e76109640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1e5c0afc40 0x7f1e5c0b2150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1e68023000 tx=0x7f1e6801af10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 -- 172.21.15.27:0/139220939 shutdown_connections 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0ba9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0ba8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/139220939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e0807390 0x55b4e0ba8a60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 -- 172.21.15.27:0/139220939 >> 172.21.15.27:0/139220939 conn(0x55b4e07fb4e0 msgr2=0x55b4e0ba4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 -- 172.21.15.27:0/139220939 shutdown_connections 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 -- 172.21.15.27:0/139220939 wait complete. 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 -- start start 2024-04-25T13:45:13.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.845+0000 7f1e7ea16740 1 Processor -- start 2024-04-25T13:45:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.846+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e75908640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e75908640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45540/0 (socket says 172.21.15.27:45540) 2024-04-25T13:45:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e75908640 1 -- 172.21.15.27:0/1293250754 learned_addr learned my addr 172.21.15.27:0/1293250754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b4e0baa780 con 0x55b4e07de4d0 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e75107640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b4e0a94d10 con 0x55b4e0807390 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b4e0baa900 con 0x55b4e07e66b0 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.847+0000 7f1e76109640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e76109640 1 -- 172.21.15.27:0/1293250754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07e66b0 msgr2=0x55b4e0bb2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e76109640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e76109640 1 -- 172.21.15.27:0/1293250754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 msgr2=0x55b4e0bb29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e76109640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e76109640 1 -- 172.21.15.27:0/1293250754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4e08e97d0 con 0x55b4e0807390 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e75107640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e75908640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e7ea16740 10 client.0 _ll_register_callbacks cb 0x55b4e0ac1210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e6801bb50 con 0x55b4e0807390 2024-04-25T13:45:13.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e76109640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb9e40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800e520 tx=0x7f1e68017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dae1f3d6e5242d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:13.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e597fa640 10 client.25868 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:13.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e597fa640 1 -- 172.21.15.27:0/1293250754 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6802b070 con 0x55b4e0807390 2024-04-25T13:45:13.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e597fa640 1 -- 172.21.15.27:0/1293250754 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1e6801abc0 con 0x55b4e0807390 2024-04-25T13:45:13.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e597fa640 1 -- 172.21.15.27:0/1293250754 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1e68009d10 con 0x55b4e0807390 2024-04-25T13:45:13.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e7ea16740 10 client.25868 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:13.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.848+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=19}) -- 0x55b4e081f3f0 con 0x55b4e0807390 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.850+0000 7f1e597fa640 1 -- 172.21.15.27:0/1293250754 <== mon.2 v2:172.21.15.45:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1e6800b040 con 0x55b4e0807390 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.850+0000 7f1e7ea16740 10 client.25868 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.850+0000 7f1e7ea16740 10 client.25868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.850+0000 7f1e7ea16740 20 client.25868 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.850+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4e07c5f60 0x55b4e0bbffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.850+0000 7f1e7ea16740 4 client.25868 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.850+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4e0b1b990 con 0x55b4e07c5f60 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.851+0000 7f1e75908640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4e07c5f60 0x55b4e0bbffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.851+0000 7f1e75908640 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4e07c5f60 0x55b4e0bbffb0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15af534ffc505d05 server_cookie=5a38bb861c5d6b86 in_seq=0 out_seq=0 2024-04-25T13:45:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.851+0000 7f1e597fa640 10 client.25868 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:13.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e597fa640 1 -- 172.21.15.27:0/1293250754 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b4e0b1b990 con 0x55b4e07c5f60 2024-04-25T13:45:13.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e597fa640 10 client.25868 handle_command_reply: tid=0 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 client.25868 shutdown 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 msgr2=0x55b4e0bb9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb9e40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800e520 tx=0x7f1e68017000 comp rx=0 tx=0).stop 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 shutdown_connections 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4e07c5f60 0x55b4e0bbffb0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb9e40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e597fa640 0 client.25868 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/1293250754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 >> 172.21.15.27:0/1293250754 conn(0x55b4e07fb4e0 msgr2=0x55b4e0ba33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 shutdown_connections 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 1 -- 172.21.15.27:0/1293250754 wait complete. 2024-04-25T13:45:13.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.852+0000 7f1e7ea16740 20 client.25868 trim_cache size 0 max 16384 2024-04-25T13:45:13.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.878+0000 7f1e7ea16740 1 Processor -- start 2024-04-25T13:45:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.878+0000 7f1e7ea16740 1 -- start start 2024-04-25T13:45:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.878+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.878+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.878+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.878+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701af710 con 0x55b4e07e66b0 2024-04-25T13:45:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.878+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701b1f90 con 0x55b4e0807390 2024-04-25T13:45:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.878+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701afd80 con 0x55b4e07de4d0 2024-04-25T13:45:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e75107640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51348/0 (socket says 172.21.15.27:51348) 2024-04-25T13:45:13.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 -- 172.21.15.27:0/3934770873 learned_addr learned my addr 172.21.15.27:0/3934770873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:13.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e75908640 1 --2- 172.21.15.27:0/3934770873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 -- 172.21.15.27:0/3934770873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 msgr2=0x55b4e0bb0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:13.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 --2- 172.21.15.27:0/3934770873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 -- 172.21.15.27:0/3934770873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07e66b0 msgr2=0x55b4e0bb14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:13.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 --2- 172.21.15.27:0/3934770873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 -- 172.21.15.27:0/3934770873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4e07df0f0 con 0x55b4e0807390 2024-04-25T13:45:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.879+0000 7f1e76109640 1 --2- 172.21.15.27:0/3934770873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb19f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800dfe0 tx=0x7f1e68030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=159c063f964ca4ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.880+0000 7f1e5b7fe640 1 -- 172.21.15.27:0/3934770873 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6801a740 con 0x55b4e0807390 2024-04-25T13:45:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.880+0000 7f1e5b7fe640 1 -- 172.21.15.27:0/3934770873 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e6801bb50 con 0x55b4e0807390 2024-04-25T13:45:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.880+0000 7f1e5b7fe640 1 -- 172.21.15.27:0/3934770873 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68004c20 con 0x55b4e0807390 2024-04-25T13:45:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 -- 172.21.15.27:0/3934770873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 msgr2=0x55b4e0bb19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/3934770873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb19f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800dfe0 tx=0x7f1e68030000 comp rx=0 tx=0).stop 2024-04-25T13:45:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 -- 172.21.15.27:0/3934770873 shutdown_connections 2024-04-25T13:45:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/3934770873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e0807390 0x55b4e0bb19f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/3934770873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e07e66b0 0x55b4e0bb14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/3934770873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 -- 172.21.15.27:0/3934770873 >> 172.21.15.27:0/3934770873 conn(0x55b4e07fb4e0 msgr2=0x55b4e0bad680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 -- 172.21.15.27:0/3934770873 shutdown_connections 2024-04-25T13:45:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.881+0000 7f1e7ea16740 1 -- 172.21.15.27:0/3934770873 wait complete. 2024-04-25T13:45:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 1 -- start start 2024-04-25T13:45:13.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 1 Processor -- start 2024-04-25T13:45:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0bba520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e0807390 0x55b4e0bb6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701b0450 con 0x55b4e0807390 2024-04-25T13:45:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701b18c0 con 0x55b4e07e66b0 2024-04-25T13:45:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701b0b20 con 0x55b4e07de4d0 2024-04-25T13:45:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e7ea16740 10 client.0 _ll_register_callbacks cb 0x7f1e6800e450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e76109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e0807390 0x55b4e0bb6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e76109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e0807390 0x55b4e0bb6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45550/0 (socket says 172.21.15.27:45550) 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e75107640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0bba520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e76109640 1 -- 172.21.15.27:0/4225302744 learned_addr learned my addr 172.21.15.27:0/4225302744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.882+0000 7f1e75908640 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e76109640 1 -- 172.21.15.27:0/4225302744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 msgr2=0x55b4e0bb9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e76109640 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e76109640 1 -- 172.21.15.27:0/4225302744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 msgr2=0x55b4e0bba520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e76109640 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0bba520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e76109640 1 -- 172.21.15.27:0/4225302744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4e0ace4b0 con 0x55b4e0807390 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e75908640 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e75107640 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0bba520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e76109640 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e0807390 0x55b4e0bb6b60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800d880 tx=0x7f1e68005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f1f27f490ac039 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e597fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e597fa640 1 -- 172.21.15.27:0/4225302744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68009d10 con 0x55b4e0807390 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e597fa640 1 -- 172.21.15.27:0/4225302744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1e68004bf0 con 0x55b4e0807390 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e7ea16740 1 -- 172.21.15.27:0/4225302744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e6801bb50 con 0x55b4e0807390 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e597fa640 1 -- 172.21.15.27:0/4225302744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1e68004e30 con 0x55b4e0807390 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e7ea16740 10 client.16170 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.883+0000 7f1e7ea16740 1 -- 172.21.15.27:0/4225302744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b4e0c26d90 con 0x55b4e0807390 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e597fa640 1 -- 172.21.15.27:0/4225302744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e6801bb50 con 0x55b4e0807390 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e597fa640 1 -- 172.21.15.27:0/4225302744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1e68004020 con 0x55b4e0807390 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e7ea16740 10 client.16170 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e7ea16740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e7ea16740 20 client.16170 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4e079b3a0 0x55b4e0c29760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e7ea16740 4 client.16170 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e7ea16740 1 -- 172.21.15.27:0/4225302744 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b4e0b15800 con 0x55b4e079b3a0 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e75908640 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4e079b3a0 0x55b4e0c29760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e597fa640 10 client.16170 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.884+0000 7f1e75908640 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4e079b3a0 0x55b4e0c29760 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e9d97b2de2fc9fb server_cookie=9ad63047dca1372e in_seq=0 out_seq=0 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.885+0000 7f1e597fa640 1 -- 172.21.15.27:0/4225302744 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b4e0b15800 con 0x55b4e079b3a0 2024-04-25T13:45:13.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.885+0000 7f1e597fa640 10 client.16170 handle_command_reply: tid=0 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.885+0000 7f1e7ea16740 1 client.16170 shutdown 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.885+0000 7f1e7ea16740 1 -- 172.21.15.27:0/4225302744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e0807390 msgr2=0x55b4e0bb6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.885+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e0807390 0x55b4e0bb6b60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800d880 tx=0x7f1e68005cf0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.886+0000 7f1e7ea16740 1 -- 172.21.15.27:0/4225302744 shutdown_connections 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.886+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4e079b3a0 0x55b4e0c29760 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.886+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4e0807390 0x55b4e0bb6b60 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.886+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4e07e66b0 0x55b4e0bba520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.886+0000 7f1e7ea16740 1 --2- 172.21.15.27:0/4225302744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4e07de4d0 0x55b4e0bb9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.886+0000 7f1e7ea16740 1 -- 172.21.15.27:0/4225302744 >> 172.21.15.27:0/4225302744 conn(0x55b4e07fb4e0 msgr2=0x55b4e0bb0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.886+0000 7f1e597fa640 0 client.16170 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.887+0000 7f1e7ea16740 1 -- 172.21.15.27:0/4225302744 shutdown_connections 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.887+0000 7f1e7ea16740 1 -- 172.21.15.27:0/4225302744 wait complete. 2024-04-25T13:45:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.887+0000 7f1e7ea16740 20 client.16170 trim_cache size 0 max 16384 2024-04-25T13:45:13.897 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:45:13.897 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:45:13.897 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:45:13.897 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1e5c0afc40 msgr2=0x7f1e5c0b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1e5c0afc40 0x7f1e5c0b2150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1e68023000 tx=0x7f1e6801af10 comp rx=0 tx=0).stop 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7010a3d0 msgr2=0x7f1e701a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7010a3d0 0x7f1e701a0870 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0304b0 tx=0x7f1e6c032d20 comp rx=0 tx=0).stop 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3320058309 shutdown_connections 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1e5c0afc40 0x7f1e5c0b2150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e70110030 0x7f1e701a7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e7010b330 0x7f1e701a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 --2- 172.21.15.27:0/3320058309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7010a3d0 0x7f1e701a0870 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3320058309 >> 172.21.15.27:0/3320058309 conn(0x7f1e70078de0 msgr2=0x7f1e7010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3320058309 shutdown_connections 2024-04-25T13:45:13.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:13.888+0000 7f1e77b6c640 1 -- 172.21.15.27:0/3320058309 wait complete. 2024-04-25T13:45:13.903 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:45:13.903 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:45:14.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:13 smithi122 ceph-mon[32843]: pgmap v479: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 780 op/s 2024-04-25T13:45:14.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:13 smithi045 ceph-mon[32731]: pgmap v479: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 780 op/s 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 -- 172.21.15.27:0/717278840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 msgr2=0x7fbac8104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 --2- 172.21.15.27:0/717278840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 0x7fbac8104010 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009900 tx=0x7fbabc02a410 comp rx=0 tx=0).stop 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 -- 172.21.15.27:0/717278840 shutdown_connections 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 --2- 172.21.15.27:0/717278840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac8108900 0x7fbac810ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 --2- 172.21.15.27:0/717278840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 0x7fbac8104010 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 --2- 172.21.15.27:0/717278840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac81029c0 0x7fbac81036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 -- 172.21.15.27:0/717278840 >> 172.21.15.27:0/717278840 conn(0x7fbac80fdbd0 msgr2=0x7fbac8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 -- 172.21.15.27:0/717278840 shutdown_connections 2024-04-25T13:45:14.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.079+0000 7fbacdc05640 1 -- 172.21.15.27:0/717278840 wait complete. 2024-04-25T13:45:14.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.080+0000 7fbacdc05640 1 Processor -- start 2024-04-25T13:45:14.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.080+0000 7fbacdc05640 1 -- start start 2024-04-25T13:45:14.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.080+0000 7fbacdc05640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac81029c0 0x7fbac819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.080+0000 7fbacdc05640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 0x7fbac819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.080+0000 7fbacdc05640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac8108900 0x7fbac81a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbacdc05640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbac810e4f0 con 0x7fbac8103c00 2024-04-25T13:45:14.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 0x7fbac819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 0x7fbac819a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45576/0 (socket says 172.21.15.27:45576) 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 -- 172.21.15.27:0/3056739347 learned_addr learned my addr 172.21.15.27:0/3056739347 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac77fe640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac81029c0 0x7fbac819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbacdc05640 1 -- 172.21.15.27:0/3056739347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbac810e370 con 0x7fbac8108900 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbacdc05640 1 -- 172.21.15.27:0/3056739347 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbac810e670 con 0x7fbac81029c0 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac7fff640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac8108900 0x7fbac81a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 -- 172.21.15.27:0/3056739347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac81029c0 msgr2=0x7fbac819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac81029c0 0x7fbac819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 -- 172.21.15.27:0/3056739347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac8108900 msgr2=0x7fbac81a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac8108900 0x7fbac81a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 -- 172.21.15.27:0/3056739347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac81a1d30 con 0x7fbac8103c00 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac77fe640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac81029c0 0x7fbac819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac7fff640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac8108900 0x7fbac81a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.081+0000 7fbac6ffd640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 0x7fbac819a730 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbabc006010 tx=0x7fbabc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:14.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.082+0000 7fbac4ff9640 1 -- 172.21.15.27:0/3056739347 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc03f030 con 0x7fbac8103c00 2024-04-25T13:45:14.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.082+0000 7fbac4ff9640 1 -- 172.21.15.27:0/3056739347 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbabc031910 con 0x7fbac8103c00 2024-04-25T13:45:14.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.082+0000 7fbac4ff9640 1 -- 172.21.15.27:0/3056739347 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbabc035070 con 0x7fbac8103c00 2024-04-25T13:45:14.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.082+0000 7fbacdc05640 1 -- 172.21.15.27:0/3056739347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac81a1fc0 con 0x7fbac8103c00 2024-04-25T13:45:14.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.082+0000 7fbacdc05640 1 -- 172.21.15.27:0/3056739347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac81a2420 con 0x7fbac8103c00 2024-04-25T13:45:14.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.083+0000 7fbaae7fc640 1 -- 172.21.15.27:0/3056739347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa8005290 con 0x7fbac8103c00 2024-04-25T13:45:14.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.086+0000 7fbac4ff9640 1 -- 172.21.15.27:0/3056739347 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbabc042030 con 0x7fbac8103c00 2024-04-25T13:45:14.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.087+0000 7fbac4ff9640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afbf0 0x7fbab00b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.087+0000 7fbac4ff9640 1 -- 172.21.15.27:0/3056739347 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbabc0ff7a0 con 0x7fbac8103c00 2024-04-25T13:45:14.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.088+0000 7fbac77fe640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afbf0 0x7fbab00b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.089+0000 7fbac77fe640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afbf0 0x7fbab00b2100 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbac81035c0 tx=0x7fbab8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:14.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.091+0000 7fbac4ff9640 1 -- 172.21.15.27:0/3056739347 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbabc0c8350 con 0x7fbac8103c00 2024-04-25T13:45:14.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:13 smithi027 ceph-mon[29140]: pgmap v479: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 780 op/s 2024-04-25T13:45:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.482+0000 7fbaae7fc640 1 -- 172.21.15.27:0/3056739347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbaa8005580 con 0x7fbac8103c00 2024-04-25T13:45:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.483+0000 7fbac4ff9640 1 -- 172.21.15.27:0/3056739347 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fbabc0c7c50 con 0x7fbac8103c00 2024-04-25T13:45:14.483 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:45:14.483 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:45:14.483 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:45:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 -- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afbf0 msgr2=0x7fbab00b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afbf0 0x7fbab00b2100 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbac81035c0 tx=0x7fbab8005f90 comp rx=0 tx=0).stop 2024-04-25T13:45:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 -- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 msgr2=0x7fbac819a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 0x7fbac819a730 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbabc006010 tx=0x7fbabc0042f0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 -- 172.21.15.27:0/3056739347 shutdown_connections 2024-04-25T13:45:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbab00afbf0 0x7fbab00b2100 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbac8108900 0x7fbac81a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac8103c00 0x7fbac819a730 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 --2- 172.21.15.27:0/3056739347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbac81029c0 0x7fbac819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.486+0000 7fbaae7fc640 1 -- 172.21.15.27:0/3056739347 >> 172.21.15.27:0/3056739347 conn(0x7fbac80fdbd0 msgr2=0x7fbac80ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.487+0000 7fbaae7fc640 1 -- 172.21.15.27:0/3056739347 shutdown_connections 2024-04-25T13:45:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.487+0000 7fbaae7fc640 1 -- 172.21.15.27:0/3056739347 wait complete. 2024-04-25T13:45:14.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:45:14.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 -- 172.21.15.27:0/3378733354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0001029f0 msgr2=0x7fd0001036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 --2- 172.21.15.27:0/3378733354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0001029f0 0x7fd0001036f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcff0009900 tx=0x7fcff002a410 comp rx=0 tx=0).stop 2024-04-25T13:45:14.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 -- 172.21.15.27:0/3378733354 shutdown_connections 2024-04-25T13:45:14.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 --2- 172.21.15.27:0/3378733354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd000108930 0x7fd00010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 --2- 172.21.15.27:0/3378733354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd000103c30 0x7fd000104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 --2- 172.21.15.27:0/3378733354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0001029f0 0x7fd0001036f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 -- 172.21.15.27:0/3378733354 >> 172.21.15.27:0/3378733354 conn(0x7fd0000fdbc0 msgr2=0x7fd000100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 -- 172.21.15.27:0/3378733354 shutdown_connections 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.676+0000 7fd006496640 1 -- 172.21.15.27:0/3378733354 wait complete. 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd006496640 1 Processor -- start 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd006496640 1 -- start start 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd006496640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0001029f0 0x7fd00019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd006496640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd000103c30 0x7fd00019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd006496640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd000108930 0x7fd0001a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd006496640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd0001118b0 con 0x7fd000103c30 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd006496640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd00010d280 con 0x7fd0001029f0 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd006496640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd000111a30 con 0x7fd000108930 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd000103c30 0x7fd00019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd000103c30 0x7fd00019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45596/0 (socket says 172.21.15.27:45596) 2024-04-25T13:45:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fd004a33640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd000108930 0x7fd0001a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2729533333 learned_addr learned my addr 172.21.15.27:0/2729533333 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.677+0000 7fcffffff640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0001029f0 0x7fd00019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.678+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2729533333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd000108930 msgr2=0x7fd0001a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.678+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd000108930 0x7fd0001a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.678+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2729533333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0001029f0 msgr2=0x7fd00019c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.678+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0001029f0 0x7fd00019c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.678+0000 7fcfff7fe640 1 -- 172.21.15.27:0/2729533333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0001a3f60 con 0x7fd000103c30 2024-04-25T13:45:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.678+0000 7fd004a33640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd000108930 0x7fd0001a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.679+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd000103c30 0x7fd00019c960 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9f0 tx=0x7fcff400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.680+0000 7fcffd7fa640 1 -- 172.21.15.27:0/2729533333 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff400cd80 con 0x7fd000103c30 2024-04-25T13:45:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.680+0000 7fcffd7fa640 1 -- 172.21.15.27:0/2729533333 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff40108e0 con 0x7fd000103c30 2024-04-25T13:45:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.680+0000 7fcffd7fa640 1 -- 172.21.15.27:0/2729533333 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4018980 con 0x7fd000103c30 2024-04-25T13:45:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.680+0000 7fd006496640 1 -- 172.21.15.27:0/2729533333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0001a4250 con 0x7fd000103c30 2024-04-25T13:45:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.680+0000 7fd006496640 1 -- 172.21.15.27:0/2729533333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0001a4750 con 0x7fd000103c30 2024-04-25T13:45:14.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.684+0000 7fd00d424740 1 Processor -- start 2024-04-25T13:45:14.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.684+0000 7fcffd7fa640 1 -- 172.21.15.27:0/2729533333 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcff4010430 con 0x7fd000103c30 2024-04-25T13:45:14.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.684+0000 7fd00d424740 1 -- start start 2024-04-25T13:45:14.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fd00d424740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906cb8390 0x55d907063020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fd00d424740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906c8f4d0 0x55d9070635a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fd00d424740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c976b0 0x55d907064230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c976b0 0x55d907064230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcffffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906c8f4d0 0x55d9070635a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcffffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906c8f4d0 0x55d9070635a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51392/0 (socket says 172.21.15.27:51392) 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fd004a33640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906cb8390 0x55d907063020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcffffff640 1 -- 172.21.15.27:0/3885305353 learned_addr learned my addr 172.21.15.27:0/3885305353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c976b0 0x55d907064230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45602/0 (socket says 172.21.15.27:45602) 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fd00d424740 1 -- 172.21.15.27:0/3885305353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d906fbed80 con 0x55d906c976b0 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fd00d424740 1 -- 172.21.15.27:0/3885305353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d906f78a20 con 0x55d906c8f4d0 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fd00d424740 1 -- 172.21.15.27:0/3885305353 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d906fbad40 con 0x55d906cb8390 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcffffff640 1 -- 172.21.15.27:0/3885305353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906cb8390 msgr2=0x55d907063020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcffffff640 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906cb8390 0x55d907063020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcffffff640 1 -- 172.21.15.27:0/3885305353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c976b0 msgr2=0x55d907064230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcffffff640 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c976b0 0x55d907064230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcffffff640 1 -- 172.21.15.27:0/3885305353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d906f83c30 con 0x55d906c8f4d0 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fd004a33640 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906cb8390 0x55d907063020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.685+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c976b0 0x55d907064230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.686+0000 7fcffffff640 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906c8f4d0 0x55d9070635a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcff0009230 tx=0x7fcff0009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=324e34690f6d9840 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.686+0000 7fcffd7fa640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfe80afcc0 0x7fcfe80b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.686+0000 7fd004a33640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfe80afcc0 0x7fcfe80b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.686+0000 7fcfe67fc640 1 -- 172.21.15.27:0/3885305353 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0035030 con 0x55d906c8f4d0 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.686+0000 7fcffd7fa640 1 -- 172.21.15.27:0/2729533333 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcff40dee60 con 0x7fd000103c30 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.686+0000 7fcfe67fc640 1 -- 172.21.15.27:0/3885305353 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff0040070 con 0x55d906c8f4d0 2024-04-25T13:45:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.686+0000 7fcfe67fc640 1 -- 172.21.15.27:0/3885305353 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0003970 con 0x55d906c8f4d0 2024-04-25T13:45:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd004a33640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfe80afcc0 0x7fcfe80b21d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009fd0 tx=0x7fcff8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd00d424740 1 -- 172.21.15.27:0/3885305353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906c8f4d0 msgr2=0x55d9070635a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd00d424740 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906c8f4d0 0x55d9070635a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcff0009230 tx=0x7fcff0009620 comp rx=0 tx=0).stop 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd00d424740 1 -- 172.21.15.27:0/3885305353 shutdown_connections 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd00d424740 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c976b0 0x55d907064230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd00d424740 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906c8f4d0 0x55d9070635a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd00d424740 1 --2- 172.21.15.27:0/3885305353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906cb8390 0x55d907063020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd00d424740 1 -- 172.21.15.27:0/3885305353 >> 172.21.15.27:0/3885305353 conn(0x55d906cac4e0 msgr2=0x55d907056020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.688+0000 7fd00d424740 1 -- 172.21.15.27:0/3885305353 shutdown_connections 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.689+0000 7fd00d424740 1 -- 172.21.15.27:0/3885305353 wait complete. 2024-04-25T13:45:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.689+0000 7fd00d424740 1 -- start start 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.689+0000 7fd00d424740 1 Processor -- start 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fd00d424740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fd00d424740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d90706d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fd00d424740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d907074210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fd00d424740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d907064ce0 con 0x55d906c8f4d0 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fd00d424740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d906f45dc0 con 0x55d906cb8390 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fd00d424740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d907064e60 con 0x55d906c976b0 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcffffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcffffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45614/0 (socket says 172.21.15.27:45614) 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcffffff640 1 -- 172.21.15.27:0/1649254059 learned_addr learned my addr 172.21.15.27:0/1649254059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fd00d424740 10 client.0 _ll_register_callbacks cb 0x55d906f61600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d90706d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fd004a33640 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d907074210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcffffff640 1 -- 172.21.15.27:0/1649254059 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 msgr2=0x55d90706d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcffffff640 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d90706d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcffffff640 1 -- 172.21.15.27:0/1649254059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 msgr2=0x55d907074210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcffffff640 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d907074210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.690+0000 7fcffffff640 1 -- 172.21.15.27:0/1649254059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d906cd03f0 con 0x55d906c8f4d0 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.691+0000 7fcffffff640 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706cd70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcff0005ab0 tx=0x7fcff0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a684a37e0216f1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.691+0000 7fcfdf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.691+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/1649254059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0053060 con 0x55d906c8f4d0 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.691+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/1649254059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcff00036f0 con 0x55d906c8f4d0 2024-04-25T13:45:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.691+0000 7fd00d424740 1 -- 172.21.15.27:0/1649254059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcff0040070 con 0x55d906c8f4d0 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.692+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/1649254059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcff0035070 con 0x55d906c8f4d0 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.692+0000 7fd00d424740 10 client.16185 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.692+0000 7fd00d424740 1 -- 172.21.15.27:0/1649254059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d906d9a7d0 con 0x55d906c8f4d0 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/1649254059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff0040070 con 0x55d906c8f4d0 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/1649254059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcff004d030 con 0x55d906c8f4d0 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fd00d424740 10 client.16185 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fd00d424740 10 client.16185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fd00d424740 20 client.16185 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:14.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fd00d424740 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d906c76f60 0x55d90707a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fd00d424740 4 client.16185 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fd00d424740 1 -- 172.21.15.27:0/1649254059 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d906fcc990 con 0x55d906c76f60 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.693+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d906c76f60 0x55d90707a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.694+0000 7fcfdf7fe640 10 client.16185 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.694+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d906c76f60 0x55d90707a520 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8b705758cb685b8 server_cookie=b144e41afea683d4 in_seq=0 out_seq=0 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.694+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/1649254059 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d906fcc990 con 0x55d906c76f60 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.694+0000 7fcfdf7fe640 10 client.16185 handle_command_reply: tid=0 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.694+0000 7fd00d424740 1 client.16185 shutdown 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 -- 172.21.15.27:0/1649254059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 msgr2=0x55d90706cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706cd70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcff0005ab0 tx=0x7fcff0004270 comp rx=0 tx=0).stop 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 -- 172.21.15.27:0/1649254059 shutdown_connections 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d906c76f60 0x55d90707a520 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d907074210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d90706d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 --2- 172.21.15.27:0/1649254059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706cd70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fcfdf7fe640 0 client.16185 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 -- 172.21.15.27:0/1649254059 >> 172.21.15.27:0/1649254059 conn(0x55d906cac4e0 msgr2=0x55d907058f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 -- 172.21.15.27:0/1649254059 shutdown_connections 2024-04-25T13:45:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 1 -- 172.21.15.27:0/1649254059 wait complete. 2024-04-25T13:45:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.695+0000 7fd00d424740 20 client.16185 trim_cache size 0 max 16384 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd00d424740 1 Processor -- start 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd00d424740 1 -- start start 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd00d424740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd00d424740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d907074130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd00d424740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d907054610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd00d424740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd0001a7020 con 0x55d906c8f4d0 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd00d424740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0001a98a0 con 0x55d906cb8390 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd00d424740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd0001a7690 con 0x55d906c976b0 2024-04-25T13:45:14.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fcffffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d907074130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fcfff7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d907074130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:35450/0 (socket says 172.21.15.27:35450) 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fcffffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45622/0 (socket says 172.21.15.27:45622) 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fcfff7fe640 1 -- 172.21.15.27:0/688274522 learned_addr learned my addr 172.21.15.27:0/688274522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.715+0000 7fd004a33640 1 --2- 172.21.15.27:0/688274522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d907054610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfff7fe640 1 -- 172.21.15.27:0/688274522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 msgr2=0x55d907054610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/688274522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d907054610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfff7fe640 1 -- 172.21.15.27:0/688274522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 msgr2=0x55d90706d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/688274522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfff7fe640 1 -- 172.21.15.27:0/688274522 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d90706d590 con 0x55d906c976b0 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/688274522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d907074130 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcff40a7350 tx=0x7fcff40dc800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4fe60f67d28f199 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfe5ffb640 1 -- 172.21.15.27:0/688274522 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff40b0170 con 0x55d906c976b0 2024-04-25T13:45:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfe5ffb640 1 -- 172.21.15.27:0/688274522 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff40a39b0 con 0x55d906c976b0 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.716+0000 7fcfe5ffb640 1 -- 172.21.15.27:0/688274522 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4018d80 con 0x55d906c976b0 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 -- 172.21.15.27:0/688274522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 msgr2=0x55d907074130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 --2- 172.21.15.27:0/688274522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d907074130 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcff40a7350 tx=0x7fcff40dc800 comp rx=0 tx=0).stop 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 -- 172.21.15.27:0/688274522 shutdown_connections 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 --2- 172.21.15.27:0/688274522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d907054610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 --2- 172.21.15.27:0/688274522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d907074130 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 --2- 172.21.15.27:0/688274522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d90706d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 -- 172.21.15.27:0/688274522 >> 172.21.15.27:0/688274522 conn(0x55d906cac4e0 msgr2=0x55d90706c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 -- 172.21.15.27:0/688274522 shutdown_connections 2024-04-25T13:45:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 -- 172.21.15.27:0/688274522 wait complete. 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.717+0000 7fd00d424740 1 -- start start 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.718+0000 7fd00d424740 1 Processor -- start 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.718+0000 7fd00d424740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d907071280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fcffffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d907071280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fcffffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d907071280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45628/0 (socket says 172.21.15.27:45628) 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fd00d424740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d9070717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fcffffff640 1 -- 172.21.15.27:0/855534439 learned_addr learned my addr 172.21.15.27:0/855534439 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d9070717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fd00d424740 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d9070726f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd0001a7d60 con 0x55d906c8f4d0 2024-04-25T13:45:14.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0001a91d0 con 0x55d906cb8390 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd0001a8430 con 0x55d906c976b0 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fd00d424740 10 client.0 _ll_register_callbacks cb 0x7fcff0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.719+0000 7fd004a33640 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d9070726f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.720+0000 7fcfff7fe640 1 -- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 msgr2=0x55d9070726f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.720+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d9070726f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.720+0000 7fcfff7fe640 1 -- 172.21.15.27:0/855534439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 msgr2=0x55d907071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.720+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d907071280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.720+0000 7fcfff7fe640 1 -- 172.21.15.27:0/855534439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d907072cb0 con 0x55d906c976b0 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.720+0000 7fcfff7fe640 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d9070717c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcff40a4630 tx=0x7fcff401f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7c04d6965361642 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.720+0000 7fcfdf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:14.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.723+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/855534439 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff40b0170 con 0x55d906c976b0 2024-04-25T13:45:14.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.724+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/855534439 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcff40a1b50 con 0x55d906c976b0 2024-04-25T13:45:14.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.724+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcff40a39b0 con 0x55d906c976b0 2024-04-25T13:45:14.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.725+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/855534439 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcff40a1d90 con 0x55d906c976b0 2024-04-25T13:45:14.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.725+0000 7fd00d424740 10 client.25921 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:14.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.725+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d907073f30 con 0x55d906c976b0 2024-04-25T13:45:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/855534439 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff40bc200 con 0x55d906c976b0 2024-04-25T13:45:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/855534439 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcff40ab170 con 0x55d906c976b0 2024-04-25T13:45:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fd00d424740 10 client.25921 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fd00d424740 10 client.25921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fd00d424740 20 client.25921 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fd00d424740 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d906c4c3a0 0x55d9070e4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fd00d424740 4 client.25921 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:45:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fcff40a39b0 con 0x55d906c4c3a0 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.726+0000 7fcffffff640 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d906c4c3a0 0x55d9070e4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fcfdf7fe640 10 client.25921 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fcffffff640 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d906c4c3a0 0x55d9070e4990 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96562d998e41209f server_cookie=c86e746c75848af2 in_seq=0 out_seq=0 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fcfdf7fe640 1 -- 172.21.15.27:0/855534439 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fcff40a39b0 con 0x55d906c4c3a0 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fcfdf7fe640 10 client.25921 handle_command_reply: tid=0 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 client.25921 shutdown 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 msgr2=0x55d9070717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d9070717c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcff40a4630 tx=0x7fcff401f000 comp rx=0 tx=0).stop 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 shutdown_connections 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d906c4c3a0 0x55d9070e4990 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d906cb8390 0x55d9070726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d906c976b0 0x55d9070717c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 --2- 172.21.15.27:0/855534439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d906c8f4d0 0x55d907071280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fcfdf7fe640 0 client.25921 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 >> 172.21.15.27:0/855534439 conn(0x55d906cac4e0 msgr2=0x55d907054610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:14.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 shutdown_connections 2024-04-25T13:45:14.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 1 -- 172.21.15.27:0/855534439 wait complete. 2024-04-25T13:45:14.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.727+0000 7fd00d424740 20 client.25921 trim_cache size 0 max 16384 2024-04-25T13:45:14.729 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:45:14.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.728+0000 7fd006496640 1 -- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfe80afcc0 msgr2=0x7fcfe80b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.728+0000 7fd006496640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfe80afcc0 0x7fcfe80b21d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009fd0 tx=0x7fcff8009500 comp rx=0 tx=0).stop 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.728+0000 7fd006496640 1 -- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd000103c30 msgr2=0x7fd00019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.728+0000 7fd006496640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd000103c30 0x7fd00019c960 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9f0 tx=0x7fcff400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.730+0000 7fd006496640 1 -- 172.21.15.27:0/2729533333 shutdown_connections 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.730+0000 7fd006496640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfe80afcc0 0x7fcfe80b21d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.730+0000 7fd006496640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd000108930 0x7fd0001a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.730+0000 7fd006496640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd000103c30 0x7fd00019c960 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.730+0000 7fd006496640 1 --2- 172.21.15.27:0/2729533333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0001029f0 0x7fd00019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.730+0000 7fd006496640 1 -- 172.21.15.27:0/2729533333 >> 172.21.15.27:0/2729533333 conn(0x7fd0000fdbc0 msgr2=0x7fd000104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.730+0000 7fd006496640 1 -- 172.21.15.27:0/2729533333 shutdown_connections 2024-04-25T13:45:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:14.730+0000 7fd006496640 1 -- 172.21.15.27:0/2729533333 wait complete. 2024-04-25T13:45:14.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:14 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3056739347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:15.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:14 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3056739347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:15.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:14 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3056739347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:15.745 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:45:15.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:45:15.745 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:45:15.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.928+0000 7ff9837fe640 1 -- 172.21.15.27:0/2753898773 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9740318a0 con 0x7ff98410b310 2024-04-25T13:45:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.929+0000 7ff98ac51640 1 -- 172.21.15.27:0/2753898773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98410b310 msgr2=0x7ff98410b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.929+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2753898773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98410b310 0x7ff98410b7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff974009900 tx=0x7ff97402a410 comp rx=0 tx=0).stop 2024-04-25T13:45:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.929+0000 7ff98ac51640 1 -- 172.21.15.27:0/2753898773 shutdown_connections 2024-04-25T13:45:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.929+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2753898773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff98406b620 0x7ff98406bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.929+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2753898773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98410b310 0x7ff98410b7a0 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff974009900 tx=0x7ff97402a410 comp rx=0 tx=0).stop 2024-04-25T13:45:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.929+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2753898773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff98410c890 0x7ff98410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.929+0000 7ff98ac51640 1 -- 172.21.15.27:0/2753898773 >> 172.21.15.27:0/2753898773 conn(0x7ff984071600 msgr2=0x7ff984073ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.931+0000 7ff98ac51640 1 -- 172.21.15.27:0/2753898773 shutdown_connections 2024-04-25T13:45:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.931+0000 7ff98ac51640 1 -- 172.21.15.27:0/2753898773 wait complete. 2024-04-25T13:45:15.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.932+0000 7ff98ac51640 1 Processor -- start 2024-04-25T13:45:15.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.932+0000 7ff98ac51640 1 -- start start 2024-04-25T13:45:15.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.932+0000 7ff98ac51640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98406b620 0x7ff9841a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.932+0000 7ff98ac51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff98410c890 0x7ff9841a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.932+0000 7ff98ac51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9841aa2c0 0x7ff9841ac6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.932+0000 7ff98ac51640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff98411a130 con 0x7ff9841aa2c0 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.932+0000 7ff98ac51640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff984119fb0 con 0x7ff98410c890 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.932+0000 7ff98ac51640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff98411a2b0 con 0x7ff98406b620 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff98410c890 0x7ff9841a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98406b620 0x7ff9841a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98406b620 0x7ff9841a4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:35472/0 (socket says 172.21.15.27:35472) 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff98410c890 0x7ff9841a53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51426/0 (socket says 172.21.15.27:51426) 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 -- 172.21.15.27:0/2858497958 learned_addr learned my addr 172.21.15.27:0/2858497958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff9891ee640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9841aa2c0 0x7ff9841ac6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 -- 172.21.15.27:0/2858497958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98406b620 msgr2=0x7ff9841a4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98406b620 0x7ff9841a4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 -- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9841aa2c0 msgr2=0x7ff9841ac6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9841aa2c0 0x7ff9841ac6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 -- 172.21.15.27:0/2858497958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9841ace10 con 0x7ff98410c890 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff983fff640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff98410c890 0x7ff9841a53c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff97402aa00 tx=0x7ff974003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.933+0000 7ff981ffb640 1 -- 172.21.15.27:0/2858497958 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97403f020 con 0x7ff98410c890 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.934+0000 7ff981ffb640 1 -- 172.21.15.27:0/2858497958 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff974031c60 con 0x7ff98410c890 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.934+0000 7ff98ac51640 1 -- 172.21.15.27:0/2858497958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9841ad0a0 con 0x7ff98410c890 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.934+0000 7ff981ffb640 1 -- 172.21.15.27:0/2858497958 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974035070 con 0x7ff98410c890 2024-04-25T13:45:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.934+0000 7ff98ac51640 1 -- 172.21.15.27:0/2858497958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9841ad540 con 0x7ff98410c890 2024-04-25T13:45:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.937+0000 7ff981ffb640 1 -- 172.21.15.27:0/2858497958 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff974042020 con 0x7ff98410c890 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.937+0000 7ff991be5740 1 Processor -- start 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.937+0000 7ff991be5740 1 -- start start 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.937+0000 7ff991be5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x5583227044a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.937+0000 7ff991be5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55832233a550 0x558322704a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.937+0000 7ff991be5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x55832270b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.937+0000 7ff991be5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558322669b00 con 0x558322363410 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.937+0000 7ff991be5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558322605920 con 0x55832233a550 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff991be5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558322665ac0 con 0x558322342730 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff983fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x55832270b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55832233a550 0x558322704a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9891ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x5583227044a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55832233a550 0x558322704a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51434/0 (socket says 172.21.15.27:51434) 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 -- 172.21.15.27:0/3763213346 learned_addr learned my addr 172.21.15.27:0/3763213346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 -- 172.21.15.27:0/3763213346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 msgr2=0x55832270b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x55832270b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 -- 172.21.15.27:0/3763213346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 msgr2=0x5583227044a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x5583227044a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 -- 172.21.15.27:0/3763213346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583224437e0 con 0x55832233a550 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff981ffb640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff96c0afaf0 0x7ff96c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff983fff640 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x55832270b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9891ee640 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x5583227044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9889ed640 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55832233a550 0x558322704a20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff97800bc30 tx=0x7ff9780107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28b82753015b840f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff981ffb640 1 -- 172.21.15.27:0/2858497958 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff9740ff240 con 0x7ff98410c890 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.938+0000 7ff9891ee640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff96c0afaf0 0x7ff96c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.939+0000 7ff96affd640 1 -- 172.21.15.27:0/3763213346 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97801c480 con 0x55832233a550 2024-04-25T13:45:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.939+0000 7ff96affd640 1 -- 172.21.15.27:0/3763213346 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97801cad0 con 0x55832233a550 2024-04-25T13:45:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.939+0000 7ff96affd640 1 -- 172.21.15.27:0/3763213346 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978011b50 con 0x55832233a550 2024-04-25T13:45:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.940+0000 7ff991be5740 1 -- 172.21.15.27:0/3763213346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55832233a550 msgr2=0x558322704a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.940+0000 7ff991be5740 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55832233a550 0x558322704a20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff97800bc30 tx=0x7ff9780107a0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.941+0000 7ff991be5740 1 -- 172.21.15.27:0/3763213346 shutdown_connections 2024-04-25T13:45:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.941+0000 7ff991be5740 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x55832270b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.941+0000 7ff991be5740 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55832233a550 0x558322704a20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.941+0000 7ff991be5740 1 --2- 172.21.15.27:0/3763213346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x5583227044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.941+0000 7ff991be5740 1 -- 172.21.15.27:0/3763213346 >> 172.21.15.27:0/3763213346 conn(0x558322357560 msgr2=0x5583226ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.941+0000 7ff9891ee640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff96c0afaf0 0x7ff96c0b2000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x558322705670 tx=0x7ff97c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.942+0000 7ff991be5740 1 -- 172.21.15.27:0/3763213346 shutdown_connections 2024-04-25T13:45:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.942+0000 7ff991be5740 1 -- 172.21.15.27:0/3763213346 wait complete. 2024-04-25T13:45:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.942+0000 7ff991be5740 1 -- start start 2024-04-25T13:45:15.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.942+0000 7ff991be5740 1 Processor -- start 2024-04-25T13:45:15.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff991be5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55832233a550 0x55832270e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff991be5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x55832270e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55832233a550 0x55832270e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55832233a550 0x55832270e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:35490/0 (socket says 172.21.15.27:35490) 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff991be5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x558322715740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff983fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x55832270e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff983fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x55832270e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51436/0 (socket says 172.21.15.27:51436) 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff9889ed640 1 -- 172.21.15.27:0/1654760217 learned_addr learned my addr 172.21.15.27:0/1654760217 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55832270c370 con 0x558322363410 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5583226035d0 con 0x558322342730 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.943+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55832270c4f0 con 0x55832233a550 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff991be5740 10 client.0 _ll_register_callbacks cb 0x5583225f1080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff9891ee640 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x558322715740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff9889ed640 1 -- 172.21.15.27:0/1654760217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 msgr2=0x55832270e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:15.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff9889ed640 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x55832270e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff9889ed640 1 -- 172.21.15.27:0/1654760217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 msgr2=0x558322715740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff9889ed640 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x558322715740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff9889ed640 1 -- 172.21.15.27:0/1654760217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583224a3f70 con 0x55832233a550 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff9889ed640 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55832233a550 0x55832270e2a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff97800bc00 tx=0x7ff97800bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83e08620838bb9b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff963fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff963fff640 1 -- 172.21.15.27:0/1654760217 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97802b060 con 0x55832233a550 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff963fff640 1 -- 172.21.15.27:0/1654760217 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff97801c690 con 0x55832233a550 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.944+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff97801cad0 con 0x55832233a550 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff963fff640 1 -- 172.21.15.27:0/1654760217 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff97801c890 con 0x55832233a550 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff991be5740 10 client.25933 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:15.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55832233b170 con 0x55832233a550 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff963fff640 1 -- 172.21.15.27:0/1654760217 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff978011d40 con 0x55832233a550 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff963fff640 1 -- 172.21.15.27:0/1654760217 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff9780046c0 con 0x55832233a550 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff991be5740 10 client.25933 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff991be5740 10 client.25933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff991be5740 20 client.25933 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff991be5740 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5583223221f0 0x55832271b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff991be5740 4 client.25933 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff97801cad0 con 0x5583223221f0 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.945+0000 7ff983fff640 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5583223221f0 0x55832271b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.946+0000 7ff963fff640 10 client.25933 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.946+0000 7ff983fff640 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5583223221f0 0x55832271b8f0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=578bc773d57963ee server_cookie=6d2cd220fe7d58c5 in_seq=0 out_seq=0 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff963fff640 1 -- 172.21.15.27:0/1654760217 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7ff97801cad0 con 0x5583223221f0 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff963fff640 10 client.25933 handle_command_reply: tid=0 2024-04-25T13:45:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 client.25933 shutdown 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55832233a550 msgr2=0x55832270e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55832233a550 0x55832270e2a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff97800bc00 tx=0x7ff97800bf50 comp rx=0 tx=0).stop 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 shutdown_connections 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5583223221f0 0x55832271b8f0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558322363410 0x558322715740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff963fff640 0 client.25933 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x55832270e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 --2- 172.21.15.27:0/1654760217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55832233a550 0x55832270e2a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 >> 172.21.15.27:0/1654760217 conn(0x558322357560 msgr2=0x5583226feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 shutdown_connections 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 1 -- 172.21.15.27:0/1654760217 wait complete. 2024-04-25T13:45:15.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.947+0000 7ff991be5740 20 client.25933 trim_cache size 0 max 16384 2024-04-25T13:45:15.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff991be5740 1 Processor -- start 2024-04-25T13:45:15.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff991be5740 1 -- start start 2024-04-25T13:45:15.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff991be5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5583223221f0 0x55832270e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff991be5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x55832270cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff991be5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x558322703340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5583223221f0 0x55832270e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5583223221f0 0x55832270e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:35506/0 (socket says 172.21.15.27:35506) 2024-04-25T13:45:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff9889ed640 1 -- 172.21.15.27:0/2884786926 learned_addr learned my addr 172.21.15.27:0/2884786926 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff991be5740 1 -- 172.21.15.27:0/2884786926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9841b41c0 con 0x55832233a550 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff991be5740 1 -- 172.21.15.27:0/2884786926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9841b6a40 con 0x558322342730 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff991be5740 1 -- 172.21.15.27:0/2884786926 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff9841b4830 con 0x5583223221f0 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.973+0000 7ff983fff640 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x55832270cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.974+0000 7ff9891ee640 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x558322703340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.974+0000 7ff983fff640 1 -- 172.21.15.27:0/2884786926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5583223221f0 msgr2=0x55832270e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.974+0000 7ff983fff640 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5583223221f0 0x55832270e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.974+0000 7ff983fff640 1 -- 172.21.15.27:0/2884786926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 msgr2=0x558322703340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.974+0000 7ff983fff640 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x558322703340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.974+0000 7ff983fff640 1 -- 172.21.15.27:0/2884786926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583227009c0 con 0x55832233a550 2024-04-25T13:45:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.974+0000 7ff983fff640 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x55832270cf30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff9740fcdc0 tx=0x7ff974102d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52af9dfd449dbf31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.975+0000 7ff96a7fc640 1 -- 172.21.15.27:0/2884786926 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9740c3610 con 0x55832233a550 2024-04-25T13:45:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.975+0000 7ff96a7fc640 1 -- 172.21.15.27:0/2884786926 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9740c7260 con 0x55832233a550 2024-04-25T13:45:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.975+0000 7ff96a7fc640 1 -- 172.21.15.27:0/2884786926 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9740c6ec0 con 0x55832233a550 2024-04-25T13:45:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 -- 172.21.15.27:0/2884786926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 msgr2=0x55832270cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x55832270cf30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff9740fcdc0 tx=0x7ff974102d20 comp rx=0 tx=0).stop 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 -- 172.21.15.27:0/2884786926 shutdown_connections 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558322342730 0x558322703340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x55832270cf30 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 --2- 172.21.15.27:0/2884786926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5583223221f0 0x55832270e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 -- 172.21.15.27:0/2884786926 >> 172.21.15.27:0/2884786926 conn(0x558322357560 msgr2=0x558322702c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 -- 172.21.15.27:0/2884786926 shutdown_connections 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 -- 172.21.15.27:0/2884786926 wait complete. 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.976+0000 7ff991be5740 1 -- start start 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff991be5740 1 Processor -- start 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff991be5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5583223221f0 0x558322713190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff991be5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x5583227136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff991be5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x558322714590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff991be5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9841b4f00 con 0x55832233a550 2024-04-25T13:45:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff991be5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9841b6370 con 0x5583223221f0 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff991be5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff9841b55d0 con 0x558322342730 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff991be5740 10 client.0 _ll_register_callbacks cb 0x7ff97800ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5583223221f0 0x558322713190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff9889ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5583223221f0 0x558322713190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51466/0 (socket says 172.21.15.27:51466) 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff9889ed640 1 -- 172.21.15.27:0/3472450156 learned_addr learned my addr 172.21.15.27:0/3472450156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff9891ee640 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x558322714590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.977+0000 7ff983fff640 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x5583227136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff9889ed640 1 -- 172.21.15.27:0/3472450156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 msgr2=0x558322714590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff9889ed640 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x558322714590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff9889ed640 1 -- 172.21.15.27:0/3472450156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 msgr2=0x5583227136d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff9889ed640 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x5583227136d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff9889ed640 1 -- 172.21.15.27:0/3472450156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558322714b50 con 0x5583223221f0 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff983fff640 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x5583227136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:15.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff9889ed640 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5583223221f0 0x558322713190 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff978004c80 tx=0x7ff97800b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d372fd2cf6d7dac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff963fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff963fff640 1 -- 172.21.15.27:0/3472450156 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978036070 con 0x5583223221f0 2024-04-25T13:45:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff963fff640 1 -- 172.21.15.27:0/3472450156 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff97801c690 con 0x5583223221f0 2024-04-25T13:45:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff991be5740 1 -- 172.21.15.27:0/3472450156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9740c7260 con 0x5583223221f0 2024-04-25T13:45:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff963fff640 1 -- 172.21.15.27:0/3472450156 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff97802b070 con 0x5583223221f0 2024-04-25T13:45:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff991be5740 10 client.25916 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff991be5740 1 -- 172.21.15.27:0/3472450156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x558322715e10 con 0x5583223221f0 2024-04-25T13:45:15.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.978+0000 7ff963fff640 1 -- 172.21.15.27:0/3472450156 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9740c7260 con 0x5583223221f0 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.979+0000 7ff963fff640 1 -- 172.21.15.27:0/3472450156 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff9780041d0 con 0x5583223221f0 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.979+0000 7ff991be5740 10 client.25916 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.979+0000 7ff991be5740 10 client.25916 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.979+0000 7ff991be5740 20 client.25916 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.979+0000 7ff991be5740 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558322363410 0x558322785f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.979+0000 7ff991be5740 4 client.25916 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.979+0000 7ff991be5740 1 -- 172.21.15.27:0/3472450156 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558322677710 con 0x558322363410 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.979+0000 7ff983fff640 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558322363410 0x558322785f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.980+0000 7ff963fff640 10 client.25916 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.980+0000 7ff983fff640 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558322363410 0x558322785f40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f14c9f3ff5a8da7 server_cookie=820cc1301e8db8d1 in_seq=0 out_seq=0 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.981+0000 7ff963fff640 1 -- 172.21.15.27:0/3472450156 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558322677710 con 0x558322363410 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.981+0000 7ff963fff640 10 client.25916 handle_command_reply: tid=0 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.981+0000 7ff991be5740 1 client.25916 shutdown 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.981+0000 7ff991be5740 1 -- 172.21.15.27:0/3472450156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5583223221f0 msgr2=0x558322713190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.981+0000 7ff991be5740 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5583223221f0 0x558322713190 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff978004c80 tx=0x7ff97800b980 comp rx=0 tx=0).stop 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff991be5740 1 -- 172.21.15.27:0/3472450156 shutdown_connections 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff991be5740 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558322363410 0x558322785f40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff991be5740 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558322342730 0x558322714590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff991be5740 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55832233a550 0x5583227136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff991be5740 1 --2- 172.21.15.27:0/3472450156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5583223221f0 0x558322713190 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff963fff640 0 client.25916 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff991be5740 1 -- 172.21.15.27:0/3472450156 >> 172.21.15.27:0/3472450156 conn(0x558322357560 msgr2=0x558322703660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff991be5740 1 -- 172.21.15.27:0/3472450156 shutdown_connections 2024-04-25T13:45:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.982+0000 7ff991be5740 1 -- 172.21.15.27:0/3472450156 wait complete. 2024-04-25T13:45:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.983+0000 7ff991be5740 20 client.25916 trim_cache size 0 max 16384 2024-04-25T13:45:15.985 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:45:15.985 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:45:15.985 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "c2ee29e6-9dd2-4e19-b57a-6814ef6dfac2", 2024-04-25T13:45:15.985 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:45:15.985 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:45:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 -- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff96c0afaf0 msgr2=0x7ff96c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff96c0afaf0 0x7ff96c0b2000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x558322705670 tx=0x7ff97c009500 comp rx=0 tx=0).stop 2024-04-25T13:45:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 -- 172.21.15.27:0/2858497958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff98410c890 msgr2=0x7ff9841a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff98410c890 0x7ff9841a53c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff97402aa00 tx=0x7ff974003c10 comp rx=0 tx=0).stop 2024-04-25T13:45:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 -- 172.21.15.27:0/2858497958 shutdown_connections 2024-04-25T13:45:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff96c0afaf0 0x7ff96c0b2000 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9841aa2c0 0x7ff9841ac6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff98410c890 0x7ff9841a53c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 --2- 172.21.15.27:0/2858497958 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff98406b620 0x7ff9841a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.985+0000 7ff98ac51640 1 -- 172.21.15.27:0/2858497958 >> 172.21.15.27:0/2858497958 conn(0x7ff984071600 msgr2=0x7ff9841131e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.986+0000 7ff98ac51640 1 -- 172.21.15.27:0/2858497958 shutdown_connections 2024-04-25T13:45:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:15.986+0000 7ff98ac51640 1 -- 172.21.15.27:0/2858497958 wait complete. 2024-04-25T13:45:16.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:15 smithi027 ceph-mon[29140]: pgmap v480: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.4 MiB/s wr, 664 op/s 2024-04-25T13:45:16.001 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:15 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:45:16.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:45:16.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:45:16.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:15 smithi122 ceph-mon[32843]: pgmap v480: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.4 MiB/s wr, 664 op/s 2024-04-25T13:45:16.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:15 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:45:16.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:15 smithi045 ceph-mon[32731]: pgmap v480: 97 pgs: 97 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.4 MiB/s wr, 664 op/s 2024-04-25T13:45:16.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:15 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:45:16.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 -- 172.21.15.27:0/1652438281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3e41080b0 msgr2=0x7fa3e410a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/1652438281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3e41080b0 0x7fa3e410a4d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009be0 tx=0x7fa3dc02a6f0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 -- 172.21.15.27:0/1652438281 shutdown_connections 2024-04-25T13:45:16.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/1652438281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3e41080b0 0x7fa3e410a4d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/1652438281 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e41033b0 0x7fa3e41037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/1652438281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4102170 0x7fa3e4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 -- 172.21.15.27:0/1652438281 >> 172.21.15.27:0/1652438281 conn(0x7fa3e40fd340 msgr2=0x7fa3e40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 -- 172.21.15.27:0/1652438281 shutdown_connections 2024-04-25T13:45:16.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.186+0000 7fa3e90b1640 1 -- 172.21.15.27:0/1652438281 wait complete. 2024-04-25T13:45:16.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.188+0000 7fa3e90b1640 1 Processor -- start 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.188+0000 7fa3e90b1640 1 -- start start 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.188+0000 7fa3e90b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3e4102170 0x7fa3e419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.188+0000 7fa3e90b1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e41033b0 0x7fa3e419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e90b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e41080b0 0x7fa3e41a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e90b1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e410d460 con 0x7fa3e41033b0 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e90b1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e410d2e0 con 0x7fa3e4102170 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e90b1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e410d5e0 con 0x7fa3e41080b0 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e41033b0 0x7fa3e419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e41033b0 0x7fa3e419c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45700/0 (socket says 172.21.15.27:45700) 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e259c640 1 -- 172.21.15.27:0/4175535522 learned_addr learned my addr 172.21.15.27:0/4175535522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:16.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e359e640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e41080b0 0x7fa3e41a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3e4102170 0x7fa3e419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e259c640 1 -- 172.21.15.27:0/4175535522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e41080b0 msgr2=0x7fa3e41a3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e259c640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e41080b0 0x7fa3e41a3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e259c640 1 -- 172.21.15.27:0/4175535522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3e4102170 msgr2=0x7fa3e419c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e259c640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3e4102170 0x7fa3e419c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e259c640 1 -- 172.21.15.27:0/4175535522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e41a3f80 con 0x7fa3e41033b0 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.189+0000 7fa3e359e640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e41080b0 0x7fa3e41a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.190+0000 7fa3e259c640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e41033b0 0x7fa3e419c980 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800e9c0 tx=0x7fa3d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.190+0000 7fa3cbfff640 1 -- 172.21.15.27:0/4175535522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d800ccc0 con 0x7fa3e41033b0 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.190+0000 7fa3e90b1640 1 -- 172.21.15.27:0/4175535522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e41a4270 con 0x7fa3e41033b0 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.190+0000 7fa3e90b1640 1 -- 172.21.15.27:0/4175535522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e41a4890 con 0x7fa3e41033b0 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.190+0000 7fa3cbfff640 1 -- 172.21.15.27:0/4175535522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3d80054d0 con 0x7fa3e41033b0 2024-04-25T13:45:16.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.190+0000 7fa3cbfff640 1 -- 172.21.15.27:0/4175535522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8018430 con 0x7fa3e41033b0 2024-04-25T13:45:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.194+0000 7fa3f003a740 1 Processor -- start 2024-04-25T13:45:16.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3f003a740 1 -- start start 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b18270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f977454d0 0x559f97b187f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b19480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3f003a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559f97a74d80 con 0x559f9774d6b0 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3f003a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559f97a2e8b0 con 0x559f977454d0 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3f003a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559f97a70d40 con 0x559f9776e390 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3e2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f977454d0 0x559f97b187f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3e2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f977454d0 0x559f97b187f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51508/0 (socket says 172.21.15.27:51508) 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3e359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b18270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.195+0000 7fa3e2d9d640 1 -- 172.21.15.27:0/3076372678 learned_addr learned my addr 172.21.15.27:0/3076372678 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.196+0000 7fa3e359e640 1 -- 172.21.15.27:0/3076372678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f977454d0 msgr2=0x559f97b187f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.196+0000 7fa3e359e640 1 --2- 172.21.15.27:0/3076372678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f977454d0 0x559f97b187f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.196+0000 7fa3e359e640 1 -- 172.21.15.27:0/3076372678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 msgr2=0x559f97b19480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:45:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.196+0000 7fa3e359e640 1 --2- 172.21.15.27:0/3076372678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b19480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.196+0000 7fa3e359e640 1 -- 172.21.15.27:0/3076372678 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f979ffa40 con 0x559f9776e390 2024-04-25T13:45:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.196+0000 7fa3e359e640 1 --2- 172.21.15.27:0/3076372678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b18270 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc006010 tx=0x7fa3dc030ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=289a9d50db3257b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.197+0000 7fa3cbfff640 1 -- 172.21.15.27:0/4175535522 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa3d80046c0 con 0x7fa3e41033b0 2024-04-25T13:45:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.198+0000 7fa3c97fa640 1 -- 172.21.15.27:0/3076372678 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc041030 con 0x559f9776e390 2024-04-25T13:45:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.198+0000 7fa3c97fa640 1 -- 172.21.15.27:0/3076372678 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3dc003e20 con 0x559f9776e390 2024-04-25T13:45:16.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.198+0000 7fa3c97fa640 1 -- 172.21.15.27:0/3076372678 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc036040 con 0x559f9776e390 2024-04-25T13:45:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3cbfff640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3cc0afb50 0x7fa3cc0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3cbfff640 1 -- 172.21.15.27:0/4175535522 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa3d80defe0 con 0x7fa3e41033b0 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3cc0afb50 0x7fa3cc0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3f003a740 1 -- 172.21.15.27:0/3076372678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 msgr2=0x559f97b18270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3f003a740 1 --2- 172.21.15.27:0/3076372678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b18270 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc006010 tx=0x7fa3dc030ab0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3f003a740 1 -- 172.21.15.27:0/3076372678 shutdown_connections 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3f003a740 1 --2- 172.21.15.27:0/3076372678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b19480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3f003a740 1 --2- 172.21.15.27:0/3076372678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f977454d0 0x559f97b187f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3f003a740 1 --2- 172.21.15.27:0/3076372678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b18270 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.199+0000 7fa3f003a740 1 -- 172.21.15.27:0/3076372678 >> 172.21.15.27:0/3076372678 conn(0x559f977624e0 msgr2=0x559f97b0b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3f003a740 1 -- 172.21.15.27:0/3076372678 shutdown_connections 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3f003a740 1 -- 172.21.15.27:0/3076372678 wait complete. 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3cc0afb50 0x7fa3cc0b2060 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x559f97b17f90 tx=0x7fa3d4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3f003a740 1 -- start start 2024-04-25T13:45:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3f003a740 1 Processor -- start 2024-04-25T13:45:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b221d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9774d6b0 0x559f97b22710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b29670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.200+0000 7fa3f003a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559f97b19f30 con 0x559f977454d0 2024-04-25T13:45:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3f003a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559f979fbd10 con 0x559f9774d6b0 2024-04-25T13:45:16.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3f003a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559f97b1a0b0 con 0x559f9776e390 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3f003a740 10 client.0 _ll_register_callbacks cb 0x559f97a28210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b221d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b221d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45708/0 (socket says 172.21.15.27:45708) 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e2d9d640 1 -- 172.21.15.27:0/2760313722 learned_addr learned my addr 172.21.15.27:0/2760313722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e359e640 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b29670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e259c640 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9774d6b0 0x559f97b22710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e2d9d640 1 -- 172.21.15.27:0/2760313722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 msgr2=0x559f97b29670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b29670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e2d9d640 1 -- 172.21.15.27:0/2760313722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9774d6b0 msgr2=0x559f97b22710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9774d6b0 0x559f97b22710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.201+0000 7fa3e2d9d640 1 -- 172.21.15.27:0/2760313722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f978507d0 con 0x559f977454d0 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.202+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b221d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40167d0 tx=0x7fa3d4004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa41c64eff8192ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.202+0000 7fa3c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:45:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.203+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/2760313722 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d401c050 con 0x559f977454d0 2024-04-25T13:45:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.203+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/2760313722 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa3d4014930 con 0x559f977454d0 2024-04-25T13:45:16.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.204+0000 7fa3f003a740 1 -- 172.21.15.27:0/2760313722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3dc003e20 con 0x559f977454d0 2024-04-25T13:45:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.204+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/2760313722 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa3d4014b30 con 0x559f977454d0 2024-04-25T13:45:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3f003a740 10 client.16215 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3f003a740 1 -- 172.21.15.27:0/2760313722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559f977863f0 con 0x559f977454d0 2024-04-25T13:45:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/2760313722 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3dc003e20 con 0x559f977454d0 2024-04-25T13:45:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/2760313722 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa3d40064c0 con 0x559f977454d0 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3f003a740 10 client.16215 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3f003a740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3f003a740 20 client.16215 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3f003a740 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559f9772cf60 0x559f97b2fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3f003a740 4 client.16215 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.205+0000 7fa3f003a740 1 -- 172.21.15.27:0/2760313722 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f97a82990 con 0x559f9772cf60 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.206+0000 7fa3e259c640 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559f9772cf60 0x559f97b2fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.206+0000 7fa3e259c640 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559f9772cf60 0x559f97b2fc60 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2599fd8aab76656 server_cookie=9cb30d63eb9e772f in_seq=0 out_seq=0 2024-04-25T13:45:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.206+0000 7fa3c6ff5640 10 client.16215 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.207+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/2760313722 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559f97a82990 con 0x559f9772cf60 2024-04-25T13:45:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.207+0000 7fa3c6ff5640 10 client.16215 handle_command_reply: tid=0 2024-04-25T13:45:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.207+0000 7fa3f003a740 1 client.16215 shutdown 2024-04-25T13:45:16.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 -- 172.21.15.27:0/2760313722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 msgr2=0x559f97b221d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b221d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40167d0 tx=0x7fa3d4004700 comp rx=0 tx=0).stop 2024-04-25T13:45:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 -- 172.21.15.27:0/2760313722 shutdown_connections 2024-04-25T13:45:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559f9772cf60 0x559f97b2fc60 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9776e390 0x559f97b29670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9774d6b0 0x559f97b22710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 --2- 172.21.15.27:0/2760313722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b221d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 -- 172.21.15.27:0/2760313722 >> 172.21.15.27:0/2760313722 conn(0x559f977624e0 msgr2=0x559f97b0a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3c6ff5640 0 client.16215 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:16.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.208+0000 7fa3f003a740 1 -- 172.21.15.27:0/2760313722 shutdown_connections 2024-04-25T13:45:16.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.209+0000 7fa3f003a740 1 -- 172.21.15.27:0/2760313722 wait complete. 2024-04-25T13:45:16.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.209+0000 7fa3f003a740 20 client.16215 trim_cache size 0 max 16384 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3f003a740 1 Processor -- start 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3f003a740 1 -- start start 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f977454d0 0x559f97b1d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b18230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b0e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3f003a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d000c390 con 0x559f9774d6b0 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3f003a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d000ec10 con 0x559f9776e390 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b18230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3e259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b18230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45716/0 (socket says 172.21.15.27:45716) 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3e259c640 1 -- 172.21.15.27:0/1632024724 learned_addr learned my addr 172.21.15.27:0/1632024724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3e359e640 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b0e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3f003a740 1 -- 172.21.15.27:0/1632024724 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d000ca00 con 0x559f977454d0 2024-04-25T13:45:16.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.235+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f977454d0 0x559f97b1d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.236+0000 7fa3e259c640 1 -- 172.21.15.27:0/1632024724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f977454d0 msgr2=0x559f97b1d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:16.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.236+0000 7fa3e259c640 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f977454d0 0x559f97b1d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.236+0000 7fa3e259c640 1 -- 172.21.15.27:0/1632024724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 msgr2=0x559f97b0e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:16.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.236+0000 7fa3e259c640 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b0e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.236+0000 7fa3e259c640 1 -- 172.21.15.27:0/1632024724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f977460f0 con 0x559f9774d6b0 2024-04-25T13:45:16.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.236+0000 7fa3e359e640 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b0e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:16.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.236+0000 7fa3e259c640 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b18230 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa3d80a4700 tx=0x7fa3d80dc980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6762c0db6ba03e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:16.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.237+0000 7fa3c8ff9640 1 -- 172.21.15.27:0/1632024724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80a7b90 con 0x559f9774d6b0 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.237+0000 7fa3c8ff9640 1 -- 172.21.15.27:0/1632024724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3d80a7490 con 0x559f9774d6b0 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.237+0000 7fa3c8ff9640 1 -- 172.21.15.27:0/1632024724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80aebc0 con 0x559f9774d6b0 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 -- 172.21.15.27:0/1632024724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 msgr2=0x559f97b18230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b18230 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa3d80a4700 tx=0x7fa3d80dc980 comp rx=0 tx=0).stop 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 -- 172.21.15.27:0/1632024724 shutdown_connections 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b0e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f9774d6b0 0x559f97b18230 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1632024724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f977454d0 0x559f97b1d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 -- 172.21.15.27:0/1632024724 >> 172.21.15.27:0/1632024724 conn(0x559f977624e0 msgr2=0x559f97b0e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 -- 172.21.15.27:0/1632024724 shutdown_connections 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 -- 172.21.15.27:0/1632024724 wait complete. 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.238+0000 7fa3f003a740 1 -- start start 2024-04-25T13:45:16.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.239+0000 7fa3f003a740 1 Processor -- start 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.239+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b28a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.239+0000 7fa3f003a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9774d6b0 0x559f97b25600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.239+0000 7fa3e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b28a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.239+0000 7fa3e2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b28a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45730/0 (socket says 172.21.15.27:45730) 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.239+0000 7fa3e2d9d640 1 -- 172.21.15.27:0/1400156766 learned_addr learned my addr 172.21.15.27:0/1400156766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.239+0000 7fa3e259c640 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9774d6b0 0x559f97b25600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.240+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b25b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.240+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d000d0d0 con 0x559f977454d0 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.240+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d000e540 con 0x559f9776e390 2024-04-25T13:45:16.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.240+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d000d7a0 con 0x559f9774d6b0 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.240+0000 7fa3e359e640 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b25b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.241+0000 7fa3f003a740 10 client.0 _ll_register_callbacks cb 0x7fa3d4016480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.241+0000 7fa3e359e640 1 -- 172.21.15.27:0/1400156766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9774d6b0 msgr2=0x559f97b25600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.241+0000 7fa3e359e640 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9774d6b0 0x559f97b25600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.241+0000 7fa3e359e640 1 -- 172.21.15.27:0/1400156766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 msgr2=0x559f97b28a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.241+0000 7fa3e359e640 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b28a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.241+0000 7fa3e359e640 1 -- 172.21.15.27:0/1400156766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f97a354b0 con 0x559f9776e390 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.241+0000 7fa3e359e640 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b25b40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x559f97b192c0 tx=0x7fa3dc000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82dcf78e668f8ad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.241+0000 7fa3c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:16.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.242+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/1400156766 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc036040 con 0x559f9776e390 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.242+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/1400156766 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa3dc003970 con 0x559f9776e390 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.242+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3d80a7490 con 0x559f9776e390 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.242+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/1400156766 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa3dc055050 con 0x559f9776e390 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.242+0000 7fa3f003a740 10 client.25934 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.242+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=19}) -- 0x559f97b96d70 con 0x559f9776e390 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.242+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/1400156766 <== mon.2 v2:172.21.15.45:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa3dc003b70 con 0x559f9776e390 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.243+0000 7fa3f003a740 10 client.25934 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.243+0000 7fa3f003a740 10 client.25934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.243+0000 7fa3f003a740 20 client.25934 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.243+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559f977023a0 0x559f97b991f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.243+0000 7fa3f003a740 4 client.25934 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.243+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559f97a7c800 con 0x559f977023a0 2024-04-25T13:45:16.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.243+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559f977023a0 0x559f97b991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.244+0000 7fa3e2d9d640 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559f977023a0 0x559f97b991f0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61d592221185506d server_cookie=7c2b8c5dad6c74c9 in_seq=0 out_seq=0 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.244+0000 7fa3c6ff5640 10 client.25934 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.244+0000 7fa3c6ff5640 1 -- 172.21.15.27:0/1400156766 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559f97a7c800 con 0x559f977023a0 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3c6ff5640 10 client.25934 handle_command_reply: tid=0 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 client.25934 shutdown 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 msgr2=0x559f97b25b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b25b40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x559f97b192c0 tx=0x7fa3dc000f40 comp rx=0 tx=0).stop 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 shutdown_connections 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559f977023a0 0x559f97b991f0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f9776e390 0x559f97b25b40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559f9774d6b0 0x559f97b25600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3c6ff5640 0 client.25934 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 --2- 172.21.15.27:0/1400156766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559f977454d0 0x559f97b28a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 >> 172.21.15.27:0/1400156766 conn(0x559f977624e0 msgr2=0x559f97b0eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 shutdown_connections 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 1 -- 172.21.15.27:0/1400156766 wait complete. 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.245+0000 7fa3f003a740 20 client.25934 trim_cache size 0 max 16384 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "c2ee29e6-9dd2-4e19-b57a-6814ef6dfac2": { 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "c2ee29e6-9dd2-4e19-b57a-6814ef6dfac2", 2024-04-25T13:45:16.247 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:45:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:45:16.248 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:45:16.248 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.246+0000 7fa3e90b1640 1 -- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3cc0afb50 msgr2=0x7fa3cc0b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.246+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3cc0afb50 0x7fa3cc0b2060 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x559f97b17f90 tx=0x7fa3d4008040 comp rx=0 tx=0).stop 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.246+0000 7fa3e90b1640 1 -- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e41033b0 msgr2=0x7fa3e419c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.246+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e41033b0 0x7fa3e419c980 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800e9c0 tx=0x7fa3d800ee80 comp rx=0 tx=0).stop 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.247+0000 7fa3e90b1640 1 -- 172.21.15.27:0/4175535522 shutdown_connections 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.247+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa3cc0afb50 0x7fa3cc0b2060 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.247+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e41080b0 0x7fa3e41a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.247+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e41033b0 0x7fa3e419c980 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.247+0000 7fa3e90b1640 1 --2- 172.21.15.27:0/4175535522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3e4102170 0x7fa3e419c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.247+0000 7fa3e90b1640 1 -- 172.21.15.27:0/4175535522 >> 172.21.15.27:0/4175535522 conn(0x7fa3e40fd340 msgr2=0x7fa3e4104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.248+0000 7fa3e90b1640 1 -- 172.21.15.27:0/4175535522 shutdown_connections 2024-04-25T13:45:16.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:16.248+0000 7fa3e90b1640 1 -- 172.21.15.27:0/4175535522 wait complete. 2024-04-25T13:45:16.263 INFO:tasks.cephfs.filesystem:scrub status for tag:c2ee29e6-9dd2-4e19-b57a-6814ef6dfac2 - {'path': '/', 'tag': 'c2ee29e6-9dd2-4e19-b57a-6814ef6dfac2', 'options': 'recursive,force'} 2024-04-25T13:45:17.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:16 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:45:17.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:16 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:45:17.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:16 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:45:17.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:16 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:45:17.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:16 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:45:17.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:16 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:45:17.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:16 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:45:17.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:16 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:45:17.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:16 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:45:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 250.0 s, 2864.6 tps, lat 34.890 ms stddev 39.001 2024-04-25T13:45:18.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:17 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:45:18.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:17 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:18.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:17 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:45:18.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:17 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:18.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:17 smithi122 ceph-mon[32843]: pgmap v481: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 817 op/s 2024-04-25T13:45:18.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:17 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:45:18.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:17 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:18.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:17 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:45:18.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:17 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:18.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:17 smithi045 ceph-mon[32731]: pgmap v481: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 817 op/s 2024-04-25T13:45:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:17 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:45:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:17 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:17 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:45:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:17 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:17 smithi027 ceph-mon[29140]: pgmap v481: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 817 op/s 2024-04-25T13:45:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.941+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2589992900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae0102400 msgr2=0x7f3ae0104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.941+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2589992900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae0102400 0x7f3ae0104820 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ad401a910 tx=0x7f3ad403b450 comp rx=0 tx=0).stop 2024-04-25T13:45:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.943+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2589992900 shutdown_connections 2024-04-25T13:45:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.943+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2589992900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ae01076c0 0x7f3ae0107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.943+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2589992900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ae0104d60 0x7f3ae0107180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.943+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2589992900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae0102400 0x7f3ae0104820 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.943+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2589992900 >> 172.21.15.27:0/2589992900 conn(0x7f3ae00fdc10 msgr2=0x7f3ae01000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.947+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2589992900 shutdown_connections 2024-04-25T13:45:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.949+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2589992900 wait complete. 2024-04-25T13:45:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.950+0000 7f3ae55f8640 1 Processor -- start 2024-04-25T13:45:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.950+0000 7f3ae55f8640 1 -- start start 2024-04-25T13:45:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.950+0000 7f3ae55f8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ae0102400 0x7f3ae006c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.950+0000 7f3ae55f8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ae0104d60 0x7f3ae006cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.950+0000 7f3ae55f8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae01076c0 0x7f3ae0073b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.950+0000 7f3ae55f8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae010b0c0 con 0x7f3ae0102400 2024-04-25T13:45:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.950+0000 7f3ae55f8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae010af40 con 0x7f3ae01076c0 2024-04-25T13:45:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.950+0000 7f3ae55f8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae010b240 con 0x7f3ae0104d60 2024-04-25T13:45:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.951+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ae0102400 0x7f3ae006c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.951+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ae0102400 0x7f3ae006c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45742/0 (socket says 172.21.15.27:45742) 2024-04-25T13:45:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.951+0000 7f3adeffd640 1 -- 172.21.15.27:0/2906122284 learned_addr learned my addr 172.21.15.27:0/2906122284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.951+0000 7f3ade7fc640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ae0104d60 0x7f3ae006cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.953+0000 7f3adeffd640 1 -- 172.21.15.27:0/2906122284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ae0104d60 msgr2=0x7f3ae006cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.953+0000 7f3adeffd640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ae0104d60 0x7f3ae006cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.953+0000 7f3adeffd640 1 -- 172.21.15.27:0/2906122284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae01076c0 msgr2=0x7f3ae0073b90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:45:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.953+0000 7f3adeffd640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae01076c0 0x7f3ae0073b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.953+0000 7f3adeffd640 1 -- 172.21.15.27:0/2906122284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae0074230 con 0x7f3ae0102400 2024-04-25T13:45:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.955+0000 7f3adeffd640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ae0102400 0x7f3ae006c820 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4006010 tx=0x7f3ad4004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.957+0000 7f3ac7fff640 1 -- 172.21.15.27:0/2906122284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad404f070 con 0x7f3ae0102400 2024-04-25T13:45:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.957+0000 7f3ac7fff640 1 -- 172.21.15.27:0/2906122284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad40409c0 con 0x7f3ae0102400 2024-04-25T13:45:19.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.957+0000 7f3ac7fff640 1 -- 172.21.15.27:0/2906122284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4046030 con 0x7f3ae0102400 2024-04-25T13:45:19.959 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.959+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2906122284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae00744c0 con 0x7f3ae0102400 2024-04-25T13:45:19.959 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.959+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2906122284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae00749b0 con 0x7f3ae0102400 2024-04-25T13:45:19.962 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.962+0000 7f3aec589740 1 Processor -- start 2024-04-25T13:45:19.963 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.964+0000 7f3ac7fff640 1 -- 172.21.15.27:0/2906122284 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3ad4052070 con 0x7f3ae0102400 2024-04-25T13:45:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3aec589740 1 -- start start 2024-04-25T13:45:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3ac7fff640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3ac80afcf0 0x7f3ac80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3aec589740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d177f1b0 0x55e0d1b1f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3aec589740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b1fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3aec589740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 0x55e0d1b26df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3aec589740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e0d1a85460 con 0x55e0d175e4d0 2024-04-25T13:45:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3aec589740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e0d1a3f590 con 0x55e0d177f1b0 2024-04-25T13:45:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3aec589740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e0d1a81420 con 0x55e0d17562f0 2024-04-25T13:45:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d177f1b0 0x55e0d1b1f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d177f1b0 0x55e0d1b1f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51568/0 (socket says 172.21.15.27:51568) 2024-04-25T13:45:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.965+0000 7f3adeffd640 1 -- 172.21.15.27:0/3558507900 learned_addr learned my addr 172.21.15.27:0/3558507900 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3ac7fff640 1 -- 172.21.15.27:0/2906122284 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3ad4110080 con 0x7f3ae0102400 2024-04-25T13:45:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3ade7fc640 1 --2- 172.21.15.27:0/3558507900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b1fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3adeffd640 1 -- 172.21.15.27:0/3558507900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 msgr2=0x55e0d1b1fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3adeffd640 1 --2- 172.21.15.27:0/3558507900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b1fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3adeffd640 1 -- 172.21.15.27:0/3558507900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 msgr2=0x55e0d1b26df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3adeffd640 1 --2- 172.21.15.27:0/3558507900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 0x55e0d1b26df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3adeffd640 1 -- 172.21.15.27:0/3558507900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0d1a4c0d0 con 0x55e0d177f1b0 2024-04-25T13:45:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3adeffd640 1 --2- 172.21.15.27:0/3558507900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d177f1b0 0x55e0d1b1f970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4004e80 tx=0x7f3ad401a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c91c33d310af52c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.966+0000 7f3adf7fe640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3ac80afcf0 0x7f3ac80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.967+0000 7f3ac57fa640 1 -- 172.21.15.27:0/3558507900 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40bc110 con 0x55e0d177f1b0 2024-04-25T13:45:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.967+0000 7f3ac57fa640 1 -- 172.21.15.27:0/3558507900 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad4042400 con 0x55e0d177f1b0 2024-04-25T13:45:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.967+0000 7f3ac57fa640 1 -- 172.21.15.27:0/3558507900 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40dc8e0 con 0x55e0d177f1b0 2024-04-25T13:45:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.968+0000 7f3adf7fe640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3ac80afcf0 0x7f3ac80b2200 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3ae006b4a0 tx=0x7f3ad800b250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.968+0000 7f3aec589740 1 -- 172.21.15.27:0/3558507900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d177f1b0 msgr2=0x55e0d1b1f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.968+0000 7f3aec589740 1 --2- 172.21.15.27:0/3558507900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d177f1b0 0x55e0d1b1f970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4004e80 tx=0x7f3ad401a270 comp rx=0 tx=0).stop 2024-04-25T13:45:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.969+0000 7f3aec589740 1 -- 172.21.15.27:0/3558507900 shutdown_connections 2024-04-25T13:45:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.969+0000 7f3aec589740 1 --2- 172.21.15.27:0/3558507900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 0x55e0d1b26df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.969+0000 7f3aec589740 1 --2- 172.21.15.27:0/3558507900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b1fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.969+0000 7f3aec589740 1 --2- 172.21.15.27:0/3558507900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d177f1b0 0x55e0d1b1f970 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.969+0000 7f3aec589740 1 -- 172.21.15.27:0/3558507900 >> 172.21.15.27:0/3558507900 conn(0x55e0d1773300 msgr2=0x55e0d1b1b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.969+0000 7f3aec589740 1 -- 172.21.15.27:0/3558507900 shutdown_connections 2024-04-25T13:45:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.970+0000 7f3aec589740 1 -- 172.21.15.27:0/3558507900 wait complete. 2024-04-25T13:45:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.970+0000 7f3aec589740 1 -- start start 2024-04-25T13:45:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.971+0000 7f3aec589740 1 Processor -- start 2024-04-25T13:45:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.973+0000 7f3aec589740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d17562f0 0x55e0d1b29280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.973+0000 7f3aec589740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 0x55e0d1b297c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.973+0000 7f3aec589740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d177f1b0 0x55e0d1b30720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.973+0000 7f3aec589740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e0d1b27600 con 0x55e0d17562f0 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.973+0000 7f3aec589740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e0d1a0cb60 con 0x55e0d175e4d0 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.973+0000 7f3aec589740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e0d1b27780 con 0x55e0d177f1b0 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d17562f0 0x55e0d1b29280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3aec589740 10 client.0 _ll_register_callbacks cb 0x55e0d1a312b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d17562f0 0x55e0d1b29280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45752/0 (socket says 172.21.15.27:45752) 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adeffd640 1 -- 172.21.15.27:0/1673832831 learned_addr learned my addr 172.21.15.27:0/1673832831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adf7fe640 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d177f1b0 0x55e0d1b30720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3ade7fc640 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 0x55e0d1b297c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adf7fe640 1 -- 172.21.15.27:0/1673832831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 msgr2=0x55e0d1b297c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adf7fe640 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 0x55e0d1b297c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adf7fe640 1 -- 172.21.15.27:0/1673832831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d17562f0 msgr2=0x55e0d1b29280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adf7fe640 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d17562f0 0x55e0d1b29280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adf7fe640 1 -- 172.21.15.27:0/1673832831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0d1797210 con 0x55e0d177f1b0 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.974+0000 7f3adeffd640 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d17562f0 0x55e0d1b29280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.975+0000 7f3ac2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.975+0000 7f3adf7fe640 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d177f1b0 0x55e0d1b30720 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800ceb0 tx=0x7f3ad8006c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f9b6b9b8b004e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.975+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1673832831 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad801e070 con 0x55e0d177f1b0 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.975+0000 7f3aec589740 1 -- 172.21.15.27:0/1673832831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e0d1a93070 con 0x55e0d177f1b0 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.975+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1673832831 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3ad801a620 con 0x55e0d177f1b0 2024-04-25T13:45:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.975+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1673832831 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f3ad801e070 con 0x55e0d177f1b0 2024-04-25T13:45:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.976+0000 7f3aec589740 10 client.25969 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.976+0000 7f3aec589740 1 -- 172.21.15.27:0/1673832831 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e0d1863620 con 0x55e0d177f1b0 2024-04-25T13:45:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.976+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1673832831 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e0d1a93070 con 0x55e0d177f1b0 2024-04-25T13:45:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.976+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1673832831 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f3ad8023b10 con 0x55e0d177f1b0 2024-04-25T13:45:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.976+0000 7f3aec589740 10 client.25969 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.976+0000 7f3aec589740 10 client.25969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.976+0000 7f3aec589740 20 client.25969 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.976+0000 7f3aec589740 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e0d173dd80 0x55e0d1b37040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.977+0000 7f3aec589740 4 client.25969 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.977+0000 7f3aec589740 1 -- 172.21.15.27:0/1673832831 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0d1a8cee0 con 0x55e0d173dd80 2024-04-25T13:45:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.977+0000 7f3adeffd640 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e0d173dd80 0x55e0d1b37040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.977+0000 7f3ac2ff5640 10 client.25969 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.978+0000 7f3adeffd640 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e0d173dd80 0x55e0d1b37040 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=254761274d59b3cf server_cookie=3355e7444c2d9f42 in_seq=0 out_seq=0 2024-04-25T13:45:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1673832831 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e0d1a8cee0 con 0x55e0d173dd80 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3ac2ff5640 10 client.25969 handle_command_reply: tid=0 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 client.25969 shutdown 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 -- 172.21.15.27:0/1673832831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d177f1b0 msgr2=0x55e0d1b30720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d177f1b0 0x55e0d1b30720 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800ceb0 tx=0x7f3ad8006c70 comp rx=0 tx=0).stop 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 -- 172.21.15.27:0/1673832831 shutdown_connections 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e0d173dd80 0x55e0d1b37040 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3ac2ff5640 0 client.25969 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d177f1b0 0x55e0d1b30720 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 0x55e0d1b297c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 --2- 172.21.15.27:0/1673832831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d17562f0 0x55e0d1b29280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 -- 172.21.15.27:0/1673832831 >> 172.21.15.27:0/1673832831 conn(0x55e0d1773300 msgr2=0x55e0d1b1e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 -- 172.21.15.27:0/1673832831 shutdown_connections 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.979+0000 7f3aec589740 1 -- 172.21.15.27:0/1673832831 wait complete. 2024-04-25T13:45:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:19.980+0000 7f3aec589740 20 client.25969 trim_cache size 0 max 16384 2024-04-25T13:45:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.006+0000 7f3aec589740 1 Processor -- start 2024-04-25T13:45:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.006+0000 7f3aec589740 1 -- start start 2024-04-25T13:45:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.006+0000 7f3aec589740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d173dd80 0x55e0d1b27ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.006+0000 7f3aec589740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b30640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.006+0000 7f3aec589740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 0x55e0d1b1e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.006+0000 7f3aec589740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01afb10 con 0x55e0d175e4d0 2024-04-25T13:45:20.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.006+0000 7f3aec589740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01b23f0 con 0x55e0d173dd80 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3aec589740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01b01e0 con 0x55e0d17562f0 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 0x55e0d1b1e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d173dd80 0x55e0d1b27ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d173dd80 0x55e0d1b27ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51600/0 (socket says 172.21.15.27:51600) 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 -- 172.21.15.27:0/2173954346 learned_addr learned my addr 172.21.15.27:0/2173954346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3ade7fc640 1 --2- 172.21.15.27:0/2173954346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b30640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 -- 172.21.15.27:0/2173954346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 msgr2=0x55e0d1b30640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 --2- 172.21.15.27:0/2173954346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b30640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 -- 172.21.15.27:0/2173954346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 msgr2=0x55e0d1b1e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 --2- 172.21.15.27:0/2173954346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 0x55e0d1b1e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 -- 172.21.15.27:0/2173954346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0d1b30b80 con 0x55e0d173dd80 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3adeffd640 1 --2- 172.21.15.27:0/2173954346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d173dd80 0x55e0d1b27ca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40d8230 tx=0x7f3ad40d7da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1799939068e7f321 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3ac4ff9640 1 -- 172.21.15.27:0/2173954346 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40e1390 con 0x55e0d173dd80 2024-04-25T13:45:20.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.007+0000 7f3ac4ff9640 1 -- 172.21.15.27:0/2173954346 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad40c0020 con 0x55e0d173dd80 2024-04-25T13:45:20.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3ac4ff9640 1 -- 172.21.15.27:0/2173954346 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40d8790 con 0x55e0d173dd80 2024-04-25T13:45:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3aec589740 1 -- 172.21.15.27:0/2173954346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d173dd80 msgr2=0x55e0d1b27ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3aec589740 1 --2- 172.21.15.27:0/2173954346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d173dd80 0x55e0d1b27ca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40d8230 tx=0x7f3ad40d7da0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3aec589740 1 -- 172.21.15.27:0/2173954346 shutdown_connections 2024-04-25T13:45:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3aec589740 1 --2- 172.21.15.27:0/2173954346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d175e4d0 0x55e0d1b1e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3aec589740 1 --2- 172.21.15.27:0/2173954346 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b30640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3aec589740 1 --2- 172.21.15.27:0/2173954346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d173dd80 0x55e0d1b27ca0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3aec589740 1 -- 172.21.15.27:0/2173954346 >> 172.21.15.27:0/2173954346 conn(0x55e0d1773300 msgr2=0x55e0d1b239b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.008+0000 7f3aec589740 1 -- 172.21.15.27:0/2173954346 shutdown_connections 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 -- 172.21.15.27:0/2173954346 wait complete. 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 -- start start 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 Processor -- start 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d173dd80 0x55e0d1b2d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b2dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 0x55e0d1b9d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01b08b0 con 0x55e0d173dd80 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01b1d20 con 0x55e0d175e4d0 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01b0f80 con 0x55e0d17562f0 2024-04-25T13:45:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3aec589740 10 client.0 _ll_register_callbacks cb 0x7f3ad800dd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d173dd80 0x55e0d1b2d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3adeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d173dd80 0x55e0d1b2d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45758/0 (socket says 172.21.15.27:45758) 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.009+0000 7f3adeffd640 1 -- 172.21.15.27:0/1149788734 learned_addr learned my addr 172.21.15.27:0/1149788734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3adf7fe640 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 0x55e0d1b9d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3ade7fc640 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b2dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3adeffd640 1 -- 172.21.15.27:0/1149788734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 msgr2=0x55e0d1b2dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3adeffd640 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b2dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3adeffd640 1 -- 172.21.15.27:0/1149788734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 msgr2=0x55e0d1b9d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3adeffd640 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 0x55e0d1b9d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3adeffd640 1 -- 172.21.15.27:0/1149788734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0d1b9d920 con 0x55e0d173dd80 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3adeffd640 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d173dd80 0x55e0d1b2d580 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4004e80 tx=0x7f3ad40d8120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e359f168be9d6d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.010+0000 7f3ac2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:45:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.011+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1149788734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40d8530 con 0x55e0d173dd80 2024-04-25T13:45:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.011+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1149788734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3ad40d8790 con 0x55e0d173dd80 2024-04-25T13:45:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.011+0000 7f3aec589740 1 -- 172.21.15.27:0/1149788734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ad40c0020 con 0x55e0d173dd80 2024-04-25T13:45:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.011+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1149788734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f3ad40e1390 con 0x55e0d173dd80 2024-04-25T13:45:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.011+0000 7f3aec589740 10 client.16233 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.011+0000 7f3aec589740 1 -- 172.21.15.27:0/1149788734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e0d1b9ea60 con 0x55e0d173dd80 2024-04-25T13:45:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.011+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1149788734 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ad40c0020 con 0x55e0d173dd80 2024-04-25T13:45:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1149788734 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f3ad40d6480 con 0x55e0d173dd80 2024-04-25T13:45:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3aec589740 10 client.16233 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3aec589740 10 client.16233 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3aec589740 20 client.16233 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3aec589740 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e0d177f1b0 0x55e0d1ba10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3aec589740 4 client.16233 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:45:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3aec589740 1 -- 172.21.15.27:0/1149788734 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e0d1a085f0 con 0x55e0d177f1b0 2024-04-25T13:45:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3ade7fc640 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e0d177f1b0 0x55e0d1ba10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.012+0000 7f3ac2ff5640 10 client.16233 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.013+0000 7f3ade7fc640 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e0d177f1b0 0x55e0d1ba10c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b21a897d10143e4 server_cookie=4f993320e77402c9 in_seq=0 out_seq=0 2024-04-25T13:45:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3ac2ff5640 1 -- 172.21.15.27:0/1149788734 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e0d1a085f0 con 0x55e0d177f1b0 2024-04-25T13:45:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3ac2ff5640 10 client.16233 handle_command_reply: tid=0 2024-04-25T13:45:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3aec589740 1 client.16233 shutdown 2024-04-25T13:45:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3aec589740 1 -- 172.21.15.27:0/1149788734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d173dd80 msgr2=0x55e0d1b2d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3aec589740 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d173dd80 0x55e0d1b2d580 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4004e80 tx=0x7f3ad40d8120 comp rx=0 tx=0).stop 2024-04-25T13:45:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3aec589740 1 -- 172.21.15.27:0/1149788734 shutdown_connections 2024-04-25T13:45:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3aec589740 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e0d177f1b0 0x55e0d1ba10c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3aec589740 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0d175e4d0 0x55e0d1b9d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3ac2ff5640 0 client.16233 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3aec589740 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e0d17562f0 0x55e0d1b2dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.033+0000 7f3aec589740 1 --2- 172.21.15.27:0/1149788734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e0d173dd80 0x55e0d1b2d580 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.034+0000 7f3aec589740 1 -- 172.21.15.27:0/1149788734 >> 172.21.15.27:0/1149788734 conn(0x55e0d1773300 msgr2=0x55e0d1b1e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.034+0000 7f3aec589740 1 -- 172.21.15.27:0/1149788734 shutdown_connections 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.034+0000 7f3aec589740 1 -- 172.21.15.27:0/1149788734 wait complete. 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.034+0000 7f3aec589740 20 client.16233 trim_cache size 0 max 16384 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:45:20.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:45:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:45:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:45:20.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3ac80afcf0 msgr2=0x7f3ac80b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3ac80afcf0 0x7f3ac80b2200 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3ae006b4a0 tx=0x7f3ad800b250 comp rx=0 tx=0).stop 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ae0102400 msgr2=0x7f3ae006c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ae0102400 0x7f3ae006c820 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4006010 tx=0x7f3ad4004960 comp rx=0 tx=0).stop 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2906122284 shutdown_connections 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f3ac80afcf0 0x7f3ac80b2200 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ae01076c0 0x7f3ae0073b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ae0104d60 0x7f3ae006cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 --2- 172.21.15.27:0/2906122284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ae0102400 0x7f3ae006c820 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2906122284 >> 172.21.15.27:0/2906122284 conn(0x7f3ae00fdc10 msgr2=0x7f3ae0100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2906122284 shutdown_connections 2024-04-25T13:45:20.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:20.035+0000 7f3ae55f8640 1 -- 172.21.15.27:0/2906122284 wait complete. 2024-04-25T13:45:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:19 smithi122 ceph-mon[32843]: pgmap v482: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.5 MiB/s wr, 676 op/s 2024-04-25T13:45:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:19 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:45:20.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:19 smithi045 ceph-mon[32731]: pgmap v482: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.5 MiB/s wr, 676 op/s 2024-04-25T13:45:20.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:19 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:45:20.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:19 smithi027 ceph-mon[29140]: pgmap v482: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 8.5 MiB/s wr, 676 op/s 2024-04-25T13:45:20.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:19 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:45:22.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:21 smithi122 ceph-mon[32843]: pgmap v483: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 10 MiB/s wr, 849 op/s 2024-04-25T13:45:22.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:21 smithi045 ceph-mon[32731]: pgmap v483: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 10 MiB/s wr, 849 op/s 2024-04-25T13:45:22.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:21 smithi027 ceph-mon[29140]: pgmap v483: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 10 MiB/s wr, 849 op/s 2024-04-25T13:45:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 255.0 s, 2345.8 tps, lat 42.456 ms stddev 46.767 2024-04-25T13:45:24.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:23 smithi122 ceph-mon[32843]: pgmap v484: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-25T13:45:24.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:23 smithi045 ceph-mon[32731]: pgmap v484: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-25T13:45:24.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:23 smithi027 ceph-mon[29140]: pgmap v484: 97 pgs: 97 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-25T13:45:26.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:25 smithi122 ceph-mon[32843]: pgmap v485: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 8.6 MiB/s wr, 802 op/s 2024-04-25T13:45:26.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:25 smithi045 ceph-mon[32731]: pgmap v485: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 8.6 MiB/s wr, 802 op/s 2024-04-25T13:45:26.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:25 smithi027 ceph-mon[29140]: pgmap v485: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 8.6 MiB/s wr, 802 op/s 2024-04-25T13:45:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 260.0 s, 2941.0 tps, lat 34.064 ms stddev 36.044 2024-04-25T13:45:28.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:27 smithi045 ceph-mon[32731]: pgmap v486: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 930 op/s 2024-04-25T13:45:28.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:27 smithi027 ceph-mon[29140]: pgmap v486: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 930 op/s 2024-04-25T13:45:28.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:27 smithi122 ceph-mon[32843]: pgmap v486: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 930 op/s 2024-04-25T13:45:30.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:29 smithi045 ceph-mon[32731]: pgmap v487: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.9 MiB/s wr, 725 op/s 2024-04-25T13:45:30.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:29 smithi027 ceph-mon[29140]: pgmap v487: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.9 MiB/s wr, 725 op/s 2024-04-25T13:45:30.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:29 smithi122 ceph-mon[32843]: pgmap v487: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.9 MiB/s wr, 725 op/s 2024-04-25T13:45:32.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:31 smithi045 ceph-mon[32731]: pgmap v488: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.2 MiB/s wr, 767 op/s 2024-04-25T13:45:32.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:31 smithi027 ceph-mon[29140]: pgmap v488: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.2 MiB/s wr, 767 op/s 2024-04-25T13:45:32.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:31 smithi122 ceph-mon[32843]: pgmap v488: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.2 MiB/s wr, 767 op/s 2024-04-25T13:45:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 265.0 s, 2547.0 tps, lat 39.306 ms stddev 42.276 2024-04-25T13:45:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.191+0000 7f5f7336c640 1 -- 172.21.15.27:0/1868400964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c108960 msgr2=0x7f5f6c10ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.191+0000 7f5f7336c640 1 --2- 172.21.15.27:0/1868400964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c108960 0x7f5f6c10ad80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009860 tx=0x7f5f5c02a380 comp rx=0 tx=0).stop 2024-04-25T13:45:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.193+0000 7f5f7336c640 1 -- 172.21.15.27:0/1868400964 shutdown_connections 2024-04-25T13:45:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.193+0000 7f5f7336c640 1 --2- 172.21.15.27:0/1868400964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c108960 0x7f5f6c10ad80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.193+0000 7f5f7336c640 1 --2- 172.21.15.27:0/1868400964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c103c60 0x7f5f6c104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.193+0000 7f5f7336c640 1 --2- 172.21.15.27:0/1868400964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f6c102a20 0x7f5f6c103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.193+0000 7f5f7336c640 1 -- 172.21.15.27:0/1868400964 >> 172.21.15.27:0/1868400964 conn(0x7f5f6c0fdc10 msgr2=0x7f5f6c1000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.195+0000 7f5f7336c640 1 -- 172.21.15.27:0/1868400964 shutdown_connections 2024-04-25T13:45:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.196+0000 7f5f7336c640 1 -- 172.21.15.27:0/1868400964 wait complete. 2024-04-25T13:45:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.197+0000 7f5f7336c640 1 Processor -- start 2024-04-25T13:45:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.197+0000 7f5f7336c640 1 -- start start 2024-04-25T13:45:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f7336c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c102a20 0x7f5f6c1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f7336c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c103c60 0x7f5f6c1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f7336c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f6c108960 0x7f5f6c1a7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f7336c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c10cc60 con 0x7f5f6c102a20 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f7336c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c10cae0 con 0x7f5f6c108960 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f7336c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c10cde0 con 0x7f5f6c103c60 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f71909640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f6c108960 0x7f5f6c1a7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f71909640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f6c108960 0x7f5f6c1a7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38430/0 (socket says 172.21.15.27:38430) 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f71909640 1 -- 172.21.15.27:0/2273435387 learned_addr learned my addr 172.21.15.27:0/2273435387 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f70907640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c103c60 0x7f5f6c1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.198+0000 7f5f71108640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c102a20 0x7f5f6c1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f71909640 1 -- 172.21.15.27:0/2273435387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c103c60 msgr2=0x7f5f6c1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f71909640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c103c60 0x7f5f6c1a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f71909640 1 -- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c102a20 msgr2=0x7f5f6c1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f71909640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c102a20 0x7f5f6c1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f71909640 1 -- 172.21.15.27:0/2273435387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c110880 con 0x7f5f6c108960 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f70907640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c103c60 0x7f5f6c1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f71909640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f6c108960 0x7f5f6c1a7d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c030380 tx=0x7f5f5c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f627fc640 1 -- 172.21.15.27:0/2273435387 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c037050 con 0x7f5f6c108960 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f627fc640 1 -- 172.21.15.27:0/2273435387 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f5c043070 con 0x7f5f6c108960 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.199+0000 7f5f7336c640 1 -- 172.21.15.27:0/2273435387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c110b10 con 0x7f5f6c108960 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.200+0000 7f5f627fc640 1 -- 172.21.15.27:0/2273435387 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c030d30 con 0x7f5f6c108960 2024-04-25T13:45:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.200+0000 7f5f7336c640 1 -- 172.21.15.27:0/2273435387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c111000 con 0x7f5f6c108960 2024-04-25T13:45:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.203+0000 7f5f7a21c740 1 Processor -- start 2024-04-25T13:45:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.204+0000 7f5f7a21c740 1 -- start start 2024-04-25T13:45:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.204+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235ba11b0 0x558235f41970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.204+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b782f0 0x558235f41ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.204+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f48df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.204+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558235ea7460 con 0x558235b804d0 2024-04-25T13:45:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.204+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558235e61590 con 0x558235ba11b0 2024-04-25T13:45:33.206 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.204+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558235ea3420 con 0x558235b782f0 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.205+0000 7f5f70907640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235ba11b0 0x558235f41970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.205+0000 7f5f70907640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235ba11b0 0x558235f41970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38440/0 (socket says 172.21.15.27:38440) 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.205+0000 7f5f70907640 1 -- 172.21.15.27:0/3504793305 learned_addr learned my addr 172.21.15.27:0/3504793305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.205+0000 7f5f70907640 1 -- 172.21.15.27:0/3504793305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b782f0 msgr2=0x558235f41ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.205+0000 7f5f70907640 1 --2- 172.21.15.27:0/3504793305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b782f0 0x558235f41ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.205+0000 7f5f70907640 1 -- 172.21.15.27:0/3504793305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 msgr2=0x558235f48df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.205+0000 7f5f70907640 1 --2- 172.21.15.27:0/3504793305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f48df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.205+0000 7f5f70907640 1 -- 172.21.15.27:0/3504793305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558235e6e0d0 con 0x558235ba11b0 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.206+0000 7f5f70907640 1 --2- 172.21.15.27:0/3504793305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235ba11b0 0x558235f41970 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800bf80 tx=0x7f5f68012a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c4fa2e6af620cf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.206+0000 7f5f627fc640 1 -- 172.21.15.27:0/2273435387 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5f5c0037f0 con 0x7f5f6c108960 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.206+0000 7f5f577fe640 1 -- 172.21.15.27:0/3504793305 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6801f070 con 0x558235ba11b0 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.206+0000 7f5f577fe640 1 -- 172.21.15.27:0/3504793305 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f68010ca0 con 0x558235ba11b0 2024-04-25T13:45:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.206+0000 7f5f577fe640 1 -- 172.21.15.27:0/3504793305 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6801ae30 con 0x558235ba11b0 2024-04-25T13:45:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.207+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3504793305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235ba11b0 msgr2=0x558235f41970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.207+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3504793305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235ba11b0 0x558235f41970 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800bf80 tx=0x7f5f68012a10 comp rx=0 tx=0).stop 2024-04-25T13:45:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.207+0000 7f5f627fc640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f580afc20 0x7f5f580b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3504793305 shutdown_connections 2024-04-25T13:45:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3504793305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f48df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3504793305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b782f0 0x558235f41ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3504793305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235ba11b0 0x558235f41970 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3504793305 >> 172.21.15.27:0/3504793305 conn(0x558235b95300 msgr2=0x558235f3d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3504793305 shutdown_connections 2024-04-25T13:45:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f627fc640 1 -- 172.21.15.27:0/2273435387 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5f5c0ff990 con 0x7f5f6c108960 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3504793305 wait complete. 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.208+0000 7f5f7a21c740 1 -- start start 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f7a21c740 1 Processor -- start 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f70907640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f580afc20 0x7f5f580b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f4b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f4bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 0x558235f52b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558235f49840 con 0x558235ba11b0 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558235e2eb60 con 0x558235b782f0 2024-04-25T13:45:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558235f499c0 con 0x558235b804d0 2024-04-25T13:45:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.209+0000 7f5f7a21c740 10 client.0 _ll_register_callbacks cb 0x558235e532b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f70907640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f4bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f70907640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f4bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43288/0 (socket says 172.21.15.27:43288) 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f70907640 1 -- 172.21.15.27:0/4177020194 learned_addr learned my addr 172.21.15.27:0/4177020194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f71909640 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 0x558235f52b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f71108640 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f4b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f70907640 1 -- 172.21.15.27:0/4177020194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 msgr2=0x558235f4b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f70907640 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f4b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f70907640 1 -- 172.21.15.27:0/4177020194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 msgr2=0x558235f52b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f70907640 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 0x558235f52b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.210+0000 7f5f70907640 1 -- 172.21.15.27:0/4177020194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558235bb9210 con 0x558235b804d0 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.211+0000 7f5f70907640 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f4bc20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5f6801c000 tx=0x7f5f68012a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f63736cb6d03c804 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.211+0000 7f5f54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:33.213 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.211+0000 7f5f54ff9640 1 -- 172.21.15.27:0/4177020194 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68037060 con 0x558235b804d0 2024-04-25T13:45:33.213 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.211+0000 7f5f54ff9640 1 -- 172.21.15.27:0/4177020194 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5f68010a00 con 0x558235b804d0 2024-04-25T13:45:33.213 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.211+0000 7f5f7a21c740 1 -- 172.21.15.27:0/4177020194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f68010ca0 con 0x558235b804d0 2024-04-25T13:45:33.215 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.211+0000 7f5f70907640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f580afc20 0x7f5f580b2130 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x558235f42b40 tx=0x7f5f6801a510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:33.215 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f54ff9640 1 -- 172.21.15.27:0/4177020194 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f5f6801f040 con 0x558235b804d0 2024-04-25T13:45:33.215 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f7a21c740 10 client.25984 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:33.215 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f7a21c740 1 -- 172.21.15.27:0/4177020194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x558235c85620 con 0x558235b804d0 2024-04-25T13:45:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f54ff9640 1 -- 172.21.15.27:0/4177020194 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f68010ca0 con 0x558235b804d0 2024-04-25T13:45:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f54ff9640 1 -- 172.21.15.27:0/4177020194 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f5f68025070 con 0x558235b804d0 2024-04-25T13:45:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f7a21c740 10 client.25984 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f7a21c740 10 client.25984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f7a21c740 20 client.25984 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558235b5fd80 0x558235f59100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f7a21c740 4 client.25984 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.212+0000 7f5f7a21c740 1 -- 172.21.15.27:0/4177020194 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558235eb5070 con 0x558235b5fd80 2024-04-25T13:45:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.213+0000 7f5f71108640 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558235b5fd80 0x558235f59100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.213+0000 7f5f54ff9640 10 client.25984 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.213+0000 7f5f71108640 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558235b5fd80 0x558235f59100 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b9800eb626b423 server_cookie=9161fb852a0a241d in_seq=0 out_seq=0 2024-04-25T13:45:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f54ff9640 1 -- 172.21.15.27:0/4177020194 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558235eb5070 con 0x558235b5fd80 2024-04-25T13:45:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f54ff9640 10 client.25984 handle_command_reply: tid=0 2024-04-25T13:45:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 client.25984 shutdown 2024-04-25T13:45:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 -- 172.21.15.27:0/4177020194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 msgr2=0x558235f4bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f4bc20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5f6801c000 tx=0x7f5f68012a10 comp rx=0 tx=0).stop 2024-04-25T13:45:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 -- 172.21.15.27:0/4177020194 shutdown_connections 2024-04-25T13:45:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558235b5fd80 0x558235f59100 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 0x558235f52b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f4bc20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/4177020194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f4b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 -- 172.21.15.27:0/4177020194 >> 172.21.15.27:0/4177020194 conn(0x558235b95300 msgr2=0x558235f3c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f54ff9640 0 client.25984 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 -- 172.21.15.27:0/4177020194 shutdown_connections 2024-04-25T13:45:33.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 1 -- 172.21.15.27:0/4177020194 wait complete. 2024-04-25T13:45:33.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.214+0000 7f5f7a21c740 20 client.25984 trim_cache size 0 max 16384 2024-04-25T13:45:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.235+0000 7f5f7a21c740 1 Processor -- start 2024-04-25T13:45:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.235+0000 7f5f7a21c740 1 -- start start 2024-04-25T13:45:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.235+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f3cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.235+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f52aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.235+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 0x558235f3d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.235+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c117de0 con 0x558235ba11b0 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.235+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1ba920 con 0x558235b782f0 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.235+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c118400 con 0x558235b804d0 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f3cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f3cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38446/0 (socket says 172.21.15.27:38446) 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 -- 172.21.15.27:0/458043117 learned_addr learned my addr 172.21.15.27:0/458043117 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f70907640 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f52aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71909640 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 0x558235f3d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 -- 172.21.15.27:0/458043117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 msgr2=0x558235f52aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f52aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 -- 172.21.15.27:0/458043117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 msgr2=0x558235f3d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 0x558235f3d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 -- 172.21.15.27:0/458043117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558235f52fe0 con 0x558235b782f0 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f71108640 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f3cbe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5f64000f80 tx=0x7f5f6400d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12d21800f303c83f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.236+0000 7f5f56ffd640 1 -- 172.21.15.27:0/458043117 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f640169f0 con 0x558235b782f0 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.237+0000 7f5f56ffd640 1 -- 172.21.15.27:0/458043117 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f6401f070 con 0x558235b782f0 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.237+0000 7f5f56ffd640 1 -- 172.21.15.27:0/458043117 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64018400 con 0x558235b782f0 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.237+0000 7f5f7a21c740 1 -- 172.21.15.27:0/458043117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 msgr2=0x558235f3cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.237+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f3cbe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5f64000f80 tx=0x7f5f6400d040 comp rx=0 tx=0).stop 2024-04-25T13:45:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 -- 172.21.15.27:0/458043117 shutdown_connections 2024-04-25T13:45:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235ba11b0 0x558235f3d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235b804d0 0x558235f52aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/458043117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f3cbe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 -- 172.21.15.27:0/458043117 >> 172.21.15.27:0/458043117 conn(0x558235b95300 msgr2=0x558235f45b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 -- 172.21.15.27:0/458043117 shutdown_connections 2024-04-25T13:45:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 -- 172.21.15.27:0/458043117 wait complete. 2024-04-25T13:45:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 -- start start 2024-04-25T13:45:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.238+0000 7f5f7a21c740 1 Processor -- start 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f4ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f50470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f7a21c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235ba11b0 0x558235f513a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c118ad0 con 0x558235b804d0 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1ba250 con 0x558235b782f0 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f7a21c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1b94b0 con 0x558235ba11b0 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f70907640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f50470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f7a21c740 10 client.0 _ll_register_callbacks cb 0x7f5f6802c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f71909640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235ba11b0 0x558235f513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f71108640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f4ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f70907640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f50470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34440/0 (socket says 172.21.15.27:34440) 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f70907640 1 -- 172.21.15.27:0/3438137827 learned_addr learned my addr 172.21.15.27:0/3438137827 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f71909640 1 -- 172.21.15.27:0/3438137827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 msgr2=0x558235f4ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f71909640 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f4ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f71909640 1 -- 172.21.15.27:0/3438137827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 msgr2=0x558235f50470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f71909640 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f50470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f71909640 1 -- 172.21.15.27:0/3438137827 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558235f51960 con 0x558235ba11b0 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.239+0000 7f5f70907640 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f50470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f71909640 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235ba11b0 0x558235f513a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0c7b40 tx=0x7f5f5c0095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8d15ac3ee1a161 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f54ff9640 1 -- 172.21.15.27:0/3438137827 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0ca070 con 0x558235ba11b0 2024-04-25T13:45:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f54ff9640 1 -- 172.21.15.27:0/3438137827 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5f5c0d0a20 con 0x558235ba11b0 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3438137827 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f6401f070 con 0x558235ba11b0 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f54ff9640 1 -- 172.21.15.27:0/3438137827 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f5f5c0305f0 con 0x558235ba11b0 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f7a21c740 10 client.25990 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3438137827 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x558235f52c60 con 0x558235ba11b0 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f54ff9640 1 -- 172.21.15.27:0/3438137827 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f6401f070 con 0x558235ba11b0 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.240+0000 7f5f54ff9640 1 -- 172.21.15.27:0/3438137827 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f5f5c0c5d90 con 0x558235ba11b0 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f7a21c740 10 client.25990 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f7a21c740 10 client.25990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f7a21c740 20 client.25990 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558235b351c0 0x558235fc3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f7a21c740 4 client.25990 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3438137827 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558235eaeee0 con 0x558235b351c0 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f71108640 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558235b351c0 0x558235fc3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f54ff9640 10 client.25990 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.241+0000 7f5f71108640 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558235b351c0 0x558235fc3b60 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66a346e538b1fde server_cookie=fd7b87bbfc2e467d in_seq=0 out_seq=0 2024-04-25T13:45:33.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.262+0000 7f5f54ff9640 1 -- 172.21.15.27:0/3438137827 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558235eaeee0 con 0x558235b351c0 2024-04-25T13:45:33.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.262+0000 7f5f54ff9640 10 client.25990 handle_command_reply: tid=0 2024-04-25T13:45:33.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 client.25990 shutdown 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3438137827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235ba11b0 msgr2=0x558235f513a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235ba11b0 0x558235f513a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0c7b40 tx=0x7f5f5c0095b0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3438137827 shutdown_connections 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558235b351c0 0x558235fc3b60 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f54ff9640 0 client.25990 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558235ba11b0 0x558235f513a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558235b804d0 0x558235f50470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 --2- 172.21.15.27:0/3438137827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558235b782f0 0x558235f4ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.263+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3438137827 >> 172.21.15.27:0/3438137827 conn(0x558235b95300 msgr2=0x558235f3cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.264+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3438137827 shutdown_connections 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.264+0000 7f5f7a21c740 1 -- 172.21.15.27:0/3438137827 wait complete. 2024-04-25T13:45:33.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.264+0000 7f5f7a21c740 20 client.25990 trim_cache size 0 max 16384 2024-04-25T13:45:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:45:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:45:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:45:33.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:45:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 -- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f580afc20 msgr2=0x7f5f580b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f580afc20 0x7f5f580b2130 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x558235f42b40 tx=0x7f5f6801a510 comp rx=0 tx=0).stop 2024-04-25T13:45:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 -- 172.21.15.27:0/2273435387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f6c108960 msgr2=0x7f5f6c1a7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f6c108960 0x7f5f6c1a7d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c030380 tx=0x7f5f5c004270 comp rx=0 tx=0).stop 2024-04-25T13:45:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 -- 172.21.15.27:0/2273435387 shutdown_connections 2024-04-25T13:45:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5f580afc20 0x7f5f580b2130 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f6c108960 0x7f5f6c1a7d20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f6c103c60 0x7f5f6c1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 --2- 172.21.15.27:0/2273435387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c102a20 0x7f5f6c1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 -- 172.21.15.27:0/2273435387 >> 172.21.15.27:0/2273435387 conn(0x7f5f6c0fdc10 msgr2=0x7f5f6c10d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 -- 172.21.15.27:0/2273435387 shutdown_connections 2024-04-25T13:45:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:33.265+0000 7f5f7336c640 1 -- 172.21.15.27:0/2273435387 wait complete. 2024-04-25T13:45:34.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:33 smithi045 ceph-mon[32731]: pgmap v489: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 721 op/s 2024-04-25T13:45:34.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:33 smithi027 ceph-mon[29140]: pgmap v489: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 721 op/s 2024-04-25T13:45:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:33 smithi122 ceph-mon[32843]: pgmap v489: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 721 op/s 2024-04-25T13:45:36.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:35 smithi045 ceph-mon[32731]: pgmap v490: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 673 op/s 2024-04-25T13:45:36.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:35 smithi027 ceph-mon[29140]: pgmap v490: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 673 op/s 2024-04-25T13:45:36.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:35 smithi122 ceph-mon[32843]: pgmap v490: 97 pgs: 97 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 673 op/s 2024-04-25T13:45:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 270.0 s, 2210.4 tps, lat 45.044 ms stddev 51.068 2024-04-25T13:45:38.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:37 smithi045 ceph-mon[32731]: pgmap v491: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.8 MiB/s wr, 743 op/s 2024-04-25T13:45:38.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:37 smithi027 ceph-mon[29140]: pgmap v491: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.8 MiB/s wr, 743 op/s 2024-04-25T13:45:38.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:37 smithi122 ceph-mon[32843]: pgmap v491: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.8 MiB/s wr, 743 op/s 2024-04-25T13:45:40.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:39 smithi027 ceph-mon[29140]: pgmap v492: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.0 MiB/s wr, 612 op/s 2024-04-25T13:45:40.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:39 smithi122 ceph-mon[32843]: pgmap v492: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.0 MiB/s wr, 612 op/s 2024-04-25T13:45:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:39 smithi045 ceph-mon[32731]: pgmap v492: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.0 MiB/s wr, 612 op/s 2024-04-25T13:45:42.109 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:41 smithi122 ceph-mon[32843]: pgmap v493: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-04-25T13:45:42.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:41 smithi027 ceph-mon[29140]: pgmap v493: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-04-25T13:45:42.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:41 smithi045 ceph-mon[32731]: pgmap v493: 97 pgs: 97 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-04-25T13:45:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 275.0 s, 2734.6 tps, lat 36.683 ms stddev 40.365 2024-04-25T13:45:44.128 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:43 smithi045 ceph-mon[32731]: pgmap v494: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-04-25T13:45:44.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:43 smithi027 ceph-mon[29140]: pgmap v494: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-04-25T13:45:44.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:43 smithi122 ceph-mon[32843]: pgmap v494: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-04-25T13:45:46.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:45 smithi027 ceph-mon[29140]: pgmap v495: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 MiB/s wr, 678 op/s 2024-04-25T13:45:46.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:45:46.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:45:46.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:45 smithi122 ceph-mon[32843]: pgmap v495: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 MiB/s wr, 678 op/s 2024-04-25T13:45:46.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:45 smithi045 ceph-mon[32731]: pgmap v495: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 MiB/s wr, 678 op/s 2024-04-25T13:45:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.412+0000 7f92d4f00640 1 -- 172.21.15.27:0/3557869631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0102a20 msgr2=0x7f92d0103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.412+0000 7f92d4f00640 1 --2- 172.21.15.27:0/3557869631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0102a20 0x7f92d0103720 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009900 tx=0x7f92c002a400 comp rx=0 tx=0).stop 2024-04-25T13:45:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.413+0000 7f92d4f00640 1 -- 172.21.15.27:0/3557869631 shutdown_connections 2024-04-25T13:45:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.413+0000 7f92d4f00640 1 --2- 172.21.15.27:0/3557869631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92d006a3c0 0x7f92d006a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.413+0000 7f92d4f00640 1 --2- 172.21.15.27:0/3557869631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d0103c60 0x7f92d0104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.413+0000 7f92d4f00640 1 --2- 172.21.15.27:0/3557869631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d0102a20 0x7f92d0103720 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.413+0000 7f92d4f00640 1 -- 172.21.15.27:0/3557869631 >> 172.21.15.27:0/3557869631 conn(0x7f92d00fdc10 msgr2=0x7f92d01000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.414+0000 7f92d4f00640 1 -- 172.21.15.27:0/3557869631 shutdown_connections 2024-04-25T13:45:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.414+0000 7f92d4f00640 1 -- 172.21.15.27:0/3557869631 wait complete. 2024-04-25T13:45:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92d4f00640 1 Processor -- start 2024-04-25T13:45:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92d4f00640 1 -- start start 2024-04-25T13:45:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92d4f00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d006a3c0 0x7f92d01a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92d4f00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d0102a20 0x7f92d01a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92d4f00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92d0103c60 0x7f92d01a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92d4f00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f92d010d4c0 con 0x7f92d0103c60 2024-04-25T13:45:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92d4f00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f92d010d340 con 0x7f92d0102a20 2024-04-25T13:45:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92d4f00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f92d010d640 con 0x7f92d006a3c0 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92cdd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d0102a20 0x7f92d01a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92ced9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92d0103c60 0x7f92d01a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92cdd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d0102a20 0x7f92d01a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:32836/0 (socket says 172.21.15.27:32836) 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92cdd9b640 1 -- 172.21.15.27:0/1749847851 learned_addr learned my addr 172.21.15.27:0/1749847851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.415+0000 7f92ce59c640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d006a3c0 0x7f92d01a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92cdd9b640 1 -- 172.21.15.27:0/1749847851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d006a3c0 msgr2=0x7f92d01a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d006a3c0 0x7f92d01a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92cdd9b640 1 -- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92d0103c60 msgr2=0x7f92d01a7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92d0103c60 0x7f92d01a7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92cdd9b640 1 -- 172.21.15.27:0/1749847851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d01a83c0 con 0x7f92d0102a20 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92ce59c640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d006a3c0 0x7f92d01a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92ced9d640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92d0103c60 0x7f92d01a7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d0102a20 0x7f92d01a0dc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f92c400c890 tx=0x7f92c400cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92bf7fe640 1 -- 172.21.15.27:0/1749847851 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c4004220 con 0x7f92d0102a20 2024-04-25T13:45:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92bf7fe640 1 -- 172.21.15.27:0/1749847851 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c40043c0 con 0x7f92d0102a20 2024-04-25T13:45:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92d4f00640 1 -- 172.21.15.27:0/1749847851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d01a86b0 con 0x7f92d0102a20 2024-04-25T13:45:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92bf7fe640 1 -- 172.21.15.27:0/1749847851 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c4004a00 con 0x7f92d0102a20 2024-04-25T13:45:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.416+0000 7f92d4f00640 1 -- 172.21.15.27:0/1749847851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d01a8c30 con 0x7f92d0102a20 2024-04-25T13:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.421+0000 7f92dbdb1740 1 Processor -- start 2024-04-25T13:45:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.421+0000 7f92dbdb1740 1 -- start start 2024-04-25T13:45:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.421+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d07b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.421+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.421+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 0x55e84d082df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.421+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e84cfe1460 con 0x55e84ccba4d0 2024-04-25T13:45:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.421+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e84cf9b590 con 0x55e84ccb22f0 2024-04-25T13:45:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.421+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e84cfdd420 con 0x55e84ccdb1b0 2024-04-25T13:45:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ce59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ced9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d07b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ced9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d07b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33750/0 (socket says 172.21.15.27:33750) 2024-04-25T13:45:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ced9d640 1 -- 172.21.15.27:0/975736474 learned_addr learned my addr 172.21.15.27:0/975736474 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ce59c640 1 -- 172.21.15.27:0/975736474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 msgr2=0x55e84d07b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ce59c640 1 --2- 172.21.15.27:0/975736474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d07b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ce59c640 1 -- 172.21.15.27:0/975736474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 msgr2=0x55e84d082df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:45:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ce59c640 1 --2- 172.21.15.27:0/975736474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 0x55e84d082df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.422+0000 7f92ce59c640 1 -- 172.21.15.27:0/975736474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e84cfa80d0 con 0x55e84ccb22f0 2024-04-25T13:45:46.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.423+0000 7f92bf7fe640 1 -- 172.21.15.27:0/1749847851 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f92c4004ba0 con 0x7f92d0102a20 2024-04-25T13:45:46.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.423+0000 7f92ce59c640 1 --2- 172.21.15.27:0/975736474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07bef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009230 tx=0x7f92c0003740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b1b1629323fc0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.424+0000 7f92bf7fe640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f92b80afae0 0x7f92b80b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.424+0000 7f92bcff9640 1 -- 172.21.15.27:0/975736474 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0035030 con 0x55e84ccb22f0 2024-04-25T13:45:46.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.424+0000 7f92bcff9640 1 -- 172.21.15.27:0/975736474 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c0040070 con 0x55e84ccb22f0 2024-04-25T13:45:46.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.424+0000 7f92ced9d640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f92b80afae0 0x7f92b80b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.424+0000 7f92bf7fe640 1 -- 172.21.15.27:0/1749847851 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f92c40de8f0 con 0x7f92d0102a20 2024-04-25T13:45:46.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92bcff9640 1 -- 172.21.15.27:0/975736474 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0045470 con 0x55e84ccb22f0 2024-04-25T13:45:46.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92dbdb1740 1 -- 172.21.15.27:0/975736474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 msgr2=0x55e84d07bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92ced9d640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f92b80afae0 0x7f92b80b1ff0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009fd0 tx=0x7f92c800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/975736474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07bef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009230 tx=0x7f92c0003740 comp rx=0 tx=0).stop 2024-04-25T13:45:46.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92dbdb1740 1 -- 172.21.15.27:0/975736474 shutdown_connections 2024-04-25T13:45:46.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/975736474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 0x55e84d082df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/975736474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07bef0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/975736474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d07b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.425+0000 7f92dbdb1740 1 -- 172.21.15.27:0/975736474 >> 172.21.15.27:0/975736474 conn(0x55e84cccf300 msgr2=0x55e84d077300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.427+0000 7f92dbdb1740 1 -- 172.21.15.27:0/975736474 shutdown_connections 2024-04-25T13:45:46.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.427+0000 7f92dbdb1740 1 -- 172.21.15.27:0/975736474 wait complete. 2024-04-25T13:45:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.428+0000 7f92dbdb1740 1 -- start start 2024-04-25T13:45:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.428+0000 7f92dbdb1740 1 Processor -- start 2024-04-25T13:45:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d0856a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 0x55e84d085be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d08cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e84d083880 con 0x55e84ccba4d0 2024-04-25T13:45:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e84cf68b60 con 0x55e84ccb22f0 2024-04-25T13:45:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e84d083a00 con 0x55e84ccdb1b0 2024-04-25T13:45:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92dbdb1740 10 client.0 _ll_register_callbacks cb 0x55e84cf8d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d0856a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ced9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d08cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d0856a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:32850/0 (socket says 172.21.15.27:32850) 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 -- 172.21.15.27:0/2936895439 learned_addr learned my addr 172.21.15.27:0/2936895439 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 0x55e84d085be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 -- 172.21.15.27:0/2936895439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 msgr2=0x55e84d08cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d08cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 -- 172.21.15.27:0/2936895439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 msgr2=0x55e84d085be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 0x55e84d085be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 -- 172.21.15.27:0/2936895439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e84ccf3210 con 0x55e84ccb22f0 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92ce59c640 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d0856a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f92c0004650 tx=0x7f92c00317d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da3e6e22886aedd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.429+0000 7f92b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.430+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2936895439 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0053060 con 0x55e84ccb22f0 2024-04-25T13:45:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.430+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2936895439 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f92c0003d40 con 0x55e84ccb22f0 2024-04-25T13:45:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.430+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2936895439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92c0040070 con 0x55e84ccb22f0 2024-04-25T13:45:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.431+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2936895439 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f92c0035070 con 0x55e84ccb22f0 2024-04-25T13:45:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.431+0000 7f92dbdb1740 10 client.25973 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.431+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2936895439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e84cdbf620 con 0x55e84ccb22f0 2024-04-25T13:45:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.431+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2936895439 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c0045920 con 0x55e84ccb22f0 2024-04-25T13:45:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.431+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2936895439 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f92c004d030 con 0x55e84ccb22f0 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.431+0000 7f92dbdb1740 10 client.25973 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.431+0000 7f92dbdb1740 10 client.25973 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.431+0000 7f92dbdb1740 20 client.25973 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.432+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e84cc99d80 0x55e84d092f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.432+0000 7f92dbdb1740 4 client.25973 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.432+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2936895439 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f92c0040070 con 0x55e84cc99d80 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.432+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e84cc99d80 0x55e84d092f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.432+0000 7f92b5ffb640 10 client.25973 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.432+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e84cc99d80 0x55e84d092f90 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c96f26041a986844 server_cookie=77c6f7bc704ad518 in_seq=0 out_seq=0 2024-04-25T13:45:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.433+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2936895439 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f92c0040070 con 0x55e84cc99d80 2024-04-25T13:45:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.433+0000 7f92b5ffb640 10 client.25973 handle_command_reply: tid=0 2024-04-25T13:45:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.433+0000 7f92dbdb1740 1 client.25973 shutdown 2024-04-25T13:45:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.433+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2936895439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 msgr2=0x55e84d0856a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.433+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d0856a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f92c0004650 tx=0x7f92c00317d0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2936895439 shutdown_connections 2024-04-25T13:45:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e84cc99d80 0x55e84d092f90 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccdb1b0 0x55e84d08cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccba4d0 0x55e84d085be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2936895439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d0856a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2936895439 >> 172.21.15.27:0/2936895439 conn(0x55e84cccf300 msgr2=0x55e84d07a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92b5ffb640 0 client.25973 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2936895439 shutdown_connections 2024-04-25T13:45:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2936895439 wait complete. 2024-04-25T13:45:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.435+0000 7f92dbdb1740 20 client.25973 trim_cache size 0 max 16384 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 -- 172.21.15.27:0/532878635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820106a40 msgr2=0x7fe820108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 --2- 172.21.15.27:0/532878635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820106a40 0x7fe820108f20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009900 tx=0x7fe80c02a3f0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 -- 172.21.15.27:0/532878635 shutdown_connections 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 --2- 172.21.15.27:0/532878635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8201095b0 0x7fe82010ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 --2- 172.21.15.27:0/532878635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820106a40 0x7fe820108f20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 --2- 172.21.15.27:0/532878635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe820104050 0x7fe820106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 -- 172.21.15.27:0/532878635 >> 172.21.15.27:0/532878635 conn(0x7fe8200fd9f0 msgr2=0x7fe8200ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 -- 172.21.15.27:0/532878635 shutdown_connections 2024-04-25T13:45:46.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.441+0000 7fe825ac4640 1 -- 172.21.15.27:0/532878635 wait complete. 2024-04-25T13:45:46.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe825ac4640 1 Processor -- start 2024-04-25T13:45:46.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe825ac4640 1 -- start start 2024-04-25T13:45:46.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe825ac4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820104050 0x7fe820199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe825ac4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe820106a40 0x7fe82019a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe825ac4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8201095b0 0x7fe8201a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe825ac4640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe82010f420 con 0x7fe820106a40 2024-04-25T13:45:46.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe825ac4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe82010f2a0 con 0x7fe820104050 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe825ac4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe82010f5a0 con 0x7fe8201095b0 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe81ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe820106a40 0x7fe82019a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe824ac2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820104050 0x7fe820199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe81ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe820106a40 0x7fe82019a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35506/0 (socket says 172.21.15.27:35506) 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.442+0000 7fe8252c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8201095b0 0x7fe8201a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe81ffff640 1 -- 172.21.15.27:0/383999858 learned_addr learned my addr 172.21.15.27:0/383999858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe824ac2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820104050 0x7fe820199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:32874/0 (socket says 172.21.15.27:32874) 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe8252c3640 1 -- 172.21.15.27:0/383999858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820104050 msgr2=0x7fe820199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe8252c3640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820104050 0x7fe820199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe8252c3640 1 -- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe820106a40 msgr2=0x7fe82019a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe8252c3640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe820106a40 0x7fe82019a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe8252c3640 1 -- 172.21.15.27:0/383999858 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe820101ec0 con 0x7fe8201095b0 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe81ffff640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe820106a40 0x7fe82019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe8252c3640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8201095b0 0x7fe8201a14c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe818009fd0 tx=0x7fe81800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe81dffb640 1 -- 172.21.15.27:0/383999858 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81800ce20 con 0x7fe8201095b0 2024-04-25T13:45:46.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe81dffb640 1 -- 172.21.15.27:0/383999858 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe818005020 con 0x7fe8201095b0 2024-04-25T13:45:46.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe825ac4640 1 -- 172.21.15.27:0/383999858 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8201021b0 con 0x7fe8201095b0 2024-04-25T13:45:46.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.443+0000 7fe81dffb640 1 -- 172.21.15.27:0/383999858 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe818010690 con 0x7fe8201095b0 2024-04-25T13:45:46.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.444+0000 7fe825ac4640 1 -- 172.21.15.27:0/383999858 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8201026b0 con 0x7fe8201095b0 2024-04-25T13:45:46.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.448+0000 7fe81dffb640 1 -- 172.21.15.27:0/383999858 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe818010830 con 0x7fe8201095b0 2024-04-25T13:45:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.448+0000 7fe82dbd8740 1 Processor -- start 2024-04-25T13:45:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.448+0000 7fe82dbd8740 1 -- start start 2024-04-25T13:45:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.448+0000 7fe81dffb640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe8100afb30 0x7fe8100b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4fc390 0x55a79e89dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4d34d0 0x55a79e89e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e89ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a79e802d80 con 0x55a79e4db6b0 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55a79e7bc8b0 con 0x55a79e4d34d0 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55a79e7fed40 con 0x55a79e4fc390 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe81ffff640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe8100afb30 0x7fe8100b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe81dffb640 1 -- 172.21.15.27:0/383999858 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe8180df2d0 con 0x7fe8201095b0 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe81ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4d34d0 0x55a79e89e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe8252c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e89ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe81ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4d34d0 0x55a79e89e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:32878/0 (socket says 172.21.15.27:32878) 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe81ffff640 1 -- 172.21.15.27:0/3726838681 learned_addr learned my addr 172.21.15.27:0/3726838681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe824ac2640 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4fc390 0x55a79e89dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe8252c3640 1 -- 172.21.15.27:0/3726838681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4fc390 msgr2=0x55a79e89dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe8252c3640 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4fc390 0x55a79e89dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe8252c3640 1 -- 172.21.15.27:0/3726838681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4d34d0 msgr2=0x55a79e89e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe8252c3640 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4d34d0 0x55a79e89e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe8252c3640 1 -- 172.21.15.27:0/3726838681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a79e78da40 con 0x55a79e4db6b0 2024-04-25T13:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.449+0000 7fe81ffff640 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4d34d0 0x55a79e89e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.450+0000 7fe8252c3640 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e89ee60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe8180dd170 tx=0x7fe8180a7430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4524a163bb6bac1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.450+0000 7fe81ffff640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe8100afb30 0x7fe8100b2040 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005710 tx=0x7fe80c0035f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.450+0000 7fe80affd640 1 -- 172.21.15.27:0/3726838681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8180b0640 con 0x55a79e4db6b0 2024-04-25T13:45:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.450+0000 7fe80affd640 1 -- 172.21.15.27:0/3726838681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8180051d0 con 0x55a79e4db6b0 2024-04-25T13:45:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.450+0000 7fe80affd640 1 -- 172.21.15.27:0/3726838681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe818090310 con 0x55a79e4db6b0 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.451+0000 7fe82dbd8740 1 -- 172.21.15.27:0/3726838681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 msgr2=0x55a79e89ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.451+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e89ee60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe8180dd170 tx=0x7fe8180a7430 comp rx=0 tx=0).stop 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.452+0000 7fe82dbd8740 1 -- 172.21.15.27:0/3726838681 shutdown_connections 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.452+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e89ee60 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.452+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4d34d0 0x55a79e89e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.452+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/3726838681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4fc390 0x55a79e89dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.452+0000 7fe82dbd8740 1 -- 172.21.15.27:0/3726838681 >> 172.21.15.27:0/3726838681 conn(0x55a79e4f04e0 msgr2=0x55a79e899460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.452+0000 7fe82dbd8740 1 -- 172.21.15.27:0/3726838681 shutdown_connections 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.452+0000 7fe82dbd8740 1 -- 172.21.15.27:0/3726838681 wait complete. 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 1 -- start start 2024-04-25T13:45:46.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 1 Processor -- start 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e8a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e8aed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a79e89f910 con 0x55a79e4fc390 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55a79e789d10 con 0x55a79e4db6b0 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55a79e89fa90 con 0x55a79e4d34d0 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe8252c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e8aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe81ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e8a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.453+0000 7fe82dbd8740 10 client.0 _ll_register_callbacks cb 0x55a79e7b6210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe81ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e8a7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:32894/0 (socket says 172.21.15.27:32894) 2024-04-25T13:45:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe81ffff640 1 -- 172.21.15.27:0/2226254212 learned_addr learned my addr 172.21.15.27:0/2226254212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe824ac2640 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe81ffff640 1 -- 172.21.15.27:0/2226254212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 msgr2=0x55a79e8a7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe81ffff640 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe81ffff640 1 -- 172.21.15.27:0/2226254212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 msgr2=0x55a79e8aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe81ffff640 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e8aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe81ffff640 1 -- 172.21.15.27:0/2226254212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a79e5de7d0 con 0x55a79e4db6b0 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe8252c3640 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e8aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe81ffff640 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e8a7dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005880 tx=0x7fe80c02a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83fedf7c940ba26d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe7fffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe7fffff640 1 -- 172.21.15.27:0/2226254212 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c050070 con 0x55a79e4db6b0 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe7fffff640 1 -- 172.21.15.27:0/2226254212 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe80c04c680 con 0x55a79e4db6b0 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.454+0000 7fe82dbd8740 1 -- 172.21.15.27:0/2226254212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8180051d0 con 0x55a79e4db6b0 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe7fffff640 1 -- 172.21.15.27:0/2226254212 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe80c05c050 con 0x55a79e4db6b0 2024-04-25T13:45:46.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe82dbd8740 10 client.25985 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe82dbd8740 1 -- 172.21.15.27:0/2226254212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55a79e5143f0 con 0x55a79e4db6b0 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe7fffff640 1 -- 172.21.15.27:0/2226254212 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8180051d0 con 0x55a79e4db6b0 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe7fffff640 1 -- 172.21.15.27:0/2226254212 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe80c03c9f0 con 0x55a79e4db6b0 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe82dbd8740 10 client.25985 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe82dbd8740 10 client.25985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe82dbd8740 20 client.25985 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a79e4baf60 0x55a79e8b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe82dbd8740 4 client.25985 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe82dbd8740 1 -- 172.21.15.27:0/2226254212 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a79e810990 con 0x55a79e4baf60 2024-04-25T13:45:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.455+0000 7fe824ac2640 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a79e4baf60 0x55a79e8b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.456+0000 7fe7fffff640 10 client.25985 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.456+0000 7fe824ac2640 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a79e4baf60 0x55a79e8b51b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71df7378c2b00907 server_cookie=ca266ee519de9d0a in_seq=0 out_seq=0 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.457+0000 7fe7fffff640 1 -- 172.21.15.27:0/2226254212 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a79e810990 con 0x55a79e4baf60 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.457+0000 7fe7fffff640 10 client.25985 handle_command_reply: tid=0 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.457+0000 7fe82dbd8740 1 client.25985 shutdown 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 -- 172.21.15.27:0/2226254212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 msgr2=0x55a79e8a7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e8a7dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005880 tx=0x7fe80c02a3f0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 -- 172.21.15.27:0/2226254212 shutdown_connections 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a79e4baf60 0x55a79e8b51b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e8aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e8a7dc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/2226254212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 -- 172.21.15.27:0/2226254212 >> 172.21.15.27:0/2226254212 conn(0x55a79e4f04e0 msgr2=0x55a79e898420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe7fffff640 0 client.25985 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 -- 172.21.15.27:0/2226254212 shutdown_connections 2024-04-25T13:45:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 1 -- 172.21.15.27:0/2226254212 wait complete. 2024-04-25T13:45:46.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.458+0000 7fe82dbd8740 20 client.25985 trim_cache size 0 max 16384 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.461+0000 7f92dbdb1740 1 Processor -- start 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.461+0000 7f92dbdb1740 1 -- start start 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.461+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.461+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d076c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.461+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d077150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.461+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f92d01af910 con 0x55e84ccdb1b0 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.461+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f92d01b2190 con 0x55e84ccb22f0 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.461+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f92d01aff80 con 0x55e84ccba4d0 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92ce59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92cdd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d076c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92ce59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:32902/0 (socket says 172.21.15.27:32902) 2024-04-25T13:45:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92ce59c640 1 -- 172.21.15.27:0/797692873 learned_addr learned my addr 172.21.15.27:0/797692873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92ced9d640 1 --2- 172.21.15.27:0/797692873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92cdd9b640 1 -- 172.21.15.27:0/797692873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 msgr2=0x55e84d07a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/797692873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92cdd9b640 1 -- 172.21.15.27:0/797692873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 msgr2=0x55e84d077150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/797692873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d077150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92cdd9b640 1 -- 172.21.15.27:0/797692873 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e84d0804b0 con 0x55e84ccba4d0 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/797692873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d076c10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f92c40058c0 tx=0x7f92c4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=518a7ad96ee6299c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92b7fff640 1 -- 172.21.15.27:0/797692873 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c401d040 con 0x55e84ccba4d0 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92b7fff640 1 -- 172.21.15.27:0/797692873 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c40ab1b0 con 0x55e84ccba4d0 2024-04-25T13:45:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.462+0000 7f92b7fff640 1 -- 172.21.15.27:0/797692873 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c40a2820 con 0x55e84ccba4d0 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.464+0000 7f92dbdb1740 1 -- 172.21.15.27:0/797692873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 msgr2=0x55e84d076c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.464+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/797692873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d076c10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f92c40058c0 tx=0x7f92c4000bc0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.465+0000 7f92dbdb1740 1 -- 172.21.15.27:0/797692873 shutdown_connections 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.465+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/797692873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d077150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.465+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/797692873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d076c10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.465+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/797692873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d07a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.465+0000 7f92dbdb1740 1 -- 172.21.15.27:0/797692873 >> 172.21.15.27:0/797692873 conn(0x55e84cccf300 msgr2=0x55e84d07a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.465+0000 7f92dbdb1740 1 -- 172.21.15.27:0/797692873 shutdown_connections 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.465+0000 7f92dbdb1740 1 -- 172.21.15.27:0/797692873 wait complete. 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.465+0000 7f92dbdb1740 1 -- start start 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92dbdb1740 1 Processor -- start 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d089270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d0897b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92dbdb1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d08a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92ce59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d089270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92cdd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d0897b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f92d01b0650 con 0x55e84ccdb1b0 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92ce59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d089270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:32914/0 (socket says 172.21.15.27:32914) 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92cdd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d0897b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33840/0 (socket says 172.21.15.27:33840) 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92dbdb1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f92d01b1ac0 con 0x55e84ccb22f0 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92ce59c640 1 -- 172.21.15.27:0/2204627050 learned_addr learned my addr 172.21.15.27:0/2204627050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f92d01b0d20 con 0x55e84ccba4d0 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92dbdb1740 10 client.0 _ll_register_callbacks cb 0x7f92c8009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.466+0000 7f92ced9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d08a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92ce59c640 1 -- 172.21.15.27:0/2204627050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 msgr2=0x55e84d0897b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.469 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92ce59c640 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d0897b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92ce59c640 1 -- 172.21.15.27:0/2204627050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 msgr2=0x55e84d08a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92ce59c640 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d08a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92ce59c640 1 -- 172.21.15.27:0/2204627050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e84d08aca0 con 0x55e84ccb22f0 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92ced9d640 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d08a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d0897b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92ce59c640 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d089270 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f92c002a9f0 tx=0x7f92c00058c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41104885a3d2906d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2204627050 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0053030 con 0x55e84ccb22f0 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2204627050 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f92c0031a40 con 0x55e84ccb22f0 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92c40ab1b0 con 0x55e84ccb22f0 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2204627050 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f92c005d080 con 0x55e84ccb22f0 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92dbdb1740 10 client.25991 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:46.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.467+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e84d08bf10 con 0x55e84ccb22f0 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2204627050 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c40ab1b0 con 0x55e84ccb22f0 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2204627050 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f92c004d040 con 0x55e84ccb22f0 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92dbdb1740 10 client.25991 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92dbdb1740 10 client.25991 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92dbdb1740 20 client.25991 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e84cc6f1c0 0x55e84d0fd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92dbdb1740 4 client.25991 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e84cfef070 con 0x55e84cc6f1c0 2024-04-25T13:45:46.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.468+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e84cc6f1c0 0x55e84d0fd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.472 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.470+0000 7f92cdd9b640 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e84cc6f1c0 0x55e84d0fd330 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95ce8f3480ffa7e2 server_cookie=2f4a88464f482bb0 in_seq=0 out_seq=0 2024-04-25T13:45:46.472 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.470+0000 7f92b5ffb640 10 client.25991 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.488+0000 7f92b5ffb640 1 -- 172.21.15.27:0/2204627050 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e84cfef070 con 0x55e84cc6f1c0 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.488+0000 7f92b5ffb640 10 client.25991 handle_command_reply: tid=0 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.488+0000 7f92dbdb1740 1 client.25991 shutdown 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.488+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 msgr2=0x55e84d089270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.488+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d089270 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f92c002a9f0 tx=0x7f92c00058c0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 shutdown_connections 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e84cc6f1c0 0x55e84d0fd330 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e84ccdb1b0 0x55e84d08a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e84ccba4d0 0x55e84d0897b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 1 --2- 172.21.15.27:0/2204627050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e84ccb22f0 0x55e84d089270 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 >> 172.21.15.27:0/2204627050 conn(0x55e84cccf300 msgr2=0x55e84d083df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92b5ffb640 0 client.25991 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 shutdown_connections 2024-04-25T13:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 1 -- 172.21.15.27:0/2204627050 wait complete. 2024-04-25T13:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.489+0000 7f92dbdb1740 20 client.25991 trim_cache size 0 max 16384 2024-04-25T13:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:45:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:45:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.490+0000 7fe82dbd8740 1 Processor -- start 2024-04-25T13:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 -- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f92b80afae0 msgr2=0x7f92b80b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f92b80afae0 0x7f92b80b1ff0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009fd0 tx=0x7f92c800d040 comp rx=0 tx=0).stop 2024-04-25T13:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 -- 172.21.15.27:0/1749847851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d0102a20 msgr2=0x7f92d01a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d0102a20 0x7f92d01a0dc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f92c400c890 tx=0x7f92c400cd50 comp rx=0 tx=0).stop 2024-04-25T13:45:46.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.490+0000 7fe82dbd8740 1 -- start start 2024-04-25T13:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 -- 172.21.15.27:0/1749847851 shutdown_connections 2024-04-25T13:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f92b80afae0 0x7f92b80b1ff0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92d0103c60 0x7f92d01a7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92d0102a20 0x7f92d01a0dc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 --2- 172.21.15.27:0/1749847851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92d006a3c0 0x7f92d01a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 -- 172.21.15.27:0/1749847851 >> 172.21.15.27:0/1749847851 conn(0x7f92d00fdc10 msgr2=0x7f92d010de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e89dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.491+0000 7f92d4f00640 1 -- 172.21.15.27:0/1749847851 shutdown_connections 2024-04-25T13:45:46.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e899480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe824ac2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe824ac2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33856/0 (socket says 172.21.15.27:33856) 2024-04-25T13:45:46.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe824ac2640 1 -- 172.21.15.27:0/1363865078 learned_addr learned my addr 172.21.15.27:0/1363865078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe82dbd8740 1 -- 172.21.15.27:0/1363865078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8201ab080 con 0x55a79e4fc390 2024-04-25T13:45:46.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe82dbd8740 1 -- 172.21.15.27:0/1363865078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe8201ad900 con 0x55a79e4db6b0 2024-04-25T13:45:46.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe82dbd8740 1 -- 172.21.15.27:0/1363865078 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe8201ab6f0 con 0x55a79e4d34d0 2024-04-25T13:45:46.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.491+0000 7fe81ffff640 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e89dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.492+0000 7fe8252c3640 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e899480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:46.492+0000 7f92d4f00640 1 -- 172.21.15.27:0/1749847851 wait complete. 2024-04-25T13:45:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.492+0000 7fe824ac2640 1 -- 172.21.15.27:0/1363865078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 msgr2=0x55a79e89dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.492+0000 7fe824ac2640 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e89dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.492+0000 7fe824ac2640 1 -- 172.21.15.27:0/1363865078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 msgr2=0x55a79e899480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.492+0000 7fe824ac2640 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e899480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.492+0000 7fe824ac2640 1 -- 172.21.15.27:0/1363865078 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a79e4d40f0 con 0x55a79e4d34d0 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.492+0000 7fe824ac2640 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a2c40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe814009730 tx=0x7fe814005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee8b5041be5a7e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.493+0000 7fe80a7fc640 1 -- 172.21.15.27:0/1363865078 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814004250 con 0x55a79e4d34d0 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.493+0000 7fe80a7fc640 1 -- 172.21.15.27:0/1363865078 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8140043f0 con 0x55a79e4d34d0 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.493+0000 7fe80a7fc640 1 -- 172.21.15.27:0/1363865078 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814003d40 con 0x55a79e4d34d0 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 -- 172.21.15.27:0/1363865078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 msgr2=0x55a79e8a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a2c40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe814009730 tx=0x7fe814005f90 comp rx=0 tx=0).stop 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 -- 172.21.15.27:0/1363865078 shutdown_connections 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4fc390 0x55a79e899480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4db6b0 0x55a79e89dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/1363865078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8a2c40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 -- 172.21.15.27:0/1363865078 >> 172.21.15.27:0/1363865078 conn(0x55a79e4f04e0 msgr2=0x55a79e898d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 -- 172.21.15.27:0/1363865078 shutdown_connections 2024-04-25T13:45:46.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.494+0000 7fe82dbd8740 1 -- 172.21.15.27:0/1363865078 wait complete. 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.495+0000 7fe82dbd8740 1 -- start start 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.495+0000 7fe82dbd8740 1 Processor -- start 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e8abe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe82dbd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4fc390 0x55a79e8ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe8201abdc0 con 0x55a79e4db6b0 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe8201ad230 con 0x55a79e4fc390 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe82dbd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe8201ac490 con 0x55a79e4d34d0 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe82dbd8740 10 client.0 _ll_register_callbacks cb 0x7fe80c0057b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe81ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e8abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe824ac2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe8252c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4fc390 0x55a79e8ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe824ac2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33866/0 (socket says 172.21.15.27:33866) 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe81ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e8abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35548/0 (socket says 172.21.15.27:35548) 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.496+0000 7fe824ac2640 1 -- 172.21.15.27:0/818375064 learned_addr learned my addr 172.21.15.27:0/818375064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe824ac2640 1 -- 172.21.15.27:0/818375064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4fc390 msgr2=0x55a79e8ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe824ac2640 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4fc390 0x55a79e8ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe824ac2640 1 -- 172.21.15.27:0/818375064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 msgr2=0x55a79e8abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe824ac2640 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e8abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe824ac2640 1 -- 172.21.15.27:0/818375064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a79e7c34b0 con 0x55a79e4d34d0 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe81ffff640 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e8abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe824ac2640 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8af270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe814003790 tx=0x7fe8140037c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9186d42c0fc4d8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe7fffff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe7fffff640 1 -- 172.21.15.27:0/818375064 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814003cc0 con 0x55a79e4d34d0 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe7fffff640 1 -- 172.21.15.27:0/818375064 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe81401f430 con 0x55a79e4d34d0 2024-04-25T13:45:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe82dbd8740 1 -- 172.21.15.27:0/818375064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8140043f0 con 0x55a79e4d34d0 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe7fffff640 1 -- 172.21.15.27:0/818375064 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe814003e60 con 0x55a79e4d34d0 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe82dbd8740 10 client.26026 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.497+0000 7fe82dbd8740 1 -- 172.21.15.27:0/818375064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55a79e91c300 con 0x55a79e4d34d0 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe7fffff640 1 -- 172.21.15.27:0/818375064 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8140043f0 con 0x55a79e4d34d0 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe7fffff640 1 -- 172.21.15.27:0/818375064 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe81402abe0 con 0x55a79e4d34d0 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe82dbd8740 10 client.26026 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe82dbd8740 10 client.26026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe82dbd8740 20 client.26026 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a79e4903a0 0x55a79e91ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe82dbd8740 4 client.26026 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe82dbd8740 1 -- 172.21.15.27:0/818375064 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a79e80a800 con 0x55a79e4903a0 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.498+0000 7fe81ffff640 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a79e4903a0 0x55a79e91ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.499+0000 7fe81ffff640 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a79e4903a0 0x55a79e91ec90 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3586e72b814c9ae server_cookie=4895d13a256079d1 in_seq=0 out_seq=0 2024-04-25T13:45:46.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.499+0000 7fe7fffff640 10 client.26026 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe7fffff640 1 -- 172.21.15.27:0/818375064 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a79e80a800 con 0x55a79e4903a0 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe7fffff640 10 client.26026 handle_command_reply: tid=0 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 client.26026 shutdown 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 -- 172.21.15.27:0/818375064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 msgr2=0x55a79e8af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8af270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe814003790 tx=0x7fe8140037c0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 -- 172.21.15.27:0/818375064 shutdown_connections 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a79e4903a0 0x55a79e91ec90 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a79e4fc390 0x55a79e8ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a79e4db6b0 0x55a79e8abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 --2- 172.21.15.27:0/818375064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a79e4d34d0 0x55a79e8af270 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe82dbd8740 1 -- 172.21.15.27:0/818375064 >> 172.21.15.27:0/818375064 conn(0x55a79e4f04e0 msgr2=0x55a79e8979f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.500+0000 7fe7fffff640 0 client.26026 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.501+0000 7fe82dbd8740 1 -- 172.21.15.27:0/818375064 shutdown_connections 2024-04-25T13:45:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.501+0000 7fe82dbd8740 1 -- 172.21.15.27:0/818375064 wait complete. 2024-04-25T13:45:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.501+0000 7fe82dbd8740 20 client.26026 trim_cache size 0 max 16384 2024-04-25T13:45:46.506 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:45:46.506 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:45:46.507 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:45:46.507 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 -- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe8100afb30 msgr2=0x7fe8100b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe8100afb30 0x7fe8100b2040 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005710 tx=0x7fe80c0035f0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 -- 172.21.15.27:0/383999858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8201095b0 msgr2=0x7fe8201a14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8201095b0 0x7fe8201a14c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe818009fd0 tx=0x7fe81800ef10 comp rx=0 tx=0).stop 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 -- 172.21.15.27:0/383999858 shutdown_connections 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe8100afb30 0x7fe8100b2040 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8201095b0 0x7fe8201a14c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe820106a40 0x7fe82019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 --2- 172.21.15.27:0/383999858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe820104050 0x7fe820199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 -- 172.21.15.27:0/383999858 >> 172.21.15.27:0/383999858 conn(0x7fe8200fd9f0 msgr2=0x7fe820104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 -- 172.21.15.27:0/383999858 shutdown_connections 2024-04-25T13:45:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.502+0000 7fe825ac4640 1 -- 172.21.15.27:0/383999858 wait complete. 2024-04-25T13:45:46.517 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:45:46.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:45:46.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.681+0000 7f1947772640 1 -- 172.21.15.27:0/3415050936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194010a420 msgr2=0x7f194010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.681+0000 7f1947772640 1 --2- 172.21.15.27:0/3415050936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194010a420 0x7f194010add0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1938009ca0 tx=0x7f193802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.681+0000 7f1947772640 1 -- 172.21.15.27:0/3415050936 shutdown_connections 2024-04-25T13:45:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.681+0000 7f1947772640 1 --2- 172.21.15.27:0/3415050936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1940110030 0x7f1940110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.682+0000 7f1947772640 1 --2- 172.21.15.27:0/3415050936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194010b310 0x7f194010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.682+0000 7f1947772640 1 --2- 172.21.15.27:0/3415050936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f194010a420 0x7f194010add0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.682+0000 7f1947772640 1 -- 172.21.15.27:0/3415050936 >> 172.21.15.27:0/3415050936 conn(0x7f19400704e0 msgr2=0x7f1940072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.682+0000 7f1947772640 1 -- 172.21.15.27:0/3415050936 shutdown_connections 2024-04-25T13:45:46.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.682+0000 7f1947772640 1 -- 172.21.15.27:0/3415050936 wait complete. 2024-04-25T13:45:46.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.683+0000 7f1947772640 1 Processor -- start 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.683+0000 7f1947772640 1 -- start start 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.683+0000 7f1947772640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194010a420 0x7f19401a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.683+0000 7f1947772640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194010b310 0x7f19401a9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.683+0000 7f1947772640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1940110030 0x7f19401b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.683+0000 7f1944d0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194010b310 0x7f19401a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.683+0000 7f1944d0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194010b310 0x7f19401a9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35566/0 (socket says 172.21.15.27:35566) 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.683+0000 7f194550e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194010a420 0x7f19401a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f194550e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194010a420 0x7f19401a90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33886/0 (socket says 172.21.15.27:33886) 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1944d0d640 1 -- 172.21.15.27:0/1565576233 learned_addr learned my addr 172.21.15.27:0/1565576233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f194011e490 con 0x7f194010b310 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f194011e310 con 0x7f1940110030 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f194011e610 con 0x7f194010a420 2024-04-25T13:45:46.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1945d0f640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1940110030 0x7f19401b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1944d0d640 1 -- 172.21.15.27:0/1565576233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194010a420 msgr2=0x7f19401a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1944d0d640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194010a420 0x7f19401a90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1944d0d640 1 -- 172.21.15.27:0/1565576233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1940110030 msgr2=0x7f19401b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1944d0d640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1940110030 0x7f19401b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1944d0d640 1 -- 172.21.15.27:0/1565576233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19401b0c10 con 0x7f194010b310 2024-04-25T13:45:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1945d0f640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1940110030 0x7f19401b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:46.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1944d0d640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194010b310 0x7f19401a9610 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9f0 tx=0x7f193000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f19367fc640 1 -- 172.21.15.27:0/1565576233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193000cd80 con 0x7f194010b310 2024-04-25T13:45:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f19367fc640 1 -- 172.21.15.27:0/1565576233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19300054d0 con 0x7f194010b310 2024-04-25T13:45:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f19367fc640 1 -- 172.21.15.27:0/1565576233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1930010690 con 0x7f194010b310 2024-04-25T13:45:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.684+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19401b0f00 con 0x7f194010b310 2024-04-25T13:45:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.685+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19401b1370 con 0x7f194010b310 2024-04-25T13:45:46.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.686+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194010ee40 con 0x7f194010b310 2024-04-25T13:45:46.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.688+0000 7f19367fc640 1 -- 172.21.15.27:0/1565576233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1930004e30 con 0x7f194010b310 2024-04-25T13:45:46.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.689+0000 7f19367fc640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f192c0afb70 0x7f192c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:46.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.689+0000 7f19367fc640 1 -- 172.21.15.27:0/1565576233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f19300df240 con 0x7f194010b310 2024-04-25T13:45:46.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.690+0000 7f194550e640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f192c0afb70 0x7f192c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:46.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.690+0000 7f194550e640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f192c0afb70 0x7f192c0b2080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f19380094e0 tx=0x7f1938004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:46.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:46.694+0000 7f19367fc640 1 -- 172.21.15.27:0/1565576233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1930053ff0 con 0x7f194010b310 2024-04-25T13:45:47.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.055+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f194002db50 con 0x7f194010b310 2024-04-25T13:45:47.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.055+0000 7f19367fc640 1 -- 172.21.15.27:0/1565576233 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f19300aee20 con 0x7f194010b310 2024-04-25T13:45:47.055 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:45:47.055 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:45:47.056 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:45:47.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.058+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f192c0afb70 msgr2=0x7f192c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.058+0000 7f1947772640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f192c0afb70 0x7f192c0b2080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f19380094e0 tx=0x7f1938004a40 comp rx=0 tx=0).stop 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.058+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194010b310 msgr2=0x7f19401a9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.058+0000 7f1947772640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194010b310 0x7f19401a9610 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9f0 tx=0x7f193000edf0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.059+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 shutdown_connections 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.059+0000 7f1947772640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f192c0afb70 0x7f192c0b2080 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.059+0000 7f1947772640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1940110030 0x7f19401b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.059+0000 7f1947772640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f194010b310 0x7f19401a9610 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.059+0000 7f1947772640 1 --2- 172.21.15.27:0/1565576233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f194010a420 0x7f19401a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.059+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 >> 172.21.15.27:0/1565576233 conn(0x7f19400704e0 msgr2=0x7f194010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.059+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 shutdown_connections 2024-04-25T13:45:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.059+0000 7f1947772640 1 -- 172.21.15.27:0/1565576233 wait complete. 2024-04-25T13:45:47.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:45:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.238+0000 7f7096634640 1 -- 172.21.15.27:0/3192069857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70901029e0 msgr2=0x7f70901036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.238+0000 7f7096634640 1 --2- 172.21.15.27:0/3192069857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70901029e0 0x7f70901036e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f708000dbf0 tx=0x7f708002e2a0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.239+0000 7f7096634640 1 -- 172.21.15.27:0/3192069857 shutdown_connections 2024-04-25T13:45:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.239+0000 7f7096634640 1 --2- 172.21.15.27:0/3192069857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7090108920 0x7f709010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.239+0000 7f7096634640 1 --2- 172.21.15.27:0/3192069857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7090103c20 0x7f7090104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.239+0000 7f7096634640 1 --2- 172.21.15.27:0/3192069857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70901029e0 0x7f70901036e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.239+0000 7f7096634640 1 -- 172.21.15.27:0/3192069857 >> 172.21.15.27:0/3192069857 conn(0x7f70900fdbd0 msgr2=0x7f7090100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.240+0000 7f7096634640 1 -- 172.21.15.27:0/3192069857 shutdown_connections 2024-04-25T13:45:47.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.240+0000 7f7096634640 1 -- 172.21.15.27:0/3192069857 wait complete. 2024-04-25T13:45:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.241+0000 7f7096634640 1 Processor -- start 2024-04-25T13:45:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.241+0000 7f7096634640 1 -- start start 2024-04-25T13:45:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.241+0000 7f7096634640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70901029e0 0x7f70901a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.241+0000 7f7096634640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7090103c20 0x7f70901a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.241+0000 7f7096634640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7090108920 0x7f70901a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7096634640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f709010d3a0 con 0x7f70901029e0 2024-04-25T13:45:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7096634640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f709010d220 con 0x7f7090103c20 2024-04-25T13:45:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7096634640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f709010d520 con 0x7f7090108920 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7090103c20 0x7f70901a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70901029e0 0x7f70901a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7090103c20 0x7f70901a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:32982/0 (socket says 172.21.15.27:32982) 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708f7fe640 1 -- 172.21.15.27:0/1027144411 learned_addr learned my addr 172.21.15.27:0/1027144411 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7094bd1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7090108920 0x7f70901a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70901029e0 0x7f70901a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35584/0 (socket says 172.21.15.27:35584) 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7094bd1640 1 -- 172.21.15.27:0/1027144411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7090103c20 msgr2=0x7f70901a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7094bd1640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7090103c20 0x7f70901a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7094bd1640 1 -- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70901029e0 msgr2=0x7f70901a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7094bd1640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70901029e0 0x7f70901a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7094bd1640 1 -- 172.21.15.27:0/1027144411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70901a8610 con 0x7f7090108920 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708f7fe640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7090103c20 0x7f70901a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7094bd1640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7090108920 0x7f70901a7fa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7088007b20 tx=0x7f708800bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708d7fa640 1 -- 172.21.15.27:0/1027144411 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088019070 con 0x7f7090108920 2024-04-25T13:45:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708d7fa640 1 -- 172.21.15.27:0/1027144411 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f708800b230 con 0x7f7090108920 2024-04-25T13:45:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f7096634640 1 -- 172.21.15.27:0/1027144411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70901a8900 con 0x7f7090108920 2024-04-25T13:45:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.242+0000 7f708d7fa640 1 -- 172.21.15.27:0/1027144411 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088004020 con 0x7f7090108920 2024-04-25T13:45:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.243+0000 7f7096634640 1 -- 172.21.15.27:0/1027144411 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70901a8e00 con 0x7f7090108920 2024-04-25T13:45:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.245+0000 7f708d7fa640 1 -- 172.21.15.27:0/1027144411 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f70880041c0 con 0x7f7090108920 2024-04-25T13:45:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.246+0000 7f709d5c1740 1 Processor -- start 2024-04-25T13:45:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.246+0000 7f709d5c1740 1 -- start start 2024-04-25T13:45:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.246+0000 7f708d7fa640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70780afb70 0x7f70780b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.246+0000 7f708d7fa640 1 -- 172.21.15.27:0/1027144411 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f70880deb90 con 0x7f7090108920 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.247+0000 7f708f7fe640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70780afb70 0x7f70780b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.247+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d60390 0x55d5a6102810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.247+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6102d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.247+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6103a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.247+0000 7f709d5c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d5a6066d80 con 0x55d5a5d374d0 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.247+0000 7f709d5c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d5a6020a20 con 0x55d5a5d60390 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.247+0000 7f709d5c1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d5a6062d40 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.247+0000 7f7094bd1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f7094bd1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33896/0 (socket says 172.21.15.27:33896) 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f7094bd1640 1 -- 172.21.15.27:0/3334936091 learned_addr learned my addr 172.21.15.27:0/3334936091 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:47.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f708ffff640 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d60390 0x55d5a6102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f708f7fe640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70780afb70 0x7f70780b2080 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f70901a1b70 tx=0x7f7084009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f708f7fe640 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f7094bd1640 1 -- 172.21.15.27:0/3334936091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d60390 msgr2=0x55d5a6102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f7094bd1640 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d60390 0x55d5a6102810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f7094bd1640 1 -- 172.21.15.27:0/3334936091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 msgr2=0x55d5a6102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f7094bd1640 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6102d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f7094bd1640 1 -- 172.21.15.27:0/3334936091 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5a602bc30 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f7094bd1640 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6103a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f70880a42b0 tx=0x7f70880a6c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae1909f7a35947d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f70767fc640 1 -- 172.21.15.27:0/3334936091 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70880ab3f0 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f70767fc640 1 -- 172.21.15.27:0/3334936091 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70880afea0 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.248+0000 7f70767fc640 1 -- 172.21.15.27:0/3334936091 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708800fe30 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.249+0000 7f709d5c1740 1 -- 172.21.15.27:0/3334936091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d3f6b0 msgr2=0x55d5a6103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.249+0000 7f709d5c1740 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6103a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f70880a42b0 tx=0x7f70880a6c10 comp rx=0 tx=0).stop 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.250+0000 7f709d5c1740 1 -- 172.21.15.27:0/3334936091 shutdown_connections 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.250+0000 7f709d5c1740 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6103a20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.250+0000 7f709d5c1740 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.250+0000 7f709d5c1740 1 --2- 172.21.15.27:0/3334936091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d60390 0x55d5a6102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.250+0000 7f709d5c1740 1 -- 172.21.15.27:0/3334936091 >> 172.21.15.27:0/3334936091 conn(0x55d5a5d544e0 msgr2=0x55d5a60fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.250+0000 7f709d5c1740 1 -- 172.21.15.27:0/3334936091 shutdown_connections 2024-04-25T13:45:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.250+0000 7f709d5c1740 1 -- 172.21.15.27:0/3334936091 wait complete. 2024-04-25T13:45:47.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.251+0000 7f709d5c1740 1 -- start start 2024-04-25T13:45:47.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.251+0000 7f709d5c1740 1 Processor -- start 2024-04-25T13:45:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a610c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f708ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f708ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35596/0 (socket says 172.21.15.27:35596) 2024-04-25T13:45:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6113820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f709d5c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d5a6104530 con 0x55d5a5d374d0 2024-04-25T13:45:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f709d5c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d5a5feddc0 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f709d5c1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d5a61046b0 con 0x55d5a5d60390 2024-04-25T13:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f708f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a610c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f709d5c1740 10 client.0 _ll_register_callbacks cb 0x55d5a6009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f708f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a610c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33002/0 (socket says 172.21.15.27:33002) 2024-04-25T13:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f708f7fe640 1 -- 172.21.15.27:0/2160943481 learned_addr learned my addr 172.21.15.27:0/2160943481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.252+0000 7f7094bd1640 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f7094bd1640 1 -- 172.21.15.27:0/2160943481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 msgr2=0x55d5a610c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f7094bd1640 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a610c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f7094bd1640 1 -- 172.21.15.27:0/2160943481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 msgr2=0x55d5a610c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f7094bd1640 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f7094bd1640 1 -- 172.21.15.27:0/2160943481 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5a5d783f0 con 0x55d5a5d60390 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f7094bd1640 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6113820 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f70880a42b0 tx=0x7f70880a36b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff5b981a2f535f3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f706f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f706f7fe640 1 -- 172.21.15.27:0/2160943481 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70880c5030 con 0x55d5a5d60390 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f706f7fe640 1 -- 172.21.15.27:0/2160943481 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f70880c1050 con 0x55d5a5d60390 2024-04-25T13:45:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f709d5c1740 1 -- 172.21.15.27:0/2160943481 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70880afea0 con 0x55d5a5d60390 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f706f7fe640 1 -- 172.21.15.27:0/2160943481 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f70880ab3f0 con 0x55d5a5d60390 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f709d5c1740 10 client.26044 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.253+0000 7f709d5c1740 1 -- 172.21.15.27:0/2160943481 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d5a5e427d0 con 0x55d5a5d60390 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f706f7fe640 1 -- 172.21.15.27:0/2160943481 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70880b2020 con 0x55d5a5d60390 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f706f7fe640 1 -- 172.21.15.27:0/2160943481 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f70880c9070 con 0x55d5a5d60390 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f709d5c1740 10 client.26044 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f709d5c1740 10 client.26044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f709d5c1740 20 client.26044 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d5a5d1ef60 0x55d5a6119c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f709d5c1740 4 client.26044 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f709d5c1740 1 -- 172.21.15.27:0/2160943481 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5a6074990 con 0x55d5a5d1ef60 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.254+0000 7f708ffff640 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d5a5d1ef60 0x55d5a6119c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.255+0000 7f706f7fe640 10 client.26044 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:47.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.255+0000 7f708ffff640 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d5a5d1ef60 0x55d5a6119c70 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ecdc555538d0fa server_cookie=57cded0741d2bb55 in_seq=0 out_seq=0 2024-04-25T13:45:47.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.255+0000 7f706f7fe640 1 -- 172.21.15.27:0/2160943481 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d5a6074990 con 0x55d5a5d1ef60 2024-04-25T13:45:47.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.255+0000 7f706f7fe640 10 client.26044 handle_command_reply: tid=0 2024-04-25T13:45:47.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.255+0000 7f709d5c1740 1 client.26044 shutdown 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 -- 172.21.15.27:0/2160943481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 msgr2=0x55d5a6113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6113820 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f70880a42b0 tx=0x7f70880a36b0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 -- 172.21.15.27:0/2160943481 shutdown_connections 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d5a5d1ef60 0x55d5a6119c70 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6113820 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a610c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2160943481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 -- 172.21.15.27:0/2160943481 >> 172.21.15.27:0/2160943481 conn(0x55d5a5d544e0 msgr2=0x55d5a60fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f706f7fe640 0 client.26044 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 -- 172.21.15.27:0/2160943481 shutdown_connections 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 1 -- 172.21.15.27:0/2160943481 wait complete. 2024-04-25T13:45:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.256+0000 7f709d5c1740 20 client.26044 trim_cache size 0 max 16384 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f709d5c1740 1 Processor -- start 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f709d5c1740 1 -- start start 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6113740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a610b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f709d5c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f70901af6e0 con 0x55d5a5d374d0 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f709d5c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70901b1f60 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f709d5c1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f70901afd50 con 0x55d5a5d60390 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f708ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f708f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6113740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f708f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6113740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33010/0 (socket says 172.21.15.27:33010) 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f708ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35612/0 (socket says 172.21.15.27:35612) 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f708f7fe640 1 -- 172.21.15.27:0/3495646293 learned_addr learned my addr 172.21.15.27:0/3495646293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.278+0000 7f7094bd1640 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a610b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f708f7fe640 1 -- 172.21.15.27:0/3495646293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 msgr2=0x55d5a610b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f708f7fe640 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a610b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f708f7fe640 1 -- 172.21.15.27:0/3495646293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 msgr2=0x55d5a610acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f708f7fe640 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f708f7fe640 1 -- 172.21.15.27:0/3495646293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5a6113c80 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f708ffff640 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f708f7fe640 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6113740 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f708400f830 tx=0x7f7084014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16b9e3bab144de05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f7075ffb640 1 -- 172.21.15.27:0/3495646293 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708402d070 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f7075ffb640 1 -- 172.21.15.27:0/3495646293 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f708401bd20 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.279+0000 7f7075ffb640 1 -- 172.21.15.27:0/3495646293 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7084026430 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.280+0000 7f709d5c1740 1 -- 172.21.15.27:0/3495646293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 msgr2=0x55d5a6113740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6113740 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f708400f830 tx=0x7f7084014000 comp rx=0 tx=0).stop 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 -- 172.21.15.27:0/3495646293 shutdown_connections 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a610b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6113740 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 --2- 172.21.15.27:0/3495646293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a610acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 -- 172.21.15.27:0/3495646293 >> 172.21.15.27:0/3495646293 conn(0x55d5a5d544e0 msgr2=0x55d5a61070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 -- 172.21.15.27:0/3495646293 shutdown_connections 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 -- 172.21.15.27:0/3495646293 wait complete. 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 -- start start 2024-04-25T13:45:47.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.281+0000 7f709d5c1740 1 Processor -- start 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6113610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6110200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f709d5c1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6110780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f709d5c1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f70901b0420 con 0x55d5a5d374d0 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f709d5c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f70901b1890 con 0x55d5a5d3f6b0 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f709d5c1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f70901b0af0 con 0x55d5a5d60390 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6113610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35614/0 (socket says 172.21.15.27:35614) 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708ffff640 1 -- 172.21.15.27:0/2652835000 learned_addr learned my addr 172.21.15.27:0/2652835000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f709d5c1740 10 client.0 _ll_register_callbacks cb 0x7f708800b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708f7fe640 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6110200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f7094bd1640 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6110780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708ffff640 1 -- 172.21.15.27:0/2652835000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 msgr2=0x55d5a6110780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:47.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708ffff640 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6110780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708ffff640 1 -- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 msgr2=0x55d5a6110200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:47.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708ffff640 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6110200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.282+0000 7f708ffff640 1 -- 172.21.15.27:0/2652835000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5a617fe70 con 0x55d5a5d374d0 2024-04-25T13:45:47.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.283+0000 7f708f7fe640 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6110200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:47.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.283+0000 7f708ffff640 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6113610 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7080038040 tx=0x7f708000d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400b820166db5f94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.284+0000 7f706f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.284+0000 7f706f7fe640 1 -- 172.21.15.27:0/2652835000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080036840 con 0x55d5a5d374d0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.284+0000 7f706f7fe640 1 -- 172.21.15.27:0/2652835000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f7080004040 con 0x55d5a5d374d0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.284+0000 7f709d5c1740 1 -- 172.21.15.27:0/2652835000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f708401bd20 con 0x55d5a5d374d0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.285+0000 7f706f7fe640 1 -- 172.21.15.27:0/2652835000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f70800369e0 con 0x55d5a5d374d0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.285+0000 7f709d5c1740 10 client.16302 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.285+0000 7f709d5c1740 1 -- 172.21.15.27:0/2652835000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d5a6181100 con 0x55d5a5d374d0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.285+0000 7f706f7fe640 1 -- 172.21.15.27:0/2652835000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f708401bd20 con 0x55d5a5d374d0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.285+0000 7f706f7fe640 1 -- 172.21.15.27:0/2652835000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f7080004850 con 0x55d5a5d374d0 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f709d5c1740 10 client.16302 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f709d5c1740 10 client.16302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f709d5c1740 20 client.16302 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d5a5cf43a0 0x55d5a6183af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:47.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f709d5c1740 4 client.16302 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f709d5c1740 1 -- 172.21.15.27:0/2652835000 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d5a6103120 con 0x55d5a5cf43a0 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f708f7fe640 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d5a5cf43a0 0x55d5a6183af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f706f7fe640 10 client.16302 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.286+0000 7f708f7fe640 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d5a5cf43a0 0x55d5a6183af0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa8676d699b0d99c server_cookie=8cf33fcb37f7adcd in_seq=0 out_seq=0 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f706f7fe640 1 -- 172.21.15.27:0/2652835000 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d5a6103120 con 0x55d5a5cf43a0 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f706f7fe640 10 client.16302 handle_command_reply: tid=0 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 client.16302 shutdown 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 -- 172.21.15.27:0/2652835000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 msgr2=0x55d5a6113610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6113610 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7080038040 tx=0x7f708000d280 comp rx=0 tx=0).stop 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 -- 172.21.15.27:0/2652835000 shutdown_connections 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d5a5cf43a0 0x55d5a6183af0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d5a5d60390 0x55d5a6110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5a5d3f6b0 0x55d5a6110200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 --2- 172.21.15.27:0/2652835000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d5a5d374d0 0x55d5a6113610 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f706f7fe640 0 client.16302 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:47.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 -- 172.21.15.27:0/2652835000 >> 172.21.15.27:0/2652835000 conn(0x55d5a5d544e0 msgr2=0x55d5a610c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:47.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 -- 172.21.15.27:0/2652835000 shutdown_connections 2024-04-25T13:45:47.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 1 -- 172.21.15.27:0/2652835000 wait complete. 2024-04-25T13:45:47.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.287+0000 7f709d5c1740 20 client.16302 trim_cache size 0 max 16384 2024-04-25T13:45:47.290 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 -- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70780afb70 msgr2=0x7f70780b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70780afb70 0x7f70780b2080 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f70901a1b70 tx=0x7f7084009260 comp rx=0 tx=0).stop 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 -- 172.21.15.27:0/1027144411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7090108920 msgr2=0x7f70901a7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7090108920 0x7f70901a7fa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7088007b20 tx=0x7f708800bf60 comp rx=0 tx=0).stop 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 -- 172.21.15.27:0/1027144411 shutdown_connections 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f70780afb70 0x7f70780b2080 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7090108920 0x7f70901a7fa0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7090103c20 0x7f70901a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 --2- 172.21.15.27:0/1027144411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70901029e0 0x7f70901a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:47.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 -- 172.21.15.27:0/1027144411 >> 172.21.15.27:0/1027144411 conn(0x7f70900fdbd0 msgr2=0x7f7090104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:47.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 -- 172.21.15.27:0/1027144411 shutdown_connections 2024-04-25T13:45:47.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:47.288+0000 7f7096634640 1 -- 172.21.15.27:0/1027144411 wait complete. 2024-04-25T13:45:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 280.0 s, 2722.2 tps, lat 36.676 ms stddev 38.685 2024-04-25T13:45:48.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:47 smithi027 ceph-mon[29140]: pgmap v496: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-25T13:45:48.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:47 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1565576233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:48.301 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:45:48.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:45:48.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:45:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:47 smithi122 ceph-mon[32843]: pgmap v496: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-25T13:45:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:47 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1565576233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:48.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:47 smithi045 ceph-mon[32731]: pgmap v496: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-25T13:45:48.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:47 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1565576233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:48.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.481+0000 7fc6e07c5640 1 -- 172.21.15.27:0/2605890473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6d810a420 msgr2=0x7fc6d810add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.481+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/2605890473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6d810a420 0x7fc6d810add0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009900 tx=0x7fc6d402a420 comp rx=0 tx=0).stop 2024-04-25T13:45:48.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.483+0000 7fc6e07c5640 1 -- 172.21.15.27:0/2605890473 shutdown_connections 2024-04-25T13:45:48.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.483+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/2605890473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d8110030 0x7fc6d8110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.483+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/2605890473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6d810b310 0x7fc6d810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.483+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/2605890473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6d810a420 0x7fc6d810add0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.483+0000 7fc6e07c5640 1 -- 172.21.15.27:0/2605890473 >> 172.21.15.27:0/2605890473 conn(0x7fc6d80704e0 msgr2=0x7fc6d8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.488+0000 7fc6e07c5640 1 -- 172.21.15.27:0/2605890473 shutdown_connections 2024-04-25T13:45:48.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.490+0000 7fc6e07c5640 1 -- 172.21.15.27:0/2605890473 wait complete. 2024-04-25T13:45:48.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.491+0000 7fc6e07c5640 1 Processor -- start 2024-04-25T13:45:48.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.492+0000 7fc6e07c5640 1 -- start start 2024-04-25T13:45:48.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.492+0000 7fc6e07c5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d810a420 0x7fc6d8117120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.492+0000 7fc6e07c5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6d810b310 0x7fc6d8117660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.493+0000 7fc6de561640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d810a420 0x7fc6d8117120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.493+0000 7fc6de561640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d810a420 0x7fc6d8117120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33020/0 (socket says 172.21.15.27:33020) 2024-04-25T13:45:48.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.493+0000 7fc6e07c5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6d8110030 0x7fc6d81185c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.493+0000 7fc6e07c5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d811e5c0 con 0x7fc6d810b310 2024-04-25T13:45:48.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.493+0000 7fc6e07c5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d811e440 con 0x7fc6d810a420 2024-04-25T13:45:48.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.493+0000 7fc6e07c5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d811e740 con 0x7fc6d8110030 2024-04-25T13:45:48.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.493+0000 7fc6de561640 1 -- 172.21.15.27:0/262106186 learned_addr learned my addr 172.21.15.27:0/262106186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.493+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6d810b310 0x7fc6d8117660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.494+0000 7fc6de561640 1 -- 172.21.15.27:0/262106186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6d8110030 msgr2=0x7fc6d81185c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:45:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.494+0000 7fc6de561640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6d8110030 0x7fc6d81185c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.494+0000 7fc6de561640 1 -- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6d810b310 msgr2=0x7fc6d8117660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.494+0000 7fc6de561640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6d810b310 0x7fc6d8117660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.494+0000 7fc6de561640 1 -- 172.21.15.27:0/262106186 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d8118c30 con 0x7fc6d810a420 2024-04-25T13:45:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.494+0000 7fc6de561640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d810a420 0x7fc6d8117120 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4031440 tx=0x7fc6d4003e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.495+0000 7fc6cf7fe640 1 -- 172.21.15.27:0/262106186 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d403f070 con 0x7fc6d810a420 2024-04-25T13:45:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.496+0000 7fc6cf7fe640 1 -- 172.21.15.27:0/262106186 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6d403b460 con 0x7fc6d810a420 2024-04-25T13:45:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.496+0000 7fc6cf7fe640 1 -- 172.21.15.27:0/262106186 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4035030 con 0x7fc6d810a420 2024-04-25T13:45:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.496+0000 7fc6e07c5640 1 -- 172.21.15.27:0/262106186 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d8118ec0 con 0x7fc6d810a420 2024-04-25T13:45:48.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.496+0000 7fc6e07c5640 1 -- 172.21.15.27:0/262106186 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d81193e0 con 0x7fc6d810a420 2024-04-25T13:45:48.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.501+0000 7fc6e767a740 1 Processor -- start 2024-04-25T13:45:48.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6e767a740 1 -- start start 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9dd34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9dd3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9dda920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6e767a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ecb9d38b00 con 0x55ecb9a09550 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6e767a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ecb9cd4920 con 0x55ecb9a11730 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6e767a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ecb9d34ac0 con 0x55ecb9a32410 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6ded62640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9dd34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6ded62640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9dd34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33940/0 (socket says 172.21.15.27:33940) 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.502+0000 7fc6ded62640 1 -- 172.21.15.27:0/136364818 learned_addr learned my addr 172.21.15.27:0/136364818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.503+0000 7fc6ded62640 1 -- 172.21.15.27:0/136364818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 msgr2=0x55ecb9dda920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.503+0000 7fc6ded62640 1 --2- 172.21.15.27:0/136364818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9dda920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.503+0000 7fc6ded62640 1 -- 172.21.15.27:0/136364818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 msgr2=0x55ecb9dd3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.503+0000 7fc6ded62640 1 --2- 172.21.15.27:0/136364818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9dd3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.503+0000 7fc6ded62640 1 -- 172.21.15.27:0/136364818 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb9b127e0 con 0x55ecb9a32410 2024-04-25T13:45:48.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.504+0000 7fc6cf7fe640 1 -- 172.21.15.27:0/262106186 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc6d4042070 con 0x7fc6d810a420 2024-04-25T13:45:48.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.504+0000 7fc6ded62640 1 --2- 172.21.15.27:0/136364818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9dd34a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80096f0 tx=0x7fc6c8009bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=501ce802f315f2a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.504+0000 7fc6ccff9640 1 -- 172.21.15.27:0/136364818 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8015020 con 0x55ecb9a32410 2024-04-25T13:45:48.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.504+0000 7fc6ccff9640 1 -- 172.21.15.27:0/136364818 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6c800bd80 con 0x55ecb9a32410 2024-04-25T13:45:48.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.504+0000 7fc6ccff9640 1 -- 172.21.15.27:0/136364818 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8010380 con 0x55ecb9a32410 2024-04-25T13:45:48.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.505+0000 7fc6cf7fe640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc6c40afd00 0x7fc6c40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.505+0000 7fc6cf7fe640 1 -- 172.21.15.27:0/262106186 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc6d40ff660 con 0x7fc6d810a420 2024-04-25T13:45:48.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.505+0000 7fc6e767a740 1 -- 172.21.15.27:0/136364818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 msgr2=0x55ecb9dd34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.505+0000 7fc6e767a740 1 --2- 172.21.15.27:0/136364818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9dd34a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80096f0 tx=0x7fc6c8009bb0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.506+0000 7fc6e767a740 1 -- 172.21.15.27:0/136364818 shutdown_connections 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.506+0000 7fc6e767a740 1 --2- 172.21.15.27:0/136364818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9dda920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.506+0000 7fc6e767a740 1 --2- 172.21.15.27:0/136364818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9dd3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.506+0000 7fc6e767a740 1 --2- 172.21.15.27:0/136364818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9dd34a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.506+0000 7fc6e767a740 1 -- 172.21.15.27:0/136364818 >> 172.21.15.27:0/136364818 conn(0x55ecb9a26560 msgr2=0x55ecb9dcee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.506+0000 7fc6de561640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc6c40afd00 0x7fc6c40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.506+0000 7fc6e767a740 1 -- 172.21.15.27:0/136364818 shutdown_connections 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.506+0000 7fc6e767a740 1 -- 172.21.15.27:0/136364818 wait complete. 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.507+0000 7fc6e767a740 1 -- start start 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.507+0000 7fc6de561640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc6c40afd00 0x7fc6c40b2210 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40c7810 tx=0x7fc6d40c3a00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.507+0000 7fc6e767a740 1 Processor -- start 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9ddd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9ddd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9de4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6e767a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ecb9ddb370 con 0x55ecb9a09550 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6e767a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ecb9cd25d0 con 0x55ecb9a11730 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6e767a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ecb9ddb4f0 con 0x55ecb9a32410 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6e767a740 10 client.0 _ll_register_callbacks cb 0x55ecb9cc0080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ddd60640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9ddd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ddd60640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9ddd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33032/0 (socket says 172.21.15.27:33032) 2024-04-25T13:45:48.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ddd60640 1 -- 172.21.15.27:0/1042284944 learned_addr learned my addr 172.21.15.27:0/1042284944 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ded62640 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9ddd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ded62640 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9de4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ddd60640 1 -- 172.21.15.27:0/1042284944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 msgr2=0x55ecb9de4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9de4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ddd60640 1 -- 172.21.15.27:0/1042284944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 msgr2=0x55ecb9ddd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9ddd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.508+0000 7fc6ddd60640 1 -- 172.21.15.27:0/1042284944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb9b72f70 con 0x55ecb9a11730 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.509+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9ddd810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c9f0 tx=0x7fc6d000ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a095df489c97565e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.509+0000 7fc6c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.509+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1042284944 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d001a060 con 0x55ecb9a11730 2024-04-25T13:45:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.509+0000 7fc6e767a740 1 -- 172.21.15.27:0/1042284944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6c800bd80 con 0x55ecb9a11730 2024-04-25T13:45:48.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.509+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1042284944 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc6d0007950 con 0x55ecb9a11730 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.511+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1042284944 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc6d0007b50 con 0x55ecb9a11730 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.511+0000 7fc6e767a740 10 client.26027 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.511+0000 7fc6e767a740 1 -- 172.21.15.27:0/1042284944 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ecb9a0a170 con 0x55ecb9a11730 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.511+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1042284944 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6c800bd80 con 0x55ecb9a11730 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.512+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1042284944 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc6d0015cd0 con 0x55ecb9a11730 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.512+0000 7fc6e767a740 10 client.26027 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.512+0000 7fc6e767a740 10 client.26027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.512+0000 7fc6e767a740 20 client.26027 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.512+0000 7fc6e767a740 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ecb99f11f0 0x55ecb9deace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.512+0000 7fc6e767a740 4 client.26027 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.512+0000 7fc6e767a740 1 -- 172.21.15.27:0/1042284944 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecb9d46710 con 0x55ecb99f11f0 2024-04-25T13:45:48.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.513+0000 7fc6ded62640 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ecb99f11f0 0x55ecb9deace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.513+0000 7fc6ded62640 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ecb99f11f0 0x55ecb9deace0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e75444a07d7db751 server_cookie=a02e05fc5a606056 in_seq=0 out_seq=0 2024-04-25T13:45:48.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.514+0000 7fc6c1ffb640 10 client.26027 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/1042284944 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ecb9d46710 con 0x55ecb99f11f0 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6c1ffb640 10 client.26027 handle_command_reply: tid=0 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 client.26027 shutdown 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 -- 172.21.15.27:0/1042284944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 msgr2=0x55ecb9ddd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9ddd810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c9f0 tx=0x7fc6d000ceb0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 -- 172.21.15.27:0/1042284944 shutdown_connections 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ecb99f11f0 0x55ecb9deace0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9de4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9ddd810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 --2- 172.21.15.27:0/1042284944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9ddd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6e767a740 1 -- 172.21.15.27:0/1042284944 >> 172.21.15.27:0/1042284944 conn(0x55ecb9a26560 msgr2=0x55ecb9dcdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.516+0000 7fc6c1ffb640 0 client.26027 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:48.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.517+0000 7fc6e767a740 1 -- 172.21.15.27:0/1042284944 shutdown_connections 2024-04-25T13:45:48.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.517+0000 7fc6e767a740 1 -- 172.21.15.27:0/1042284944 wait complete. 2024-04-25T13:45:48.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.517+0000 7fc6e767a740 20 client.26027 trim_cache size 0 max 16384 2024-04-25T13:45:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.540+0000 7fc6e767a740 1 Processor -- start 2024-04-25T13:45:48.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.540+0000 7fc6e767a740 1 -- start start 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9dd7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9dd23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6ded62640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9dd7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6ded62640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9dd7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35654/0 (socket says 172.21.15.27:35654) 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6ded62640 1 -- 172.21.15.27:0/803371362 learned_addr learned my addr 172.21.15.27:0/803371362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9dd23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6e767a740 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9df04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6e767a740 1 -- 172.21.15.27:0/803371362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc00c390 con 0x55ecb9a09550 2024-04-25T13:45:48.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6e767a740 1 -- 172.21.15.27:0/803371362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc00ec10 con 0x55ecb9a11730 2024-04-25T13:45:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.541+0000 7fc6e767a740 1 -- 172.21.15.27:0/803371362 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc00ca00 con 0x55ecb9a32410 2024-04-25T13:45:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.542+0000 7fc6ded62640 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9df04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.542+0000 7fc6ddd60640 1 -- 172.21.15.27:0/803371362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 msgr2=0x55ecb9df04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.542+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9df04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.542+0000 7fc6ddd60640 1 -- 172.21.15.27:0/803371362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 msgr2=0x55ecb9dd7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.542+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9dd7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.542+0000 7fc6ddd60640 1 -- 172.21.15.27:0/803371362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb9dd46a0 con 0x55ecb9a11730 2024-04-25T13:45:48.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.542+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9dd23f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0002230 tx=0x7fc6d0007000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5231fb1ad5f2c1b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.543+0000 7fc6c3fff640 1 -- 172.21.15.27:0/803371362 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0025070 con 0x55ecb9a11730 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.543+0000 7fc6c3fff640 1 -- 172.21.15.27:0/803371362 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6d00098c0 con 0x55ecb9a11730 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.543+0000 7fc6c3fff640 1 -- 172.21.15.27:0/803371362 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0021690 con 0x55ecb9a11730 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 -- 172.21.15.27:0/803371362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 msgr2=0x55ecb9dd23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9dd23f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0002230 tx=0x7fc6d0007000 comp rx=0 tx=0).stop 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 -- 172.21.15.27:0/803371362 shutdown_connections 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a32410 0x55ecb9df04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9dd23f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 --2- 172.21.15.27:0/803371362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a09550 0x55ecb9dd7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 -- 172.21.15.27:0/803371362 >> 172.21.15.27:0/803371362 conn(0x55ecb9a26560 msgr2=0x55ecb9dd1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 -- 172.21.15.27:0/803371362 shutdown_connections 2024-04-25T13:45:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 -- 172.21.15.27:0/803371362 wait complete. 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 -- start start 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.544+0000 7fc6e767a740 1 Processor -- start 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a09550 0x55ecb9de1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9de24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6e767a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a32410 0x55ecb9e51080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6e767a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc00d0d0 con 0x55ecb9a32410 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6e767a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc00e540 con 0x55ecb9a11730 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ddd60640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9de24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ddd60640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9de24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33052/0 (socket says 172.21.15.27:33052) 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ddd60640 1 -- 172.21.15.27:0/151756983 learned_addr learned my addr 172.21.15.27:0/151756983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ded62640 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a09550 0x55ecb9de1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc00d7a0 con 0x55ecb9a09550 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ded62640 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a32410 0x55ecb9e51080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6e767a740 10 client.0 _ll_register_callbacks cb 0x7fc6d000c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ddd60640 1 -- 172.21.15.27:0/151756983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a09550 msgr2=0x55ecb9de1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a09550 0x55ecb9de1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ddd60640 1 -- 172.21.15.27:0/151756983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a32410 msgr2=0x55ecb9e51080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a32410 0x55ecb9e51080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.545+0000 7fc6ddd60640 1 -- 172.21.15.27:0/151756983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb9e515c0 con 0x55ecb9a11730 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6ddd60640 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9de24a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c760 tx=0x7fc6d0009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e628d44c6c7441d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/151756983 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0025070 con 0x55ecb9a11730 2024-04-25T13:45:48.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/151756983 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc6d0016480 con 0x55ecb9a11730 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6d00098c0 con 0x55ecb9a11730 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/151756983 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc6d0038070 con 0x55ecb9a11730 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6e767a740 10 client.26033 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ecb9e52780 con 0x55ecb9a11730 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.546+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/151756983 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6d0007680 con 0x55ecb9a11730 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.547+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/151756983 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc6d0030030 con 0x55ecb9a11730 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.547+0000 7fc6e767a740 10 client.26033 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.547+0000 7fc6e767a740 10 client.26033 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.547+0000 7fc6e767a740 20 client.26033 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.547+0000 7fc6e767a740 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ecb99c6630 0x55ecb9e55090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.547+0000 7fc6e767a740 4 client.26033 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.547+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fc6d00098c0 con 0x55ecb99c6630 2024-04-25T13:45:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.547+0000 7fc6ded62640 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ecb99c6630 0x55ecb9e55090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.548+0000 7fc6ded62640 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ecb99c6630 0x55ecb9e55090 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b71345edc8724db server_cookie=76137497964afba7 in_seq=0 out_seq=0 2024-04-25T13:45:48.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.548+0000 7fc6c1ffb640 10 client.26033 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:48.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6c1ffb640 1 -- 172.21.15.27:0/151756983 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fc6d00098c0 con 0x55ecb99c6630 2024-04-25T13:45:48.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6c1ffb640 10 client.26033 handle_command_reply: tid=0 2024-04-25T13:45:48.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 client.26033 shutdown 2024-04-25T13:45:48.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 msgr2=0x55ecb9de24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9de24a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000c760 tx=0x7fc6d0009790 comp rx=0 tx=0).stop 2024-04-25T13:45:48.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 shutdown_connections 2024-04-25T13:45:48.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ecb99c6630 0x55ecb9e55090 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ecb9a32410 0x55ecb9e51080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ecb9a11730 0x55ecb9de24a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6c1ffb640 0 client.26033 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:48.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 --2- 172.21.15.27:0/151756983 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ecb9a09550 0x55ecb9de1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 >> 172.21.15.27:0/151756983 conn(0x55ecb9a26560 msgr2=0x55ecb9ddc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 shutdown_connections 2024-04-25T13:45:48.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 1 -- 172.21.15.27:0/151756983 wait complete. 2024-04-25T13:45:48.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.550+0000 7fc6e767a740 20 client.26033 trim_cache size 0 max 16384 2024-04-25T13:45:48.555 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:45:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:45:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "4c91d8bf-24ef-4cdb-9d12-6cfbdaee0a2f", 2024-04-25T13:45:48.556 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:45:48.556 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:45:48.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 -- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc6c40afd00 msgr2=0x7fc6c40b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc6c40afd00 0x7fc6c40b2210 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40c7810 tx=0x7fc6d40c3a00 comp rx=0 tx=0).stop 2024-04-25T13:45:48.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 -- 172.21.15.27:0/262106186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d810a420 msgr2=0x7fc6d8117120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d810a420 0x7fc6d8117120 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4031440 tx=0x7fc6d4003e30 comp rx=0 tx=0).stop 2024-04-25T13:45:48.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 -- 172.21.15.27:0/262106186 shutdown_connections 2024-04-25T13:45:48.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc6c40afd00 0x7fc6c40b2210 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc6d8110030 0x7fc6d81185c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6d810b310 0x7fc6d8117660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 --2- 172.21.15.27:0/262106186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6d810a420 0x7fc6d8117120 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 -- 172.21.15.27:0/262106186 >> 172.21.15.27:0/262106186 conn(0x7fc6d80704e0 msgr2=0x7fc6d810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 -- 172.21.15.27:0/262106186 shutdown_connections 2024-04-25T13:45:48.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.551+0000 7fc6e07c5640 1 -- 172.21.15.27:0/262106186 wait complete. 2024-04-25T13:45:48.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:45:48.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:45:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 -- 172.21.15.27:0/399279578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c1098a0 msgr2=0x7f850c10bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 --2- 172.21.15.27:0/399279578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c1098a0 0x7f850c10bcc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8504009ca0 tx=0x7f850402a8b0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 -- 172.21.15.27:0/399279578 shutdown_connections 2024-04-25T13:45:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 --2- 172.21.15.27:0/399279578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c1098a0 0x7f850c10bcc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 --2- 172.21.15.27:0/399279578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f850c106f40 0x7f850c109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 --2- 172.21.15.27:0/399279578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f850c1045e0 0x7f850c106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 -- 172.21.15.27:0/399279578 >> 172.21.15.27:0/399279578 conn(0x7f850c0fdbe0 msgr2=0x7f850c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 -- 172.21.15.27:0/399279578 shutdown_connections 2024-04-25T13:45:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.736+0000 7f8512bf3640 1 -- 172.21.15.27:0/399279578 wait complete. 2024-04-25T13:45:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f8512bf3640 1 Processor -- start 2024-04-25T13:45:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f8512bf3640 1 -- start start 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f8512bf3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f850c1045e0 0x7f850c103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f8512bf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c106f40 0x7f850c1020d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f8512bf3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f850c1098a0 0x7f850c102610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f8512bf3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f850c10f600 con 0x7f850c1098a0 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f8512bf3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f850c10f480 con 0x7f850c106f40 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f8512bf3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f850c10f780 con 0x7f850c1045e0 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f850bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c106f40 0x7f850c1020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f850bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c106f40 0x7f850c1020d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33080/0 (socket says 172.21.15.27:33080) 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f850bfff640 1 -- 172.21.15.27:0/3440510172 learned_addr learned my addr 172.21.15.27:0/3440510172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.737+0000 7f851098f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f850c1045e0 0x7f850c103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f8511190640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f850c1098a0 0x7f850c102610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f851098f640 1 -- 172.21.15.27:0/3440510172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c106f40 msgr2=0x7f850c1020d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f851098f640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c106f40 0x7f850c1020d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f851098f640 1 -- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f850c1098a0 msgr2=0x7f850c102610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f851098f640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f850c1098a0 0x7f850c102610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f851098f640 1 -- 172.21.15.27:0/3440510172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c19e0c0 con 0x7f850c1045e0 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f851098f640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f850c1045e0 0x7f850c103a30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f850000d6a0 tx=0x7f850000db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f8509ffb640 1 -- 172.21.15.27:0/3440510172 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500017070 con 0x7f850c1045e0 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f8509ffb640 1 -- 172.21.15.27:0/3440510172 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8500009820 con 0x7f850c1045e0 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.738+0000 7f8512bf3640 1 -- 172.21.15.27:0/3440510172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850c102c70 con 0x7f850c1045e0 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.739+0000 7f8509ffb640 1 -- 172.21.15.27:0/3440510172 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500004c40 con 0x7f850c1045e0 2024-04-25T13:45:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.739+0000 7f8512bf3640 1 -- 172.21.15.27:0/3440510172 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850c1a8cc0 con 0x7f850c1045e0 2024-04-25T13:45:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.741+0000 7f8509ffb640 1 -- 172.21.15.27:0/3440510172 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8500005220 con 0x7f850c1045e0 2024-04-25T13:45:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.742+0000 7f8519b82740 1 Processor -- start 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.742+0000 7f8519b82740 1 -- start start 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8509ffb640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84fc0afaf0 0x7f84fc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8519b82740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367d1390 0x559436b72a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8519b82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367a84d0 0x559436b72fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8519b82740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b73c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8519b82740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559436ad7d80 con 0x5594367d1390 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8519b82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559436a918b0 con 0x5594367a84d0 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8519b82740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559436ad3d40 con 0x5594367b06b0 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8509ffb640 1 -- 172.21.15.27:0/3440510172 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f85000de3e0 con 0x7f850c1045e0 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f850bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367a84d0 0x559436b72fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f850bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367a84d0 0x559436b72fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33090/0 (socket says 172.21.15.27:33090) 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f850bfff640 1 -- 172.21.15.27:0/551814033 learned_addr learned my addr 172.21.15.27:0/551814033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f851098f640 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367d1390 0x559436b72a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8511190640 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b73c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f8511190640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84fc0afaf0 0x7f84fc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f850bfff640 1 -- 172.21.15.27:0/551814033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 msgr2=0x559436b73c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f850bfff640 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b73c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f850bfff640 1 -- 172.21.15.27:0/551814033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367d1390 msgr2=0x559436b72a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f850bfff640 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367d1390 0x559436b72a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f850bfff640 1 -- 172.21.15.27:0/551814033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559436a62a40 con 0x5594367a84d0 2024-04-25T13:45:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.743+0000 7f851098f640 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367d1390 0x559436b72a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.744+0000 7f850bfff640 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367a84d0 0x559436b72fe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009d70 tx=0x7f84f8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5eaada9acb7ff87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.744+0000 7f84f6ffd640 1 -- 172.21.15.27:0/551814033 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f801e070 con 0x5594367a84d0 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.744+0000 7f84f6ffd640 1 -- 172.21.15.27:0/551814033 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f8019ae0 con 0x5594367a84d0 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.744+0000 7f84f6ffd640 1 -- 172.21.15.27:0/551814033 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f800eb60 con 0x5594367a84d0 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.745+0000 7f8519b82740 1 -- 172.21.15.27:0/551814033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367a84d0 msgr2=0x559436b72fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.745+0000 7f8519b82740 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367a84d0 0x559436b72fe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009d70 tx=0x7f84f8011810 comp rx=0 tx=0).stop 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.746+0000 7f8511190640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84fc0afaf0 0x7f84fc0b2000 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8504030860 tx=0x7f8504030f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.746+0000 7f8519b82740 1 -- 172.21.15.27:0/551814033 shutdown_connections 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.746+0000 7f8519b82740 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b73c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.746+0000 7f8519b82740 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367a84d0 0x559436b72fe0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.746+0000 7f8519b82740 1 --2- 172.21.15.27:0/551814033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367d1390 0x559436b72a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.746+0000 7f8519b82740 1 -- 172.21.15.27:0/551814033 >> 172.21.15.27:0/551814033 conn(0x5594367c54e0 msgr2=0x559436b6e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.746+0000 7f8519b82740 1 -- 172.21.15.27:0/551814033 shutdown_connections 2024-04-25T13:45:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.746+0000 7f8519b82740 1 -- 172.21.15.27:0/551814033 wait complete. 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.747+0000 7f8519b82740 1 -- start start 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.747+0000 7f8519b82740 1 Processor -- start 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f8519b82740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 0x559436b7c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f8519b82740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b7cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f8519b82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b83b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f8519b82740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559436b74780 con 0x5594367a84d0 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f8519b82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559436a5ed10 con 0x5594367d1390 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f8519b82740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559436b74900 con 0x5594367b06b0 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b7cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f851098f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 0x559436b7c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f8519b82740 10 client.0 _ll_register_callbacks cb 0x559436a8b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b7cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:34004/0 (socket says 172.21.15.27:34004) 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 -- 172.21.15.27:0/2950291443 learned_addr learned my addr 172.21.15.27:0/2950291443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f8511190640 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b83b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 -- 172.21.15.27:0/2950291443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 msgr2=0x559436b83b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b83b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 -- 172.21.15.27:0/2950291443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 msgr2=0x559436b7c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 0x559436b7c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 -- 172.21.15.27:0/2950291443 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594368b37d0 con 0x5594367b06b0 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.748+0000 7f850bfff640 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b7cc30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f84f8004270 tx=0x7f84f80042a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e116462f765e0f9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.749+0000 7f84ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.749+0000 7f84ebfff640 1 -- 172.21.15.27:0/2950291443 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8027020 con 0x5594367b06b0 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.749+0000 7f8519b82740 1 -- 172.21.15.27:0/2950291443 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84f8019ae0 con 0x5594367b06b0 2024-04-25T13:45:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.749+0000 7f84ebfff640 1 -- 172.21.15.27:0/2950291443 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f84f8019630 con 0x5594367b06b0 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.749+0000 7f84ebfff640 1 -- 172.21.15.27:0/2950291443 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f84f801e040 con 0x5594367b06b0 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.749+0000 7f8519b82740 10 client.26077 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.749+0000 7f8519b82740 1 -- 172.21.15.27:0/2950291443 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5594367e93f0 con 0x5594367b06b0 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.749+0000 7f84ebfff640 1 -- 172.21.15.27:0/2950291443 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f8019ae0 con 0x5594367b06b0 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.750+0000 7f84ebfff640 1 -- 172.21.15.27:0/2950291443 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f84f8005170 con 0x5594367b06b0 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.750+0000 7f8519b82740 10 client.26077 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.750+0000 7f8519b82740 10 client.26077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.750+0000 7f8519b82740 20 client.26077 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.750+0000 7f8519b82740 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55943678ff60 0x559436b89d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.750+0000 7f8519b82740 4 client.26077 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.750+0000 7f8519b82740 1 -- 172.21.15.27:0/2950291443 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559436ae5990 con 0x55943678ff60 2024-04-25T13:45:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.750+0000 7f851098f640 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55943678ff60 0x559436b89d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.751+0000 7f851098f640 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55943678ff60 0x559436b89d40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26ff7f68840df902 server_cookie=569d7ede2173a59d in_seq=0 out_seq=0 2024-04-25T13:45:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.751+0000 7f84ebfff640 10 client.26077 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.752+0000 7f84ebfff640 1 -- 172.21.15.27:0/2950291443 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559436ae5990 con 0x55943678ff60 2024-04-25T13:45:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.752+0000 7f84ebfff640 10 client.26077 handle_command_reply: tid=0 2024-04-25T13:45:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.752+0000 7f8519b82740 1 client.26077 shutdown 2024-04-25T13:45:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f8519b82740 1 -- 172.21.15.27:0/2950291443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 msgr2=0x559436b7cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f8519b82740 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b7cc30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f84f8004270 tx=0x7f84f80042a0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f8519b82740 1 -- 172.21.15.27:0/2950291443 shutdown_connections 2024-04-25T13:45:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f8519b82740 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55943678ff60 0x559436b89d40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f8519b82740 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b83b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f8519b82740 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b7cc30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f8519b82740 1 --2- 172.21.15.27:0/2950291443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 0x559436b7c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f8519b82740 1 -- 172.21.15.27:0/2950291443 >> 172.21.15.27:0/2950291443 conn(0x5594367c54e0 msgr2=0x559436b73b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.753+0000 7f84ebfff640 0 client.26077 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.754+0000 7f8519b82740 1 -- 172.21.15.27:0/2950291443 shutdown_connections 2024-04-25T13:45:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.754+0000 7f8519b82740 1 -- 172.21.15.27:0/2950291443 wait complete. 2024-04-25T13:45:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.754+0000 7f8519b82740 20 client.26077 trim_cache size 0 max 16384 2024-04-25T13:45:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.782+0000 7f8519b82740 1 Processor -- start 2024-04-25T13:45:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.782+0000 7f8519b82740 1 -- start start 2024-04-25T13:45:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.782+0000 7f8519b82740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 0x559436b76730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.782+0000 7f8519b82740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b83ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.782+0000 7f8519b82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b83ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.782+0000 7f8519b82740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1af780 con 0x5594367a84d0 2024-04-25T13:45:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.782+0000 7f8519b82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1b2000 con 0x5594367d1390 2024-04-25T13:45:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.782+0000 7f8519b82740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1afdf0 con 0x5594367b06b0 2024-04-25T13:45:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f850bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b83ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f850bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b83ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:34006/0 (socket says 172.21.15.27:34006) 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f850bfff640 1 -- 172.21.15.27:0/1768737495 learned_addr learned my addr 172.21.15.27:0/1768737495 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f851098f640 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 0x559436b76730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f8511190640 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b83ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f850bfff640 1 -- 172.21.15.27:0/1768737495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 msgr2=0x559436b83ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f850bfff640 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b83ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f850bfff640 1 -- 172.21.15.27:0/1768737495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 msgr2=0x559436b76730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f850bfff640 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 0x559436b76730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.784+0000 7f850bfff640 1 -- 172.21.15.27:0/1768737495 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559436b7cdd0 con 0x5594367b06b0 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.785+0000 7f850bfff640 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b83ab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f84f80054e0 tx=0x7f84f8012f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65b37b21bde56cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.785+0000 7f84f67fc640 1 -- 172.21.15.27:0/1768737495 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8027040 con 0x5594367b06b0 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.785+0000 7f84f67fc640 1 -- 172.21.15.27:0/1768737495 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f8004f70 con 0x5594367b06b0 2024-04-25T13:45:48.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.785+0000 7f84f67fc640 1 -- 172.21.15.27:0/1768737495 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8003ac0 con 0x5594367b06b0 2024-04-25T13:45:48.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 -- 172.21.15.27:0/1768737495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 msgr2=0x559436b83ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b83ab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f84f80054e0 tx=0x7f84f8012f90 comp rx=0 tx=0).stop 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 -- 172.21.15.27:0/1768737495 shutdown_connections 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b83ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367b06b0 0x559436b83ab0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 --2- 172.21.15.27:0/1768737495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367a84d0 0x559436b76730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 -- 172.21.15.27:0/1768737495 >> 172.21.15.27:0/1768737495 conn(0x5594367c54e0 msgr2=0x559436b774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 -- 172.21.15.27:0/1768737495 shutdown_connections 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 -- 172.21.15.27:0/1768737495 wait complete. 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.786+0000 7f8519b82740 1 -- start start 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.787+0000 7f8519b82740 1 Processor -- start 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.787+0000 7f8519b82740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367a84d0 0x559436b840c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.787+0000 7f8519b82740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367b06b0 0x559436b80cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.787+0000 7f8519b82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b811f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.787+0000 7f8519b82740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1b04c0 con 0x5594367b06b0 2024-04-25T13:45:48.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.787+0000 7f8519b82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1b1930 con 0x5594367d1390 2024-04-25T13:45:48.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.787+0000 7f8519b82740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1b0b90 con 0x5594367a84d0 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.787+0000 7f8519b82740 10 client.0 _ll_register_callbacks cb 0x7f84f8009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f8511190640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b811f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f850bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367b06b0 0x559436b80cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f850bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367b06b0 0x559436b80cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35706/0 (socket says 172.21.15.27:35706) 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f850bfff640 1 -- 172.21.15.27:0/124620321 learned_addr learned my addr 172.21.15.27:0/124620321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f851098f640 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367a84d0 0x559436b840c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f851098f640 1 -- 172.21.15.27:0/124620321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 msgr2=0x559436b811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f851098f640 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b811f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f851098f640 1 -- 172.21.15.27:0/124620321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367b06b0 msgr2=0x559436b80cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f851098f640 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367b06b0 0x559436b80cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f851098f640 1 -- 172.21.15.27:0/124620321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594367a90f0 con 0x5594367a84d0 2024-04-25T13:45:48.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f8511190640 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b811f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.788+0000 7f850bfff640 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367b06b0 0x559436b80cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f851098f640 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367a84d0 0x559436b840c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f85000dbf60 tx=0x7f85000a2220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edffbd16a9b4ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f84ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f84ebfff640 1 -- 172.21.15.27:0/124620321 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500020040 con 0x5594367a84d0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f84ebfff640 1 -- 172.21.15.27:0/124620321 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8500004de0 con 0x5594367a84d0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f8519b82740 1 -- 172.21.15.27:0/124620321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84f8004f70 con 0x5594367a84d0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f84ebfff640 1 -- 172.21.15.27:0/124620321 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f85000a69a0 con 0x5594367a84d0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f8519b82740 10 client.26083 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f8519b82740 1 -- 172.21.15.27:0/124620321 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559436a984b0 con 0x5594367a84d0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f84ebfff640 1 -- 172.21.15.27:0/124620321 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f8004f70 con 0x5594367a84d0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.789+0000 7f84ebfff640 1 -- 172.21.15.27:0/124620321 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f85000a2c30 con 0x5594367a84d0 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.790+0000 7f8519b82740 10 client.26083 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.790+0000 7f8519b82740 10 client.26083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.790+0000 7f8519b82740 20 client.26083 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.790+0000 7f8519b82740 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5594367653a0 0x559436bf38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.790+0000 7f8519b82740 4 client.26083 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:45:48.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.790+0000 7f8519b82740 1 -- 172.21.15.27:0/124620321 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559436adf800 con 0x5594367653a0 2024-04-25T13:45:48.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.790+0000 7f850bfff640 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5594367653a0 0x559436bf38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.791+0000 7f850bfff640 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5594367653a0 0x559436bf38b0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f741dca02d1ab24 server_cookie=7051752726808dba in_seq=0 out_seq=0 2024-04-25T13:45:48.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.791+0000 7f84ebfff640 10 client.26083 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:48.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.791+0000 7f84ebfff640 1 -- 172.21.15.27:0/124620321 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559436adf800 con 0x5594367653a0 2024-04-25T13:45:48.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.791+0000 7f84ebfff640 10 client.26083 handle_command_reply: tid=0 2024-04-25T13:45:48.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.791+0000 7f8519b82740 1 client.26083 shutdown 2024-04-25T13:45:48.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 -- 172.21.15.27:0/124620321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367a84d0 msgr2=0x559436b840c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367a84d0 0x559436b840c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f85000dbf60 tx=0x7f85000a2220 comp rx=0 tx=0).stop 2024-04-25T13:45:48.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 -- 172.21.15.27:0/124620321 shutdown_connections 2024-04-25T13:45:48.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5594367653a0 0x559436bf38b0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f84ebfff640 0 client.26083 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:48.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594367d1390 0x559436b811f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5594367b06b0 0x559436b80cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 --2- 172.21.15.27:0/124620321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5594367a84d0 0x559436b840c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 -- 172.21.15.27:0/124620321 >> 172.21.15.27:0/124620321 conn(0x5594367c54e0 msgr2=0x559436b710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 -- 172.21.15.27:0/124620321 shutdown_connections 2024-04-25T13:45:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 1 -- 172.21.15.27:0/124620321 wait complete. 2024-04-25T13:45:48.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.792+0000 7f8519b82740 20 client.26083 trim_cache size 0 max 16384 2024-04-25T13:45:48.795 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:45:48.795 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:45:48.795 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:45:48.795 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:45:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 -- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84fc0afaf0 msgr2=0x7f84fc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84fc0afaf0 0x7f84fc0b2000 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8504030860 tx=0x7f8504030f50 comp rx=0 tx=0).stop 2024-04-25T13:45:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 -- 172.21.15.27:0/3440510172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f850c1045e0 msgr2=0x7f850c103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f850c1045e0 0x7f850c103a30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f850000d6a0 tx=0x7f850000db60 comp rx=0 tx=0).stop 2024-04-25T13:45:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 -- 172.21.15.27:0/3440510172 shutdown_connections 2024-04-25T13:45:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84fc0afaf0 0x7f84fc0b2000 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f850c1098a0 0x7f850c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f850c106f40 0x7f850c1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 --2- 172.21.15.27:0/3440510172 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f850c1045e0 0x7f850c103a30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 -- 172.21.15.27:0/3440510172 >> 172.21.15.27:0/3440510172 conn(0x7f850c0fdbe0 msgr2=0x7f850c105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 -- 172.21.15.27:0/3440510172 shutdown_connections 2024-04-25T13:45:48.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.793+0000 7f8512bf3640 1 -- 172.21.15.27:0/3440510172 wait complete. 2024-04-25T13:45:48.807 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:45:48.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:45:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.962+0000 7f751b642640 1 -- 172.21.15.27:0/3000891183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 msgr2=0x7f75141090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.962+0000 7f751b642640 1 --2- 172.21.15.27:0/3000891183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 0x7f75141090e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7504009860 tx=0x7f750402a340 comp rx=0 tx=0).stop 2024-04-25T13:45:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.962+0000 7f751b642640 1 -- 172.21.15.27:0/3000891183 shutdown_connections 2024-04-25T13:45:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.962+0000 7f751b642640 1 --2- 172.21.15.27:0/3000891183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7514109770 0x7f751410bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.962+0000 7f751b642640 1 --2- 172.21.15.27:0/3000891183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 0x7f75141090e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.962+0000 7f751b642640 1 --2- 172.21.15.27:0/3000891183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7514101e00 0x7f7514102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.962+0000 7f751b642640 1 -- 172.21.15.27:0/3000891183 >> 172.21.15.27:0/3000891183 conn(0x7f75140fd330 msgr2=0x7f75140ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.963+0000 7f751b642640 1 -- 172.21.15.27:0/3000891183 shutdown_connections 2024-04-25T13:45:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.963+0000 7f751b642640 1 -- 172.21.15.27:0/3000891183 wait complete. 2024-04-25T13:45:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.963+0000 7f751b642640 1 Processor -- start 2024-04-25T13:45:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.963+0000 7f751b642640 1 -- start start 2024-04-25T13:45:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f751b642640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7514101e00 0x7f751419c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f751b642640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 0x7f751419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f751b642640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7514109770 0x7f75141a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f751b642640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7514103340 con 0x7f7514106c00 2024-04-25T13:45:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f751b642640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75141031c0 con 0x7f7514109770 2024-04-25T13:45:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f751b642640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f75141034c0 con 0x7f7514101e00 2024-04-25T13:45:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7518bdd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 0x7f751419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7518bdd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 0x7f751419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35722/0 (socket says 172.21.15.27:35722) 2024-04-25T13:45:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7518bdd640 1 -- 172.21.15.27:0/3488533310 learned_addr learned my addr 172.21.15.27:0/3488533310 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7519bdf640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7514109770 0x7f75141a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f75193de640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7514101e00 0x7f751419c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7519bdf640 1 -- 172.21.15.27:0/3488533310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7514101e00 msgr2=0x7f751419c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7519bdf640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7514101e00 0x7f751419c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7519bdf640 1 -- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 msgr2=0x7f751419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7519bdf640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 0x7f751419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.964+0000 7f7519bdf640 1 -- 172.21.15.27:0/3488533310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75141a3e60 con 0x7f7514109770 2024-04-25T13:45:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.965+0000 7f7519bdf640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7514109770 0x7f75141a38a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f750c00cab0 tx=0x7f750c00cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.965+0000 7f750a7fc640 1 -- 172.21.15.27:0/3488533310 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c00ae20 con 0x7f7514109770 2024-04-25T13:45:48.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.965+0000 7f750a7fc640 1 -- 172.21.15.27:0/3488533310 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f750c00f900 con 0x7f7514109770 2024-04-25T13:45:48.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.965+0000 7f750a7fc640 1 -- 172.21.15.27:0/3488533310 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c018a40 con 0x7f7514109770 2024-04-25T13:45:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.965+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75141a4150 con 0x7f7514109770 2024-04-25T13:45:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.966+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75141a4650 con 0x7f7514109770 2024-04-25T13:45:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.967+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75141a4910 con 0x7f7514109770 2024-04-25T13:45:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.969+0000 7f750a7fc640 1 -- 172.21.15.27:0/3488533310 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f750c00fab0 con 0x7f7514109770 2024-04-25T13:45:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.970+0000 7f750a7fc640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75000afcc0 0x7f75000b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.970+0000 7f75193de640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75000afcc0 0x7f75000b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:48.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.970+0000 7f750a7fc640 1 -- 172.21.15.27:0/3488533310 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f750c0df080 con 0x7f7514109770 2024-04-25T13:45:48.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.971+0000 7f75193de640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75000afcc0 0x7f75000b21d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7514104ad0 tx=0x7f7510005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:48.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:48.973+0000 7f750a7fc640 1 -- 172.21.15.27:0/3488533310 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f750c014030 con 0x7f7514109770 2024-04-25T13:45:49.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:48 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:45:49.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:48 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:49.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:48 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:45:49.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:48 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:49.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:48 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:45:49.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:48 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:45:49.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:48 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:49.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:48 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:45:49.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:48 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:49.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:48 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:45:49.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:48 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:45:49.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:48 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:49.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:48 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:45:49.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:48 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:49.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:48 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:45:49.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.336+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75141a4ae0 con 0x7f7514109770 2024-04-25T13:45:49.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.336+0000 7f750a7fc640 1 -- 172.21.15.27:0/3488533310 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f750c0a7d30 con 0x7f7514109770 2024-04-25T13:45:49.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:45:49.336 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:45:49.337 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:45:49.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75000afcc0 msgr2=0x7f75000b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75000afcc0 0x7f75000b21d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7514104ad0 tx=0x7f7510005f90 comp rx=0 tx=0).stop 2024-04-25T13:45:49.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7514109770 msgr2=0x7f75141a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7514109770 0x7f75141a38a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f750c00cab0 tx=0x7f750c00cf70 comp rx=0 tx=0).stop 2024-04-25T13:45:49.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 shutdown_connections 2024-04-25T13:45:49.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f75000afcc0 0x7f75000b21d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7514109770 0x7f75141a38a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7514106c00 0x7f751419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 --2- 172.21.15.27:0/3488533310 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7514101e00 0x7f751419c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.339+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 >> 172.21.15.27:0/3488533310 conn(0x7f75140fd330 msgr2=0x7f7514107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:49.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.340+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 shutdown_connections 2024-04-25T13:45:49.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.340+0000 7f751b642640 1 -- 172.21.15.27:0/3488533310 wait complete. 2024-04-25T13:45:49.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:45:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.521+0000 7f31c7227640 1 -- 172.21.15.27:0/3584093421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31c0110030 msgr2=0x7f31c0112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.521+0000 7f31c7227640 1 --2- 172.21.15.27:0/3584093421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31c0110030 0x7f31c0112450 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009ca0 tx=0x7f31b002a8f0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.522+0000 7f31c7227640 1 -- 172.21.15.27:0/3584093421 shutdown_connections 2024-04-25T13:45:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.522+0000 7f31c7227640 1 --2- 172.21.15.27:0/3584093421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31c0110030 0x7f31c0112450 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.522+0000 7f31c7227640 1 --2- 172.21.15.27:0/3584093421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c010b330 0x7f31c010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.522+0000 7f31c7227640 1 --2- 172.21.15.27:0/3584093421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c010a3d0 0x7f31c010adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.522+0000 7f31c7227640 1 -- 172.21.15.27:0/3584093421 >> 172.21.15.27:0/3584093421 conn(0x7f31c0078de0 msgr2=0x7f31c007b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:49.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.527+0000 7f31c7227640 1 -- 172.21.15.27:0/3584093421 shutdown_connections 2024-04-25T13:45:49.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.528+0000 7f31c7227640 1 -- 172.21.15.27:0/3584093421 wait complete. 2024-04-25T13:45:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.529+0000 7f31c7227640 1 Processor -- start 2024-04-25T13:45:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.529+0000 7f31c7227640 1 -- start start 2024-04-25T13:45:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.529+0000 7f31c7227640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31c010a3d0 0x7f31c01a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.529+0000 7f31c7227640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c010b330 0x7f31c01a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.529+0000 7f31c7227640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c0110030 0x7f31c01a7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.529+0000 7f31c7227640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31c0115c70 con 0x7f31c010b330 2024-04-25T13:45:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.529+0000 7f31c7227640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31c0115af0 con 0x7f31c0110030 2024-04-25T13:45:49.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.529+0000 7f31c7227640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f31c0115df0 con 0x7f31c010a3d0 2024-04-25T13:45:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.530+0000 7f31c57c4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c0110030 0x7f31c01a7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.530+0000 7f31c57c4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c0110030 0x7f31c01a7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33150/0 (socket says 172.21.15.27:33150) 2024-04-25T13:45:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.530+0000 7f31c57c4640 1 -- 172.21.15.27:0/181548274 learned_addr learned my addr 172.21.15.27:0/181548274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.531+0000 7f31bffff640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c010b330 0x7f31c01a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.531+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31c010a3d0 0x7f31c01a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.532+0000 7f31c57c4640 1 -- 172.21.15.27:0/181548274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31c010a3d0 msgr2=0x7f31c01a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.532+0000 7f31c57c4640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31c010a3d0 0x7f31c01a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.532+0000 7f31c57c4640 1 -- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c010b330 msgr2=0x7f31c01a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.532+0000 7f31c57c4640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c010b330 0x7f31c01a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.532+0000 7f31c57c4640 1 -- 172.21.15.27:0/181548274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c01a8350 con 0x7f31c0110030 2024-04-25T13:45:49.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.534+0000 7f31c57c4640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c0110030 0x7f31c01a7ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f31b0031a90 tx=0x7f31b0003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:49.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.536+0000 7f31bdffb640 1 -- 172.21.15.27:0/181548274 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b0004240 con 0x7f31c0110030 2024-04-25T13:45:49.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.536+0000 7f31bdffb640 1 -- 172.21.15.27:0/181548274 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b002d070 con 0x7f31c0110030 2024-04-25T13:45:49.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.536+0000 7f31c7227640 1 -- 172.21.15.27:0/181548274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c01a85e0 con 0x7f31c0110030 2024-04-25T13:45:49.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.536+0000 7f31bdffb640 1 -- 172.21.15.27:0/181548274 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b003c410 con 0x7f31c0110030 2024-04-25T13:45:49.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.536+0000 7f31c7227640 1 -- 172.21.15.27:0/181548274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c01a8bb0 con 0x7f31c0110030 2024-04-25T13:45:49.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.541+0000 7f31ce1bc740 1 Processor -- start 2024-04-25T13:45:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.541+0000 7f31bdffb640 1 -- 172.21.15.27:0/181548274 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f31b003c5b0 con 0x7f31c0110030 2024-04-25T13:45:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.541+0000 7f31ce1bc740 1 -- start start 2024-04-25T13:45:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a87d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a87dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a87ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55648a7e1d80 con 0x55648a4db390 2024-04-25T13:45:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55648a79ba20 con 0x55648a4b24d0 2024-04-25T13:45:49.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55648a7ddd40 con 0x55648a4ba6b0 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31c4fc3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a87dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31c4fc3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a87dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33154/0 (socket says 172.21.15.27:33154) 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31c4fc3640 1 -- 172.21.15.27:0/2280727392 learned_addr learned my addr 172.21.15.27:0/2280727392 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31bffff640 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a87ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.542+0000 7f31bffff640 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a87d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.543+0000 7f31c4fc3640 1 -- 172.21.15.27:0/2280727392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 msgr2=0x55648a87ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.543+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a87ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.543+0000 7f31c4fc3640 1 -- 172.21.15.27:0/2280727392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 msgr2=0x55648a87d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.543+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a87d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.543+0000 7f31c4fc3640 1 -- 172.21.15.27:0/2280727392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55648a7a6c30 con 0x55648a4b24d0 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.543+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a87dd90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31b800b7c0 tx=0x7f31b800bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc1c9215ff064a81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:49.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.543+0000 7f31bdffb640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31ac0afd00 0x7f31ac0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.543+0000 7f31b6ffd640 1 -- 172.21.15.27:0/2280727392 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b800d8d0 con 0x55648a4b24d0 2024-04-25T13:45:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.544+0000 7f31bffff640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31ac0afd00 0x7f31ac0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.544+0000 7f31b6ffd640 1 -- 172.21.15.27:0/2280727392 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8018070 con 0x55648a4b24d0 2024-04-25T13:45:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.544+0000 7f31b6ffd640 1 -- 172.21.15.27:0/2280727392 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b80123e0 con 0x55648a4b24d0 2024-04-25T13:45:49.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.544+0000 7f31bdffb640 1 -- 172.21.15.27:0/181548274 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f31b0100450 con 0x7f31c0110030 2024-04-25T13:45:49.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.545+0000 7f31ce1bc740 1 -- 172.21.15.27:0/2280727392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 msgr2=0x55648a87dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.545+0000 7f31bffff640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31ac0afd00 0x7f31ac0b2210 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f31a8005d90 tx=0x7f31a8005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:49.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.545+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a87dd90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31b800b7c0 tx=0x7f31b800bc80 comp rx=0 tx=0).stop 2024-04-25T13:45:49.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.545+0000 7f31ce1bc740 1 -- 172.21.15.27:0/2280727392 shutdown_connections 2024-04-25T13:45:49.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.545+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a87ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.545+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a87dd90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.545+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/2280727392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a87d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.545+0000 7f31ce1bc740 1 -- 172.21.15.27:0/2280727392 >> 172.21.15.27:0/2280727392 conn(0x55648a4cf4e0 msgr2=0x55648a879020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:49.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.546+0000 7f31ce1bc740 1 -- 172.21.15.27:0/2280727392 shutdown_connections 2024-04-25T13:45:49.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.546+0000 7f31ce1bc740 1 -- 172.21.15.27:0/2280727392 wait complete. 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.546+0000 7f31ce1bc740 1 -- start start 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.546+0000 7f31ce1bc740 1 Processor -- start 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a887550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a887a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a88e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55648a87f530 con 0x55648a4db390 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55648a768dc0 con 0x55648a4b24d0 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55648a87f6b0 con 0x55648a4ba6b0 2024-04-25T13:45:49.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31ce1bc740 10 client.0 _ll_register_callbacks cb 0x55648a784600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:49.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31c57c4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a88e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31c4fc3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a887550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31bffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a887a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31c4fc3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a887550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33170/0 (socket says 172.21.15.27:33170) 2024-04-25T13:45:49.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31c57c4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a88e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35746/0 (socket says 172.21.15.27:35746) 2024-04-25T13:45:49.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.547+0000 7f31c4fc3640 1 -- 172.21.15.27:0/3300935806 learned_addr learned my addr 172.21.15.27:0/3300935806 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:49.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31c4fc3640 1 -- 172.21.15.27:0/3300935806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 msgr2=0x55648a887a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:49.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a887a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31c4fc3640 1 -- 172.21.15.27:0/3300935806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 msgr2=0x55648a88e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a88e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31c4fc3640 1 -- 172.21.15.27:0/3300935806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55648a4f33f0 con 0x55648a4b24d0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31bffff640 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a887a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a887550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f31b800b790 tx=0x7f31b8013dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5271abc07fc0824f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f319ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f319ffff640 1 -- 172.21.15.27:0/3300935806 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8023070 con 0x55648a4b24d0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f319ffff640 1 -- 172.21.15.27:0/3300935806 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f31b800d8d0 con 0x55648a4b24d0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31ce1bc740 1 -- 172.21.15.27:0/3300935806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b8018070 con 0x55648a4b24d0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f319ffff640 1 -- 172.21.15.27:0/3300935806 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f31b802f050 con 0x55648a4b24d0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31ce1bc740 10 client.26063 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.548+0000 7f31ce1bc740 1 -- 172.21.15.27:0/3300935806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55648a5bd7d0 con 0x55648a4b24d0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.549+0000 7f319ffff640 1 -- 172.21.15.27:0/3300935806 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8018070 con 0x55648a4b24d0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.549+0000 7f319ffff640 1 -- 172.21.15.27:0/3300935806 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f31b801cc10 con 0x55648a4b24d0 2024-04-25T13:45:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.549+0000 7f31ce1bc740 10 client.26063 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:49.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.550+0000 7f31ce1bc740 10 client.26063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:49.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.550+0000 7f31ce1bc740 20 client.26063 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:49.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.550+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55648a499f60 0x55648a894ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.550+0000 7f31ce1bc740 4 client.26063 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:49.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.550+0000 7f31ce1bc740 1 -- 172.21.15.27:0/3300935806 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55648a7ef990 con 0x55648a499f60 2024-04-25T13:45:49.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.550+0000 7f31bffff640 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55648a499f60 0x55648a894ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.550+0000 7f31bffff640 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55648a499f60 0x55648a894ea0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12a7ec2ad2649a38 server_cookie=154cf62e0a390acc in_seq=0 out_seq=0 2024-04-25T13:45:49.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.550+0000 7f319ffff640 10 client.26063 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:49.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f319ffff640 1 -- 172.21.15.27:0/3300935806 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55648a7ef990 con 0x55648a499f60 2024-04-25T13:45:49.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f319ffff640 10 client.26063 handle_command_reply: tid=0 2024-04-25T13:45:49.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 client.26063 shutdown 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 -- 172.21.15.27:0/3300935806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 msgr2=0x55648a887550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a887550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f31b800b790 tx=0x7f31b8013dc0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 -- 172.21.15.27:0/3300935806 shutdown_connections 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55648a499f60 0x55648a894ea0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4db390 0x55648a88e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f319ffff640 0 client.26063 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a887a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/3300935806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4b24d0 0x55648a887550 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 -- 172.21.15.27:0/3300935806 >> 172.21.15.27:0/3300935806 conn(0x55648a4cf4e0 msgr2=0x55648a87bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.551+0000 7f31ce1bc740 1 -- 172.21.15.27:0/3300935806 shutdown_connections 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.552+0000 7f31ce1bc740 1 -- 172.21.15.27:0/3300935806 wait complete. 2024-04-25T13:45:49.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.552+0000 7f31ce1bc740 20 client.26063 trim_cache size 0 max 16384 2024-04-25T13:45:49.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31ce1bc740 1 Processor -- start 2024-04-25T13:45:49.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31ce1bc740 1 -- start start 2024-04-25T13:45:49.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4b24d0 0x55648a887830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a88e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8995b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01af770 con 0x55648a4b24d0 2024-04-25T13:45:49.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01b1ff0 con 0x55648a4db390 2024-04-25T13:45:49.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01afde0 con 0x55648a4ba6b0 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31c4fc3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4b24d0 0x55648a887830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31c4fc3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4b24d0 0x55648a887830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35760/0 (socket says 172.21.15.27:35760) 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31c4fc3640 1 -- 172.21.15.27:0/1512085676 learned_addr learned my addr 172.21.15.27:0/1512085676 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.572+0000 7f31c57c4640 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31bffff640 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a88e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31c4fc3640 1 -- 172.21.15.27:0/1512085676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 msgr2=0x55648a88e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a88e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31c4fc3640 1 -- 172.21.15.27:0/1512085676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 msgr2=0x55648a8995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31c4fc3640 1 -- 172.21.15.27:0/1512085676 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55648a88ee50 con 0x55648a4b24d0 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31bffff640 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a88e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4b24d0 0x55648a887830 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f31b801b5a0 tx=0x7f31b800b160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ae18c247000c28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31b67fc640 1 -- 172.21.15.27:0/1512085676 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8020070 con 0x55648a4b24d0 2024-04-25T13:45:49.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31b67fc640 1 -- 172.21.15.27:0/1512085676 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8018050 con 0x55648a4b24d0 2024-04-25T13:45:49.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.573+0000 7f31b67fc640 1 -- 172.21.15.27:0/1512085676 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8023040 con 0x55648a4b24d0 2024-04-25T13:45:49.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.574+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1512085676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4b24d0 msgr2=0x55648a887830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.574+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4b24d0 0x55648a887830 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f31b801b5a0 tx=0x7f31b800b160 comp rx=0 tx=0).stop 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.575+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1512085676 shutdown_connections 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.575+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.575+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4ba6b0 0x55648a88e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.575+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1512085676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4b24d0 0x55648a887830 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.575+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1512085676 >> 172.21.15.27:0/1512085676 conn(0x55648a4cf4e0 msgr2=0x55648a887120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.575+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1512085676 shutdown_connections 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.575+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1512085676 wait complete. 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 1 -- start start 2024-04-25T13:45:49.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 1 Processor -- start 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4b24d0 0x55648a88d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4ba6b0 0x55648a88ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8fb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01b04b0 con 0x55648a4ba6b0 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01b1920 con 0x55648a4db390 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01b0b80 con 0x55648a4b24d0 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31ce1bc740 10 client.0 _ll_register_callbacks cb 0x7f31a800de30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.576+0000 7f31bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4ba6b0 0x55648a88ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31c57c4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8fb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4ba6b0 0x55648a88ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35764/0 (socket says 172.21.15.27:35764) 2024-04-25T13:45:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31bffff640 1 -- 172.21.15.27:0/1909679859 learned_addr learned my addr 172.21.15.27:0/1909679859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4b24d0 0x55648a88d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31c57c4640 1 -- 172.21.15.27:0/1909679859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4b24d0 msgr2=0x55648a88d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31c57c4640 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4b24d0 0x55648a88d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31c57c4640 1 -- 172.21.15.27:0/1909679859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4ba6b0 msgr2=0x55648a88ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31c57c4640 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4ba6b0 0x55648a88ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31c57c4640 1 -- 172.21.15.27:0/1909679859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55648a8fb6a0 con 0x55648a4db390 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31bffff640 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4ba6b0 0x55648a88ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f31c57c4640 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8fb160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f31b00c8600 tx=0x7f31b0030a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa2b3f215209cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f319ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f319ffff640 1 -- 172.21.15.27:0/1909679859 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b00098d0 con 0x55648a4db390 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.577+0000 7f319ffff640 1 -- 172.21.15.27:0/1909679859 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f31b0030680 con 0x55648a4db390 2024-04-25T13:45:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.578+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1909679859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31b8018050 con 0x55648a4db390 2024-04-25T13:45:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f319ffff640 1 -- 172.21.15.27:0/1909679859 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f31b00d10a0 con 0x55648a4db390 2024-04-25T13:45:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f31ce1bc740 10 client.26069 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1909679859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55648a8fc830 con 0x55648a4db390 2024-04-25T13:45:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f319ffff640 1 -- 172.21.15.27:0/1909679859 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b8018050 con 0x55648a4db390 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f319ffff640 1 -- 172.21.15.27:0/1909679859 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f31b00c8b60 con 0x55648a4db390 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f31ce1bc740 10 client.26069 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f31ce1bc740 10 client.26069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f31ce1bc740 20 client.26069 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55648a46f3a0 0x55648a8ff1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f31ce1bc740 4 client.26069 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.579+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1909679859 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55648a7e9800 con 0x55648a46f3a0 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.580+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55648a46f3a0 0x55648a8ff1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.580+0000 7f319ffff640 10 client.26069 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.580+0000 7f31c4fc3640 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55648a46f3a0 0x55648a8ff1d0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=930b01b92314587 server_cookie=fef9b4d17bb950bc in_seq=0 out_seq=0 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.580+0000 7f319ffff640 1 -- 172.21.15.27:0/1909679859 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55648a7e9800 con 0x55648a46f3a0 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.580+0000 7f319ffff640 10 client.26069 handle_command_reply: tid=0 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.580+0000 7f31ce1bc740 1 client.26069 shutdown 2024-04-25T13:45:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1909679859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 msgr2=0x55648a8fb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8fb160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f31b00c8600 tx=0x7f31b0030a20 comp rx=0 tx=0).stop 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1909679859 shutdown_connections 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55648a46f3a0 0x55648a8ff1d0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f319ffff640 0 client.26069 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55648a4db390 0x55648a8fb160 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55648a4ba6b0 0x55648a88ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 --2- 172.21.15.27:0/1909679859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55648a4b24d0 0x55648a88d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1909679859 >> 172.21.15.27:0/1909679859 conn(0x55648a4cf4e0 msgr2=0x55648a87be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1909679859 shutdown_connections 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 1 -- 172.21.15.27:0/1909679859 wait complete. 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.581+0000 7f31ce1bc740 20 client.26069 trim_cache size 0 max 16384 2024-04-25T13:45:49.583 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:45:49.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 -- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31ac0afd00 msgr2=0x7f31ac0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31ac0afd00 0x7f31ac0b2210 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f31a8005d90 tx=0x7f31a8005e40 comp rx=0 tx=0).stop 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 -- 172.21.15.27:0/181548274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c0110030 msgr2=0x7f31c01a7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c0110030 0x7f31c01a7ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f31b0031a90 tx=0x7f31b0003ce0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 -- 172.21.15.27:0/181548274 shutdown_connections 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f31ac0afd00 0x7f31ac0b2210 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31c0110030 0x7f31c01a7ce0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c010b330 0x7f31c01a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 --2- 172.21.15.27:0/181548274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f31c010a3d0 0x7f31c01a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 -- 172.21.15.27:0/181548274 >> 172.21.15.27:0/181548274 conn(0x7f31c0078de0 msgr2=0x7f31c010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 -- 172.21.15.27:0/181548274 shutdown_connections 2024-04-25T13:45:49.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:49.582+0000 7f31c7227640 1 -- 172.21.15.27:0/181548274 wait complete. 2024-04-25T13:45:50.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:49 smithi027 ceph-mon[29140]: pgmap v497: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.5 MiB/s wr, 622 op/s 2024-04-25T13:45:50.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:49 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3488533310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:50.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:49 smithi122 ceph-mon[32843]: pgmap v497: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.5 MiB/s wr, 622 op/s 2024-04-25T13:45:50.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:49 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3488533310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:50.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:49 smithi045 ceph-mon[32731]: pgmap v497: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.5 MiB/s wr, 622 op/s 2024-04-25T13:45:50.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:49 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3488533310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:45:50.595 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:45:50.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:45:50.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 -- 172.21.15.27:0/1109611168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 msgr2=0x7fba101037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 --2- 172.21.15.27:0/1109611168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 0x7fba101037c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800ba90 tx=0x7fb9f802c170 comp rx=0 tx=0).stop 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 -- 172.21.15.27:0/1109611168 shutdown_connections 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 --2- 172.21.15.27:0/1109611168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba101080b0 0x7fba1010a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 --2- 172.21.15.27:0/1109611168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 0x7fba101037c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 --2- 172.21.15.27:0/1109611168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba10102170 0x7fba10102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 -- 172.21.15.27:0/1109611168 >> 172.21.15.27:0/1109611168 conn(0x7fba100fd340 msgr2=0x7fba100ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 -- 172.21.15.27:0/1109611168 shutdown_connections 2024-04-25T13:45:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.752+0000 7fba15234640 1 -- 172.21.15.27:0/1109611168 wait complete. 2024-04-25T13:45:50.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba15234640 1 Processor -- start 2024-04-25T13:45:50.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba15234640 1 -- start start 2024-04-25T13:45:50.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba15234640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba10102170 0x7fba1019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba15234640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 0x7fba1019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba15234640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba101080b0 0x7fba101a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba15234640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba1010d450 con 0x7fba10102170 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba15234640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba1010d2d0 con 0x7fba101080b0 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba15234640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fba1010d5d0 con 0x7fba101033b0 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba0ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba10102170 0x7fba1019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba07fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 0x7fba1019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba101080b0 0x7fba101a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba07fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 0x7fba1019c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:34106/0 (socket says 172.21.15.27:34106) 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba101080b0 0x7fba101a38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33206/0 (socket says 172.21.15.27:33206) 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.753+0000 7fba07fff640 1 -- 172.21.15.27:0/3905435001 learned_addr learned my addr 172.21.15.27:0/3905435001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0ed9d640 1 -- 172.21.15.27:0/3905435001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 msgr2=0x7fba1019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0ed9d640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 0x7fba1019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0ed9d640 1 -- 172.21.15.27:0/3905435001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba101080b0 msgr2=0x7fba101a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0ed9d640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba101080b0 0x7fba101a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0ed9d640 1 -- 172.21.15.27:0/3905435001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba101a3f50 con 0x7fba10102170 2024-04-25T13:45:50.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0ed9d640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba10102170 0x7fba1019c410 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fba10101a40 tx=0x7fba0000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:50.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0cd99640 1 -- 172.21.15.27:0/3905435001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0000cc40 con 0x7fba10102170 2024-04-25T13:45:50.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba15234640 1 -- 172.21.15.27:0/3905435001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba101a4240 con 0x7fba10102170 2024-04-25T13:45:50.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0cd99640 1 -- 172.21.15.27:0/3905435001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba0000cde0 con 0x7fba10102170 2024-04-25T13:45:50.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba0cd99640 1 -- 172.21.15.27:0/3905435001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00010430 con 0x7fba10102170 2024-04-25T13:45:50.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.754+0000 7fba15234640 1 -- 172.21.15.27:0/3905435001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba101a47e0 con 0x7fba10102170 2024-04-25T13:45:50.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.757+0000 7fba1c1be740 1 Processor -- start 2024-04-25T13:45:50.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.757+0000 7fba1c1be740 1 -- start start 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54cb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba1c1be740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d8f5429b00 con 0x55d8f5123410 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba1c1be740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d8f53c5920 con 0x55d8f50fa550 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba1c1be740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d8f5425ac0 con 0x55d8f5102730 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54c44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35790/0 (socket says 172.21.15.27:35790) 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0f59e640 1 -- 172.21.15.27:0/2793867323 learned_addr learned my addr 172.21.15.27:0/2793867323 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0cd99640 1 -- 172.21.15.27:0/3905435001 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fba00003e70 con 0x7fba10102170 2024-04-25T13:45:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba07fff640 1 --2- 172.21.15.27:0/2793867323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0f59e640 1 -- 172.21.15.27:0/2793867323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 msgr2=0x55d8f54cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2793867323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0f59e640 1 -- 172.21.15.27:0/2793867323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 msgr2=0x55d8f54c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2793867323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.758+0000 7fba0f59e640 1 -- 172.21.15.27:0/2793867323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f52037e0 con 0x55d8f5123410 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.759+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2793867323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54c44a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fba08009fd0 tx=0x7fba080118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3007208587b46ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.759+0000 7fba057fa640 1 -- 172.21.15.27:0/2793867323 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08004d20 con 0x55d8f5123410 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.759+0000 7fba0cd99640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9fc0afcf0 0x7fb9fc0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.759+0000 7fba057fa640 1 -- 172.21.15.27:0/2793867323 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba08004050 con 0x55d8f5123410 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.759+0000 7fba057fa640 1 -- 172.21.15.27:0/2793867323 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08005280 con 0x55d8f5123410 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.759+0000 7fba0ed9d640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9fc0afcf0 0x7fb9fc0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.759+0000 7fba0cd99640 1 -- 172.21.15.27:0/3905435001 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fba00003c30 con 0x7fba10102170 2024-04-25T13:45:50.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.761+0000 7fba1c1be740 1 -- 172.21.15.27:0/2793867323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 msgr2=0x55d8f54c44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.761+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2793867323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54c44a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fba08009fd0 tx=0x7fba080118d0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.761+0000 7fba1c1be740 1 -- 172.21.15.27:0/2793867323 shutdown_connections 2024-04-25T13:45:50.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.761+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2793867323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.761+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2793867323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.761+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2793867323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54c44a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.761+0000 7fba1c1be740 1 -- 172.21.15.27:0/2793867323 >> 172.21.15.27:0/2793867323 conn(0x55d8f5117560 msgr2=0x55d8f54bfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:50.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.762+0000 7fba0ed9d640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9fc0afcf0 0x7fb9fc0b2200 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fba000a3370 tx=0x7fba0000cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:50.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.762+0000 7fba1c1be740 1 -- 172.21.15.27:0/2793867323 shutdown_connections 2024-04-25T13:45:50.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.762+0000 7fba1c1be740 1 -- 172.21.15.27:0/2793867323 wait complete. 2024-04-25T13:45:50.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.763+0000 7fba1c1be740 1 -- start start 2024-04-25T13:45:50.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.763+0000 7fba1c1be740 1 Processor -- start 2024-04-25T13:45:50.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.763+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f50fa550 0x55d8f54ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5102730 0x55d8f54ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba1c1be740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d8f54cc370 con 0x55d8f5123410 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba1c1be740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d8f53c35d0 con 0x55d8f5102730 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba1c1be740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d8f54cc4f0 con 0x55d8f50fa550 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba1c1be740 10 client.0 _ll_register_callbacks cb 0x55d8f53b1080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5102730 0x55d8f54ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5102730 0x55d8f54ce5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33224/0 (socket says 172.21.15.27:33224) 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35800/0 (socket says 172.21.15.27:35800) 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba07fff640 1 -- 172.21.15.27:0/1022959664 learned_addr learned my addr 172.21.15.27:0/1022959664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba0f59e640 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f50fa550 0x55d8f54ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba0f59e640 1 -- 172.21.15.27:0/1022959664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f50fa550 msgr2=0x55d8f54ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.764+0000 7fba0f59e640 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f50fa550 0x55d8f54ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fba0f59e640 1 -- 172.21.15.27:0/1022959664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5102730 msgr2=0x55d8f54ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fba0f59e640 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5102730 0x55d8f54ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fba0f59e640 1 -- 172.21.15.27:0/1022959664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f5263f70 con 0x55d8f5123410 2024-04-25T13:45:50.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fba07fff640 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5102730 0x55d8f54ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:50.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fba0f59e640 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d5530 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fba08003fd0 tx=0x7fba08000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aef139d2025a14f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:50.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fb9f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:45:50.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fb9f67fc640 1 -- 172.21.15.27:0/1022959664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08004a40 con 0x55d8f5123410 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fb9f67fc640 1 -- 172.21.15.27:0/1022959664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fba08004be0 con 0x55d8f5123410 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fba1c1be740 1 -- 172.21.15.27:0/1022959664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba08004050 con 0x55d8f5123410 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fb9f67fc640 1 -- 172.21.15.27:0/1022959664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fba08004de0 con 0x55d8f5123410 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fba1c1be740 10 client.16365 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.765+0000 7fba1c1be740 1 -- 172.21.15.27:0/1022959664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d8f50fb170 con 0x55d8f5123410 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fb9f67fc640 1 -- 172.21.15.27:0/1022959664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba08004050 con 0x55d8f5123410 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fb9f67fc640 1 -- 172.21.15.27:0/1022959664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fba0800f4e0 con 0x55d8f5123410 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fba1c1be740 10 client.16365 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fba1c1be740 10 client.16365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fba1c1be740 20 client.16365 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fba1c1be740 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d8f50e21f0 0x55d8f54db790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fba1c1be740 4 client.16365 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:50.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fba1c1be740 1 -- 172.21.15.27:0/1022959664 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8f5437710 con 0x55d8f50e21f0 2024-04-25T13:45:50.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.766+0000 7fba07fff640 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d8f50e21f0 0x55d8f54db790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.767+0000 7fb9f67fc640 10 client.16365 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:50.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.767+0000 7fba07fff640 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d8f50e21f0 0x55d8f54db790 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73bdb5dd88e8a97e server_cookie=4d84e72f6ab97858 in_seq=0 out_seq=0 2024-04-25T13:45:50.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.768+0000 7fb9f67fc640 1 -- 172.21.15.27:0/1022959664 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d8f5437710 con 0x55d8f50e21f0 2024-04-25T13:45:50.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.768+0000 7fb9f67fc640 10 client.16365 handle_command_reply: tid=0 2024-04-25T13:45:50.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.768+0000 7fba1c1be740 1 client.16365 shutdown 2024-04-25T13:45:50.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.769+0000 7fba1c1be740 1 -- 172.21.15.27:0/1022959664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 msgr2=0x55d8f54d5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.769+0000 7fba1c1be740 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d5530 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fba08003fd0 tx=0x7fba08000f40 comp rx=0 tx=0).stop 2024-04-25T13:45:50.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.769+0000 7fba1c1be740 1 -- 172.21.15.27:0/1022959664 shutdown_connections 2024-04-25T13:45:50.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.769+0000 7fba1c1be740 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d8f50e21f0 0x55d8f54db790 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.770+0000 7fba1c1be740 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d5530 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.770+0000 7fba1c1be740 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5102730 0x55d8f54ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.770+0000 7fba1c1be740 1 --2- 172.21.15.27:0/1022959664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f50fa550 0x55d8f54ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.770+0000 7fba1c1be740 1 -- 172.21.15.27:0/1022959664 >> 172.21.15.27:0/1022959664 conn(0x55d8f5117560 msgr2=0x55d8f54beef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:50.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.770+0000 7fb9f67fc640 0 client.16365 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:50.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.770+0000 7fba1c1be740 1 -- 172.21.15.27:0/1022959664 shutdown_connections 2024-04-25T13:45:50.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.770+0000 7fba1c1be740 1 -- 172.21.15.27:0/1022959664 wait complete. 2024-04-25T13:45:50.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.770+0000 7fba1c1be740 20 client.16365 trim_cache size 0 max 16384 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.803+0000 7fba1c1be740 1 Processor -- start 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.803+0000 7fba1c1be740 1 -- start start 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f50fa550 0x55d8f54c3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54e0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5123410 0x55d8f54e14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba1c1be740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba101ab4b0 con 0x55d8f50fa550 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba1c1be740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba101add30 con 0x55d8f5123410 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba1c1be740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fba101abb20 con 0x55d8f5102730 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f50fa550 0x55d8f54c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba07fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54e0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f50fa550 0x55d8f54c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35802/0 (socket says 172.21.15.27:35802) 2024-04-25T13:45:50.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba0f59e640 1 -- 172.21.15.27:0/2619942247 learned_addr learned my addr 172.21.15.27:0/2619942247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:50.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.804+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2619942247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5123410 0x55d8f54e14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba0f59e640 1 -- 172.21.15.27:0/2619942247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 msgr2=0x55d8f54e0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:50.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2619942247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54e0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba0f59e640 1 -- 172.21.15.27:0/2619942247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5123410 msgr2=0x55d8f54e14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2619942247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5123410 0x55d8f54e14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba0f59e640 1 -- 172.21.15.27:0/2619942247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f54c8f40 con 0x55d8f50fa550 2024-04-25T13:45:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2619942247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f50fa550 0x55d8f54c3340 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fba08010010 tx=0x7fba08009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7671f1560b708699 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba04ff9640 1 -- 172.21.15.27:0/2619942247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08009b30 con 0x55d8f50fa550 2024-04-25T13:45:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba04ff9640 1 -- 172.21.15.27:0/2619942247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba0800f4e0 con 0x55d8f50fa550 2024-04-25T13:45:50.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.805+0000 7fba04ff9640 1 -- 172.21.15.27:0/2619942247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba080286e0 con 0x55d8f50fa550 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.806+0000 7fba1c1be740 1 -- 172.21.15.27:0/2619942247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f50fa550 msgr2=0x55d8f54c3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.806+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2619942247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f50fa550 0x55d8f54c3340 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fba08010010 tx=0x7fba08009230 comp rx=0 tx=0).stop 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.807+0000 7fba1c1be740 1 -- 172.21.15.27:0/2619942247 shutdown_connections 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.807+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2619942247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f5123410 0x55d8f54e14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.807+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2619942247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54e0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.807+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2619942247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f50fa550 0x55d8f54c3340 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.807+0000 7fba1c1be740 1 -- 172.21.15.27:0/2619942247 >> 172.21.15.27:0/2619942247 conn(0x55d8f5117560 msgr2=0x55d8f54c2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.807+0000 7fba1c1be740 1 -- 172.21.15.27:0/2619942247 shutdown_connections 2024-04-25T13:45:50.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.807+0000 7fba1c1be740 1 -- 172.21.15.27:0/2619942247 wait complete. 2024-04-25T13:45:50.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.808+0000 7fba1c1be740 1 -- start start 2024-04-25T13:45:50.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.808+0000 7fba1c1be740 1 Processor -- start 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54d3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba1c1be740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54d3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba0f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54d3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33248/0 (socket says 172.21.15.27:33248) 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba0f59e640 1 -- 172.21.15.27:0/2454965508 learned_addr learned my addr 172.21.15.27:0/2454965508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fba101ac1f0 con 0x55d8f5123410 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fba101ad660 con 0x55d8f50fa550 2024-04-25T13:45:50.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.809+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fba101ac8c0 con 0x55d8f5102730 2024-04-25T13:45:50.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fba1c1be740 10 client.0 _ll_register_callbacks cb 0x7fba08003bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:50.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fba07fff640 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fba0f59e640 1 -- 172.21.15.27:0/2454965508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 msgr2=0x55d8f54d3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:50.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54d3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fba0f59e640 1 -- 172.21.15.27:0/2454965508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 msgr2=0x55d8f54d4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:50.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fba0f59e640 1 -- 172.21.15.27:0/2454965508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f54d5230 con 0x55d8f50fa550 2024-04-25T13:45:50.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fba0f59e640 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54d3800 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fba08011dd0 tx=0x7fba0800fb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c2eb9e46eb055 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:50.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.810+0000 7fb9f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:50.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.811+0000 7fb9f67fc640 1 -- 172.21.15.27:0/2454965508 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0800fda0 con 0x55d8f50fa550 2024-04-25T13:45:50.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.811+0000 7fb9f67fc640 1 -- 172.21.15.27:0/2454965508 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fba08028480 con 0x55d8f50fa550 2024-04-25T13:45:50.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.811+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba0800f4e0 con 0x55d8f50fa550 2024-04-25T13:45:50.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.811+0000 7fb9f67fc640 1 -- 172.21.15.27:0/2454965508 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fba0800f690 con 0x55d8f50fa550 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.811+0000 7fba1c1be740 10 client.26084 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.811+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d8f5543dd0 con 0x55d8f50fa550 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fb9f67fc640 1 -- 172.21.15.27:0/2454965508 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba0800f4e0 con 0x55d8f50fa550 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fb9f67fc640 1 -- 172.21.15.27:0/2454965508 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fba0803d470 con 0x55d8f50fa550 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fba1c1be740 10 client.26084 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fba1c1be740 10 client.26084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fba1c1be740 20 client.26084 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d8f50b7630 0x55d8f5546690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fba1c1be740 4 client.26084 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d8f5431580 con 0x55d8f50b7630 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.812+0000 7fba07fff640 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d8f50b7630 0x55d8f5546690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.813+0000 7fba07fff640 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d8f50b7630 0x55d8f5546690 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5ae897b14d11a3a server_cookie=2e8800778b8dfc12 in_seq=0 out_seq=0 2024-04-25T13:45:50.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.813+0000 7fb9f67fc640 10 client.26084 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fb9f67fc640 1 -- 172.21.15.27:0/2454965508 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d8f5431580 con 0x55d8f50b7630 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fb9f67fc640 10 client.26084 handle_command_reply: tid=0 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 client.26084 shutdown 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 msgr2=0x55d8f54d3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54d3800 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fba08011dd0 tx=0x7fba0800fb90 comp rx=0 tx=0).stop 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 shutdown_connections 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d8f50b7630 0x55d8f5546690 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d8f5123410 0x55d8f54d4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d8f5102730 0x55d8f54d3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 --2- 172.21.15.27:0/2454965508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d8f50fa550 0x55d8f54d3800 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 >> 172.21.15.27:0/2454965508 conn(0x55d8f5117560 msgr2=0x55d8f54c3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fb9f67fc640 0 client.26084 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 shutdown_connections 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.814+0000 7fba1c1be740 1 -- 172.21.15.27:0/2454965508 wait complete. 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba1c1be740 20 client.26084 trim_cache size 0 max 16384 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "ce4fbb66-386e-4b55-ad1e-4273b5528ff4", 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:45:50.816 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:45:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 -- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9fc0afcf0 msgr2=0x7fb9fc0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9fc0afcf0 0x7fb9fc0b2200 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fba000a3370 tx=0x7fba0000cf90 comp rx=0 tx=0).stop 2024-04-25T13:45:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 -- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba10102170 msgr2=0x7fba1019c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba10102170 0x7fba1019c410 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fba10101a40 tx=0x7fba0000ee50 comp rx=0 tx=0).stop 2024-04-25T13:45:50.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 -- 172.21.15.27:0/3905435001 shutdown_connections 2024-04-25T13:45:50.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb9fc0afcf0 0x7fb9fc0b2200 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba101080b0 0x7fba101a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba101033b0 0x7fba1019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 --2- 172.21.15.27:0/3905435001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba10102170 0x7fba1019c410 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 -- 172.21.15.27:0/3905435001 >> 172.21.15.27:0/3905435001 conn(0x7fba100fd340 msgr2=0x7fba10104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:50.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 -- 172.21.15.27:0/3905435001 shutdown_connections 2024-04-25T13:45:50.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.815+0000 7fba15234640 1 -- 172.21.15.27:0/3905435001 wait complete. 2024-04-25T13:45:50.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:45:50.830 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:45:50.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:50 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:45:50.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:50 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:50.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:50 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:45:50.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.998+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1658523495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a4110870 msgr2=0x7fa1a4112c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:50.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.998+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1658523495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a4110870 0x7fa1a4112c90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009be0 tx=0x7fa19c02a6c0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.999+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1658523495 shutdown_connections 2024-04-25T13:45:50.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.999+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1658523495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a4110870 0x7fa1a4112c90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.999+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1658523495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a410bb70 0x7fa1a410bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.999+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1658523495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a410ad00 0x7fa1a410b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:50.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:50.999+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1658523495 >> 172.21.15.27:0/1658523495 conn(0x7fa1a4070940 msgr2=0x7fa1a4072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.006+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1658523495 shutdown_connections 2024-04-25T13:45:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.009+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1658523495 wait complete. 2024-04-25T13:45:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.010+0000 7fa1a8e51640 1 Processor -- start 2024-04-25T13:45:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.010+0000 7fa1a8e51640 1 -- start start 2024-04-25T13:45:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a8e51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a410ad00 0x7fa1a41a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a8e51640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a410bb70 0x7fa1a41a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a8e51640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a4110870 0x7fa1a41b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a8e51640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a4115ca0 con 0x7fa1a4110870 2024-04-25T13:45:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a8e51640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a4115b20 con 0x7fa1a410ad00 2024-04-25T13:45:51.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a8e51640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a4115e20 con 0x7fa1a410bb70 2024-04-25T13:45:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a410bb70 0x7fa1a41a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a1d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a410bb70 0x7fa1a41a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:34146/0 (socket says 172.21.15.27:34146) 2024-04-25T13:45:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.011+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/1064827706 learned_addr learned my addr 172.21.15.27:0/1064827706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.013+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/1064827706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a410ad00 msgr2=0x7fa1a41a08d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:45:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.013+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a410ad00 0x7fa1a41a08d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.013+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a4110870 msgr2=0x7fa1a41b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.013+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a4110870 0x7fa1a41b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.013+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/1064827706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a41b94d0 con 0x7fa1a410bb70 2024-04-25T13:45:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.014+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a410bb70 0x7fa1a41a0e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa19800d950 tx=0x7fa19800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.019+0000 7fa1937fe640 1 -- 172.21.15.27:0/1064827706 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa198017070 con 0x7fa1a410bb70 2024-04-25T13:45:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.019+0000 7fa1937fe640 1 -- 172.21.15.27:0/1064827706 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa198009c30 con 0x7fa1a410bb70 2024-04-25T13:45:51.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.020+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1064827706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a41b9760 con 0x7fa1a410bb70 2024-04-25T13:45:51.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.020+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1064827706 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a41b9c80 con 0x7fa1a410bb70 2024-04-25T13:45:51.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.021+0000 7fa1937fe640 1 -- 172.21.15.27:0/1064827706 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19800f620 con 0x7fa1a410bb70 2024-04-25T13:45:51.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.028+0000 7fa1937fe640 1 -- 172.21.15.27:0/1064827706 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa19800f7c0 con 0x7fa1a410bb70 2024-04-25T13:45:51.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.029+0000 7fa1afde8740 1 Processor -- start 2024-04-25T13:45:51.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.029+0000 7fa1afde8740 1 -- start start 2024-04-25T13:45:51.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.029+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca874ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.029+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 0x557ca874efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.029+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca874fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.029+0000 7fa1afde8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557ca86b3d80 con 0x557ca83ad390 2024-04-25T13:45:51.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.029+0000 7fa1afde8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557ca866d8b0 con 0x557ca838c6b0 2024-04-25T13:45:51.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.029+0000 7fa1afde8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557ca86afd40 con 0x557ca83844d0 2024-04-25T13:45:51.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.030+0000 7fa1937fe640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa18c0afd00 0x7fa18c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.031+0000 7fa1937fe640 1 -- 172.21.15.27:0/1064827706 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa1980df3b0 con 0x7fa1a410bb70 2024-04-25T13:45:51.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.032+0000 7fa1a1d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca874fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.032+0000 7fa1a1d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca874fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33260/0 (socket says 172.21.15.27:33260) 2024-04-25T13:45:51.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.032+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/3962395149 learned_addr learned my addr 172.21.15.27:0/3962395149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:51.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.032+0000 7fa1a2d9d640 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca874ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.032+0000 7fa1a2d9d640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa18c0afd00 0x7fa18c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.033+0000 7fa1a259c640 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 0x557ca874efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.035+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/3962395149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 msgr2=0x557ca874efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:51.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.035+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 0x557ca874efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.035+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/3962395149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 msgr2=0x557ca874ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:51.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.035+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca874ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.035+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/3962395149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ca863ea40 con 0x557ca838c6b0 2024-04-25T13:45:51.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.036+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca874fc70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa1980dcdc0 tx=0x7fa198010c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f5cc71dcab6acf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.038+0000 7fa190ff9640 1 -- 172.21.15.27:0/3962395149 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa198020070 con 0x557ca838c6b0 2024-04-25T13:45:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.038+0000 7fa190ff9640 1 -- 172.21.15.27:0/3962395149 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1980a7f60 con 0x557ca838c6b0 2024-04-25T13:45:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.038+0000 7fa190ff9640 1 -- 172.21.15.27:0/3962395149 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1980c8b10 con 0x557ca838c6b0 2024-04-25T13:45:51.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.040+0000 7fa1afde8740 1 -- 172.21.15.27:0/3962395149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 msgr2=0x557ca874fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.040+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca874fc70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa1980dcdc0 tx=0x7fa198010c40 comp rx=0 tx=0).stop 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.041+0000 7fa1afde8740 1 -- 172.21.15.27:0/3962395149 shutdown_connections 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.041+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca874fc70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.041+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 0x557ca874efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.041+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3962395149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca874ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.041+0000 7fa1afde8740 1 -- 172.21.15.27:0/3962395149 >> 172.21.15.27:0/3962395149 conn(0x557ca83a14e0 msgr2=0x557ca874a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.041+0000 7fa1a2d9d640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa18c0afd00 0x7fa18c0b2210 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009420 tx=0x7fa19c004a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.042+0000 7fa1afde8740 1 -- 172.21.15.27:0/3962395149 shutdown_connections 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.042+0000 7fa1afde8740 1 -- 172.21.15.27:0/3962395149 wait complete. 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.043+0000 7fa1afde8740 1 -- start start 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.043+0000 7fa1afde8740 1 Processor -- start 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.043+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83844d0 0x557ca877aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.043+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca877aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.043+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83ad390 0x557ca8781f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.043+0000 7fa1afde8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557ca8750780 con 0x557ca83844d0 2024-04-25T13:45:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.043+0000 7fa1afde8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557ca863ad10 con 0x557ca838c6b0 2024-04-25T13:45:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.043+0000 7fa1afde8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557ca8750900 con 0x557ca83ad390 2024-04-25T13:45:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.044+0000 7fa1afde8740 10 client.0 _ll_register_callbacks cb 0x557ca8667210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.044+0000 7fa1a2d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83ad390 0x557ca8781f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.044+0000 7fa1a259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83844d0 0x557ca877aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.044+0000 7fa1a259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83844d0 0x557ca877aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35842/0 (socket says 172.21.15.27:35842) 2024-04-25T13:45:51.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.044+0000 7fa1a259c640 1 -- 172.21.15.27:0/3582278612 learned_addr learned my addr 172.21.15.27:0/3582278612 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:51.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.046+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca877aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.046+0000 7fa1a259c640 1 -- 172.21.15.27:0/3582278612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83ad390 msgr2=0x557ca8781f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:51.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.046+0000 7fa1a259c640 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83ad390 0x557ca8781f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.046+0000 7fa1a259c640 1 -- 172.21.15.27:0/3582278612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 msgr2=0x557ca877aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:51.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.046+0000 7fa1a259c640 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca877aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.046+0000 7fa1a259c640 1 -- 172.21.15.27:0/3582278612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ca848f7d0 con 0x557ca83844d0 2024-04-25T13:45:51.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.047+0000 7fa1a259c640 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83844d0 0x557ca877aab0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa19400f7e0 tx=0x7fa19400fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697b63d821c418e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:51.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.047+0000 7fa189ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:45:51.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.048+0000 7fa189ffb640 1 -- 172.21.15.27:0/3582278612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19400bd50 con 0x557ca83844d0 2024-04-25T13:45:51.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.048+0000 7fa1afde8740 1 -- 172.21.15.27:0/3582278612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1980a7f60 con 0x557ca83844d0 2024-04-25T13:45:51.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.048+0000 7fa189ffb640 1 -- 172.21.15.27:0/3582278612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa194005020 con 0x557ca83844d0 2024-04-25T13:45:51.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.048+0000 7fa189ffb640 1 -- 172.21.15.27:0/3582278612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa194005220 con 0x557ca83844d0 2024-04-25T13:45:51.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.050+0000 7fa1afde8740 10 client.16377 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:51.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.050+0000 7fa1afde8740 1 -- 172.21.15.27:0/3582278612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557ca83c53f0 con 0x557ca83844d0 2024-04-25T13:45:51.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.051+0000 7fa189ffb640 1 -- 172.21.15.27:0/3582278612 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1980a7f60 con 0x557ca83844d0 2024-04-25T13:45:51.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.051+0000 7fa189ffb640 1 -- 172.21.15.27:0/3582278612 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa194016030 con 0x557ca83844d0 2024-04-25T13:45:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.051+0000 7fa1afde8740 10 client.16377 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.051+0000 7fa1afde8740 10 client.16377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.051+0000 7fa1afde8740 20 client.16377 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.051+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ca836bf60 0x557ca8788480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.051+0000 7fa1afde8740 4 client.16377 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:51.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.051+0000 7fa1afde8740 1 -- 172.21.15.27:0/3582278612 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ca86c1990 con 0x557ca836bf60 2024-04-25T13:45:51.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.053+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ca836bf60 0x557ca8788480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.054+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ca836bf60 0x557ca8788480 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72771ed00b3515c8 server_cookie=7ae5a1c20828ce62 in_seq=0 out_seq=0 2024-04-25T13:45:51.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.054+0000 7fa189ffb640 10 client.16377 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.055+0000 7fa189ffb640 1 -- 172.21.15.27:0/3582278612 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557ca86c1990 con 0x557ca836bf60 2024-04-25T13:45:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.055+0000 7fa189ffb640 10 client.16377 handle_command_reply: tid=0 2024-04-25T13:45:51.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.055+0000 7fa1afde8740 1 client.16377 shutdown 2024-04-25T13:45:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.056+0000 7fa1afde8740 1 -- 172.21.15.27:0/3582278612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83844d0 msgr2=0x557ca877aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.056+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83844d0 0x557ca877aab0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa19400f7e0 tx=0x7fa19400fca0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.056+0000 7fa1afde8740 1 -- 172.21.15.27:0/3582278612 shutdown_connections 2024-04-25T13:45:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.056+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ca836bf60 0x557ca8788480 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.056+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83ad390 0x557ca8781f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.056+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca877aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.056+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3582278612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83844d0 0x557ca877aab0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.056+0000 7fa1afde8740 1 -- 172.21.15.27:0/3582278612 >> 172.21.15.27:0/3582278612 conn(0x557ca83a14e0 msgr2=0x557ca87493f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:51.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.057+0000 7fa189ffb640 0 client.16377 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:51.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.057+0000 7fa1afde8740 1 -- 172.21.15.27:0/3582278612 shutdown_connections 2024-04-25T13:45:51.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.057+0000 7fa1afde8740 1 -- 172.21.15.27:0/3582278612 wait complete. 2024-04-25T13:45:51.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.061+0000 7fa1afde8740 20 client.16377 trim_cache size 0 max 16384 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1afde8740 1 Processor -- start 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1afde8740 1 -- start start 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 0x557ca874a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca87577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca8781e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1afde8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa18400c470 con 0x557ca83ad390 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1afde8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa18400ecf0 con 0x557ca838c6b0 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1afde8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa18400cae0 con 0x557ca83844d0 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1a2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca8781e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1a2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca8781e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35846/0 (socket says 172.21.15.27:35846) 2024-04-25T13:45:51.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1a2d9d640 1 -- 172.21.15.27:0/338525706 learned_addr learned my addr 172.21.15.27:0/338525706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca87577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.084+0000 7fa1a259c640 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 0x557ca874a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/338525706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 msgr2=0x557ca874a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 0x557ca874a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/338525706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 msgr2=0x557ca8781e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca8781e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa1a1d9b640 1 -- 172.21.15.27:0/338525706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ca83850f0 con 0x557ca838c6b0 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca87577d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x557ca8750430 tx=0x7fa1980aef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8da6fa6c646c45b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa18bfff640 1 -- 172.21.15.27:0/338525706 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa198020040 con 0x557ca838c6b0 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa18bfff640 1 -- 172.21.15.27:0/338525706 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa198009de0 con 0x557ca838c6b0 2024-04-25T13:45:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.085+0000 7fa18bfff640 1 -- 172.21.15.27:0/338525706 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1980a8b90 con 0x557ca838c6b0 2024-04-25T13:45:51.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 -- 172.21.15.27:0/338525706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 msgr2=0x557ca87577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:51.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca87577d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x557ca8750430 tx=0x7fa1980aef90 comp rx=0 tx=0).stop 2024-04-25T13:45:51.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 -- 172.21.15.27:0/338525706 shutdown_connections 2024-04-25T13:45:51.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca8781e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca838c6b0 0x557ca87577d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 --2- 172.21.15.27:0/338525706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca83844d0 0x557ca874a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 -- 172.21.15.27:0/338525706 >> 172.21.15.27:0/338525706 conn(0x557ca83a14e0 msgr2=0x557ca87570c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 -- 172.21.15.27:0/338525706 shutdown_connections 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.086+0000 7fa1afde8740 1 -- 172.21.15.27:0/338525706 wait complete. 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.087+0000 7fa1afde8740 1 -- start start 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.087+0000 7fa1afde8740 1 Processor -- start 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.087+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca83844d0 0x557ca877d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca838c6b0 0x557ca877dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1afde8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca87ee820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1afde8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa18400d1b0 con 0x557ca83ad390 2024-04-25T13:45:51.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1afde8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa18400e620 con 0x557ca83844d0 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1afde8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa18400d880 con 0x557ca838c6b0 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca87ee820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca83844d0 0x557ca877d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca87ee820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35848/0 (socket says 172.21.15.27:35848) 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a2d9d640 1 -- 172.21.15.27:0/3526641566 learned_addr learned my addr 172.21.15.27:0/3526641566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1afde8740 10 client.0 _ll_register_callbacks cb 0x7fa19400d400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca838c6b0 0x557ca877dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a259c640 1 -- 172.21.15.27:0/3526641566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca838c6b0 msgr2=0x557ca877dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a259c640 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca838c6b0 0x557ca877dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a259c640 1 -- 172.21.15.27:0/3526641566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 msgr2=0x557ca87ee820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a259c640 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca87ee820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a259c640 1 -- 172.21.15.27:0/3526641566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ca86744b0 con 0x557ca83844d0 2024-04-25T13:45:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca838c6b0 0x557ca877dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a2d9d640 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca87ee820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa1a259c640 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca83844d0 0x557ca877d830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa194023610 tx=0x7fa194023a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7447e0e764df38f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.088+0000 7fa189ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa189ffb640 1 -- 172.21.15.27:0/3526641566 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194029040 con 0x557ca83844d0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa189ffb640 1 -- 172.21.15.27:0/3526641566 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa19400bd50 con 0x557ca83844d0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 1 -- 172.21.15.27:0/3526641566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa198009de0 con 0x557ca83844d0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa189ffb640 1 -- 172.21.15.27:0/3526641566 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa194000e60 con 0x557ca83844d0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 10 client.26099 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 1 -- 172.21.15.27:0/3526641566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557ca87efe10 con 0x557ca83844d0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa189ffb640 1 -- 172.21.15.27:0/3526641566 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa198009de0 con 0x557ca83844d0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa189ffb640 1 -- 172.21.15.27:0/3526641566 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa194016040 con 0x557ca83844d0 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 10 client.26099 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 10 client.26099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 20 client.26099 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ca83413a0 0x557ca87f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 4 client.26099 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.089+0000 7fa1afde8740 1 -- 172.21.15.27:0/3526641566 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557ca86bb800 con 0x557ca83413a0 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.090+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ca83413a0 0x557ca87f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.090+0000 7fa189ffb640 10 client.26099 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.090+0000 7fa1a1d9b640 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ca83413a0 0x557ca87f27d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3e608c1b6fc93cd server_cookie=25aa39257e735f46 in_seq=0 out_seq=0 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.091+0000 7fa189ffb640 1 -- 172.21.15.27:0/3526641566 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557ca86bb800 con 0x557ca83413a0 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.091+0000 7fa189ffb640 10 client.26099 handle_command_reply: tid=0 2024-04-25T13:45:51.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.091+0000 7fa1afde8740 1 client.26099 shutdown 2024-04-25T13:45:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.091+0000 7fa1afde8740 1 -- 172.21.15.27:0/3526641566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca83844d0 msgr2=0x557ca877d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.091+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca83844d0 0x557ca877d830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa194023610 tx=0x7fa194023a30 comp rx=0 tx=0).stop 2024-04-25T13:45:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.093+0000 7fa1afde8740 1 -- 172.21.15.27:0/3526641566 shutdown_connections 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.093+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557ca83413a0 0x557ca87f27d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.093+0000 7fa189ffb640 0 client.26099 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.093+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557ca83ad390 0x557ca87ee820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.093+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557ca838c6b0 0x557ca877dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.093+0000 7fa1afde8740 1 --2- 172.21.15.27:0/3526641566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ca83844d0 0x557ca877d830 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.093+0000 7fa1afde8740 1 -- 172.21.15.27:0/3526641566 >> 172.21.15.27:0/3526641566 conn(0x557ca83a14e0 msgr2=0x557ca874d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.093+0000 7fa1afde8740 1 -- 172.21.15.27:0/3526641566 shutdown_connections 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.094+0000 7fa1afde8740 1 -- 172.21.15.27:0/3526641566 wait complete. 2024-04-25T13:45:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.094+0000 7fa1afde8740 20 client.26099 trim_cache size 0 max 16384 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout: "ce4fbb66-386e-4b55-ad1e-4273b5528ff4": { 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "ce4fbb66-386e-4b55-ad1e-4273b5528ff4", 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:45:51.097 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:45:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.094+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa18c0afd00 msgr2=0x7fa18c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.094+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa18c0afd00 0x7fa18c0b2210 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009420 tx=0x7fa19c004a60 comp rx=0 tx=0).stop 2024-04-25T13:45:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.094+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1064827706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a410bb70 msgr2=0x7fa1a41a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.094+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a410bb70 0x7fa1a41a0e10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa19800d950 tx=0x7fa19800de10 comp rx=0 tx=0).stop 2024-04-25T13:45:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.095+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1064827706 shutdown_connections 2024-04-25T13:45:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.095+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa18c0afd00 0x7fa18c0b2210 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.095+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a4110870 0x7fa1a41b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.095+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1a410bb70 0x7fa1a41a0e10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.095+0000 7fa1a8e51640 1 --2- 172.21.15.27:0/1064827706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a410ad00 0x7fa1a41a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.095+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1064827706 >> 172.21.15.27:0/1064827706 conn(0x7fa1a4070940 msgr2=0x7fa1a410c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.095+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1064827706 shutdown_connections 2024-04-25T13:45:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:45:51.095+0000 7fa1a8e51640 1 -- 172.21.15.27:0/1064827706 wait complete. 2024-04-25T13:45:51.106 INFO:tasks.cephfs.filesystem:scrub status for tag:ce4fbb66-386e-4b55-ad1e-4273b5528ff4 - {'path': '/', 'tag': 'ce4fbb66-386e-4b55-ad1e-4273b5528ff4', 'options': 'recursive,force'} 2024-04-25T13:45:51.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:50 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:45:51.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:50 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:51.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:50 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:45:51.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:50 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:45:51.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:50 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:51.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:50 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:45:52.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:51 smithi027 ceph-mon[29140]: pgmap v498: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-04-25T13:45:52.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:51 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:52.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:51 smithi122 ceph-mon[32843]: pgmap v498: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-04-25T13:45:52.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:51 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:52.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:51 smithi045 ceph-mon[32731]: pgmap v498: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-04-25T13:45:52.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:51 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:45:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 285.0 s, 2465.4 tps, lat 40.659 ms stddev 43.988 2024-04-25T13:45:54.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:53 smithi027 ceph-mon[29140]: pgmap v499: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 7.8 MiB/s wr, 864 op/s 2024-04-25T13:45:54.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:53 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:45:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:53 smithi122 ceph-mon[32843]: pgmap v499: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 7.8 MiB/s wr, 864 op/s 2024-04-25T13:45:54.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:53 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:45:54.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:53 smithi045 ceph-mon[32731]: pgmap v499: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 7.8 MiB/s wr, 864 op/s 2024-04-25T13:45:54.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:53 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:45:56.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:55 smithi027 ceph-mon[29140]: pgmap v500: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 6.7 MiB/s wr, 790 op/s 2024-04-25T13:45:56.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:55 smithi122 ceph-mon[32843]: pgmap v500: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 6.7 MiB/s wr, 790 op/s 2024-04-25T13:45:56.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:55 smithi045 ceph-mon[32731]: pgmap v500: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 6.7 MiB/s wr, 790 op/s 2024-04-25T13:45:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 290.0 s, 2212.0 tps, lat 44.716 ms stddev 50.708 2024-04-25T13:45:58.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:57 smithi027 ceph-mon[29140]: pgmap v501: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 8.1 MiB/s wr, 944 op/s 2024-04-25T13:45:58.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:57 smithi122 ceph-mon[32843]: pgmap v501: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 8.1 MiB/s wr, 944 op/s 2024-04-25T13:45:58.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:57 smithi045 ceph-mon[32731]: pgmap v501: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 8.1 MiB/s wr, 944 op/s 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 -- 172.21.15.27:0/1345370821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f049010f760 msgr2=0x7f0490111b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 --2- 172.21.15.27:0/1345370821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f049010f760 0x7f0490111b80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f048c009900 tx=0x7f048c02a410 comp rx=0 tx=0).stop 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 -- 172.21.15.27:0/1345370821 shutdown_connections 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 --2- 172.21.15.27:0/1345370821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04901120c0 0x7f0490114560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 --2- 172.21.15.27:0/1345370821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f049010f760 0x7f0490111b80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 --2- 172.21.15.27:0/1345370821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0490109fc0 0x7f049010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 -- 172.21.15.27:0/1345370821 >> 172.21.15.27:0/1345370821 conn(0x7f0490070540 msgr2=0x7f04900729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 -- 172.21.15.27:0/1345370821 shutdown_connections 2024-04-25T13:45:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.641+0000 7f04973c5640 1 -- 172.21.15.27:0/1345370821 wait complete. 2024-04-25T13:45:59.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.642+0000 7f04973c5640 1 Processor -- start 2024-04-25T13:45:59.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.642+0000 7f04973c5640 1 -- start start 2024-04-25T13:45:59.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.642+0000 7f04973c5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0490109fc0 0x7f049010cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.642+0000 7f04973c5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f049010f760 0x7f049010b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.642+0000 7f04973c5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04901120c0 0x7f049010bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.642+0000 7f04973c5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0490117ed0 con 0x7f049010f760 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.642+0000 7f04973c5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0490117d50 con 0x7f0490109fc0 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.642+0000 7f04973c5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0490118050 con 0x7f04901120c0 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0490109fc0 0x7f049010cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0490109fc0 0x7f049010cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49908/0 (socket says 172.21.15.27:49908) 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 -- 172.21.15.27:0/142474341 learned_addr learned my addr 172.21.15.27:0/142474341 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0494960640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f049010f760 0x7f049010b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495962640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04901120c0 0x7f049010bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 -- 172.21.15.27:0/142474341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04901120c0 msgr2=0x7f049010bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04901120c0 0x7f049010bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 -- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f049010f760 msgr2=0x7f049010b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f049010f760 0x7f049010b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 -- 172.21.15.27:0/142474341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04901a69d0 con 0x7f0490109fc0 2024-04-25T13:45:59.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.643+0000 7f0495161640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0490109fc0 0x7f049010cfc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f048000d740 tx=0x7f048000db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:59.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.644+0000 7f04867fc640 1 -- 172.21.15.27:0/142474341 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480017070 con 0x7f0490109fc0 2024-04-25T13:45:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.644+0000 7f04867fc640 1 -- 172.21.15.27:0/142474341 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0480009820 con 0x7f0490109fc0 2024-04-25T13:45:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.644+0000 7f04867fc640 1 -- 172.21.15.27:0/142474341 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480004ce0 con 0x7f0490109fc0 2024-04-25T13:45:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.644+0000 7f04973c5640 1 -- 172.21.15.27:0/142474341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f049010c200 con 0x7f0490109fc0 2024-04-25T13:45:59.645 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.644+0000 7f04973c5640 1 -- 172.21.15.27:0/142474341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04901b15d0 con 0x7f0490109fc0 2024-04-25T13:45:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.647+0000 7f04867fc640 1 -- 172.21.15.27:0/142474341 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0480005220 con 0x7f0490109fc0 2024-04-25T13:45:59.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.648+0000 7f049e278740 1 Processor -- start 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.648+0000 7f049e278740 1 -- start start 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f049e278740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a2241b0 0x558a2a5c4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f049e278740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a1fb2f0 0x558a2a5c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f049e278740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f049e278740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558a2a52a460 con 0x558a2a2241b0 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f049e278740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558a2a4e4590 con 0x558a2a2034d0 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f049e278740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558a2a526420 con 0x558a2a1fb2f0 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f04867fc640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f047c0afb60 0x7f047c0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f0495962640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a2241b0 0x558a2a5c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f0494960640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a1fb2f0 0x558a2a5c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f0494960640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a1fb2f0 0x558a2a5c4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57084/0 (socket says 172.21.15.27:57084) 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f0494960640 1 -- 172.21.15.27:0/2910666718 learned_addr learned my addr 172.21.15.27:0/2910666718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f04867fc640 1 -- 172.21.15.27:0/142474341 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f04800de4b0 con 0x7f0490109fc0 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.649+0000 7f0495161640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f047c0afb60 0x7f047c0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0495962640 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0494960640 1 -- 172.21.15.27:0/2910666718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 msgr2=0x558a2a5cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0494960640 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0494960640 1 -- 172.21.15.27:0/2910666718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a2241b0 msgr2=0x558a2a5c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0494960640 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a2241b0 0x558a2a5c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0494960640 1 -- 172.21.15.27:0/2910666718 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a2a4f10d0 con 0x558a2a1fb2f0 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0495962640 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0495161640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f047c0afb60 0x7f047c0b2070 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f04800dc350 tx=0x7f04800c9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.650+0000 7f0494960640 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a1fb2f0 0x558a2a5c4ef0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f048c031500 tx=0x7f048c031b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78120b8ff123cb08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:59.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.651+0000 7f047b7fe640 1 -- 172.21.15.27:0/2910666718 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c035030 con 0x558a2a1fb2f0 2024-04-25T13:45:59.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.651+0000 7f047b7fe640 1 -- 172.21.15.27:0/2910666718 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f048c040070 con 0x558a2a1fb2f0 2024-04-25T13:45:59.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.651+0000 7f047b7fe640 1 -- 172.21.15.27:0/2910666718 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c003950 con 0x558a2a1fb2f0 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 -- 172.21.15.27:0/2910666718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a1fb2f0 msgr2=0x558a2a5c4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a1fb2f0 0x558a2a5c4ef0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f048c031500 tx=0x7f048c031b10 comp rx=0 tx=0).stop 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 -- 172.21.15.27:0/2910666718 shutdown_connections 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a1fb2f0 0x558a2a5c4ef0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 --2- 172.21.15.27:0/2910666718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a2241b0 0x558a2a5c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 -- 172.21.15.27:0/2910666718 >> 172.21.15.27:0/2910666718 conn(0x558a2a218300 msgr2=0x558a2a5c0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 -- 172.21.15.27:0/2910666718 shutdown_connections 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 -- 172.21.15.27:0/2910666718 wait complete. 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.652+0000 7f049e278740 1 -- start start 2024-04-25T13:45:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f049e278740 1 Processor -- start 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f049e278740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f049e278740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f049e278740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2241b0 0x558a2a5d5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f049e278740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558a2a5cc7e0 con 0x558a2a1fb2f0 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f049e278740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558a2a4b1b60 con 0x558a2a2034d0 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f049e278740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558a2a5cc960 con 0x558a2a2241b0 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f0494960640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f0495962640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f0494960640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5ceb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49918/0 (socket says 172.21.15.27:49918) 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f0495962640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35096/0 (socket says 172.21.15.27:35096) 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f049e278740 10 client.0 _ll_register_callbacks cb 0x558a2a4d62b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.653+0000 7f0494960640 1 -- 172.21.15.27:0/4225201279 learned_addr learned my addr 172.21.15.27:0/4225201279 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.654+0000 7f0495962640 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2241b0 0x558a2a5d5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.654+0000 7f0494960640 1 -- 172.21.15.27:0/4225201279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2241b0 msgr2=0x558a2a5d5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.654+0000 7f0494960640 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2241b0 0x558a2a5d5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.654+0000 7f0494960640 1 -- 172.21.15.27:0/4225201279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 msgr2=0x558a2a5ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.654+0000 7f0494960640 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.654+0000 7f0494960640 1 -- 172.21.15.27:0/4225201279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a2a23c210 con 0x558a2a2034d0 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.654+0000 7f0495962640 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.654+0000 7f0494960640 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5ceb80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f048c009a20 tx=0x7f048c0317d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ad2c0e399a7b7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.655+0000 7f0478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.655+0000 7f0478ff9640 1 -- 172.21.15.27:0/4225201279 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c053060 con 0x558a2a2034d0 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.655+0000 7f0478ff9640 1 -- 172.21.15.27:0/4225201279 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f048c0036d0 con 0x558a2a2034d0 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.655+0000 7f049e278740 1 -- 172.21.15.27:0/4225201279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f048c040070 con 0x558a2a2034d0 2024-04-25T13:45:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.655+0000 7f0478ff9640 1 -- 172.21.15.27:0/4225201279 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f048c035070 con 0x558a2a2034d0 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f049e278740 10 client.26108 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f049e278740 1 -- 172.21.15.27:0/4225201279 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x558a2a308620 con 0x558a2a2034d0 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f0478ff9640 1 -- 172.21.15.27:0/4225201279 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f048c003d80 con 0x558a2a2034d0 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f0478ff9640 1 -- 172.21.15.27:0/4225201279 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f048c04d030 con 0x558a2a2034d0 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f049e278740 10 client.26108 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f049e278740 10 client.26108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f049e278740 20 client.26108 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f049e278740 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558a2a1e2d80 0x558a2a5dbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.656+0000 7f049e278740 4 client.26108 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:45:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.657+0000 7f049e278740 1 -- 172.21.15.27:0/4225201279 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f048c040070 con 0x558a2a1e2d80 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.657+0000 7f0495962640 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558a2a1e2d80 0x558a2a5dbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.657+0000 7f0478ff9640 10 client.26108 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.657+0000 7f0495962640 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558a2a1e2d80 0x558a2a5dbea0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b11bb05a5d5a6a server_cookie=8dbb1a8eb6aef59d in_seq=0 out_seq=0 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.658+0000 7f0478ff9640 1 -- 172.21.15.27:0/4225201279 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f048c040070 con 0x558a2a1e2d80 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.658+0000 7f0478ff9640 10 client.26108 handle_command_reply: tid=0 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.658+0000 7f049e278740 1 client.26108 shutdown 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 -- 172.21.15.27:0/4225201279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 msgr2=0x558a2a5ceb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5ceb80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f048c009a20 tx=0x7f048c0317d0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 -- 172.21.15.27:0/4225201279 shutdown_connections 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558a2a1e2d80 0x558a2a5dbea0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2241b0 0x558a2a5d5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f0478ff9640 0 client.26108 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a2034d0 0x558a2a5ceb80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 --2- 172.21.15.27:0/4225201279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 -- 172.21.15.27:0/4225201279 >> 172.21.15.27:0/4225201279 conn(0x558a2a218300 msgr2=0x558a2a5c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 -- 172.21.15.27:0/4225201279 shutdown_connections 2024-04-25T13:45:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.659+0000 7f049e278740 1 -- 172.21.15.27:0/4225201279 wait complete. 2024-04-25T13:45:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.660+0000 7f049e278740 20 client.26108 trim_cache size 0 max 16384 2024-04-25T13:45:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.684+0000 7f049e278740 1 Processor -- start 2024-04-25T13:45:59.684 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.685+0000 7f049e278740 1 -- start start 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.685+0000 7f049e278740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5bfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.685+0000 7f049e278740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.686+0000 7f0495962640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5bfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.686+0000 7f0495962640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5bfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49930/0 (socket says 172.21.15.27:49930) 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.686+0000 7f0495962640 1 -- 172.21.15.27:0/1419607269 learned_addr learned my addr 172.21.15.27:0/1419607269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.686+0000 7f049e278740 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 0x558a2a5c0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.686+0000 7f0494960640 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.686+0000 7f049e278740 1 -- 172.21.15.27:0/1419607269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f04901b8070 con 0x558a2a1fb2f0 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.686+0000 7f049e278740 1 -- 172.21.15.27:0/1419607269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04901ba8f0 con 0x558a2a1e2d80 2024-04-25T13:45:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.686+0000 7f049e278740 1 -- 172.21.15.27:0/1419607269 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f04901b86e0 con 0x558a2a2034d0 2024-04-25T13:45:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.687+0000 7f0494960640 1 -- 172.21.15.27:0/1419607269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 msgr2=0x558a2a5c0190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:45:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.687+0000 7f0494960640 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 0x558a2a5c0190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.687+0000 7f0494960640 1 -- 172.21.15.27:0/1419607269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 msgr2=0x558a2a5bfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.687+0000 7f0494960640 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5bfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.687+0000 7f0494960640 1 -- 172.21.15.27:0/1419607269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a2a5ccaa0 con 0x558a2a1fb2f0 2024-04-25T13:45:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.687+0000 7f0495962640 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5bfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:45:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.687+0000 7f0494960640 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d5a00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f048c02a930 tx=0x7f048c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc938a7282b8cf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:59.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.688+0000 7f047affd640 1 -- 172.21.15.27:0/1419607269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c053040 con 0x558a2a1fb2f0 2024-04-25T13:45:59.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.688+0000 7f047affd640 1 -- 172.21.15.27:0/1419607269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f048c040070 con 0x558a2a1fb2f0 2024-04-25T13:45:59.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.688+0000 7f047affd640 1 -- 172.21.15.27:0/1419607269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c05d080 con 0x558a2a1fb2f0 2024-04-25T13:45:59.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.689+0000 7f049e278740 1 -- 172.21.15.27:0/1419607269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 msgr2=0x558a2a5d5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.689+0000 7f049e278740 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d5a00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f048c02a930 tx=0x7f048c031400 comp rx=0 tx=0).stop 2024-04-25T13:45:59.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.689+0000 7f049e278740 1 -- 172.21.15.27:0/1419607269 shutdown_connections 2024-04-25T13:45:59.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.689+0000 7f049e278740 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 0x558a2a5c0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.689+0000 7f049e278740 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d5a00 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.689+0000 7f049e278740 1 --2- 172.21.15.27:0/1419607269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5bfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.690+0000 7f049e278740 1 -- 172.21.15.27:0/1419607269 >> 172.21.15.27:0/1419607269 conn(0x558a2a218300 msgr2=0x558a2a5c8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:59.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.690+0000 7f049e278740 1 -- 172.21.15.27:0/1419607269 shutdown_connections 2024-04-25T13:45:59.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.690+0000 7f049e278740 1 -- 172.21.15.27:0/1419607269 wait complete. 2024-04-25T13:45:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.691+0000 7f049e278740 1 -- start start 2024-04-25T13:45:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.691+0000 7f049e278740 1 Processor -- start 2024-04-25T13:45:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f049e278740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5d5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f049e278740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f049e278740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 0x558a2a6425f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f049e278740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f04901b8db0 con 0x558a2a1fb2f0 2024-04-25T13:45:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f049e278740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04901ba220 con 0x558a2a1e2d80 2024-04-25T13:45:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f0495962640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5d5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f0495962640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5d5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49940/0 (socket says 172.21.15.27:49940) 2024-04-25T13:45:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f0495962640 1 -- 172.21.15.27:0/1179132539 learned_addr learned my addr 172.21.15.27:0/1179132539 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:45:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f0494960640 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.692+0000 7f049e278740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f04901b9480 con 0x558a2a2034d0 2024-04-25T13:45:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.693+0000 7f0495962640 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 0x558a2a6425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.693+0000 7f049e278740 10 client.0 _ll_register_callbacks cb 0x7f0488009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:45:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0494960640 1 -- 172.21.15.27:0/1179132539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 msgr2=0x558a2a6425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0494960640 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 0x558a2a6425f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0494960640 1 -- 172.21.15.27:0/1179132539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 msgr2=0x558a2a5d5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:45:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0494960640 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5d5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0494960640 1 -- 172.21.15.27:0/1179132539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a2a5d5f40 con 0x558a2a1fb2f0 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0494960640 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d56d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x558a2a5bbc10 tx=0x7f048c003800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92822c1476319d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0478ff9640 1 -- 172.21.15.27:0/1179132539 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c05d070 con 0x558a2a1fb2f0 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.694+0000 7f0478ff9640 1 -- 172.21.15.27:0/1179132539 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f048c003e00 con 0x558a2a1fb2f0 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.695+0000 7f049e278740 1 -- 172.21.15.27:0/1179132539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f048c040070 con 0x558a2a1fb2f0 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.695+0000 7f0478ff9640 1 -- 172.21.15.27:0/1179132539 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f048c053040 con 0x558a2a1fb2f0 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.695+0000 7f049e278740 10 client.16401 fetch_fsmap learned FSMap version 19 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.695+0000 7f049e278740 1 -- 172.21.15.27:0/1179132539 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x558a2a643c40 con 0x558a2a1fb2f0 2024-04-25T13:45:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.695+0000 7f0478ff9640 1 -- 172.21.15.27:0/1179132539 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f048c040070 con 0x558a2a1fb2f0 2024-04-25T13:45:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.696+0000 7f0478ff9640 1 -- 172.21.15.27:0/1179132539 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f048c04d050 con 0x558a2a1fb2f0 2024-04-25T13:45:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.696+0000 7f049e278740 10 client.16401 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:45:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.696+0000 7f049e278740 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:45:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.696+0000 7f049e278740 20 client.16401 populate_metadata read hostname 'smithi027' 2024-04-25T13:45:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.696+0000 7f049e278740 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558a2a2241b0 0x558a2a6464d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:45:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.696+0000 7f049e278740 4 client.16401 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:45:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.696+0000 7f049e278740 1 -- 172.21.15.27:0/1179132539 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558a2a538070 con 0x558a2a2241b0 2024-04-25T13:45:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.697+0000 7f0495962640 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558a2a2241b0 0x558a2a6464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:45:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.697+0000 7f0478ff9640 10 client.16401 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.697+0000 7f0495962640 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558a2a2241b0 0x558a2a6464d0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a63764fce4b31e2c server_cookie=a2764f0e80a9f987 in_seq=0 out_seq=0 2024-04-25T13:45:59.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.712+0000 7f0478ff9640 1 -- 172.21.15.27:0/1179132539 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558a2a538070 con 0x558a2a2241b0 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.712+0000 7f0478ff9640 10 client.16401 handle_command_reply: tid=0 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.712+0000 7f049e278740 1 client.16401 shutdown 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.712+0000 7f049e278740 1 -- 172.21.15.27:0/1179132539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 msgr2=0x558a2a5d56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.712+0000 7f049e278740 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d56d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x558a2a5bbc10 tx=0x7f048c003800 comp rx=0 tx=0).stop 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 1 -- 172.21.15.27:0/1179132539 shutdown_connections 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558a2a2241b0 0x558a2a6464d0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558a2a2034d0 0x558a2a6425f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558a2a1fb2f0 0x558a2a5d56d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 1 --2- 172.21.15.27:0/1179132539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a2a1e2d80 0x558a2a5d5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f0478ff9640 0 client.16401 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 1 -- 172.21.15.27:0/1179132539 >> 172.21.15.27:0/1179132539 conn(0x558a2a218300 msgr2=0x558a2a5c3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 1 -- 172.21.15.27:0/1179132539 shutdown_connections 2024-04-25T13:45:59.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 1 -- 172.21.15.27:0/1179132539 wait complete. 2024-04-25T13:45:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f049e278740 20 client.16401 trim_cache size 0 max 16384 2024-04-25T13:45:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:45:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:45:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:45:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f04973c5640 1 -- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f047c0afb60 msgr2=0x7f047c0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f04973c5640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f047c0afb60 0x7f047c0b2070 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f04800dc350 tx=0x7f04800c9040 comp rx=0 tx=0).stop 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f04973c5640 1 -- 172.21.15.27:0/142474341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0490109fc0 msgr2=0x7f049010cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f04973c5640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0490109fc0 0x7f049010cfc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f048000d740 tx=0x7f048000db40 comp rx=0 tx=0).stop 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f04973c5640 1 -- 172.21.15.27:0/142474341 shutdown_connections 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f04973c5640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f047c0afb60 0x7f047c0b2070 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f04973c5640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04901120c0 0x7f049010bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.713+0000 7f04973c5640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f049010f760 0x7f049010b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.714+0000 7f04973c5640 1 --2- 172.21.15.27:0/142474341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0490109fc0 0x7f049010cfc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:45:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.714+0000 7f04973c5640 1 -- 172.21.15.27:0/142474341 >> 172.21.15.27:0/142474341 conn(0x7f0490070540 msgr2=0x7f0490110680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:45:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.714+0000 7f04973c5640 1 -- 172.21.15.27:0/142474341 shutdown_connections 2024-04-25T13:45:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:45:59.714+0000 7f04973c5640 1 -- 172.21.15.27:0/142474341 wait complete. 2024-04-25T13:46:00.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:45:59 smithi027 ceph-mon[29140]: pgmap v502: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 6.5 MiB/s wr, 789 op/s 2024-04-25T13:46:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:45:59 smithi122 ceph-mon[32843]: pgmap v502: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 6.5 MiB/s wr, 789 op/s 2024-04-25T13:46:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:45:59 smithi045 ceph-mon[32731]: pgmap v502: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 6.5 MiB/s wr, 789 op/s 2024-04-25T13:46:02.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:01 smithi027 ceph-mon[29140]: pgmap v503: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 7.3 MiB/s wr, 876 op/s 2024-04-25T13:46:02.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:01 smithi122 ceph-mon[32843]: pgmap v503: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 7.3 MiB/s wr, 876 op/s 2024-04-25T13:46:02.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:01 smithi045 ceph-mon[32731]: pgmap v503: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 7.3 MiB/s wr, 876 op/s 2024-04-25T13:46:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 295.0 s, 2764.4 tps, lat 36.557 ms stddev 41.101 2024-04-25T13:46:04.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:03 smithi027 ceph-mon[29140]: pgmap v504: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 7.6 MiB/s wr, 885 op/s 2024-04-25T13:46:04.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:03 smithi122 ceph-mon[32843]: pgmap v504: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 7.6 MiB/s wr, 885 op/s 2024-04-25T13:46:04.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:03 smithi045 ceph-mon[32731]: pgmap v504: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 7.6 MiB/s wr, 885 op/s 2024-04-25T13:46:06.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:05 smithi027 ceph-mon[29140]: pgmap v505: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.0 MiB/s wr, 669 op/s 2024-04-25T13:46:06.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:05 smithi122 ceph-mon[32843]: pgmap v505: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.0 MiB/s wr, 669 op/s 2024-04-25T13:46:06.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:05 smithi045 ceph-mon[32731]: pgmap v505: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.0 MiB/s wr, 669 op/s 2024-04-25T13:46:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 300.0 s, 2461.2 tps, lat 40.582 ms stddev 45.640 2024-04-25T13:46:08.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:07 smithi027 ceph-mon[29140]: pgmap v506: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.0 MiB/s wr, 818 op/s 2024-04-25T13:46:08.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:07 smithi122 ceph-mon[32843]: pgmap v506: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.0 MiB/s wr, 818 op/s 2024-04-25T13:46:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:07 smithi045 ceph-mon[32731]: pgmap v506: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.0 MiB/s wr, 818 op/s 2024-04-25T13:46:10.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:09 smithi027 ceph-mon[29140]: pgmap v507: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 589 op/s 2024-04-25T13:46:10.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:09 smithi122 ceph-mon[32843]: pgmap v507: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 589 op/s 2024-04-25T13:46:10.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:09 smithi045 ceph-mon[32731]: pgmap v507: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 589 op/s 2024-04-25T13:46:12.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:11 smithi027 ceph-mon[29140]: pgmap v508: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 675 op/s 2024-04-25T13:46:12.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:11 smithi122 ceph-mon[32843]: pgmap v508: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 675 op/s 2024-04-25T13:46:12.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:11 smithi045 ceph-mon[32731]: pgmap v508: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 675 op/s 2024-04-25T13:46:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 -- 172.21.15.27:0/1917103793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 msgr2=0x7f2054104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 --2- 172.21.15.27:0/1917103793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 0x7f2054104070 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2044009900 tx=0x7f204402a3f0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 -- 172.21.15.27:0/1917103793 shutdown_connections 2024-04-25T13:46:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 --2- 172.21.15.27:0/1917103793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2054108960 0x7f205410ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 --2- 172.21.15.27:0/1917103793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 0x7f2054104070 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 --2- 172.21.15.27:0/1917103793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2054102a20 0x7f2054103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 -- 172.21.15.27:0/1917103793 >> 172.21.15.27:0/1917103793 conn(0x7f20540fdc10 msgr2=0x7f20541000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:12.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 -- 172.21.15.27:0/1917103793 shutdown_connections 2024-04-25T13:46:12.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.858+0000 7f205c11f640 1 -- 172.21.15.27:0/1917103793 wait complete. 2024-04-25T13:46:12.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f205c11f640 1 Processor -- start 2024-04-25T13:46:12.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f205c11f640 1 -- start start 2024-04-25T13:46:12.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f205c11f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2054102a20 0x7f205406a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f205c11f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 0x7f205406ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f205c11f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2054108960 0x7f2054073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f205c11f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f205406ce20 con 0x7f2054103c60 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f205c11f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f205406cca0 con 0x7f2054108960 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f205c11f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f205406cfa0 con 0x7f2054102a20 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f20596ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 0x7f205406ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f20596ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 0x7f205406ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56068/0 (socket says 172.21.15.27:56068) 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.859+0000 7f20596ba640 1 -- 172.21.15.27:0/4084187556 learned_addr learned my addr 172.21.15.27:0/4084187556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.860+0000 7f2059ebb640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2054102a20 0x7f205406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.860+0000 7f205a6bc640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2054108960 0x7f2054073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.860+0000 7f205a6bc640 1 -- 172.21.15.27:0/4084187556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2054102a20 msgr2=0x7f205406a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.860+0000 7f205a6bc640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2054102a20 0x7f205406a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.860+0000 7f205a6bc640 1 -- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 msgr2=0x7f205406ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.860+0000 7f205a6bc640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 0x7f205406ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.860+0000 7f205a6bc640 1 -- 172.21.15.27:0/4084187556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20540736d0 con 0x7f2054108960 2024-04-25T13:46:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.861+0000 7f2059ebb640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2054102a20 0x7f205406a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.861+0000 7f20596ba640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 0x7f205406ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.861+0000 7f205a6bc640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2054108960 0x7f2054073060 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f204c009fd0 tx=0x7f204c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.861+0000 7f204affd640 1 -- 172.21.15.27:0/4084187556 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c00ce20 con 0x7f2054108960 2024-04-25T13:46:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.861+0000 7f204affd640 1 -- 172.21.15.27:0/4084187556 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f204c005020 con 0x7f2054108960 2024-04-25T13:46:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.862+0000 7f205c11f640 1 -- 172.21.15.27:0/4084187556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20540739c0 con 0x7f2054108960 2024-04-25T13:46:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.862+0000 7f204affd640 1 -- 172.21.15.27:0/4084187556 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c010690 con 0x7f2054108960 2024-04-25T13:46:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.862+0000 7f205c11f640 1 -- 172.21.15.27:0/4084187556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2054073ec0 con 0x7f2054108960 2024-04-25T13:46:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.865+0000 7f204affd640 1 -- 172.21.15.27:0/4084187556 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f204c0054d0 con 0x7f2054108960 2024-04-25T13:46:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.865+0000 7f2062fd2740 1 Processor -- start 2024-04-25T13:46:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f2062fd2740 1 -- start start 2024-04-25T13:46:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28651b0 0x561cd2c05970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c05ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f2062fd2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561cd2b6b460 con 0x561cd283c2f0 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f2062fd2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561cd2b25590 con 0x561cd28444d0 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f2062fd2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561cd2b67420 con 0x561cd28651b0 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f2059ebb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c05ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.866+0000 7f204affd640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f20400afae0 0x7f20400b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.867+0000 7f20596ba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.867+0000 7f20596ba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28651b0 0x561cd2c05970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.867+0000 7f2059ebb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c05ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56074/0 (socket says 172.21.15.27:56074) 2024-04-25T13:46:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.867+0000 7f2059ebb640 1 -- 172.21.15.27:0/167956639 learned_addr learned my addr 172.21.15.27:0/167956639 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.867+0000 7f204affd640 1 -- 172.21.15.27:0/4084187556 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f204c0df2e0 con 0x7f2054108960 2024-04-25T13:46:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.867+0000 7f205a6bc640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f20400afae0 0x7f20400b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.867+0000 7f2059ebb640 1 -- 172.21.15.27:0/167956639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28651b0 msgr2=0x561cd2c05970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.867+0000 7f2059ebb640 1 --2- 172.21.15.27:0/167956639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28651b0 0x561cd2c05970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.868+0000 7f2059ebb640 1 -- 172.21.15.27:0/167956639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 msgr2=0x561cd2c0cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.868+0000 7f2059ebb640 1 --2- 172.21.15.27:0/167956639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.868+0000 7f2059ebb640 1 -- 172.21.15.27:0/167956639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cd2b320d0 con 0x561cd283c2f0 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.868+0000 7f2059ebb640 1 --2- 172.21.15.27:0/167956639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c05ef0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f20500114f0 tx=0x7f20500119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ed3a95038ddc60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.869+0000 7f205a6bc640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f20400afae0 0x7f20400b1ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f204c01a040 tx=0x7f204c0057c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.869+0000 7f203ffff640 1 -- 172.21.15.27:0/167956639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205001e070 con 0x561cd283c2f0 2024-04-25T13:46:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.869+0000 7f203ffff640 1 -- 172.21.15.27:0/167956639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20500035f0 con 0x561cd283c2f0 2024-04-25T13:46:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.869+0000 7f203ffff640 1 -- 172.21.15.27:0/167956639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205000f880 con 0x561cd283c2f0 2024-04-25T13:46:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 -- 172.21.15.27:0/167956639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 msgr2=0x561cd2c05ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 --2- 172.21.15.27:0/167956639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c05ef0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f20500114f0 tx=0x7f20500119b0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 -- 172.21.15.27:0/167956639 shutdown_connections 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 --2- 172.21.15.27:0/167956639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 --2- 172.21.15.27:0/167956639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c05ef0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 --2- 172.21.15.27:0/167956639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28651b0 0x561cd2c05970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 -- 172.21.15.27:0/167956639 >> 172.21.15.27:0/167956639 conn(0x561cd2859300 msgr2=0x561cd2c01300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 -- 172.21.15.27:0/167956639 shutdown_connections 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 -- 172.21.15.27:0/167956639 wait complete. 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 -- start start 2024-04-25T13:46:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.870+0000 7f2062fd2740 1 Processor -- start 2024-04-25T13:46:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.871+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c0f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.871+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.871+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c16af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.871+0000 7f2062fd2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561cd2c0d840 con 0x561cd28651b0 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.871+0000 7f2062fd2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561cd2af2b60 con 0x561cd28444d0 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.871+0000 7f2062fd2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561cd2c0d9c0 con 0x561cd283c2f0 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.871+0000 7f2062fd2740 10 client.0 _ll_register_callbacks cb 0x561cd2b172b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c16af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c16af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56084/0 (socket says 172.21.15.27:56084) 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 -- 172.21.15.27:0/3849918391 learned_addr learned my addr 172.21.15.27:0/3849918391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c0f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 -- 172.21.15.27:0/3849918391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 msgr2=0x561cd2c0f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c0f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 -- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 msgr2=0x561cd2c0fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 -- 172.21.15.27:0/3849918391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cd287d210 con 0x561cd28651b0 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f203d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.872+0000 7f20596ba640 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c16af0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2044031440 tx=0x7f204403f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71de712bbe7541c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.873+0000 7f203d7fa640 1 -- 172.21.15.27:0/3849918391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044035040 con 0x561cd28651b0 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.873+0000 7f2062fd2740 1 -- 172.21.15.27:0/3849918391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20500035f0 con 0x561cd28651b0 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.873+0000 7f203d7fa640 1 -- 172.21.15.27:0/3849918391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2044046070 con 0x561cd28651b0 2024-04-25T13:46:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.873+0000 7f203d7fa640 1 -- 172.21.15.27:0/3849918391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2044035040 con 0x561cd28651b0 2024-04-25T13:46:12.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.873+0000 7f2062fd2740 10 client.16413 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:12.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.873+0000 7f2062fd2740 1 -- 172.21.15.27:0/3849918391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x561cd2949620 con 0x561cd28651b0 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.874+0000 7f203d7fa640 1 -- 172.21.15.27:0/3849918391 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20500035f0 con 0x561cd28651b0 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.874+0000 7f203d7fa640 1 -- 172.21.15.27:0/3849918391 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f204404e4c0 con 0x561cd28651b0 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.874+0000 7f2062fd2740 10 client.16413 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.875+0000 7f2062fd2740 10 client.16413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.875+0000 7f2062fd2740 20 client.16413 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.875+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561cd2823d80 0x561cd2c1d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.875+0000 7f2062fd2740 4 client.16413 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.875+0000 7f2062fd2740 1 -- 172.21.15.27:0/3849918391 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cd2b79070 con 0x561cd2823d80 2024-04-25T13:46:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.875+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561cd2823d80 0x561cd2c1d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.876+0000 7f203d7fa640 10 client.16413 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.876+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561cd2823d80 0x561cd2c1d2d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15eba4c86b709398 server_cookie=6464d0819bd83928 in_seq=0 out_seq=0 2024-04-25T13:46:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.877+0000 7f203d7fa640 1 -- 172.21.15.27:0/3849918391 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561cd2b79070 con 0x561cd2823d80 2024-04-25T13:46:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.877+0000 7f203d7fa640 10 client.16413 handle_command_reply: tid=0 2024-04-25T13:46:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.877+0000 7f2062fd2740 1 client.16413 shutdown 2024-04-25T13:46:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 -- 172.21.15.27:0/3849918391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 msgr2=0x561cd2c16af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c16af0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2044031440 tx=0x7f204403f040 comp rx=0 tx=0).stop 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 -- 172.21.15.27:0/3849918391 shutdown_connections 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561cd2823d80 0x561cd2c1d2d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c16af0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f203d7fa640 0 client.16413 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c0fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3849918391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c0f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 -- 172.21.15.27:0/3849918391 >> 172.21.15.27:0/3849918391 conn(0x561cd2859300 msgr2=0x561cd2c04210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 -- 172.21.15.27:0/3849918391 shutdown_connections 2024-04-25T13:46:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 1 -- 172.21.15.27:0/3849918391 wait complete. 2024-04-25T13:46:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.878+0000 7f2062fd2740 20 client.16413 trim_cache size 0 max 16384 2024-04-25T13:46:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.905+0000 7f2062fd2740 1 Processor -- start 2024-04-25T13:46:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f2062fd2740 1 -- start start 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c04100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28444d0 0x561cd2c16a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28651b0 0x561cd2c04640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f2062fd2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f20541b3b80 con 0x561cd283c2f0 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f2062fd2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20541b6400 con 0x561cd28651b0 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f2062fd2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f20541b41f0 con 0x561cd28444d0 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f20596ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c04100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f2059ebb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28444d0 0x561cd2c16a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f20596ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c04100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56100/0 (socket says 172.21.15.27:56100) 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f20596ba640 1 -- 172.21.15.27:0/3631911915 learned_addr learned my addr 172.21.15.27:0/3631911915 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.906+0000 7f20596ba640 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28651b0 0x561cd2c04640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.907+0000 7f2059ebb640 1 -- 172.21.15.27:0/3631911915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28651b0 msgr2=0x561cd2c04640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.907+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28651b0 0x561cd2c04640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.907+0000 7f2059ebb640 1 -- 172.21.15.27:0/3631911915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 msgr2=0x561cd2c04100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.907+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c04100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.907+0000 7f2059ebb640 1 -- 172.21.15.27:0/3631911915 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cd2c10790 con 0x561cd28444d0 2024-04-25T13:46:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.907+0000 7f20596ba640 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28651b0 0x561cd2c04640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.907+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28444d0 0x561cd2c16a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2050009710 tx=0x7f205000f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67ee9f9426a48fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.908+0000 7f203f7fe640 1 -- 172.21.15.27:0/3631911915 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205001e040 con 0x561cd28444d0 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.908+0000 7f203f7fe640 1 -- 172.21.15.27:0/3631911915 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2050003da0 con 0x561cd28444d0 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.908+0000 7f203f7fe640 1 -- 172.21.15.27:0/3631911915 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205000f620 con 0x561cd28444d0 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 -- 172.21.15.27:0/3631911915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28444d0 msgr2=0x561cd2c16a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28444d0 0x561cd2c16a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2050009710 tx=0x7f205000f3c0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 -- 172.21.15.27:0/3631911915 shutdown_connections 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28651b0 0x561cd2c04640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd28444d0 0x561cd2c16a80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3631911915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd283c2f0 0x561cd2c04100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 -- 172.21.15.27:0/3631911915 >> 172.21.15.27:0/3631911915 conn(0x561cd2859300 msgr2=0x561cd2c09ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 -- 172.21.15.27:0/3631911915 shutdown_connections 2024-04-25T13:46:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 -- 172.21.15.27:0/3631911915 wait complete. 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.909+0000 7f2062fd2740 1 -- start start 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2062fd2740 1 Processor -- start 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c14770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c14cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2062fd2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c15be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2062fd2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f20541b48c0 con 0x561cd28651b0 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2062fd2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20541b5d30 con 0x561cd28444d0 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2062fd2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f20541b4f90 con 0x561cd283c2f0 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2062fd2740 10 client.0 _ll_register_callbacks cb 0x7f2050005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2059ebb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c14cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f20596ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c15be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.910+0000 7f2059ebb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c14cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38362/0 (socket says 172.21.15.27:38362) 2024-04-25T13:46:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f2059ebb640 1 -- 172.21.15.27:0/3592122871 learned_addr learned my addr 172.21.15.27:0/3592122871 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f20596ba640 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c14770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f2059ebb640 1 -- 172.21.15.27:0/3592122871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 msgr2=0x561cd2c14770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c14770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f2059ebb640 1 -- 172.21.15.27:0/3592122871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 msgr2=0x561cd2c15be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c15be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f2059ebb640 1 -- 172.21.15.27:0/3592122871 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cd2c161a0 con 0x561cd28444d0 2024-04-25T13:46:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f20596ba640 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c14770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.911+0000 7f2059ebb640 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c14cb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f205001c040 tx=0x7f20500051f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=709b9a4e2cf2404c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.912+0000 7f203d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.912+0000 7f203d7fa640 1 -- 172.21.15.27:0/3592122871 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205002f060 con 0x561cd28444d0 2024-04-25T13:46:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.912+0000 7f203d7fa640 1 -- 172.21.15.27:0/3592122871 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f20500054c0 con 0x561cd28444d0 2024-04-25T13:46:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.912+0000 7f2062fd2740 1 -- 172.21.15.27:0/3592122871 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2050003da0 con 0x561cd28444d0 2024-04-25T13:46:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.912+0000 7f203d7fa640 1 -- 172.21.15.27:0/3592122871 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f205001e040 con 0x561cd28444d0 2024-04-25T13:46:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.912+0000 7f2062fd2740 10 client.26132 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.912+0000 7f2062fd2740 1 -- 172.21.15.27:0/3592122871 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x561cd2c8e260 con 0x561cd28444d0 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.912+0000 7f203d7fa640 1 -- 172.21.15.27:0/3592122871 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2050003da0 con 0x561cd28444d0 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.913+0000 7f203d7fa640 1 -- 172.21.15.27:0/3592122871 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f205000fb70 con 0x561cd28444d0 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.913+0000 7f2062fd2740 10 client.26132 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.913+0000 7f2062fd2740 10 client.26132 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.913+0000 7f2062fd2740 20 client.26132 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.913+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561cd27f91c0 0x561cd2c90860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.913+0000 7f2062fd2740 4 client.26132 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:46:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.913+0000 7f2062fd2740 1 -- 172.21.15.27:0/3592122871 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cd2b72ee0 con 0x561cd27f91c0 2024-04-25T13:46:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.914+0000 7f20596ba640 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561cd27f91c0 0x561cd2c90860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.914+0000 7f20596ba640 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561cd27f91c0 0x561cd2c90860 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9050bdb9886f23b2 server_cookie=d0063d18d314da19 in_seq=0 out_seq=0 2024-04-25T13:46:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.914+0000 7f203d7fa640 10 client.26132 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:12.916 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 305.0 s, 3566.6 tps, lat 28.107 ms stddev 32.224 2024-04-25T13:46:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.937+0000 7f203d7fa640 1 -- 172.21.15.27:0/3592122871 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561cd2b72ee0 con 0x561cd27f91c0 2024-04-25T13:46:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.937+0000 7f203d7fa640 10 client.26132 handle_command_reply: tid=0 2024-04-25T13:46:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.937+0000 7f2062fd2740 1 client.26132 shutdown 2024-04-25T13:46:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.937+0000 7f2062fd2740 1 -- 172.21.15.27:0/3592122871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 msgr2=0x561cd2c14cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.937+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c14cb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f205001c040 tx=0x7f20500051f0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.938+0000 7f2062fd2740 1 -- 172.21.15.27:0/3592122871 shutdown_connections 2024-04-25T13:46:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.938+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561cd27f91c0 0x561cd2c90860 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.938+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561cd28651b0 0x561cd2c15be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.938+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cd28444d0 0x561cd2c14cb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.938+0000 7f2062fd2740 1 --2- 172.21.15.27:0/3592122871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561cd283c2f0 0x561cd2c14770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.938+0000 7f2062fd2740 1 -- 172.21.15.27:0/3592122871 >> 172.21.15.27:0/3592122871 conn(0x561cd2859300 msgr2=0x561cd2c04100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.938+0000 7f203d7fa640 0 client.26132 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.939+0000 7f2062fd2740 1 -- 172.21.15.27:0/3592122871 shutdown_connections 2024-04-25T13:46:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.939+0000 7f2062fd2740 1 -- 172.21.15.27:0/3592122871 wait complete. 2024-04-25T13:46:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.939+0000 7f2062fd2740 20 client.26132 trim_cache size 0 max 16384 2024-04-25T13:46:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:46:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:46:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:46:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 -- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f20400afae0 msgr2=0x7f20400b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f20400afae0 0x7f20400b1ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f204c01a040 tx=0x7f204c0057c0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 -- 172.21.15.27:0/4084187556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2054108960 msgr2=0x7f2054073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2054108960 0x7f2054073060 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f204c009fd0 tx=0x7f204c00ef10 comp rx=0 tx=0).stop 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 -- 172.21.15.27:0/4084187556 shutdown_connections 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f20400afae0 0x7f20400b1ff0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2054108960 0x7f2054073060 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2054103c60 0x7f205406ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 --2- 172.21.15.27:0/4084187556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2054102a20 0x7f205406a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.940+0000 7f205c11f640 1 -- 172.21.15.27:0/4084187556 >> 172.21.15.27:0/4084187556 conn(0x7f20540fdc10 msgr2=0x7f2054104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.941+0000 7f205c11f640 1 -- 172.21.15.27:0/4084187556 shutdown_connections 2024-04-25T13:46:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:12.941+0000 7f205c11f640 1 -- 172.21.15.27:0/4084187556 wait complete. 2024-04-25T13:46:14.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:13 smithi122 ceph-mon[32843]: pgmap v509: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 673 op/s 2024-04-25T13:46:14.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:13 smithi045 ceph-mon[32731]: pgmap v509: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 673 op/s 2024-04-25T13:46:14.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:13 smithi027 ceph-mon[29140]: pgmap v509: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 673 op/s 2024-04-25T13:46:16.202 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:15 smithi027 ceph-mon[29140]: pgmap v510: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-25T13:46:16.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:15 smithi122 ceph-mon[32843]: pgmap v510: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-25T13:46:16.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:15 smithi045 ceph-mon[32731]: pgmap v510: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-25T13:46:17.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:16 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:46:17.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:16 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:46:17.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:16 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:46:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 310.0 s, 2964.2 tps, lat 33.478 ms stddev 37.457 2024-04-25T13:46:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:17 smithi122 ceph-mon[32843]: pgmap v511: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 845 op/s 2024-04-25T13:46:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:46:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:46:18.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:17 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:46:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:17 smithi045 ceph-mon[32731]: pgmap v511: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 845 op/s 2024-04-25T13:46:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:46:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:46:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:17 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:46:18.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:17 smithi027 ceph-mon[29140]: pgmap v511: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 845 op/s 2024-04-25T13:46:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:46:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:46:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:17 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:46:20.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:19 smithi122 ceph-mon[32843]: pgmap v512: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.5 MiB/s wr, 634 op/s 2024-04-25T13:46:20.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:19 smithi045 ceph-mon[32731]: pgmap v512: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.5 MiB/s wr, 634 op/s 2024-04-25T13:46:20.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:19 smithi027 ceph-mon[29140]: pgmap v512: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.5 MiB/s wr, 634 op/s 2024-04-25T13:46:21.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:46:21.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:46:21.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.272+0000 7fbedd7f3640 1 -- 172.21.15.27:0/3430688610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbed8103c20 msgr2=0x7fbed8104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.272+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/3430688610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbed8103c20 0x7fbed8104030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbec0009860 tx=0x7fbec002a340 comp rx=0 tx=0).stop 2024-04-25T13:46:21.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.272+0000 7fbedd7f3640 1 -- 172.21.15.27:0/3430688610 shutdown_connections 2024-04-25T13:46:21.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.272+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/3430688610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed8108920 0x7fbed810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.272+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/3430688610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbed8103c20 0x7fbed8104030 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.272+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/3430688610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed81029e0 0x7fbed81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.272+0000 7fbedd7f3640 1 -- 172.21.15.27:0/3430688610 >> 172.21.15.27:0/3430688610 conn(0x7fbed80fdbd0 msgr2=0x7fbed8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:21.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.278+0000 7fbedd7f3640 1 -- 172.21.15.27:0/3430688610 shutdown_connections 2024-04-25T13:46:21.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.281+0000 7fbedd7f3640 1 -- 172.21.15.27:0/3430688610 wait complete. 2024-04-25T13:46:21.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.281+0000 7fbedd7f3640 1 Processor -- start 2024-04-25T13:46:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.285+0000 7fbedd7f3640 1 -- start start 2024-04-25T13:46:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.285+0000 7fbedd7f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbed81029e0 0x7fbed8198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.285+0000 7fbedd7f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8103c20 0x7fbed81988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.285+0000 7fbedd7f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed8108920 0x7fbed81a8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.285+0000 7fbedd7f3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbed810f650 con 0x7fbed8103c20 2024-04-25T13:46:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.285+0000 7fbedd7f3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbed810f4d0 con 0x7fbed8108920 2024-04-25T13:46:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.285+0000 7fbedd7f3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbed810f7d0 con 0x7fbed81029e0 2024-04-25T13:46:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.287+0000 7fbed77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed8108920 0x7fbed81a8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.287+0000 7fbed77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed8108920 0x7fbed81a8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37716/0 (socket says 172.21.15.27:37716) 2024-04-25T13:46:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.287+0000 7fbed77fe640 1 -- 172.21.15.27:0/2621311833 learned_addr learned my addr 172.21.15.27:0/2621311833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:21.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.287+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbed81029e0 0x7fbed8198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.289+0000 7fbecffff640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8103c20 0x7fbed81988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.289+0000 7fbed77fe640 1 -- 172.21.15.27:0/2621311833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbed81029e0 msgr2=0x7fbed8198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.289+0000 7fbed77fe640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbed81029e0 0x7fbed8198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.289+0000 7fbed77fe640 1 -- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8103c20 msgr2=0x7fbed81988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.289+0000 7fbed77fe640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8103c20 0x7fbed81988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.289+0000 7fbed77fe640 1 -- 172.21.15.27:0/2621311833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed806c6f0 con 0x7fbed8108920 2024-04-25T13:46:21.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.291+0000 7fbed77fe640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed8108920 0x7fbed81a8020 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbed000eaa0 tx=0x7fbed000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:21.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.297+0000 7fbed4ff9640 1 -- 172.21.15.27:0/2621311833 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed000cdb0 con 0x7fbed8108920 2024-04-25T13:46:21.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.297+0000 7fbed4ff9640 1 -- 172.21.15.27:0/2621311833 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed0005020 con 0x7fbed8108920 2024-04-25T13:46:21.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.297+0000 7fbed4ff9640 1 -- 172.21.15.27:0/2621311833 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed00116d0 con 0x7fbed8108920 2024-04-25T13:46:21.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.297+0000 7fbedd7f3640 1 -- 172.21.15.27:0/2621311833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed806c9e0 con 0x7fbed8108920 2024-04-25T13:46:21.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.297+0000 7fbedd7f3640 1 -- 172.21.15.27:0/2621311833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed806cee0 con 0x7fbed8108920 2024-04-25T13:46:21.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.304+0000 7fbee4778740 1 Processor -- start 2024-04-25T13:46:21.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.305+0000 7fbee4778740 1 -- start start 2024-04-25T13:46:21.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.305+0000 7fbee4778740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0aa390 0x55f38d44ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.305+0000 7fbee4778740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0814d0 0x55f38d44bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.305+0000 7fbee4778740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d44cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.305+0000 7fbee4778740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f38d3b0d80 con 0x55f38d0814d0 2024-04-25T13:46:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.305+0000 7fbee4778740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f38d36a8b0 con 0x55f38d0aa390 2024-04-25T13:46:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.305+0000 7fbee4778740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f38d3acd40 con 0x55f38d0896b0 2024-04-25T13:46:21.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.308+0000 7fbed4ff9640 1 -- 172.21.15.27:0/2621311833 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbed0011870 con 0x7fbed8108920 2024-04-25T13:46:21.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.309+0000 7fbed4ff9640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbec40afd00 0x7fbec40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.309+0000 7fbed4ff9640 1 -- 172.21.15.27:0/2621311833 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbed00deda0 con 0x7fbed8108920 2024-04-25T13:46:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.311+0000 7fbecffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0aa390 0x55f38d44ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.311+0000 7fbecffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d44cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.311+0000 7fbecffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0aa390 0x55f38d44ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37722/0 (socket says 172.21.15.27:37722) 2024-04-25T13:46:21.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.311+0000 7fbecffff640 1 -- 172.21.15.27:0/4055256184 learned_addr learned my addr 172.21.15.27:0/4055256184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:21.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.313+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/4055256184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0814d0 0x55f38d44bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.314+0000 7fbed77fe640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbec40afd00 0x7fbec40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.316+0000 7fbed77fe640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbec40afd00 0x7fbec40b2210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbed00a7ec0 tx=0x7fbed000cf50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:21.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.316+0000 7fbecffff640 1 -- 172.21.15.27:0/4055256184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0aa390 msgr2=0x55f38d44ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:21.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.316+0000 7fbecffff640 1 --2- 172.21.15.27:0/4055256184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0aa390 0x55f38d44ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.316+0000 7fbecffff640 1 -- 172.21.15.27:0/4055256184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0814d0 msgr2=0x55f38d44bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:21.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.316+0000 7fbecffff640 1 --2- 172.21.15.27:0/4055256184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0814d0 0x55f38d44bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.316+0000 7fbecffff640 1 -- 172.21.15.27:0/4055256184 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f38d33ba40 con 0x55f38d0896b0 2024-04-25T13:46:21.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.323+0000 7fbecffff640 1 --2- 172.21.15.27:0/4055256184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d44cc70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbec00304b0 tx=0x7fbec002a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bfce05c7de99531 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:21.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.325+0000 7fbecd7fa640 1 -- 172.21.15.27:0/4055256184 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0045070 con 0x55f38d0896b0 2024-04-25T13:46:21.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.325+0000 7fbecd7fa640 1 -- 172.21.15.27:0/4055256184 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec0004030 con 0x55f38d0896b0 2024-04-25T13:46:21.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.325+0000 7fbecd7fa640 1 -- 172.21.15.27:0/4055256184 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0037040 con 0x55f38d0896b0 2024-04-25T13:46:21.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.327+0000 7fbee4778740 1 -- 172.21.15.27:0/4055256184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 msgr2=0x55f38d44cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.327+0000 7fbee4778740 1 --2- 172.21.15.27:0/4055256184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d44cc70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbec00304b0 tx=0x7fbec002a340 comp rx=0 tx=0).stop 2024-04-25T13:46:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.330+0000 7fbee4778740 1 -- 172.21.15.27:0/4055256184 shutdown_connections 2024-04-25T13:46:21.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.330+0000 7fbee4778740 1 --2- 172.21.15.27:0/4055256184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d44cc70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.330+0000 7fbee4778740 1 --2- 172.21.15.27:0/4055256184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0814d0 0x55f38d44bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.330+0000 7fbee4778740 1 --2- 172.21.15.27:0/4055256184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0aa390 0x55f38d44ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.330+0000 7fbee4778740 1 -- 172.21.15.27:0/4055256184 >> 172.21.15.27:0/4055256184 conn(0x55f38d09e4e0 msgr2=0x55f38d447270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:21.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.336+0000 7fbee4778740 1 -- 172.21.15.27:0/4055256184 shutdown_connections 2024-04-25T13:46:21.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.337+0000 7fbee4778740 1 -- 172.21.15.27:0/4055256184 wait complete. 2024-04-25T13:46:21.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.337+0000 7fbee4778740 1 -- start start 2024-04-25T13:46:21.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbee4778740 1 Processor -- start 2024-04-25T13:46:21.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbee4778740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0814d0 0x55f38d4669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbee4778740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d466f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbee4778740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d46de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbee4778740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f38d44d720 con 0x55f38d0aa390 2024-04-25T13:46:21.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbee4778740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f38d337d10 con 0x55f38d0896b0 2024-04-25T13:46:21.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbee4778740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f38d44d8a0 con 0x55f38d0814d0 2024-04-25T13:46:21.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbee4778740 10 client.0 _ll_register_callbacks cb 0x55f38d364210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:21.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbecffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0814d0 0x55f38d4669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbecffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0814d0 0x55f38d4669e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59868/0 (socket says 172.21.15.27:59868) 2024-04-25T13:46:21.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbecffff640 1 -- 172.21.15.27:0/966106455 learned_addr learned my addr 172.21.15.27:0/966106455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbecffff640 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d46de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.338+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d466f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbecffff640 1 -- 172.21.15.27:0/966106455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 msgr2=0x55f38d466f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbecffff640 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d466f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbecffff640 1 -- 172.21.15.27:0/966106455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 msgr2=0x55f38d46de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbecffff640 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d46de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbecffff640 1 -- 172.21.15.27:0/966106455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f38d18c7d0 con 0x55f38d0814d0 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbecffff640 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0814d0 0x55f38d4669e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbec002a930 tx=0x7fbec002a840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6e79a2d42db56fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbebe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbebe7fc640 1 -- 172.21.15.27:0/966106455 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0037070 con 0x55f38d0814d0 2024-04-25T13:46:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbebe7fc640 1 -- 172.21.15.27:0/966106455 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbec0030550 con 0x55f38d0814d0 2024-04-25T13:46:21.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.339+0000 7fbee4778740 1 -- 172.21.15.27:0/966106455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec0004030 con 0x55f38d0814d0 2024-04-25T13:46:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.341+0000 7fbebe7fc640 1 -- 172.21.15.27:0/966106455 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fbec0045040 con 0x55f38d0814d0 2024-04-25T13:46:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.341+0000 7fbee4778740 10 client.26185 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.341+0000 7fbee4778740 1 -- 172.21.15.27:0/966106455 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f38d0c23f0 con 0x55f38d0814d0 2024-04-25T13:46:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbebe7fc640 1 -- 172.21.15.27:0/966106455 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec0004030 con 0x55f38d0814d0 2024-04-25T13:46:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbebe7fc640 1 -- 172.21.15.27:0/966106455 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fbec004b070 con 0x55f38d0814d0 2024-04-25T13:46:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbee4778740 10 client.26185 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbee4778740 10 client.26185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbee4778740 20 client.26185 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbee4778740 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f38d068f60 0x55f38d4740e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbee4778740 4 client.26185 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:21.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbee4778740 1 -- 172.21.15.27:0/966106455 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f38d3be990 con 0x55f38d068f60 2024-04-25T13:46:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.342+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f38d068f60 0x55f38d4740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.343+0000 7fbebe7fc640 10 client.26185 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.343+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f38d068f60 0x55f38d4740e0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12310f5ddd0bdf7c server_cookie=53fb26174b186609 in_seq=0 out_seq=0 2024-04-25T13:46:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbebe7fc640 1 -- 172.21.15.27:0/966106455 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f38d3be990 con 0x55f38d068f60 2024-04-25T13:46:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbebe7fc640 10 client.26185 handle_command_reply: tid=0 2024-04-25T13:46:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 client.26185 shutdown 2024-04-25T13:46:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 -- 172.21.15.27:0/966106455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0814d0 msgr2=0x55f38d4669e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0814d0 0x55f38d4669e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbec002a930 tx=0x7fbec002a840 comp rx=0 tx=0).stop 2024-04-25T13:46:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 -- 172.21.15.27:0/966106455 shutdown_connections 2024-04-25T13:46:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f38d068f60 0x55f38d4740e0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d46de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d466f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 --2- 172.21.15.27:0/966106455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0814d0 0x55f38d4669e0 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbec002a930 tx=0x7fbec002a840 comp rx=0 tx=0).stop 2024-04-25T13:46:21.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbee4778740 1 -- 172.21.15.27:0/966106455 >> 172.21.15.27:0/966106455 conn(0x55f38d09e4e0 msgr2=0x55f38d4463f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:21.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.344+0000 7fbebe7fc640 0 client.26185 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:21.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.346+0000 7fbee4778740 1 -- 172.21.15.27:0/966106455 shutdown_connections 2024-04-25T13:46:21.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.346+0000 7fbee4778740 1 -- 172.21.15.27:0/966106455 wait complete. 2024-04-25T13:46:21.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.347+0000 7fbee4778740 20 client.26185 trim_cache size 0 max 16384 2024-04-25T13:46:21.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.374+0000 7fbee4778740 1 Processor -- start 2024-04-25T13:46:21.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.374+0000 7fbee4778740 1 -- start start 2024-04-25T13:46:21.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbee4778740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d4665b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbee4778740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d466af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbee4778740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d03e3a0 0x55f38d46dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbee4778740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb800c380 con 0x55f38d0aa390 2024-04-25T13:46:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbee4778740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb800ec00 con 0x55f38d0896b0 2024-04-25T13:46:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbee4778740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb800c9f0 con 0x55f38d03e3a0 2024-04-25T13:46:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbecffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d4665b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbed6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d466af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbed6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d466af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58416/0 (socket says 172.21.15.27:58416) 2024-04-25T13:46:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbecffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d4665b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37734/0 (socket says 172.21.15.27:37734) 2024-04-25T13:46:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbed6ffd640 1 -- 172.21.15.27:0/3541035510 learned_addr learned my addr 172.21.15.27:0/3541035510 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbecffff640 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d03e3a0 0x55f38d46dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbed6ffd640 1 -- 172.21.15.27:0/3541035510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d03e3a0 msgr2=0x55f38d46dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d03e3a0 0x55f38d46dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbed6ffd640 1 -- 172.21.15.27:0/3541035510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 msgr2=0x55f38d4665b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d4665b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbed6ffd640 1 -- 172.21.15.27:0/3541035510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f38d0820f0 con 0x55f38d0aa390 2024-04-25T13:46:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.375+0000 7fbecffff640 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d4665b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.376+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d466af0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbec8000c00 tx=0x7fbec8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39d2f8fdac9350e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.376+0000 7fbeccff9640 1 -- 172.21.15.27:0/3541035510 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8004030 con 0x55f38d0aa390 2024-04-25T13:46:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.376+0000 7fbeccff9640 1 -- 172.21.15.27:0/3541035510 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec8025020 con 0x55f38d0aa390 2024-04-25T13:46:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.376+0000 7fbeccff9640 1 -- 172.21.15.27:0/3541035510 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec80210e0 con 0x55f38d0aa390 2024-04-25T13:46:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.377+0000 7fbee4778740 1 -- 172.21.15.27:0/3541035510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 msgr2=0x55f38d466af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.377+0000 7fbee4778740 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d466af0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbec8000c00 tx=0x7fbec8009260 comp rx=0 tx=0).stop 2024-04-25T13:46:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.377+0000 7fbee4778740 1 -- 172.21.15.27:0/3541035510 shutdown_connections 2024-04-25T13:46:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.377+0000 7fbee4778740 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d03e3a0 0x55f38d46dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.377+0000 7fbee4778740 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d466af0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.377+0000 7fbee4778740 1 --2- 172.21.15.27:0/3541035510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d0896b0 0x55f38d4665b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.377+0000 7fbee4778740 1 -- 172.21.15.27:0/3541035510 >> 172.21.15.27:0/3541035510 conn(0x55f38d09e4e0 msgr2=0x55f38d44b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.378+0000 7fbee4778740 1 -- 172.21.15.27:0/3541035510 shutdown_connections 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.378+0000 7fbee4778740 1 -- 172.21.15.27:0/3541035510 wait complete. 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.378+0000 7fbee4778740 1 -- start start 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.378+0000 7fbee4778740 1 Processor -- start 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbee4778740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d03e3a0 0x55f38d46dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbee4778740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d46e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbee4778740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d46c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbed6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d46e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbed6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d46e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59882/0 (socket says 172.21.15.27:59882) 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbecffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d03e3a0 0x55f38d46dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbecffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d03e3a0 0x55f38d46dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37744/0 (socket says 172.21.15.27:37744) 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbed6ffd640 1 -- 172.21.15.27:0/222316043 learned_addr learned my addr 172.21.15.27:0/222316043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb800d0c0 con 0x55f38d0aa390 2024-04-25T13:46:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb800e530 con 0x55f38d03e3a0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb800d790 con 0x55f38d0896b0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.379+0000 7fbecffff640 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d46c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.380+0000 7fbecffff640 1 -- 172.21.15.27:0/222316043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 msgr2=0x55f38d46e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.380+0000 7fbecffff640 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d46e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.380+0000 7fbecffff640 1 -- 172.21.15.27:0/222316043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 msgr2=0x55f38d46c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.380+0000 7fbecffff640 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d46c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.380+0000 7fbecffff640 1 -- 172.21.15.27:0/222316043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f38d3714b0 con 0x55f38d03e3a0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.380+0000 7fbecffff640 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d03e3a0 0x55f38d46dfa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbec002c040 tx=0x7fbec0035040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e075f610bf074aa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.380+0000 7fbebe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.380+0000 7fbebe7fc640 1 -- 172.21.15.27:0/222316043 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0037070 con 0x55f38d03e3a0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.381+0000 7fbebe7fc640 1 -- 172.21.15.27:0/222316043 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbec0004450 con 0x55f38d03e3a0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.381+0000 7fbee4778740 10 client.0 _ll_register_callbacks cb 0x7fbec0043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:21.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.381+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec8025020 con 0x55f38d03e3a0 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.381+0000 7fbebe7fc640 1 -- 172.21.15.27:0/222316043 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fbec0004ef0 con 0x55f38d03e3a0 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.381+0000 7fbee4778740 10 client.26150 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.381+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=19}) -- 0x55f38d4da6a0 con 0x55f38d03e3a0 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.382+0000 7fbebe7fc640 1 -- 172.21.15.27:0/222316043 <== mon.2 v2:172.21.15.45:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fbec004b040 con 0x55f38d03e3a0 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.382+0000 7fbee4778740 10 client.26150 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.382+0000 7fbee4778740 10 client.26150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.382+0000 7fbee4778740 20 client.26150 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.382+0000 7fbee4778740 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f38d4dae90 0x55f38d4dd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.382+0000 7fbee4778740 4 client.26150 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.382+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fbec8025020 con 0x55f38d4dae90 2024-04-25T13:46:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.382+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f38d4dae90 0x55f38d4dd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.383+0000 7fbed6ffd640 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f38d4dae90 0x55f38d4dd3a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a21d34624e41fdce server_cookie=f92bcca1f063de8d in_seq=0 out_seq=0 2024-04-25T13:46:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.383+0000 7fbebe7fc640 10 client.26150 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.383+0000 7fbebe7fc640 1 -- 172.21.15.27:0/222316043 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fbec8025020 con 0x55f38d4dae90 2024-04-25T13:46:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.383+0000 7fbebe7fc640 10 client.26150 handle_command_reply: tid=0 2024-04-25T13:46:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.383+0000 7fbee4778740 1 client.26150 shutdown 2024-04-25T13:46:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.383+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d03e3a0 msgr2=0x55f38d46dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.383+0000 7fbee4778740 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d03e3a0 0x55f38d46dfa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbec002c040 tx=0x7fbec0035040 comp rx=0 tx=0).stop 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 shutdown_connections 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f38d4dae90 0x55f38d4dd3a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f38d0aa390 0x55f38d46c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbebe7fc640 0 client.26150 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f38d0896b0 0x55f38d46e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 1 --2- 172.21.15.27:0/222316043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f38d03e3a0 0x55f38d46dfa0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 >> 172.21.15.27:0/222316043 conn(0x55f38d09e4e0 msgr2=0x55f38d4665b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 shutdown_connections 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 1 -- 172.21.15.27:0/222316043 wait complete. 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.384+0000 7fbee4778740 20 client.26150 trim_cache size 0 max 16384 2024-04-25T13:46:21.388 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:46:21.389 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:46:21.389 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:46:21.389 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.386+0000 7fbedd7f3640 1 -- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbec40afd00 msgr2=0x7fbec40b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.386+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbec40afd00 0x7fbec40b2210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbed00a7ec0 tx=0x7fbed000cf50 comp rx=0 tx=0).stop 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.386+0000 7fbedd7f3640 1 -- 172.21.15.27:0/2621311833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed8108920 msgr2=0x7fbed81a8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.386+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed8108920 0x7fbed81a8020 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbed000eaa0 tx=0x7fbed000ef60 comp rx=0 tx=0).stop 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.387+0000 7fbedd7f3640 1 -- 172.21.15.27:0/2621311833 shutdown_connections 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.387+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbec40afd00 0x7fbec40b2210 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.387+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbed8108920 0x7fbed81a8020 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.387+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed8103c20 0x7fbed81988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.387+0000 7fbedd7f3640 1 --2- 172.21.15.27:0/2621311833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbed81029e0 0x7fbed8198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.387+0000 7fbedd7f3640 1 -- 172.21.15.27:0/2621311833 >> 172.21.15.27:0/2621311833 conn(0x7fbed80fdbd0 msgr2=0x7fbed80fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.388+0000 7fbedd7f3640 1 -- 172.21.15.27:0/2621311833 shutdown_connections 2024-04-25T13:46:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.388+0000 7fbedd7f3640 1 -- 172.21.15.27:0/2621311833 wait complete. 2024-04-25T13:46:21.405 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:46:21.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:46:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 -- 172.21.15.27:0/3054986038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 msgr2=0x7f677c104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 --2- 172.21.15.27:0/3054986038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 0x7f677c104030 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6770009900 tx=0x7f677002a410 comp rx=0 tx=0).stop 2024-04-25T13:46:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 -- 172.21.15.27:0/3054986038 shutdown_connections 2024-04-25T13:46:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 --2- 172.21.15.27:0/3054986038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677c108920 0x7f677c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 --2- 172.21.15.27:0/3054986038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 0x7f677c104030 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 --2- 172.21.15.27:0/3054986038 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677c1029e0 0x7f677c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 -- 172.21.15.27:0/3054986038 >> 172.21.15.27:0/3054986038 conn(0x7f677c0fdbd0 msgr2=0x7f677c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 -- 172.21.15.27:0/3054986038 shutdown_connections 2024-04-25T13:46:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.570+0000 7f678191c640 1 -- 172.21.15.27:0/3054986038 wait complete. 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f678191c640 1 Processor -- start 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f678191c640 1 -- start start 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f678191c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677c1029e0 0x7f677c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f678191c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 0x7f677c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f678191c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677c108920 0x7f677c1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f678191c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f677c10e4e0 con 0x7f677c103c20 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f678191c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f677c10e360 con 0x7f677c108920 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f678191c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f677c10e660 con 0x7f677c1029e0 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f677a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 0x7f677c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f677a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 0x7f677c19a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58440/0 (socket says 172.21.15.27:58440) 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f677affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677c1029e0 0x7f677c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.572+0000 7f677a7fc640 1 -- 172.21.15.27:0/2598966726 learned_addr learned my addr 172.21.15.27:0/2598966726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f677b7fe640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677c108920 0x7f677c1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f677affd640 1 -- 172.21.15.27:0/2598966726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677c108920 msgr2=0x7f677c1a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f677affd640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677c108920 0x7f677c1a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f677affd640 1 -- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 msgr2=0x7f677c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f677affd640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 0x7f677c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f677affd640 1 -- 172.21.15.27:0/2598966726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f677c1a1da0 con 0x7f677c1029e0 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f677a7fc640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 0x7f677c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f677affd640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677c1029e0 0x7f677c19a260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f677c1022b0 tx=0x7f676c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f678091a640 1 -- 172.21.15.27:0/2598966726 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c00cc40 con 0x7f677c1029e0 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f678091a640 1 -- 172.21.15.27:0/2598966726 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f676c00cde0 con 0x7f677c1029e0 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f678091a640 1 -- 172.21.15.27:0/2598966726 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c010430 con 0x7f677c1029e0 2024-04-25T13:46:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f677c1a2090 con 0x7f677c1029e0 2024-04-25T13:46:21.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.573+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f677c1a2630 con 0x7f677c1029e0 2024-04-25T13:46:21.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.574+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f677c06a190 con 0x7f677c1029e0 2024-04-25T13:46:21.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.577+0000 7f678091a640 1 -- 172.21.15.27:0/2598966726 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f676c003eb0 con 0x7f677c1029e0 2024-04-25T13:46:21.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.577+0000 7f678091a640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67640afb70 0x7f67640b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:21.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.577+0000 7f678091a640 1 -- 172.21.15.27:0/2598966726 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f676c0df0b0 con 0x7f677c1029e0 2024-04-25T13:46:21.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.578+0000 7f677a7fc640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67640afb70 0x7f67640b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:21.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.578+0000 7f677a7fc640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67640afb70 0x7f67640b2080 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6770031440 tx=0x7f67700314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:21.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.581+0000 7f678091a640 1 -- 172.21.15.27:0/2598966726 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f676c0a7d50 con 0x7f677c1029e0 2024-04-25T13:46:21.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.953+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f677c1a28f0 con 0x7f677c1029e0 2024-04-25T13:46:21.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.955+0000 7f678091a640 1 -- 172.21.15.27:0/2598966726 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f676c0a7650 con 0x7f677c1029e0 2024-04-25T13:46:21.955 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:46:21.955 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:46:21.955 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67640afb70 msgr2=0x7f67640b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67640afb70 0x7f67640b2080 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6770031440 tx=0x7f67700314a0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677c1029e0 msgr2=0x7f677c19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677c1029e0 0x7f677c19a260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f677c1022b0 tx=0x7f676c00ee50 comp rx=0 tx=0).stop 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 shutdown_connections 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67640afb70 0x7f67640b2080 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f677c108920 0x7f677c1a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f677c103c20 0x7f677c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 --2- 172.21.15.27:0/2598966726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677c1029e0 0x7f677c19a260 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:21.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 >> 172.21.15.27:0/2598966726 conn(0x7f677c0fdbd0 msgr2=0x7f677c074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:21.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 shutdown_connections 2024-04-25T13:46:21.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:21.958+0000 7f678191c640 1 -- 172.21.15.27:0/2598966726 wait complete. 2024-04-25T13:46:21.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:46:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.159+0000 7ff6f2b26640 1 -- 172.21.15.27:0/3662994537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10b310 msgr2=0x7ff6ec10b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.159+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/3662994537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10b310 0x7ff6ec10b720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009ca0 tx=0x7ff6e402a8f0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.162+0000 7ff6f2b26640 1 -- 172.21.15.27:0/3662994537 shutdown_connections 2024-04-25T13:46:22.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.162+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/3662994537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6ec110030 0x7ff6ec110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.162+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/3662994537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10b310 0x7ff6ec10b720 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.162+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/3662994537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6ec10a420 0x7ff6ec10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.162+0000 7ff6f2b26640 1 -- 172.21.15.27:0/3662994537 >> 172.21.15.27:0/3662994537 conn(0x7ff6ec0704e0 msgr2=0x7ff6ec072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:22.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.168+0000 7ff6f2b26640 1 -- 172.21.15.27:0/3662994537 shutdown_connections 2024-04-25T13:46:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.170+0000 7ff6f2b26640 1 -- 172.21.15.27:0/3662994537 wait complete. 2024-04-25T13:46:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.170+0000 7ff6f2b26640 1 Processor -- start 2024-04-25T13:46:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.175+0000 7ff6f2b26640 1 -- start start 2024-04-25T13:46:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.175+0000 7ff6f2b26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10a420 0x7ff6ec11b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.175+0000 7ff6f2b26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6ec10b310 0x7ff6ec11a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.175+0000 7ff6f2b26640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6ec110030 0x7ff6ec11a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.175+0000 7ff6f2b26640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec1123c0 con 0x7ff6ec10b310 2024-04-25T13:46:22.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.175+0000 7ff6f2b26640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec112240 con 0x7ff6ec10a420 2024-04-25T13:46:22.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.175+0000 7ff6f2b26640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec112540 con 0x7ff6ec110030 2024-04-25T13:46:22.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.176+0000 7ff6f10c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6ec110030 0x7ff6ec11a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.176+0000 7ff6f10c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6ec110030 0x7ff6ec11a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59934/0 (socket says 172.21.15.27:59934) 2024-04-25T13:46:22.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.176+0000 7ff6f10c3640 1 -- 172.21.15.27:0/2804850463 learned_addr learned my addr 172.21.15.27:0/2804850463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:22.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.176+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10a420 0x7ff6ec11b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.177+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6ec10b310 0x7ff6ec11a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.177+0000 7ff6f08c2640 1 -- 172.21.15.27:0/2804850463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6ec110030 msgr2=0x7ff6ec11a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.177+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6ec110030 0x7ff6ec11a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.177+0000 7ff6f08c2640 1 -- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6ec10b310 msgr2=0x7ff6ec11a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.177+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6ec10b310 0x7ff6ec11a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.177+0000 7ff6f08c2640 1 -- 172.21.15.27:0/2804850463 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ec11aae0 con 0x7ff6ec10a420 2024-04-25T13:46:22.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.178+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10a420 0x7ff6ec11b9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0009fd0 tx=0x7ff6e000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.179+0000 7ff6e9ffb640 1 -- 172.21.15.27:0/2804850463 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0015020 con 0x7ff6ec10a420 2024-04-25T13:46:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.179+0000 7ff6f2b26640 1 -- 172.21.15.27:0/2804850463 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ec11add0 con 0x7ff6ec10a420 2024-04-25T13:46:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.179+0000 7ff6f2b26640 1 -- 172.21.15.27:0/2804850463 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ec1b9e20 con 0x7ff6ec10a420 2024-04-25T13:46:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.180+0000 7ff6e9ffb640 1 -- 172.21.15.27:0/2804850463 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6e0011ce0 con 0x7ff6ec10a420 2024-04-25T13:46:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.180+0000 7ff6e9ffb640 1 -- 172.21.15.27:0/2804850463 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0010e00 con 0x7ff6ec10a420 2024-04-25T13:46:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.184+0000 7ff6f99d8740 1 Processor -- start 2024-04-25T13:46:22.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.185+0000 7ff6e9ffb640 1 -- 172.21.15.27:0/2804850463 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff6e0018020 con 0x7ff6ec10a420 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6e9ffb640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6d40afc30 0x7ff6d40b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6f99d8740 1 -- start start 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ab7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846ec4d0 0x560f84ab7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846f46b0 0x560f84ab8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6e9ffb640 1 -- 172.21.15.27:0/2804850463 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff6e00de310 con 0x7ff6ec10a420 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6f99d8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560f84a1bd80 con 0x560f846f46b0 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6f99d8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560f849d5a20 con 0x560f84715390 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.186+0000 7ff6f99d8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x560f84a17d40 con 0x560f846ec4d0 2024-04-25T13:46:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ab7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f10c3640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6d40afc30 0x7ff6d40b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ab7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37802/0 (socket says 172.21.15.27:37802) 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 -- 172.21.15.27:0/781990054 learned_addr learned my addr 172.21.15.27:0/781990054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f10c3640 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846f46b0 0x560f84ab8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846ec4d0 0x560f84ab7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 -- 172.21.15.27:0/781990054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846ec4d0 msgr2=0x560f84ab7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846ec4d0 0x560f84ab7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 -- 172.21.15.27:0/781990054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846f46b0 msgr2=0x560f84ab8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846f46b0 0x560f84ab8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 -- 172.21.15.27:0/781990054 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f849e0c30 con 0x560f84715390 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f10c3640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6d40afc30 0x7ff6d40b2140 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc006ed0 tx=0x7ff6dc0099b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.187+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ab7810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0006d00 tx=0x7ff6e00a5f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b96195158581e40f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.188+0000 7ff6d2ffd640 1 -- 172.21.15.27:0/781990054 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00c90f0 con 0x560f84715390 2024-04-25T13:46:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.188+0000 7ff6d2ffd640 1 -- 172.21.15.27:0/781990054 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6e00a67c0 con 0x560f84715390 2024-04-25T13:46:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.188+0000 7ff6d2ffd640 1 -- 172.21.15.27:0/781990054 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e008f0d0 con 0x560f84715390 2024-04-25T13:46:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.189+0000 7ff6f99d8740 1 -- 172.21.15.27:0/781990054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 msgr2=0x560f84ab7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.189+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ab7810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0006d00 tx=0x7ff6e00a5f30 comp rx=0 tx=0).stop 2024-04-25T13:46:22.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.190+0000 7ff6f99d8740 1 -- 172.21.15.27:0/781990054 shutdown_connections 2024-04-25T13:46:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.190+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846f46b0 0x560f84ab8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.190+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846ec4d0 0x560f84ab7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.190+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/781990054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ab7810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.190+0000 7ff6f99d8740 1 -- 172.21.15.27:0/781990054 >> 172.21.15.27:0/781990054 conn(0x560f847094e0 msgr2=0x560f84ab3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.190+0000 7ff6f99d8740 1 -- 172.21.15.27:0/781990054 shutdown_connections 2024-04-25T13:46:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.191+0000 7ff6f99d8740 1 -- 172.21.15.27:0/781990054 wait complete. 2024-04-25T13:46:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.191+0000 7ff6f99d8740 1 -- start start 2024-04-25T13:46:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.191+0000 7ff6f99d8740 1 Processor -- start 2024-04-25T13:46:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.192+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84ac13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.192+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.192+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f84715390 0x560f84ac8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.192+0000 7ff6ebfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6ebfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37810/0 (socket says 172.21.15.27:37810) 2024-04-25T13:46:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6ebfff640 1 -- 172.21.15.27:0/2040917165 learned_addr learned my addr 172.21.15.27:0/2040917165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560f84ab9530 con 0x560f846ec4d0 2024-04-25T13:46:22.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560f849a2dc0 con 0x560f846f46b0 2024-04-25T13:46:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x560f84ab96b0 con 0x560f84715390 2024-04-25T13:46:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6f99d8740 10 client.0 _ll_register_callbacks cb 0x560f849be600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84ac13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6f10c3640 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f84715390 0x560f84ac8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6ebfff640 1 -- 172.21.15.27:0/2040917165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f84715390 msgr2=0x560f84ac8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f84715390 0x560f84ac8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6ebfff640 1 -- 172.21.15.27:0/2040917165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 msgr2=0x560f84ac13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84ac13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6ebfff640 1 -- 172.21.15.27:0/2040917165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f8472d3f0 con 0x560f846f46b0 2024-04-25T13:46:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.193+0000 7ff6f10c3640 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f84715390 0x560f84ac8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:46:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac1920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009bf0 tx=0x7ff6e4031dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=313acc215174dbfb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6cbfff640 1 -- 172.21.15.27:0/2040917165 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4030800 con 0x560f846f46b0 2024-04-25T13:46:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6e00a67c0 con 0x560f846f46b0 2024-04-25T13:46:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6cbfff640 1 -- 172.21.15.27:0/2040917165 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff6e40309a0 con 0x560f846f46b0 2024-04-25T13:46:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6cbfff640 1 -- 172.21.15.27:0/2040917165 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff6e4030ba0 con 0x560f846f46b0 2024-04-25T13:46:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6f99d8740 10 client.26168 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560f847f77d0 con 0x560f846f46b0 2024-04-25T13:46:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.194+0000 7ff6cbfff640 1 -- 172.21.15.27:0/2040917165 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6e00a67c0 con 0x560f846f46b0 2024-04-25T13:46:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6cbfff640 1 -- 172.21.15.27:0/2040917165 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff6e4042070 con 0x560f846f46b0 2024-04-25T13:46:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6f99d8740 10 client.26168 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6f99d8740 10 client.26168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6f99d8740 20 client.26168 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560f846d3f60 0x560f84acf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6f99d8740 4 client.26168 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:22.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f84a29990 con 0x560f846d3f60 2024-04-25T13:46:22.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560f846d3f60 0x560f84acf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.195+0000 7ff6cbfff640 10 client.26168 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:22.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.196+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560f846d3f60 0x560f84acf140 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bb89c05db358001 server_cookie=fba3ea8a9e6ac170 in_seq=0 out_seq=0 2024-04-25T13:46:22.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6cbfff640 1 -- 172.21.15.27:0/2040917165 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560f84a29990 con 0x560f846d3f60 2024-04-25T13:46:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6cbfff640 10 client.26168 handle_command_reply: tid=0 2024-04-25T13:46:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 client.26168 shutdown 2024-04-25T13:46:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 msgr2=0x560f84ac1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac1920 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009bf0 tx=0x7ff6e4031dd0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 shutdown_connections 2024-04-25T13:46:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560f846d3f60 0x560f84acf140 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f84715390 0x560f84ac8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac1920 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/2040917165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84ac13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 >> 172.21.15.27:0/2040917165 conn(0x560f847094e0 msgr2=0x560f84ab2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.197+0000 7ff6cbfff640 0 client.26168 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.198+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 shutdown_connections 2024-04-25T13:46:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.198+0000 7ff6f99d8740 1 -- 172.21.15.27:0/2040917165 wait complete. 2024-04-25T13:46:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.198+0000 7ff6f99d8740 20 client.26168 trim_cache size 0 max 16384 2024-04-25T13:46:22.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.220+0000 7ff6f99d8740 1 Processor -- start 2024-04-25T13:46:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.221+0000 7ff6f99d8740 1 -- start start 2024-04-25T13:46:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.221+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84abfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.221+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846f46b0 0x560f84ac02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.221+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ac87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.221+0000 7ff6f99d8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d800c390 con 0x560f846ec4d0 2024-04-25T13:46:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.221+0000 7ff6f08c2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84abfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f08c2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84abfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58472/0 (socket says 172.21.15.27:58472) 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f08c2640 1 -- 172.21.15.27:0/3696979543 learned_addr learned my addr 172.21.15.27:0/3696979543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846f46b0 0x560f84ac02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f99d8740 1 -- 172.21.15.27:0/3696979543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d800ec10 con 0x560f84715390 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f99d8740 1 -- 172.21.15.27:0/3696979543 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d800ca00 con 0x560f846f46b0 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f10c3640 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ac87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f08c2640 1 -- 172.21.15.27:0/3696979543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846f46b0 msgr2=0x560f84ac02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846f46b0 0x560f84ac02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f08c2640 1 -- 172.21.15.27:0/3696979543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 msgr2=0x560f84ac87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ac87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f08c2640 1 -- 172.21.15.27:0/3696979543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f84ac8ce0 con 0x560f846ec4d0 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846f46b0 0x560f84ac02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f10c3640 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ac87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84abfd70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000bd70 tx=0x7ff6e00af040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75dd26723c9b75be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:22.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6d27fc640 1 -- 172.21.15.27:0/3696979543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00af620 con 0x560f846ec4d0 2024-04-25T13:46:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6d27fc640 1 -- 172.21.15.27:0/3696979543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6e00a67c0 con 0x560f846ec4d0 2024-04-25T13:46:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.222+0000 7ff6d27fc640 1 -- 172.21.15.27:0/3696979543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e008f310 con 0x560f846ec4d0 2024-04-25T13:46:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.223+0000 7ff6f99d8740 1 -- 172.21.15.27:0/3696979543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 msgr2=0x560f84abfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.223+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84abfd70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000bd70 tx=0x7ff6e00af040 comp rx=0 tx=0).stop 2024-04-25T13:46:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.223+0000 7ff6f99d8740 1 -- 172.21.15.27:0/3696979543 shutdown_connections 2024-04-25T13:46:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.223+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f84715390 0x560f84ac87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.223+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846f46b0 0x560f84ac02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.223+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/3696979543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f846ec4d0 0x560f84abfd70 secure :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000bd70 tx=0x7ff6e00af040 comp rx=0 tx=0).stop 2024-04-25T13:46:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.223+0000 7ff6f99d8740 1 -- 172.21.15.27:0/3696979543 >> 172.21.15.27:0/3696979543 conn(0x560f847094e0 msgr2=0x560f84abc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.225+0000 7ff6f99d8740 1 -- 172.21.15.27:0/3696979543 shutdown_connections 2024-04-25T13:46:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.225+0000 7ff6f99d8740 1 -- 172.21.15.27:0/3696979543 wait complete. 2024-04-25T13:46:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.225+0000 7ff6f99d8740 1 -- start start 2024-04-25T13:46:22.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.226+0000 7ff6f99d8740 1 Processor -- start 2024-04-25T13:46:22.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.226+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.226+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f84715390 0x560f84ac7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f99d8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846a93a0 0x560f84b354e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f08c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f08c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37828/0 (socket says 172.21.15.27:37828) 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f99d8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d800d0d0 con 0x560f84715390 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d800e540 con 0x560f846f46b0 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d800d7a0 con 0x560f846a93a0 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f08c2640 1 -- 172.21.15.27:0/865182224 learned_addr learned my addr 172.21.15.27:0/865182224 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f84715390 0x560f84ac7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f99d8740 10 client.0 _ll_register_callbacks cb 0x7ff6e4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6f10c3640 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846a93a0 0x560f84b354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6ebfff640 1 -- 172.21.15.27:0/865182224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846a93a0 msgr2=0x560f84b354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846a93a0 0x560f84b354e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6ebfff640 1 -- 172.21.15.27:0/865182224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 msgr2=0x560f84ac73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6ebfff640 1 -- 172.21.15.27:0/865182224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f84b35a20 con 0x560f84715390 2024-04-25T13:46:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6ebfff640 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f84715390 0x560f84ac7950 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4043de0 tx=0x7ff6e402ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc4923b45aa2db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.227+0000 7ff6cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:46:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.228+0000 7ff6cbfff640 1 -- 172.21.15.27:0/865182224 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e403a440 con 0x560f84715390 2024-04-25T13:46:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.228+0000 7ff6cbfff640 1 -- 172.21.15.27:0/865182224 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff6e403a5e0 con 0x560f84715390 2024-04-25T13:46:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.228+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6e00a67c0 con 0x560f84715390 2024-04-25T13:46:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.228+0000 7ff6cbfff640 1 -- 172.21.15.27:0/865182224 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff6e405e070 con 0x560f84715390 2024-04-25T13:46:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.228+0000 7ff6f99d8740 10 client.16458 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.228+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560f84b36b90 con 0x560f84715390 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.230+0000 7ff6cbfff640 1 -- 172.21.15.27:0/865182224 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6e00a67c0 con 0x560f84715390 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.230+0000 7ff6cbfff640 1 -- 172.21.15.27:0/865182224 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff6e4056030 con 0x560f84715390 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.230+0000 7ff6f99d8740 10 client.16458 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.230+0000 7ff6f99d8740 10 client.16458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.230+0000 7ff6f99d8740 20 client.16458 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.230+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560f84b37540 0x560f84b39960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.230+0000 7ff6f99d8740 4 client.16458 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.230+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f84a23800 con 0x560f84b37540 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.231+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560f84b37540 0x560f84b39960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.231+0000 7ff6f08c2640 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560f84b37540 0x560f84b39960 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4125d9ef33a32dc server_cookie=afa7291d506211e in_seq=0 out_seq=0 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.231+0000 7ff6cbfff640 10 client.16458 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.231+0000 7ff6cbfff640 1 -- 172.21.15.27:0/865182224 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f84a23800 con 0x560f84b37540 2024-04-25T13:46:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.231+0000 7ff6cbfff640 10 client.16458 handle_command_reply: tid=0 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.231+0000 7ff6f99d8740 1 client.16458 shutdown 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f84715390 msgr2=0x560f84ac7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f84715390 0x560f84ac7950 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4043de0 tx=0x7ff6e402ae70 comp rx=0 tx=0).stop 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 shutdown_connections 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560f84b37540 0x560f84b39960 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560f846a93a0 0x560f84b354e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6cbfff640 0 client.16458 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560f84715390 0x560f84ac7950 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6f99d8740 1 --2- 172.21.15.27:0/865182224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560f846f46b0 0x560f84ac73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.232+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 >> 172.21.15.27:0/865182224 conn(0x560f847094e0 msgr2=0x560f84abfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.233+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 shutdown_connections 2024-04-25T13:46:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.233+0000 7ff6f99d8740 1 -- 172.21.15.27:0/865182224 wait complete. 2024-04-25T13:46:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.233+0000 7ff6f99d8740 20 client.16458 trim_cache size 0 max 16384 2024-04-25T13:46:22.235 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:46:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.234+0000 7ff6f2b26640 1 -- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6d40afc30 msgr2=0x7ff6d40b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.234+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6d40afc30 0x7ff6d40b2140 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc006ed0 tx=0x7ff6dc0099b0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.234+0000 7ff6f2b26640 1 -- 172.21.15.27:0/2804850463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10a420 msgr2=0x7ff6ec11b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.234+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10a420 0x7ff6ec11b9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0009fd0 tx=0x7ff6e000b870 comp rx=0 tx=0).stop 2024-04-25T13:46:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.235+0000 7ff6f2b26640 1 -- 172.21.15.27:0/2804850463 shutdown_connections 2024-04-25T13:46:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.235+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6d40afc30 0x7ff6d40b2140 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.235+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff6ec110030 0x7ff6ec11a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.235+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6ec10b310 0x7ff6ec11a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.235+0000 7ff6f2b26640 1 --2- 172.21.15.27:0/2804850463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ec10a420 0x7ff6ec11b9c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.235+0000 7ff6f2b26640 1 -- 172.21.15.27:0/2804850463 >> 172.21.15.27:0/2804850463 conn(0x7ff6ec0704e0 msgr2=0x7ff6ec10e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.235+0000 7ff6f2b26640 1 -- 172.21.15.27:0/2804850463 shutdown_connections 2024-04-25T13:46:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:22.235+0000 7ff6f2b26640 1 -- 172.21.15.27:0/2804850463 wait complete. 2024-04-25T13:46:22.251 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:21 smithi027 ceph-mon[29140]: pgmap v513: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.0 MiB/s wr, 761 op/s 2024-04-25T13:46:22.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:21 smithi122 ceph-mon[32843]: pgmap v513: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.0 MiB/s wr, 761 op/s 2024-04-25T13:46:22.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:21 smithi045 ceph-mon[32731]: pgmap v513: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.0 MiB/s wr, 761 op/s 2024-04-25T13:46:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 315.0 s, 3283.2 tps, lat 30.661 ms stddev 32.750 2024-04-25T13:46:23.253 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:46:23.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:46:23.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:46:23.281 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:22 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2598966726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:46:23.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:22 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2598966726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:46:23.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:22 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2598966726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:46:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.428+0000 7efcbee70640 1 -- 172.21.15.27:0/253966807 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb8110030 msgr2=0x7efcb8110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.428+0000 7efcbee70640 1 --2- 172.21.15.27:0/253966807 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb8110030 0x7efcb8110440 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009ca0 tx=0x7efcb002a8e0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.428+0000 7efcbee70640 1 -- 172.21.15.27:0/253966807 shutdown_connections 2024-04-25T13:46:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.428+0000 7efcbee70640 1 --2- 172.21.15.27:0/253966807 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb8110030 0x7efcb8110440 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.428+0000 7efcbee70640 1 --2- 172.21.15.27:0/253966807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb810b310 0x7efcb810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.428+0000 7efcbee70640 1 --2- 172.21.15.27:0/253966807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb810a420 0x7efcb810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.428+0000 7efcbee70640 1 -- 172.21.15.27:0/253966807 >> 172.21.15.27:0/253966807 conn(0x7efcb80704e0 msgr2=0x7efcb8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.430+0000 7efcbee70640 1 -- 172.21.15.27:0/253966807 shutdown_connections 2024-04-25T13:46:23.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.430+0000 7efcbee70640 1 -- 172.21.15.27:0/253966807 wait complete. 2024-04-25T13:46:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.431+0000 7efcbee70640 1 Processor -- start 2024-04-25T13:46:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.431+0000 7efcbee70640 1 -- start start 2024-04-25T13:46:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcbee70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb810a420 0x7efcb81a9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcbee70640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb810b310 0x7efcb81a9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcbee70640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb8110030 0x7efcb81b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb810b310 0x7efcb81a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcbd40d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb8110030 0x7efcb81b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcbcc0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb810a420 0x7efcb81a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcbcc0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb810a420 0x7efcb81a9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37858/0 (socket says 172.21.15.27:37858) 2024-04-25T13:46:23.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcbcc0c640 1 -- 172.21.15.27:0/2349873437 learned_addr learned my addr 172.21.15.27:0/2349873437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.432+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efcb811e570 con 0x7efcb810b310 2024-04-25T13:46:23.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.433+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efcb811e3f0 con 0x7efcb810a420 2024-04-25T13:46:23.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.433+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efcb811e6f0 con 0x7efcb8110030 2024-04-25T13:46:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.435+0000 7efcb7fff640 1 -- 172.21.15.27:0/2349873437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb8110030 msgr2=0x7efcb81b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.435+0000 7efcb7fff640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb8110030 0x7efcb81b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.435+0000 7efcb7fff640 1 -- 172.21.15.27:0/2349873437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb810a420 msgr2=0x7efcb81a9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.435+0000 7efcb7fff640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb810a420 0x7efcb81a9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.435+0000 7efcb7fff640 1 -- 172.21.15.27:0/2349873437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb81b0af0 con 0x7efcb810b310 2024-04-25T13:46:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.436+0000 7efcb7fff640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb810b310 0x7efcb81a9540 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7efcac00b3f0 tx=0x7efcac00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.436+0000 7efcb5ffb640 1 -- 172.21.15.27:0/2349873437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac015020 con 0x7efcb810b310 2024-04-25T13:46:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.436+0000 7efcb5ffb640 1 -- 172.21.15.27:0/2349873437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcac011ce0 con 0x7efcb810b310 2024-04-25T13:46:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.436+0000 7efcb5ffb640 1 -- 172.21.15.27:0/2349873437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac010da0 con 0x7efcb810b310 2024-04-25T13:46:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.436+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb81b0de0 con 0x7efcb810b310 2024-04-25T13:46:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.437+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb81b12d0 con 0x7efcb810b310 2024-04-25T13:46:23.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.441+0000 7efcc5df5740 1 Processor -- start 2024-04-25T13:46:23.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.441+0000 7efcc5df5740 1 -- start start 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcb5ffb640 1 -- 172.21.15.27:0/2349873437 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7efcac018020 con 0x7efcb810b310 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a101730 0x55ee9a4ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcc5df5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ee9a428b00 con 0x55ee9a0f9550 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcc5df5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ee9a3c4920 con 0x55ee9a101730 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcc5df5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ee9a424ac0 con 0x55ee9a122410 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcbd40d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcbd40d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59992/0 (socket says 172.21.15.27:59992) 2024-04-25T13:46:23.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcbd40d640 1 -- 172.21.15.27:0/839761152 learned_addr learned my addr 172.21.15.27:0/839761152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.442+0000 7efcb7fff640 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a101730 0x55ee9a4ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.443+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.443+0000 7efcb5ffb640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efca00afc30 0x7efca00b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.443+0000 7efcbd40d640 1 -- 172.21.15.27:0/839761152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a101730 msgr2=0x55ee9a4ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.443+0000 7efcbd40d640 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a101730 0x55ee9a4ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.443+0000 7efcbd40d640 1 -- 172.21.15.27:0/839761152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 msgr2=0x55ee9a4c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.443+0000 7efcbd40d640 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.443+0000 7efcbd40d640 1 -- 172.21.15.27:0/839761152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee9a2027e0 con 0x55ee9a122410 2024-04-25T13:46:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.443+0000 7efcbd40d640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efca00afc30 0x7efca00b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.444+0000 7efcb5ffb640 1 -- 172.21.15.27:0/2349873437 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efcac0de2c0 con 0x7efcb810b310 2024-04-25T13:46:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.444+0000 7efcbd40d640 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4c34a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efcb00316b0 tx=0x7efcb00316e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b0ae7da45cded2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.444+0000 7efc9effd640 1 -- 172.21.15.27:0/839761152 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0004510 con 0x55ee9a122410 2024-04-25T13:46:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.444+0000 7efc9effd640 1 -- 172.21.15.27:0/839761152 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb0036040 con 0x55ee9a122410 2024-04-25T13:46:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.444+0000 7efc9effd640 1 -- 172.21.15.27:0/839761152 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0046440 con 0x55ee9a122410 2024-04-25T13:46:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 -- 172.21.15.27:0/839761152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 msgr2=0x55ee9a4c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcbd40d640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efca00afc30 0x7efca00b2140 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efcb004b000 tx=0x7efcb0048000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4c34a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efcb00316b0 tx=0x7efcb00316e0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 -- 172.21.15.27:0/839761152 shutdown_connections 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a101730 0x55ee9a4ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 --2- 172.21.15.27:0/839761152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4c34a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 -- 172.21.15.27:0/839761152 >> 172.21.15.27:0/839761152 conn(0x55ee9a116560 msgr2=0x55ee9a4bee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 -- 172.21.15.27:0/839761152 shutdown_connections 2024-04-25T13:46:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.445+0000 7efcc5df5740 1 -- 172.21.15.27:0/839761152 wait complete. 2024-04-25T13:46:23.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.446+0000 7efcc5df5740 1 -- start start 2024-04-25T13:46:23.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.446+0000 7efcc5df5740 1 Processor -- start 2024-04-25T13:46:23.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a101730 0x55ee9a4cd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a122410 0x55ee9a4d4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcc5df5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ee9a4cb370 con 0x55ee9a0f9550 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcc5df5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ee9a3c25d0 con 0x55ee9a122410 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcc5df5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ee9a4cb4f0 con 0x55ee9a101730 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcc5df5740 10 client.0 _ll_register_callbacks cb 0x55ee9a3b0080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a101730 0x55ee9a4cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a101730 0x55ee9a4cd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:60002/0 (socket says 172.21.15.27:60002) 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcb7fff640 1 -- 172.21.15.27:0/671349834 learned_addr learned my addr 172.21.15.27:0/671349834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcbd40d640 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a122410 0x55ee9a4d4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.447+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efcb7fff640 1 -- 172.21.15.27:0/671349834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a122410 msgr2=0x55ee9a4d4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efcb7fff640 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a122410 0x55ee9a4d4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efcb7fff640 1 -- 172.21.15.27:0/671349834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 msgr2=0x55ee9a4cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efcb7fff640 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efcb7fff640 1 -- 172.21.15.27:0/671349834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee9a262f70 con 0x55ee9a101730 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efcb7fff640 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a101730 0x55ee9a4cd800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efcac004810 tx=0x7efcac0e17b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fbddd67553065d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efc97fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efc97fff640 1 -- 172.21.15.27:0/671349834 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac0a60c0 con 0x55ee9a101730 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efc97fff640 1 -- 172.21.15.27:0/671349834 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efcac011830 con 0x55ee9a101730 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efcc5df5740 1 -- 172.21.15.27:0/671349834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcb0036040 con 0x55ee9a101730 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.448+0000 7efc97fff640 1 -- 172.21.15.27:0/671349834 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7efcac0c90d0 con 0x55ee9a101730 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcc5df5740 10 client.26224 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcc5df5740 1 -- 172.21.15.27:0/671349834 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ee9a0fa170 con 0x55ee9a101730 2024-04-25T13:46:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efc97fff640 1 -- 172.21.15.27:0/671349834 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb0036040 con 0x55ee9a101730 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efc97fff640 1 -- 172.21.15.27:0/671349834 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7efcac0a20e0 con 0x55ee9a101730 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcc5df5740 10 client.26224 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcc5df5740 10 client.26224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcc5df5740 20 client.26224 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcc5df5740 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ee9a0e11f0 0x55ee9a4dacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcc5df5740 4 client.26224 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcc5df5740 1 -- 172.21.15.27:0/671349834 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee9a436710 con 0x55ee9a0e11f0 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.449+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ee9a0e11f0 0x55ee9a4dacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.450+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ee9a0e11f0 0x55ee9a4dacf0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3215709bf2d317c4 server_cookie=324aa2fdbafbce4f in_seq=0 out_seq=0 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.450+0000 7efc97fff640 10 client.26224 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efc97fff640 1 -- 172.21.15.27:0/671349834 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ee9a436710 con 0x55ee9a0e11f0 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efc97fff640 10 client.26224 handle_command_reply: tid=0 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 client.26224 shutdown 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 -- 172.21.15.27:0/671349834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a101730 msgr2=0x55ee9a4cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a101730 0x55ee9a4cd800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efcac004810 tx=0x7efcac0e17b0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 -- 172.21.15.27:0/671349834 shutdown_connections 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ee9a0e11f0 0x55ee9a4dacf0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a122410 0x55ee9a4d4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a101730 0x55ee9a4cd800 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 --2- 172.21.15.27:0/671349834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efcc5df5740 1 -- 172.21.15.27:0/671349834 >> 172.21.15.27:0/671349834 conn(0x55ee9a116560 msgr2=0x55ee9a4bdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.451+0000 7efc97fff640 0 client.26224 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:23.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.452+0000 7efcc5df5740 1 -- 172.21.15.27:0/671349834 shutdown_connections 2024-04-25T13:46:23.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.452+0000 7efcc5df5740 1 -- 172.21.15.27:0/671349834 wait complete. 2024-04-25T13:46:23.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.452+0000 7efcc5df5740 20 client.26224 trim_cache size 0 max 16384 2024-04-25T13:46:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.478+0000 7efcc5df5740 1 Processor -- start 2024-04-25T13:46:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.478+0000 7efcc5df5740 1 -- start start 2024-04-25T13:46:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.478+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.478+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a122410 0x55ee9a4cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.478+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a0b6630 0x55ee9a4bd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcc5df5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81b7e20 con 0x55ee9a122410 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcc5df5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81ba6a0 con 0x55ee9a0f9550 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcc5df5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81b8490 con 0x55ee9a0b6630 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a122410 0x55ee9a4cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcbd40d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a0b6630 0x55ee9a4bd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a122410 0x55ee9a4cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58506/0 (socket says 172.21.15.27:58506) 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcb7fff640 1 -- 172.21.15.27:0/244687360 learned_addr learned my addr 172.21.15.27:0/244687360 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcb7fff640 1 -- 172.21.15.27:0/244687360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a0b6630 msgr2=0x55ee9a4bd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcb7fff640 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a0b6630 0x55ee9a4bd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcb7fff640 1 -- 172.21.15.27:0/244687360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0f9550 msgr2=0x55ee9a4cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcb7fff640 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.479+0000 7efcb7fff640 1 -- 172.21.15.27:0/244687360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee9a4c8130 con 0x55ee9a122410 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.480+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.480+0000 7efcbd40d640 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a0b6630 0x55ee9a4bd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.480+0000 7efcb7fff640 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a122410 0x55ee9a4cbe00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7efcac0a24b0 tx=0x7efcac010840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bc89e92513d649 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.481+0000 7efc9e7fc640 1 -- 172.21.15.27:0/244687360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac0a5ee0 con 0x55ee9a122410 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.481+0000 7efc9e7fc640 1 -- 172.21.15.27:0/244687360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcac0a6280 con 0x55ee9a122410 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.481+0000 7efc9e7fc640 1 -- 172.21.15.27:0/244687360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac0a6770 con 0x55ee9a122410 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 -- 172.21.15.27:0/244687360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a122410 msgr2=0x55ee9a4cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a122410 0x55ee9a4cbe00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7efcac0a24b0 tx=0x7efcac010840 comp rx=0 tx=0).stop 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 -- 172.21.15.27:0/244687360 shutdown_connections 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a0b6630 0x55ee9a4bd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a122410 0x55ee9a4cbe00 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 --2- 172.21.15.27:0/244687360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0f9550 0x55ee9a4cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 -- 172.21.15.27:0/244687360 >> 172.21.15.27:0/244687360 conn(0x55ee9a116560 msgr2=0x55ee9a4be7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 -- 172.21.15.27:0/244687360 shutdown_connections 2024-04-25T13:46:23.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 -- 172.21.15.27:0/244687360 wait complete. 2024-04-25T13:46:23.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 -- start start 2024-04-25T13:46:23.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.482+0000 7efcc5df5740 1 Processor -- start 2024-04-25T13:46:23.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0b6630 0x55ee9a4d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcc5df5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4d2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcc5df5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81b8b60 con 0x55ee9a0f9550 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcc5df5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81b9fd0 con 0x55ee9a0b6630 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcc5df5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81b9230 con 0x55ee9a122410 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcc5df5740 10 client.0 _ll_register_callbacks cb 0x7efcac0e1970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcb7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4d1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58508/0 (socket says 172.21.15.27:58508) 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcb7fff640 1 -- 172.21.15.27:0/2053467150 learned_addr learned my addr 172.21.15.27:0/2053467150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcbd40d640 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4d2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.483+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0b6630 0x55ee9a4d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efcbcc0c640 1 -- 172.21.15.27:0/2053467150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 msgr2=0x55ee9a4d2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4d2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efcbcc0c640 1 -- 172.21.15.27:0/2053467150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 msgr2=0x55ee9a4d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efcbcc0c640 1 -- 172.21.15.27:0/2053467150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee9a4cc340 con 0x55ee9a0b6630 2024-04-25T13:46:23.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efcbcc0c640 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0b6630 0x55ee9a4d1910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55ee9a4db3f0 tx=0x7efca80157f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a4bcbf22346cab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efc97fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:23.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efc97fff640 1 -- 172.21.15.27:0/2053467150 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca80059c0 con 0x55ee9a0b6630 2024-04-25T13:46:23.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.484+0000 7efc97fff640 1 -- 172.21.15.27:0/2053467150 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efca8005b60 con 0x55ee9a0b6630 2024-04-25T13:46:23.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 1 -- 172.21.15.27:0/2053467150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcac0a6280 con 0x55ee9a0b6630 2024-04-25T13:46:23.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efc97fff640 1 -- 172.21.15.27:0/2053467150 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7efca8029050 con 0x55ee9a0b6630 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 10 client.26189 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 1 -- 172.21.15.27:0/2053467150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ee9a5431f0 con 0x55ee9a0b6630 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efc97fff640 1 -- 172.21.15.27:0/2053467150 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcac0a6280 con 0x55ee9a0b6630 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efc97fff640 1 -- 172.21.15.27:0/2053467150 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7efca801cce0 con 0x55ee9a0b6630 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 10 client.26189 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 10 client.26189 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 20 client.26189 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ee9a543b50 0x55ee9a545f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 4 client.26189 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.485+0000 7efcc5df5740 1 -- 172.21.15.27:0/2053467150 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ee9a430580 con 0x55ee9a543b50 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.486+0000 7efcb7fff640 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ee9a543b50 0x55ee9a545f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.486+0000 7efc97fff640 10 client.26189 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.486+0000 7efcb7fff640 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ee9a543b50 0x55ee9a545f70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82779a136a6c93dc server_cookie=ae053dbc7c1e0dae in_seq=0 out_seq=0 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efc97fff640 1 -- 172.21.15.27:0/2053467150 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ee9a430580 con 0x55ee9a543b50 2024-04-25T13:46:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efc97fff640 10 client.26189 handle_command_reply: tid=0 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 client.26189 shutdown 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 -- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0b6630 msgr2=0x55ee9a4d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0b6630 0x55ee9a4d1910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55ee9a4db3f0 tx=0x7efca80157f0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 -- 172.21.15.27:0/2053467150 shutdown_connections 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ee9a543b50 0x55ee9a545f70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ee9a122410 0x55ee9a4d2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ee9a0f9550 0x55ee9a4d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 --2- 172.21.15.27:0/2053467150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee9a0b6630 0x55ee9a4d1910 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.487+0000 7efcc5df5740 1 -- 172.21.15.27:0/2053467150 >> 172.21.15.27:0/2053467150 conn(0x55ee9a116560 msgr2=0x55ee9a4c1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.488+0000 7efc97fff640 0 client.26189 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.488+0000 7efcc5df5740 1 -- 172.21.15.27:0/2053467150 shutdown_connections 2024-04-25T13:46:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.488+0000 7efcc5df5740 1 -- 172.21.15.27:0/2053467150 wait complete. 2024-04-25T13:46:23.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.489+0000 7efcc5df5740 20 client.26189 trim_cache size 0 max 16384 2024-04-25T13:46:23.493 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:46:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:46:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "08a2d3a8-050a-48f8-b402-01beda6bc529", 2024-04-25T13:46:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:46:23.493 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.490+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efca00afc30 msgr2=0x7efca00b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.490+0000 7efcbee70640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efca00afc30 0x7efca00b2140 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efcb004b000 tx=0x7efcb0048000 comp rx=0 tx=0).stop 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.490+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb810b310 msgr2=0x7efcb81a9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.490+0000 7efcbee70640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb810b310 0x7efcb81a9540 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7efcac00b3f0 tx=0x7efcac00b8b0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.490+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 shutdown_connections 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.490+0000 7efcbee70640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efca00afc30 0x7efca00b2140 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.491+0000 7efcbee70640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcb8110030 0x7efcb81b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.491+0000 7efcbee70640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcb810b310 0x7efcb81a9540 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.491+0000 7efcbee70640 1 --2- 172.21.15.27:0/2349873437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb810a420 0x7efcb81a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.491+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 >> 172.21.15.27:0/2349873437 conn(0x7efcb80704e0 msgr2=0x7efcb8072970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.491+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 shutdown_connections 2024-04-25T13:46:23.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.491+0000 7efcbee70640 1 -- 172.21.15.27:0/2349873437 wait complete. 2024-04-25T13:46:23.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:46:23.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:46:23.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.733+0000 7f1aa4784640 1 -- 172.21.15.27:0/2968983467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c1029e0 msgr2=0x7f1a9c1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.733+0000 7f1aa4784640 1 --2- 172.21.15.27:0/2968983467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c1029e0 0x7f1a9c1036e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009900 tx=0x7f1a9802a410 comp rx=0 tx=0).stop 2024-04-25T13:46:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.733+0000 7f1aa4784640 1 -- 172.21.15.27:0/2968983467 shutdown_connections 2024-04-25T13:46:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.733+0000 7f1aa4784640 1 --2- 172.21.15.27:0/2968983467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a9c108920 0x7f1a9c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.733+0000 7f1aa4784640 1 --2- 172.21.15.27:0/2968983467 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a9c103c20 0x7f1a9c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.734+0000 7f1aa4784640 1 --2- 172.21.15.27:0/2968983467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c1029e0 0x7f1a9c1036e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.734+0000 7f1aa4784640 1 -- 172.21.15.27:0/2968983467 >> 172.21.15.27:0/2968983467 conn(0x7f1a9c0fdbd0 msgr2=0x7f1a9c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.734+0000 7f1aa4784640 1 -- 172.21.15.27:0/2968983467 shutdown_connections 2024-04-25T13:46:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.734+0000 7f1aa4784640 1 -- 172.21.15.27:0/2968983467 wait complete. 2024-04-25T13:46:23.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.734+0000 7f1aa4784640 1 Processor -- start 2024-04-25T13:46:23.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.734+0000 7f1aa4784640 1 -- start start 2024-04-25T13:46:23.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa4784640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a9c1029e0 0x7f1a9c19c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa4784640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c103c20 0x7f1a9c19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa4784640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a9c108920 0x7f1a9c1a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa4784640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c10cf40 con 0x7f1a9c103c20 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa4784640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c10cdc0 con 0x7f1a9c1029e0 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa4784640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c10d0c0 con 0x7f1a9c108920 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa1d1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c103c20 0x7f1a9c19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa1d1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c103c20 0x7f1a9c19cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33868/0 (socket says 172.21.15.27:33868) 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa1d1f640 1 -- 172.21.15.27:0/380647828 learned_addr learned my addr 172.21.15.27:0/380647828 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa2520640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a9c1029e0 0x7f1a9c19c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa2d21640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a9c108920 0x7f1a9c1a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa1d1f640 1 -- 172.21.15.27:0/380647828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a9c108920 msgr2=0x7f1a9c1a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a9c108920 0x7f1a9c1a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa1d1f640 1 -- 172.21.15.27:0/380647828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a9c1029e0 msgr2=0x7f1a9c19c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a9c1029e0 0x7f1a9c19c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa1d1f640 1 -- 172.21.15.27:0/380647828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9c1a4270 con 0x7f1a9c103c20 2024-04-25T13:46:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.735+0000 7f1aa2520640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a9c1029e0 0x7f1a9c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.736+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c103c20 0x7f1a9c19cc70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00e9f0 tx=0x7f1a8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.736+0000 7f1a937fe640 1 -- 172.21.15.27:0/380647828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c00cd80 con 0x7f1a9c103c20 2024-04-25T13:46:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.736+0000 7f1a937fe640 1 -- 172.21.15.27:0/380647828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a8c0054d0 con 0x7f1a9c103c20 2024-04-25T13:46:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.736+0000 7f1a937fe640 1 -- 172.21.15.27:0/380647828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c010690 con 0x7f1a9c103c20 2024-04-25T13:46:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.736+0000 7f1aa4784640 1 -- 172.21.15.27:0/380647828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a9c1a4560 con 0x7f1a9c103c20 2024-04-25T13:46:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.736+0000 7f1aa4784640 1 -- 172.21.15.27:0/380647828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a9c1a4ab0 con 0x7f1a9c103c20 2024-04-25T13:46:23.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.740+0000 7f1a937fe640 1 -- 172.21.15.27:0/380647828 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1a8c010830 con 0x7f1a9c103c20 2024-04-25T13:46:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.741+0000 7f1aab632740 1 Processor -- start 2024-04-25T13:46:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aab632740 1 -- start start 2024-04-25T13:46:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aab632740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2be2390 0x5615c2f8c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aab632740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 0x5615c2f8c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aab632740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f8d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aab632740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5615c2ee8d80 con 0x5615c2be2390 2024-04-25T13:46:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aab632740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5615c2ea28b0 con 0x5615c2bb94d0 2024-04-25T13:46:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aab632740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5615c2ee4d40 con 0x5615c2bc16b0 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aa2d21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2be2390 0x5615c2f8c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aa2d21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2be2390 0x5615c2f8c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33880/0 (socket says 172.21.15.27:33880) 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.742+0000 7f1aa2d21640 1 -- 172.21.15.27:0/1641962836 learned_addr learned my addr 172.21.15.27:0/1641962836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1a937fe640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a880afc30 0x7f1a880b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f8d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa2d21640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a880afc30 0x7f1a880b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa2520640 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 0x5615c2f8c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa2d21640 1 -- 172.21.15.27:0/1641962836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 msgr2=0x5615c2f8d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa2d21640 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f8d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa2d21640 1 -- 172.21.15.27:0/1641962836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 msgr2=0x5615c2f8c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa2d21640 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 0x5615c2f8c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa2d21640 1 -- 172.21.15.27:0/1641962836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615c2e73a40 con 0x5615c2be2390 2024-04-25T13:46:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f8d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:46:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.743+0000 7f1a937fe640 1 -- 172.21.15.27:0/380647828 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a8c0defc0 con 0x7f1a9c103c20 2024-04-25T13:46:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.744+0000 7f1aa2d21640 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2be2390 0x5615c2f8c270 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1a94011b70 tx=0x7f1a9400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2998250ae50336 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.744+0000 7f1aa2d21640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a880afc30 0x7f1a880b2140 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1a9400fc90 tx=0x7f1a9401f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.744+0000 7f1a90ff9640 1 -- 172.21.15.27:0/1641962836 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9401a040 con 0x5615c2be2390 2024-04-25T13:46:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.744+0000 7f1a90ff9640 1 -- 172.21.15.27:0/1641962836 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a94022030 con 0x5615c2be2390 2024-04-25T13:46:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.744+0000 7f1a90ff9640 1 -- 172.21.15.27:0/1641962836 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9400e5d0 con 0x5615c2be2390 2024-04-25T13:46:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 -- 172.21.15.27:0/1641962836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2be2390 msgr2=0x5615c2f8c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2be2390 0x5615c2f8c270 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1a94011b70 tx=0x7f1a9400f790 comp rx=0 tx=0).stop 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 -- 172.21.15.27:0/1641962836 shutdown_connections 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f8d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 0x5615c2f8c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 --2- 172.21.15.27:0/1641962836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2be2390 0x5615c2f8c270 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 -- 172.21.15.27:0/1641962836 >> 172.21.15.27:0/1641962836 conn(0x5615c2bd64e0 msgr2=0x5615c2f7f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 -- 172.21.15.27:0/1641962836 shutdown_connections 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.745+0000 7f1aab632740 1 -- 172.21.15.27:0/1641962836 wait complete. 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 1 -- start start 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 1 Processor -- start 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f96650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2be2390 0x5615c2f9d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5615c2f8df30 con 0x5615c2bb94d0 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5615c2e6fd10 con 0x5615c2be2390 2024-04-25T13:46:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5615c2f8e0b0 con 0x5615c2bc16b0 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa2520640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aab632740 10 client.0 _ll_register_callbacks cb 0x5615c2e9c210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa1d1f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f96650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa2520640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33896/0 (socket says 172.21.15.27:33896) 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa2520640 1 -- 172.21.15.27:0/166778133 learned_addr learned my addr 172.21.15.27:0/166778133 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa2d21640 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2be2390 0x5615c2f9d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa1d1f640 1 -- 172.21.15.27:0/166778133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2be2390 msgr2=0x5615c2f9d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2be2390 0x5615c2f9d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa1d1f640 1 -- 172.21.15.27:0/166778133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 msgr2=0x5615c2f96110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.746+0000 7f1aa1d1f640 1 -- 172.21.15.27:0/166778133 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615c2cc47d0 con 0x5615c2bc16b0 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f96650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c0a7400 tx=0x7f1a8c0a6eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f49069e497f2568c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1a85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1a85ffb640 1 -- 172.21.15.27:0/166778133 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0a4d00 con 0x5615c2bc16b0 2024-04-25T13:46:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1a85ffb640 1 -- 172.21.15.27:0/166778133 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a8c0a3a60 con 0x5615c2bc16b0 2024-04-25T13:46:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 1 -- 172.21.15.27:0/166778133 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a94022030 con 0x5615c2bc16b0 2024-04-25T13:46:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1a85ffb640 1 -- 172.21.15.27:0/166778133 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1a8c0a7240 con 0x5615c2bc16b0 2024-04-25T13:46:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 10 client.26242 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 1 -- 172.21.15.27:0/166778133 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5615c2bfa3f0 con 0x5615c2bc16b0 2024-04-25T13:46:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1a85ffb640 1 -- 172.21.15.27:0/166778133 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a94022030 con 0x5615c2bc16b0 2024-04-25T13:46:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1a85ffb640 1 -- 172.21.15.27:0/166778133 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1a8c0a5d10 con 0x5615c2bc16b0 2024-04-25T13:46:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 10 client.26242 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 10 client.26242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 20 client.26242 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5615c2ba0f60 0x5615c2fa3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 4 client.26242 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.747+0000 7f1aab632740 1 -- 172.21.15.27:0/166778133 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615c2ef6990 con 0x5615c2ba0f60 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.748+0000 7f1aa2520640 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5615c2ba0f60 0x5615c2fa3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.748+0000 7f1a85ffb640 10 client.26242 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.749+0000 7f1aa2520640 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5615c2ba0f60 0x5615c2fa3a80 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfe7a032bff1dde7 server_cookie=2007d2f943390b21 in_seq=0 out_seq=0 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.749+0000 7f1a85ffb640 1 -- 172.21.15.27:0/166778133 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5615c2ef6990 con 0x5615c2ba0f60 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.749+0000 7f1a85ffb640 10 client.26242 handle_command_reply: tid=0 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 client.26242 shutdown 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 -- 172.21.15.27:0/166778133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 msgr2=0x5615c2f96650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f96650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c0a7400 tx=0x7f1a8c0a6eb0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 -- 172.21.15.27:0/166778133 shutdown_connections 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5615c2ba0f60 0x5615c2fa3a80 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1a85ffb640 0 client.26242 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2be2390 0x5615c2f9d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2bc16b0 0x5615c2f96650 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 --2- 172.21.15.27:0/166778133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 -- 172.21.15.27:0/166778133 >> 172.21.15.27:0/166778133 conn(0x5615c2bd64e0 msgr2=0x5615c2f82180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 -- 172.21.15.27:0/166778133 shutdown_connections 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 1 -- 172.21.15.27:0/166778133 wait complete. 2024-04-25T13:46:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.750+0000 7f1aab632740 20 client.26242 trim_cache size 0 max 16384 2024-04-25T13:46:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.776+0000 7f1aab632740 1 Processor -- start 2024-04-25T13:46:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.776+0000 7f1aab632740 1 -- start start 2024-04-25T13:46:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aab632740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f7f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aab632740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aab632740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bc16b0 0x5615c2f82070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aab632740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c1a6f90 con 0x5615c2bc16b0 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aab632740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c1a9810 con 0x5615c2bb94d0 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aab632740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c1a7600 con 0x5615c2ba0f60 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2d21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bc16b0 0x5615c2f82070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2520640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f7f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2d21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bc16b0 0x5615c2f82070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33908/0 (socket says 172.21.15.27:33908) 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2d21640 1 -- 172.21.15.27:0/3406921117 learned_addr learned my addr 172.21.15.27:0/3406921117 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/3406921117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2520640 1 -- 172.21.15.27:0/3406921117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 msgr2=0x5615c2f96110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2520640 1 --2- 172.21.15.27:0/3406921117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2520640 1 -- 172.21.15.27:0/3406921117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bc16b0 msgr2=0x5615c2f82070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2520640 1 --2- 172.21.15.27:0/3406921117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bc16b0 0x5615c2f82070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2520640 1 -- 172.21.15.27:0/3406921117 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615c2f915d0 con 0x5615c2ba0f60 2024-04-25T13:46:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.777+0000 7f1aa2520640 1 --2- 172.21.15.27:0/3406921117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f7f3b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1a98031500 tx=0x7f1a98031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dbb845accef95ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.778+0000 7f1a87fff640 1 -- 172.21.15.27:0/3406921117 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98035040 con 0x5615c2ba0f60 2024-04-25T13:46:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.778+0000 7f1a87fff640 1 -- 172.21.15.27:0/3406921117 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a9804a070 con 0x5615c2ba0f60 2024-04-25T13:46:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.778+0000 7f1a87fff640 1 -- 172.21.15.27:0/3406921117 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a980455e0 con 0x5615c2ba0f60 2024-04-25T13:46:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.779+0000 7f1aab632740 1 -- 172.21.15.27:0/3406921117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 msgr2=0x5615c2f7f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.779+0000 7f1aab632740 1 --2- 172.21.15.27:0/3406921117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f7f3b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1a98031500 tx=0x7f1a98031400 comp rx=0 tx=0).stop 2024-04-25T13:46:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.779+0000 7f1aab632740 1 -- 172.21.15.27:0/3406921117 shutdown_connections 2024-04-25T13:46:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.779+0000 7f1aab632740 1 --2- 172.21.15.27:0/3406921117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bc16b0 0x5615c2f82070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.779+0000 7f1aab632740 1 --2- 172.21.15.27:0/3406921117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bb94d0 0x5615c2f96110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.779+0000 7f1aab632740 1 --2- 172.21.15.27:0/3406921117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f7f3b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.779+0000 7f1aab632740 1 -- 172.21.15.27:0/3406921117 >> 172.21.15.27:0/3406921117 conn(0x5615c2bd64e0 msgr2=0x5615c2f7eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.780+0000 7f1aab632740 1 -- 172.21.15.27:0/3406921117 shutdown_connections 2024-04-25T13:46:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.780+0000 7f1aab632740 1 -- 172.21.15.27:0/3406921117 wait complete. 2024-04-25T13:46:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 1 -- start start 2024-04-25T13:46:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 1 Processor -- start 2024-04-25T13:46:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f9a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f9a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bc16b0 0x5615c2f9b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c1a7cd0 con 0x5615c2bb94d0 2024-04-25T13:46:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c1a9140 con 0x5615c2bc16b0 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c1a83a0 con 0x5615c2ba0f60 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aab632740 10 client.0 _ll_register_callbacks cb 0x7f1a8c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2520640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f9a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa1d1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f9a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2520640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f9a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:48488/0 (socket says 172.21.15.27:48488) 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa1d1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f9a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33910/0 (socket says 172.21.15.27:33910) 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2520640 1 -- 172.21.15.27:0/3985517245 learned_addr learned my addr 172.21.15.27:0/3985517245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2d21640 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bc16b0 0x5615c2f9b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2520640 1 -- 172.21.15.27:0/3985517245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bc16b0 msgr2=0x5615c2f9b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2520640 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bc16b0 0x5615c2f9b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2520640 1 -- 172.21.15.27:0/3985517245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 msgr2=0x5615c2f9a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2520640 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f9a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.781+0000 7f1aa2520640 1 -- 172.21.15.27:0/3985517245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615c2bba0f0 con 0x5615c2ba0f60 2024-04-25T13:46:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.782+0000 7f1aa2520640 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f9a0e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009230 tx=0x7f1a980035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20e2132131240f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.782+0000 7f1a85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.782+0000 7f1a85ffb640 1 -- 172.21.15.27:0/3985517245 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98035040 con 0x5615c2ba0f60 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.782+0000 7f1a85ffb640 1 -- 172.21.15.27:0/3985517245 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a98045380 con 0x5615c2ba0f60 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.782+0000 7f1aab632740 1 -- 172.21.15.27:0/3985517245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a9804a070 con 0x5615c2ba0f60 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.782+0000 7f1a85ffb640 1 -- 172.21.15.27:0/3985517245 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1a98045580 con 0x5615c2ba0f60 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.782+0000 7f1aab632740 10 client.26248 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.782+0000 7f1aab632740 1 -- 172.21.15.27:0/3985517245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5615c2ea94b0 con 0x5615c2ba0f60 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.783+0000 7f1a85ffb640 1 -- 172.21.15.27:0/3985517245 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a9804a070 con 0x5615c2ba0f60 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.783+0000 7f1a85ffb640 1 -- 172.21.15.27:0/3985517245 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1a9803c440 con 0x5615c2ba0f60 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.783+0000 7f1aab632740 10 client.26248 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.783+0000 7f1aab632740 10 client.26248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.783+0000 7f1aab632740 20 client.26248 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.783+0000 7f1aab632740 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5615c2be2390 0x5615c300e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.783+0000 7f1aab632740 4 client.26248 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:46:23.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.783+0000 7f1aab632740 1 -- 172.21.15.27:0/3985517245 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5615c2ef0800 con 0x5615c2be2390 2024-04-25T13:46:23.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.784+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5615c2be2390 0x5615c300e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:23.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.784+0000 7f1a85ffb640 10 client.26248 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:23.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.784+0000 7f1aa1d1f640 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5615c2be2390 0x5615c300e0a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5597271816b3f354 server_cookie=d14ba28fb40fd327 in_seq=0 out_seq=0 2024-04-25T13:46:23.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.785+0000 7f1a85ffb640 1 -- 172.21.15.27:0/3985517245 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5615c2ef0800 con 0x5615c2be2390 2024-04-25T13:46:23.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.785+0000 7f1a85ffb640 10 client.26248 handle_command_reply: tid=0 2024-04-25T13:46:23.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.785+0000 7f1aab632740 1 client.26248 shutdown 2024-04-25T13:46:23.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 -- 172.21.15.27:0/3985517245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 msgr2=0x5615c2f9a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f9a0e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009230 tx=0x7f1a980035f0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 -- 172.21.15.27:0/3985517245 shutdown_connections 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5615c2be2390 0x5615c300e0a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615c2bc16b0 0x5615c2f9b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5615c2bb94d0 0x5615c2f9a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 --2- 172.21.15.27:0/3985517245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5615c2ba0f60 0x5615c2f9a0e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 -- 172.21.15.27:0/3985517245 >> 172.21.15.27:0/3985517245 conn(0x5615c2bd64e0 msgr2=0x5615c2f82070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1a85ffb640 0 client.26248 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 -- 172.21.15.27:0/3985517245 shutdown_connections 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 1 -- 172.21.15.27:0/3985517245 wait complete. 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.786+0000 7f1aab632740 20 client.26248 trim_cache size 0 max 16384 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:46:23.788 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:46:23.789 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:46:23.789 INFO:teuthology.orchestra.run.smithi027.stdout: "08a2d3a8-050a-48f8-b402-01beda6bc529": { 2024-04-25T13:46:23.789 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:46:23.789 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "08a2d3a8-050a-48f8-b402-01beda6bc529", 2024-04-25T13:46:23.789 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:46:23.789 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:46:23.789 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:46:23.789 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 -- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a880afc30 msgr2=0x7f1a880b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a880afc30 0x7f1a880b2140 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1a9400fc90 tx=0x7f1a9401f000 comp rx=0 tx=0).stop 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 -- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c103c20 msgr2=0x7f1a9c19cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c103c20 0x7f1a9c19cc70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00e9f0 tx=0x7f1a8c00edf0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 -- 172.21.15.27:0/380647828 shutdown_connections 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a880afc30 0x7f1a880b2140 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a9c108920 0x7f1a9c1a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a9c103c20 0x7f1a9c19cc70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 --2- 172.21.15.27:0/380647828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a9c1029e0 0x7f1a9c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:23.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 -- 172.21.15.27:0/380647828 >> 172.21.15.27:0/380647828 conn(0x7f1a9c0fdbd0 msgr2=0x7f1a9c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:23.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 -- 172.21.15.27:0/380647828 shutdown_connections 2024-04-25T13:46:23.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:23.787+0000 7f1aa4784640 1 -- 172.21.15.27:0/380647828 wait complete. 2024-04-25T13:46:23.800 INFO:tasks.cephfs.filesystem:scrub status for tag:08a2d3a8-050a-48f8-b402-01beda6bc529 - {'path': '/', 'tag': '08a2d3a8-050a-48f8-b402-01beda6bc529', 'options': 'recursive,force'} 2024-04-25T13:46:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:23 smithi122 ceph-mon[32843]: pgmap v514: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.4 MiB/s wr, 799 op/s 2024-04-25T13:46:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:23 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:46:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:23 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:23 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:46:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:23 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:23 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:46:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:23 smithi045 ceph-mon[32731]: pgmap v514: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.4 MiB/s wr, 799 op/s 2024-04-25T13:46:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:23 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:46:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:23 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:23 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:46:24.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:23 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:24.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:23 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:46:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:23 smithi027 ceph-mon[29140]: pgmap v514: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.4 MiB/s wr, 799 op/s 2024-04-25T13:46:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:23 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:46:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:23 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:23 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:46:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:23 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:23 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.082+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/2973370061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0108990 msgr2=0x7fa1d010adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.082+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/2973370061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0108990 0x7fa1d010adb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009be0 tx=0x7fa1c802a6c0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/2973370061 shutdown_connections 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/2973370061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0108990 0x7fa1d010adb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/2973370061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1d0103c90 0x7fa1d01040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/2973370061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1d0102a50 0x7fa1d0103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/2973370061 >> 172.21.15.27:0/2973370061 conn(0x7fa1d00fdc60 msgr2=0x7fa1d01000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/2973370061 shutdown_connections 2024-04-25T13:46:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/2973370061 wait complete. 2024-04-25T13:46:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 Processor -- start 2024-04-25T13:46:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.083+0000 7fa1d5eaf640 1 -- start start 2024-04-25T13:46:26.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1d5eaf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0102a50 0x7fa1d0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1d5eaf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1d0103c90 0x7fa1d0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1d5eaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1d0108990 0x7fa1d019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1d5eaf640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d010d470 con 0x7fa1d0108990 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1d5eaf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d010d2f0 con 0x7fa1d0102a50 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1d5eaf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d010d5f0 con 0x7fa1d0103c90 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1d0108990 0x7fa1d019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0102a50 0x7fa1d0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0102a50 0x7fa1d0197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37928/0 (socket says 172.21.15.27:37928) 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1d0108990 0x7fa1d019f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33932/0 (socket says 172.21.15.27:33932) 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 -- 172.21.15.27:0/1460620955 learned_addr learned my addr 172.21.15.27:0/1460620955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1d0103c90 0x7fa1d0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 -- 172.21.15.27:0/1460620955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1d0103c90 msgr2=0x7fa1d0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1d0103c90 0x7fa1d0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 -- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1d0108990 msgr2=0x7fa1d019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1d0108990 0x7fa1d019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 -- 172.21.15.27:0/1460620955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d019fb20 con 0x7fa1d0102a50 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.084+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0102a50 0x7fa1d0197fe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa1c000e9f0 tx=0x7fa1c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.085+0000 7fa1ccff9640 1 -- 172.21.15.27:0/1460620955 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c000ccc0 con 0x7fa1d0102a50 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.085+0000 7fa1ccff9640 1 -- 172.21.15.27:0/1460620955 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1c00035f0 con 0x7fa1d0102a50 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.085+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/1460620955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d019fe10 con 0x7fa1d0102a50 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.085+0000 7fa1ccff9640 1 -- 172.21.15.27:0/1460620955 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c000ce60 con 0x7fa1d0102a50 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.085+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/1460620955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d01a0220 con 0x7fa1d0102a50 2024-04-25T13:46:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.087+0000 7fa1dcd65740 1 Processor -- start 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.088+0000 7fa1ccff9640 1 -- 172.21.15.27:0/1460620955 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa1c0003aa0 con 0x7fa1d0102a50 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.088+0000 7fa1dcd65740 1 -- start start 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e94dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 0x56204e94e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e94a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56204e8b0460 con 0x56204e5aa1b0 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56204e86a590 con 0x56204e5812f0 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56204e8ac420 con 0x56204e5894d0 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1ceffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e94a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1ceffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e94a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:48524/0 (socket says 172.21.15.27:48524) 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1ceffd640 1 -- 172.21.15.27:0/4272905489 learned_addr learned my addr 172.21.15.27:0/4272905489 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1cffff640 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e94dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.089+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 0x56204e94e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1ceffd640 1 -- 172.21.15.27:0/4272905489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 msgr2=0x56204e94e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 0x56204e94e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1ceffd640 1 -- 172.21.15.27:0/4272905489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 msgr2=0x56204e94dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e94dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1ceffd640 1 -- 172.21.15.27:0/4272905489 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56204e8770d0 con 0x56204e5894d0 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1ccff9640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1b80afb60 0x7fa1b80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e94a9e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4009c80 tx=0x7fa1c4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=112054e7fecb9648 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1cffff640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1b80afb60 0x7fa1b80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1ccff9640 1 -- 172.21.15.27:0/1460620955 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa1c00def80 con 0x7fa1d0102a50 2024-04-25T13:46:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1b5ffb640 1 -- 172.21.15.27:0/4272905489 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4004220 con 0x56204e5894d0 2024-04-25T13:46:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1b5ffb640 1 -- 172.21.15.27:0/4272905489 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1c40043c0 con 0x56204e5894d0 2024-04-25T13:46:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.090+0000 7fa1b5ffb640 1 -- 172.21.15.27:0/4272905489 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c40197b0 con 0x56204e5894d0 2024-04-25T13:46:26.094 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.091+0000 7fa1dcd65740 1 -- 172.21.15.27:0/4272905489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 msgr2=0x56204e94a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.094 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.091+0000 7fa1cffff640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1b80afb60 0x7fa1b80b2070 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x56204e94bac0 tx=0x7fa1c8030930 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:26.094 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.091+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e94a9e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4009c80 tx=0x7fa1c4011810 comp rx=0 tx=0).stop 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 -- 172.21.15.27:0/4272905489 shutdown_connections 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e94a9e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 0x56204e94e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/4272905489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e94dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 -- 172.21.15.27:0/4272905489 >> 172.21.15.27:0/4272905489 conn(0x56204e59e300 msgr2=0x56204e946300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 -- 172.21.15.27:0/4272905489 shutdown_connections 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 -- 172.21.15.27:0/4272905489 wait complete. 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 -- start start 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.092+0000 7fa1dcd65740 1 Processor -- start 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 0x56204e9544e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e954a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e959970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56204e837b60 con 0x56204e5aa1b0 2024-04-25T13:46:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e954a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1cf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 0x56204e9544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e954a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:48536/0 (socket says 172.21.15.27:48536) 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 -- 172.21.15.27:0/2986486557 learned_addr learned my addr 172.21.15.27:0/2986486557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1cffff640 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e959970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56204e94e9b0 con 0x56204e5812f0 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56204e959eb0 con 0x56204e5894d0 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1dcd65740 10 client.0 _ll_register_callbacks cb 0x56204e85c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 -- 172.21.15.27:0/2986486557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 msgr2=0x56204e9544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 0x56204e9544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 -- 172.21.15.27:0/2986486557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 msgr2=0x56204e959970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e959970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 -- 172.21.15.27:0/2986486557 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56204e5c2210 con 0x56204e5894d0 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1c40043c0 con 0x56204e5894d0 2024-04-25T13:46:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e954a20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1c401b040 tx=0x7fa1c4005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0812cd79f4d05a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.093+0000 7fa1aeffd640 10 client.26260 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.094+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2986486557 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4032050 con 0x56204e5894d0 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.094+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2986486557 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa1c40035f0 con 0x56204e5894d0 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.094+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2986486557 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa1c4028070 con 0x56204e5894d0 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.094+0000 7fa1dcd65740 10 client.26260 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.094+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x56204e68e620 con 0x56204e5894d0 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.094+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2986486557 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa1c4003850 con 0x56204e5894d0 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.095+0000 7fa1dcd65740 10 client.26260 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.095+0000 7fa1dcd65740 10 client.26260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.095+0000 7fa1dcd65740 20 client.26260 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.095+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56204e568d80 0x56204e972fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.095+0000 7fa1dcd65740 4 client.26260 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.095+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56204e8be070 con 0x56204e568d80 2024-04-25T13:46:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.095+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56204e568d80 0x56204e972fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.096+0000 7fa1aeffd640 10 client.26260 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.096+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56204e568d80 0x56204e972fd0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4506b844643d5d0e server_cookie=998eb046e775b294 in_seq=0 out_seq=0 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.096+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2986486557 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56204e8be070 con 0x56204e568d80 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.096+0000 7fa1aeffd640 10 client.26260 handle_command_reply: tid=0 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 client.26260 shutdown 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 msgr2=0x56204e954a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e954a20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa1c401b040 tx=0x7fa1c4005250 comp rx=0 tx=0).stop 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 shutdown_connections 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56204e568d80 0x56204e972fd0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e959970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e954a20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2986486557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5812f0 0x56204e9544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1aeffd640 0 client.26260 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:26.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 >> 172.21.15.27:0/2986486557 conn(0x56204e59e300 msgr2=0x56204e945250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:26.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 shutdown_connections 2024-04-25T13:46:26.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2986486557 wait complete. 2024-04-25T13:46:26.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.097+0000 7fa1dcd65740 20 client.26260 trim_cache size 0 max 16384 2024-04-25T13:46:26.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1dcd65740 1 Processor -- start 2024-04-25T13:46:26.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1dcd65740 1 -- start start 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5812f0 0x56204e945c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5894d0 0x56204e946150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e944820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d01a7010 con 0x56204e5aa1b0 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d01a9890 con 0x56204e5894d0 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d01a7680 con 0x56204e5812f0 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e944820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1cffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e944820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33952/0 (socket says 172.21.15.27:33952) 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1cffff640 1 -- 172.21.15.27:0/3959709043 learned_addr learned my addr 172.21.15.27:0/3959709043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5812f0 0x56204e945c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.116+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5894d0 0x56204e946150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.117+0000 7fa1cf7fe640 1 -- 172.21.15.27:0/3959709043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5894d0 msgr2=0x56204e946150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.117+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5894d0 0x56204e946150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.117+0000 7fa1cf7fe640 1 -- 172.21.15.27:0/3959709043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 msgr2=0x56204e944820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.117+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e944820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.117+0000 7fa1cf7fe640 1 -- 172.21.15.27:0/3959709043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56204e94fdf0 con 0x56204e5812f0 2024-04-25T13:46:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.117+0000 7fa1cffff640 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e944820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.118+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5812f0 0x56204e945c10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x56204e94bcd0 tx=0x7fa1c00e1350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5433745caba99f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.118+0000 7fa1b57fa640 1 -- 172.21.15.27:0/3959709043 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c00a6d80 con 0x56204e5812f0 2024-04-25T13:46:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.119+0000 7fa1b57fa640 1 -- 172.21.15.27:0/3959709043 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1c00a6f40 con 0x56204e5812f0 2024-04-25T13:46:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.119+0000 7fa1b57fa640 1 -- 172.21.15.27:0/3959709043 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c00a6ba0 con 0x56204e5812f0 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.119+0000 7fa1dcd65740 1 -- 172.21.15.27:0/3959709043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5812f0 msgr2=0x56204e945c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.119+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5812f0 0x56204e945c10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x56204e94bcd0 tx=0x7fa1c00e1350 comp rx=0 tx=0).stop 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.119+0000 7fa1dcd65740 1 -- 172.21.15.27:0/3959709043 shutdown_connections 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.119+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5aa1b0 0x56204e944820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.119+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5894d0 0x56204e946150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.119+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/3959709043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5812f0 0x56204e945c10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 -- 172.21.15.27:0/3959709043 >> 172.21.15.27:0/3959709043 conn(0x56204e59e300 msgr2=0x56204e952b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 -- 172.21.15.27:0/3959709043 shutdown_connections 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 -- 172.21.15.27:0/3959709043 wait complete. 2024-04-25T13:46:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 -- start start 2024-04-25T13:46:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 Processor -- start 2024-04-25T13:46:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5812f0 0x56204e982090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e9825d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5aa1b0 0x56204e983500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d01a7d50 con 0x56204e5812f0 2024-04-25T13:46:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d01a91c0 con 0x56204e5aa1b0 2024-04-25T13:46:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.120+0000 7fa1dcd65740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d01a8420 con 0x56204e5894d0 2024-04-25T13:46:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1dcd65740 10 client.0 _ll_register_callbacks cb 0x7fa1c4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5812f0 0x56204e982090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5aa1b0 0x56204e983500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5812f0 0x56204e982090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33960/0 (socket says 172.21.15.27:33960) 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cf7fe640 1 -- 172.21.15.27:0/2078304006 learned_addr learned my addr 172.21.15.27:0/2078304006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e9825d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cffff640 1 -- 172.21.15.27:0/2078304006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 msgr2=0x56204e9825d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cffff640 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e9825d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cffff640 1 -- 172.21.15.27:0/2078304006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5812f0 msgr2=0x56204e982090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cffff640 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5812f0 0x56204e982090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cffff640 1 -- 172.21.15.27:0/2078304006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56204e983ac0 con 0x56204e5aa1b0 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5812f0 0x56204e982090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.121+0000 7fa1ceffd640 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e9825d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:46:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.122+0000 7fa1cffff640 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5aa1b0 0x56204e983500 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa1c802abe0 tx=0x7fa1c8009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ed7fdecf1621bfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.122+0000 7fa1aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.122+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2078304006 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8059060 con 0x56204e5aa1b0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.122+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2078304006 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa1c8031440 con 0x56204e5aa1b0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.122+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2078304006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1c00a6f40 con 0x56204e5aa1b0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2078304006 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa1c8036070 con 0x56204e5aa1b0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1dcd65740 10 client.26222 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2078304006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56204e9db0c0 con 0x56204e5aa1b0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2078304006 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1c00a6f40 con 0x56204e5aa1b0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2078304006 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa1c8031c70 con 0x56204e5aa1b0 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1dcd65740 10 client.26222 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1dcd65740 10 client.26222 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1dcd65740 20 client.26222 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56204e53e1c0 0x56204e9dda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1dcd65740 4 client.26222 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:46:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.123+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2078304006 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56204e8b7ee0 con 0x56204e53e1c0 2024-04-25T13:46:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.124+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56204e53e1c0 0x56204e9dda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.124+0000 7fa1cf7fe640 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56204e53e1c0 0x56204e9dda00 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afac0af1d157c157 server_cookie=d48cb9bbf3bc4edb in_seq=0 out_seq=0 2024-04-25T13:46:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.124+0000 7fa1aeffd640 10 client.26222 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1aeffd640 1 -- 172.21.15.27:0/2078304006 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56204e8b7ee0 con 0x56204e53e1c0 2024-04-25T13:46:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1aeffd640 10 client.26222 handle_command_reply: tid=0 2024-04-25T13:46:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 client.26222 shutdown 2024-04-25T13:46:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2078304006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5aa1b0 msgr2=0x56204e983500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5aa1b0 0x56204e983500 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa1c802abe0 tx=0x7fa1c8009500 comp rx=0 tx=0).stop 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2078304006 shutdown_connections 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56204e53e1c0 0x56204e9dda00 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56204e5aa1b0 0x56204e983500 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1aeffd640 0 client.26222 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56204e5894d0 0x56204e9825d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 --2- 172.21.15.27:0/2078304006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56204e5812f0 0x56204e982090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2078304006 >> 172.21.15.27:0/2078304006 conn(0x56204e59e300 msgr2=0x56204e952740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2078304006 shutdown_connections 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 1 -- 172.21.15.27:0/2078304006 wait complete. 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.139+0000 7fa1dcd65740 20 client.26222 trim_cache size 0 max 16384 2024-04-25T13:46:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:46:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:46:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:46:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:46:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1b80afb60 msgr2=0x7fa1b80b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1b80afb60 0x7fa1b80b2070 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x56204e94bac0 tx=0x7fa1c8030930 comp rx=0 tx=0).stop 2024-04-25T13:46:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/1460620955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0102a50 msgr2=0x7fa1d0197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0102a50 0x7fa1d0197fe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa1c000e9f0 tx=0x7fa1c000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/1460620955 shutdown_connections 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1b80afb60 0x7fa1b80b2070 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1d0108990 0x7fa1d019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa1d0103c90 0x7fa1d0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 --2- 172.21.15.27:0/1460620955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d0102a50 0x7fa1d0197fe0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/1460620955 >> 172.21.15.27:0/1460620955 conn(0x7fa1d00fdc60 msgr2=0x7fa1d0104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/1460620955 shutdown_connections 2024-04-25T13:46:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:26.140+0000 7fa1d5eaf640 1 -- 172.21.15.27:0/1460620955 wait complete. 2024-04-25T13:46:26.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:25 smithi122 ceph-mon[32843]: pgmap v515: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.1 MiB/s wr, 768 op/s 2024-04-25T13:46:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:25 smithi045 ceph-mon[32731]: pgmap v515: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.1 MiB/s wr, 768 op/s 2024-04-25T13:46:26.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:25 smithi027 ceph-mon[29140]: pgmap v515: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.1 MiB/s wr, 768 op/s 2024-04-25T13:46:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 320.0 s, 2377.2 tps, lat 42.024 ms stddev 48.722 2024-04-25T13:46:28.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:27 smithi122 ceph-mon[32843]: pgmap v516: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 8.2 MiB/s wr, 878 op/s 2024-04-25T13:46:28.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:27 smithi045 ceph-mon[32731]: pgmap v516: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 8.2 MiB/s wr, 878 op/s 2024-04-25T13:46:28.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:27 smithi027 ceph-mon[29140]: pgmap v516: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 8.2 MiB/s wr, 878 op/s 2024-04-25T13:46:30.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:29 smithi122 ceph-mon[32843]: pgmap v517: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.2 MiB/s wr, 703 op/s 2024-04-25T13:46:30.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:29 smithi045 ceph-mon[32731]: pgmap v517: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.2 MiB/s wr, 703 op/s 2024-04-25T13:46:30.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:29 smithi027 ceph-mon[29140]: pgmap v517: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.2 MiB/s wr, 703 op/s 2024-04-25T13:46:32.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:31 smithi122 ceph-mon[32843]: pgmap v518: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 8.5 MiB/s wr, 813 op/s 2024-04-25T13:46:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:31 smithi045 ceph-mon[32731]: pgmap v518: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 8.5 MiB/s wr, 813 op/s 2024-04-25T13:46:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:31 smithi027 ceph-mon[29140]: pgmap v518: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 8.5 MiB/s wr, 813 op/s 2024-04-25T13:46:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 325.0 s, 2635.0 tps, lat 37.764 ms stddev 42.422 2024-04-25T13:46:34.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:33 smithi122 ceph-mon[32843]: pgmap v519: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 872 op/s 2024-04-25T13:46:34.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:33 smithi045 ceph-mon[32731]: pgmap v519: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 872 op/s 2024-04-25T13:46:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:33 smithi027 ceph-mon[29140]: pgmap v519: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 872 op/s 2024-04-25T13:46:36.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:35 smithi122 ceph-mon[32843]: pgmap v520: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-25T13:46:36.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:35 smithi045 ceph-mon[32731]: pgmap v520: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-25T13:46:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:35 smithi027 ceph-mon[29140]: pgmap v520: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-25T13:46:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 330.0 s, 2563.6 tps, lat 39.218 ms stddev 44.012 2024-04-25T13:46:38.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:37 smithi122 ceph-mon[32843]: pgmap v521: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 17 MiB/s wr, 981 op/s 2024-04-25T13:46:38.330 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:37 smithi045 ceph-mon[32731]: pgmap v521: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 17 MiB/s wr, 981 op/s 2024-04-25T13:46:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:37 smithi027 ceph-mon[29140]: pgmap v521: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 17 MiB/s wr, 981 op/s 2024-04-25T13:46:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 -- 172.21.15.27:0/871502672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8103c80 msgr2=0x7fbff8104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/871502672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8103c80 0x7fbff8104090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009900 tx=0x7fbfec02a410 comp rx=0 tx=0).stop 2024-04-25T13:46:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 -- 172.21.15.27:0/871502672 shutdown_connections 2024-04-25T13:46:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/871502672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbff8108980 0x7fbff810ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/871502672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8103c80 0x7fbff8104090 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/871502672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8102a40 0x7fbff8103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 -- 172.21.15.27:0/871502672 >> 172.21.15.27:0/871502672 conn(0x7fbff80fdc30 msgr2=0x7fbff81000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 -- 172.21.15.27:0/871502672 shutdown_connections 2024-04-25T13:46:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.283+0000 7fbfffcb6640 1 -- 172.21.15.27:0/871502672 wait complete. 2024-04-25T13:46:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbfffcb6640 1 Processor -- start 2024-04-25T13:46:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbfffcb6640 1 -- start start 2024-04-25T13:46:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbfffcb6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8102a40 0x7fbff8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbfffcb6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8103c80 0x7fbff8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbfffcb6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbff8108980 0x7fbff81a7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbfffcb6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbff810f670 con 0x7fbff8103c80 2024-04-25T13:46:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbfffcb6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbff810f4f0 con 0x7fbff8102a40 2024-04-25T13:46:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbfffcb6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbff810f7f0 con 0x7fbff8108980 2024-04-25T13:46:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbffda52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8102a40 0x7fbff8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbffda52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8102a40 0x7fbff8198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45098/0 (socket says 172.21.15.27:45098) 2024-04-25T13:46:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.285+0000 7fbffda52640 1 -- 172.21.15.27:0/1630330335 learned_addr learned my addr 172.21.15.27:0/1630330335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffe253640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbff8108980 0x7fbff81a7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffd251640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8103c80 0x7fbff8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffda52640 1 -- 172.21.15.27:0/1630330335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbff8108980 msgr2=0x7fbff81a7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffda52640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbff8108980 0x7fbff81a7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffda52640 1 -- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8103c80 msgr2=0x7fbff8198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffda52640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8103c80 0x7fbff8198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffda52640 1 -- 172.21.15.27:0/1630330335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff81a8360 con 0x7fbff8102a40 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffd251640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8103c80 0x7fbff8198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.286+0000 7fbffda52640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8102a40 0x7fbff8198010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800eb90 tx=0x7fbfe800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.287+0000 7fbff6ffd640 1 -- 172.21.15.27:0/1630330335 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8003b70 con 0x7fbff8102a40 2024-04-25T13:46:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.287+0000 7fbff6ffd640 1 -- 172.21.15.27:0/1630330335 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfe800ee90 con 0x7fbff8102a40 2024-04-25T13:46:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.287+0000 7fbff6ffd640 1 -- 172.21.15.27:0/1630330335 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80197b0 con 0x7fbff8102a40 2024-04-25T13:46:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.287+0000 7fbfffcb6640 1 -- 172.21.15.27:0/1630330335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff81a8650 con 0x7fbff8102a40 2024-04-25T13:46:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.287+0000 7fbfffcb6640 1 -- 172.21.15.27:0/1630330335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff81a8b50 con 0x7fbff8102a40 2024-04-25T13:46:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.290+0000 7fbff6ffd640 1 -- 172.21.15.27:0/1630330335 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbfe8004030 con 0x7fbff8102a40 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.291+0000 7fbff6ffd640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbfe00afbe0 0x7fbfe00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fc006b69740 1 Processor -- start 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fbffd251640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbfe00afbe0 0x7fbfe00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fc006b69740 1 -- start start 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fbff6ffd640 1 -- 172.21.15.27:0/1630330335 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbfe80df590 con 0x7fbff8102a40 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fc006b69740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e07e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fc006b69740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcb52f0 0x56169e07eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fc006b69740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcbd4d0 0x56169e085df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fc006b69740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56169dfe4460 con 0x56169dcb52f0 2024-04-25T13:46:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fc006b69740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56169df9e590 con 0x56169dcde1b0 2024-04-25T13:46:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fc006b69740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56169dfe0420 con 0x56169dcbd4d0 2024-04-25T13:46:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fbffda52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e07e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fbffda52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e07e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45110/0 (socket says 172.21.15.27:45110) 2024-04-25T13:46:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.292+0000 7fbffda52640 1 -- 172.21.15.27:0/1435435240 learned_addr learned my addr 172.21.15.27:0/1435435240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffe253640 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcbd4d0 0x56169e085df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffd251640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbfe00afbe0 0x7fbfe00b20f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbfec006010 tx=0x7fbfec000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffd251640 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcb52f0 0x56169e07eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffda52640 1 -- 172.21.15.27:0/1435435240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcbd4d0 msgr2=0x56169e085df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffda52640 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcbd4d0 0x56169e085df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffda52640 1 -- 172.21.15.27:0/1435435240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcb52f0 msgr2=0x56169e07eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffda52640 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcb52f0 0x56169e07eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffda52640 1 -- 172.21.15.27:0/1435435240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56169dfab0d0 con 0x56169dcde1b0 2024-04-25T13:46:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffd251640 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcb52f0 0x56169e07eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:46:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbffda52640 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e07e970 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009270 tx=0x7fbfe80a7390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62889f1064e4d7f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbfdffff640 1 -- 172.21.15.27:0/1435435240 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80b0040 con 0x56169dcde1b0 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbfdffff640 1 -- 172.21.15.27:0/1435435240 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfe80a7550 con 0x56169dcde1b0 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.293+0000 7fbfdffff640 1 -- 172.21.15.27:0/1435435240 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80a3cd0 con 0x56169dcde1b0 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.294+0000 7fc006b69740 1 -- 172.21.15.27:0/1435435240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 msgr2=0x56169e07e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.295+0000 7fc006b69740 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e07e970 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009270 tx=0x7fbfe80a7390 comp rx=0 tx=0).stop 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.295+0000 7fc006b69740 1 -- 172.21.15.27:0/1435435240 shutdown_connections 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.295+0000 7fc006b69740 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcbd4d0 0x56169e085df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.295+0000 7fc006b69740 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcb52f0 0x56169e07eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.295+0000 7fc006b69740 1 --2- 172.21.15.27:0/1435435240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e07e970 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.295+0000 7fc006b69740 1 -- 172.21.15.27:0/1435435240 >> 172.21.15.27:0/1435435240 conn(0x56169dcd2300 msgr2=0x56169e07a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.295+0000 7fc006b69740 1 -- 172.21.15.27:0/1435435240 shutdown_connections 2024-04-25T13:46:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.295+0000 7fc006b69740 1 -- 172.21.15.27:0/1435435240 wait complete. 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.296+0000 7fc006b69740 1 -- start start 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.296+0000 7fc006b69740 1 Processor -- start 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.297+0000 7fc006b69740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 0x56169e0886a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.297+0000 7fc006b69740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e088be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.297+0000 7fc006b69740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.297+0000 7fbffd251640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e088be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffda52640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 0x56169e0886a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffd251640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e088be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48130/0 (socket says 172.21.15.27:48130) 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffd251640 1 -- 172.21.15.27:0/927348591 learned_addr learned my addr 172.21.15.27:0/927348591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56169e086840 con 0x56169dcbd4d0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56169df6bb60 con 0x56169dcde1b0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffe253640 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56169e0869c0 con 0x56169dcb52f0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fc006b69740 10 client.0 _ll_register_callbacks cb 0x56169df902b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffd251640 1 -- 172.21.15.27:0/927348591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 msgr2=0x56169e0886a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffd251640 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 0x56169e0886a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffd251640 1 -- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 msgr2=0x56169e08fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffd251640 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffd251640 1 -- 172.21.15.27:0/927348591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56169dcf6210 con 0x56169dcbd4d0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffe253640 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbffd251640 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e088be0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009900 tx=0x7fbfec009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5707352fd87964b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.298+0000 7fbfdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.299+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/927348591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec04d070 con 0x56169dcbd4d0 2024-04-25T13:46:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.299+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfe80a7550 con 0x56169dcbd4d0 2024-04-25T13:46:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.299+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/927348591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbfec004450 con 0x56169dcbd4d0 2024-04-25T13:46:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.299+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/927348591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fbfec059050 con 0x56169dcbd4d0 2024-04-25T13:46:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.299+0000 7fc006b69740 10 client.16518 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.299+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56169ddc2620 con 0x56169dcbd4d0 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.300+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/927348591 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfe80a7550 con 0x56169dcbd4d0 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.300+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/927348591 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fbfec004cb0 con 0x56169dcbd4d0 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.300+0000 7fc006b69740 10 client.16518 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.300+0000 7fc006b69740 10 client.16518 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.300+0000 7fc006b69740 20 client.16518 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.300+0000 7fc006b69740 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56169dc9cd80 0x56169e096390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.300+0000 7fc006b69740 4 client.16518 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.300+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56169dff2070 con 0x56169dc9cd80 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.301+0000 7fbffda52640 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56169dc9cd80 0x56169e096390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.301+0000 7fbfdd7fa640 10 client.16518 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.301+0000 7fbffda52640 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56169dc9cd80 0x56169e096390 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2834311154328e73 server_cookie=c1a8ac1bcad72995 in_seq=0 out_seq=0 2024-04-25T13:46:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/927348591 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56169dff2070 con 0x56169dc9cd80 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fbfdd7fa640 10 client.16518 handle_command_reply: tid=0 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 client.16518 shutdown 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 msgr2=0x56169e088be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e088be0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009900 tx=0x7fbfec009260 comp rx=0 tx=0).stop 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 shutdown_connections 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56169dc9cd80 0x56169e096390 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e088be0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 --2- 172.21.15.27:0/927348591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 0x56169e0886a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fbfdd7fa640 0 client.16518 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 >> 172.21.15.27:0/927348591 conn(0x56169dcd2300 msgr2=0x56169e079250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 shutdown_connections 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.302+0000 7fc006b69740 1 -- 172.21.15.27:0/927348591 wait complete. 2024-04-25T13:46:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.303+0000 7fc006b69740 20 client.16518 trim_cache size 0 max 16384 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.326+0000 7fc006b69740 1 Processor -- start 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.326+0000 7fc006b69740 1 -- start start 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fc006b69740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcb52f0 0x56169e07a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fc006b69740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e08fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fc006b69740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcde1b0 0x56169e086d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fc006b69740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81af950 con 0x56169dcbd4d0 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fc006b69740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b21d0 con 0x56169dcb52f0 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fc006b69740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81affc0 con 0x56169dcde1b0 2024-04-25T13:46:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fbffd251640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e08fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fbffda52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcb52f0 0x56169e07a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fbffda52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcb52f0 0x56169e07a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45130/0 (socket says 172.21.15.27:45130) 2024-04-25T13:46:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fbffda52640 1 -- 172.21.15.27:0/2387225459 learned_addr learned my addr 172.21.15.27:0/2387225459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.327+0000 7fbffe253640 1 --2- 172.21.15.27:0/2387225459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcde1b0 0x56169e086d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbffe253640 1 -- 172.21.15.27:0/2387225459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcb52f0 msgr2=0x56169e07a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbffe253640 1 --2- 172.21.15.27:0/2387225459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcb52f0 0x56169e07a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbffe253640 1 -- 172.21.15.27:0/2387225459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 msgr2=0x56169e08fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbffe253640 1 --2- 172.21.15.27:0/2387225459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e08fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbffe253640 1 -- 172.21.15.27:0/2387225459 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56169e0834f0 con 0x56169dcde1b0 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbffe253640 1 --2- 172.21.15.27:0/2387225459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcde1b0 0x56169e086d60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbff000bd30 tx=0x7fbff0012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36b7b026d03d9bb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbfdf7fe640 1 -- 172.21.15.27:0/2387225459 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff000be20 con 0x56169dcde1b0 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbfdf7fe640 1 -- 172.21.15.27:0/2387225459 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbff00054d0 con 0x56169dcde1b0 2024-04-25T13:46:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.328+0000 7fbfdf7fe640 1 -- 172.21.15.27:0/2387225459 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff001b670 con 0x56169dcde1b0 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 -- 172.21.15.27:0/2387225459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcde1b0 msgr2=0x56169e086d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 --2- 172.21.15.27:0/2387225459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcde1b0 0x56169e086d60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbff000bd30 tx=0x7fbff0012820 comp rx=0 tx=0).stop 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 -- 172.21.15.27:0/2387225459 shutdown_connections 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 --2- 172.21.15.27:0/2387225459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcde1b0 0x56169e086d60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 --2- 172.21.15.27:0/2387225459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e08fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 --2- 172.21.15.27:0/2387225459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcb52f0 0x56169e07a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 -- 172.21.15.27:0/2387225459 >> 172.21.15.27:0/2387225459 conn(0x56169dcd2300 msgr2=0x56169e079ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 -- 172.21.15.27:0/2387225459 shutdown_connections 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 -- 172.21.15.27:0/2387225459 wait complete. 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 -- start start 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.330+0000 7fc006b69740 1 Processor -- start 2024-04-25T13:46:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fc006b69740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 0x56169e08ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fc006b69740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e08d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fc006b69740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fc006b69740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b0690 con 0x56169dcbd4d0 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fc006b69740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b1b00 con 0x56169dcde1b0 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fc006b69740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b0d60 con 0x56169dcb52f0 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fc006b69740 10 client.0 _ll_register_callbacks cb 0x7fbfe80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffe253640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffe253640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45142/0 (socket says 172.21.15.27:45142) 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffe253640 1 -- 172.21.15.27:0/586533469 learned_addr learned my addr 172.21.15.27:0/586533469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffd251640 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e08d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffda52640 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 0x56169e08ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffe253640 1 -- 172.21.15.27:0/586533469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 msgr2=0x56169e08ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffe253640 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 0x56169e08ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffe253640 1 -- 172.21.15.27:0/586533469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 msgr2=0x56169e08d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffe253640 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e08d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.331+0000 7fbffe253640 1 -- 172.21.15.27:0/586533469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56169e08ffa0 con 0x56169dcde1b0 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fbffe253640 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08e280 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbff001d040 tx=0x7fbff00056f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=786ecbe802606cd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fbfdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/586533469 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0029020 con 0x56169dcde1b0 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/586533469 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbff0004850 con 0x56169dcde1b0 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fc006b69740 1 -- 172.21.15.27:0/586533469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbff00054d0 con 0x56169dcde1b0 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/586533469 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fbff0005850 con 0x56169dcde1b0 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fc006b69740 10 client.26240 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fc006b69740 1 -- 172.21.15.27:0/586533469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56169e08f930 con 0x56169dcde1b0 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.332+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/586533469 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbff00054d0 con 0x56169dcde1b0 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.333+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/586533469 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fbff001b410 con 0x56169dcde1b0 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.333+0000 7fc006b69740 10 client.26240 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.333+0000 7fc006b69740 10 client.26240 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.333+0000 7fc006b69740 20 client.26240 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.333+0000 7fc006b69740 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56169dc721c0 0x56169e100b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.333+0000 7fc006b69740 4 client.26240 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.333+0000 7fc006b69740 1 -- 172.21.15.27:0/586533469 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56169dfebee0 con 0x56169dc721c0 2024-04-25T13:46:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.333+0000 7fbffda52640 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56169dc721c0 0x56169e100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.334+0000 7fbffda52640 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56169dc721c0 0x56169e100b10 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e753571483e3db server_cookie=92fa34e8856ab44 in_seq=0 out_seq=0 2024-04-25T13:46:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.334+0000 7fbfdd7fa640 10 client.26240 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.353+0000 7fbfdd7fa640 1 -- 172.21.15.27:0/586533469 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56169dfebee0 con 0x56169dc721c0 2024-04-25T13:46:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.353+0000 7fbfdd7fa640 10 client.26240 handle_command_reply: tid=0 2024-04-25T13:46:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.353+0000 7fc006b69740 1 client.26240 shutdown 2024-04-25T13:46:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.353+0000 7fc006b69740 1 -- 172.21.15.27:0/586533469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 msgr2=0x56169e08e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.353+0000 7fc006b69740 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08e280 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbff001d040 tx=0x7fbff00056f0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fc006b69740 1 -- 172.21.15.27:0/586533469 shutdown_connections 2024-04-25T13:46:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fc006b69740 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56169dc721c0 0x56169e100b10 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fc006b69740 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56169dcde1b0 0x56169e08e280 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fbfdd7fa640 0 client.26240 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fc006b69740 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56169dcbd4d0 0x56169e08d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fc006b69740 1 --2- 172.21.15.27:0/586533469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56169dcb52f0 0x56169e08ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fc006b69740 1 -- 172.21.15.27:0/586533469 >> 172.21.15.27:0/586533469 conn(0x56169dcd2300 msgr2=0x56169e086d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fc006b69740 1 -- 172.21.15.27:0/586533469 shutdown_connections 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.354+0000 7fc006b69740 1 -- 172.21.15.27:0/586533469 wait complete. 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.355+0000 7fc006b69740 20 client.26240 trim_cache size 0 max 16384 2024-04-25T13:46:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:46:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:46:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:46:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 -- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbfe00afbe0 msgr2=0x7fbfe00b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbfe00afbe0 0x7fbfe00b20f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbfec006010 tx=0x7fbfec000bc0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 -- 172.21.15.27:0/1630330335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8102a40 msgr2=0x7fbff8198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8102a40 0x7fbff8198010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800eb90 tx=0x7fbfe800c6a0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 -- 172.21.15.27:0/1630330335 shutdown_connections 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbfe00afbe0 0x7fbfe00b20f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbff8108980 0x7fbff81a7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8103c80 0x7fbff8198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 --2- 172.21.15.27:0/1630330335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbff8102a40 0x7fbff8198010 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 -- 172.21.15.27:0/1630330335 >> 172.21.15.27:0/1630330335 conn(0x7fbff80fdc30 msgr2=0x7fbff80fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 -- 172.21.15.27:0/1630330335 shutdown_connections 2024-04-25T13:46:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:39.357+0000 7fbfffcb6640 1 -- 172.21.15.27:0/1630330335 wait complete. 2024-04-25T13:46:40.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:39 smithi122 ceph-mon[32843]: pgmap v522: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 17 MiB/s wr, 758 op/s 2024-04-25T13:46:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:39 smithi045 ceph-mon[32731]: pgmap v522: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 17 MiB/s wr, 758 op/s 2024-04-25T13:46:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:39 smithi027 ceph-mon[29140]: pgmap v522: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 17 MiB/s wr, 758 op/s 2024-04-25T13:46:42.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:41 smithi122 ceph-mon[32843]: pgmap v523: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 840 op/s 2024-04-25T13:46:42.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:41 smithi045 ceph-mon[32731]: pgmap v523: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 840 op/s 2024-04-25T13:46:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:41 smithi027 ceph-mon[29140]: pgmap v523: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 840 op/s 2024-04-25T13:46:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 335.0 s, 3069.2 tps, lat 32.587 ms stddev 38.034 2024-04-25T13:46:44.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:43 smithi122 ceph-mon[32843]: pgmap v524: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 MiB/s wr, 862 op/s 2024-04-25T13:46:44.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:43 smithi045 ceph-mon[32731]: pgmap v524: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 MiB/s wr, 862 op/s 2024-04-25T13:46:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:43 smithi027 ceph-mon[29140]: pgmap v524: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 MiB/s wr, 862 op/s 2024-04-25T13:46:46.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:45 smithi122 ceph-mon[32843]: pgmap v525: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 773 op/s 2024-04-25T13:46:46.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:45 smithi045 ceph-mon[32731]: pgmap v525: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 773 op/s 2024-04-25T13:46:46.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:45 smithi027 ceph-mon[29140]: pgmap v525: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 773 op/s 2024-04-25T13:46:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 340.0 s, 2529.0 tps, lat 39.525 ms stddev 42.760 2024-04-25T13:46:48.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:47 smithi122 ceph-mon[32843]: pgmap v526: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 25 MiB/s wr, 894 op/s 2024-04-25T13:46:48.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:47 smithi045 ceph-mon[32731]: pgmap v526: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 25 MiB/s wr, 894 op/s 2024-04-25T13:46:48.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:47 smithi027 ceph-mon[29140]: pgmap v526: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 25 MiB/s wr, 894 op/s 2024-04-25T13:46:50.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:49 smithi122 ceph-mon[32843]: pgmap v527: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 660 op/s 2024-04-25T13:46:50.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:49 smithi045 ceph-mon[32731]: pgmap v527: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 660 op/s 2024-04-25T13:46:50.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:49 smithi027 ceph-mon[29140]: pgmap v527: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 660 op/s 2024-04-25T13:46:52.310 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:51 smithi122 ceph-mon[32843]: pgmap v528: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 20 MiB/s wr, 753 op/s 2024-04-25T13:46:52.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:51 smithi045 ceph-mon[32731]: pgmap v528: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 20 MiB/s wr, 753 op/s 2024-04-25T13:46:52.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:51 smithi027 ceph-mon[29140]: pgmap v528: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 20 MiB/s wr, 753 op/s 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.505+0000 7f6818508640 1 -- 172.21.15.27:0/2112641617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 msgr2=0x7f6810103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.505+0000 7f6818508640 1 --2- 172.21.15.27:0/2112641617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 0x7f6810103710 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f680c009900 tx=0x7f680c02a410 comp rx=0 tx=0).stop 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 -- 172.21.15.27:0/2112641617 shutdown_connections 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 --2- 172.21.15.27:0/2112641617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6810108950 0x7f681010ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 --2- 172.21.15.27:0/2112641617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6810103c50 0x7f6810104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 --2- 172.21.15.27:0/2112641617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 0x7f6810103710 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 -- 172.21.15.27:0/2112641617 >> 172.21.15.27:0/2112641617 conn(0x7f68100fdc00 msgr2=0x7f68101000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 -- 172.21.15.27:0/2112641617 shutdown_connections 2024-04-25T13:46:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 -- 172.21.15.27:0/2112641617 wait complete. 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 Processor -- start 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.506+0000 7f6818508640 1 -- start start 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6818508640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 0x7f6810075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6818508640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6810103c50 0x7f681019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6818508640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6810108950 0x7f68101a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6818508640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f681010e510 con 0x7f6810102a10 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6818508640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f681010e390 con 0x7f6810108950 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6818508640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f681010e690 con 0x7f6810103c50 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f68162a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 0x7f6810075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6816aa5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6810108950 0x7f68101a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f68162a4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 0x7f6810075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51336/0 (socket says 172.21.15.27:51336) 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f68162a4640 1 -- 172.21.15.27:0/921496027 learned_addr learned my addr 172.21.15.27:0/921496027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6815aa3640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6810103c50 0x7f681019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6815aa3640 1 -- 172.21.15.27:0/921496027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6810108950 msgr2=0x7f68101a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6815aa3640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6810108950 0x7f68101a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6815aa3640 1 -- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 msgr2=0x7f6810075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6815aa3640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 0x7f6810075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.509 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.507+0000 7f6815aa3640 1 -- 172.21.15.27:0/921496027 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68101a1d60 con 0x7f6810103c50 2024-04-25T13:46:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.508+0000 7f6815aa3640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6810103c50 0x7f681019eff0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f680000e9f0 tx=0x7f680000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.508+0000 7f68077fe640 1 -- 172.21.15.27:0/921496027 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680000ccc0 con 0x7f6810103c50 2024-04-25T13:46:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.508+0000 7f68077fe640 1 -- 172.21.15.27:0/921496027 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68000108e0 con 0x7f6810103c50 2024-04-25T13:46:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.508+0000 7f68077fe640 1 -- 172.21.15.27:0/921496027 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6800018880 con 0x7f6810103c50 2024-04-25T13:46:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.508+0000 7f6818508640 1 -- 172.21.15.27:0/921496027 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68101a1ff0 con 0x7f6810103c50 2024-04-25T13:46:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.509+0000 7f6818508640 1 -- 172.21.15.27:0/921496027 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68101a2460 con 0x7f6810103c50 2024-04-25T13:46:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.511+0000 7f68077fe640 1 -- 172.21.15.27:0/921496027 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6800018a20 con 0x7f6810103c50 2024-04-25T13:46:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.512+0000 7f68077fe640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67fc0afbe0 0x7f67fc0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.512+0000 7f68077fe640 1 -- 172.21.15.27:0/921496027 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f68000deef0 con 0x7f6810103c50 2024-04-25T13:46:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.512+0000 7f68162a4640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67fc0afbe0 0x7f67fc0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.513+0000 7f681f3b7740 1 Processor -- start 2024-04-25T13:46:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.513+0000 7f681f3b7740 1 -- start start 2024-04-25T13:46:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.513+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d2531b0 0x55f52d5fe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.513+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d600d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.513+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d603510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f681f3b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f52d559460 con 0x55f52d2324d0 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f681f3b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f52d513590 con 0x55f52d22a2f0 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f681f3b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f52d555420 con 0x55f52d2531b0 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f6816aa5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d603510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f6816aa5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d603510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51340/0 (socket says 172.21.15.27:51340) 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f6816aa5640 1 -- 172.21.15.27:0/3393004762 learned_addr learned my addr 172.21.15.27:0/3393004762 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f68162a4640 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d2531b0 0x55f52d5fe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f68162a4640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67fc0afbe0 0x7f67fc0b20f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f680c0057c0 tx=0x7f680c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f6815aa3640 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d600d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f68162a4640 1 -- 172.21.15.27:0/3393004762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 msgr2=0x55f52d600d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f68162a4640 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d600d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f68162a4640 1 -- 172.21.15.27:0/3393004762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 msgr2=0x55f52d603510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f68162a4640 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d603510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f68162a4640 1 -- 172.21.15.27:0/3393004762 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f52d5200d0 con 0x55f52d2531b0 2024-04-25T13:46:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.514+0000 7f6816aa5640 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d603510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.515+0000 7f68162a4640 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d2531b0 0x55f52d5fe7d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f680c000c00 tx=0x7f680c03f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9882a992728d63d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.515+0000 7f6804bfe640 1 -- 172.21.15.27:0/3393004762 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c035020 con 0x55f52d2531b0 2024-04-25T13:46:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.515+0000 7f6804bfe640 1 -- 172.21.15.27:0/3393004762 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f680c031790 con 0x55f52d2531b0 2024-04-25T13:46:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.515+0000 7f6804bfe640 1 -- 172.21.15.27:0/3393004762 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c004a20 con 0x55f52d2531b0 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.516+0000 7f681f3b7740 1 -- 172.21.15.27:0/3393004762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d2531b0 msgr2=0x55f52d5fe7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.516+0000 7f681f3b7740 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d2531b0 0x55f52d5fe7d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f680c000c00 tx=0x7f680c03f000 comp rx=0 tx=0).stop 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.516+0000 7f681f3b7740 1 -- 172.21.15.27:0/3393004762 shutdown_connections 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.516+0000 7f681f3b7740 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d603510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.516+0000 7f681f3b7740 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d600d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.516+0000 7f681f3b7740 1 --2- 172.21.15.27:0/3393004762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d2531b0 0x55f52d5fe7d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.516+0000 7f681f3b7740 1 -- 172.21.15.27:0/3393004762 >> 172.21.15.27:0/3393004762 conn(0x55f52d247300 msgr2=0x55f52d5ef280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.517+0000 7f681f3b7740 1 -- 172.21.15.27:0/3393004762 shutdown_connections 2024-04-25T13:46:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.517+0000 7f681f3b7740 1 -- 172.21.15.27:0/3393004762 wait complete. 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.517+0000 7f681f3b7740 1 -- start start 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f681f3b7740 1 Processor -- start 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d22a2f0 0x55f52d605e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d6063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2531b0 0x55f52d60d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f681f3b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f52d603fd0 con 0x55f52d2324d0 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f681f3b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f52d4e0b60 con 0x55f52d2531b0 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f681f3b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f52d604150 con 0x55f52d22a2f0 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f68162a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d22a2f0 0x55f52d605e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f681f3b7740 10 client.0 _ll_register_callbacks cb 0x55f52d5052b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f68162a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d22a2f0 0x55f52d605e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:49192/0 (socket says 172.21.15.27:49192) 2024-04-25T13:46:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f6815aa3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d6063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f68162a4640 1 -- 172.21.15.27:0/4056771405 learned_addr learned my addr 172.21.15.27:0/4056771405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f6816aa5640 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2531b0 0x55f52d60d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f68162a4640 1 -- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2531b0 msgr2=0x55f52d60d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f68162a4640 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2531b0 0x55f52d60d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f68162a4640 1 -- 172.21.15.27:0/4056771405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 msgr2=0x55f52d6063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f68162a4640 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d6063a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f68162a4640 1 -- 172.21.15.27:0/4056771405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f52d26b210 con 0x55f52d22a2f0 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f6815aa3640 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d6063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.518+0000 7f6816aa5640 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2531b0 0x55f52d60d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f68162a4640 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d22a2f0 0x55f52d605e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f680c031e60 tx=0x7f680c031e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2851386fb75c0b62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f67f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f67f9ffb640 1 -- 172.21.15.27:0/4056771405 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c035030 con 0x55f52d22a2f0 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f67f9ffb640 1 -- 172.21.15.27:0/4056771405 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f680c003800 con 0x55f52d22a2f0 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f681f3b7740 1 -- 172.21.15.27:0/4056771405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f680c031790 con 0x55f52d22a2f0 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f67f9ffb640 1 -- 172.21.15.27:0/4056771405 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f680c062070 con 0x55f52d22a2f0 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f681f3b7740 10 client.26296 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f681f3b7740 1 -- 172.21.15.27:0/4056771405 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f52d337620 con 0x55f52d22a2f0 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f67f9ffb640 1 -- 172.21.15.27:0/4056771405 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f680c031790 con 0x55f52d22a2f0 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.519+0000 7f67f9ffb640 1 -- 172.21.15.27:0/4056771405 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f680c03a040 con 0x55f52d22a2f0 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f681f3b7740 10 client.26296 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f681f3b7740 10 client.26296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f681f3b7740 20 client.26296 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f681f3b7740 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f52d211d80 0x55f52d613540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f681f3b7740 4 client.26296 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f681f3b7740 1 -- 172.21.15.27:0/4056771405 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f52d567070 con 0x55f52d211d80 2024-04-25T13:46:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f6815aa3640 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f52d211d80 0x55f52d613540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f67f9ffb640 10 client.26296 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.520+0000 7f6815aa3640 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f52d211d80 0x55f52d613540 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e946d5892dc29605 server_cookie=af846aefff027a9a in_seq=0 out_seq=0 2024-04-25T13:46:52.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.521+0000 7f67f9ffb640 1 -- 172.21.15.27:0/4056771405 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f52d567070 con 0x55f52d211d80 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.521+0000 7f67f9ffb640 10 client.26296 handle_command_reply: tid=0 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.521+0000 7f681f3b7740 1 client.26296 shutdown 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f681f3b7740 1 -- 172.21.15.27:0/4056771405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d22a2f0 msgr2=0x55f52d605e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f681f3b7740 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d22a2f0 0x55f52d605e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f680c031e60 tx=0x7f680c031e90 comp rx=0 tx=0).stop 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f681f3b7740 1 -- 172.21.15.27:0/4056771405 shutdown_connections 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f681f3b7740 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f52d211d80 0x55f52d613540 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f681f3b7740 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2531b0 0x55f52d60d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f67f9ffb640 0 client.26296 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f681f3b7740 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d6063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f681f3b7740 1 --2- 172.21.15.27:0/4056771405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d22a2f0 0x55f52d605e60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.522+0000 7f681f3b7740 1 -- 172.21.15.27:0/4056771405 >> 172.21.15.27:0/4056771405 conn(0x55f52d247300 msgr2=0x55f52d5ee250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.523+0000 7f681f3b7740 1 -- 172.21.15.27:0/4056771405 shutdown_connections 2024-04-25T13:46:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.523+0000 7f681f3b7740 1 -- 172.21.15.27:0/4056771405 wait complete. 2024-04-25T13:46:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.523+0000 7f681f3b7740 20 client.26296 trim_cache size 0 max 16384 2024-04-25T13:46:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.546+0000 7f681f3b7740 1 Processor -- start 2024-04-25T13:46:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.546+0000 7f681f3b7740 1 -- start start 2024-04-25T13:46:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.546+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d606450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.546+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d60d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.546+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d5edf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.546+0000 7f681f3b7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f68101a70d0 con 0x55f52d2324d0 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.546+0000 7f681f3b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68101a9930 con 0x55f52d22a2f0 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.546+0000 7f681f3b7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f68101a7720 con 0x55f52d211d80 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f6815aa3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d60d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d606450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f6816aa5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d5edf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d606450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:49204/0 (socket says 172.21.15.27:49204) 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f6815aa3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d60d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:50804/0 (socket says 172.21.15.27:50804) 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 -- 172.21.15.27:0/981265755 learned_addr learned my addr 172.21.15.27:0/981265755 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 -- 172.21.15.27:0/981265755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 msgr2=0x55f52d60d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 --2- 172.21.15.27:0/981265755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d60d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 -- 172.21.15.27:0/981265755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 msgr2=0x55f52d5edf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 --2- 172.21.15.27:0/981265755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d5edf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 -- 172.21.15.27:0/981265755 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f52d606fa0 con 0x55f52d211d80 2024-04-25T13:46:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f6816aa5640 1 --2- 172.21.15.27:0/981265755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d5edf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f68162a4640 1 --2- 172.21.15.27:0/981265755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d606450 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f680c004de0 tx=0x7f680c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52fad1f1d5f0e1ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f67fbfff640 1 -- 172.21.15.27:0/981265755 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c06c070 con 0x55f52d211d80 2024-04-25T13:46:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f67fbfff640 1 -- 172.21.15.27:0/981265755 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f680c004fd0 con 0x55f52d211d80 2024-04-25T13:46:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.547+0000 7f67fbfff640 1 -- 172.21.15.27:0/981265755 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c035070 con 0x55f52d211d80 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.548+0000 7f681f3b7740 1 -- 172.21.15.27:0/981265755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 msgr2=0x55f52d606450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.549+0000 7f681f3b7740 1 --2- 172.21.15.27:0/981265755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d606450 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f680c004de0 tx=0x7f680c009800 comp rx=0 tx=0).stop 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.549+0000 7f681f3b7740 1 -- 172.21.15.27:0/981265755 shutdown_connections 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.549+0000 7f681f3b7740 1 --2- 172.21.15.27:0/981265755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d2324d0 0x55f52d5edf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.549+0000 7f681f3b7740 1 --2- 172.21.15.27:0/981265755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d22a2f0 0x55f52d60d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.549+0000 7f681f3b7740 1 --2- 172.21.15.27:0/981265755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d606450 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.549+0000 7f681f3b7740 1 -- 172.21.15.27:0/981265755 >> 172.21.15.27:0/981265755 conn(0x55f52d247300 msgr2=0x55f52d5ed820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.550+0000 7f681f3b7740 1 -- 172.21.15.27:0/981265755 shutdown_connections 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.550+0000 7f681f3b7740 1 -- 172.21.15.27:0/981265755 wait complete. 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.550+0000 7f681f3b7740 1 -- start start 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f681f3b7740 1 Processor -- start 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d60d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d22a2f0 0x55f52d609d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f681f3b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2324d0 0x55f52d60a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f6815aa3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d22a2f0 0x55f52d609d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f68162a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d60d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f68162a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d60d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:49210/0 (socket says 172.21.15.27:49210) 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f6815aa3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d22a2f0 0x55f52d609d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51368/0 (socket says 172.21.15.27:51368) 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f68162a4640 1 -- 172.21.15.27:0/976067604 learned_addr learned my addr 172.21.15.27:0/976067604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f68101a7df0 con 0x55f52d22a2f0 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68101a9260 con 0x55f52d2324d0 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f68101a84c0 con 0x55f52d211d80 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.551+0000 7f6816aa5640 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2324d0 0x55f52d60a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f681f3b7740 10 client.0 _ll_register_callbacks cb 0x7f6808009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f68162a4640 1 -- 172.21.15.27:0/976067604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2324d0 msgr2=0x55f52d60a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f68162a4640 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2324d0 0x55f52d60a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f68162a4640 1 -- 172.21.15.27:0/976067604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d22a2f0 msgr2=0x55f52d609d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f68162a4640 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d22a2f0 0x55f52d609d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f68162a4640 1 -- 172.21.15.27:0/976067604 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f52d6796e0 con 0x55f52d211d80 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f680c004fd0 con 0x55f52d211d80 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f68162a4640 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d60d0e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f680c0054e0 tx=0x7f680c003ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bc6c45b3df0f9b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f67f9ffb640 10 client.26302 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f67f9ffb640 1 -- 172.21.15.27:0/976067604 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c076050 con 0x55f52d211d80 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f67f9ffb640 1 -- 172.21.15.27:0/976067604 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f680c0048d0 con 0x55f52d211d80 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f67f9ffb640 1 -- 172.21.15.27:0/976067604 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f680c06c070 con 0x55f52d211d80 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f681f3b7740 10 client.26302 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.552+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x55f52d67a6b0 con 0x55f52d211d80 2024-04-25T13:46:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.553+0000 7f67f9ffb640 1 -- 172.21.15.27:0/976067604 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f680c03d4d0 con 0x55f52d211d80 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.553+0000 7f681f3b7740 10 client.26302 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.553+0000 7f681f3b7740 10 client.26302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.553+0000 7f681f3b7740 20 client.26302 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.553+0000 7f681f3b7740 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f52d2531b0 0x55f52d67cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.553+0000 7f681f3b7740 4 client.26302 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.553+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f52d560ee0 con 0x55f52d2531b0 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.554+0000 7f6815aa3640 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f52d2531b0 0x55f52d67cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.555+0000 7f67f9ffb640 10 client.26302 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.555+0000 7f6815aa3640 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f52d2531b0 0x55f52d67cf00 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb10b49cbf43ab85 server_cookie=f42137e311371d2b in_seq=0 out_seq=0 2024-04-25T13:46:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.574+0000 7f67f9ffb640 1 -- 172.21.15.27:0/976067604 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f52d560ee0 con 0x55f52d2531b0 2024-04-25T13:46:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.574+0000 7f67f9ffb640 10 client.26302 handle_command_reply: tid=0 2024-04-25T13:46:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.575+0000 7f681f3b7740 1 client.26302 shutdown 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.575+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 msgr2=0x55f52d60d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.575+0000 7f681f3b7740 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d60d0e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f680c0054e0 tx=0x7f680c003ed0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.575+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 shutdown_connections 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.575+0000 7f681f3b7740 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f52d2531b0 0x55f52d67cf00 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.575+0000 7f681f3b7740 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f52d2324d0 0x55f52d60a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f67f9ffb640 0 client.26302 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f681f3b7740 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f52d22a2f0 0x55f52d609d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f681f3b7740 1 --2- 172.21.15.27:0/976067604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f52d211d80 0x55f52d60d0e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 >> 172.21.15.27:0/976067604 conn(0x55f52d247300 msgr2=0x55f52d5ee250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 shutdown_connections 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f681f3b7740 1 -- 172.21.15.27:0/976067604 wait complete. 2024-04-25T13:46:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f681f3b7740 20 client.26302 trim_cache size 0 max 16384 2024-04-25T13:46:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:46:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:46:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:46:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:46:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f6818508640 1 -- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67fc0afbe0 msgr2=0x7f67fc0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f6818508640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67fc0afbe0 0x7f67fc0b20f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f680c0057c0 tx=0x7f680c005d50 comp rx=0 tx=0).stop 2024-04-25T13:46:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f6818508640 1 -- 172.21.15.27:0/921496027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6810103c50 msgr2=0x7f681019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.576+0000 7f6818508640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6810103c50 0x7f681019eff0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f680000e9f0 tx=0x7f680000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.577+0000 7f6818508640 1 -- 172.21.15.27:0/921496027 shutdown_connections 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.577+0000 7f6818508640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f67fc0afbe0 0x7f67fc0b20f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.577+0000 7f6818508640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6810108950 0x7f68101a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.577+0000 7f6818508640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6810103c50 0x7f681019eff0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.577+0000 7f6818508640 1 --2- 172.21.15.27:0/921496027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6810102a10 0x7f6810075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.577+0000 7f6818508640 1 -- 172.21.15.27:0/921496027 >> 172.21.15.27:0/921496027 conn(0x7f68100fdc00 msgr2=0x7f6810104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.577+0000 7f6818508640 1 -- 172.21.15.27:0/921496027 shutdown_connections 2024-04-25T13:46:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:46:52.577+0000 7f6818508640 1 -- 172.21.15.27:0/921496027 wait complete. 2024-04-25T13:46:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 345.0 s, 2452.2 tps, lat 40.527 ms stddev 45.745 2024-04-25T13:46:53.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:46:53.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:46:53.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.959+0000 7efcdc04c640 1 -- 172.21.15.27:0/541310261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4103ae0 msgr2=0x7efcd41047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:53.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.959+0000 7efcdc04c640 1 --2- 172.21.15.27:0/541310261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4103ae0 0x7efcd41047e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efccc009900 tx=0x7efccc02a410 comp rx=0 tx=0).stop 2024-04-25T13:46:53.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.959+0000 7efcdc04c640 1 -- 172.21.15.27:0/541310261 shutdown_connections 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.959+0000 7efcdc04c640 1 --2- 172.21.15.27:0/541310261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4109a20 0x7efcd410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.959+0000 7efcdc04c640 1 --2- 172.21.15.27:0/541310261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd4104d20 0x7efcd4105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.959+0000 7efcdc04c640 1 --2- 172.21.15.27:0/541310261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4103ae0 0x7efcd41047e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.959+0000 7efcdc04c640 1 -- 172.21.15.27:0/541310261 >> 172.21.15.27:0/541310261 conn(0x7efcd40fecd0 msgr2=0x7efcd4101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.959+0000 7efcdc04c640 1 -- 172.21.15.27:0/541310261 shutdown_connections 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 -- 172.21.15.27:0/541310261 wait complete. 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 Processor -- start 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 -- start start 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd4103ae0 0x7efcd4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4104d20 0x7efcd41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4109a20 0x7efcd419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410d430 con 0x7efcd4103ae0 2024-04-25T13:46:53.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410d2b0 con 0x7efcd4104d20 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.960+0000 7efcdc04c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efcd410d5b0 con 0x7efcd4109a20 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcd95e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4104d20 0x7efcd41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcd95e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4104d20 0x7efcd41985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33200/0 (socket says 172.21.15.27:33200) 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcda5e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4109a20 0x7efcd419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcd95e7640 1 -- 172.21.15.27:0/746279221 learned_addr learned my addr 172.21.15.27:0/746279221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcd9de8640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd4103ae0 0x7efcd4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcda5e9640 1 -- 172.21.15.27:0/746279221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4104d20 msgr2=0x7efcd41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcda5e9640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4104d20 0x7efcd41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcda5e9640 1 -- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd4103ae0 msgr2=0x7efcd4198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcda5e9640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd4103ae0 0x7efcd4198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcda5e9640 1 -- 172.21.15.27:0/746279221 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd419fb40 con 0x7efcd4109a20 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcd95e7640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4104d20 0x7efcd41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcda5e9640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4109a20 0x7efcd419f4d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009fd0 tx=0x7efcc400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.961+0000 7efcd2ffd640 1 -- 172.21.15.27:0/746279221 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc400ce20 con 0x7efcd4109a20 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.962+0000 7efcd2ffd640 1 -- 172.21.15.27:0/746279221 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcc4005020 con 0x7efcd4109a20 2024-04-25T13:46:53.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.962+0000 7efcdc04c640 1 -- 172.21.15.27:0/746279221 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd419fe30 con 0x7efcd4109a20 2024-04-25T13:46:53.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.962+0000 7efcd2ffd640 1 -- 172.21.15.27:0/746279221 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4010690 con 0x7efcd4109a20 2024-04-25T13:46:53.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.962+0000 7efcdc04c640 1 -- 172.21.15.27:0/746279221 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd41a0240 con 0x7efcd4109a20 2024-04-25T13:46:53.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.964+0000 7efcd2ffd640 1 -- 172.21.15.27:0/746279221 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7efcc40054d0 con 0x7efcd4109a20 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.964+0000 7efce2fd9740 1 Processor -- start 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efce2fd9740 1 -- start start 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949aa7390 0x55b949e48a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a7e4d0 0x55b949e48fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 0x55b949e49c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efce2fd9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b949dadd80 con 0x55b949a866b0 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efce2fd9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b949d678b0 con 0x55b949a7e4d0 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efce2fd9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b949da9d40 con 0x55b949aa7390 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949aa7390 0x55b949e48a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949aa7390 0x55b949e48a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47412/0 (socket says 172.21.15.27:47412) 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 -- 172.21.15.27:0/2495465978 learned_addr learned my addr 172.21.15.27:0/2495465978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd9de8640 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a7e4d0 0x55b949e48fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 0x55b949e49c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd2ffd640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efcbc0afaf0 0x7efcbc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 -- 172.21.15.27:0/2495465978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a7e4d0 msgr2=0x55b949e48fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a7e4d0 0x55b949e48fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 -- 172.21.15.27:0/2495465978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 msgr2=0x55b949e49c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 0x55b949e49c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcda5e9640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efcbc0afaf0 0x7efcbc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd95e7640 1 -- 172.21.15.27:0/2495465978 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b949d38a40 con 0x55b949aa7390 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.965+0000 7efcd2ffd640 1 -- 172.21.15.27:0/746279221 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efcc40df2b0 con 0x7efcd4109a20 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.966+0000 7efcd95e7640 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949aa7390 0x55b949e48a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009ec0 tx=0x7efcc8011040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c8a60f78f9f4e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.966+0000 7efcbbfff640 1 -- 172.21.15.27:0/2495465978 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc801db80 con 0x55b949aa7390 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.966+0000 7efcbbfff640 1 -- 172.21.15.27:0/2495465978 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcc801dd20 con 0x55b949aa7390 2024-04-25T13:46:53.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.966+0000 7efcbbfff640 1 -- 172.21.15.27:0/2495465978 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc800e660 con 0x55b949aa7390 2024-04-25T13:46:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 -- 172.21.15.27:0/2495465978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949aa7390 msgr2=0x55b949e48a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efcda5e9640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efcbc0afaf0 0x7efcbc0b2000 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efcc400c760 tx=0x7efcc401b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949aa7390 0x55b949e48a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009ec0 tx=0x7efcc8011040 comp rx=0 tx=0).stop 2024-04-25T13:46:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 -- 172.21.15.27:0/2495465978 shutdown_connections 2024-04-25T13:46:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 0x55b949e49c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a7e4d0 0x55b949e48fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2495465978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949aa7390 0x55b949e48a60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 -- 172.21.15.27:0/2495465978 >> 172.21.15.27:0/2495465978 conn(0x55b949a9b4e0 msgr2=0x55b949e44270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 -- 172.21.15.27:0/2495465978 shutdown_connections 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 -- 172.21.15.27:0/2495465978 wait complete. 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 -- start start 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.967+0000 7efce2fd9740 1 Processor -- start 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a7e4d0 0x55b949e529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 0x55b949e52f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949aa7390 0x55b949e59e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efce2fd9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b949e4a780 con 0x55b949a866b0 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efce2fd9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b949d34d10 con 0x55b949aa7390 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efce2fd9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b949e4a900 con 0x55b949a7e4d0 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efce2fd9740 10 client.0 _ll_register_callbacks cb 0x55b949d61210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a7e4d0 0x55b949e529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd9de8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 0x55b949e52f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a7e4d0 0x55b949e529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47414/0 (socket says 172.21.15.27:47414) 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 -- 172.21.15.27:0/2670929669 learned_addr learned my addr 172.21.15.27:0/2670929669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949aa7390 0x55b949e59e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 -- 172.21.15.27:0/2670929669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949aa7390 msgr2=0x55b949e59e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949aa7390 0x55b949e59e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 -- 172.21.15.27:0/2670929669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 msgr2=0x55b949e52f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 0x55b949e52f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 -- 172.21.15.27:0/2670929669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b949b897d0 con 0x55b949a7e4d0 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcd95e7640 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a7e4d0 0x55b949e529d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efcc801e010 tx=0x7efcc8009770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7c59e087f02be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.968+0000 7efcb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.969+0000 7efcb97fa640 1 -- 172.21.15.27:0/2670929669 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc8005520 con 0x55b949a7e4d0 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.969+0000 7efcb97fa640 1 -- 172.21.15.27:0/2670929669 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efcc80046c0 con 0x55b949a7e4d0 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.969+0000 7efce2fd9740 1 -- 172.21.15.27:0/2670929669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcc801dd20 con 0x55b949a7e4d0 2024-04-25T13:46:53.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.969+0000 7efcb97fa640 1 -- 172.21.15.27:0/2670929669 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7efcc80048c0 con 0x55b949a7e4d0 2024-04-25T13:46:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.969+0000 7efce2fd9740 10 client.26314 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.969+0000 7efce2fd9740 1 -- 172.21.15.27:0/2670929669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b949abf3f0 con 0x55b949a7e4d0 2024-04-25T13:46:53.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.969+0000 7efcb97fa640 1 -- 172.21.15.27:0/2670929669 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcc801dd20 con 0x55b949a7e4d0 2024-04-25T13:46:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.972+0000 7efcb97fa640 1 -- 172.21.15.27:0/2670929669 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7efcc801b070 con 0x55b949a7e4d0 2024-04-25T13:46:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.972+0000 7efce2fd9740 10 client.26314 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:53.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.972+0000 7efce2fd9740 10 client.26314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.972+0000 7efce2fd9740 20 client.26314 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.972+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b949a65f60 0x55b949e5fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.972+0000 7efce2fd9740 4 client.26314 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.972+0000 7efce2fd9740 1 -- 172.21.15.27:0/2670929669 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b949dbb990 con 0x55b949a65f60 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.972+0000 7efcd9de8640 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b949a65f60 0x55b949e5fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.973+0000 7efcb97fa640 10 client.26314 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.973+0000 7efcd9de8640 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b949a65f60 0x55b949e5fff0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ae03fdf85823253 server_cookie=2f668ffb2f53c379 in_seq=0 out_seq=0 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.973+0000 7efcb97fa640 1 -- 172.21.15.27:0/2670929669 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b949dbb990 con 0x55b949a65f60 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.973+0000 7efcb97fa640 10 client.26314 handle_command_reply: tid=0 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.973+0000 7efce2fd9740 1 client.26314 shutdown 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.973+0000 7efce2fd9740 1 -- 172.21.15.27:0/2670929669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a7e4d0 msgr2=0x55b949e529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.973+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a7e4d0 0x55b949e529d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efcc801e010 tx=0x7efcc8009770 comp rx=0 tx=0).stop 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 1 -- 172.21.15.27:0/2670929669 shutdown_connections 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b949a65f60 0x55b949e5fff0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949aa7390 0x55b949e59e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efcb97fa640 0 client.26314 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a866b0 0x55b949e52f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 1 --2- 172.21.15.27:0/2670929669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a7e4d0 0x55b949e529d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 1 -- 172.21.15.27:0/2670929669 >> 172.21.15.27:0/2670929669 conn(0x55b949a9b4e0 msgr2=0x55b949e47180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 1 -- 172.21.15.27:0/2670929669 shutdown_connections 2024-04-25T13:46:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 1 -- 172.21.15.27:0/2670929669 wait complete. 2024-04-25T13:46:53.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.974+0000 7efce2fd9740 20 client.26314 trim_cache size 0 max 16384 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efce2fd9740 1 Processor -- start 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efce2fd9740 1 -- start start 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e43ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 0x55b949e525a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efce2fd9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efcd41a6f00 con 0x55b949a7e4d0 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efce2fd9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efcd41a9780 con 0x55b949a65f60 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efce2fd9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efcd41a7570 con 0x55b949a866b0 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efcd9de8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efcd9de8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e441e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60478/0 (socket says 172.21.15.27:60478) 2024-04-25T13:46:54.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efcd95e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e43ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efcd9de8640 1 -- 172.21.15.27:0/3326377506 learned_addr learned my addr 172.21.15.27:0/3326377506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efcd95e7640 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 0x55b949e525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:53.999+0000 7efcd9de8640 1 -- 172.21.15.27:0/3326377506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 msgr2=0x55b949e525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.000+0000 7efcd9de8640 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 0x55b949e525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.000+0000 7efcd9de8640 1 -- 172.21.15.27:0/3326377506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 msgr2=0x55b949e43ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.000+0000 7efcd9de8640 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e43ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.000+0000 7efcd9de8640 1 -- 172.21.15.27:0/3326377506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b949a7f0f0 con 0x55b949a7e4d0 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.000+0000 7efcd95e7640 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e43ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.000+0000 7efcd9de8640 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e441e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efccc005ab0 tx=0x7efccc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425d0273c7251a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.001+0000 7efcbb7fe640 1 -- 172.21.15.27:0/3326377506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efccc035040 con 0x55b949a7e4d0 2024-04-25T13:46:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.001+0000 7efcbb7fe640 1 -- 172.21.15.27:0/3326377506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efccc04b070 con 0x55b949a7e4d0 2024-04-25T13:46:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.001+0000 7efcbb7fe640 1 -- 172.21.15.27:0/3326377506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efccc0035f0 con 0x55b949a7e4d0 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 -- 172.21.15.27:0/3326377506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 msgr2=0x55b949e441e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e441e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efccc005ab0 tx=0x7efccc000bc0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 -- 172.21.15.27:0/3326377506 shutdown_connections 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 0x55b949e525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e441e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 --2- 172.21.15.27:0/3326377506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e43ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 -- 172.21.15.27:0/3326377506 >> 172.21.15.27:0/3326377506 conn(0x55b949a9b4e0 msgr2=0x55b949e51020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 -- 172.21.15.27:0/3326377506 shutdown_connections 2024-04-25T13:46:54.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.004+0000 7efce2fd9740 1 -- 172.21.15.27:0/3326377506 wait complete. 2024-04-25T13:46:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 1 -- start start 2024-04-25T13:46:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 1 Processor -- start 2024-04-25T13:46:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e566f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 0x55b949ec6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efcd41a7c40 con 0x55b949a7e4d0 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efcd41a90b0 con 0x55b949a65f60 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efcd41a8310 con 0x55b949a866b0 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efce2fd9740 10 client.0 _ll_register_callbacks cb 0x7efcc8009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.005+0000 7efcd9de8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd95e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd9de8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e566f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60480/0 (socket says 172.21.15.27:60480) 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd9de8640 1 -- 172.21.15.27:0/1877338403 learned_addr learned my addr 172.21.15.27:0/1877338403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd95e7640 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 0x55b949ec6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd95e7640 1 -- 172.21.15.27:0/1877338403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 msgr2=0x55b949ec6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd95e7640 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 0x55b949ec6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd95e7640 1 -- 172.21.15.27:0/1877338403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 msgr2=0x55b949e566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd95e7640 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd95e7640 1 -- 172.21.15.27:0/1877338403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b949d6e4b0 con 0x55b949a65f60 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcd95e7640 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e561b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efcc800feb0 tx=0x7efcc8009590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc2a70091b9d140b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcb97fa640 1 -- 172.21.15.27:0/1877338403 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc803e040 con 0x55b949a65f60 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcb97fa640 1 -- 172.21.15.27:0/1877338403 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efcc8004a60 con 0x55b949a65f60 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efce2fd9740 1 -- 172.21.15.27:0/1877338403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efccc04b070 con 0x55b949a65f60 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efcb97fa640 1 -- 172.21.15.27:0/1877338403 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7efcc8004c60 con 0x55b949a65f60 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efce2fd9740 10 client.26276 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.006+0000 7efce2fd9740 1 -- 172.21.15.27:0/1877338403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b949ec7460 con 0x55b949a65f60 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efcb97fa640 1 -- 172.21.15.27:0/1877338403 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efccc04b070 con 0x55b949a65f60 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efcb97fa640 1 -- 172.21.15.27:0/1877338403 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7efcc8044040 con 0x55b949a65f60 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efce2fd9740 10 client.26276 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efce2fd9740 10 client.26276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efce2fd9740 20 client.26276 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efce2fd9740 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b949aa7390 0x55b949ec9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efce2fd9740 4 client.26276 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efce2fd9740 1 -- 172.21.15.27:0/1877338403 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b949db5800 con 0x55b949aa7390 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efcd9de8640 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b949aa7390 0x55b949ec9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efcb97fa640 10 client.26276 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.007+0000 7efcd9de8640 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b949aa7390 0x55b949ec9df0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9d115007432c38f server_cookie=7b7f27aa440f7db5 in_seq=0 out_seq=0 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efcb97fa640 1 -- 172.21.15.27:0/1877338403 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b949db5800 con 0x55b949aa7390 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efcb97fa640 10 client.26276 handle_command_reply: tid=0 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 client.26276 shutdown 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 -- 172.21.15.27:0/1877338403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 msgr2=0x55b949e561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e561b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efcc800feb0 tx=0x7efcc8009590 comp rx=0 tx=0).stop 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 -- 172.21.15.27:0/1877338403 shutdown_connections 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b949aa7390 0x55b949ec9df0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b949a866b0 0x55b949ec6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efcb97fa640 0 client.26276 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b949a7e4d0 0x55b949e566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 --2- 172.21.15.27:0/1877338403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b949a65f60 0x55b949e561b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 -- 172.21.15.27:0/1877338403 >> 172.21.15.27:0/1877338403 conn(0x55b949a9b4e0 msgr2=0x55b949e43c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 -- 172.21.15.27:0/1877338403 shutdown_connections 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 1 -- 172.21.15.27:0/1877338403 wait complete. 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.008+0000 7efce2fd9740 20 client.26276 trim_cache size 0 max 16384 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:46:54.012 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:46:54.013 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:46:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 -- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efcbc0afaf0 msgr2=0x7efcbc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efcbc0afaf0 0x7efcbc0b2000 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efcc400c760 tx=0x7efcc401b2c0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 -- 172.21.15.27:0/746279221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4109a20 msgr2=0x7efcd419f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4109a20 0x7efcd419f4d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009fd0 tx=0x7efcc400ef10 comp rx=0 tx=0).stop 2024-04-25T13:46:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 -- 172.21.15.27:0/746279221 shutdown_connections 2024-04-25T13:46:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efcbc0afaf0 0x7efcbc0b2000 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efcd4109a20 0x7efcd419f4d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcd4104d20 0x7efcd41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 --2- 172.21.15.27:0/746279221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcd4103ae0 0x7efcd4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 -- 172.21.15.27:0/746279221 >> 172.21.15.27:0/746279221 conn(0x7efcd40fecd0 msgr2=0x7efcd4105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 -- 172.21.15.27:0/746279221 shutdown_connections 2024-04-25T13:46:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.009+0000 7efcdc04c640 1 -- 172.21.15.27:0/746279221 wait complete. 2024-04-25T13:46:54.024 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:46:54.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:46:54.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:53 smithi027 ceph-mon[29140]: pgmap v529: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 20 MiB/s wr, 795 op/s 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.190+0000 7f693f09f640 1 -- 172.21.15.27:0/1043405842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 msgr2=0x7f693810bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.190+0000 7f693f09f640 1 --2- 172.21.15.27:0/1043405842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 0x7f693810bc60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6930009be0 tx=0x7f693002a6c0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.190+0000 7f693f09f640 1 -- 172.21.15.27:0/1043405842 shutdown_connections 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.190+0000 7f693f09f640 1 --2- 172.21.15.27:0/1043405842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 0x7f693810bc60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.190+0000 7f693f09f640 1 --2- 172.21.15.27:0/1043405842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6938106ea0 0x7f69381092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.190+0000 7f693f09f640 1 --2- 172.21.15.27:0/1043405842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6938101700 0x7f6938101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.190+0000 7f693f09f640 1 -- 172.21.15.27:0/1043405842 >> 172.21.15.27:0/1043405842 conn(0x7f69380fcef0 msgr2=0x7f69380ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.191+0000 7f693f09f640 1 -- 172.21.15.27:0/1043405842 shutdown_connections 2024-04-25T13:46:54.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.191+0000 7f693f09f640 1 -- 172.21.15.27:0/1043405842 wait complete. 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693f09f640 1 Processor -- start 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693f09f640 1 -- start start 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693f09f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6938101700 0x7f6938104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693f09f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6938106ea0 0x7f6938102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693f09f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 0x7f69381032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693f09f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f693810f5e0 con 0x7f6938109800 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693f09f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f693810f460 con 0x7f6938101700 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693f09f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f693810f760 con 0x7f6938106ea0 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693d63c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 0x7f69381032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f6937fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6938106ea0 0x7f6938102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693d63c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 0x7f69381032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60492/0 (socket says 172.21.15.27:60492) 2024-04-25T13:46:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693d63c640 1 -- 172.21.15.27:0/4108950747 learned_addr learned my addr 172.21.15.27:0/4108950747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:54.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.192+0000 7f693ce3b640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6938101700 0x7f6938104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.193+0000 7f6937fff640 1 -- 172.21.15.27:0/4108950747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6938101700 msgr2=0x7f6938104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.193+0000 7f6937fff640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6938101700 0x7f6938104700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.193+0000 7f6937fff640 1 -- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 msgr2=0x7f69381032e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.193+0000 7f6937fff640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 0x7f69381032e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.193+0000 7f6937fff640 1 -- 172.21.15.27:0/4108950747 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6938103a10 con 0x7f6938106ea0 2024-04-25T13:46:54.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.193+0000 7f693d63c640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 0x7f69381032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:54.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.193+0000 7f6937fff640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6938106ea0 0x7f6938102da0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6924005fd0 tx=0x7f692400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.194+0000 7f6935ffb640 1 -- 172.21.15.27:0/4108950747 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692400cd00 con 0x7f6938106ea0 2024-04-25T13:46:54.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.194+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69381a4270 con 0x7f6938106ea0 2024-04-25T13:46:54.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.194+0000 7f6935ffb640 1 -- 172.21.15.27:0/4108950747 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69240035f0 con 0x7f6938106ea0 2024-04-25T13:46:54.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.194+0000 7f6935ffb640 1 -- 172.21.15.27:0/4108950747 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924010690 con 0x7f6938106ea0 2024-04-25T13:46:54.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.194+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69381a4760 con 0x7f6938106ea0 2024-04-25T13:46:54.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.195+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69380fea10 con 0x7f6938106ea0 2024-04-25T13:46:54.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.197+0000 7f6935ffb640 1 -- 172.21.15.27:0/4108950747 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6924003aa0 con 0x7f6938106ea0 2024-04-25T13:46:54.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.198+0000 7f6935ffb640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69280afbf0 0x7f69280b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.198+0000 7f6935ffb640 1 -- 172.21.15.27:0/4108950747 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f69240df020 con 0x7f6938106ea0 2024-04-25T13:46:54.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.198+0000 7f693ce3b640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69280afbf0 0x7f69280b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.198+0000 7f693ce3b640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69280afbf0 0x7f69280b2100 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f69381028a0 tx=0x7f692c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.201+0000 7f6935ffb640 1 -- 172.21.15.27:0/4108950747 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6924076ff0 con 0x7f6938106ea0 2024-04-25T13:46:54.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:53 smithi122 ceph-mon[32843]: pgmap v529: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 20 MiB/s wr, 795 op/s 2024-04-25T13:46:54.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:53 smithi045 ceph-mon[32731]: pgmap v529: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 20 MiB/s wr, 795 op/s 2024-04-25T13:46:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.562+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f693802db50 con 0x7f6938106ea0 2024-04-25T13:46:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.564+0000 7f6935ffb640 1 -- 172.21.15.27:0/4108950747 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f6924009510 con 0x7f6938106ea0 2024-04-25T13:46:54.566 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:46:54.566 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:46:54.567 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:46:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69280afbf0 msgr2=0x7f69280b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69280afbf0 0x7f69280b2100 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f69381028a0 tx=0x7f692c005f90 comp rx=0 tx=0).stop 2024-04-25T13:46:54.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6938106ea0 msgr2=0x7f6938102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6938106ea0 0x7f6938102da0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6924005fd0 tx=0x7f692400ee50 comp rx=0 tx=0).stop 2024-04-25T13:46:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 shutdown_connections 2024-04-25T13:46:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69280afbf0 0x7f69280b2100 secure :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f69381028a0 tx=0x7f692c005f90 comp rx=0 tx=0).stop 2024-04-25T13:46:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6938109800 0x7f69381032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6938106ea0 0x7f6938102da0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 --2- 172.21.15.27:0/4108950747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6938101700 0x7f6938104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 >> 172.21.15.27:0/4108950747 conn(0x7f69380fcef0 msgr2=0x7f6938107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.569+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 shutdown_connections 2024-04-25T13:46:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.570+0000 7f693f09f640 1 -- 172.21.15.27:0/4108950747 wait complete. 2024-04-25T13:46:54.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:46:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.751+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1873391933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd60810b1b0 msgr2=0x7fd60810b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.751+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1873391933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd60810b1b0 0x7fd60810b5c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009900 tx=0x7fd5fc02a400 comp rx=0 tx=0).stop 2024-04-25T13:46:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.752+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1873391933 shutdown_connections 2024-04-25T13:46:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.752+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1873391933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd60810feb0 0x7fd6081122d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.752+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1873391933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd60810b1b0 0x7fd60810b5c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.752+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1873391933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd608109fc0 0x7fd60810ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.752+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1873391933 >> 172.21.15.27:0/1873391933 conn(0x7fd608078bb0 msgr2=0x7fd6081079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.752+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1873391933 shutdown_connections 2024-04-25T13:46:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.752+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1873391933 wait complete. 2024-04-25T13:46:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.752+0000 7fd60d9ff640 1 Processor -- start 2024-04-25T13:46:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.753+0000 7fd60d9ff640 1 -- start start 2024-04-25T13:46:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.753+0000 7fd60d9ff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd608109fc0 0x7fd6081a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.753+0000 7fd60d9ff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd60810b1b0 0x7fd6081a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.753+0000 7fd60d9ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd60810feb0 0x7fd6081a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.753+0000 7fd6067fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd60810b1b0 0x7fd6081a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd6067fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd60810b1b0 0x7fd6081a0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47472/0 (socket says 172.21.15.27:47472) 2024-04-25T13:46:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd6067fc640 1 -- 172.21.15.27:0/1761758130 learned_addr learned my addr 172.21.15.27:0/1761758130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:54.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.753+0000 7fd60d9ff640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd608115ae0 con 0x7fd608109fc0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd608115960 con 0x7fd60810feb0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd608115c60 con 0x7fd60810b1b0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd6077fe640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd60810feb0 0x7fd6081a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd6067fc640 1 -- 172.21.15.27:0/1761758130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd60810feb0 msgr2=0x7fd6081a79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd6067fc640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd60810feb0 0x7fd6081a79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd6067fc640 1 -- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd608109fc0 msgr2=0x7fd6081a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd6067fc640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd608109fc0 0x7fd6081a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.754+0000 7fd6067fc640 1 -- 172.21.15.27:0/1761758130 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60806c6f0 con 0x7fd60810b1b0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.755+0000 7fd6067fc640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd60810b1b0 0x7fd6081a0a20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009850 tx=0x7fd5fc003e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.755+0000 7fd60c9fd640 1 -- 172.21.15.27:0/1761758130 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc050070 con 0x7fd60810b1b0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.755+0000 7fd60c9fd640 1 -- 172.21.15.27:0/1761758130 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5fc031d60 con 0x7fd60810b1b0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.755+0000 7fd60c9fd640 1 -- 172.21.15.27:0/1761758130 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc035030 con 0x7fd60810b1b0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.755+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60806c980 con 0x7fd60810b1b0 2024-04-25T13:46:54.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.755+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60806ce20 con 0x7fd60810b1b0 2024-04-25T13:46:54.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.759+0000 7fd60c9fd640 1 -- 172.21.15.27:0/1761758130 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd5fc053070 con 0x7fd60810b1b0 2024-04-25T13:46:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.759+0000 7fd614984740 1 Processor -- start 2024-04-25T13:46:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.760+0000 7fd614984740 1 -- start start 2024-04-25T13:46:54.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd614984740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 0x563c10e0e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd614984740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e0ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd614984740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a4b6b0 0x563c10e0fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd614984740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563c10d72d80 con 0x563c10a6c390 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd614984740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563c10d2ca20 con 0x563c10a434d0 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd614984740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563c10d6ed40 con 0x563c10a4b6b0 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd606ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e0ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd60c9fd640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd5f00afbe0 0x7fd5f00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd606ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e0ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33274/0 (socket says 172.21.15.27:33274) 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd606ffd640 1 -- 172.21.15.27:0/2824392804 learned_addr learned my addr 172.21.15.27:0/2824392804 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd6067fc640 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a4b6b0 0x563c10e0fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.761+0000 7fd6077fe640 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 0x563c10e0e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.762+0000 7fd6067fc640 1 -- 172.21.15.27:0/2824392804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 msgr2=0x563c10e0ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.762+0000 7fd6067fc640 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e0ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.762+0000 7fd6067fc640 1 -- 172.21.15.27:0/2824392804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 msgr2=0x563c10e0e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.762+0000 7fd6067fc640 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 0x563c10e0e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.762+0000 7fd6067fc640 1 -- 172.21.15.27:0/2824392804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c10d37c30 con 0x563c10a4b6b0 2024-04-25T13:46:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.762+0000 7fd60c9fd640 1 -- 172.21.15.27:0/1761758130 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd5fc1106a0 con 0x7fd60810b1b0 2024-04-25T13:46:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.762+0000 7fd6067fc640 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a4b6b0 0x563c10e0fbf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0d5dc0 tx=0x7fd5fc1072a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4aefa4624a9e6f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.763+0000 7fd5edffb640 1 -- 172.21.15.27:0/2824392804 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc05a030 con 0x563c10a4b6b0 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.763+0000 7fd5edffb640 1 -- 172.21.15.27:0/2824392804 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5fc0e12f0 con 0x563c10a4b6b0 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.763+0000 7fd5edffb640 1 -- 172.21.15.27:0/2824392804 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0d8660 con 0x563c10a4b6b0 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.764+0000 7fd614984740 1 -- 172.21.15.27:0/2824392804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a4b6b0 msgr2=0x563c10e0fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.764+0000 7fd614984740 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a4b6b0 0x563c10e0fbf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0d5dc0 tx=0x7fd5fc1072a0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.764+0000 7fd614984740 1 -- 172.21.15.27:0/2824392804 shutdown_connections 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.764+0000 7fd614984740 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a4b6b0 0x563c10e0fbf0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.764+0000 7fd614984740 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e0ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.764+0000 7fd614984740 1 --2- 172.21.15.27:0/2824392804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 0x563c10e0e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.764+0000 7fd614984740 1 -- 172.21.15.27:0/2824392804 >> 172.21.15.27:0/2824392804 conn(0x563c10a604e0 msgr2=0x563c10e0a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.765+0000 7fd6077fe640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd5f00afbe0 0x7fd5f00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.769+0000 7fd614984740 1 -- 172.21.15.27:0/2824392804 shutdown_connections 2024-04-25T13:46:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.774+0000 7fd614984740 1 -- 172.21.15.27:0/2824392804 wait complete. 2024-04-25T13:46:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.774+0000 7fd614984740 1 -- start start 2024-04-25T13:46:54.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.774+0000 7fd614984740 1 Processor -- start 2024-04-25T13:46:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.778+0000 7fd614984740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a434d0 0x563c10e182c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.778+0000 7fd614984740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a4b6b0 0x563c10e18800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.778+0000 7fd614984740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 0x563c10e1f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.778+0000 7fd614984740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563c10e104c0 con 0x563c10a6c390 2024-04-25T13:46:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.778+0000 7fd614984740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563c10cf9dc0 con 0x563c10a4b6b0 2024-04-25T13:46:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.778+0000 7fd614984740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563c10e10640 con 0x563c10a434d0 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.778+0000 7fd614984740 10 client.0 _ll_register_callbacks cb 0x563c10d15600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.779+0000 7fd6067fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a4b6b0 0x563c10e18800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.779+0000 7fd6067fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a4b6b0 0x563c10e18800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33278/0 (socket says 172.21.15.27:33278) 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.779+0000 7fd6067fc640 1 -- 172.21.15.27:0/1770642512 learned_addr learned my addr 172.21.15.27:0/1770642512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.779+0000 7fd6067fc640 1 -- 172.21.15.27:0/1770642512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a434d0 msgr2=0x563c10e182c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.779+0000 7fd6067fc640 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a434d0 0x563c10e182c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.779+0000 7fd6067fc640 1 -- 172.21.15.27:0/1770642512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 msgr2=0x563c10e1f760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.779+0000 7fd6067fc640 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 0x563c10e1f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.779+0000 7fd6067fc640 1 -- 172.21.15.27:0/1770642512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c10a843f0 con 0x563c10a4b6b0 2024-04-25T13:46:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.782+0000 7fd6067fc640 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a4b6b0 0x563c10e18800 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc10e3d0 tx=0x7fd5fc04c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20598172a71da714 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.783+0000 7fd5e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.783+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/1770642512 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc05a070 con 0x563c10a4b6b0 2024-04-25T13:46:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.783+0000 7fd614984740 1 -- 172.21.15.27:0/1770642512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563c10d80990 con 0x563c10a4b6b0 2024-04-25T13:46:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.783+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/1770642512 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd5fc0d5140 con 0x563c10a4b6b0 2024-04-25T13:46:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.787+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/1770642512 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd5fc0d8660 con 0x563c10a4b6b0 2024-04-25T13:46:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.787+0000 7fd614984740 10 client.26294 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.787+0000 7fd614984740 1 -- 172.21.15.27:0/1770642512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563c10b4e7d0 con 0x563c10a4b6b0 2024-04-25T13:46:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.792+0000 7fd6077fe640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd5f00afbe0 0x7fd5f00b20f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x563c10e0fa30 tx=0x7fd600004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.793+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/1770642512 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563c10d80990 con 0x563c10a4b6b0 2024-04-25T13:46:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.794+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/1770642512 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fd5fc04a040 con 0x563c10a4b6b0 2024-04-25T13:46:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.794+0000 7fd614984740 10 client.26294 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.794+0000 7fd614984740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.794+0000 7fd614984740 20 client.26294 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.794+0000 7fd614984740 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563c10a2af60 0x563c10e36c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.794+0000 7fd614984740 4 client.26294 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.794+0000 7fd614984740 1 -- 172.21.15.27:0/1770642512 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c10d7a800 con 0x563c10a2af60 2024-04-25T13:46:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.795+0000 7fd606ffd640 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563c10a2af60 0x563c10e36c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.798+0000 7fd606ffd640 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563c10a2af60 0x563c10e36c20 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f7a297b3c9e1fe server_cookie=b76c71f103aa0c00 in_seq=0 out_seq=0 2024-04-25T13:46:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.798+0000 7fd5e6ffd640 10 client.26294 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.799+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/1770642512 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563c10d7a800 con 0x563c10a2af60 2024-04-25T13:46:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.799+0000 7fd5e6ffd640 10 client.26294 handle_command_reply: tid=0 2024-04-25T13:46:54.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.801+0000 7fd614984740 1 client.26294 shutdown 2024-04-25T13:46:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.803+0000 7fd614984740 1 -- 172.21.15.27:0/1770642512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a4b6b0 msgr2=0x563c10e18800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.803+0000 7fd614984740 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a4b6b0 0x563c10e18800 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc10e3d0 tx=0x7fd5fc04c470 comp rx=0 tx=0).stop 2024-04-25T13:46:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.806+0000 7fd614984740 1 -- 172.21.15.27:0/1770642512 shutdown_connections 2024-04-25T13:46:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.806+0000 7fd614984740 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563c10a2af60 0x563c10e36c20 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.806+0000 7fd614984740 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a6c390 0x563c10e1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.806+0000 7fd614984740 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a4b6b0 0x563c10e18800 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.806+0000 7fd614984740 1 --2- 172.21.15.27:0/1770642512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a434d0 0x563c10e182c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.806+0000 7fd614984740 1 -- 172.21.15.27:0/1770642512 >> 172.21.15.27:0/1770642512 conn(0x563c10a604e0 msgr2=0x563c10e093f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.808+0000 7fd5e6ffd640 0 client.26294 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.809+0000 7fd614984740 1 -- 172.21.15.27:0/1770642512 shutdown_connections 2024-04-25T13:46:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.810+0000 7fd614984740 1 -- 172.21.15.27:0/1770642512 wait complete. 2024-04-25T13:46:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.810+0000 7fd614984740 20 client.26294 trim_cache size 0 max 16384 2024-04-25T13:46:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.834+0000 7fd614984740 1 Processor -- start 2024-04-25T13:46:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.834+0000 7fd614984740 1 -- start start 2024-04-25T13:46:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.834+0000 7fd614984740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e16ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.834+0000 7fd614984740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.834+0000 7fd614984740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e3b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.834+0000 7fd614984740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd608073b00 con 0x563c10a4b6b0 2024-04-25T13:46:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd614984740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd60807c930 con 0x563c10a434d0 2024-04-25T13:46:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd614984740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd6080740c0 con 0x563c10a6c390 2024-04-25T13:46:54.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60520/0 (socket says 172.21.15.27:60520) 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 -- 172.21.15.27:0/988539228 learned_addr learned my addr 172.21.15.27:0/988539228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd606ffd640 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e16ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6077fe640 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e3b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 -- 172.21.15.27:0/988539228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 msgr2=0x563c10e3b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e3b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 -- 172.21.15.27:0/988539228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 msgr2=0x563c10e16ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e16ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 -- 172.21.15.27:0/988539228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c10e13bd0 con 0x563c10a4b6b0 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.835+0000 7fd6067fc640 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1f680 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0a12e0 tx=0x7fd5fc02a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff3cd031e659422 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.836+0000 7fd5ed7fa640 1 -- 172.21.15.27:0/988539228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc004480 con 0x563c10a4b6b0 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.836+0000 7fd5ed7fa640 1 -- 172.21.15.27:0/988539228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5fc0d9250 con 0x563c10a4b6b0 2024-04-25T13:46:54.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.836+0000 7fd5ed7fa640 1 -- 172.21.15.27:0/988539228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0d7db0 con 0x563c10a4b6b0 2024-04-25T13:46:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.836+0000 7fd614984740 1 -- 172.21.15.27:0/988539228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 msgr2=0x563c10e1f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.836+0000 7fd614984740 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1f680 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0a12e0 tx=0x7fd5fc02a400 comp rx=0 tx=0).stop 2024-04-25T13:46:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.837+0000 7fd614984740 1 -- 172.21.15.27:0/988539228 shutdown_connections 2024-04-25T13:46:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.837+0000 7fd614984740 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e3b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.837+0000 7fd614984740 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1f680 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.837+0000 7fd614984740 1 --2- 172.21.15.27:0/988539228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e16ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.837+0000 7fd614984740 1 -- 172.21.15.27:0/988539228 >> 172.21.15.27:0/988539228 conn(0x563c10a604e0 msgr2=0x563c10e168e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.837+0000 7fd614984740 1 -- 172.21.15.27:0/988539228 shutdown_connections 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.837+0000 7fd614984740 1 -- 172.21.15.27:0/988539228 wait complete. 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.837+0000 7fd614984740 1 -- start start 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.838+0000 7fd614984740 1 Processor -- start 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.838+0000 7fd614984740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e1c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.838+0000 7fd614984740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.838+0000 7fd614984740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e9cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.838+0000 7fd614984740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd608074790 con 0x563c10a4b6b0 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.838+0000 7fd614984740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd60807c260 con 0x563c10a434d0 2024-04-25T13:46:54.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.838+0000 7fd614984740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd60807b4c0 con 0x563c10a6c390 2024-04-25T13:46:54.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.838+0000 7fd614984740 10 client.0 _ll_register_callbacks cb 0x7fd5fc0a0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:54.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.840+0000 7fd606ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e1c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.840+0000 7fd606ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e1c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33298/0 (socket says 172.21.15.27:33298) 2024-04-25T13:46:54.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.840+0000 7fd606ffd640 1 -- 172.21.15.27:0/637576955 learned_addr learned my addr 172.21.15.27:0/637576955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.841+0000 7fd6077fe640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e9cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.841+0000 7fd6067fc640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.841+0000 7fd606ffd640 1 -- 172.21.15.27:0/637576955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 msgr2=0x563c10e9cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.841+0000 7fd606ffd640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e9cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.841+0000 7fd606ffd640 1 -- 172.21.15.27:0/637576955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 msgr2=0x563c10e1c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.841+0000 7fd606ffd640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.841+0000 7fd606ffd640 1 -- 172.21.15.27:0/637576955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c10e1f900 con 0x563c10a434d0 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.842+0000 7fd6077fe640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e9cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:46:54.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.842+0000 7fd6067fc640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:46:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.842+0000 7fd606ffd640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e1c400 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x563c10e37380 tx=0x7fd5f8012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=861620d71535d83b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.842+0000 7fd5e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.842+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/637576955 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f80049a0 con 0x563c10a434d0 2024-04-25T13:46:54.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.842+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/637576955 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd5f8004b40 con 0x563c10a434d0 2024-04-25T13:46:54.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.842+0000 7fd614984740 1 -- 172.21.15.27:0/637576955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5fc0d9250 con 0x563c10a434d0 2024-04-25T13:46:54.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.843+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/637576955 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd5f8029050 con 0x563c10a434d0 2024-04-25T13:46:54.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.843+0000 7fd614984740 10 client.26300 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:54.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.843+0000 7fd614984740 1 -- 172.21.15.27:0/637576955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563c10e9e4a0 con 0x563c10a434d0 2024-04-25T13:46:54.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.843+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/637576955 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5fc0d9250 con 0x563c10a434d0 2024-04-25T13:46:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.844+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/637576955 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fd5f801b5d0 con 0x563c10a434d0 2024-04-25T13:46:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.844+0000 7fd614984740 10 client.26300 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:54.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.844+0000 7fd614984740 10 client.26300 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.844+0000 7fd614984740 20 client.26300 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.844+0000 7fd614984740 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563c10a003a0 0x563c10ea0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.845+0000 7fd614984740 4 client.26300 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:46:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.845+0000 7fd614984740 1 -- 172.21.15.27:0/637576955 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c10cf5610 con 0x563c10a003a0 2024-04-25T13:46:54.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.845+0000 7fd6067fc640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563c10a003a0 0x563c10ea0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:54.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.845+0000 7fd6067fc640 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563c10a003a0 0x563c10ea0f10 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9cd2e8a9424157e server_cookie=f838bed9ca1489c4 in_seq=0 out_seq=0 2024-04-25T13:46:54.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.845+0000 7fd5e6ffd640 10 client.26300 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:54.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd5e6ffd640 1 -- 172.21.15.27:0/637576955 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c10cf5610 con 0x563c10a003a0 2024-04-25T13:46:54.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd5e6ffd640 10 client.26300 handle_command_reply: tid=0 2024-04-25T13:46:54.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 client.26300 shutdown 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 -- 172.21.15.27:0/637576955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 msgr2=0x563c10e1c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e1c400 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x563c10e37380 tx=0x7fd5f8012820 comp rx=0 tx=0).stop 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 -- 172.21.15.27:0/637576955 shutdown_connections 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563c10a003a0 0x563c10ea0f10 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563c10a6c390 0x563c10e9cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563c10a4b6b0 0x563c10e1c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd5e6ffd640 0 client.26300 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 --2- 172.21.15.27:0/637576955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c10a434d0 0x563c10e1c400 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 -- 172.21.15.27:0/637576955 >> 172.21.15.27:0/637576955 conn(0x563c10a604e0 msgr2=0x563c10e0cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 -- 172.21.15.27:0/637576955 shutdown_connections 2024-04-25T13:46:54.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.846+0000 7fd614984740 1 -- 172.21.15.27:0/637576955 wait complete. 2024-04-25T13:46:54.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.847+0000 7fd614984740 20 client.26300 trim_cache size 0 max 16384 2024-04-25T13:46:54.849 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.847+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd5f00afbe0 msgr2=0x7fd5f00b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.847+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd5f00afbe0 0x7fd5f00b20f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x563c10e0fa30 tx=0x7fd600004a40 comp rx=0 tx=0).stop 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.847+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd60810b1b0 msgr2=0x7fd6081a0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.847+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd60810b1b0 0x7fd6081a0a20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009850 tx=0x7fd5fc003e60 comp rx=0 tx=0).stop 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.848+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 shutdown_connections 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.848+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd5f00afbe0 0x7fd5f00b20f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.848+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd60810feb0 0x7fd6081a79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.848+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd60810b1b0 0x7fd6081a0a20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.848+0000 7fd60d9ff640 1 --2- 172.21.15.27:0/1761758130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd608109fc0 0x7fd6081a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:54.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.848+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 >> 172.21.15.27:0/1761758130 conn(0x7fd608078bb0 msgr2=0x7fd60810be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.848+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 shutdown_connections 2024-04-25T13:46:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:54.848+0000 7fd60d9ff640 1 -- 172.21.15.27:0/1761758130 wait complete. 2024-04-25T13:46:55.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:55 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/4108950747' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:46:55.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:55 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/4108950747' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:46:55.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:55 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/4108950747' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:46:55.864 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:46:55.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:46:55.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:46:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.078+0000 7fceaa6d8640 1 -- 172.21.15.27:0/19089732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4103c20 msgr2=0x7fcea4104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.078+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/19089732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4103c20 0x7fcea4104030 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009ca0 tx=0x7fce9c02e9d0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.084+0000 7fceaa6d8640 1 -- 172.21.15.27:0/19089732 shutdown_connections 2024-04-25T13:46:56.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.084+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/19089732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea4108920 0x7fcea410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.084+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/19089732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4103c20 0x7fcea4104030 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.084+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/19089732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcea41029e0 0x7fcea41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.084+0000 7fceaa6d8640 1 -- 172.21.15.27:0/19089732 >> 172.21.15.27:0/19089732 conn(0x7fcea40fdbd0 msgr2=0x7fcea4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.087+0000 7fceaa6d8640 1 -- 172.21.15.27:0/19089732 shutdown_connections 2024-04-25T13:46:56.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.087+0000 7fceaa6d8640 1 -- 172.21.15.27:0/19089732 wait complete. 2024-04-25T13:46:56.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.088+0000 7fceaa6d8640 1 Processor -- start 2024-04-25T13:46:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.088+0000 7fceaa6d8640 1 -- start start 2024-04-25T13:46:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fceaa6d8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcea41029e0 0x7fcea4071e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fceaa6d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea4103c20 0x7fcea4072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fceaa6d8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4108920 0x7fcea40732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fceaa6d8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcea4115d60 con 0x7fcea4108920 2024-04-25T13:46:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fceaa6d8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcea4115be0 con 0x7fcea4103c20 2024-04-25T13:46:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fceaa6d8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcea4115ee0 con 0x7fcea41029e0 2024-04-25T13:46:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea4103c20 0x7fcea4072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea4103c20 0x7fcea4072370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33310/0 (socket says 172.21.15.27:33310) 2024-04-25T13:46:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fcea37fe640 1 -- 172.21.15.27:0/3312329500 learned_addr learned my addr 172.21.15.27:0/3312329500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.089+0000 7fcea8c75640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4108920 0x7fcea40732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.090+0000 7fcea8c75640 1 -- 172.21.15.27:0/3312329500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcea41029e0 msgr2=0x7fcea4071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.090+0000 7fcea8c75640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcea41029e0 0x7fcea4071e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.090+0000 7fcea8c75640 1 -- 172.21.15.27:0/3312329500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea4103c20 msgr2=0x7fcea4072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.090+0000 7fcea8c75640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea4103c20 0x7fcea4072370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.090+0000 7fcea8c75640 1 -- 172.21.15.27:0/3312329500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea4073940 con 0x7fcea4108920 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.090+0000 7fcea8c75640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4108920 0x7fcea40732d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fce9800b3e0 tx=0x7fce9800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.091+0000 7fcea17fa640 1 -- 172.21.15.27:0/3312329500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98015020 con 0x7fcea4108920 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.091+0000 7fceaa6d8640 1 -- 172.21.15.27:0/3312329500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea4073c30 con 0x7fcea4108920 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.091+0000 7fceaa6d8640 1 -- 172.21.15.27:0/3312329500 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea4074250 con 0x7fcea4108920 2024-04-25T13:46:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.091+0000 7fcea17fa640 1 -- 172.21.15.27:0/3312329500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce98011ce0 con 0x7fcea4108920 2024-04-25T13:46:56.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.092+0000 7fcea17fa640 1 -- 172.21.15.27:0/3312329500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98010e10 con 0x7fcea4108920 2024-04-25T13:46:56.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.094+0000 7fceb1583740 1 Processor -- start 2024-04-25T13:46:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.096+0000 7fcea17fa640 1 -- 172.21.15.27:0/3312329500 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fce98018020 con 0x7fcea4108920 2024-04-25T13:46:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.096+0000 7fceb1583740 1 -- start start 2024-04-25T13:46:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.096+0000 7fceb1583740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea31410 0x558d3edd24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fceb1583740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3edd2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fceb1583740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3edd9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fceb1583740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558d3ed37b00 con 0x558d3ea10730 2024-04-25T13:46:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fceb1583740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558d3ecd3920 con 0x558d3ea31410 2024-04-25T13:46:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fceb1583740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558d3ed33ac0 con 0x558d3ea08550 2024-04-25T13:46:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea31410 0x558d3edd24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea31410 0x558d3edd24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33322/0 (socket says 172.21.15.27:33322) 2024-04-25T13:46:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fcea37fe640 1 -- 172.21.15.27:0/3987619437 learned_addr learned my addr 172.21.15.27:0/3987619437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.097+0000 7fcea3fff640 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3edd2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea37fe640 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3edd9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea37fe640 1 -- 172.21.15.27:0/3987619437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 msgr2=0x558d3edd2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea37fe640 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3edd2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea37fe640 1 -- 172.21.15.27:0/3987619437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 msgr2=0x558d3edd9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea37fe640 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3edd9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea37fe640 1 -- 172.21.15.27:0/3987619437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d3eb117e0 con 0x558d3ea31410 2024-04-25T13:46:56.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea17fa640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce8c0afc30 0x7fce8c0b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea37fe640 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea31410 0x558d3edd24a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fce9c000f80 tx=0x7fce9c00f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a18789830c9526a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea17fa640 1 -- 172.21.15.27:0/3312329500 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fce980de320 con 0x7fcea4108920 2024-04-25T13:46:56.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.098+0000 7fcea3fff640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce8c0afc30 0x7fce8c0b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.099+0000 7fce8a7fc640 1 -- 172.21.15.27:0/3987619437 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c037070 con 0x558d3ea31410 2024-04-25T13:46:56.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.099+0000 7fce8a7fc640 1 -- 172.21.15.27:0/3987619437 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce9c032d70 con 0x558d3ea31410 2024-04-25T13:46:56.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.099+0000 7fce8a7fc640 1 -- 172.21.15.27:0/3987619437 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c004290 con 0x558d3ea31410 2024-04-25T13:46:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 -- 172.21.15.27:0/3987619437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea31410 msgr2=0x558d3edd24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea31410 0x558d3edd24a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fce9c000f80 tx=0x7fce9c00f040 comp rx=0 tx=0).stop 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fcea3fff640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce8c0afc30 0x7fce8c0b2140 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x558d3edd3880 tx=0x7fce940090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 -- 172.21.15.27:0/3987619437 shutdown_connections 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3edd9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3edd2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 --2- 172.21.15.27:0/3987619437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea31410 0x558d3edd24a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 -- 172.21.15.27:0/3987619437 >> 172.21.15.27:0/3987619437 conn(0x558d3ea25560 msgr2=0x558d3edcde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 -- 172.21.15.27:0/3987619437 shutdown_connections 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.100+0000 7fceb1583740 1 -- 172.21.15.27:0/3987619437 wait complete. 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 1 -- start start 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 1 Processor -- start 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3eddc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea10730 0x558d3eddc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3e9f01f0 0x558d3ede3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x558d3edda360 con 0x558d3ea08550 2024-04-25T13:46:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558d3ecd15d0 con 0x558d3ea10730 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x558d3edda4e0 con 0x558d3e9f01f0 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fceb1583740 10 client.0 _ll_register_callbacks cb 0x558d3ecbf080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea10730 0x558d3eddc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.101+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3eddc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea10730 0x558d3eddc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33334/0 (socket says 172.21.15.27:33334) 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea37fe640 1 -- 172.21.15.27:0/3093921417 learned_addr learned my addr 172.21.15.27:0/3093921417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea8c75640 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3e9f01f0 0x558d3ede3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea37fe640 1 -- 172.21.15.27:0/3093921417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3e9f01f0 msgr2=0x558d3ede3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea37fe640 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3e9f01f0 0x558d3ede3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea37fe640 1 -- 172.21.15.27:0/3093921417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 msgr2=0x558d3eddc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea37fe640 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3eddc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea37fe640 1 -- 172.21.15.27:0/3093921417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d3eb71f70 con 0x558d3ea10730 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.102+0000 7fcea8c75640 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3e9f01f0 0x558d3ede3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:46:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.103+0000 7fcea37fe640 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea10730 0x558d3eddc9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00f010 tx=0x7fce9c005c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d29e1cfbca8cbd60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.103+0000 7fce837fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.104+0000 7fce837fe640 1 -- 172.21.15.27:0/3093921417 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c04e070 con 0x558d3ea10730 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.104+0000 7fce837fe640 1 -- 172.21.15.27:0/3093921417 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fce9c033630 con 0x558d3ea10730 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.104+0000 7fceb1583740 1 -- 172.21.15.27:0/3093921417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce9c032d70 con 0x558d3ea10730 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.104+0000 7fce837fe640 1 -- 172.21.15.27:0/3093921417 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fce9c037050 con 0x558d3ea10730 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.104+0000 7fceb1583740 10 client.26309 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.104+0000 7fceb1583740 1 -- 172.21.15.27:0/3093921417 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x558d3ea09170 con 0x558d3ea10730 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.104+0000 7fce837fe640 1 -- 172.21.15.27:0/3093921417 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce9c032d70 con 0x558d3ea10730 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.105+0000 7fce837fe640 1 -- 172.21.15.27:0/3093921417 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fce9c048020 con 0x558d3ea10730 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.105+0000 7fceb1583740 10 client.26309 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.105+0000 7fceb1583740 10 client.26309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.105+0000 7fceb1583740 20 client.26309 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.105+0000 7fceb1583740 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558d3e9c5630 0x558d3ede99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.105+0000 7fceb1583740 4 client.26309 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.105+0000 7fceb1583740 1 -- 172.21.15.27:0/3093921417 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d3ed45710 con 0x558d3e9c5630 2024-04-25T13:46:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.106+0000 7fcea3fff640 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558d3e9c5630 0x558d3ede99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.106+0000 7fce837fe640 10 client.26309 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.106+0000 7fcea3fff640 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558d3e9c5630 0x558d3ede99e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b47a7dc559c731e8 server_cookie=c5ab34e83f935e4f in_seq=0 out_seq=0 2024-04-25T13:46:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.107+0000 7fce837fe640 1 -- 172.21.15.27:0/3093921417 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558d3ed45710 con 0x558d3e9c5630 2024-04-25T13:46:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.107+0000 7fce837fe640 10 client.26309 handle_command_reply: tid=0 2024-04-25T13:46:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.107+0000 7fceb1583740 1 client.26309 shutdown 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 -- 172.21.15.27:0/3093921417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea10730 msgr2=0x558d3eddc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea10730 0x558d3eddc9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00f010 tx=0x7fce9c005c30 comp rx=0 tx=0).stop 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 -- 172.21.15.27:0/3093921417 shutdown_connections 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558d3e9c5630 0x558d3ede99e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3e9f01f0 0x558d3ede3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ea10730 0x558d3eddc9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 --2- 172.21.15.27:0/3093921417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3eddc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 -- 172.21.15.27:0/3093921417 >> 172.21.15.27:0/3093921417 conn(0x558d3ea25560 msgr2=0x558d3edccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fce837fe640 0 client.26309 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 -- 172.21.15.27:0/3093921417 shutdown_connections 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 1 -- 172.21.15.27:0/3093921417 wait complete. 2024-04-25T13:46:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.108+0000 7fceb1583740 20 client.26309 trim_cache size 0 max 16384 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.137+0000 7fceb1583740 1 Processor -- start 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fceb1583740 1 -- start start 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fceb1583740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3e9f01f0 0x558d3edcdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fceb1583740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3ede38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fceb1583740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea10730 0x558d3ede3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fceb1583740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fce9000c330 con 0x558d3ea08550 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fceb1583740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce9000ebb0 con 0x558d3e9f01f0 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fceb1583740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fce9000c9a0 con 0x558d3ea10730 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3e9f01f0 0x558d3edcdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3e9f01f0 0x558d3edcdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33348/0 (socket says 172.21.15.27:33348) 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fcea3fff640 1 -- 172.21.15.27:0/3536245623 learned_addr learned my addr 172.21.15.27:0/3536245623 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3ede38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fcea8c75640 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea10730 0x558d3ede3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fcea3fff640 1 -- 172.21.15.27:0/3536245623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea10730 msgr2=0x558d3ede3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.138+0000 7fcea3fff640 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea10730 0x558d3ede3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.139+0000 7fcea3fff640 1 -- 172.21.15.27:0/3536245623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 msgr2=0x558d3ede38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.139+0000 7fcea3fff640 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3ede38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.139+0000 7fcea3fff640 1 -- 172.21.15.27:0/3536245623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d3eddb1a0 con 0x558d3e9f01f0 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.139+0000 7fcea37fe640 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3ede38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.139+0000 7fcea3fff640 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3e9f01f0 0x558d3edcdf00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcea41035e0 tx=0x7fce94011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0b9a1265322af38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.139+0000 7fce89ffb640 1 -- 172.21.15.27:0/3536245623 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9402b020 con 0x558d3e9f01f0 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.139+0000 7fce89ffb640 1 -- 172.21.15.27:0/3536245623 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce9401c040 con 0x558d3e9f01f0 2024-04-25T13:46:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.139+0000 7fce89ffb640 1 -- 172.21.15.27:0/3536245623 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce94027ad0 con 0x558d3e9f01f0 2024-04-25T13:46:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.140+0000 7fceb1583740 1 -- 172.21.15.27:0/3536245623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3e9f01f0 msgr2=0x558d3edcdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.140+0000 7fceb1583740 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3e9f01f0 0x558d3edcdf00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcea41035e0 tx=0x7fce94011000 comp rx=0 tx=0).stop 2024-04-25T13:46:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.140+0000 7fceb1583740 1 -- 172.21.15.27:0/3536245623 shutdown_connections 2024-04-25T13:46:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.140+0000 7fceb1583740 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea10730 0x558d3ede3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.140+0000 7fceb1583740 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea08550 0x558d3ede38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.140+0000 7fceb1583740 1 --2- 172.21.15.27:0/3536245623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3e9f01f0 0x558d3edcdf00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.140+0000 7fceb1583740 1 -- 172.21.15.27:0/3536245623 >> 172.21.15.27:0/3536245623 conn(0x558d3ea25560 msgr2=0x558d3edcd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.141+0000 7fceb1583740 1 -- 172.21.15.27:0/3536245623 shutdown_connections 2024-04-25T13:46:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.141+0000 7fceb1583740 1 -- 172.21.15.27:0/3536245623 wait complete. 2024-04-25T13:46:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.141+0000 7fceb1583740 1 -- start start 2024-04-25T13:46:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fceb1583740 1 Processor -- start 2024-04-25T13:46:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fceb1583740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3ede1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fceb1583740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3ede2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fceb1583740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ede2f70 0x558d3ede3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fceb1583740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fce9000d070 con 0x558d3ea10730 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fceb1583740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce9000e4e0 con 0x558d3ede2f70 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fceb1583740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fce9000d740 con 0x558d3ea08550 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fceb1583740 10 client.0 _ll_register_callbacks cb 0x7fce9c032c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3ede2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3ede2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60578/0 (socket says 172.21.15.27:60578) 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.142+0000 7fcea37fe640 1 -- 172.21.15.27:0/559263016 learned_addr learned my addr 172.21.15.27:0/559263016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.143+0000 7fcea3fff640 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3ede1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.143+0000 7fcea3fff640 1 -- 172.21.15.27:0/559263016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ede2f70 msgr2=0x558d3ede3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.143+0000 7fcea3fff640 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ede2f70 0x558d3ede3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.143+0000 7fcea3fff640 1 -- 172.21.15.27:0/559263016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 msgr2=0x558d3ede2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.143+0000 7fcea3fff640 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3ede2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.144+0000 7fcea3fff640 1 -- 172.21.15.27:0/559263016 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d3ede3940 con 0x558d3ea08550 2024-04-25T13:46:56.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.144+0000 7fcea37fe640 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3ede2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.144+0000 7fcea3fff640 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3ede1b50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x558d3ecd3980 tx=0x7fce94027400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1afdbdb2a56bcc15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.144+0000 7fce837fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.144+0000 7fce837fe640 1 -- 172.21.15.27:0/559263016 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9402b040 con 0x558d3ea08550 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.144+0000 7fceb1583740 1 -- 172.21.15.27:0/559263016 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce9401c040 con 0x558d3ea08550 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.145+0000 7fce837fe640 1 -- 172.21.15.27:0/559263016 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fce94015c10 con 0x558d3ea08550 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.145+0000 7fce837fe640 1 -- 172.21.15.27:0/559263016 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fce94040070 con 0x558d3ea08550 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.145+0000 7fceb1583740 10 client.26347 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.145+0000 7fceb1583740 1 -- 172.21.15.27:0/559263016 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x558d3ee521f0 con 0x558d3ea08550 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.145+0000 7fce837fe640 1 -- 172.21.15.27:0/559263016 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce9401c040 con 0x558d3ea08550 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.145+0000 7fce837fe640 1 -- 172.21.15.27:0/559263016 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fce94022430 con 0x558d3ea08550 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.146+0000 7fceb1583740 10 client.26347 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.146+0000 7fceb1583740 10 client.26347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.146+0000 7fceb1583740 20 client.26347 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.146+0000 7fceb1583740 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558d3ee52b70 0x558d3ee54f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.146+0000 7fceb1583740 4 client.26347 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:46:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.146+0000 7fceb1583740 1 -- 172.21.15.27:0/559263016 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558d3ed3f580 con 0x558d3ee52b70 2024-04-25T13:46:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.146+0000 7fcea37fe640 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558d3ee52b70 0x558d3ee54f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.147+0000 7fcea37fe640 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558d3ee52b70 0x558d3ee54f90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aef585f0b68589ac server_cookie=20762649d7bf5257 in_seq=0 out_seq=0 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.147+0000 7fce837fe640 10 client.26347 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.148+0000 7fce837fe640 1 -- 172.21.15.27:0/559263016 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558d3ed3f580 con 0x558d3ee52b70 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.148+0000 7fce837fe640 10 client.26347 handle_command_reply: tid=0 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.148+0000 7fceb1583740 1 client.26347 shutdown 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 -- 172.21.15.27:0/559263016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 msgr2=0x558d3ede1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3ede1b50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x558d3ecd3980 tx=0x7fce94027400 comp rx=0 tx=0).stop 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 -- 172.21.15.27:0/559263016 shutdown_connections 2024-04-25T13:46:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558d3ee52b70 0x558d3ee54f90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d3ede2f70 0x558d3ede3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fce837fe640 0 client.26347 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558d3ea10730 0x558d3ede2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 --2- 172.21.15.27:0/559263016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558d3ea08550 0x558d3ede1b50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 -- 172.21.15.27:0/559263016 >> 172.21.15.27:0/559263016 conn(0x558d3ea25560 msgr2=0x558d3edee060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 -- 172.21.15.27:0/559263016 shutdown_connections 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 1 -- 172.21.15.27:0/559263016 wait complete. 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.149+0000 7fceb1583740 20 client.26347 trim_cache size 0 max 16384 2024-04-25T13:46:56.150 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:46:56.151 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:46:56.151 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "2f25edda-82c7-4d3c-8e81-d222941a24b9", 2024-04-25T13:46:56.151 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:46:56.151 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:46:56.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 -- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce8c0afc30 msgr2=0x7fce8c0b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce8c0afc30 0x7fce8c0b2140 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x558d3edd3880 tx=0x7fce940090e0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 -- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4108920 msgr2=0x7fcea40732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4108920 0x7fcea40732d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fce9800b3e0 tx=0x7fce9800b8a0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 -- 172.21.15.27:0/3312329500 shutdown_connections 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fce8c0afc30 0x7fce8c0b2140 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea4108920 0x7fcea40732d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea4103c20 0x7fcea4072370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 --2- 172.21.15.27:0/3312329500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcea41029e0 0x7fcea4071e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 -- 172.21.15.27:0/3312329500 >> 172.21.15.27:0/3312329500 conn(0x7fcea40fdbd0 msgr2=0x7fcea4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 -- 172.21.15.27:0/3312329500 shutdown_connections 2024-04-25T13:46:56.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.151+0000 7fceaa6d8640 1 -- 172.21.15.27:0/3312329500 wait complete. 2024-04-25T13:46:56.167 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:56 smithi027 ceph-mon[29140]: pgmap v530: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 709 op/s 2024-04-25T13:46:56.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:46:56.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:46:56.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:56 smithi122 ceph-mon[32843]: pgmap v530: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 709 op/s 2024-04-25T13:46:56.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:56 smithi045 ceph-mon[32731]: pgmap v530: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 709 op/s 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 -- 172.21.15.27:0/1330813442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65810bc30 msgr2=0x7fb65810e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/1330813442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65810bc30 0x7fb65810e050 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb650009ca0 tx=0x7fb65002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 -- 172.21.15.27:0/1330813442 shutdown_connections 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/1330813442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb65810bc30 0x7fb65810e050 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/1330813442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb658106f30 0x7fb658107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/1330813442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb658105cf0 0x7fb6581069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 -- 172.21.15.27:0/1330813442 >> 172.21.15.27:0/1330813442 conn(0x7fb658100f00 msgr2=0x7fb658103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 -- 172.21.15.27:0/1330813442 shutdown_connections 2024-04-25T13:46:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.357+0000 7fb65f8d8640 1 -- 172.21.15.27:0/1330813442 wait complete. 2024-04-25T13:46:56.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.358+0000 7fb65f8d8640 1 Processor -- start 2024-04-25T13:46:56.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.358+0000 7fb65f8d8640 1 -- start start 2024-04-25T13:46:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65f8d8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb658105cf0 0x7fb65819c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65d674640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb658105cf0 0x7fb65819c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65d674640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb658105cf0 0x7fb65819c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47598/0 (socket says 172.21.15.27:47598) 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65d674640 1 -- 172.21.15.27:0/451862680 learned_addr learned my addr 172.21.15.27:0/451862680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb658106f30 0x7fb65819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65810bc30 0x7fb6581a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb658111800 con 0x7fb658106f30 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb658111680 con 0x7fb65810bc30 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb658111980 con 0x7fb658105cf0 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.359+0000 7fb65de75640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65810bc30 0x7fb6581a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb65ce73640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb658106f30 0x7fb65819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb65d674640 1 -- 172.21.15.27:0/451862680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65810bc30 msgr2=0x7fb6581a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb65d674640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65810bc30 0x7fb6581a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb65d674640 1 -- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb658106f30 msgr2=0x7fb65819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb65d674640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb658106f30 0x7fb65819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb65d674640 1 -- 172.21.15.27:0/451862680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6581a3ee0 con 0x7fb658105cf0 2024-04-25T13:46:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb65d674640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb658105cf0 0x7fb65819c3a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb65400e9f0 tx=0x7fb65400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb64e7fc640 1 -- 172.21.15.27:0/451862680 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65400cc40 con 0x7fb658105cf0 2024-04-25T13:46:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb64e7fc640 1 -- 172.21.15.27:0/451862680 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb65400cde0 con 0x7fb658105cf0 2024-04-25T13:46:56.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.360+0000 7fb64e7fc640 1 -- 172.21.15.27:0/451862680 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654010430 con 0x7fb658105cf0 2024-04-25T13:46:56.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.361+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6581a41d0 con 0x7fb658105cf0 2024-04-25T13:46:56.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.361+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6581a4670 con 0x7fb658105cf0 2024-04-25T13:46:56.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.365+0000 7fb64e7fc640 1 -- 172.21.15.27:0/451862680 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb654004000 con 0x7fb658105cf0 2024-04-25T13:46:56.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb666786740 1 Processor -- start 2024-04-25T13:46:56.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb666786740 1 -- start start 2024-04-25T13:46:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb666786740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12de390 0x5581c167fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb666786740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12b54d0 0x5581c167ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb666786740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 0x5581c1680c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb666786740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5581c15e4d80 con 0x5581c12bd6b0 2024-04-25T13:46:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb64e7fc640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb6440afbb0 0x7fb6440b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb666786740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5581c159e8b0 con 0x5581c12b54d0 2024-04-25T13:46:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.366+0000 7fb666786740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5581c15e0d40 con 0x5581c12de390 2024-04-25T13:46:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb64e7fc640 1 -- 172.21.15.27:0/451862680 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb6540def20 con 0x7fb658105cf0 2024-04-25T13:46:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65de75640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12de390 0x5581c167fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65de75640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12de390 0x5581c167fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47600/0 (socket says 172.21.15.27:47600) 2024-04-25T13:46:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65de75640 1 -- 172.21.15.27:0/2384860809 learned_addr learned my addr 172.21.15.27:0/2384860809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65ce73640 1 --2- 172.21.15.27:0/2384860809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 0x5581c1680c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65de75640 1 -- 172.21.15.27:0/2384860809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12b54d0 msgr2=0x5581c167ffe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:46:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65de75640 1 --2- 172.21.15.27:0/2384860809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12b54d0 0x5581c167ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65de75640 1 -- 172.21.15.27:0/2384860809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 msgr2=0x5581c1680c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65de75640 1 --2- 172.21.15.27:0/2384860809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 0x5581c1680c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.367+0000 7fb65de75640 1 -- 172.21.15.27:0/2384860809 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c156fa40 con 0x5581c12de390 2024-04-25T13:46:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.368+0000 7fb65de75640 1 --2- 172.21.15.27:0/2384860809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12de390 0x5581c167fa60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb650009900 tx=0x7fb650004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8418b0b6d3d032 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.368+0000 7fb65de75640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb6440afbb0 0x7fb6440b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.368+0000 7fb6437fe640 1 -- 172.21.15.27:0/2384860809 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650031450 con 0x5581c12de390 2024-04-25T13:46:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.368+0000 7fb6437fe640 1 -- 172.21.15.27:0/2384860809 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb650036070 con 0x5581c12de390 2024-04-25T13:46:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.368+0000 7fb6437fe640 1 -- 172.21.15.27:0/2384860809 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650004710 con 0x5581c12de390 2024-04-25T13:46:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.369+0000 7fb65de75640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb6440afbb0 0x7fb6440b20c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb650046000 tx=0x7fb650004990 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 -- 172.21.15.27:0/2384860809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12de390 msgr2=0x5581c167fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 --2- 172.21.15.27:0/2384860809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12de390 0x5581c167fa60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb650009900 tx=0x7fb650004ab0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 -- 172.21.15.27:0/2384860809 shutdown_connections 2024-04-25T13:46:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 --2- 172.21.15.27:0/2384860809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 0x5581c1680c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 --2- 172.21.15.27:0/2384860809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12b54d0 0x5581c167ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 --2- 172.21.15.27:0/2384860809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12de390 0x5581c167fa60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 -- 172.21.15.27:0/2384860809 >> 172.21.15.27:0/2384860809 conn(0x5581c12d24e0 msgr2=0x5581c167b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 -- 172.21.15.27:0/2384860809 shutdown_connections 2024-04-25T13:46:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 -- 172.21.15.27:0/2384860809 wait complete. 2024-04-25T13:46:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.370+0000 7fb666786740 1 -- start start 2024-04-25T13:46:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.371+0000 7fb666786740 1 Processor -- start 2024-04-25T13:46:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.371+0000 7fb666786740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12b54d0 0x5581c16899d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.371+0000 7fb666786740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 0x5581c1689f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.371+0000 7fb666786740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12de390 0x5581c1690e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.371+0000 7fb666786740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5581c1681720 con 0x5581c12bd6b0 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.371+0000 7fb666786740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5581c156bd10 con 0x5581c12de390 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.371+0000 7fb666786740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5581c16818a0 con 0x5581c12b54d0 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb666786740 10 client.0 _ll_register_callbacks cb 0x5581c1598210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65d674640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12b54d0 0x5581c16899d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65d674640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12b54d0 0x5581c16899d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47616/0 (socket says 172.21.15.27:47616) 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65d674640 1 -- 172.21.15.27:0/851000726 learned_addr learned my addr 172.21.15.27:0/851000726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65ce73640 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 0x5581c1689f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65d674640 1 -- 172.21.15.27:0/851000726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12de390 msgr2=0x5581c1690e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65d674640 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12de390 0x5581c1690e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65d674640 1 -- 172.21.15.27:0/851000726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 msgr2=0x5581c1689f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65d674640 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 0x5581c1689f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.372+0000 7fb65d674640 1 -- 172.21.15.27:0/851000726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c13c07d0 con 0x5581c12b54d0 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.373+0000 7fb65d674640 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12b54d0 0x5581c16899d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb6540a7360 tx=0x7fb6540a6c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=570f004257797a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.373+0000 7fb640ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.373+0000 7fb640ff9640 1 -- 172.21.15.27:0/851000726 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6540c0020 con 0x5581c12b54d0 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.373+0000 7fb666786740 1 -- 172.21.15.27:0/851000726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb650036070 con 0x5581c12b54d0 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.373+0000 7fb640ff9640 1 -- 172.21.15.27:0/851000726 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb6540042f0 con 0x5581c12b54d0 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb640ff9640 1 -- 172.21.15.27:0/851000726 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb6540c8050 con 0x5581c12b54d0 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb666786740 10 client.26359 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb666786740 1 -- 172.21.15.27:0/851000726 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5581c12f63f0 con 0x5581c12b54d0 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb640ff9640 1 -- 172.21.15.27:0/851000726 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6540a73d0 con 0x5581c12b54d0 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb640ff9640 1 -- 172.21.15.27:0/851000726 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb6540a1b60 con 0x5581c12b54d0 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb666786740 10 client.26359 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb666786740 10 client.26359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb666786740 20 client.26359 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb666786740 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5581c129cf60 0x5581c16972c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb666786740 4 client.26359 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:46:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb666786740 1 -- 172.21.15.27:0/851000726 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb650036070 con 0x5581c129cf60 2024-04-25T13:46:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.374+0000 7fb65ce73640 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5581c129cf60 0x5581c16972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.375+0000 7fb640ff9640 10 client.26359 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.375+0000 7fb65ce73640 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5581c129cf60 0x5581c16972c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b33867b62b62e1a2 server_cookie=da9349e88d6a0fb in_seq=0 out_seq=0 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.376+0000 7fb640ff9640 1 -- 172.21.15.27:0/851000726 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fb650036070 con 0x5581c129cf60 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.376+0000 7fb640ff9640 10 client.26359 handle_command_reply: tid=0 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.376+0000 7fb666786740 1 client.26359 shutdown 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.376+0000 7fb666786740 1 -- 172.21.15.27:0/851000726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12b54d0 msgr2=0x5581c16899d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.377+0000 7fb666786740 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12b54d0 0x5581c16899d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb6540a7360 tx=0x7fb6540a6c00 comp rx=0 tx=0).stop 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.377+0000 7fb666786740 1 -- 172.21.15.27:0/851000726 shutdown_connections 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.377+0000 7fb666786740 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5581c129cf60 0x5581c16972c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.377+0000 7fb666786740 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12de390 0x5581c1690e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.377+0000 7fb666786740 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12bd6b0 0x5581c1689f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.377+0000 7fb666786740 1 --2- 172.21.15.27:0/851000726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12b54d0 0x5581c16899d0 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb6540a7360 tx=0x7fb6540a6c00 comp rx=0 tx=0).stop 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.377+0000 7fb666786740 1 -- 172.21.15.27:0/851000726 >> 172.21.15.27:0/851000726 conn(0x5581c12d24e0 msgr2=0x5581c167a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.377+0000 7fb640ff9640 0 client.26359 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:56.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.378+0000 7fb666786740 1 -- 172.21.15.27:0/851000726 shutdown_connections 2024-04-25T13:46:56.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.378+0000 7fb666786740 1 -- 172.21.15.27:0/851000726 wait complete. 2024-04-25T13:46:56.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.378+0000 7fb666786740 20 client.26359 trim_cache size 0 max 16384 2024-04-25T13:46:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.409+0000 7fb666786740 1 Processor -- start 2024-04-25T13:46:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.409+0000 7fb666786740 1 -- start start 2024-04-25T13:46:56.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb666786740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12bd6b0 0x5581c1684da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb666786740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 0x5581c1690e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb666786740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12723a0 0x5581c1691340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb666786740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ab440 con 0x5581c12de390 2024-04-25T13:46:56.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb666786740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581adcc0 con 0x5581c12723a0 2024-04-25T13:46:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb666786740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581abab0 con 0x5581c12bd6b0 2024-04-25T13:46:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb65d674640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12bd6b0 0x5581c1684da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb65d674640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12bd6b0 0x5581c1684da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47632/0 (socket says 172.21.15.27:47632) 2024-04-25T13:46:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.410+0000 7fb65d674640 1 -- 172.21.15.27:0/384319441 learned_addr learned my addr 172.21.15.27:0/384319441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb65de75640 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12723a0 0x5581c1691340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb65ce73640 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 0x5581c1690e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb65de75640 1 -- 172.21.15.27:0/384319441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12bd6b0 msgr2=0x5581c1684da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb65de75640 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12bd6b0 0x5581c1684da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb65de75640 1 -- 172.21.15.27:0/384319441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 msgr2=0x5581c1690e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb65de75640 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 0x5581c1690e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb65de75640 1 -- 172.21.15.27:0/384319441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c12b60f0 con 0x5581c12723a0 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb65de75640 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12723a0 0x5581c1691340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb650006010 tx=0x7fb65004f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a739006001adf478 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb642ffd640 1 -- 172.21.15.27:0/384319441 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650005bd0 con 0x5581c12723a0 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb642ffd640 1 -- 172.21.15.27:0/384319441 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb650036070 con 0x5581c12723a0 2024-04-25T13:46:56.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.411+0000 7fb642ffd640 1 -- 172.21.15.27:0/384319441 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650004040 con 0x5581c12723a0 2024-04-25T13:46:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 -- 172.21.15.27:0/384319441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12723a0 msgr2=0x5581c1691340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12723a0 0x5581c1691340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb650006010 tx=0x7fb65004f000 comp rx=0 tx=0).stop 2024-04-25T13:46:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 -- 172.21.15.27:0/384319441 shutdown_connections 2024-04-25T13:46:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12723a0 0x5581c1691340 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 0x5581c1690e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 --2- 172.21.15.27:0/384319441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12bd6b0 0x5581c1684da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 -- 172.21.15.27:0/384319441 >> 172.21.15.27:0/384319441 conn(0x5581c12d24e0 msgr2=0x5581c167eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 -- 172.21.15.27:0/384319441 shutdown_connections 2024-04-25T13:46:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.413+0000 7fb666786740 1 -- 172.21.15.27:0/384319441 wait complete. 2024-04-25T13:46:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 1 -- start start 2024-04-25T13:46:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 1 Processor -- start 2024-04-25T13:46:56.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12723a0 0x5581c168ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12bd6b0 0x5581c168ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 0x5581c168fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ac180 con 0x5581c12de390 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ad5f0 con 0x5581c12bd6b0 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ac850 con 0x5581c12723a0 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.414+0000 7fb666786740 10 client.0 _ll_register_callbacks cb 0x7fb6540046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12bd6b0 0x5581c168ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12bd6b0 0x5581c168ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33394/0 (socket says 172.21.15.27:33394) 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65de75640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 0x5581c168fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 -- 172.21.15.27:0/3144228641 learned_addr learned my addr 172.21.15.27:0/3144228641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65d674640 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12723a0 0x5581c168ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 -- 172.21.15.27:0/3144228641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12723a0 msgr2=0x5581c168ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12723a0 0x5581c168ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 -- 172.21.15.27:0/3144228641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 msgr2=0x5581c168fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 0x5581c168fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 -- 172.21.15.27:0/3144228641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c15a54b0 con 0x5581c12bd6b0 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb65ce73640 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12bd6b0 0x5581c168ef50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb64800bde0 tx=0x7fb64801da00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afcb4179d3451cc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb640ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb640ff9640 1 -- 172.21.15.27:0/3144228641 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648011de0 con 0x5581c12bd6b0 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb640ff9640 1 -- 172.21.15.27:0/3144228641 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb648004030 con 0x5581c12bd6b0 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.415+0000 7fb666786740 1 -- 172.21.15.27:0/3144228641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb650036070 con 0x5581c12bd6b0 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb640ff9640 1 -- 172.21.15.27:0/3144228641 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb64802c070 con 0x5581c12bd6b0 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb666786740 10 client.26324 fetch_fsmap learned FSMap version 19 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb666786740 1 -- 172.21.15.27:0/3144228641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5581c1691590 con 0x5581c12bd6b0 2024-04-25T13:46:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb640ff9640 1 -- 172.21.15.27:0/3144228641 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb650036070 con 0x5581c12bd6b0 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb640ff9640 1 -- 172.21.15.27:0/3144228641 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb648025410 con 0x5581c12bd6b0 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb666786740 10 client.26324 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb666786740 10 client.26324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb666786740 20 client.26324 populate_metadata read hostname 'smithi027' 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb666786740 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5581c1708250 0x5581c170a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb666786740 4 client.26324 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb666786740 1 -- 172.21.15.27:0/3144228641 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581c15f2990 con 0x5581c1708250 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.416+0000 7fb65d674640 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5581c1708250 0x5581c170a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb640ff9640 10 client.26324 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb65d674640 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5581c1708250 0x5581c170a670 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8922a036dc00cd1c server_cookie=8bc3f095aac2669f in_seq=0 out_seq=0 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb640ff9640 1 -- 172.21.15.27:0/3144228641 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5581c15f2990 con 0x5581c1708250 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb640ff9640 10 client.26324 handle_command_reply: tid=0 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 client.26324 shutdown 2024-04-25T13:46:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 -- 172.21.15.27:0/3144228641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12bd6b0 msgr2=0x5581c168ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12bd6b0 0x5581c168ef50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb64800bde0 tx=0x7fb64801da00 comp rx=0 tx=0).stop 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 -- 172.21.15.27:0/3144228641 shutdown_connections 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5581c1708250 0x5581c170a670 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5581c12de390 0x5581c168fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581c12bd6b0 0x5581c168ef50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb640ff9640 0 client.26324 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 --2- 172.21.15.27:0/3144228641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5581c12723a0 0x5581c168ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 -- 172.21.15.27:0/3144228641 >> 172.21.15.27:0/3144228641 conn(0x5581c12d24e0 msgr2=0x5581c167e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 -- 172.21.15.27:0/3144228641 shutdown_connections 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 1 -- 172.21.15.27:0/3144228641 wait complete. 2024-04-25T13:46:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.417+0000 7fb666786740 20 client.26324 trim_cache size 0 max 16384 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout: "2f25edda-82c7-4d3c-8e81-d222941a24b9": { 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "2f25edda-82c7-4d3c-8e81-d222941a24b9", 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:46:56.422 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:46:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb6440afbb0 msgr2=0x7fb6440b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb6440afbb0 0x7fb6440b20c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb650046000 tx=0x7fb650004990 comp rx=0 tx=0).stop 2024-04-25T13:46:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb658105cf0 msgr2=0x7fb65819c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:46:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb658105cf0 0x7fb65819c3a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb65400e9f0 tx=0x7fb65400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 shutdown_connections 2024-04-25T13:46:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb6440afbb0 0x7fb6440b20c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65810bc30 0x7fb6581a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb658106f30 0x7fb65819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 --2- 172.21.15.27:0/451862680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb658105cf0 0x7fb65819c3a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:46:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.418+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 >> 172.21.15.27:0/451862680 conn(0x7fb658100f00 msgr2=0x7fb658107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:46:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.419+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 shutdown_connections 2024-04-25T13:46:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:46:56.419+0000 7fb65f8d8640 1 -- 172.21.15.27:0/451862680 wait complete. 2024-04-25T13:46:56.435 INFO:tasks.cephfs.filesystem:scrub status for tag:2f25edda-82c7-4d3c-8e81-d222941a24b9 - {'path': '/', 'tag': '2f25edda-82c7-4d3c-8e81-d222941a24b9', 'options': 'recursive,force'} 2024-04-25T13:46:57.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:57 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:46:57.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:57 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:57.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:57 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:46:57.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:57 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:57.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:57 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:46:57.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:57 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:57.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:57 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:46:57.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:57 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:57.403 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:57 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:46:57.403 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:57 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:57.403 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:57 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:46:57.403 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:57 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:46:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 350.0 s, 2352.6 tps, lat 39.409 ms stddev 57.795 2024-04-25T13:46:58.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:58 smithi122 ceph-mon[32843]: pgmap v531: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 17 MiB/s wr, 832 op/s 2024-04-25T13:46:58.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:58 smithi045 ceph-mon[32731]: pgmap v531: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 17 MiB/s wr, 832 op/s 2024-04-25T13:46:58.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:58 smithi027 ceph-mon[29140]: pgmap v531: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 17 MiB/s wr, 832 op/s 2024-04-25T13:46:59.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:46:59 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:46:59.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:46:59 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:46:59.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:46:59 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:47:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:00 smithi122 ceph-mon[32843]: pgmap v532: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 MiB/s wr, 688 op/s 2024-04-25T13:47:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:00 smithi045 ceph-mon[32731]: pgmap v532: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 MiB/s wr, 688 op/s 2024-04-25T13:47:00.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:00 smithi027 ceph-mon[29140]: pgmap v532: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 MiB/s wr, 688 op/s 2024-04-25T13:47:02.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:02 smithi122 ceph-mon[32843]: pgmap v533: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 28 MiB/s wr, 1.22k op/s 2024-04-25T13:47:02.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:02 smithi045 ceph-mon[32731]: pgmap v533: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 28 MiB/s wr, 1.22k op/s 2024-04-25T13:47:02.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:02 smithi027 ceph-mon[29140]: pgmap v533: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 28 MiB/s wr, 1.22k op/s 2024-04-25T13:47:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 355.0 s, 2788.6 tps, lat 38.503 ms stddev 57.398 2024-04-25T13:47:04.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:04 smithi122 ceph-mon[32843]: pgmap v534: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 43 MiB/s wr, 1.69k op/s 2024-04-25T13:47:04.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:04 smithi045 ceph-mon[32731]: pgmap v534: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 43 MiB/s wr, 1.69k op/s 2024-04-25T13:47:04.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:04 smithi027 ceph-mon[29140]: pgmap v534: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 43 MiB/s wr, 1.69k op/s 2024-04-25T13:47:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 -- 172.21.15.27:0/1135567102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31810b9f0 msgr2=0x7fb31810df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 --2- 172.21.15.27:0/1135567102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31810b9f0 0x7fb31810df00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb310009be0 tx=0x7fb31002a6c0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 -- 172.21.15.27:0/1135567102 shutdown_connections 2024-04-25T13:47:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 --2- 172.21.15.27:0/1135567102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31810b9f0 0x7fb31810df00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 --2- 172.21.15.27:0/1135567102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb318108ec0 0x7fb31810b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 --2- 172.21.15.27:0/1135567102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318104010 0x7fb318104420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 -- 172.21.15.27:0/1135567102 >> 172.21.15.27:0/1135567102 conn(0x7fb3180fd390 msgr2=0x7fb3180ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 -- 172.21.15.27:0/1135567102 shutdown_connections 2024-04-25T13:47:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.723+0000 7fb31f30d640 1 -- 172.21.15.27:0/1135567102 wait complete. 2024-04-25T13:47:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.724+0000 7fb31f30d640 1 Processor -- start 2024-04-25T13:47:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31f30d640 1 -- start start 2024-04-25T13:47:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31f30d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318104010 0x7fb31819c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31f30d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb318108ec0 0x7fb31819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31f30d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31810b9f0 0x7fb3181a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31f30d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb318111870 con 0x7fb318104010 2024-04-25T13:47:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31f30d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181116f0 con 0x7fb318108ec0 2024-04-25T13:47:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31f30d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181119f0 con 0x7fb31810b9f0 2024-04-25T13:47:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31c8a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb318108ec0 0x7fb31819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31c8a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb318108ec0 0x7fb31819c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43222/0 (socket says 172.21.15.27:43222) 2024-04-25T13:47:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31c8a8640 1 -- 172.21.15.27:0/2486616354 learned_addr learned my addr 172.21.15.27:0/2486616354 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.725+0000 7fb31d8aa640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31810b9f0 0x7fb3181a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.726+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318104010 0x7fb31819c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.726+0000 7fb31c8a8640 1 -- 172.21.15.27:0/2486616354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31810b9f0 msgr2=0x7fb3181a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.726+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31810b9f0 0x7fb3181a3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.726+0000 7fb31c8a8640 1 -- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318104010 msgr2=0x7fb31819c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.726+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318104010 0x7fb31819c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.726+0000 7fb31c8a8640 1 -- 172.21.15.27:0/2486616354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3181a3ef0 con 0x7fb318108ec0 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.726+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb318108ec0 0x7fb31819c8f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb3080142d0 tx=0x7fb308014790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.727+0000 7fb30e7fc640 1 -- 172.21.15.27:0/2486616354 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30801f070 con 0x7fb318108ec0 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.727+0000 7fb30e7fc640 1 -- 172.21.15.27:0/2486616354 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb308005020 con 0x7fb318108ec0 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.727+0000 7fb30e7fc640 1 -- 172.21.15.27:0/2486616354 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308004af0 con 0x7fb318108ec0 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.727+0000 7fb31f30d640 1 -- 172.21.15.27:0/2486616354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb318104a80 con 0x7fb318108ec0 2024-04-25T13:47:05.727 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.727+0000 7fb31f30d640 1 -- 172.21.15.27:0/2486616354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb318104f80 con 0x7fb318108ec0 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.730+0000 7fb3261c3740 1 Processor -- start 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb3261c3740 1 -- start start 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b38970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55638476f2f0 0x556384b38ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b3fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb3261c3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556384a9e460 con 0x55638476f2f0 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb3261c3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556384a58590 con 0x5563847774d0 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb3261c3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556384a9a420 con 0x5563847981b0 2024-04-25T13:47:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb30e7fc640 1 -- 172.21.15.27:0/2486616354 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb3080054b0 con 0x7fb318108ec0 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb31d0a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55638476f2f0 0x556384b38ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb31c8a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b3fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb31d0a9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55638476f2f0 0x556384b38ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43794/0 (socket says 172.21.15.27:43794) 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb31d0a9640 1 -- 172.21.15.27:0/1937720085 learned_addr learned my addr 172.21.15.27:0/1937720085 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.731+0000 7fb31d8aa640 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b38970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb31d0a9640 1 -- 172.21.15.27:0/1937720085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 msgr2=0x556384b38970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b38970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb31d0a9640 1 -- 172.21.15.27:0/1937720085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 msgr2=0x556384b3fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b3fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb31d0a9640 1 -- 172.21.15.27:0/1937720085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556384a650d0 con 0x55638476f2f0 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb31d8aa640 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b38970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b3fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb30e7fc640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3040afb60 0x7fb3040b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55638476f2f0 0x556384b38ef0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb314009b30 tx=0x7fb314010370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66aab17a862aae77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb30e7fc640 1 -- 172.21.15.27:0/2486616354 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb3080e5fc0 con 0x7fb318108ec0 2024-04-25T13:47:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb3037fe640 1 -- 172.21.15.27:0/1937720085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314010d40 con 0x55638476f2f0 2024-04-25T13:47:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.732+0000 7fb3037fe640 1 -- 172.21.15.27:0/1937720085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3140054d0 con 0x55638476f2f0 2024-04-25T13:47:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.733+0000 7fb3037fe640 1 -- 172.21.15.27:0/1937720085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3140126b0 con 0x55638476f2f0 2024-04-25T13:47:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.733+0000 7fb31d8aa640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3040afb60 0x7fb3040b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.733+0000 7fb31d8aa640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3040afb60 0x7fb3040b2070 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb310006010 tx=0x7fb310030970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.733+0000 7fb3261c3740 1 -- 172.21.15.27:0/1937720085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55638476f2f0 msgr2=0x556384b38ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.733+0000 7fb3261c3740 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55638476f2f0 0x556384b38ef0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb314009b30 tx=0x7fb314010370 comp rx=0 tx=0).stop 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.734+0000 7fb3261c3740 1 -- 172.21.15.27:0/1937720085 shutdown_connections 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.734+0000 7fb3261c3740 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b3fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.734+0000 7fb3261c3740 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55638476f2f0 0x556384b38ef0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.734+0000 7fb3261c3740 1 --2- 172.21.15.27:0/1937720085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b38970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.734+0000 7fb3261c3740 1 -- 172.21.15.27:0/1937720085 >> 172.21.15.27:0/1937720085 conn(0x55638478c300 msgr2=0x556384b34300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.734+0000 7fb3261c3740 1 -- 172.21.15.27:0/1937720085 shutdown_connections 2024-04-25T13:47:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.736+0000 7fb3261c3740 1 -- 172.21.15.27:0/1937720085 wait complete. 2024-04-25T13:47:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.736+0000 7fb3261c3740 1 -- start start 2024-04-25T13:47:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.737+0000 7fb3261c3740 1 Processor -- start 2024-04-25T13:47:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.737+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.737+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847774d0 0x556384b42be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.737+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b49ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.737+0000 7fb3261c3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556384b40840 con 0x5563847774d0 2024-04-25T13:47:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb3261c3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556384a25b60 con 0x55638476f2f0 2024-04-25T13:47:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31c8a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847774d0 0x556384b42be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31c8a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847774d0 0x556384b42be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43806/0 (socket says 172.21.15.27:43806) 2024-04-25T13:47:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31c8a8640 1 -- 172.21.15.27:0/4044562557 learned_addr learned my addr 172.21.15.27:0/4044562557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31d8aa640 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b49ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556384b409c0 con 0x5563847981b0 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb3261c3740 10 client.0 _ll_register_callbacks cb 0x556384a4a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31d0a9640 1 -- 172.21.15.27:0/4044562557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 msgr2=0x556384b49ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b49ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31d0a9640 1 -- 172.21.15.27:0/4044562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847774d0 msgr2=0x556384b42be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847774d0 0x556384b42be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31d0a9640 1 -- 172.21.15.27:0/4044562557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563847b0210 con 0x55638476f2f0 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847774d0 0x556384b42be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b426a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x556384b39d50 tx=0x7fb314010d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3be7f1f581dd5236 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb300ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.738+0000 7fb300ff9640 1 -- 172.21.15.27:0/4044562557 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31402a060 con 0x55638476f2f0 2024-04-25T13:47:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb300ff9640 1 -- 172.21.15.27:0/4044562557 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb314012450 con 0x55638476f2f0 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3140054d0 con 0x55638476f2f0 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb300ff9640 1 -- 172.21.15.27:0/4044562557 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb3140126b0 con 0x55638476f2f0 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 10 client.26333 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55638487c620 con 0x55638476f2f0 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb300ff9640 1 -- 172.21.15.27:0/4044562557 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3140054d0 con 0x55638476f2f0 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb300ff9640 1 -- 172.21.15.27:0/4044562557 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb31400f040 con 0x55638476f2f0 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 10 client.26333 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 10 client.26333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 20 client.26333 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556384756d80 0x556384b4fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 4 client.26333 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556384aac070 con 0x556384756d80 2024-04-25T13:47:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.739+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556384756d80 0x556384b4fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.740+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556384756d80 0x556384b4fff0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d7a458e1722267 server_cookie=863cede17b597cec in_seq=0 out_seq=0 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.740+0000 7fb300ff9640 10 client.26333 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.741+0000 7fb300ff9640 1 -- 172.21.15.27:0/4044562557 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556384aac070 con 0x556384756d80 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.741+0000 7fb300ff9640 10 client.26333 handle_command_reply: tid=0 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.741+0000 7fb3261c3740 1 client.26333 shutdown 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 msgr2=0x556384b426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b426a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x556384b39d50 tx=0x7fb314010d70 comp rx=0 tx=0).stop 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 shutdown_connections 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556384756d80 0x556384b4fff0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847981b0 0x556384b49ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847774d0 0x556384b42be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 --2- 172.21.15.27:0/4044562557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b426a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 >> 172.21.15.27:0/4044562557 conn(0x55638478c300 msgr2=0x556384b37210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb300ff9640 0 client.26333 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 shutdown_connections 2024-04-25T13:47:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 1 -- 172.21.15.27:0/4044562557 wait complete. 2024-04-25T13:47:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.742+0000 7fb3261c3740 20 client.26333 trim_cache size 0 max 16384 2024-04-25T13:47:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.763+0000 7fb3261c3740 1 Processor -- start 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.763+0000 7fb3261c3740 1 -- start start 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.763+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b556d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847774d0 0x556384b49a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 0x556384b55c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb3261c3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181a7050 con 0x5563847981b0 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31d0a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb3261c3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181a98d0 con 0x55638476f2f0 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb3261c3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181a76c0 con 0x5563847774d0 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31d0a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43258/0 (socket says 172.21.15.27:43258) 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31d0a9640 1 -- 172.21.15.27:0/3191848990 learned_addr learned my addr 172.21.15.27:0/3191848990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847774d0 0x556384b49a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31d8aa640 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 0x556384b55c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31c8a8640 1 -- 172.21.15.27:0/3191848990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 msgr2=0x556384b556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31c8a8640 1 -- 172.21.15.27:0/3191848990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 msgr2=0x556384b55c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 0x556384b55c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.764+0000 7fb31c8a8640 1 -- 172.21.15.27:0/3191848990 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556384b39af0 con 0x5563847774d0 2024-04-25T13:47:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.765+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847774d0 0x556384b49a60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb3080aa5f0 tx=0x7fb308097040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1112956df1d6cef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.766+0000 7fb302ffd640 1 -- 172.21.15.27:0/3191848990 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3080ae080 con 0x5563847774d0 2024-04-25T13:47:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.766+0000 7fb302ffd640 1 -- 172.21.15.27:0/3191848990 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3080aec70 con 0x5563847774d0 2024-04-25T13:47:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.766+0000 7fb302ffd640 1 -- 172.21.15.27:0/3191848990 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3080e9ca0 con 0x5563847774d0 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 -- 172.21.15.27:0/3191848990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847774d0 msgr2=0x556384b49a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847774d0 0x556384b49a60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb3080aa5f0 tx=0x7fb308097040 comp rx=0 tx=0).stop 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 -- 172.21.15.27:0/3191848990 shutdown_connections 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 0x556384b55c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5563847774d0 0x556384b49a60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 --2- 172.21.15.27:0/3191848990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638476f2f0 0x556384b556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 -- 172.21.15.27:0/3191848990 >> 172.21.15.27:0/3191848990 conn(0x55638478c300 msgr2=0x556384b3cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 -- 172.21.15.27:0/3191848990 shutdown_connections 2024-04-25T13:47:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.767+0000 7fb3261c3740 1 -- 172.21.15.27:0/3191848990 wait complete. 2024-04-25T13:47:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 1 -- start start 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 1 Processor -- start 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55638476f2f0 0x556384b49700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b47d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 0x556384b482b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181a7d90 con 0x5563847981b0 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181a9200 con 0x5563847774d0 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181a8460 con 0x55638476f2f0 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb3261c3740 10 client.0 _ll_register_callbacks cb 0x7fb3080e9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb31c8a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b47d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb31d8aa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 0x556384b482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb31d0a9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55638476f2f0 0x556384b49700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb31c8a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b47d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43274/0 (socket says 172.21.15.27:43274) 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.768+0000 7fb31c8a8640 1 -- 172.21.15.27:0/2235145722 learned_addr learned my addr 172.21.15.27:0/2235145722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.769+0000 7fb31c8a8640 1 -- 172.21.15.27:0/2235145722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55638476f2f0 msgr2=0x556384b49700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.769+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55638476f2f0 0x556384b49700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.769+0000 7fb31c8a8640 1 -- 172.21.15.27:0/2235145722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 msgr2=0x556384b482b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.769+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 0x556384b482b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.769+0000 7fb31c8a8640 1 -- 172.21.15.27:0/2235145722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556384b487f0 con 0x5563847774d0 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.769+0000 7fb31c8a8640 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b47d70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb3080ae500 tx=0x7fb3080e9660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c418d20b62abb1d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.769+0000 7fb300ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.770+0000 7fb300ff9640 1 -- 172.21.15.27:0/2235145722 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3080c7020 con 0x5563847774d0 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.770+0000 7fb300ff9640 1 -- 172.21.15.27:0/2235145722 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb3080e80f0 con 0x5563847774d0 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.770+0000 7fb3261c3740 1 -- 172.21.15.27:0/2235145722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3080aec70 con 0x5563847774d0 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.770+0000 7fb300ff9640 1 -- 172.21.15.27:0/2235145722 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb3080c8e60 con 0x5563847774d0 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.770+0000 7fb3261c3740 10 client.26339 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.770+0000 7fb3261c3740 1 -- 172.21.15.27:0/2235145722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556384bb6e80 con 0x5563847774d0 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb300ff9640 1 -- 172.21.15.27:0/2235145722 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3080aec70 con 0x5563847774d0 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb300ff9640 1 -- 172.21.15.27:0/2235145722 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb30801a070 con 0x5563847774d0 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb3261c3740 10 client.26339 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb3261c3740 10 client.26339 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb3261c3740 20 client.26339 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb3261c3740 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55638472c1c0 0x556384bb97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:05.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb3261c3740 4 client.26339 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:47:05.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb3261c3740 1 -- 172.21.15.27:0/2235145722 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556384aa5ee0 con 0x55638472c1c0 2024-04-25T13:47:05.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.771+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55638472c1c0 0x556384bb97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:05.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.772+0000 7fb300ff9640 10 client.26339 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:05.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.772+0000 7fb31d0a9640 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55638472c1c0 0x556384bb97f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27cdbc0848b11e70 server_cookie=cd9cd5e4b0cd24f2 in_seq=0 out_seq=0 2024-04-25T13:47:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.824+0000 7fb300ff9640 1 -- 172.21.15.27:0/2235145722 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556384aa5ee0 con 0x55638472c1c0 2024-04-25T13:47:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.824+0000 7fb300ff9640 10 client.26339 handle_command_reply: tid=0 2024-04-25T13:47:05.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.824+0000 7fb3261c3740 1 client.26339 shutdown 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.824+0000 7fb3261c3740 1 -- 172.21.15.27:0/2235145722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 msgr2=0x556384b47d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.824+0000 7fb3261c3740 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b47d70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb3080ae500 tx=0x7fb3080e9660 comp rx=0 tx=0).stop 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 1 -- 172.21.15.27:0/2235145722 shutdown_connections 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55638472c1c0 0x556384bb97f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5563847981b0 0x556384b482b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563847774d0 0x556384b47d70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb300ff9640 0 client.26339 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 1 --2- 172.21.15.27:0/2235145722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55638476f2f0 0x556384b49700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 1 -- 172.21.15.27:0/2235145722 >> 172.21.15.27:0/2235145722 conn(0x55638478c300 msgr2=0x556384b37100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 1 -- 172.21.15.27:0/2235145722 shutdown_connections 2024-04-25T13:47:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 1 -- 172.21.15.27:0/2235145722 wait complete. 2024-04-25T13:47:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.825+0000 7fb3261c3740 20 client.26339 trim_cache size 0 max 16384 2024-04-25T13:47:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:47:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:47:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:47:05.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:47:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 -- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3040afb60 msgr2=0x7fb3040b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3040afb60 0x7fb3040b2070 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb310006010 tx=0x7fb310030970 comp rx=0 tx=0).stop 2024-04-25T13:47:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 -- 172.21.15.27:0/2486616354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb318108ec0 msgr2=0x7fb31819c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb318108ec0 0x7fb31819c8f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb3080142d0 tx=0x7fb308014790 comp rx=0 tx=0).stop 2024-04-25T13:47:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 -- 172.21.15.27:0/2486616354 shutdown_connections 2024-04-25T13:47:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3040afb60 0x7fb3040b2070 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31810b9f0 0x7fb3181a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb318108ec0 0x7fb31819c8f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 --2- 172.21.15.27:0/2486616354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318104010 0x7fb31819c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 -- 172.21.15.27:0/2486616354 >> 172.21.15.27:0/2486616354 conn(0x7fb3180fd390 msgr2=0x7fb318109b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 -- 172.21.15.27:0/2486616354 shutdown_connections 2024-04-25T13:47:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:05.826+0000 7fb31f30d640 1 -- 172.21.15.27:0/2486616354 wait complete. 2024-04-25T13:47:06.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:06 smithi122 ceph-mon[32843]: pgmap v535: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 41 MiB/s wr, 1.62k op/s 2024-04-25T13:47:06.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:06 smithi045 ceph-mon[32731]: pgmap v535: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 41 MiB/s wr, 1.62k op/s 2024-04-25T13:47:06.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:06 smithi027 ceph-mon[29140]: pgmap v535: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 41 MiB/s wr, 1.62k op/s 2024-04-25T13:47:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 360.0 s, 2476.2 tps, lat 40.539 ms stddev 43.307 2024-04-25T13:47:08.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:08 smithi122 ceph-mon[32843]: pgmap v536: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 42 MiB/s wr, 1.74k op/s 2024-04-25T13:47:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:08 smithi045 ceph-mon[32731]: pgmap v536: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 42 MiB/s wr, 1.74k op/s 2024-04-25T13:47:08.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:08 smithi027 ceph-mon[29140]: pgmap v536: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 42 MiB/s wr, 1.74k op/s 2024-04-25T13:47:10.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:10 smithi122 ceph-mon[32843]: pgmap v537: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 39 MiB/s wr, 1.53k op/s 2024-04-25T13:47:10.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:10 smithi045 ceph-mon[32731]: pgmap v537: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 39 MiB/s wr, 1.53k op/s 2024-04-25T13:47:10.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:10 smithi027 ceph-mon[29140]: pgmap v537: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 39 MiB/s wr, 1.53k op/s 2024-04-25T13:47:12.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:12 smithi122 ceph-mon[32843]: pgmap v538: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 40 MiB/s wr, 1.61k op/s 2024-04-25T13:47:12.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:12 smithi045 ceph-mon[32731]: pgmap v538: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 40 MiB/s wr, 1.61k op/s 2024-04-25T13:47:12.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:12 smithi027 ceph-mon[29140]: pgmap v538: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 40 MiB/s wr, 1.61k op/s 2024-04-25T13:47:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 365.0 s, 3122.6 tps, lat 32.117 ms stddev 36.908 2024-04-25T13:47:14.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:14 smithi122 ceph-mon[32843]: pgmap v539: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 27 MiB/s wr, 1.20k op/s 2024-04-25T13:47:14.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:14 smithi045 ceph-mon[32731]: pgmap v539: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 27 MiB/s wr, 1.20k op/s 2024-04-25T13:47:14.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:14 smithi027 ceph-mon[29140]: pgmap v539: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 27 MiB/s wr, 1.20k op/s 2024-04-25T13:47:16.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:16 smithi045 ceph-mon[32731]: pgmap v540: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 9.6 MiB/s wr, 647 op/s 2024-04-25T13:47:16.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:16 smithi027 ceph-mon[29140]: pgmap v540: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 9.6 MiB/s wr, 647 op/s 2024-04-25T13:47:16.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:16 smithi122 ceph-mon[32843]: pgmap v540: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 9.6 MiB/s wr, 647 op/s 2024-04-25T13:47:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 370.0 s, 2802.6 tps, lat 35.360 ms stddev 39.989 2024-04-25T13:47:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:18 smithi045 ceph-mon[32731]: pgmap v541: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 834 op/s 2024-04-25T13:47:18.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:18 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:47:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:18 smithi027 ceph-mon[29140]: pgmap v541: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 834 op/s 2024-04-25T13:47:18.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:18 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:47:18.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:18 smithi122 ceph-mon[32843]: pgmap v541: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 834 op/s 2024-04-25T13:47:18.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:18 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:47:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.957+0000 7f64852ee640 1 -- 172.21.15.27:0/62240387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6480110060 msgr2=0x7f6480112480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.957+0000 7f64852ee640 1 --2- 172.21.15.27:0/62240387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6480110060 0x7f6480112480 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6470009900 tx=0x7f647002a410 comp rx=0 tx=0).stop 2024-04-25T13:47:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.958+0000 7f64852ee640 1 -- 172.21.15.27:0/62240387 shutdown_connections 2024-04-25T13:47:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.958+0000 7f64852ee640 1 --2- 172.21.15.27:0/62240387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6480110060 0x7f6480112480 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.958+0000 7f64852ee640 1 --2- 172.21.15.27:0/62240387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f648010b360 0x7f648010b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.958+0000 7f64852ee640 1 --2- 172.21.15.27:0/62240387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f648010a320 0x7f648010ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.958+0000 7f64852ee640 1 -- 172.21.15.27:0/62240387 >> 172.21.15.27:0/62240387 conn(0x7f6480078de0 msgr2=0x7f648007b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.958+0000 7f64852ee640 1 -- 172.21.15.27:0/62240387 shutdown_connections 2024-04-25T13:47:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.958+0000 7f64852ee640 1 -- 172.21.15.27:0/62240387 wait complete. 2024-04-25T13:47:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.959+0000 7f64852ee640 1 Processor -- start 2024-04-25T13:47:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.959+0000 7f64852ee640 1 -- start start 2024-04-25T13:47:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.959+0000 7f64852ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648010a320 0x7f64801a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.959+0000 7f64852ee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f648010b360 0x7f64801a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.959+0000 7f64852ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6480110060 0x7f64801a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.959+0000 7f64852ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6480115cc0 con 0x7f648010a320 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.959+0000 7f64852ee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6480115b40 con 0x7f6480110060 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.959+0000 7f64852ee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6480115e40 con 0x7f648010b360 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648010a320 0x7f64801a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648010a320 0x7f64801a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36152/0 (socket says 172.21.15.27:36152) 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f648010b360 0x7f64801a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647effd640 1 -- 172.21.15.27:0/1824951733 learned_addr learned my addr 172.21.15.27:0/1824951733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647f7fe640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6480110060 0x7f64801a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647f7fe640 1 -- 172.21.15.27:0/1824951733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f648010b360 msgr2=0x7f64801a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647f7fe640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f648010b360 0x7f64801a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647f7fe640 1 -- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648010a320 msgr2=0x7f64801a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647f7fe640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648010a320 0x7f64801a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647f7fe640 1 -- 172.21.15.27:0/1824951733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64801a8330 con 0x7f6480110060 2024-04-25T13:47:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647e7fc640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f648010b360 0x7f64801a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f647f7fe640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6480110060 0x7f64801a7cc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6470009230 tx=0x7f6470003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.960+0000 7f6477fff640 1 -- 172.21.15.27:0/1824951733 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647003f070 con 0x7f6480110060 2024-04-25T13:47:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.961+0000 7f6477fff640 1 -- 172.21.15.27:0/1824951733 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6470031c50 con 0x7f6480110060 2024-04-25T13:47:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.961+0000 7f6477fff640 1 -- 172.21.15.27:0/1824951733 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470035030 con 0x7f6480110060 2024-04-25T13:47:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.961+0000 7f64852ee640 1 -- 172.21.15.27:0/1824951733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64801a85c0 con 0x7f6480110060 2024-04-25T13:47:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.961+0000 7f64852ee640 1 -- 172.21.15.27:0/1824951733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64801a8a30 con 0x7f6480110060 2024-04-25T13:47:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.965+0000 7f648c19e740 1 Processor -- start 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.965+0000 7f648c19e740 1 -- start start 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f648c19e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 0x55cf6e63b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f648c19e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f648c19e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e27a4d0 0x55cf6e642df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f648c19e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cf6e5a1460 con 0x55cf6e29b1b0 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f648c19e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cf6e55b590 con 0x55cf6e27a4d0 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f648c19e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55cf6e59d420 con 0x55cf6e2722f0 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f6477fff640 1 -- 172.21.15.27:0/1824951733 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6470042070 con 0x7f6480110060 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:45542/0 (socket says 172.21.15.27:45542) 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 0x55cf6e63b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647effd640 1 -- 172.21.15.27:0/1580871924 learned_addr learned my addr 172.21.15.27:0/1580871924 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647e7fc640 1 --2- 172.21.15.27:0/1580871924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e27a4d0 0x55cf6e642df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647effd640 1 -- 172.21.15.27:0/1580871924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e27a4d0 msgr2=0x55cf6e642df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647effd640 1 --2- 172.21.15.27:0/1580871924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e27a4d0 0x55cf6e642df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647effd640 1 -- 172.21.15.27:0/1580871924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 msgr2=0x55cf6e63b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647effd640 1 --2- 172.21.15.27:0/1580871924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 0x55cf6e63b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.966+0000 7f647effd640 1 -- 172.21.15.27:0/1580871924 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf6e5680d0 con 0x55cf6e2722f0 2024-04-25T13:47:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.967+0000 7f647effd640 1 --2- 172.21.15.27:0/1580871924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63bef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f64780114f0 tx=0x7f64780119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f07411636676ba0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.967+0000 7f64757fa640 1 -- 172.21.15.27:0/1580871924 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478004b20 con 0x55cf6e2722f0 2024-04-25T13:47:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.967+0000 7f64757fa640 1 -- 172.21.15.27:0/1580871924 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6478004cc0 con 0x55cf6e2722f0 2024-04-25T13:47:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.967+0000 7f64757fa640 1 -- 172.21.15.27:0/1580871924 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478005480 con 0x55cf6e2722f0 2024-04-25T13:47:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.967+0000 7f6477fff640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f646c0afbe0 0x7f646c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.967+0000 7f647e7fc640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f646c0afbe0 0x7f646c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.967+0000 7f6477fff640 1 -- 172.21.15.27:0/1824951733 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f64700ff200 con 0x7f6480110060 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.968+0000 7f648c19e740 1 -- 172.21.15.27:0/1580871924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 msgr2=0x55cf6e63bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.968+0000 7f648c19e740 1 --2- 172.21.15.27:0/1580871924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63bef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f64780114f0 tx=0x7f64780119b0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.968+0000 7f648c19e740 1 -- 172.21.15.27:0/1580871924 shutdown_connections 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.968+0000 7f648c19e740 1 --2- 172.21.15.27:0/1580871924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e27a4d0 0x55cf6e642df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.968+0000 7f648c19e740 1 --2- 172.21.15.27:0/1580871924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63bef0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.968+0000 7f648c19e740 1 --2- 172.21.15.27:0/1580871924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 0x55cf6e63b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.968+0000 7f648c19e740 1 -- 172.21.15.27:0/1580871924 >> 172.21.15.27:0/1580871924 conn(0x55cf6e28f300 msgr2=0x55cf6e637300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.969+0000 7f648c19e740 1 -- 172.21.15.27:0/1580871924 shutdown_connections 2024-04-25T13:47:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.969+0000 7f647e7fc640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f646c0afbe0 0x7f646c0b20f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55cf6e63cb40 tx=0x7f6468009b30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.969+0000 7f648c19e740 1 -- 172.21.15.27:0/1580871924 wait complete. 2024-04-25T13:47:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 1 -- start start 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 1 Processor -- start 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e2722f0 0x55cf6e6455e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 0x55cf6e645b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 0x55cf6e64ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55cf6e643840 con 0x55cf6e29b1b0 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55cf6e528b60 con 0x55cf6e2722f0 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55cf6e6439c0 con 0x55cf6e27a4d0 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f648c19e740 10 client.0 _ll_register_callbacks cb 0x55cf6e54d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e2722f0 0x55cf6e6455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 0x55cf6e64ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 0x55cf6e645b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e2722f0 0x55cf6e6455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52760/0 (socket says 172.21.15.27:52760) 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647effd640 1 -- 172.21.15.27:0/414892586 learned_addr learned my addr 172.21.15.27:0/414892586 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647effd640 1 -- 172.21.15.27:0/414892586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 msgr2=0x55cf6e645b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647effd640 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 0x55cf6e645b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647effd640 1 -- 172.21.15.27:0/414892586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 msgr2=0x55cf6e64ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647effd640 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 0x55cf6e64ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.970+0000 7f647effd640 1 -- 172.21.15.27:0/414892586 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf6e2b3210 con 0x55cf6e2722f0 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f647effd640 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e2722f0 0x55cf6e6455e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f647801b040 tx=0x7f647800fe40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9d48450d727f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f64667fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f64667fc640 1 -- 172.21.15.27:0/414892586 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478028020 con 0x55cf6e2722f0 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f64667fc640 1 -- 172.21.15.27:0/414892586 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6478016040 con 0x55cf6e2722f0 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f648c19e740 1 -- 172.21.15.27:0/414892586 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6478004cc0 con 0x55cf6e2722f0 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f64667fc640 1 -- 172.21.15.27:0/414892586 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6478004e70 con 0x55cf6e2722f0 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f648c19e740 10 client.26351 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f648c19e740 1 -- 172.21.15.27:0/414892586 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55cf6e37f620 con 0x55cf6e2722f0 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f64667fc640 1 -- 172.21.15.27:0/414892586 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6478004cc0 con 0x55cf6e2722f0 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.971+0000 7f64667fc640 1 -- 172.21.15.27:0/414892586 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6478005800 con 0x55cf6e2722f0 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f648c19e740 10 client.26351 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f648c19e740 10 client.26351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f648c19e740 20 client.26351 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f648c19e740 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf6e259d80 0x55cf6e653130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f648c19e740 4 client.26351 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f648c19e740 1 -- 172.21.15.27:0/414892586 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf6e5af070 con 0x55cf6e259d80 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f647e7fc640 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf6e259d80 0x55cf6e653130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f647e7fc640 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf6e259d80 0x55cf6e653130 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60eb3d26644bcaa4 server_cookie=bef161393689582c in_seq=0 out_seq=0 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.972+0000 7f64667fc640 10 client.26351 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f64667fc640 1 -- 172.21.15.27:0/414892586 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cf6e5af070 con 0x55cf6e259d80 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f64667fc640 10 client.26351 handle_command_reply: tid=0 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 client.26351 shutdown 2024-04-25T13:47:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 -- 172.21.15.27:0/414892586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e2722f0 msgr2=0x55cf6e6455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e2722f0 0x55cf6e6455e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f647801b040 tx=0x7f647800fe40 comp rx=0 tx=0).stop 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 -- 172.21.15.27:0/414892586 shutdown_connections 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf6e259d80 0x55cf6e653130 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e29b1b0 0x55cf6e64ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 0x55cf6e645b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f64667fc640 0 client.26351 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 --2- 172.21.15.27:0/414892586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e2722f0 0x55cf6e6455e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 -- 172.21.15.27:0/414892586 >> 172.21.15.27:0/414892586 conn(0x55cf6e28f300 msgr2=0x55cf6e63a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 -- 172.21.15.27:0/414892586 shutdown_connections 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 1 -- 172.21.15.27:0/414892586 wait complete. 2024-04-25T13:47:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.973+0000 7f648c19e740 20 client.26351 trim_cache size 0 max 16384 2024-04-25T13:47:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.993+0000 7f648c19e740 1 Processor -- start 2024-04-25T13:47:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.994+0000 7f648c19e740 1 -- start start 2024-04-25T13:47:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.994+0000 7f648c19e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.994+0000 7f648c19e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e27a4d0 0x55cf6e64c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f648c19e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e636bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f648c19e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f64801af7c0 con 0x55cf6e27a4d0 2024-04-25T13:47:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f648c19e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64801b2040 con 0x55cf6e29b1b0 2024-04-25T13:47:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f648c19e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f64801afe30 con 0x55cf6e2722f0 2024-04-25T13:47:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e27a4d0 0x55cf6e64c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e27a4d0 0x55cf6e64c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36176/0 (socket says 172.21.15.27:36176) 2024-04-25T13:47:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647e7fc640 1 -- 172.21.15.27:0/3408547471 learned_addr learned my addr 172.21.15.27:0/3408547471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647effd640 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647f7fe640 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e636bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647e7fc640 1 -- 172.21.15.27:0/3408547471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 msgr2=0x55cf6e63a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647e7fc640 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647e7fc640 1 -- 172.21.15.27:0/3408547471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 msgr2=0x55cf6e636bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647e7fc640 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e636bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647e7fc640 1 -- 172.21.15.27:0/3408547471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf6e63f540 con 0x55cf6e27a4d0 2024-04-25T13:47:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.995+0000 7f647f7fe640 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e636bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:47:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.996+0000 7f647effd640 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.996+0000 7f647e7fc640 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e27a4d0 0x55cf6e64c9a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6468009710 tx=0x7f64680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c093142cedd9f062 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.996+0000 7f6474ff9640 1 -- 172.21.15.27:0/3408547471 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646801ea90 con 0x55cf6e27a4d0 2024-04-25T13:47:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.996+0000 7f6474ff9640 1 -- 172.21.15.27:0/3408547471 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64680054d0 con 0x55cf6e27a4d0 2024-04-25T13:47:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.996+0000 7f6474ff9640 1 -- 172.21.15.27:0/3408547471 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646802c6b0 con 0x55cf6e27a4d0 2024-04-25T13:47:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.997+0000 7f648c19e740 1 -- 172.21.15.27:0/3408547471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e27a4d0 msgr2=0x55cf6e64c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.997+0000 7f648c19e740 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e27a4d0 0x55cf6e64c9a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6468009710 tx=0x7f64680046c0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.997+0000 7f648c19e740 1 -- 172.21.15.27:0/3408547471 shutdown_connections 2024-04-25T13:47:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.997+0000 7f648c19e740 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e636bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.997+0000 7f648c19e740 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e27a4d0 0x55cf6e64c9a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.997+0000 7f648c19e740 1 --2- 172.21.15.27:0/3408547471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e2722f0 0x55cf6e63a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.997+0000 7f648c19e740 1 -- 172.21.15.27:0/3408547471 >> 172.21.15.27:0/3408547471 conn(0x55cf6e28f300 msgr2=0x55cf6e63a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 -- 172.21.15.27:0/3408547471 shutdown_connections 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 -- 172.21.15.27:0/3408547471 wait complete. 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 -- start start 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 Processor -- start 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e2722f0 0x55cf6e649930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 0x55cf6e649e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e6b9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f64801b0500 con 0x55cf6e2722f0 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64801b1970 con 0x55cf6e29b1b0 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f64801b0bd0 con 0x55cf6e27a4d0 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f648c19e740 10 client.0 _ll_register_callbacks cb 0x7f6468009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f647effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e2722f0 0x55cf6e649930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f647effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e2722f0 0x55cf6e649930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36178/0 (socket says 172.21.15.27:36178) 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.998+0000 7f647effd640 1 -- 172.21.15.27:0/1830657604 learned_addr learned my addr 172.21.15.27:0/1830657604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f647f7fe640 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e6b9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f647e7fc640 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 0x55cf6e649e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f647effd640 1 -- 172.21.15.27:0/1830657604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 msgr2=0x55cf6e649e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f647effd640 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 0x55cf6e649e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f647effd640 1 -- 172.21.15.27:0/1830657604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 msgr2=0x55cf6e6b9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f647effd640 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e6b9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f647effd640 1 -- 172.21.15.27:0/1830657604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf6e64cee0 con 0x55cf6e2722f0 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f647effd640 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e2722f0 0x55cf6e649930 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6478003d70 tx=0x7f6478004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619d03be904f3cf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:18.999+0000 7f64667fc640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.000+0000 7f64667fc640 1 -- 172.21.15.27:0/1830657604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647800f600 con 0x55cf6e2722f0 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.000+0000 7f64667fc640 1 -- 172.21.15.27:0/1830657604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f647800f7a0 con 0x55cf6e2722f0 2024-04-25T13:47:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.000+0000 7f648c19e740 1 -- 172.21.15.27:0/1830657604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64680054d0 con 0x55cf6e2722f0 2024-04-25T13:47:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.000+0000 7f64667fc640 1 -- 172.21.15.27:0/1830657604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f647800f9a0 con 0x55cf6e2722f0 2024-04-25T13:47:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.000+0000 7f648c19e740 10 client.16647 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.000+0000 7f648c19e740 1 -- 172.21.15.27:0/1830657604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55cf6e6ba9f0 con 0x55cf6e2722f0 2024-04-25T13:47:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.000+0000 7f64667fc640 1 -- 172.21.15.27:0/1830657604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64680054d0 con 0x55cf6e2722f0 2024-04-25T13:47:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.000+0000 7f64667fc640 1 -- 172.21.15.27:0/1830657604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6478043cc0 con 0x55cf6e2722f0 2024-04-25T13:47:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.001+0000 7f648c19e740 10 client.16647 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.001+0000 7f648c19e740 10 client.16647 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.001+0000 7f648c19e740 20 client.16647 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.001+0000 7f648c19e740 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf6e22f1c0 0x55cf6e6bd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.001+0000 7f648c19e740 4 client.16647 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:47:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.002+0000 7f648c19e740 1 -- 172.21.15.27:0/1830657604 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf6e5a8ee0 con 0x55cf6e22f1c0 2024-04-25T13:47:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.002+0000 7f647e7fc640 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf6e22f1c0 0x55cf6e6bd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.002+0000 7f647e7fc640 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf6e22f1c0 0x55cf6e6bd430 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f6d6a1fd2f0d834 server_cookie=eb2e46faa9b67748 in_seq=0 out_seq=0 2024-04-25T13:47:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.002+0000 7f64667fc640 10 client.16647 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.030+0000 7f64667fc640 1 -- 172.21.15.27:0/1830657604 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf6e5a8ee0 con 0x55cf6e22f1c0 2024-04-25T13:47:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.030+0000 7f64667fc640 10 client.16647 handle_command_reply: tid=0 2024-04-25T13:47:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.030+0000 7f648c19e740 1 client.16647 shutdown 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 -- 172.21.15.27:0/1830657604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e2722f0 msgr2=0x55cf6e649930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e2722f0 0x55cf6e649930 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6478003d70 tx=0x7f6478004540 comp rx=0 tx=0).stop 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 -- 172.21.15.27:0/1830657604 shutdown_connections 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55cf6e22f1c0 0x55cf6e6bd430 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f64667fc640 0 client.16647 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf6e29b1b0 0x55cf6e6b9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55cf6e27a4d0 0x55cf6e649e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 --2- 172.21.15.27:0/1830657604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cf6e2722f0 0x55cf6e649930 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 -- 172.21.15.27:0/1830657604 >> 172.21.15.27:0/1830657604 conn(0x55cf6e28f300 msgr2=0x55cf6e645120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:19.031 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 -- 172.21.15.27:0/1830657604 shutdown_connections 2024-04-25T13:47:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 1 -- 172.21.15.27:0/1830657604 wait complete. 2024-04-25T13:47:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.031+0000 7f648c19e740 20 client.16647 trim_cache size 0 max 16384 2024-04-25T13:47:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:47:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:47:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:47:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:47:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 -- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f646c0afbe0 msgr2=0x7f646c0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f646c0afbe0 0x7f646c0b20f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55cf6e63cb40 tx=0x7f6468009b30 comp rx=0 tx=0).stop 2024-04-25T13:47:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 -- 172.21.15.27:0/1824951733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6480110060 msgr2=0x7f64801a7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6480110060 0x7f64801a7cc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6470009230 tx=0x7f6470003b90 comp rx=0 tx=0).stop 2024-04-25T13:47:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 -- 172.21.15.27:0/1824951733 shutdown_connections 2024-04-25T13:47:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f646c0afbe0 0x7f646c0b20f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6480110060 0x7f64801a7cc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f648010b360 0x7f64801a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 --2- 172.21.15.27:0/1824951733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648010a320 0x7f64801a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 -- 172.21.15.27:0/1824951733 >> 172.21.15.27:0/1824951733 conn(0x7f6480078de0 msgr2=0x7f648010c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 -- 172.21.15.27:0/1824951733 shutdown_connections 2024-04-25T13:47:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:19.032+0000 7f64852ee640 1 -- 172.21.15.27:0/1824951733 wait complete. 2024-04-25T13:47:19.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T13:47:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T13:47:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:47:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:47:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:47:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T13:47:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T13:47:19.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:47:19.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:47:19.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi027 to 176.2M 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: Unable to set osd_memory_target on smithi027 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:47:19.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:47:20.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:20 smithi122 ceph-mon[32843]: pgmap v542: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 680 op/s 2024-04-25T13:47:20.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:20 smithi045 ceph-mon[32731]: pgmap v542: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 680 op/s 2024-04-25T13:47:20.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:20 smithi027 ceph-mon[29140]: pgmap v542: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 680 op/s 2024-04-25T13:47:22.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:22 smithi122 ceph-mon[32843]: pgmap v543: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 12 MiB/s wr, 809 op/s 2024-04-25T13:47:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:22 smithi045 ceph-mon[32731]: pgmap v543: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 12 MiB/s wr, 809 op/s 2024-04-25T13:47:22.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:22 smithi027 ceph-mon[29140]: pgmap v543: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 12 MiB/s wr, 809 op/s 2024-04-25T13:47:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 375.0 s, 2907.6 tps, lat 34.673 ms stddev 41.616 2024-04-25T13:47:24.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:24 smithi122 ceph-mon[32843]: pgmap v544: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 858 op/s 2024-04-25T13:47:24.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:24 smithi045 ceph-mon[32731]: pgmap v544: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 858 op/s 2024-04-25T13:47:24.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:24 smithi027 ceph-mon[29140]: pgmap v544: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 858 op/s 2024-04-25T13:47:26.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:47:26.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:47:26.463 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:26 smithi027 ceph-mon[29140]: pgmap v545: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 733 op/s 2024-04-25T13:47:26.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:26 smithi122 ceph-mon[32843]: pgmap v545: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 733 op/s 2024-04-25T13:47:26.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:26 smithi045 ceph-mon[32731]: pgmap v545: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 733 op/s 2024-04-25T13:47:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.602+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2385629293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010b310 msgr2=0x7f17e010b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.602+0000 7f17de7fc640 1 -- 172.21.15.27:0/2385629293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0035020 con 0x7f17e010b310 2024-04-25T13:47:26.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.602+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2385629293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010b310 0x7f17e010b720 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009900 tx=0x7f17d002a440 comp rx=0 tx=0).stop 2024-04-25T13:47:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.605+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2385629293 shutdown_connections 2024-04-25T13:47:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.605+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2385629293 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17e0110030 0x7f17e0110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.605+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2385629293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010b310 0x7f17e010b720 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.605+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2385629293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e010a420 0x7f17e010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.605+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2385629293 >> 172.21.15.27:0/2385629293 conn(0x7f17e00704e0 msgr2=0x7f17e0072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:26.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.611+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2385629293 shutdown_connections 2024-04-25T13:47:26.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.612+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2385629293 wait complete. 2024-04-25T13:47:26.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.612+0000 7f17e5ab1640 1 Processor -- start 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.613+0000 7f17e5ab1640 1 -- start start 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.613+0000 7f17e5ab1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010a420 0x7f17e011b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.613+0000 7f17e5ab1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e010b310 0x7f17e011bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.613+0000 7f17e5ab1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17e0110030 0x7f17e0119c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.613+0000 7f17e5ab1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17e0111f90 con 0x7f17e010a420 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.613+0000 7f17e5ab1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f17e0111e10 con 0x7f17e010b310 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.613+0000 7f17e5ab1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f17e0112110 con 0x7f17e0110030 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.614+0000 7f17df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010a420 0x7f17e011b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.614+0000 7f17df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010a420 0x7f17e011b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46784/0 (socket says 172.21.15.27:46784) 2024-04-25T13:47:26.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.614+0000 7f17df7fe640 1 -- 172.21.15.27:0/2188545907 learned_addr learned my addr 172.21.15.27:0/2188545907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:26.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.615+0000 7f17df7fe640 1 -- 172.21.15.27:0/2188545907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17e0110030 msgr2=0x7f17e0119c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:47:26.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.615+0000 7f17df7fe640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17e0110030 0x7f17e0119c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.615+0000 7f17df7fe640 1 -- 172.21.15.27:0/2188545907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e010b310 msgr2=0x7f17e011bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.615+0000 7f17df7fe640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e010b310 0x7f17e011bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.616+0000 7f17df7fe640 1 -- 172.21.15.27:0/2188545907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e011a300 con 0x7f17e010a420 2024-04-25T13:47:26.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.621+0000 7f17df7fe640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010a420 0x7f17e011b5f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f17c800b470 tx=0x7f17c800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:26.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.622+0000 7f17dcff9640 1 -- 172.21.15.27:0/2188545907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c8015020 con 0x7f17e010a420 2024-04-25T13:47:26.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.622+0000 7f17dcff9640 1 -- 172.21.15.27:0/2188545907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c8011d40 con 0x7f17e010a420 2024-04-25T13:47:26.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.622+0000 7f17dcff9640 1 -- 172.21.15.27:0/2188545907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c8015020 con 0x7f17e010a420 2024-04-25T13:47:26.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.623+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2188545907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e011a5f0 con 0x7f17e010a420 2024-04-25T13:47:26.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.623+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2188545907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e01b9f00 con 0x7f17e010a420 2024-04-25T13:47:26.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.627+0000 7f17eca39740 1 Processor -- start 2024-04-25T13:47:26.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.627+0000 7f17eca39740 1 -- start start 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.627+0000 7f17eca39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d8661f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17eca39740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e2b4d0 0x55d8661f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17eca39740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e336b0 0x55d8661f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17eca39740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d86615ad80 con 0x55d865e2b4d0 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17eca39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d8661148b0 con 0x55d865e54390 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17eca39740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d866156d40 con 0x55d865e336b0 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17deffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e336b0 0x55d8661f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17deffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e336b0 0x55d8661f6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58474/0 (socket says 172.21.15.27:58474) 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17deffd640 1 -- 172.21.15.27:0/1064547096 learned_addr learned my addr 172.21.15.27:0/1064547096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:26.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17dffff640 1 --2- 172.21.15.27:0/1064547096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d8661f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17deffd640 1 -- 172.21.15.27:0/1064547096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 msgr2=0x55d8661f5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17deffd640 1 --2- 172.21.15.27:0/1064547096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d8661f5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17deffd640 1 -- 172.21.15.27:0/1064547096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e2b4d0 msgr2=0x55d8661f5fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17deffd640 1 --2- 172.21.15.27:0/1064547096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e2b4d0 0x55d8661f5fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.628+0000 7f17deffd640 1 -- 172.21.15.27:0/1064547096 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8660e5a40 con 0x55d865e336b0 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.629+0000 7f17deffd640 1 --2- 172.21.15.27:0/1064547096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e336b0 0x55d8661f6c70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009230 tx=0x7f17d00040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0763ae40252741a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.629+0000 7f17d5ffb640 1 -- 172.21.15.27:0/1064547096 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d003e030 con 0x55d865e336b0 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.629+0000 7f17d5ffb640 1 -- 172.21.15.27:0/1064547096 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d00042e0 con 0x55d865e336b0 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.629+0000 7f17d5ffb640 1 -- 172.21.15.27:0/1064547096 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0035040 con 0x55d865e336b0 2024-04-25T13:47:26.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.629+0000 7f17dcff9640 1 -- 172.21.15.27:0/2188545907 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f17c8018020 con 0x7f17e010a420 2024-04-25T13:47:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.630+0000 7f17dcff9640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f17c00aff20 0x7f17c00b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.630+0000 7f17eca39740 1 -- 172.21.15.27:0/1064547096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e336b0 msgr2=0x55d8661f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.630+0000 7f17eca39740 1 --2- 172.21.15.27:0/1064547096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e336b0 0x55d8661f6c70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009230 tx=0x7f17d00040c0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.631+0000 7f17eca39740 1 -- 172.21.15.27:0/1064547096 shutdown_connections 2024-04-25T13:47:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.631+0000 7f17eca39740 1 --2- 172.21.15.27:0/1064547096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e336b0 0x55d8661f6c70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.631+0000 7f17eca39740 1 --2- 172.21.15.27:0/1064547096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e2b4d0 0x55d8661f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.631+0000 7f17dffff640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f17c00aff20 0x7f17c00b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.631+0000 7f17eca39740 1 --2- 172.21.15.27:0/1064547096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d8661f5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.631+0000 7f17eca39740 1 -- 172.21.15.27:0/1064547096 >> 172.21.15.27:0/1064547096 conn(0x55d865e484e0 msgr2=0x55d8661f1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.631+0000 7f17dcff9640 1 -- 172.21.15.27:0/2188545907 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f17c8011690 con 0x7f17e010a420 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.631+0000 7f17dffff640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f17c00aff20 0x7f17c00b2430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55d8661f6ab0 tx=0x7f17d8009430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.632+0000 7f17eca39740 1 -- 172.21.15.27:0/1064547096 shutdown_connections 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.632+0000 7f17eca39740 1 -- 172.21.15.27:0/1064547096 wait complete. 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.632+0000 7f17eca39740 1 -- start start 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.632+0000 7f17eca39740 1 Processor -- start 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17eca39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e2b4d0 0x55d8661ff640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17eca39740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661ffb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17eca39740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e54390 0x55d866206ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17eca39740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d8661f7720 con 0x55d865e336b0 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17eca39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d8660e1d10 con 0x55d865e2b4d0 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17eca39740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d8661f78a0 con 0x55d865e54390 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17eca39740 10 client.0 _ll_register_callbacks cb 0x55d86610e210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661ffb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17df7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e2b4d0 0x55d8661ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661ffb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46800/0 (socket says 172.21.15.27:46800) 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17df7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e2b4d0 0x55d8661ff640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34348/0 (socket says 172.21.15.27:34348) 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17deffd640 1 -- 172.21.15.27:0/377161940 learned_addr learned my addr 172.21.15.27:0/377161940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.633+0000 7f17dffff640 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e54390 0x55d866206ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.634+0000 7f17deffd640 1 -- 172.21.15.27:0/377161940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e54390 msgr2=0x55d866206ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.634+0000 7f17deffd640 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e54390 0x55d866206ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.634+0000 7f17deffd640 1 -- 172.21.15.27:0/377161940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e2b4d0 msgr2=0x55d8661ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.634+0000 7f17deffd640 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e2b4d0 0x55d8661ff640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.634+0000 7f17deffd640 1 -- 172.21.15.27:0/377161940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d865f367d0 con 0x55d865e336b0 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.634+0000 7f17deffd640 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661ffb80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f17d0031ca0 tx=0x7f17d0031cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b3cb0e10e23b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.634+0000 7f17ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.634+0000 7f17ceffd640 1 -- 172.21.15.27:0/377161940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d003e040 con 0x55d865e336b0 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.635+0000 7f17eca39740 1 -- 172.21.15.27:0/377161940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d00042e0 con 0x55d865e336b0 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.635+0000 7f17ceffd640 1 -- 172.21.15.27:0/377161940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f17d00037f0 con 0x55d865e336b0 2024-04-25T13:47:26.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.635+0000 7f17ceffd640 1 -- 172.21.15.27:0/377161940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f17d0053070 con 0x55d865e336b0 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.635+0000 7f17eca39740 10 client.16656 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.635+0000 7f17eca39740 1 -- 172.21.15.27:0/377161940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d865e6c3f0 con 0x55d865e336b0 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.635+0000 7f17ceffd640 1 -- 172.21.15.27:0/377161940 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d00042e0 con 0x55d865e336b0 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.636+0000 7f17ceffd640 1 -- 172.21.15.27:0/377161940 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f17d004c9e0 con 0x55d865e336b0 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.636+0000 7f17eca39740 10 client.16656 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.637+0000 7f17eca39740 10 client.16656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.637+0000 7f17eca39740 20 client.16656 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.637+0000 7f17eca39740 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d865e12f60 0x55d86620cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.637+0000 7f17eca39740 4 client.16656 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.637+0000 7f17eca39740 1 -- 172.21.15.27:0/377161940 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d866168990 con 0x55d865e12f60 2024-04-25T13:47:26.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.637+0000 7f17df7fe640 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d865e12f60 0x55d86620cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.638+0000 7f17df7fe640 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d865e12f60 0x55d86620cba0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7149db5b8ad25b8 server_cookie=a4a25dfaad090ff9 in_seq=0 out_seq=0 2024-04-25T13:47:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.638+0000 7f17ceffd640 10 client.16656 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.640+0000 7f17ceffd640 1 -- 172.21.15.27:0/377161940 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d866168990 con 0x55d865e12f60 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.640+0000 7f17ceffd640 10 client.16656 handle_command_reply: tid=0 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.640+0000 7f17eca39740 1 client.16656 shutdown 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 -- 172.21.15.27:0/377161940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 msgr2=0x55d8661ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661ffb80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f17d0031ca0 tx=0x7f17d0031cd0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 -- 172.21.15.27:0/377161940 shutdown_connections 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d865e12f60 0x55d86620cba0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e54390 0x55d866206ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661ffb80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17ceffd640 0 client.16656 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:26.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 --2- 172.21.15.27:0/377161940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e2b4d0 0x55d8661ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 -- 172.21.15.27:0/377161940 >> 172.21.15.27:0/377161940 conn(0x55d865e484e0 msgr2=0x55d8661f4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 -- 172.21.15.27:0/377161940 shutdown_connections 2024-04-25T13:47:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.641+0000 7f17eca39740 1 -- 172.21.15.27:0/377161940 wait complete. 2024-04-25T13:47:26.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.642+0000 7f17eca39740 20 client.16656 trim_cache size 0 max 16384 2024-04-25T13:47:26.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17eca39740 1 Processor -- start 2024-04-25T13:47:26.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17eca39740 1 -- start start 2024-04-25T13:47:26.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17eca39740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8661f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17eca39740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17eca39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d866206a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17eca39740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17e01c0900 con 0x55d865e336b0 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17eca39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f17e01c3180 con 0x55d865e54390 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17eca39740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f17e01c0f70 con 0x55d865e2b4d0 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17df7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8661f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17df7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8661f0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58500/0 (socket says 172.21.15.27:58500) 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.672+0000 7f17df7fe640 1 -- 172.21.15.27:0/3039499349 learned_addr learned my addr 172.21.15.27:0/3039499349 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17dffff640 1 --2- 172.21.15.27:0/3039499349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d866206a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17df7fe640 1 -- 172.21.15.27:0/3039499349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 msgr2=0x55d866206a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:26.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17df7fe640 1 --2- 172.21.15.27:0/3039499349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d866206a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17df7fe640 1 -- 172.21.15.27:0/3039499349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 msgr2=0x55d8661f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17df7fe640 1 --2- 172.21.15.27:0/3039499349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17df7fe640 1 -- 172.21.15.27:0/3039499349 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8661f7080 con 0x55d865e2b4d0 2024-04-25T13:47:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17df7fe640 1 --2- 172.21.15.27:0/3039499349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8661f0ca0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f17c80a3510 tx=0x7f17c80abb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2be67f2ea3b63f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17d57fa640 1 -- 172.21.15.27:0/3039499349 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c80a2e60 con 0x55d865e2b4d0 2024-04-25T13:47:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17d57fa640 1 -- 172.21.15.27:0/3039499349 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c80aa030 con 0x55d865e2b4d0 2024-04-25T13:47:26.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.673+0000 7f17d57fa640 1 -- 172.21.15.27:0/3039499349 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c80a3000 con 0x55d865e2b4d0 2024-04-25T13:47:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 -- 172.21.15.27:0/3039499349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 msgr2=0x55d8661f0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 --2- 172.21.15.27:0/3039499349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8661f0ca0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f17c80a3510 tx=0x7f17c80abb40 comp rx=0 tx=0).stop 2024-04-25T13:47:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 -- 172.21.15.27:0/3039499349 shutdown_connections 2024-04-25T13:47:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 --2- 172.21.15.27:0/3039499349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d866206a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 --2- 172.21.15.27:0/3039499349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8661f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 --2- 172.21.15.27:0/3039499349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8661f0ca0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 -- 172.21.15.27:0/3039499349 >> 172.21.15.27:0/3039499349 conn(0x55d865e484e0 msgr2=0x55d8661fa3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 -- 172.21.15.27:0/3039499349 shutdown_connections 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.674+0000 7f17eca39740 1 -- 172.21.15.27:0/3039499349 wait complete. 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.675+0000 7f17eca39740 1 -- start start 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.675+0000 7f17eca39740 1 Processor -- start 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.675+0000 7f17eca39740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8662042a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.675+0000 7f17eca39740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8662047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17eca39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d866272da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17eca39740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f17e01c1640 con 0x55d865e336b0 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17eca39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f17e01c2ab0 con 0x55d865e54390 2024-04-25T13:47:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17eca39740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f17e01c1d10 con 0x55d865e2b4d0 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17eca39740 10 client.0 _ll_register_callbacks cb 0x7f17d002ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8662047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8662047e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46808/0 (socket says 172.21.15.27:46808) 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17deffd640 1 -- 172.21.15.27:0/3009758685 learned_addr learned my addr 172.21.15.27:0/3009758685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.676+0000 7f17dffff640 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d866272da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.677+0000 7f17deffd640 1 -- 172.21.15.27:0/3009758685 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 msgr2=0x55d8662042a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.677+0000 7f17deffd640 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8662042a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.677+0000 7f17deffd640 1 -- 172.21.15.27:0/3009758685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 msgr2=0x55d866272da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.677+0000 7f17deffd640 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d866272da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.677+0000 7f17deffd640 1 -- 172.21.15.27:0/3009758685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d865e2c0f0 con 0x55d865e336b0 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.677+0000 7f17deffd640 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8662047e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f17d004ce30 tx=0x7f17d0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6b9af3bb66bbec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.677+0000 7f17ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.677+0000 7f17ceffd640 1 -- 172.21.15.27:0/3009758685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d003e040 con 0x55d865e336b0 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.678+0000 7f17ceffd640 1 -- 172.21.15.27:0/3009758685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f17d002adf0 con 0x55d865e336b0 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.678+0000 7f17eca39740 1 -- 172.21.15.27:0/3009758685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17c80aa030 con 0x55d865e336b0 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.678+0000 7f17ceffd640 1 -- 172.21.15.27:0/3009758685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f17d0035040 con 0x55d865e336b0 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.678+0000 7f17eca39740 10 client.16662 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.678+0000 7f17eca39740 1 -- 172.21.15.27:0/3009758685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d86611b4b0 con 0x55d865e336b0 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17ceffd640 1 -- 172.21.15.27:0/3009758685 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c80aa030 con 0x55d865e336b0 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17ceffd640 1 -- 172.21.15.27:0/3009758685 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f17d004c480 con 0x55d865e336b0 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17eca39740 10 client.16662 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17eca39740 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17eca39740 20 client.16662 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17eca39740 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d865de83a0 0x55d866276970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17eca39740 4 client.16662 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:47:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17eca39740 1 -- 172.21.15.27:0/3009758685 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d866162800 con 0x55d865de83a0 2024-04-25T13:47:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.679+0000 7f17df7fe640 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d865de83a0 0x55d866276970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17df7fe640 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d865de83a0 0x55d866276970 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aff61773cb25e537 server_cookie=421c2fab888bd47c in_seq=0 out_seq=0 2024-04-25T13:47:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17ceffd640 10 client.16662 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17ceffd640 1 -- 172.21.15.27:0/3009758685 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d866162800 con 0x55d865de83a0 2024-04-25T13:47:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17ceffd640 10 client.16662 handle_command_reply: tid=0 2024-04-25T13:47:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 client.16662 shutdown 2024-04-25T13:47:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 -- 172.21.15.27:0/3009758685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 msgr2=0x55d8662047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8662047e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f17d004ce30 tx=0x7f17d0005fd0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 -- 172.21.15.27:0/3009758685 shutdown_connections 2024-04-25T13:47:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d865de83a0 0x55d866276970 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d865e54390 0x55d866272da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d865e336b0 0x55d8662047e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 --2- 172.21.15.27:0/3009758685 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d865e2b4d0 0x55d8662042a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 -- 172.21.15.27:0/3009758685 >> 172.21.15.27:0/3009758685 conn(0x55d865e484e0 msgr2=0x55d8661f0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17ceffd640 0 client.16662 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 -- 172.21.15.27:0/3009758685 shutdown_connections 2024-04-25T13:47:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.680+0000 7f17eca39740 1 -- 172.21.15.27:0/3009758685 wait complete. 2024-04-25T13:47:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17eca39740 20 client.16662 trim_cache size 0 max 16384 2024-04-25T13:47:26.687 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:47:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:47:26.687 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:47:26.687 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:47:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f17c00aff20 msgr2=0x7f17c00b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f17c00aff20 0x7f17c00b2430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55d8661f6ab0 tx=0x7f17d8009430 comp rx=0 tx=0).stop 2024-04-25T13:47:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010a420 msgr2=0x7f17e011b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010a420 0x7f17e011b5f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f17c800b470 tx=0x7f17c800b930 comp rx=0 tx=0).stop 2024-04-25T13:47:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2188545907 shutdown_connections 2024-04-25T13:47:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f17c00aff20 0x7f17c00b2430 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f17e0110030 0x7f17e0119c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17e010b310 0x7f17e011bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 --2- 172.21.15.27:0/2188545907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17e010a420 0x7f17e011b5f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2188545907 >> 172.21.15.27:0/2188545907 conn(0x7f17e00704e0 msgr2=0x7f17e010e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:26.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2188545907 shutdown_connections 2024-04-25T13:47:26.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.681+0000 7f17e5ab1640 1 -- 172.21.15.27:0/2188545907 wait complete. 2024-04-25T13:47:26.695 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:47:26.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:47:26.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.894+0000 7fc0a4a97640 1 -- 172.21.15.27:0/2731064081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a0106f50 msgr2=0x7fc0a0109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.894+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/2731064081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a0106f50 0x7fc0a0109370 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc094009900 tx=0x7fc09402a410 comp rx=0 tx=0).stop 2024-04-25T13:47:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.895+0000 7fc0a4a97640 1 -- 172.21.15.27:0/2731064081 shutdown_connections 2024-04-25T13:47:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.895+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/2731064081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0a01098b0 0x7fc0a010bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.895+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/2731064081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a0106f50 0x7fc0a0109370 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.895+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/2731064081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0a01045f0 0x7fc0a0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.895+0000 7fc0a4a97640 1 -- 172.21.15.27:0/2731064081 >> 172.21.15.27:0/2731064081 conn(0x7fc0a00fdbf0 msgr2=0x7fc0a01000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.895+0000 7fc0a4a97640 1 -- 172.21.15.27:0/2731064081 shutdown_connections 2024-04-25T13:47:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.895+0000 7fc0a4a97640 1 -- 172.21.15.27:0/2731064081 wait complete. 2024-04-25T13:47:26.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc0a4a97640 1 Processor -- start 2024-04-25T13:47:26.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc0a4a97640 1 -- start start 2024-04-25T13:47:26.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc0a4a97640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a01045f0 0x7fc0a019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc0a4a97640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0a0106f50 0x7fc0a019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc0a4a97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0a01098b0 0x7fc0a01a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc0a4a97640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a010f660 con 0x7fc0a01045f0 2024-04-25T13:47:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc0a4a97640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a010f4e0 con 0x7fc0a01098b0 2024-04-25T13:47:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc0a4a97640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a010f7e0 con 0x7fc0a0106f50 2024-04-25T13:47:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc09e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a01045f0 0x7fc0a019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc09e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a01045f0 0x7fc0a019a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46818/0 (socket says 172.21.15.27:46818) 2024-04-25T13:47:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc09e59c640 1 -- 172.21.15.27:0/1127874512 learned_addr learned my addr 172.21.15.27:0/1127874512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.896+0000 7fc09dd9b640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0a0106f50 0x7fc0a019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc09ed9d640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0a01098b0 0x7fc0a01a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc09e59c640 1 -- 172.21.15.27:0/1127874512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0a0106f50 msgr2=0x7fc0a019a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc09e59c640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0a0106f50 0x7fc0a019a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc09e59c640 1 -- 172.21.15.27:0/1127874512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0a01098b0 msgr2=0x7fc0a01a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc09e59c640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0a01098b0 0x7fc0a01a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc09e59c640 1 -- 172.21.15.27:0/1127874512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a01a1b80 con 0x7fc0a01045f0 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc09e59c640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a01045f0 0x7fc0a019a040 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc09000e9c0 tx=0x7fc09000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc08f7fe640 1 -- 172.21.15.27:0/1127874512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09000cd80 con 0x7fc0a01045f0 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0a01a1e70 con 0x7fc0a01045f0 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc08f7fe640 1 -- 172.21.15.27:0/1127874512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0900054d0 con 0x7fc0a01045f0 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc08f7fe640 1 -- 172.21.15.27:0/1127874512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090010430 con 0x7fc0a01045f0 2024-04-25T13:47:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.897+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0a01a22d0 con 0x7fc0a01045f0 2024-04-25T13:47:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.899+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0a00ff710 con 0x7fc0a01045f0 2024-04-25T13:47:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.901+0000 7fc08f7fe640 1 -- 172.21.15.27:0/1127874512 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc090004fb0 con 0x7fc0a01045f0 2024-04-25T13:47:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.902+0000 7fc08f7fe640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc0880afaf0 0x7fc0880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.902+0000 7fc09dd9b640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc0880afaf0 0x7fc0880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.902+0000 7fc08f7fe640 1 -- 172.21.15.27:0/1127874512 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc0900df140 con 0x7fc0a01045f0 2024-04-25T13:47:26.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.903+0000 7fc09dd9b640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc0880afaf0 0x7fc0880b2000 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc0a019b0e0 tx=0x7fc094000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:26.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:26.907+0000 7fc08f7fe640 1 -- 172.21.15.27:0/1127874512 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc090053ff0 con 0x7fc0a01045f0 2024-04-25T13:47:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.265+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0a002dda0 con 0x7fc0a01045f0 2024-04-25T13:47:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.267+0000 7fc08f7fe640 1 -- 172.21.15.27:0/1127874512 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fc0900a7d50 con 0x7fc0a01045f0 2024-04-25T13:47:27.267 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:47:27.267 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:47:27.268 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:47:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc0880afaf0 msgr2=0x7fc0880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc0880afaf0 0x7fc0880b2000 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc0a019b0e0 tx=0x7fc094000bc0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a01045f0 msgr2=0x7fc0a019a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a01045f0 0x7fc0a019a040 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc09000e9c0 tx=0x7fc09000ee80 comp rx=0 tx=0).stop 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 shutdown_connections 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc0880afaf0 0x7fc0880b2000 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0a01098b0 0x7fc0a01a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0a0106f50 0x7fc0a019a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 --2- 172.21.15.27:0/1127874512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a01045f0 0x7fc0a019a040 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.270+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 >> 172.21.15.27:0/1127874512 conn(0x7fc0a00fdbf0 msgr2=0x7fc0a01050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.271+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 shutdown_connections 2024-04-25T13:47:27.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.271+0000 7fc0a4a97640 1 -- 172.21.15.27:0/1127874512 wait complete. 2024-04-25T13:47:27.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:47:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.462+0000 7fdf8def2640 1 -- 172.21.15.27:0/1397642611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88107690 msgr2=0x7fdf88107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.462+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1397642611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88107690 0x7fdf88107aa0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fdf8001acb0 tx=0x7fdf8003b920 comp rx=0 tx=0).stop 2024-04-25T13:47:27.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.467+0000 7fdf8def2640 1 -- 172.21.15.27:0/1397642611 shutdown_connections 2024-04-25T13:47:27.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.467+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1397642611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88107690 0x7fdf88107aa0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.467+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1397642611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf88104d30 0x7fdf88107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.467+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1397642611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf881023d0 0x7fdf881047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.467+0000 7fdf8def2640 1 -- 172.21.15.27:0/1397642611 >> 172.21.15.27:0/1397642611 conn(0x7fdf880fdc00 msgr2=0x7fdf88100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:27.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.467+0000 7fdf8def2640 1 -- 172.21.15.27:0/1397642611 shutdown_connections 2024-04-25T13:47:27.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.468+0000 7fdf8def2640 1 -- 172.21.15.27:0/1397642611 wait complete. 2024-04-25T13:47:27.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.469+0000 7fdf8def2640 1 Processor -- start 2024-04-25T13:47:27.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.469+0000 7fdf8def2640 1 -- start start 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf8def2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf881023d0 0x7fdf88198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf8def2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf88104d30 0x7fdf881986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf8def2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf88107690 0x7fdf8819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf8def2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8810b060 con 0x7fdf881023d0 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf8def2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8810aee0 con 0x7fdf88107690 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf8def2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8810b1e0 con 0x7fdf88104d30 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf7effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf88104d30 0x7fdf881986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf7effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf88104d30 0x7fdf881986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58522/0 (socket says 172.21.15.27:58522) 2024-04-25T13:47:27.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf7effd640 1 -- 172.21.15.27:0/1331731672 learned_addr learned my addr 172.21.15.27:0/1331731672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:27.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.471+0000 7fdf7effd640 1 -- 172.21.15.27:0/1331731672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf88107690 msgr2=0x7fdf8819f470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:47:27.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.472+0000 7fdf7effd640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf88107690 0x7fdf8819f470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.472+0000 7fdf7effd640 1 -- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf881023d0 msgr2=0x7fdf88198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.472+0000 7fdf7effd640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf881023d0 0x7fdf88198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.472+0000 7fdf7effd640 1 -- 172.21.15.27:0/1331731672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8819fb40 con 0x7fdf88104d30 2024-04-25T13:47:27.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.472+0000 7fdf7effd640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf88104d30 0x7fdf881986b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800b450 tx=0x7fdf7800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:27.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.472+0000 7fdf857fa640 1 -- 172.21.15.27:0/1331731672 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78015020 con 0x7fdf88104d30 2024-04-25T13:47:27.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.472+0000 7fdf8def2640 1 -- 172.21.15.27:0/1331731672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8819fe30 con 0x7fdf88104d30 2024-04-25T13:47:27.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.472+0000 7fdf8def2640 1 -- 172.21.15.27:0/1331731672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf881a0240 con 0x7fdf88104d30 2024-04-25T13:47:27.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.473+0000 7fdf857fa640 1 -- 172.21.15.27:0/1331731672 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf78011d40 con 0x7fdf88104d30 2024-04-25T13:47:27.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.473+0000 7fdf857fa640 1 -- 172.21.15.27:0/1331731672 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78010e30 con 0x7fdf88104d30 2024-04-25T13:47:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.476+0000 7fdf94da7740 1 Processor -- start 2024-04-25T13:47:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.476+0000 7fdf94da7740 1 -- start start 2024-04-25T13:47:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acc4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acc4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acc5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf94da7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c98ac28d80 con 0x55c98a8f94d0 2024-04-25T13:47:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf94da7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c98abe2a20 con 0x55c98a9016b0 2024-04-25T13:47:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf94da7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55c98ac24d40 con 0x55c98a922390 2024-04-25T13:47:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acc4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acc4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acc4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46840/0 (socket says 172.21.15.27:46840) 2024-04-25T13:47:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.477+0000 7fdf877fe640 1 -- 172.21.15.27:0/2655373402 learned_addr learned my addr 172.21.15.27:0/2655373402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:27.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.478+0000 7fdf877fe640 1 -- 172.21.15.27:0/2655373402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 msgr2=0x55c98acc4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:27.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.478+0000 7fdf877fe640 1 --2- 172.21.15.27:0/2655373402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acc4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.478+0000 7fdf877fe640 1 -- 172.21.15.27:0/2655373402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 msgr2=0x55c98acc5a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:47:27.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.478+0000 7fdf877fe640 1 --2- 172.21.15.27:0/2655373402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acc5a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.478+0000 7fdf877fe640 1 -- 172.21.15.27:0/2655373402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c98abedc30 con 0x55c98a8f94d0 2024-04-25T13:47:27.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.478+0000 7fdf877fe640 1 --2- 172.21.15.27:0/2655373402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acc4d90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009be0 tx=0x7fdf7000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4612f55bf6e53c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:27.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.479+0000 7fdf7dffb640 1 -- 172.21.15.27:0/2655373402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70005020 con 0x55c98a8f94d0 2024-04-25T13:47:27.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.479+0000 7fdf7dffb640 1 -- 172.21.15.27:0/2655373402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf700047a0 con 0x55c98a8f94d0 2024-04-25T13:47:27.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.479+0000 7fdf7dffb640 1 -- 172.21.15.27:0/2655373402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70010690 con 0x55c98a8f94d0 2024-04-25T13:47:27.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.479+0000 7fdf857fa640 1 -- 172.21.15.27:0/1331731672 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdf78018020 con 0x7fdf88104d30 2024-04-25T13:47:27.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf94da7740 1 -- 172.21.15.27:0/2655373402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 msgr2=0x55c98acc4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf857fa640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf740afbb0 0x7fdf740b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf94da7740 1 --2- 172.21.15.27:0/2655373402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acc4d90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009be0 tx=0x7fdf7000e720 comp rx=0 tx=0).stop 2024-04-25T13:47:27.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf94da7740 1 -- 172.21.15.27:0/2655373402 shutdown_connections 2024-04-25T13:47:27.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf94da7740 1 --2- 172.21.15.27:0/2655373402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acc5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf87fff640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf740afbb0 0x7fdf740b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf94da7740 1 --2- 172.21.15.27:0/2655373402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acc4d90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf94da7740 1 --2- 172.21.15.27:0/2655373402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acc4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf94da7740 1 -- 172.21.15.27:0/2655373402 >> 172.21.15.27:0/2655373402 conn(0x55c98a9164e0 msgr2=0x55c98acc0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:27.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.480+0000 7fdf857fa640 1 -- 172.21.15.27:0/1331731672 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdf780de5f0 con 0x7fdf88104d30 2024-04-25T13:47:27.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.481+0000 7fdf94da7740 1 -- 172.21.15.27:0/2655373402 shutdown_connections 2024-04-25T13:47:27.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.481+0000 7fdf94da7740 1 -- 172.21.15.27:0/2655373402 wait complete. 2024-04-25T13:47:27.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.481+0000 7fdf87fff640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf740afbb0 0x7fdf740b20c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006010 tx=0x7fdf80040790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:27.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.482+0000 7fdf94da7740 1 -- start start 2024-04-25T13:47:27.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.482+0000 7fdf94da7740 1 Processor -- start 2024-04-25T13:47:27.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.482+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46844/0 (socket says 172.21.15.27:46844) 2024-04-25T13:47:27.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf94da7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c98acc6530 con 0x55c98a8f94d0 2024-04-25T13:47:27.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf94da7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c98abafdc0 con 0x55c98a9016b0 2024-04-25T13:47:27.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf94da7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55c98acc66b0 con 0x55c98a922390 2024-04-25T13:47:27.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf877fe640 1 -- 172.21.15.27:0/1357750101 learned_addr learned my addr 172.21.15.27:0/1357750101 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf94da7740 10 client.0 _ll_register_callbacks cb 0x55c98abcb600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf87fff640 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf7effd640 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf877fe640 1 -- 172.21.15.27:0/1357750101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 msgr2=0x55c98acd5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf877fe640 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf877fe640 1 -- 172.21.15.27:0/1357750101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 msgr2=0x55c98acceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf877fe640 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf877fe640 1 -- 172.21.15.27:0/1357750101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c98a93a3f0 con 0x55c98a8f94d0 2024-04-25T13:47:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.483+0000 7fdf87fff640 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:27.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.484+0000 7fdf877fe640 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acce630 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdf70005250 tx=0x7fdf70005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc29203aaf8ae86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.484+0000 7fdf66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.484+0000 7fdf66ffd640 1 -- 172.21.15.27:0/1357750101 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70028060 con 0x55c98a8f94d0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.484+0000 7fdf66ffd640 1 -- 172.21.15.27:0/1357750101 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdf70005420 con 0x55c98a8f94d0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.484+0000 7fdf94da7740 1 -- 172.21.15.27:0/1357750101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf700047a0 con 0x55c98a8f94d0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf66ffd640 1 -- 172.21.15.27:0/1357750101 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdf7001d040 con 0x55c98a8f94d0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf94da7740 10 client.16677 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf94da7740 1 -- 172.21.15.27:0/1357750101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55c98aa047d0 con 0x55c98a8f94d0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf66ffd640 1 -- 172.21.15.27:0/1357750101 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf700047a0 con 0x55c98a8f94d0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf66ffd640 1 -- 172.21.15.27:0/1357750101 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fdf700039a0 con 0x55c98a8f94d0 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf94da7740 10 client.16677 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf94da7740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf94da7740 20 client.16677 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf94da7740 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c98a8e0f60 0x55c98acdc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf94da7740 4 client.16677 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.485+0000 7fdf94da7740 1 -- 172.21.15.27:0/1357750101 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c98ac36990 con 0x55c98a8e0f60 2024-04-25T13:47:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.486+0000 7fdf7effd640 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c98a8e0f60 0x55c98acdc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.486+0000 7fdf7effd640 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c98a8e0f60 0x55c98acdc010 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74a8ab7b8883841b server_cookie=90e77d61581582dd in_seq=0 out_seq=0 2024-04-25T13:47:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.486+0000 7fdf66ffd640 10 client.16677 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.488+0000 7fdf66ffd640 1 -- 172.21.15.27:0/1357750101 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c98ac36990 con 0x55c98a8e0f60 2024-04-25T13:47:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.488+0000 7fdf66ffd640 10 client.16677 handle_command_reply: tid=0 2024-04-25T13:47:27.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.488+0000 7fdf94da7740 1 client.16677 shutdown 2024-04-25T13:47:27.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.488+0000 7fdf94da7740 1 -- 172.21.15.27:0/1357750101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 msgr2=0x55c98acce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.488+0000 7fdf94da7740 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acce630 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdf70005250 tx=0x7fdf70005280 comp rx=0 tx=0).stop 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.488+0000 7fdf94da7740 1 -- 172.21.15.27:0/1357750101 shutdown_connections 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.488+0000 7fdf94da7740 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c98a8e0f60 0x55c98acdc010 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.488+0000 7fdf94da7740 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.489+0000 7fdf94da7740 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.489+0000 7fdf94da7740 1 --2- 172.21.15.27:0/1357750101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acce630 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.489+0000 7fdf94da7740 1 -- 172.21.15.27:0/1357750101 >> 172.21.15.27:0/1357750101 conn(0x55c98a9164e0 msgr2=0x55c98acc2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.489+0000 7fdf66ffd640 0 client.16677 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.489+0000 7fdf94da7740 1 -- 172.21.15.27:0/1357750101 shutdown_connections 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.489+0000 7fdf94da7740 1 -- 172.21.15.27:0/1357750101 wait complete. 2024-04-25T13:47:27.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.489+0000 7fdf94da7740 20 client.16677 trim_cache size 0 max 16384 2024-04-25T13:47:27.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.509+0000 7fdf94da7740 1 Processor -- start 2024-04-25T13:47:27.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.509+0000 7fdf94da7740 1 -- start start 2024-04-25T13:47:27.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.509+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a8f94d0 0x55c98acbfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.509+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a9016b0 0x55c98acbff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a922390 0x55c98acccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf94da7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6800c380 con 0x55c98a922390 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf94da7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6800ec00 con 0x55c98a8f94d0 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf94da7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6800c9f0 con 0x55c98a9016b0 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a8f94d0 0x55c98acbfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a922390 0x55c98acccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a8f94d0 0x55c98acbfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34410/0 (socket says 172.21.15.27:34410) 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 -- 172.21.15.27:0/2608731702 learned_addr learned my addr 172.21.15.27:0/2608731702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf7effd640 1 --2- 172.21.15.27:0/2608731702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a9016b0 0x55c98acbff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 -- 172.21.15.27:0/2608731702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a9016b0 msgr2=0x55c98acbff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 --2- 172.21.15.27:0/2608731702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a9016b0 0x55c98acbff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 -- 172.21.15.27:0/2608731702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a922390 msgr2=0x55c98acccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 --2- 172.21.15.27:0/2608731702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a922390 0x55c98acccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 -- 172.21.15.27:0/2608731702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c98accc6d0 con 0x55c98a8f94d0 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf877fe640 1 --2- 172.21.15.27:0/2608731702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a8f94d0 0x55c98acbfa50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000f010 tx=0x7fdf70004d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a91149aeb7ca94d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:27.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.510+0000 7fdf7d7fa640 1 -- 172.21.15.27:0/2608731702 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70033040 con 0x55c98a8f94d0 2024-04-25T13:47:27.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.511+0000 7fdf7d7fa640 1 -- 172.21.15.27:0/2608731702 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf700047a0 con 0x55c98a8f94d0 2024-04-25T13:47:27.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.511+0000 7fdf7d7fa640 1 -- 172.21.15.27:0/2608731702 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70028040 con 0x55c98a8f94d0 2024-04-25T13:47:27.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.512+0000 7fdf94da7740 1 -- 172.21.15.27:0/2608731702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a8f94d0 msgr2=0x55c98acbfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.512+0000 7fdf94da7740 1 --2- 172.21.15.27:0/2608731702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a8f94d0 0x55c98acbfa50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000f010 tx=0x7fdf70004d50 comp rx=0 tx=0).stop 2024-04-25T13:47:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.512+0000 7fdf94da7740 1 -- 172.21.15.27:0/2608731702 shutdown_connections 2024-04-25T13:47:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.512+0000 7fdf94da7740 1 --2- 172.21.15.27:0/2608731702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a922390 0x55c98acccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.512+0000 7fdf94da7740 1 --2- 172.21.15.27:0/2608731702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a9016b0 0x55c98acbff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.512+0000 7fdf94da7740 1 --2- 172.21.15.27:0/2608731702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a8f94d0 0x55c98acbfa50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.512+0000 7fdf94da7740 1 -- 172.21.15.27:0/2608731702 >> 172.21.15.27:0/2608731702 conn(0x55c98a9164e0 msgr2=0x55c98acc9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.512+0000 7fdf94da7740 1 -- 172.21.15.27:0/2608731702 shutdown_connections 2024-04-25T13:47:27.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.513+0000 7fdf94da7740 1 -- 172.21.15.27:0/2608731702 wait complete. 2024-04-25T13:47:27.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.513+0000 7fdf94da7740 1 -- start start 2024-04-25T13:47:27.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf94da7740 1 Processor -- start 2024-04-25T13:47:27.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acd4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acd50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf94da7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf94da7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6800d0c0 con 0x55c98a8f94d0 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf94da7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6800e530 con 0x55c98a9016b0 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf94da7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6800d790 con 0x55c98a922390 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf94da7740 10 client.0 _ll_register_callbacks cb 0x7fdf70005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acd4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acd4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46854/0 (socket says 172.21.15.27:46854) 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf877fe640 1 -- 172.21.15.27:0/4173186593 learned_addr learned my addr 172.21.15.27:0/4173186593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf87fff640 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.514+0000 7fdf7effd640 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acd50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf877fe640 1 -- 172.21.15.27:0/4173186593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 msgr2=0x55c98acd1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf877fe640 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf877fe640 1 -- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 msgr2=0x55c98acd50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf877fe640 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acd50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf877fe640 1 -- 172.21.15.27:0/4173186593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c98acd1fc0 con 0x55c98a8f94d0 2024-04-25T13:47:27.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf7effd640 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acd50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:27.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf877fe640 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acd4ba0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x55c98ac24e20 tx=0x7fdf70004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8630954ca673a2cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:27.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:27.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf66ffd640 1 -- 172.21.15.27:0/4173186593 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70033040 con 0x55c98a8f94d0 2024-04-25T13:47:27.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf66ffd640 1 -- 172.21.15.27:0/4173186593 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdf70005230 con 0x55c98a8f94d0 2024-04-25T13:47:27.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf94da7740 1 -- 172.21.15.27:0/4173186593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf700047a0 con 0x55c98a8f94d0 2024-04-25T13:47:27.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.515+0000 7fdf66ffd640 1 -- 172.21.15.27:0/4173186593 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdf70048070 con 0x55c98a8f94d0 2024-04-25T13:47:27.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.516+0000 7fdf94da7740 10 client.16683 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.516+0000 7fdf94da7740 1 -- 172.21.15.27:0/4173186593 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55c98ad430b0 con 0x55c98a8f94d0 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.516+0000 7fdf66ffd640 1 -- 172.21.15.27:0/4173186593 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf700047a0 con 0x55c98a8f94d0 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.516+0000 7fdf66ffd640 1 -- 172.21.15.27:0/4173186593 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fdf70003850 con 0x55c98a8f94d0 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.516+0000 7fdf94da7740 10 client.16683 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.517+0000 7fdf94da7740 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.517+0000 7fdf94da7740 20 client.16683 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.517+0000 7fdf94da7740 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c98a8b63a0 0x55c98ad45a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.517+0000 7fdf94da7740 4 client.16683 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:47:27.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.517+0000 7fdf94da7740 1 -- 172.21.15.27:0/4173186593 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c98ac30800 con 0x55c98a8b63a0 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.517+0000 7fdf7effd640 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c98a8b63a0 0x55c98ad45a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf7effd640 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c98a8b63a0 0x55c98ad45a40 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7189cc83557e2538 server_cookie=4d29b34924cd83d9 in_seq=0 out_seq=0 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf66ffd640 10 client.16683 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf66ffd640 1 -- 172.21.15.27:0/4173186593 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c98ac30800 con 0x55c98a8b63a0 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf66ffd640 10 client.16683 handle_command_reply: tid=0 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 client.16683 shutdown 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 -- 172.21.15.27:0/4173186593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 msgr2=0x55c98acd4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acd4ba0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x55c98ac24e20 tx=0x7fdf70004c60 comp rx=0 tx=0).stop 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 -- 172.21.15.27:0/4173186593 shutdown_connections 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c98a8b63a0 0x55c98ad45a40 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf66ffd640 0 client.16683 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c98a922390 0x55c98acd1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c98a9016b0 0x55c98acd50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 --2- 172.21.15.27:0/4173186593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c98a8f94d0 0x55c98acd4ba0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 -- 172.21.15.27:0/4173186593 >> 172.21.15.27:0/4173186593 conn(0x55c98a9164e0 msgr2=0x55c98acccd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:27.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 -- 172.21.15.27:0/4173186593 shutdown_connections 2024-04-25T13:47:27.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 1 -- 172.21.15.27:0/4173186593 wait complete. 2024-04-25T13:47:27.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.518+0000 7fdf94da7740 20 client.16683 trim_cache size 0 max 16384 2024-04-25T13:47:27.524 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.519+0000 7fdf8def2640 1 -- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf740afbb0 msgr2=0x7fdf740b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.519+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf740afbb0 0x7fdf740b20c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006010 tx=0x7fdf80040790 comp rx=0 tx=0).stop 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.519+0000 7fdf8def2640 1 -- 172.21.15.27:0/1331731672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf88104d30 msgr2=0x7fdf881986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.519+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf88104d30 0x7fdf881986b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800b450 tx=0x7fdf7800b910 comp rx=0 tx=0).stop 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.519+0000 7fdf8def2640 1 -- 172.21.15.27:0/1331731672 shutdown_connections 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.519+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf740afbb0 0x7fdf740b20c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.519+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf88107690 0x7fdf8819f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.519+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf88104d30 0x7fdf881986b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.520+0000 7fdf8def2640 1 --2- 172.21.15.27:0/1331731672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf881023d0 0x7fdf88198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:27.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.520+0000 7fdf8def2640 1 -- 172.21.15.27:0/1331731672 >> 172.21.15.27:0/1331731672 conn(0x7fdf880fdc00 msgr2=0x7fdf88100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:27.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.521+0000 7fdf8def2640 1 -- 172.21.15.27:0/1331731672 shutdown_connections 2024-04-25T13:47:27.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:27.521+0000 7fdf8def2640 1 -- 172.21.15.27:0/1331731672 wait complete. 2024-04-25T13:47:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 380.0 s, 2273.6 tps, lat 43.495 ms stddev 52.108 2024-04-25T13:47:28.536 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:47:28.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:47:28.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:47:28.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:28 smithi122 ceph-mon[32843]: pgmap v546: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 920 op/s 2024-04-25T13:47:28.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:28 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1127874512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:47:28.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:28 smithi027 ceph-mon[29140]: pgmap v546: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 920 op/s 2024-04-25T13:47:28.565 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:28 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1127874512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:47:28.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:28 smithi045 ceph-mon[32731]: pgmap v546: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 920 op/s 2024-04-25T13:47:28.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:28 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1127874512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:47:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.694+0000 7fc11e7fc640 1 -- 172.21.15.27:0/781002166 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1100035f0 con 0x7fc120103c20 2024-04-25T13:47:28.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.696+0000 7fc125d57640 1 -- 172.21.15.27:0/781002166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 msgr2=0x7fc120104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:28.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.696+0000 7fc125d57640 1 --2- 172.21.15.27:0/781002166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 0x7fc120104030 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc110009900 tx=0x7fc11002a420 comp rx=0 tx=0).stop 2024-04-25T13:47:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.698+0000 7fc125d57640 1 -- 172.21.15.27:0/781002166 shutdown_connections 2024-04-25T13:47:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.698+0000 7fc125d57640 1 --2- 172.21.15.27:0/781002166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc120108920 0x7fc12010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.698+0000 7fc125d57640 1 --2- 172.21.15.27:0/781002166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 0x7fc120104030 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.698+0000 7fc125d57640 1 --2- 172.21.15.27:0/781002166 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1201029e0 0x7fc1201036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.698+0000 7fc125d57640 1 -- 172.21.15.27:0/781002166 >> 172.21.15.27:0/781002166 conn(0x7fc1200fdbd0 msgr2=0x7fc120100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:28.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.699+0000 7fc125d57640 1 -- 172.21.15.27:0/781002166 shutdown_connections 2024-04-25T13:47:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.700+0000 7fc125d57640 1 -- 172.21.15.27:0/781002166 wait complete. 2024-04-25T13:47:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.701+0000 7fc125d57640 1 Processor -- start 2024-04-25T13:47:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc125d57640 1 -- start start 2024-04-25T13:47:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc125d57640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1201029e0 0x7fc1201a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc125d57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 0x7fc1201a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc125d57640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc120108920 0x7fc1201a8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc125d57640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc12010e530 con 0x7fc120108920 2024-04-25T13:47:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc125d57640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc12010e3b0 con 0x7fc120103c20 2024-04-25T13:47:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc125d57640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc12010e6b0 con 0x7fc1201029e0 2024-04-25T13:47:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc11ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc120108920 0x7fc1201a8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc11effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 0x7fc1201a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc11effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 0x7fc1201a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34420/0 (socket says 172.21.15.27:34420) 2024-04-25T13:47:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.702+0000 7fc11effd640 1 -- 172.21.15.27:0/3740523295 learned_addr learned my addr 172.21.15.27:0/3740523295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.703+0000 7fc11effd640 1 -- 172.21.15.27:0/3740523295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1201029e0 msgr2=0x7fc1201a0b80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:47:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.703+0000 7fc11effd640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1201029e0 0x7fc1201a0b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.703+0000 7fc11effd640 1 -- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc120108920 msgr2=0x7fc1201a8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.703+0000 7fc11effd640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc120108920 0x7fc1201a8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.703+0000 7fc11effd640 1 -- 172.21.15.27:0/3740523295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1201a8600 con 0x7fc120103c20 2024-04-25T13:47:28.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.704+0000 7fc11effd640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 0x7fc1201a10c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc11002aa10 tx=0x7fc110003e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:28.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.704+0000 7fc11cff9640 1 -- 172.21.15.27:0/3740523295 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110040030 con 0x7fc120103c20 2024-04-25T13:47:28.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.704+0000 7fc125d57640 1 -- 172.21.15.27:0/3740523295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1201a8890 con 0x7fc120103c20 2024-04-25T13:47:28.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.704+0000 7fc125d57640 1 -- 172.21.15.27:0/3740523295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1201a8dd0 con 0x7fc120103c20 2024-04-25T13:47:28.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.704+0000 7fc11cff9640 1 -- 172.21.15.27:0/3740523295 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc110004350 con 0x7fc120103c20 2024-04-25T13:47:28.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.704+0000 7fc11cff9640 1 -- 172.21.15.27:0/3740523295 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11003c6a0 con 0x7fc120103c20 2024-04-25T13:47:28.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.708+0000 7fc12cc0a740 1 Processor -- start 2024-04-25T13:47:28.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.709+0000 7fc11cff9640 1 -- 172.21.15.27:0/3740523295 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc110043030 con 0x7fc120103c20 2024-04-25T13:47:28.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.710+0000 7fc12cc0a740 1 -- start start 2024-04-25T13:47:28.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.710+0000 7fc11cff9640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1080afc30 0x7fc1080b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.711+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x565480ffe4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.711+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c34550 0x565480ffea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.711+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c3c730 0x565481005920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.711+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565480f63b00 con 0x565480c3c730 2024-04-25T13:47:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.711+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x565480eff920 con 0x565480c34550 2024-04-25T13:47:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.711+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x565480f5fac0 con 0x565480c5d410 2024-04-25T13:47:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.711+0000 7fc11cff9640 1 -- 172.21.15.27:0/3740523295 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc110100540 con 0x7fc120103c20 2024-04-25T13:47:28.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.712+0000 7fc11f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x565480ffe4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.712+0000 7fc11f7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x565480ffe4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58564/0 (socket says 172.21.15.27:58564) 2024-04-25T13:47:28.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.712+0000 7fc11f7fe640 1 -- 172.21.15.27:0/916448588 learned_addr learned my addr 172.21.15.27:0/916448588 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.713+0000 7fc11ffff640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1080afc30 0x7fc1080b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.713+0000 7fc11ffff640 1 --2- 172.21.15.27:0/916448588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c3c730 0x565481005920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.714+0000 7fc11f7fe640 1 -- 172.21.15.27:0/916448588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c34550 msgr2=0x565480ffea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.714+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/916448588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c34550 0x565480ffea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.714+0000 7fc11f7fe640 1 -- 172.21.15.27:0/916448588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c3c730 msgr2=0x565481005920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.714+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/916448588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c3c730 0x565481005920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.714+0000 7fc11f7fe640 1 -- 172.21.15.27:0/916448588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565480d3d7e0 con 0x565480c5d410 2024-04-25T13:47:28.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.715+0000 7fc11ffff640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1080afc30 0x7fc1080b2140 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc1180127a0 tx=0x7fc11800a320 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.717+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/916448588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x565480ffe4a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc11400a940 tx=0x7fc11400ae00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ed5b9f287f0e397 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.718+0000 7fc105ffb640 1 -- 172.21.15.27:0/916448588 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11400cd80 con 0x565480c5d410 2024-04-25T13:47:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.718+0000 7fc105ffb640 1 -- 172.21.15.27:0/916448588 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc114005020 con 0x565480c5d410 2024-04-25T13:47:28.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.718+0000 7fc105ffb640 1 -- 172.21.15.27:0/916448588 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc114004a90 con 0x565480c5d410 2024-04-25T13:47:28.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.723+0000 7fc12cc0a740 1 -- 172.21.15.27:0/916448588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 msgr2=0x565480ffe4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:28.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.723+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/916448588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x565480ffe4a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc11400a940 tx=0x7fc11400ae00 comp rx=0 tx=0).stop 2024-04-25T13:47:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.725+0000 7fc12cc0a740 1 -- 172.21.15.27:0/916448588 shutdown_connections 2024-04-25T13:47:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.725+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/916448588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c3c730 0x565481005920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.725+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/916448588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c34550 0x565480ffea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.725+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/916448588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x565480ffe4a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.725+0000 7fc12cc0a740 1 -- 172.21.15.27:0/916448588 >> 172.21.15.27:0/916448588 conn(0x565480c51560 msgr2=0x565480ff9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.725+0000 7fc12cc0a740 1 -- 172.21.15.27:0/916448588 shutdown_connections 2024-04-25T13:47:28.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.726+0000 7fc12cc0a740 1 -- 172.21.15.27:0/916448588 wait complete. 2024-04-25T13:47:28.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.727+0000 7fc12cc0a740 1 -- start start 2024-04-25T13:47:28.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.727+0000 7fc12cc0a740 1 Processor -- start 2024-04-25T13:47:28.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.727+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x5654810193b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.727+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c3c730 0x5654810198f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.727+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c5d410 0x565481020850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.727+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565481017380 con 0x565480c34550 2024-04-25T13:47:28.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.727+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x565480efd5d0 con 0x565480c5d410 2024-04-25T13:47:28.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.727+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x565481017500 con 0x565480c3c730 2024-04-25T13:47:28.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.728+0000 7fc12cc0a740 10 client.0 _ll_register_callbacks cb 0x565480eeb080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:28.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c3c730 0x5654810198f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c3c730 0x5654810198f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58568/0 (socket says 172.21.15.27:58568) 2024-04-25T13:47:28.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11effd640 1 -- 172.21.15.27:0/69683171 learned_addr learned my addr 172.21.15.27:0/69683171 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:28.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11ffff640 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c5d410 0x565481020850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x5654810193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11effd640 1 -- 172.21.15.27:0/69683171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c5d410 msgr2=0x565481020850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:28.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11effd640 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c5d410 0x565481020850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11effd640 1 -- 172.21.15.27:0/69683171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 msgr2=0x5654810193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:28.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11effd640 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x5654810193b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.729+0000 7fc11effd640 1 -- 172.21.15.27:0/69683171 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565480d9df70 con 0x565480c3c730 2024-04-25T13:47:28.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.730+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x5654810193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:47:28.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.730+0000 7fc11effd640 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c3c730 0x5654810198f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc1100c5c60 tx=0x7fc1100c8160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3ba2efd19dfc144 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.730+0000 7fc0feffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:47:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.731+0000 7fc0feffd640 1 -- 172.21.15.27:0/69683171 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11004a040 con 0x565480c3c730 2024-04-25T13:47:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.731+0000 7fc0feffd640 1 -- 172.21.15.27:0/69683171 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc1100c8c50 con 0x565480c3c730 2024-04-25T13:47:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.731+0000 7fc12cc0a740 1 -- 172.21.15.27:0/69683171 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc114005020 con 0x565480c3c730 2024-04-25T13:47:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.731+0000 7fc0feffd640 1 -- 172.21.15.27:0/69683171 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc110100300 con 0x565480c3c730 2024-04-25T13:47:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.731+0000 7fc12cc0a740 10 client.26437 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:28.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.731+0000 7fc12cc0a740 1 -- 172.21.15.27:0/69683171 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x565480c35170 con 0x565480c3c730 2024-04-25T13:47:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc0feffd640 1 -- 172.21.15.27:0/69683171 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc114005020 con 0x565480c3c730 2024-04-25T13:47:28.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc0feffd640 1 -- 172.21.15.27:0/69683171 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc1100eb070 con 0x565480c3c730 2024-04-25T13:47:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc12cc0a740 10 client.26437 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc12cc0a740 10 client.26437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc12cc0a740 20 client.26437 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565480c1c1f0 0x565481026c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc12cc0a740 4 client.26437 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc12cc0a740 1 -- 172.21.15.27:0/69683171 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565480f71710 con 0x565480c1c1f0 2024-04-25T13:47:28.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.733+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565480c1c1f0 0x565481026c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.734+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565480c1c1f0 0x565481026c70 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa705f60e54e0810 server_cookie=f1ba5ed53e6a744e in_seq=0 out_seq=0 2024-04-25T13:47:28.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.734+0000 7fc0feffd640 10 client.26437 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:28.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.735+0000 7fc0feffd640 1 -- 172.21.15.27:0/69683171 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x565480f71710 con 0x565480c1c1f0 2024-04-25T13:47:28.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.735+0000 7fc0feffd640 10 client.26437 handle_command_reply: tid=0 2024-04-25T13:47:28.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.735+0000 7fc12cc0a740 1 client.26437 shutdown 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 -- 172.21.15.27:0/69683171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c3c730 msgr2=0x5654810198f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c3c730 0x5654810198f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc1100c5c60 tx=0x7fc1100c8160 comp rx=0 tx=0).stop 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 -- 172.21.15.27:0/69683171 shutdown_connections 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565480c1c1f0 0x565481026c70 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c5d410 0x565481020850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c3c730 0x5654810198f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc0feffd640 0 client.26437 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/69683171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x5654810193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 -- 172.21.15.27:0/69683171 >> 172.21.15.27:0/69683171 conn(0x565480c51560 msgr2=0x565480ff8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 -- 172.21.15.27:0/69683171 shutdown_connections 2024-04-25T13:47:28.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 1 -- 172.21.15.27:0/69683171 wait complete. 2024-04-25T13:47:28.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.736+0000 7fc12cc0a740 20 client.26437 trim_cache size 0 max 16384 2024-04-25T13:47:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.760+0000 7fc12cc0a740 1 Processor -- start 2024-04-25T13:47:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.760+0000 7fc12cc0a740 1 -- start start 2024-04-25T13:47:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.760+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c34550 0x565480ffd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.760+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x565480ffd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.760+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c5d410 0x565481020770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.760+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c00c380 con 0x565480c5d410 2024-04-25T13:47:28.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.760+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c00ec00 con 0x565480c3c730 2024-04-25T13:47:28.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.760+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c00c9f0 con 0x565480c34550 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c5d410 0x565481020770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c5d410 0x565481020770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46898/0 (socket says 172.21.15.27:46898) 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x565480ffd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11ffff640 1 -- 172.21.15.27:0/2307640184 learned_addr learned my addr 172.21.15.27:0/2307640184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11effd640 1 -- 172.21.15.27:0/2307640184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c34550 msgr2=0x565480ffd340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11effd640 1 --2- 172.21.15.27:0/2307640184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c34550 0x565480ffd340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11effd640 1 -- 172.21.15.27:0/2307640184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c5d410 msgr2=0x565481020770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11effd640 1 --2- 172.21.15.27:0/2307640184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c5d410 0x565481020770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11effd640 1 -- 172.21.15.27:0/2307640184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654810180d0 con 0x565480c3c730 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.761+0000 7fc11ffff640 1 --2- 172.21.15.27:0/2307640184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c5d410 0x565481020770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.762+0000 7fc11effd640 1 --2- 172.21.15.27:0/2307640184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x565480ffd880 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc11004b000 tx=0x7fc1100c8290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f5b11a10a59d389 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.762+0000 7fc1057fa640 1 -- 172.21.15.27:0/2307640184 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11004a070 con 0x565480c3c730 2024-04-25T13:47:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.762+0000 7fc1057fa640 1 -- 172.21.15.27:0/2307640184 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1100dfe50 con 0x565480c3c730 2024-04-25T13:47:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.762+0000 7fc1057fa640 1 -- 172.21.15.27:0/2307640184 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1100c4570 con 0x565480c3c730 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 -- 172.21.15.27:0/2307640184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 msgr2=0x565480ffd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/2307640184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x565480ffd880 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc11004b000 tx=0x7fc1100c8290 comp rx=0 tx=0).stop 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 -- 172.21.15.27:0/2307640184 shutdown_connections 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/2307640184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c5d410 0x565481020770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/2307640184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x565480ffd880 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/2307640184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c34550 0x565480ffd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 -- 172.21.15.27:0/2307640184 >> 172.21.15.27:0/2307640184 conn(0x565480c51560 msgr2=0x565480ffcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 -- 172.21.15.27:0/2307640184 shutdown_connections 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 -- 172.21.15.27:0/2307640184 wait complete. 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.763+0000 7fc12cc0a740 1 -- start start 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc12cc0a740 1 Processor -- start 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x565481020720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x56548108b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc12cc0a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x56548108d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c00d0c0 con 0x565480c34550 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c00e530 con 0x565480c3c730 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc12cc0a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c00d790 con 0x565480c5d410 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc12cc0a740 10 client.0 _ll_register_callbacks cb 0x7fc1100c87c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc11effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x56548108b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc11f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x565481020720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc11effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x56548108b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34444/0 (socket says 172.21.15.27:34444) 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc11effd640 1 -- 172.21.15.27:0/3118765831 learned_addr learned my addr 172.21.15.27:0/3118765831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc11ffff640 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x56548108d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.764+0000 7fc11effd640 1 -- 172.21.15.27:0/3118765831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 msgr2=0x56548108d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.765+0000 7fc11effd640 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x56548108d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.765+0000 7fc11effd640 1 -- 172.21.15.27:0/3118765831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 msgr2=0x565481020720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.765+0000 7fc11effd640 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x565481020720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.765+0000 7fc11effd640 1 -- 172.21.15.27:0/3118765831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56548108dd40 con 0x565480c3c730 2024-04-25T13:47:28.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.766+0000 7fc11effd640 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x56548108b000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc1100c5c60 tx=0x7fc11002a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eea869ac2ac556a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:28.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.766+0000 7fc0feffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:47:28.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc0feffd640 1 -- 172.21.15.27:0/3118765831 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11004a050 con 0x565480c3c730 2024-04-25T13:47:28.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc0feffd640 1 -- 172.21.15.27:0/3118765831 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc1100c8e60 con 0x565480c3c730 2024-04-25T13:47:28.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 1 -- 172.21.15.27:0/3118765831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1100dfe50 con 0x565480c3c730 2024-04-25T13:47:28.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc0feffd640 1 -- 172.21.15.27:0/3118765831 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc1100ccda0 con 0x565480c3c730 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 10 client.26402 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 1 -- 172.21.15.27:0/3118765831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56548108dd40 con 0x565480c3c730 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc0feffd640 1 -- 172.21.15.27:0/3118765831 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1100c4040 con 0x565480c3c730 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc0feffd640 1 -- 172.21.15.27:0/3118765831 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc1100c78c0 con 0x565480c3c730 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 10 client.26402 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 10 client.26402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 20 client.26402 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565480bf1630 0x565481091600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 4 client.26402 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.767+0000 7fc12cc0a740 1 -- 172.21.15.27:0/3118765831 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fc1100dfe50 con 0x565480bf1630 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.768+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565480bf1630 0x565481091600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:28.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.768+0000 7fc0feffd640 10 client.26402 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.768+0000 7fc11f7fe640 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565480bf1630 0x565481091600 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=244cf45ddf4be91 server_cookie=83e931a3f5d19842 in_seq=0 out_seq=0 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc0feffd640 1 -- 172.21.15.27:0/3118765831 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fc1100dfe50 con 0x565480bf1630 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc0feffd640 10 client.26402 handle_command_reply: tid=0 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 client.26402 shutdown 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 -- 172.21.15.27:0/3118765831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 msgr2=0x56548108b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x56548108b000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc1100c5c60 tx=0x7fc11002a420 comp rx=0 tx=0).stop 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 -- 172.21.15.27:0/3118765831 shutdown_connections 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565480bf1630 0x565481091600 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc0feffd640 0 client.26402 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565480c5d410 0x56548108d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565480c3c730 0x56548108b000 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 --2- 172.21.15.27:0/3118765831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565480c34550 0x565481020720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 -- 172.21.15.27:0/3118765831 >> 172.21.15.27:0/3118765831 conn(0x565480c51560 msgr2=0x565480ffd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 -- 172.21.15.27:0/3118765831 shutdown_connections 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 1 -- 172.21.15.27:0/3118765831 wait complete. 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.770+0000 7fc12cc0a740 20 client.26402 trim_cache size 0 max 16384 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "ac368991-79a7-419e-93cc-94592170b10d", 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:47:28.773 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:47:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 -- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1080afc30 msgr2=0x7fc1080b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1080afc30 0x7fc1080b2140 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc1180127a0 tx=0x7fc11800a320 comp rx=0 tx=0).stop 2024-04-25T13:47:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 -- 172.21.15.27:0/3740523295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 msgr2=0x7fc1201a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 0x7fc1201a10c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc11002aa10 tx=0x7fc110003e90 comp rx=0 tx=0).stop 2024-04-25T13:47:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 -- 172.21.15.27:0/3740523295 shutdown_connections 2024-04-25T13:47:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc1080afc30 0x7fc1080b2140 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc120108920 0x7fc1201a8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc120103c20 0x7fc1201a10c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 --2- 172.21.15.27:0/3740523295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc1201029e0 0x7fc1201a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.772+0000 7fc125d57640 1 -- 172.21.15.27:0/3740523295 >> 172.21.15.27:0/3740523295 conn(0x7fc1200fdbd0 msgr2=0x7fc1200ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.773+0000 7fc125d57640 1 -- 172.21.15.27:0/3740523295 shutdown_connections 2024-04-25T13:47:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:28.773+0000 7fc125d57640 1 -- 172.21.15.27:0/3740523295 wait complete. 2024-04-25T13:47:28.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:47:28.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.017+0000 7fb627867640 1 -- 172.21.15.27:0/2954426453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 msgr2=0x7fb6201036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.017+0000 7fb627867640 1 --2- 172.21.15.27:0/2954426453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 0x7fb6201036e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009900 tx=0x7fb61c02a410 comp rx=0 tx=0).stop 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.018+0000 7fb627867640 1 -- 172.21.15.27:0/2954426453 shutdown_connections 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.018+0000 7fb627867640 1 --2- 172.21.15.27:0/2954426453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb620108920 0x7fb62010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.018+0000 7fb627867640 1 --2- 172.21.15.27:0/2954426453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb620103c20 0x7fb620104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.018+0000 7fb627867640 1 --2- 172.21.15.27:0/2954426453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 0x7fb6201036e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.018+0000 7fb627867640 1 -- 172.21.15.27:0/2954426453 >> 172.21.15.27:0/2954426453 conn(0x7fb6200fdbd0 msgr2=0x7fb620100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.018+0000 7fb627867640 1 -- 172.21.15.27:0/2954426453 shutdown_connections 2024-04-25T13:47:29.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.018+0000 7fb627867640 1 -- 172.21.15.27:0/2954426453 wait complete. 2024-04-25T13:47:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb627867640 1 Processor -- start 2024-04-25T13:47:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb627867640 1 -- start start 2024-04-25T13:47:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb627867640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 0x7fb620198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb627867640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb620103c20 0x7fb620198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb625603640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 0x7fb620198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb625603640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 0x7fb620198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46918/0 (socket says 172.21.15.27:46918) 2024-04-25T13:47:29.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb625603640 1 -- 172.21.15.27:0/1355828705 learned_addr learned my addr 172.21.15.27:0/1355828705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb627867640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb620108920 0x7fb62019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb62010d400 con 0x7fb6201029e0 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb62010d280 con 0x7fb620103c20 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb62010d580 con 0x7fb620108920 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb625e04640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb620108920 0x7fb62019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.019+0000 7fb624e02640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb620103c20 0x7fb620198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb625e04640 1 -- 172.21.15.27:0/1355828705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb620103c20 msgr2=0x7fb620198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb625e04640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb620103c20 0x7fb620198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb625e04640 1 -- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 msgr2=0x7fb620198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb625e04640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 0x7fb620198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb625e04640 1 -- 172.21.15.27:0/1355828705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62019fe70 con 0x7fb620108920 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb625603640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 0x7fb620198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb625e04640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb620108920 0x7fb62019f800 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb618010f90 tx=0x7fb61800e490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:29.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb6167fc640 1 -- 172.21.15.27:0/1355828705 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61801a070 con 0x7fb620108920 2024-04-25T13:47:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb6167fc640 1 -- 172.21.15.27:0/1355828705 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb61800be40 con 0x7fb620108920 2024-04-25T13:47:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb6167fc640 1 -- 172.21.15.27:0/1355828705 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6180056a0 con 0x7fb620108920 2024-04-25T13:47:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6201a0160 con 0x7fb620108920 2024-04-25T13:47:29.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.020+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6201a0660 con 0x7fb620108920 2024-04-25T13:47:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.024+0000 7fb62e7f8740 1 Processor -- start 2024-04-25T13:47:29.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.024+0000 7fb6167fc640 1 -- 172.21.15.27:0/1355828705 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb618014050 con 0x7fb620108920 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.024+0000 7fb62e7f8740 1 -- start start 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 0x563e0c8c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563e0c825d80 con 0x563e0c4f64d0 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563e0c7df8b0 con 0x563e0c51f390 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563e0c821d40 con 0x563e0c4fe6b0 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb624e02640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 0x563e0c8c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb625603640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb625603640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46928/0 (socket says 172.21.15.27:46928) 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb625603640 1 -- 172.21.15.27:0/948443236 learned_addr learned my addr 172.21.15.27:0/948443236 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:29.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb624e02640 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb625603640 1 -- 172.21.15.27:0/948443236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 msgr2=0x563e0c8c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb625603640 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb625603640 1 -- 172.21.15.27:0/948443236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 msgr2=0x563e0c8c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb625603640 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 0x563e0c8c4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb625603640 1 -- 172.21.15.27:0/948443236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e0c7b0a40 con 0x563e0c4f64d0 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.025+0000 7fb624e02640 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.026+0000 7fb6167fc640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb60c0afaf0 0x7fb60c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.026+0000 7fb625603640 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8c53f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb61c02a930 tx=0x7fb61c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d122e41d4ed4700a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.026+0000 7fb60b7fe640 1 -- 172.21.15.27:0/948443236 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c035030 con 0x563e0c4f64d0 2024-04-25T13:47:29.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.026+0000 7fb60b7fe640 1 -- 172.21.15.27:0/948443236 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb61c040070 con 0x563e0c4f64d0 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.026+0000 7fb624e02640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb60c0afaf0 0x7fb60c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.026+0000 7fb6167fc640 1 -- 172.21.15.27:0/1355828705 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb6180e0cf0 con 0x7fb620108920 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.026+0000 7fb60b7fe640 1 -- 172.21.15.27:0/948443236 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c031d70 con 0x563e0c4f64d0 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 -- 172.21.15.27:0/948443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 msgr2=0x563e0c8c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8c53f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb61c02a930 tx=0x7fb61c004470 comp rx=0 tx=0).stop 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb624e02640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb60c0afaf0 0x7fb60c0b2000 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x563e0c8c5ec0 tx=0x7fb610004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 -- 172.21.15.27:0/948443236 shutdown_connections 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8c53f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/948443236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 0x563e0c8c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 -- 172.21.15.27:0/948443236 >> 172.21.15.27:0/948443236 conn(0x563e0c5134e0 msgr2=0x563e0c8bc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 -- 172.21.15.27:0/948443236 shutdown_connections 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 -- 172.21.15.27:0/948443236 wait complete. 2024-04-25T13:47:29.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 -- start start 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.027+0000 7fb62e7f8740 1 Processor -- start 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8cec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 0x563e0c8d6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563e0c8c6b80 con 0x563e0c4f64d0 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563e0c7acd10 con 0x563e0c51f390 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563e0c8c6d00 con 0x563e0c4fe6b0 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb62e7f8740 10 client.0 _ll_register_callbacks cb 0x563e0c7d9210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb625603640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb625603640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8cec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46940/0 (socket says 172.21.15.27:46940) 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb624e02640 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb625603640 1 -- 172.21.15.27:0/259914335 learned_addr learned my addr 172.21.15.27:0/259914335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:29.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb625e04640 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 0x563e0c8d6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb624e02640 1 -- 172.21.15.27:0/259914335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 msgr2=0x563e0c8d6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb624e02640 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 0x563e0c8d6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb624e02640 1 -- 172.21.15.27:0/259914335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 msgr2=0x563e0c8cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb624e02640 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.028+0000 7fb624e02640 1 -- 172.21.15.27:0/259914335 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e0c6017d0 con 0x563e0c4fe6b0 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.029+0000 7fb624e02640 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8cf1d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb6201993d0 tx=0x7fb610005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5cff32af78e131d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.029+0000 7fb608ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.029+0000 7fb608ff9640 1 -- 172.21.15.27:0/259914335 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb610022640 con 0x563e0c4fe6b0 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.029+0000 7fb62e7f8740 1 -- 172.21.15.27:0/259914335 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb61c040070 con 0x563e0c4fe6b0 2024-04-25T13:47:29.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.029+0000 7fb608ff9640 1 -- 172.21.15.27:0/259914335 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb6100227e0 con 0x563e0c4fe6b0 2024-04-25T13:47:29.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.029+0000 7fb608ff9640 1 -- 172.21.15.27:0/259914335 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb6100229e0 con 0x563e0c4fe6b0 2024-04-25T13:47:29.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.029+0000 7fb62e7f8740 10 client.26452 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:29.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.029+0000 7fb62e7f8740 1 -- 172.21.15.27:0/259914335 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563e0c5373f0 con 0x563e0c4fe6b0 2024-04-25T13:47:29.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.030+0000 7fb608ff9640 1 -- 172.21.15.27:0/259914335 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb61c040070 con 0x563e0c4fe6b0 2024-04-25T13:47:29.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.030+0000 7fb608ff9640 1 -- 172.21.15.27:0/259914335 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb610021430 con 0x563e0c4fe6b0 2024-04-25T13:47:29.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.030+0000 7fb62e7f8740 10 client.26452 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:29.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.030+0000 7fb62e7f8740 10 client.26452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:29.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.031+0000 7fb62e7f8740 20 client.26452 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:29.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.031+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563e0c4ddf60 0x563e0c8dcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.031+0000 7fb62e7f8740 4 client.26452 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.031+0000 7fb62e7f8740 1 -- 172.21.15.27:0/259914335 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e0c833990 con 0x563e0c4ddf60 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.031+0000 7fb625603640 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563e0c4ddf60 0x563e0c8dcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.032+0000 7fb608ff9640 10 client.26452 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.032+0000 7fb625603640 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563e0c4ddf60 0x563e0c8dcb40 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e62170eda277eb58 server_cookie=70704fbc79463d91 in_seq=0 out_seq=0 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb608ff9640 1 -- 172.21.15.27:0/259914335 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563e0c833990 con 0x563e0c4ddf60 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb608ff9640 10 client.26452 handle_command_reply: tid=0 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 client.26452 shutdown 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 -- 172.21.15.27:0/259914335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 msgr2=0x563e0c8cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8cf1d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb6201993d0 tx=0x7fb610005f90 comp rx=0 tx=0).stop 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 -- 172.21.15.27:0/259914335 shutdown_connections 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563e0c4ddf60 0x563e0c8dcb40 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c51f390 0x563e0c8d6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8cf1d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/259914335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb62e7f8740 1 -- 172.21.15.27:0/259914335 >> 172.21.15.27:0/259914335 conn(0x563e0c5134e0 msgr2=0x563e0c8bf180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.033+0000 7fb608ff9640 0 client.26452 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.034+0000 7fb62e7f8740 1 -- 172.21.15.27:0/259914335 shutdown_connections 2024-04-25T13:47:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.035+0000 7fb62e7f8740 1 -- 172.21.15.27:0/259914335 wait complete. 2024-04-25T13:47:29.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.035+0000 7fb62e7f8740 20 client.26452 trim_cache size 0 max 16384 2024-04-25T13:47:29.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.067+0000 7fb62e7f8740 1 Processor -- start 2024-04-25T13:47:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.067+0000 7fb62e7f8740 1 -- start start 2024-04-25T13:47:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.067+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4f64d0 0x563e0c8c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.067+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.067+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c51f390 0x563e0c8d6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.067+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201a7000 con 0x563e0c51f390 2024-04-25T13:47:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.067+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201a9880 con 0x563e0c4f64d0 2024-04-25T13:47:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.067+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201a7670 con 0x563e0c4fe6b0 2024-04-25T13:47:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.068+0000 7fb625e04640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c51f390 0x563e0c8d6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.068+0000 7fb625e04640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c51f390 0x563e0c8d6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46946/0 (socket says 172.21.15.27:46946) 2024-04-25T13:47:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.068+0000 7fb625e04640 1 -- 172.21.15.27:0/1638593942 learned_addr learned my addr 172.21.15.27:0/1638593942 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.068+0000 7fb624e02640 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.068+0000 7fb625603640 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4f64d0 0x563e0c8c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb625e04640 1 -- 172.21.15.27:0/1638593942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 msgr2=0x563e0c8c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb625e04640 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb625e04640 1 -- 172.21.15.27:0/1638593942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4f64d0 msgr2=0x563e0c8c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb625e04640 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4f64d0 0x563e0c8c99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb625e04640 1 -- 172.21.15.27:0/1638593942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e0c4f70f0 con 0x563e0c51f390 2024-04-25T13:47:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb624e02640 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb625e04640 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c51f390 0x563e0c8d6050 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb6180a8f00 tx=0x7fb6180a8dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c34977410b4c3c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb60affd640 1 -- 172.21.15.27:0/1638593942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6180ad5b0 con 0x563e0c51f390 2024-04-25T13:47:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb60affd640 1 -- 172.21.15.27:0/1638593942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6180b2060 con 0x563e0c51f390 2024-04-25T13:47:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.069+0000 7fb60affd640 1 -- 172.21.15.27:0/1638593942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6180b1bd0 con 0x563e0c51f390 2024-04-25T13:47:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.070+0000 7fb62e7f8740 1 -- 172.21.15.27:0/1638593942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c51f390 msgr2=0x563e0c8d6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.070+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c51f390 0x563e0c8d6050 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb6180a8f00 tx=0x7fb6180a8dd0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.070+0000 7fb62e7f8740 1 -- 172.21.15.27:0/1638593942 shutdown_connections 2024-04-25T13:47:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.070+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c51f390 0x563e0c8d6050 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.070+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c4fe6b0 0x563e0c8c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.070+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/1638593942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4f64d0 0x563e0c8c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.071+0000 7fb62e7f8740 1 -- 172.21.15.27:0/1638593942 >> 172.21.15.27:0/1638593942 conn(0x563e0c5134e0 msgr2=0x563e0c8cd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.071+0000 7fb62e7f8740 1 -- 172.21.15.27:0/1638593942 shutdown_connections 2024-04-25T13:47:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.071+0000 7fb62e7f8740 1 -- 172.21.15.27:0/1638593942 wait complete. 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.071+0000 7fb62e7f8740 1 -- start start 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb62e7f8740 1 Processor -- start 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8d62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4fe6b0 0x563e0c8d6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb62e7f8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c51f390 0x563e0c8d4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201a7d40 con 0x563e0c4f64d0 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201a91b0 con 0x563e0c4fe6b0 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb62e7f8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201a8410 con 0x563e0c51f390 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb62e7f8740 10 client.0 _ll_register_callbacks cb 0x7fb610009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb625e04640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c51f390 0x563e0c8d4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb625603640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8d62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb625e04640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c51f390 0x563e0c8d4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58640/0 (socket says 172.21.15.27:58640) 2024-04-25T13:47:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb625603640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8d62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46962/0 (socket says 172.21.15.27:46962) 2024-04-25T13:47:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb625e04640 1 -- 172.21.15.27:0/2989262600 learned_addr learned my addr 172.21.15.27:0/2989262600 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.072+0000 7fb624e02640 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4fe6b0 0x563e0c8d6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.073+0000 7fb625603640 1 -- 172.21.15.27:0/2989262600 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c51f390 msgr2=0x563e0c8d4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.073+0000 7fb625603640 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c51f390 0x563e0c8d4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.073+0000 7fb625603640 1 -- 172.21.15.27:0/2989262600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4fe6b0 msgr2=0x563e0c8d6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.073+0000 7fb625603640 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4fe6b0 0x563e0c8d6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.073+0000 7fb625603640 1 -- 172.21.15.27:0/2989262600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e0c7e64b0 con 0x563e0c4f64d0 2024-04-25T13:47:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.074+0000 7fb625603640 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8d62f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x563e0c7acd40 tx=0x7fb61c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f064cf70134e0f18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.074+0000 7fb608ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.074+0000 7fb608ff9640 1 -- 172.21.15.27:0/2989262600 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c056070 con 0x563e0c4f64d0 2024-04-25T13:47:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.074+0000 7fb608ff9640 1 -- 172.21.15.27:0/2989262600 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb61c0037f0 con 0x563e0c4f64d0 2024-04-25T13:47:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.074+0000 7fb62e7f8740 1 -- 172.21.15.27:0/2989262600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6180b2060 con 0x563e0c4f64d0 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb608ff9640 1 -- 172.21.15.27:0/2989262600 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb61c060050 con 0x563e0c4f64d0 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb62e7f8740 10 client.16710 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb62e7f8740 1 -- 172.21.15.27:0/2989262600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563e0c9436d0 con 0x563e0c4f64d0 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb608ff9640 1 -- 172.21.15.27:0/2989262600 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb61c040040 con 0x563e0c4f64d0 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb608ff9640 1 -- 172.21.15.27:0/2989262600 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb61c04e070 con 0x563e0c4f64d0 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb62e7f8740 10 client.16710 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb62e7f8740 10 client.16710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb62e7f8740 20 client.16710 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563e0c4b33a0 0x563e0c945bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb62e7f8740 4 client.16710 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:47:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.075+0000 7fb62e7f8740 1 -- 172.21.15.27:0/2989262600 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fb6180b2060 con 0x563e0c4b33a0 2024-04-25T13:47:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.076+0000 7fb624e02640 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563e0c4b33a0 0x563e0c945bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.077+0000 7fb624e02640 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563e0c4b33a0 0x563e0c945bc0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e30699a0008b522 server_cookie=5d805e8a377d9a0b in_seq=0 out_seq=0 2024-04-25T13:47:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.077+0000 7fb608ff9640 10 client.16710 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:29.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.077+0000 7fb608ff9640 1 -- 172.21.15.27:0/2989262600 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fb6180b2060 con 0x563e0c4b33a0 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.077+0000 7fb608ff9640 10 client.16710 handle_command_reply: tid=0 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.077+0000 7fb62e7f8740 1 client.16710 shutdown 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.077+0000 7fb62e7f8740 1 -- 172.21.15.27:0/2989262600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 msgr2=0x563e0c8d62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.077+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8d62f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x563e0c7acd40 tx=0x7fb61c02a410 comp rx=0 tx=0).stop 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 1 -- 172.21.15.27:0/2989262600 shutdown_connections 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563e0c4b33a0 0x563e0c945bc0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb608ff9640 0 client.16710 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563e0c51f390 0x563e0c8d4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563e0c4fe6b0 0x563e0c8d6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 1 --2- 172.21.15.27:0/2989262600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563e0c4f64d0 0x563e0c8d62f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 1 -- 172.21.15.27:0/2989262600 >> 172.21.15.27:0/2989262600 conn(0x563e0c5134e0 msgr2=0x563e0c8cec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 1 -- 172.21.15.27:0/2989262600 shutdown_connections 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 1 -- 172.21.15.27:0/2989262600 wait complete. 2024-04-25T13:47:29.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.078+0000 7fb62e7f8740 20 client.16710 trim_cache size 0 max 16384 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "ac368991-79a7-419e-93cc-94592170b10d": { 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "ac368991-79a7-419e-93cc-94592170b10d", 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:47:29.081 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:47:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb60c0afaf0 msgr2=0x7fb60c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb60c0afaf0 0x7fb60c0b2000 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x563e0c8c5ec0 tx=0x7fb610004730 comp rx=0 tx=0).stop 2024-04-25T13:47:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb620108920 msgr2=0x7fb62019f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb620108920 0x7fb62019f800 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb618010f90 tx=0x7fb61800e490 comp rx=0 tx=0).stop 2024-04-25T13:47:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 shutdown_connections 2024-04-25T13:47:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb60c0afaf0 0x7fb60c0b2000 secure :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0x563e0c8c5ec0 tx=0x7fb610004730 comp rx=0 tx=0).stop 2024-04-25T13:47:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb620108920 0x7fb62019f800 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb620103c20 0x7fb620198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 --2- 172.21.15.27:0/1355828705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6201029e0 0x7fb620198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.079+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 >> 172.21.15.27:0/1355828705 conn(0x7fb6200fdbd0 msgr2=0x7fb620104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.080+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 shutdown_connections 2024-04-25T13:47:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:29.080+0000 7fb627867640 1 -- 172.21.15.27:0/1355828705 wait complete. 2024-04-25T13:47:29.096 INFO:tasks.cephfs.filesystem:scrub status for tag:ac368991-79a7-419e-93cc-94592170b10d - {'path': '/', 'tag': 'ac368991-79a7-419e-93cc-94592170b10d', 'options': 'recursive,force'} 2024-04-25T13:47:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:29 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:47:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:29 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:47:29.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:29 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:47:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:29 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:47:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:29 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:47:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:29 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:47:29.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:29 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:47:29.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:29 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:47:29.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:29 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:47:30.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:30 smithi122 ceph-mon[32843]: pgmap v547: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 683 op/s 2024-04-25T13:47:30.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:30 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:47:30.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:30 smithi045 ceph-mon[32731]: pgmap v547: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 683 op/s 2024-04-25T13:47:30.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:30 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:47:30.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:30 smithi027 ceph-mon[29140]: pgmap v547: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 683 op/s 2024-04-25T13:47:30.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:30 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:47:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.175+0000 7fcb9b957640 1 -- 172.21.15.27:0/628824647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94108170 msgr2=0x7fcb9410a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.175+0000 7fcb9b957640 1 --2- 172.21.15.27:0/628824647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94108170 0x7fcb9410a590 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009ca0 tx=0x7fcb8c02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.175+0000 7fcb9b957640 1 -- 172.21.15.27:0/628824647 shutdown_connections 2024-04-25T13:47:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.175+0000 7fcb9b957640 1 --2- 172.21.15.27:0/628824647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94108170 0x7fcb9410a590 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.175+0000 7fcb9b957640 1 --2- 172.21.15.27:0/628824647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb94103470 0x7fcb94103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.175+0000 7fcb9b957640 1 --2- 172.21.15.27:0/628824647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb94102230 0x7fcb94102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.175+0000 7fcb9b957640 1 -- 172.21.15.27:0/628824647 >> 172.21.15.27:0/628824647 conn(0x7fcb940fd400 msgr2=0x7fcb940ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.175+0000 7fcb9b957640 1 -- 172.21.15.27:0/628824647 shutdown_connections 2024-04-25T13:47:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.176+0000 7fcb9b957640 1 -- 172.21.15.27:0/628824647 wait complete. 2024-04-25T13:47:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.176+0000 7fcb9b957640 1 Processor -- start 2024-04-25T13:47:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.176+0000 7fcb9b957640 1 -- start start 2024-04-25T13:47:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.176+0000 7fcb9b957640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb94102230 0x7fcb941a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.176+0000 7fcb9b957640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94103470 0x7fcb941a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb9b957640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb94108170 0x7fcb941ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb9b957640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9406cde0 con 0x7fcb94103470 2024-04-25T13:47:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb9b957640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9406cc60 con 0x7fcb94102230 2024-04-25T13:47:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb9b957640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9406cf60 con 0x7fcb94108170 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb99ef4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb94108170 0x7fcb941ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb996f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb94102230 0x7fcb941a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb996f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb94102230 0x7fcb941a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34512/0 (socket says 172.21.15.27:34512) 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb99ef4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb94108170 0x7fcb941ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58658/0 (socket says 172.21.15.27:58658) 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb996f3640 1 -- 172.21.15.27:0/3054019238 learned_addr learned my addr 172.21.15.27:0/3054019238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94103470 0x7fcb941a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb99ef4640 1 -- 172.21.15.27:0/3054019238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb94102230 msgr2=0x7fcb941a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb94102230 0x7fcb941a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb99ef4640 1 -- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94103470 msgr2=0x7fcb941a5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94103470 0x7fcb941a5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.177+0000 7fcb99ef4640 1 -- 172.21.15.27:0/3054019238 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb941ac730 con 0x7fcb94108170 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.178+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb94108170 0x7fcb941ac0c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009bf0 tx=0x7fcb8c0317e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.178+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94103470 0x7fcb941a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:47:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.178+0000 7fcb8a7fc640 1 -- 172.21.15.27:0/3054019238 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c003b70 con 0x7fcb94108170 2024-04-25T13:47:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.178+0000 7fcb8a7fc640 1 -- 172.21.15.27:0/3054019238 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c003d10 con 0x7fcb94108170 2024-04-25T13:47:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.178+0000 7fcb9b957640 1 -- 172.21.15.27:0/3054019238 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9406d7b0 con 0x7fcb94108170 2024-04-25T13:47:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.178+0000 7fcb9b957640 1 -- 172.21.15.27:0/3054019238 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb9406dc40 con 0x7fcb94108170 2024-04-25T13:47:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.178+0000 7fcb8a7fc640 1 -- 172.21.15.27:0/3054019238 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c02f9c0 con 0x7fcb94108170 2024-04-25T13:47:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.181+0000 7fcb8a7fc640 1 -- 172.21.15.27:0/3054019238 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcb8c004040 con 0x7fcb94108170 2024-04-25T13:47:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcb8a7fc640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcb800afcf0 0x7fcb800b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcb8a7fc640 1 -- 172.21.15.27:0/3054019238 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcb8c0ffed0 con 0x7fcb94108170 2024-04-25T13:47:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcba2806740 1 Processor -- start 2024-04-25T13:47:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcba2806740 1 -- start start 2024-04-25T13:47:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcb996f3640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcb800afcf0 0x7fcb800b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcba2806740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33b1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcba2806740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcba2806740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcba2806740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b3b3317460 con 0x55b3b2fe82f0 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.182+0000 7fcba2806740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b3b32d1590 con 0x55b3b2ff04d0 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcba2806740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b3b3313420 con 0x55b3b30111b0 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb99ef4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb99ef4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33b8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34514/0 (socket says 172.21.15.27:34514) 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb99ef4640 1 -- 172.21.15.27:0/1088120146 learned_addr learned my addr 172.21.15.27:0/1088120146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb996f3640 1 --2- 172.21.15.27:0/1088120146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb99ef4640 1 -- 172.21.15.27:0/1088120146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 msgr2=0x55b3b33b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/1088120146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33b1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.184 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb99ef4640 1 -- 172.21.15.27:0/1088120146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 msgr2=0x55b3b33b1ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:47:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/1088120146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.183+0000 7fcb99ef4640 1 -- 172.21.15.27:0/1088120146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b32de0d0 con 0x55b3b2ff04d0 2024-04-25T13:47:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.184+0000 7fcb996f3640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcb800afcf0 0x7fcb800b2200 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcb90005d00 tx=0x7fcb9000bf80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.185+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/1088120146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33b8df0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0c4400 tx=0x7fcb8c009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fec4225e0265c72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.185+0000 7fcb7f7fe640 1 -- 172.21.15.27:0/1088120146 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c0fde60 con 0x55b3b2ff04d0 2024-04-25T13:47:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.185+0000 7fcb7f7fe640 1 -- 172.21.15.27:0/1088120146 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c036040 con 0x55b3b2ff04d0 2024-04-25T13:47:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.185+0000 7fcb7f7fe640 1 -- 172.21.15.27:0/1088120146 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c0c8840 con 0x55b3b2ff04d0 2024-04-25T13:47:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.186+0000 7fcba2806740 1 -- 172.21.15.27:0/1088120146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 msgr2=0x55b3b33b8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.186+0000 7fcba2806740 1 --2- 172.21.15.27:0/1088120146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33b8df0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0c4400 tx=0x7fcb8c009870 comp rx=0 tx=0).stop 2024-04-25T13:47:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.187+0000 7fcba2806740 1 -- 172.21.15.27:0/1088120146 shutdown_connections 2024-04-25T13:47:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.187+0000 7fcba2806740 1 --2- 172.21.15.27:0/1088120146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33b8df0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.187+0000 7fcba2806740 1 --2- 172.21.15.27:0/1088120146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.187+0000 7fcba2806740 1 --2- 172.21.15.27:0/1088120146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.187+0000 7fcba2806740 1 -- 172.21.15.27:0/1088120146 >> 172.21.15.27:0/1088120146 conn(0x55b3b3005300 msgr2=0x55b3b33ad300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:32.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.187+0000 7fcba2806740 1 -- 172.21.15.27:0/1088120146 shutdown_connections 2024-04-25T13:47:32.188 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.187+0000 7fcba2806740 1 -- 172.21.15.27:0/1088120146 wait complete. 2024-04-25T13:47:32.188 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.188+0000 7fcba2806740 1 -- start start 2024-04-25T13:47:32.188 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.188+0000 7fcba2806740 1 Processor -- start 2024-04-25T13:47:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcba2806740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcba2806740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcba2806740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcba2806740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b3b32d1590 con 0x55b3b2fe82f0 2024-04-25T13:47:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcba2806740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b3b329eb60 con 0x55b3b2ff04d0 2024-04-25T13:47:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcba2806740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b3b33b9600 con 0x55b3b30111b0 2024-04-25T13:47:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcba2806740 10 client.0 _ll_register_callbacks cb 0x55b3b32c32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb996f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb996f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46984/0 (socket says 172.21.15.27:46984) 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb996f3640 1 -- 172.21.15.27:0/4137690085 learned_addr learned my addr 172.21.15.27:0/4137690085 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb996f3640 1 -- 172.21.15.27:0/4137690085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 msgr2=0x55b3b33c26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb996f3640 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33c26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb996f3640 1 -- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 msgr2=0x55b3b33bb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.189+0000 7fcb996f3640 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.190+0000 7fcb996f3640 1 -- 172.21.15.27:0/4137690085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b3029210 con 0x55b3b2fe82f0 2024-04-25T13:47:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.190+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:47:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.190+0000 7fcb996f3640 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bb200 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcb90016fd0 tx=0x7fcb90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65feb0984ffe84cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.191+0000 7fcb7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.191+0000 7fcb7cff9640 1 -- 172.21.15.27:0/4137690085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90025580 con 0x55b3b2fe82f0 2024-04-25T13:47:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.191+0000 7fcb7cff9640 1 -- 172.21.15.27:0/4137690085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcb90025720 con 0x55b3b2fe82f0 2024-04-25T13:47:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.192+0000 7fcba2806740 1 -- 172.21.15.27:0/4137690085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb8c036040 con 0x55b3b2fe82f0 2024-04-25T13:47:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.192+0000 7fcb7cff9640 1 -- 172.21.15.27:0/4137690085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcb90025920 con 0x55b3b2fe82f0 2024-04-25T13:47:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.192+0000 7fcba2806740 10 client.16719 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.192+0000 7fcba2806740 1 -- 172.21.15.27:0/4137690085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b3b30f5620 con 0x55b3b2fe82f0 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.192+0000 7fcb7cff9640 1 -- 172.21.15.27:0/4137690085 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c036040 con 0x55b3b2fe82f0 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.192+0000 7fcb7cff9640 1 -- 172.21.15.27:0/4137690085 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcb9002fb50 con 0x55b3b2fe82f0 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcba2806740 10 client.16719 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcba2806740 10 client.16719 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcba2806740 20 client.16719 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcba2806740 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3b2fcfd80 0x55b3b33c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcba2806740 4 client.16719 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcba2806740 1 -- 172.21.15.27:0/4137690085 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3b3325070 con 0x55b3b2fcfd80 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3b2fcfd80 0x55b3b33c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3b2fcfd80 0x55b3b33c8f50 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5545b9ef544eabcc server_cookie=e649561a18776f in_seq=0 out_seq=0 2024-04-25T13:47:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.193+0000 7fcb7cff9640 10 client.16719 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.194+0000 7fcb7cff9640 1 -- 172.21.15.27:0/4137690085 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b3b3325070 con 0x55b3b2fcfd80 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.194+0000 7fcb7cff9640 10 client.16719 handle_command_reply: tid=0 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.194+0000 7fcba2806740 1 client.16719 shutdown 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 -- 172.21.15.27:0/4137690085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 msgr2=0x55b3b33bb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bb200 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcb90016fd0 tx=0x7fcb90004730 comp rx=0 tx=0).stop 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 -- 172.21.15.27:0/4137690085 shutdown_connections 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3b2fcfd80 0x55b3b33c8f50 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33c26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 --2- 172.21.15.27:0/4137690085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bb200 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 -- 172.21.15.27:0/4137690085 >> 172.21.15.27:0/4137690085 conn(0x55b3b3005300 msgr2=0x55b3b33ac250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcb7cff9640 0 client.16719 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 -- 172.21.15.27:0/4137690085 shutdown_connections 2024-04-25T13:47:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 1 -- 172.21.15.27:0/4137690085 wait complete. 2024-04-25T13:47:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.195+0000 7fcba2806740 20 client.16719 trim_cache size 0 max 16384 2024-04-25T13:47:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.220+0000 7fcba2806740 1 Processor -- start 2024-04-25T13:47:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.221+0000 7fcba2806740 1 -- start start 2024-04-25T13:47:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcba2806740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b2fe82f0 0x55b3b33b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcba2806740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 0x55b3b33c2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcb996f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b2fe82f0 0x55b3b33b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcb996f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b2fe82f0 0x55b3b33b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58680/0 (socket says 172.21.15.27:58680) 2024-04-25T13:47:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcb996f3640 1 -- 172.21.15.27:0/3715005559 learned_addr learned my addr 172.21.15.27:0/3715005559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcba2806740 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b30111b0 0x55b3b33b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcba2806740 1 -- 172.21.15.27:0/3715005559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcb941af7e0 con 0x55b3b2ff04d0 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcba2806740 1 -- 172.21.15.27:0/3715005559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb941b2060 con 0x55b3b30111b0 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcba2806740 1 -- 172.21.15.27:0/3715005559 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcb941afe50 con 0x55b3b2fe82f0 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.222+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b30111b0 0x55b3b33b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 0x55b3b33c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb99ef4640 1 -- 172.21.15.27:0/3715005559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b2fe82f0 msgr2=0x55b3b33b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b2fe82f0 0x55b3b33b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb99ef4640 1 -- 172.21.15.27:0/3715005559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 msgr2=0x55b3b33c2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 0x55b3b33c2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb99ef4640 1 -- 172.21.15.27:0/3715005559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b33ac830 con 0x55b3b30111b0 2024-04-25T13:47:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b30111b0 0x55b3b33b9880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0c8080 tx=0x7fcb8c02adc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99520649caf01834 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb7effd640 1 -- 172.21.15.27:0/3715005559 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c0fde60 con 0x55b3b30111b0 2024-04-25T13:47:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb7effd640 1 -- 172.21.15.27:0/3715005559 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c036070 con 0x55b3b30111b0 2024-04-25T13:47:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.223+0000 7fcb7effd640 1 -- 172.21.15.27:0/3715005559 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c0c8840 con 0x55b3b30111b0 2024-04-25T13:47:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.224+0000 7fcba2806740 1 -- 172.21.15.27:0/3715005559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b30111b0 msgr2=0x55b3b33b9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.224+0000 7fcba2806740 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b30111b0 0x55b3b33b9880 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0c8080 tx=0x7fcb8c02adc0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.225+0000 7fcba2806740 1 -- 172.21.15.27:0/3715005559 shutdown_connections 2024-04-25T13:47:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.225+0000 7fcba2806740 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b30111b0 0x55b3b33b9880 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.225+0000 7fcba2806740 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 0x55b3b33c2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.225+0000 7fcba2806740 1 --2- 172.21.15.27:0/3715005559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b2fe82f0 0x55b3b33b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.225+0000 7fcba2806740 1 -- 172.21.15.27:0/3715005559 >> 172.21.15.27:0/3715005559 conn(0x55b3b3005300 msgr2=0x55b3b33b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.226+0000 7fcba2806740 1 -- 172.21.15.27:0/3715005559 shutdown_connections 2024-04-25T13:47:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.226+0000 7fcba2806740 1 -- 172.21.15.27:0/3715005559 wait complete. 2024-04-25T13:47:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.226+0000 7fcba2806740 1 -- start start 2024-04-25T13:47:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.226+0000 7fcba2806740 1 Processor -- start 2024-04-25T13:47:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcba2806740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcba2806740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcba2806740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33c0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcba2806740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcb941b0520 con 0x55b3b2ff04d0 2024-04-25T13:47:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcba2806740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb941b1990 con 0x55b3b2fe82f0 2024-04-25T13:47:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcba2806740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcb941b0bf0 con 0x55b3b30111b0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb996f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb98ef2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb996f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34536/0 (socket says 172.21.15.27:34536) 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb996f3640 1 -- 172.21.15.27:0/2535190577 learned_addr learned my addr 172.21.15.27:0/2535190577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb99ef4640 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33c0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcba2806740 10 client.0 _ll_register_callbacks cb 0x7fcb9000b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb996f3640 1 -- 172.21.15.27:0/2535190577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 msgr2=0x55b3b33c0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb996f3640 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33c0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb996f3640 1 -- 172.21.15.27:0/2535190577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 msgr2=0x55b3b33bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb996f3640 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.227+0000 7fcb996f3640 1 -- 172.21.15.27:0/2535190577 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b33c13b0 con 0x55b3b2fe82f0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcb996f3640 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bf980 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcb9002e4c0 tx=0x7fcb9002e770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4304fbc1ec4615c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcb7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcb7cff9640 1 -- 172.21.15.27:0/2535190577 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90005850 con 0x55b3b2fe82f0 2024-04-25T13:47:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcba2806740 1 -- 172.21.15.27:0/2535190577 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb8c036070 con 0x55b3b2fe82f0 2024-04-25T13:47:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcb7cff9640 1 -- 172.21.15.27:0/2535190577 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcb9001e040 con 0x55b3b2fe82f0 2024-04-25T13:47:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcb7cff9640 1 -- 172.21.15.27:0/2535190577 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcb90025cd0 con 0x55b3b2fe82f0 2024-04-25T13:47:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcba2806740 10 client.26435 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcba2806740 1 -- 172.21.15.27:0/2535190577 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b3b33c2690 con 0x55b3b2fe82f0 2024-04-25T13:47:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.228+0000 7fcb7cff9640 1 -- 172.21.15.27:0/2535190577 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c036070 con 0x55b3b2fe82f0 2024-04-25T13:47:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.229+0000 7fcb7cff9640 1 -- 172.21.15.27:0/2535190577 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcb9002f890 con 0x55b3b2fe82f0 2024-04-25T13:47:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.229+0000 7fcba2806740 10 client.26435 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.229+0000 7fcba2806740 10 client.26435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.230+0000 7fcba2806740 20 client.26435 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.230+0000 7fcba2806740 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3b2fa51c0 0x55b3b3433850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.230+0000 7fcba2806740 4 client.26435 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:47:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.230+0000 7fcba2806740 1 -- 172.21.15.27:0/2535190577 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3b331eee0 con 0x55b3b2fa51c0 2024-04-25T13:47:32.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.230+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3b2fa51c0 0x55b3b3433850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:32.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.231+0000 7fcb98ef2640 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3b2fa51c0 0x55b3b3433850 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5725bd5b00cfad4b server_cookie=976743d09b8fb29d in_seq=0 out_seq=0 2024-04-25T13:47:32.232 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.231+0000 7fcb7cff9640 10 client.26435 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcb7cff9640 1 -- 172.21.15.27:0/2535190577 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b3b331eee0 con 0x55b3b2fa51c0 2024-04-25T13:47:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcb7cff9640 10 client.26435 handle_command_reply: tid=0 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 client.26435 shutdown 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 -- 172.21.15.27:0/2535190577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2fe82f0 msgr2=0x55b3b33bf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bf980 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcb9002e4c0 tx=0x7fcb9002e770 comp rx=0 tx=0).stop 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 -- 172.21.15.27:0/2535190577 shutdown_connections 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3b2fa51c0 0x55b3b3433850 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3b30111b0 0x55b3b33c0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcb7cff9640 0 client.26435 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3b2ff04d0 0x55b3b33bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 --2- 172.21.15.27:0/2535190577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3b2fe82f0 0x55b3b33bf980 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.251+0000 7fcba2806740 1 -- 172.21.15.27:0/2535190577 >> 172.21.15.27:0/2535190577 conn(0x55b3b3005300 msgr2=0x55b3b33abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcba2806740 1 -- 172.21.15.27:0/2535190577 shutdown_connections 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcba2806740 1 -- 172.21.15.27:0/2535190577 wait complete. 2024-04-25T13:47:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcba2806740 20 client.26435 trim_cache size 0 max 16384 2024-04-25T13:47:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:47:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:47:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:47:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 -- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcb800afcf0 msgr2=0x7fcb800b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcb800afcf0 0x7fcb800b2200 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcb90005d00 tx=0x7fcb9000bf80 comp rx=0 tx=0).stop 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 -- 172.21.15.27:0/3054019238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb94108170 msgr2=0x7fcb941ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb94108170 0x7fcb941ac0c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009bf0 tx=0x7fcb8c0317e0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 -- 172.21.15.27:0/3054019238 shutdown_connections 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcb800afcf0 0x7fcb800b2200 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb94108170 0x7fcb941ac0c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb94103470 0x7fcb941a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 --2- 172.21.15.27:0/3054019238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb94102230 0x7fcb941a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 -- 172.21.15.27:0/3054019238 >> 172.21.15.27:0/3054019238 conn(0x7fcb940fd400 msgr2=0x7fcb94071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 -- 172.21.15.27:0/3054019238 shutdown_connections 2024-04-25T13:47:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:32.252+0000 7fcb9b957640 1 -- 172.21.15.27:0/3054019238 wait complete. 2024-04-25T13:47:32.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:32 smithi122 ceph-mon[32843]: pgmap v548: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.8 MiB/s wr, 796 op/s 2024-04-25T13:47:32.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:32 smithi045 ceph-mon[32731]: pgmap v548: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.8 MiB/s wr, 796 op/s 2024-04-25T13:47:32.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:32 smithi027 ceph-mon[29140]: pgmap v548: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.8 MiB/s wr, 796 op/s 2024-04-25T13:47:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 385.0 s, 2708.8 tps, lat 37.191 ms stddev 42.397 2024-04-25T13:47:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:34 smithi122 ceph-mon[32843]: pgmap v549: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 10 MiB/s wr, 832 op/s 2024-04-25T13:47:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:34 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:47:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:34 smithi045 ceph-mon[32731]: pgmap v549: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 10 MiB/s wr, 832 op/s 2024-04-25T13:47:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:34 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:47:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:34 smithi027 ceph-mon[29140]: pgmap v549: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 10 MiB/s wr, 832 op/s 2024-04-25T13:47:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:34 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:47:36.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:36 smithi122 ceph-mon[32843]: pgmap v550: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.6 MiB/s wr, 708 op/s 2024-04-25T13:47:36.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:36 smithi045 ceph-mon[32731]: pgmap v550: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.6 MiB/s wr, 708 op/s 2024-04-25T13:47:36.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:36 smithi027 ceph-mon[29140]: pgmap v550: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.6 MiB/s wr, 708 op/s 2024-04-25T13:47:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 390.0 s, 3216.8 tps, lat 31.114 ms stddev 35.503 2024-04-25T13:47:38.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:38 smithi122 ceph-mon[32843]: pgmap v551: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.9 MiB/s wr, 854 op/s 2024-04-25T13:47:38.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:38 smithi045 ceph-mon[32731]: pgmap v551: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.9 MiB/s wr, 854 op/s 2024-04-25T13:47:38.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:38 smithi027 ceph-mon[29140]: pgmap v551: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.9 MiB/s wr, 854 op/s 2024-04-25T13:47:40.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:40 smithi122 ceph-mon[32843]: pgmap v552: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.0 MiB/s wr, 667 op/s 2024-04-25T13:47:40.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:40 smithi045 ceph-mon[32731]: pgmap v552: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.0 MiB/s wr, 667 op/s 2024-04-25T13:47:40.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:40 smithi027 ceph-mon[29140]: pgmap v552: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.0 MiB/s wr, 667 op/s 2024-04-25T13:47:42.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:42 smithi122 ceph-mon[32843]: pgmap v553: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.1 MiB/s wr, 771 op/s 2024-04-25T13:47:42.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:42 smithi045 ceph-mon[32731]: pgmap v553: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.1 MiB/s wr, 771 op/s 2024-04-25T13:47:42.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:42 smithi027 ceph-mon[29140]: pgmap v553: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.1 MiB/s wr, 771 op/s 2024-04-25T13:47:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 395.0 s, 2947.8 tps, lat 33.935 ms stddev 37.522 2024-04-25T13:47:44.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:44 smithi122 ceph-mon[32843]: pgmap v554: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.9 MiB/s wr, 800 op/s 2024-04-25T13:47:44.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:44 smithi045 ceph-mon[32731]: pgmap v554: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.9 MiB/s wr, 800 op/s 2024-04-25T13:47:44.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:44 smithi027 ceph-mon[29140]: pgmap v554: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.9 MiB/s wr, 800 op/s 2024-04-25T13:47:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 -- 172.21.15.27:0/1462177262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 msgr2=0x7f5268104840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 --2- 172.21.15.27:0/1462177262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 0x7f5268104840 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5258009860 tx=0x7f525802a340 comp rx=0 tx=0).stop 2024-04-25T13:47:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 -- 172.21.15.27:0/1462177262 shutdown_connections 2024-04-25T13:47:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 --2- 172.21.15.27:0/1462177262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268109a80 0x7f526810bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 --2- 172.21.15.27:0/1462177262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5268104d80 0x7f5268105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 --2- 172.21.15.27:0/1462177262 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 0x7f5268104840 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 -- 172.21.15.27:0/1462177262 >> 172.21.15.27:0/1462177262 conn(0x7f52680fed30 msgr2=0x7f52681011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 -- 172.21.15.27:0/1462177262 shutdown_connections 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 -- 172.21.15.27:0/1462177262 wait complete. 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.391+0000 7f526dfee640 1 Processor -- start 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f526dfee640 1 -- start start 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f526dfee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 0x7f526819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f526dfee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5268104d80 0x7f526819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f526dfee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268109a80 0x7f52681a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f526dfee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f526810f640 con 0x7f5268109a80 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f526dfee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f526810f4c0 con 0x7f5268104d80 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f526dfee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f526810f7c0 con 0x7f5268103b40 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f52677fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 0x7f526819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f5267fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268109a80 0x7f52681a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f5266ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5268104d80 0x7f526819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f52677fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 0x7f526819a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:49296/0 (socket says 172.21.15.27:49296) 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.392+0000 7f52677fe640 1 -- 172.21.15.27:0/2480251672 learned_addr learned my addr 172.21.15.27:0/2480251672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f52677fe640 1 -- 172.21.15.27:0/2480251672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5268104d80 msgr2=0x7f526819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f52677fe640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5268104d80 0x7f526819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f52677fe640 1 -- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268109a80 msgr2=0x7f52681a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f52677fe640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268109a80 0x7f52681a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f52677fe640 1 -- 172.21.15.27:0/2480251672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52681a1db0 con 0x7f5268103b40 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f5266ffd640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5268104d80 0x7f526819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f52677fe640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 0x7f526819a270 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f525802a860 tx=0x7f5258032cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f5264ff9640 1 -- 172.21.15.27:0/2480251672 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258030e20 con 0x7f5268103b40 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f5264ff9640 1 -- 172.21.15.27:0/2480251672 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5258004040 con 0x7f5268103b40 2024-04-25T13:47:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.393+0000 7f5264ff9640 1 -- 172.21.15.27:0/2480251672 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258003b00 con 0x7f5268103b40 2024-04-25T13:47:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.394+0000 7f526dfee640 1 -- 172.21.15.27:0/2480251672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5268074dd0 con 0x7f5268103b40 2024-04-25T13:47:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.394+0000 7f526dfee640 1 -- 172.21.15.27:0/2480251672 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5268075270 con 0x7f5268103b40 2024-04-25T13:47:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.397+0000 7f5264ff9640 1 -- 172.21.15.27:0/2480251672 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5258003ca0 con 0x7f5268103b40 2024-04-25T13:47:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.398+0000 7f5264ff9640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52500afb60 0x7f52500b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.398+0000 7f5264ff9640 1 -- 172.21.15.27:0/2480251672 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f52580c7300 con 0x7f5268103b40 2024-04-25T13:47:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.398+0000 7f5274f72740 1 Processor -- start 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.398+0000 7f5266ffd640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52500afb60 0x7f52500b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5274f72740 1 -- start start 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5274f72740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157cb1b0 0x55c515b74180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5274f72740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b74700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5274f72740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5274f72740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c515ad1460 con 0x55c5157a22f0 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5274f72740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c515a8b590 con 0x55c5157cb1b0 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5274f72740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55c515acd420 con 0x55c5157aa4d0 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5267fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5267fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:49308/0 (socket says 172.21.15.27:49308) 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5267fff640 1 -- 172.21.15.27:0/772130712 learned_addr learned my addr 172.21.15.27:0/772130712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f52677fe640 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157cb1b0 0x55c515b74180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5266ffd640 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b74700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5267fff640 1 -- 172.21.15.27:0/772130712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157cb1b0 msgr2=0x55c515b74180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5267fff640 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157cb1b0 0x55c515b74180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5267fff640 1 -- 172.21.15.27:0/772130712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 msgr2=0x55c515b74700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5267fff640 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b74700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5267fff640 1 -- 172.21.15.27:0/772130712 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c515a980d0 con 0x55c5157aa4d0 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.399+0000 7f5266ffd640 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b74700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.400+0000 7f5267fff640 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7b600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f526000e450 tx=0x7f52600095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d47a1af16718c8bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.400+0000 7f524dffb640 1 -- 172.21.15.27:0/772130712 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526000d910 con 0x55c5157aa4d0 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.400+0000 7f524dffb640 1 -- 172.21.15.27:0/772130712 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f526001c070 con 0x55c5157aa4d0 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.400+0000 7f524dffb640 1 -- 172.21.15.27:0/772130712 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5260017d60 con 0x55c5157aa4d0 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 -- 172.21.15.27:0/772130712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 msgr2=0x55c515b7b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7b600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f526000e450 tx=0x7f52600095c0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5266ffd640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52500afb60 0x7f52500b2070 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f526819b310 tx=0x7f525c011ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 -- 172.21.15.27:0/772130712 shutdown_connections 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7b600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b74700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 --2- 172.21.15.27:0/772130712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157cb1b0 0x55c515b74180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 -- 172.21.15.27:0/772130712 >> 172.21.15.27:0/772130712 conn(0x55c5157bf300 msgr2=0x55c515b67300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 -- 172.21.15.27:0/772130712 shutdown_connections 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 -- 172.21.15.27:0/772130712 wait complete. 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.401+0000 7f5274f72740 1 -- start start 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5274f72740 1 Processor -- start 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5274f72740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b7ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5274f72740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5274f72740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157cb1b0 0x55c515b85290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5274f72740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c515b7c050 con 0x55c5157cb1b0 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5274f72740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c515a58b60 con 0x55c5157a22f0 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5274f72740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55c515b7c1d0 con 0x55c5157aa4d0 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5274f72740 10 client.0 _ll_register_callbacks cb 0x55c515a7d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f52677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b7ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5267fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157cb1b0 0x55c515b85290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f52677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b7ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:55434/0 (socket says 172.21.15.27:55434) 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f52677fe640 1 -- 172.21.15.27:0/275314181 learned_addr learned my addr 172.21.15.27:0/275314181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.402+0000 7f5266ffd640 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f52677fe640 1 -- 172.21.15.27:0/275314181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 msgr2=0x55c515b7e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f52677fe640 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f52677fe640 1 -- 172.21.15.27:0/275314181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157cb1b0 msgr2=0x55c515b85290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f52677fe640 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157cb1b0 0x55c515b85290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.408 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f52677fe640 1 -- 172.21.15.27:0/275314181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5157e3210 con 0x55c5157a22f0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f52677fe640 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b7ddf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f52580c7bb0 tx=0x7f52580c5430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d26d9240e593414 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f5246ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f5246ffd640 1 -- 172.21.15.27:0/275314181 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580c7500 con 0x55c5157a22f0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f5274f72740 1 -- 172.21.15.27:0/275314181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f526001c070 con 0x55c5157a22f0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f5246ffd640 1 -- 172.21.15.27:0/275314181 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f52580c9f70 con 0x55c5157a22f0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f5246ffd640 1 -- 172.21.15.27:0/275314181 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f5258037030 con 0x55c5157a22f0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f5274f72740 10 client.26447 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.403+0000 7f5274f72740 1 -- 172.21.15.27:0/275314181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55c5158af620 con 0x55c5157a22f0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5246ffd640 1 -- 172.21.15.27:0/275314181 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f526001c070 con 0x55c5157a22f0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5246ffd640 1 -- 172.21.15.27:0/275314181 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f52580e5140 con 0x55c5157a22f0 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5274f72740 10 client.26447 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5274f72740 10 client.26447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5274f72740 20 client.26447 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5274f72740 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c515789d80 0x55c515b8ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5274f72740 4 client.26447 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5274f72740 1 -- 172.21.15.27:0/275314181 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c515adf070 con 0x55c515789d80 2024-04-25T13:47:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.404+0000 7f5266ffd640 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c515789d80 0x55c515b8ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.405+0000 7f5246ffd640 10 client.26447 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.405+0000 7f5266ffd640 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c515789d80 0x55c515b8ba50 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94b009b10fdc8fd server_cookie=d02aa8420071cf18 in_seq=0 out_seq=0 2024-04-25T13:47:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.405+0000 7f5246ffd640 1 -- 172.21.15.27:0/275314181 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c515adf070 con 0x55c515789d80 2024-04-25T13:47:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.405+0000 7f5246ffd640 10 client.26447 handle_command_reply: tid=0 2024-04-25T13:47:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.405+0000 7f5274f72740 1 client.26447 shutdown 2024-04-25T13:47:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 -- 172.21.15.27:0/275314181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 msgr2=0x55c515b7ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b7ddf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f52580c7bb0 tx=0x7f52580c5430 comp rx=0 tx=0).stop 2024-04-25T13:47:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 -- 172.21.15.27:0/275314181 shutdown_connections 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c515789d80 0x55c515b8ba50 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157cb1b0 0x55c515b85290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5246ffd640 0 client.26447 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 --2- 172.21.15.27:0/275314181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b7ddf0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 -- 172.21.15.27:0/275314181 >> 172.21.15.27:0/275314181 conn(0x55c5157bf300 msgr2=0x55c515b66250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 -- 172.21.15.27:0/275314181 shutdown_connections 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 1 -- 172.21.15.27:0/275314181 wait complete. 2024-04-25T13:47:45.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.406+0000 7f5274f72740 20 client.26447 trim_cache size 0 max 16384 2024-04-25T13:47:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.425+0000 7f5274f72740 1 Processor -- start 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5274f72740 1 -- start start 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5274f72740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c515789d80 0x55c515b66ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5274f72740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5274f72740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5274f72740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52681a6e70 con 0x55c515789d80 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5274f72740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52681a96f0 con 0x55c5157a22f0 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5274f72740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f52681a74e0 con 0x55c5157aa4d0 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f52677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c515789d80 0x55c515b66ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f52677fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c515789d80 0x55c515b66ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41256/0 (socket says 172.21.15.27:41256) 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f52677fe640 1 -- 172.21.15.27:0/4092534241 learned_addr learned my addr 172.21.15.27:0/4092534241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5266ffd640 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f5267fff640 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f52677fe640 1 -- 172.21.15.27:0/4092534241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 msgr2=0x55c515b7d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f52677fe640 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f52677fe640 1 -- 172.21.15.27:0/4092534241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 msgr2=0x55c515b670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f52677fe640 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.426+0000 7f52677fe640 1 -- 172.21.15.27:0/4092534241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c515b7b9b0 con 0x55c515789d80 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.427+0000 7f52677fe640 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c515789d80 0x55c515b66ba0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f52580c7840 tx=0x7f5258005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ccae05b63d393 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.427+0000 7f524d7fa640 1 -- 172.21.15.27:0/4092534241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580c76a0 con 0x55c515789d80 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.427+0000 7f524d7fa640 1 -- 172.21.15.27:0/4092534241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5258007e80 con 0x55c515789d80 2024-04-25T13:47:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.427+0000 7f524d7fa640 1 -- 172.21.15.27:0/4092534241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580c7870 con 0x55c515789d80 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 -- 172.21.15.27:0/4092534241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c515789d80 msgr2=0x55c515b66ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c515789d80 0x55c515b66ba0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f52580c7840 tx=0x7f5258005da0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 -- 172.21.15.27:0/4092534241 shutdown_connections 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c5157aa4d0 0x55c515b7d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157a22f0 0x55c515b670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 --2- 172.21.15.27:0/4092534241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c515789d80 0x55c515b66ba0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 -- 172.21.15.27:0/4092534241 >> 172.21.15.27:0/4092534241 conn(0x55c5157bf300 msgr2=0x55c515b78270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 -- 172.21.15.27:0/4092534241 shutdown_connections 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.428+0000 7f5274f72740 1 -- 172.21.15.27:0/4092534241 wait complete. 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 1 -- start start 2024-04-25T13:47:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 1 Processor -- start 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c515789d80 0x55c515b805a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b80ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157aa4d0 0x55c515b839d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52681a7bb0 con 0x55c5157a22f0 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52681a9020 con 0x55c5157aa4d0 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f52681a8280 con 0x55c515789d80 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5274f72740 10 client.0 _ll_register_callbacks cb 0x7f525c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5266ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b80ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5266ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b80ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41268/0 (socket says 172.21.15.27:41268) 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5266ffd640 1 -- 172.21.15.27:0/3137924190 learned_addr learned my addr 172.21.15.27:0/3137924190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f5267fff640 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157aa4d0 0x55c515b839d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.429+0000 7f52677fe640 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c515789d80 0x55c515b805a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5266ffd640 1 -- 172.21.15.27:0/3137924190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c515789d80 msgr2=0x55c515b805a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5266ffd640 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c515789d80 0x55c515b805a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5266ffd640 1 -- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157aa4d0 msgr2=0x55c515b839d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5266ffd640 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157aa4d0 0x55c515b839d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5266ffd640 1 -- 172.21.15.27:0/3137924190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c515b83f90 con 0x55c5157a22f0 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5267fff640 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157aa4d0 0x55c515b839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5266ffd640 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b80ae0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f525c009fd0 tx=0x7f525c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e45a25e22391496 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5246ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5246ffd640 1 -- 172.21.15.27:0/3137924190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c02bad0 con 0x55c5157a22f0 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5246ffd640 1 -- 172.21.15.27:0/3137924190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f525c00f900 con 0x55c5157a22f0 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5274f72740 1 -- 172.21.15.27:0/3137924190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5258007e80 con 0x55c5157a22f0 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5246ffd640 1 -- 172.21.15.27:0/3137924190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f525c00fb00 con 0x55c5157a22f0 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5274f72740 10 client.16740 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.430+0000 7f5274f72740 1 -- 172.21.15.27:0/3137924190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55c515bf4230 con 0x55c5157a22f0 2024-04-25T13:47:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5246ffd640 1 -- 172.21.15.27:0/3137924190 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5258007e80 con 0x55c5157a22f0 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5246ffd640 1 -- 172.21.15.27:0/3137924190 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f525c032070 con 0x55c5157a22f0 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5274f72740 10 client.16740 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5274f72740 10 client.16740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5274f72740 20 client.16740 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5274f72740 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c5157cb1b0 0x55c515bf6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5274f72740 4 client.16740 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5274f72740 1 -- 172.21.15.27:0/3137924190 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c515ad8ee0 con 0x55c5157cb1b0 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f52677fe640 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c5157cb1b0 0x55c515bf6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.431+0000 7f5246ffd640 10 client.16740 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.432+0000 7f52677fe640 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c5157cb1b0 0x55c515bf6c50 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bfe0036b188a3ad server_cookie=65440c79180f3618 in_seq=0 out_seq=0 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5246ffd640 1 -- 172.21.15.27:0/3137924190 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c515ad8ee0 con 0x55c5157cb1b0 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5246ffd640 10 client.16740 handle_command_reply: tid=0 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 client.16740 shutdown 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 -- 172.21.15.27:0/3137924190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 msgr2=0x55c515b80ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b80ae0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f525c009fd0 tx=0x7f525c004730 comp rx=0 tx=0).stop 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 -- 172.21.15.27:0/3137924190 shutdown_connections 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c5157cb1b0 0x55c515bf6c50 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c5157aa4d0 0x55c515b839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c5157a22f0 0x55c515b80ae0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5246ffd640 0 client.16740 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 --2- 172.21.15.27:0/3137924190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c515789d80 0x55c515b805a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 -- 172.21.15.27:0/3137924190 >> 172.21.15.27:0/3137924190 conn(0x55c5157bf300 msgr2=0x55c515b66ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 -- 172.21.15.27:0/3137924190 shutdown_connections 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.504+0000 7f5274f72740 1 -- 172.21.15.27:0/3137924190 wait complete. 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f5274f72740 20 client.16740 trim_cache size 0 max 16384 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:47:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:47:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 -- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52500afb60 msgr2=0x7f52500b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52500afb60 0x7f52500b2070 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f526819b310 tx=0x7f525c011ad0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 -- 172.21.15.27:0/2480251672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 msgr2=0x7f526819a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 0x7f526819a270 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f525802a860 tx=0x7f5258032cd0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 -- 172.21.15.27:0/2480251672 shutdown_connections 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52500afb60 0x7f52500b2070 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268109a80 0x7f52681a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5268104d80 0x7f526819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 --2- 172.21.15.27:0/2480251672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5268103b40 0x7f526819a270 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 -- 172.21.15.27:0/2480251672 >> 172.21.15.27:0/2480251672 conn(0x7f52680fed30 msgr2=0x7f5268105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 -- 172.21.15.27:0/2480251672 shutdown_connections 2024-04-25T13:47:45.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:45.505+0000 7f526dfee640 1 -- 172.21.15.27:0/2480251672 wait complete. 2024-04-25T13:47:46.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:46 smithi122 ceph-mon[32843]: pgmap v555: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 7.9 MiB/s wr, 679 op/s 2024-04-25T13:47:46.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:46 smithi045 ceph-mon[32731]: pgmap v555: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 7.9 MiB/s wr, 679 op/s 2024-04-25T13:47:46.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:46 smithi027 ceph-mon[29140]: pgmap v555: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 7.9 MiB/s wr, 679 op/s 2024-04-25T13:47:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 400.0 s, 2275.6 tps, lat 43.463 ms stddev 47.110 2024-04-25T13:47:48.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:48 smithi122 ceph-mon[32843]: pgmap v556: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 9.5 MiB/s wr, 844 op/s 2024-04-25T13:47:48.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:48 smithi045 ceph-mon[32731]: pgmap v556: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 9.5 MiB/s wr, 844 op/s 2024-04-25T13:47:48.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:48 smithi027 ceph-mon[29140]: pgmap v556: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 9.5 MiB/s wr, 844 op/s 2024-04-25T13:47:50.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:50 smithi122 ceph-mon[32843]: pgmap v557: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.0 MiB/s wr, 683 op/s 2024-04-25T13:47:50.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:50 smithi045 ceph-mon[32731]: pgmap v557: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.0 MiB/s wr, 683 op/s 2024-04-25T13:47:50.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:50 smithi027 ceph-mon[29140]: pgmap v557: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.0 MiB/s wr, 683 op/s 2024-04-25T13:47:52.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:52 smithi122 ceph-mon[32843]: pgmap v558: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.2 MiB/s wr, 728 op/s 2024-04-25T13:47:52.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:52 smithi045 ceph-mon[32731]: pgmap v558: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.2 MiB/s wr, 728 op/s 2024-04-25T13:47:52.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:52 smithi027 ceph-mon[29140]: pgmap v558: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.2 MiB/s wr, 728 op/s 2024-04-25T13:47:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 405.0 s, 2057.8 tps, lat 49.042 ms stddev 50.676 2024-04-25T13:47:54.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:54 smithi122 ceph-mon[32843]: pgmap v559: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.2 MiB/s wr, 750 op/s 2024-04-25T13:47:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:54 smithi045 ceph-mon[32731]: pgmap v559: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.2 MiB/s wr, 750 op/s 2024-04-25T13:47:54.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:54 smithi027 ceph-mon[29140]: pgmap v559: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 8.2 MiB/s wr, 750 op/s 2024-04-25T13:47:56.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:56 smithi122 ceph-mon[32843]: pgmap v560: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 7.1 MiB/s wr, 650 op/s 2024-04-25T13:47:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:56 smithi045 ceph-mon[32731]: pgmap v560: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 7.1 MiB/s wr, 650 op/s 2024-04-25T13:47:56.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:56 smithi027 ceph-mon[29140]: pgmap v560: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 7.1 MiB/s wr, 650 op/s 2024-04-25T13:47:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 410.0 s, 2313.0 tps, lat 43.228 ms stddev 47.472 2024-04-25T13:47:58.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:47:58 smithi122 ceph-mon[32843]: pgmap v561: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.2 MiB/s wr, 806 op/s 2024-04-25T13:47:58.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:47:58 smithi045 ceph-mon[32731]: pgmap v561: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.2 MiB/s wr, 806 op/s 2024-04-25T13:47:58.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:47:58 smithi027 ceph-mon[29140]: pgmap v561: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.2 MiB/s wr, 806 op/s 2024-04-25T13:47:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.654+0000 7faada642640 1 -- 172.21.15.27:0/383142670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 msgr2=0x7faad41054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.654+0000 7faada642640 1 --2- 172.21.15.27:0/383142670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 0x7faad41054f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faac4009900 tx=0x7faac402a410 comp rx=0 tx=0).stop 2024-04-25T13:47:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.655+0000 7faada642640 1 -- 172.21.15.27:0/383142670 shutdown_connections 2024-04-25T13:47:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.655+0000 7faada642640 1 --2- 172.21.15.27:0/383142670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad4102290 0x7faad410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.655+0000 7faada642640 1 --2- 172.21.15.27:0/383142670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad4101790 0x7faad4101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.655+0000 7faada642640 1 --2- 172.21.15.27:0/383142670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 0x7faad41054f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.655+0000 7faada642640 1 -- 172.21.15.27:0/383142670 >> 172.21.15.27:0/383142670 conn(0x7faad40fd390 msgr2=0x7faad40ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.655+0000 7faada642640 1 -- 172.21.15.27:0/383142670 shutdown_connections 2024-04-25T13:47:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.655+0000 7faada642640 1 -- 172.21.15.27:0/383142670 wait complete. 2024-04-25T13:47:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.656+0000 7faada642640 1 Processor -- start 2024-04-25T13:47:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.656+0000 7faada642640 1 -- start start 2024-04-25T13:47:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.656+0000 7faada642640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad4101790 0x7faad419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.656+0000 7faada642640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad4102290 0x7faad419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.656+0000 7faada642640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 0x7faad41a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.656+0000 7faad37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad4102290 0x7faad419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faada642640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faad410f630 con 0x7faad4102290 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad37fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad4102290 0x7faad419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60740/0 (socket says 172.21.15.27:60740) 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad4101790 0x7faad419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad37fe640 1 -- 172.21.15.27:0/72028292 learned_addr learned my addr 172.21.15.27:0/72028292 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faada642640 1 -- 172.21.15.27:0/72028292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faad410f4b0 con 0x7faad4101790 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faada642640 1 -- 172.21.15.27:0/72028292 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7faad410f7b0 con 0x7faad41050e0 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad8bdf640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 0x7faad41a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad3fff640 1 -- 172.21.15.27:0/72028292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 msgr2=0x7faad41a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad3fff640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 0x7faad41a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad3fff640 1 -- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad4102290 msgr2=0x7faad419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad3fff640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad4102290 0x7faad419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad3fff640 1 -- 172.21.15.27:0/72028292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad41a1d50 con 0x7faad4101790 2024-04-25T13:47:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad8bdf640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 0x7faad41a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad37fe640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad4102290 0x7faad419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad3fff640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad4101790 0x7faad419a210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faad406a120 tx=0x7faac4031500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.657+0000 7faad17fa640 1 -- 172.21.15.27:0/72028292 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4040070 con 0x7faad4101790 2024-04-25T13:47:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.658+0000 7faada642640 1 -- 172.21.15.27:0/72028292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad41a1fe0 con 0x7faad4101790 2024-04-25T13:47:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.658+0000 7faad17fa640 1 -- 172.21.15.27:0/72028292 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faac40316d0 con 0x7faad4101790 2024-04-25T13:47:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.658+0000 7faad17fa640 1 -- 172.21.15.27:0/72028292 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4035020 con 0x7faad4101790 2024-04-25T13:47:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.658+0000 7faada642640 1 -- 172.21.15.27:0/72028292 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad41a2440 con 0x7faad4101790 2024-04-25T13:47:58.660 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.661+0000 7faae15c7740 1 Processor -- start 2024-04-25T13:47:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.662+0000 7faae15c7740 1 -- start start 2024-04-25T13:47:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.662+0000 7faae15c7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055baad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.662+0000 7faad17fa640 1 -- 172.21.15.27:0/72028292 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7faac4043070 con 0x7faad4101790 2024-04-25T13:47:58.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faae15c7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bab2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055baad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055baad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40850/0 (socket says 172.21.15.27:40850) 2024-04-25T13:47:58.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faae15c7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055ba7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faad8bdf640 1 -- 172.21.15.27:0/3645773454 learned_addr learned my addr 172.21.15.27:0/3645773454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:58.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faae15c7740 1 -- 172.21.15.27:0/3645773454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563055b0d460 con 0x5630557de2f0 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faae15c7740 1 -- 172.21.15.27:0/3645773454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563055ac7590 con 0x5630558071b0 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faae15c7740 1 -- 172.21.15.27:0/3645773454 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563055b09420 con 0x5630557e64d0 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faad37fe640 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055ba7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.663+0000 7faad3fff640 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bab2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad8bdf640 1 -- 172.21.15.27:0/3645773454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 msgr2=0x563055ba7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055ba7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad8bdf640 1 -- 172.21.15.27:0/3645773454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 msgr2=0x563055bab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bab2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad8bdf640 1 -- 172.21.15.27:0/3645773454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563055ad40d0 con 0x5630558071b0 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad3fff640 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad17fa640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faabc0afae0 0x7faabc0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055baad50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faacc011810 tx=0x7faacc011cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a07731b5643052 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad3fff640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faabc0afae0 0x7faabc0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faaba7fc640 1 -- 172.21.15.27:0/3645773454 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc011310 con 0x5630558071b0 2024-04-25T13:47:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faaba7fc640 1 -- 172.21.15.27:0/3645773454 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faacc00f9a0 con 0x5630558071b0 2024-04-25T13:47:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faad17fa640 1 -- 172.21.15.27:0/72028292 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7faac40ff4b0 con 0x7faad4101790 2024-04-25T13:47:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.664+0000 7faaba7fc640 1 -- 172.21.15.27:0/3645773454 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc005610 con 0x5630558071b0 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.665+0000 7faad3fff640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faabc0afae0 0x7faabc0b1ff0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x563055ba8cd0 tx=0x7faac402a410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.665+0000 7faae15c7740 1 -- 172.21.15.27:0/3645773454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 msgr2=0x563055baad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.665+0000 7faae15c7740 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055baad50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faacc011810 tx=0x7faacc011cd0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.665+0000 7faae15c7740 1 -- 172.21.15.27:0/3645773454 shutdown_connections 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.665+0000 7faae15c7740 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055ba7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.665+0000 7faae15c7740 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.665+0000 7faae15c7740 1 --2- 172.21.15.27:0/3645773454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055baad50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.665+0000 7faae15c7740 1 -- 172.21.15.27:0/3645773454 >> 172.21.15.27:0/3645773454 conn(0x5630557fb300 msgr2=0x563055ba3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 -- 172.21.15.27:0/3645773454 shutdown_connections 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 -- 172.21.15.27:0/3645773454 wait complete. 2024-04-25T13:47:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 -- start start 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 Processor -- start 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557de2f0 0x563055bb9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557e64d0 0x563055bba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 0x563055bc1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563055a94b60 con 0x5630558071b0 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563055bb8200 con 0x5630557e64d0 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.666+0000 7faae15c7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563055bc1690 con 0x5630557de2f0 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faae15c7740 10 client.0 _ll_register_callbacks cb 0x563055ab92b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557de2f0 0x563055bb9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557e64d0 0x563055bba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557de2f0 0x563055bb9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33488/0 (socket says 172.21.15.27:33488) 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 -- 172.21.15.27:0/1774116167 learned_addr learned my addr 172.21.15.27:0/1774116167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 0x563055bc1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 -- 172.21.15.27:0/1774116167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557e64d0 msgr2=0x563055bba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557e64d0 0x563055bba260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 -- 172.21.15.27:0/1774116167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 msgr2=0x563055bc1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 0x563055bc1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 -- 172.21.15.27:0/1774116167 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56305581f210 con 0x5630557de2f0 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad37fe640 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557e64d0 0x563055bba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faad8bdf640 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557de2f0 0x563055bb9d20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7faacc004ce0 tx=0x7faacc005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9903c8e12ae5de2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faab37fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faab37fe640 1 -- 172.21.15.27:0/1774116167 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc02d060 con 0x5630557de2f0 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faab37fe640 1 -- 172.21.15.27:0/1774116167 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7faacc003b70 con 0x5630557de2f0 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.667+0000 7faae15c7740 1 -- 172.21.15.27:0/1774116167 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faacc00f9a0 con 0x5630557de2f0 2024-04-25T13:47:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faab37fe640 1 -- 172.21.15.27:0/1774116167 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7faacc01e040 con 0x5630557de2f0 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faae15c7740 10 client.26506 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faae15c7740 1 -- 172.21.15.27:0/1774116167 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5630558eb620 con 0x5630557de2f0 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faab37fe640 1 -- 172.21.15.27:0/1774116167 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faacc00f9a0 con 0x5630557de2f0 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faab37fe640 1 -- 172.21.15.27:0/1774116167 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7faacc024070 con 0x5630557de2f0 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faae15c7740 10 client.26506 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faae15c7740 10 client.26506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faae15c7740 20 client.26506 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faae15c7740 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630557c5d80 0x563055bc7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faae15c7740 4 client.26506 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faad37fe640 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630557c5d80 0x563055bc7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.668+0000 7faae15c7740 1 -- 172.21.15.27:0/1774116167 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563055b1b070 con 0x5630557c5d80 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.669+0000 7faab37fe640 10 client.26506 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.669+0000 7faad37fe640 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630557c5d80 0x563055bc7870 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47bde81d6aad8137 server_cookie=39119b12817208af in_seq=0 out_seq=0 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faab37fe640 1 -- 172.21.15.27:0/1774116167 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563055b1b070 con 0x5630557c5d80 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faab37fe640 10 client.26506 handle_command_reply: tid=0 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 client.26506 shutdown 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 -- 172.21.15.27:0/1774116167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557de2f0 msgr2=0x563055bb9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557de2f0 0x563055bb9d20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7faacc004ce0 tx=0x7faacc005bd0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 -- 172.21.15.27:0/1774116167 shutdown_connections 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5630557c5d80 0x563055bc7870 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 0x563055bc1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557e64d0 0x563055bba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 --2- 172.21.15.27:0/1774116167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557de2f0 0x563055bb9d20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faae15c7740 1 -- 172.21.15.27:0/1774116167 >> 172.21.15.27:0/1774116167 conn(0x5630557fb300 msgr2=0x563055ba2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.670+0000 7faab37fe640 0 client.26506 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.671+0000 7faae15c7740 1 -- 172.21.15.27:0/1774116167 shutdown_connections 2024-04-25T13:47:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.671+0000 7faae15c7740 1 -- 172.21.15.27:0/1774116167 wait complete. 2024-04-25T13:47:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.671+0000 7faae15c7740 20 client.26506 trim_cache size 0 max 16384 2024-04-25T13:47:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.691+0000 7faae15c7740 1 Processor -- start 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.691+0000 7faae15c7740 1 -- start start 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.691+0000 7faae15c7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557de2f0 0x563055bba000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faae15c7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bcc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faae15c7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 0x563055bcc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faae15c7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faad41a6fa0 con 0x5630558071b0 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faae15c7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faad41a9820 con 0x5630557de2f0 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faae15c7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7faad41a7610 con 0x5630557e64d0 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 0x563055bcc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557de2f0 0x563055bba000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557de2f0 0x563055bba000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40868/0 (socket says 172.21.15.27:40868) 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 -- 172.21.15.27:0/3673563722 learned_addr learned my addr 172.21.15.27:0/3673563722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad37fe640 1 --2- 172.21.15.27:0/3673563722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bcc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 -- 172.21.15.27:0/3673563722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 msgr2=0x563055bcc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3673563722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bcc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 -- 172.21.15.27:0/3673563722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 msgr2=0x563055bcc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3673563722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 0x563055bcc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 -- 172.21.15.27:0/3673563722 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563055bb4410 con 0x5630557de2f0 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.692+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3673563722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557de2f0 0x563055bba000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faacc004d10 tx=0x7faacc011cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be895c80de0d90ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.693+0000 7faab9ffb640 1 -- 172.21.15.27:0/3673563722 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc02d070 con 0x5630557de2f0 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.693+0000 7faab9ffb640 1 -- 172.21.15.27:0/3673563722 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faacc03b070 con 0x5630557de2f0 2024-04-25T13:47:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.693+0000 7faab9ffb640 1 -- 172.21.15.27:0/3673563722 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc034d00 con 0x5630557de2f0 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.693+0000 7faae15c7740 1 -- 172.21.15.27:0/3673563722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557de2f0 msgr2=0x563055bba000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.694+0000 7faae15c7740 1 --2- 172.21.15.27:0/3673563722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557de2f0 0x563055bba000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faacc004d10 tx=0x7faacc011cd0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.694+0000 7faae15c7740 1 -- 172.21.15.27:0/3673563722 shutdown_connections 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.694+0000 7faae15c7740 1 --2- 172.21.15.27:0/3673563722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630558071b0 0x563055bcc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.694+0000 7faae15c7740 1 --2- 172.21.15.27:0/3673563722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bcc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.694+0000 7faae15c7740 1 --2- 172.21.15.27:0/3673563722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630557de2f0 0x563055bba000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.694+0000 7faae15c7740 1 -- 172.21.15.27:0/3673563722 >> 172.21.15.27:0/3673563722 conn(0x5630557fb300 msgr2=0x563055bb98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.694+0000 7faae15c7740 1 -- 172.21.15.27:0/3673563722 shutdown_connections 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.694+0000 7faae15c7740 1 -- 172.21.15.27:0/3673563722 wait complete. 2024-04-25T13:47:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 1 -- start start 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 1 Processor -- start 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bbdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bbe4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055c2dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faad41a7ce0 con 0x5630557de2f0 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faad41a9150 con 0x5630558071b0 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7faad41a83b0 con 0x5630557e64d0 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faae15c7740 10 client.0 _ll_register_callbacks cb 0x7faac8009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bbdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faad8bdf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bbdf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60768/0 (socket says 172.21.15.27:60768) 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faad8bdf640 1 -- 172.21.15.27:0/3695718214 learned_addr learned my addr 172.21.15.27:0/3695718214 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.695+0000 7faad37fe640 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bbe4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055c2dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faad37fe640 1 -- 172.21.15.27:0/3695718214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 msgr2=0x563055c2dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faad37fe640 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055c2dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faad37fe640 1 -- 172.21.15.27:0/3695718214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 msgr2=0x563055bbdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faad37fe640 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bbdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faad37fe640 1 -- 172.21.15.27:0/3695718214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563055c2e150 con 0x5630557e64d0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faad37fe640 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bbe4c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x563055bc7fe0 tx=0x7faac8012f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ff3edb64f27df27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faab37fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faab37fe640 1 -- 172.21.15.27:0/3695718214 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8022830 con 0x5630557e64d0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faae15c7740 1 -- 172.21.15.27:0/3695718214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faacc03b070 con 0x5630557e64d0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faab37fe640 1 -- 172.21.15.27:0/3695718214 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7faac80229d0 con 0x5630557e64d0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.696+0000 7faab37fe640 1 -- 172.21.15.27:0/3695718214 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7faac8032050 con 0x5630557e64d0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.697+0000 7faae15c7740 10 client.26512 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.697+0000 7faae15c7740 1 -- 172.21.15.27:0/3695718214 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563055c2f330 con 0x5630557e64d0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.697+0000 7faab37fe640 1 -- 172.21.15.27:0/3695718214 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faacc03b070 con 0x5630557e64d0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.697+0000 7faab37fe640 1 -- 172.21.15.27:0/3695718214 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7faac8010630 con 0x5630557e64d0 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.698+0000 7faae15c7740 10 client.26512 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.698+0000 7faae15c7740 10 client.26512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.698+0000 7faae15c7740 20 client.26512 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.698+0000 7faae15c7740 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56305579b1c0 0x563055c31c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.698+0000 7faae15c7740 4 client.26512 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:47:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.698+0000 7faae15c7740 1 -- 172.21.15.27:0/3695718214 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563055b14ee0 con 0x56305579b1c0 2024-04-25T13:47:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.698+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56305579b1c0 0x563055c31c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.699+0000 7faab37fe640 10 client.26512 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.699+0000 7faad8bdf640 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56305579b1c0 0x563055c31c40 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8453ebd13229a05 server_cookie=4a2d26100b88834f in_seq=0 out_seq=0 2024-04-25T13:47:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faab37fe640 1 -- 172.21.15.27:0/3695718214 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563055b14ee0 con 0x56305579b1c0 2024-04-25T13:47:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faab37fe640 10 client.26512 handle_command_reply: tid=0 2024-04-25T13:47:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 client.26512 shutdown 2024-04-25T13:47:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 -- 172.21.15.27:0/3695718214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 msgr2=0x563055bbe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bbe4c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x563055bc7fe0 tx=0x7faac8012f30 comp rx=0 tx=0).stop 2024-04-25T13:47:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 -- 172.21.15.27:0/3695718214 shutdown_connections 2024-04-25T13:47:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56305579b1c0 0x563055c31c40 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630558071b0 0x563055c2dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630557e64d0 0x563055bbe4c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faab37fe640 0 client.26512 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 --2- 172.21.15.27:0/3695718214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5630557de2f0 0x563055bbdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 -- 172.21.15.27:0/3695718214 >> 172.21.15.27:0/3695718214 conn(0x5630557fb300 msgr2=0x563055bb8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 -- 172.21.15.27:0/3695718214 shutdown_connections 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 1 -- 172.21.15.27:0/3695718214 wait complete. 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.714+0000 7faae15c7740 20 client.26512 trim_cache size 0 max 16384 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:47:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:47:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:47:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 -- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faabc0afae0 msgr2=0x7faabc0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faabc0afae0 0x7faabc0b1ff0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x563055ba8cd0 tx=0x7faac402a410 comp rx=0 tx=0).stop 2024-04-25T13:47:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 -- 172.21.15.27:0/72028292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad4101790 msgr2=0x7faad419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad4101790 0x7faad419a210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faad406a120 tx=0x7faac4031500 comp rx=0 tx=0).stop 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 -- 172.21.15.27:0/72028292 shutdown_connections 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faabc0afae0 0x7faabc0b1ff0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad41050e0 0x7faad41a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad4102290 0x7faad419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 --2- 172.21.15.27:0/72028292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faad4101790 0x7faad419a210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 -- 172.21.15.27:0/72028292 >> 172.21.15.27:0/72028292 conn(0x7faad40fd390 msgr2=0x7faad4105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.715+0000 7faada642640 1 -- 172.21.15.27:0/72028292 shutdown_connections 2024-04-25T13:47:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:47:58.716+0000 7faada642640 1 -- 172.21.15.27:0/72028292 wait complete. 2024-04-25T13:47:59.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:47:59.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:47:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.254+0000 7f0d152d1640 1 -- 172.21.15.27:0/3761143374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d101043c0 msgr2=0x7f0d10102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.254+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3761143374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d101043c0 0x7f0d10102dc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009900 tx=0x7f0d0402a410 comp rx=0 tx=0).stop 2024-04-25T13:47:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.254+0000 7f0d152d1640 1 -- 172.21.15.27:0/3761143374 shutdown_connections 2024-04-25T13:47:59.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.254+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3761143374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d10108a10 0x7f0d1010ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.254+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3761143374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d101043c0 0x7f0d10102dc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.254+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3761143374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d10103ee0 0x7f0d101027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.254+0000 7f0d152d1640 1 -- 172.21.15.27:0/3761143374 >> 172.21.15.27:0/3761143374 conn(0x7f0d100fd340 msgr2=0x7f0d100ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.255+0000 7f0d152d1640 1 -- 172.21.15.27:0/3761143374 shutdown_connections 2024-04-25T13:47:59.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.255+0000 7f0d152d1640 1 -- 172.21.15.27:0/3761143374 wait complete. 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.255+0000 7f0d152d1640 1 Processor -- start 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d152d1640 1 -- start start 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d152d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d10103ee0 0x7f0d10199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d152d1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d10108a10 0x7f0d1019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d152d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d1019f260 0x7f0d101a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d152d1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1010f620 con 0x7f0d10103ee0 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d152d1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1010f4a0 con 0x7f0d1019f260 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d152d1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1010f7a0 con 0x7f0d10108a10 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d0effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d10103ee0 0x7f0d10199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d0e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d10108a10 0x7f0d1019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d1019f260 0x7f0d101a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d0effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d10103ee0 0x7f0d10199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60786/0 (socket says 172.21.15.27:60786) 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.256+0000 7f0d0effd640 1 -- 172.21.15.27:0/3890232561 learned_addr learned my addr 172.21.15.27:0/3890232561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d0effd640 1 -- 172.21.15.27:0/3890232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d10108a10 msgr2=0x7f0d1019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d0effd640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d10108a10 0x7f0d1019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d0effd640 1 -- 172.21.15.27:0/3890232561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d1019f260 msgr2=0x7f0d101a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d0effd640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d1019f260 0x7f0d101a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d0effd640 1 -- 172.21.15.27:0/3890232561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d101a1cf0 con 0x7f0d10103ee0 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d0e7fc640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d10108a10 0x7f0d1019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d1019f260 0x7f0d101a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d0effd640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d10103ee0 0x7f0d10199da0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000e9c0 tx=0x7f0d0000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0cf7fff640 1 -- 172.21.15.27:0/3890232561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0000cd80 con 0x7f0d10103ee0 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d152d1640 1 -- 172.21.15.27:0/3890232561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d101a1fe0 con 0x7f0d10103ee0 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0cf7fff640 1 -- 172.21.15.27:0/3890232561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d000054d0 con 0x7f0d10103ee0 2024-04-25T13:47:59.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0cf7fff640 1 -- 172.21.15.27:0/3890232561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00010430 con 0x7f0d10103ee0 2024-04-25T13:47:59.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.257+0000 7f0d152d1640 1 -- 172.21.15.27:0/3890232561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d101a24a0 con 0x7f0d10103ee0 2024-04-25T13:47:59.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.259+0000 7f0d1c180740 1 Processor -- start 2024-04-25T13:47:59.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d1c180740 1 -- start start 2024-04-25T13:47:59.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb23da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb23dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae7b6b0 0x557abb23ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d1c180740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557abb1a2d80 con 0x557abae7b6b0 2024-04-25T13:47:59.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d1c180740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557abb15c8b0 con 0x557abae9c390 2024-04-25T13:47:59.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d1c180740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557abb19ed40 con 0x557abae734d0 2024-04-25T13:47:59.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0cf7fff640 1 -- 172.21.15.27:0/3890232561 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0d00005020 con 0x7f0d10103ee0 2024-04-25T13:47:59.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb23da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae7b6b0 0x557abb23ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb23da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40898/0 (socket says 172.21.15.27:40898) 2024-04-25T13:47:59.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/4042605579 learned_addr learned my addr 172.21.15.27:0/4042605579 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/4042605579 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 msgr2=0x557abb23dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/4042605579 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb23dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/4042605579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae7b6b0 msgr2=0x557abb23ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/4042605579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae7b6b0 0x557abb23ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.260+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/4042605579 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557abb12da40 con 0x557abae9c390 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.262+0000 7f0cf7fff640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0cf80afaf0 0x7f0cf80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.262+0000 7f0cf7fff640 1 -- 172.21.15.27:0/3890232561 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0d000df000 con 0x7f0d10103ee0 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.263+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/4042605579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb23da60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0d080117e0 tx=0x7f0d08011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=931c6bb29155db8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.263+0000 7f0cf57fa640 1 -- 172.21.15.27:0/4042605579 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08011250 con 0x557abae9c390 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.263+0000 7f0cf57fa640 1 -- 172.21.15.27:0/4042605579 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d0801e070 con 0x557abae9c390 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.263+0000 7f0cf57fa640 1 -- 172.21.15.27:0/4042605579 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0801aab0 con 0x557abae9c390 2024-04-25T13:47:59.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.263+0000 7f0d0effd640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0cf80afaf0 0x7f0cf80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.264+0000 7f0d0effd640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0cf80afaf0 0x7f0cf80b2000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0d000a74a0 tx=0x7f0d00005680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.264+0000 7f0d1c180740 1 -- 172.21.15.27:0/4042605579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 msgr2=0x557abb23da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 --2- 172.21.15.27:0/4042605579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb23da60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0d080117e0 tx=0x7f0d08011ca0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 -- 172.21.15.27:0/4042605579 shutdown_connections 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 --2- 172.21.15.27:0/4042605579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae7b6b0 0x557abb23ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 --2- 172.21.15.27:0/4042605579 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb23dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 --2- 172.21.15.27:0/4042605579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb23da60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 -- 172.21.15.27:0/4042605579 >> 172.21.15.27:0/4042605579 conn(0x557abae904e0 msgr2=0x557abb239270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 -- 172.21.15.27:0/4042605579 shutdown_connections 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 -- 172.21.15.27:0/4042605579 wait complete. 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 -- start start 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.265+0000 7f0d1c180740 1 Processor -- start 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae734d0 0x557abb247960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae7b6b0 0x557abb247ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb24ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d1c180740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557abb23f780 con 0x557abae734d0 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae734d0 0x557abb247960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae734d0 0x557abb247960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60796/0 (socket says 172.21.15.27:60796) 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/1162824077 learned_addr learned my addr 172.21.15.27:0/1162824077 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557abb129d10 con 0x557abae9c390 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557abb23f900 con 0x557abae7b6b0 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0e7fc640 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae7b6b0 0x557abb247ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb24ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0e7fc640 1 -- 172.21.15.27:0/1162824077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 msgr2=0x557abb24ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0e7fc640 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb24ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0e7fc640 1 -- 172.21.15.27:0/1162824077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae734d0 msgr2=0x557abb247960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0e7fc640 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae734d0 0x557abb247960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0e7fc640 1 -- 172.21.15.27:0/1162824077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557abaf7e7d0 con 0x557abae7b6b0 2024-04-25T13:47:59.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d1c180740 10 client.0 _ll_register_callbacks cb 0x557abb156210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d0801e070 con 0x557abae7b6b0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0d0e7fc640 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae7b6b0 0x557abb247ea0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d04000c00 tx=0x7f0d040092f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8de22f85c51391 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.266+0000 7f0cf2ff5640 10 client.26521 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/1162824077 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d04035040 con 0x557abae7b6b0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/1162824077 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0d04004280 con 0x557abae7b6b0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/1162824077 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0d040318a0 con 0x557abae7b6b0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0d1c180740 10 client.26521 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x557abaeb43f0 con 0x557abae7b6b0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/1162824077 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f0d04005030 con 0x557abae7b6b0 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0d1c180740 10 client.26521 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0d1c180740 10 client.26521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0d1c180740 20 client.26521 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.267+0000 7f0d1c180740 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557abae5af60 0x557abb255150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.268+0000 7f0d1c180740 4 client.26521 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:59.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.268+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557abb1b0990 con 0x557abae5af60 2024-04-25T13:47:59.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.268+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557abae5af60 0x557abb255150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.269+0000 7f0cf2ff5640 10 client.26521 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:59.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.269+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557abae5af60 0x557abb255150 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7de73143330b9167 server_cookie=24c3aa85c0d24419 in_seq=0 out_seq=0 2024-04-25T13:47:59.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/1162824077 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557abb1b0990 con 0x557abae5af60 2024-04-25T13:47:59.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0cf2ff5640 10 client.26521 handle_command_reply: tid=0 2024-04-25T13:47:59.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 client.26521 shutdown 2024-04-25T13:47:59.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae7b6b0 msgr2=0x557abb247ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae7b6b0 0x557abb247ea0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d04000c00 tx=0x7f0d040092f0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 shutdown_connections 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557abae5af60 0x557abb255150 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae9c390 0x557abb24ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0cf2ff5640 0 client.26521 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae7b6b0 0x557abb247ea0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 --2- 172.21.15.27:0/1162824077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae734d0 0x557abb247960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.270+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 >> 172.21.15.27:0/1162824077 conn(0x557abae904e0 msgr2=0x557abb2383f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.271+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 shutdown_connections 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.271+0000 7f0d1c180740 1 -- 172.21.15.27:0/1162824077 wait complete. 2024-04-25T13:47:59.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.271+0000 7f0d1c180740 20 client.26521 trim_cache size 0 max 16384 2024-04-25T13:47:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d1c180740 1 Processor -- start 2024-04-25T13:47:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d1c180740 1 -- start start 2024-04-25T13:47:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae904e0 0x557abb2379c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae5af60 0x557abb237f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb238440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d1c180740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d101a6fe0 con 0x557abae5af60 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d1c180740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d101a9860 con 0x557abae904e0 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d1c180740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0d101a7650 con 0x557abae734d0 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae5af60 0x557abb237f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae904e0 0x557abb2379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae904e0 0x557abb2379c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40910/0 (socket says 172.21.15.27:40910) 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae5af60 0x557abb237f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60812/0 (socket says 172.21.15.27:60812) 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/2246415190 learned_addr learned my addr 172.21.15.27:0/2246415190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/2246415190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb238440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/2246415190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 msgr2=0x557abb238440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/2246415190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb238440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/2246415190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae5af60 msgr2=0x557abb237f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/2246415190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae5af60 0x557abb237f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.295+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/2246415190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557abae740f0 con 0x557abae904e0 2024-04-25T13:47:59.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.296+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/2246415190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae904e0 0x557abb2379c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0d08000f80 tx=0x7f0d0800fb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18d38c40edec28e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.296+0000 7f0cf4ff9640 1 -- 172.21.15.27:0/2246415190 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08004c00 con 0x557abae904e0 2024-04-25T13:47:59.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.296+0000 7f0cf4ff9640 1 -- 172.21.15.27:0/2246415190 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d08004da0 con 0x557abae904e0 2024-04-25T13:47:59.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.296+0000 7f0cf4ff9640 1 -- 172.21.15.27:0/2246415190 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08005680 con 0x557abae904e0 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.297+0000 7f0d1c180740 1 -- 172.21.15.27:0/2246415190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae904e0 msgr2=0x557abb2379c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.297+0000 7f0d1c180740 1 --2- 172.21.15.27:0/2246415190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae904e0 0x557abb2379c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0d08000f80 tx=0x7f0d0800fb00 comp rx=0 tx=0).stop 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.298+0000 7f0d1c180740 1 -- 172.21.15.27:0/2246415190 shutdown_connections 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.298+0000 7f0d1c180740 1 --2- 172.21.15.27:0/2246415190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb238440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.298+0000 7f0d1c180740 1 --2- 172.21.15.27:0/2246415190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae5af60 0x557abb237f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.298+0000 7f0d1c180740 1 --2- 172.21.15.27:0/2246415190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae904e0 0x557abb2379c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.298+0000 7f0d1c180740 1 -- 172.21.15.27:0/2246415190 >> 172.21.15.27:0/2246415190 conn(0x557abb2344d0 msgr2=0x557abb242680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.299+0000 7f0d1c180740 1 -- 172.21.15.27:0/2246415190 shutdown_connections 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.299+0000 7f0d1c180740 1 -- 172.21.15.27:0/2246415190 wait complete. 2024-04-25T13:47:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.299+0000 7f0d1c180740 1 -- start start 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d1c180740 1 Processor -- start 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae5af60 0x557abb24ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb24bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d1c180740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae904e0 0x557abb24ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d1c180740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0d101a7d20 con 0x557abae904e0 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d1c180740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d101a9190 con 0x557abae5af60 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d1c180740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0d101a83f0 con 0x557abae734d0 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d1c180740 10 client.0 _ll_register_callbacks cb 0x7f0d080093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae904e0 0x557abb24ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb24bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae904e0 0x557abb24ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60816/0 (socket says 172.21.15.27:60816) 2024-04-25T13:47:59.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0e7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb24bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33548/0 (socket says 172.21.15.27:33548) 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0f7fe640 1 -- 172.21.15.27:0/717434842 learned_addr learned my addr 172.21.15.27:0/717434842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae5af60 0x557abb24ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0e7fc640 1 -- 172.21.15.27:0/717434842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae5af60 msgr2=0x557abb24ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0e7fc640 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae5af60 0x557abb24ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0e7fc640 1 -- 172.21.15.27:0/717434842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae904e0 msgr2=0x557abb24ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0e7fc640 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae904e0 0x557abb24ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0e7fc640 1 -- 172.21.15.27:0/717434842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557abb1634b0 con 0x557abae734d0 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.300+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae904e0 0x557abb24ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.301+0000 7f0d0e7fc640 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb24bf50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0d0402a9b0 tx=0x7f0d04009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9665152f3311d40b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.301+0000 7f0cf2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.301+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/717434842 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0405d060 con 0x557abae734d0 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.301+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/717434842 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0d04045880 con 0x557abae734d0 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.301+0000 7f0d1c180740 1 -- 172.21.15.27:0/717434842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d08004da0 con 0x557abae734d0 2024-04-25T13:47:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.301+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/717434842 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0d04035070 con 0x557abae734d0 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.301+0000 7f0d1c180740 10 client.26527 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.301+0000 7f0d1c180740 1 -- 172.21.15.27:0/717434842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557abb24e500 con 0x557abae734d0 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/717434842 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d08004da0 con 0x557abae734d0 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/717434842 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f0d04057030 con 0x557abae734d0 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0d1c180740 10 client.26527 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0d1c180740 10 client.26527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0d1c180740 20 client.26527 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0d1c180740 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557abae7b6b0 0x557abb2bfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0d1c180740 4 client.26527 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0d1c180740 1 -- 172.21.15.27:0/717434842 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557abb1aa800 con 0x557abae7b6b0 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557abae7b6b0 0x557abb2bfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0cf2ff5640 10 client.26527 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:59.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.302+0000 7f0d0f7fe640 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557abae7b6b0 0x557abb2bfa60 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a2788a32a5b415f server_cookie=d25aeb43d4af35eb in_seq=0 out_seq=0 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0cf2ff5640 1 -- 172.21.15.27:0/717434842 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557abb1aa800 con 0x557abae7b6b0 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0cf2ff5640 10 client.26527 handle_command_reply: tid=0 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 client.26527 shutdown 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 -- 172.21.15.27:0/717434842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 msgr2=0x557abb24bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb24bf50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0d0402a9b0 tx=0x7f0d04009260 comp rx=0 tx=0).stop 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 -- 172.21.15.27:0/717434842 shutdown_connections 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557abae7b6b0 0x557abb2bfa60 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557abae904e0 0x557abb24ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0cf2ff5640 0 client.26527 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557abae734d0 0x557abb24bf50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 --2- 172.21.15.27:0/717434842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557abae5af60 0x557abb24ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 -- 172.21.15.27:0/717434842 >> 172.21.15.27:0/717434842 conn(0x557abb2344d0 msgr2=0x557abb245f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 -- 172.21.15.27:0/717434842 shutdown_connections 2024-04-25T13:47:59.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 1 -- 172.21.15.27:0/717434842 wait complete. 2024-04-25T13:47:59.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d1c180740 20 client.26527 trim_cache size 0 max 16384 2024-04-25T13:47:59.308 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:47:59.308 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:47:59.308 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:47:59.308 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:47:59.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d152d1640 1 -- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0cf80afaf0 msgr2=0x7f0cf80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0cf80afaf0 0x7f0cf80b2000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0d000a74a0 tx=0x7f0d00005680 comp rx=0 tx=0).stop 2024-04-25T13:47:59.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d152d1640 1 -- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d10103ee0 msgr2=0x7f0d10199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.303+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d10103ee0 0x7f0d10199da0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000e9c0 tx=0x7f0d0000ee80 comp rx=0 tx=0).stop 2024-04-25T13:47:59.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.304+0000 7f0d152d1640 1 -- 172.21.15.27:0/3890232561 shutdown_connections 2024-04-25T13:47:59.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.304+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0cf80afaf0 0x7f0cf80b2000 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.304+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d1019f260 0x7f0d101a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.304+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d10108a10 0x7f0d1019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.304+0000 7f0d152d1640 1 --2- 172.21.15.27:0/3890232561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d10103ee0 0x7f0d10199da0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.304+0000 7f0d152d1640 1 -- 172.21.15.27:0/3890232561 >> 172.21.15.27:0/3890232561 conn(0x7f0d100fd340 msgr2=0x7f0d100ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.304+0000 7f0d152d1640 1 -- 172.21.15.27:0/3890232561 shutdown_connections 2024-04-25T13:47:59.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.304+0000 7f0d152d1640 1 -- 172.21.15.27:0/3890232561 wait complete. 2024-04-25T13:47:59.319 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:47:59.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:47:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.471+0000 7ff17852d640 1 -- 172.21.15.27:0/742260927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1701045d0 msgr2=0x7ff1701069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.471+0000 7ff17852d640 1 --2- 172.21.15.27:0/742260927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1701045d0 0x7ff1701069f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009900 tx=0x7ff16c02a410 comp rx=0 tx=0).stop 2024-04-25T13:47:59.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.471+0000 7ff17852d640 1 -- 172.21.15.27:0/742260927 shutdown_connections 2024-04-25T13:47:59.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.471+0000 7ff17852d640 1 --2- 172.21.15.27:0/742260927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff170109890 0x7ff17010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.471+0000 7ff17852d640 1 --2- 172.21.15.27:0/742260927 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff170106f30 0x7ff170109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.471+0000 7ff17852d640 1 --2- 172.21.15.27:0/742260927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1701045d0 0x7ff1701069f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.471+0000 7ff17852d640 1 -- 172.21.15.27:0/742260927 >> 172.21.15.27:0/742260927 conn(0x7ff1700fdbd0 msgr2=0x7ff170100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.472+0000 7ff17852d640 1 -- 172.21.15.27:0/742260927 shutdown_connections 2024-04-25T13:47:59.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.472+0000 7ff17852d640 1 -- 172.21.15.27:0/742260927 wait complete. 2024-04-25T13:47:59.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.472+0000 7ff17852d640 1 Processor -- start 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff17852d640 1 -- start start 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff17852d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1701045d0 0x7ff17019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff17852d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff170106f30 0x7ff17019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff17852d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff170109890 0x7ff1701a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff17852d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff17010f5e0 con 0x7ff170106f30 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff17852d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff17010f460 con 0x7ff1701045d0 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff17852d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff17010f760 con 0x7ff170109890 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff170106f30 0x7ff17019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff170106f30 0x7ff17019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60822/0 (socket says 172.21.15.27:60822) 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 -- 172.21.15.27:0/2454603278 learned_addr learned my addr 172.21.15.27:0/2454603278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff1762c9640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1701045d0 0x7ff17019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff176aca640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff170109890 0x7ff1701a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 -- 172.21.15.27:0/2454603278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff170109890 msgr2=0x7ff1701a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff170109890 0x7ff1701a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 -- 172.21.15.27:0/2454603278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1701045d0 msgr2=0x7ff17019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1701045d0 0x7ff17019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 -- 172.21.15.27:0/2454603278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1701a1d50 con 0x7ff170106f30 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff1762c9640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1701045d0 0x7ff17019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.473+0000 7ff175ac8640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff170106f30 0x7ff17019a750 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff16000e9f0 tx=0x7ff16000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.474+0000 7ff1677fe640 1 -- 172.21.15.27:0/2454603278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16000cd80 con 0x7ff170106f30 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.474+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1701a1fe0 con 0x7ff170106f30 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.474+0000 7ff1677fe640 1 -- 172.21.15.27:0/2454603278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1600054d0 con 0x7ff170106f30 2024-04-25T13:47:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.474+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1701a24d0 con 0x7ff170106f30 2024-04-25T13:47:59.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.474+0000 7ff1677fe640 1 -- 172.21.15.27:0/2454603278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160018630 con 0x7ff170106f30 2024-04-25T13:47:59.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.475+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1700ff6f0 con 0x7ff170106f30 2024-04-25T13:47:59.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.478+0000 7ff1677fe640 1 -- 172.21.15.27:0/2454603278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff160004e70 con 0x7ff170106f30 2024-04-25T13:47:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.479+0000 7ff1677fe640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff15c0afaf0 0x7ff15c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.479+0000 7ff1677fe640 1 -- 172.21.15.27:0/2454603278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff1600df700 con 0x7ff170106f30 2024-04-25T13:47:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.479+0000 7ff1762c9640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff15c0afaf0 0x7ff15c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.480+0000 7ff1762c9640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff15c0afaf0 0x7ff15c0b2000 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009230 tx=0x7ff16c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.482+0000 7ff1677fe640 1 -- 172.21.15.27:0/2454603278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1600a8310 con 0x7ff170106f30 2024-04-25T13:47:59.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.798+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff17002db50 con 0x7ff170106f30 2024-04-25T13:47:59.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.801+0000 7ff1677fe640 1 -- 172.21.15.27:0/2454603278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7ff1600a7c10 con 0x7ff170106f30 2024-04-25T13:47:59.801 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:47:59.801 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:47:59.801 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:47:59.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.805+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff15c0afaf0 msgr2=0x7ff15c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.805+0000 7ff17852d640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff15c0afaf0 0x7ff15c0b2000 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009230 tx=0x7ff16c031400 comp rx=0 tx=0).stop 2024-04-25T13:47:59.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.805+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff170106f30 msgr2=0x7ff17019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.805+0000 7ff17852d640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff170106f30 0x7ff17019a750 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff16000e9f0 tx=0x7ff16000edf0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.805+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 shutdown_connections 2024-04-25T13:47:59.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.806+0000 7ff17852d640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff15c0afaf0 0x7ff15c0b2000 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.806+0000 7ff17852d640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff170109890 0x7ff1701a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.806+0000 7ff17852d640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff170106f30 0x7ff17019a750 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.806+0000 7ff17852d640 1 --2- 172.21.15.27:0/2454603278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1701045d0 0x7ff17019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.806+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 >> 172.21.15.27:0/2454603278 conn(0x7ff1700fdbd0 msgr2=0x7ff1701052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.807+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 shutdown_connections 2024-04-25T13:47:59.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.807+0000 7ff17852d640 1 -- 172.21.15.27:0/2454603278 wait complete. 2024-04-25T13:47:59.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:47:59.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.974+0000 7f192d97d640 1 -- 172.21.15.27:0/3646843597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 msgr2=0x7f19281066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.974+0000 7f192d97d640 1 --2- 172.21.15.27:0/3646843597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 0x7f19281066a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1918009900 tx=0x7f191802a3f0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 -- 172.21.15.27:0/3646843597 shutdown_connections 2024-04-25T13:47:59.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 --2- 172.21.15.27:0/3646843597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1928109780 0x7f192810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 --2- 172.21.15.27:0/3646843597 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1928106c70 0x7f19281090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 --2- 172.21.15.27:0/3646843597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 0x7f19281066a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 -- 172.21.15.27:0/3646843597 >> 172.21.15.27:0/3646843597 conn(0x7f19280fdbe0 msgr2=0x7f1928100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 -- 172.21.15.27:0/3646843597 shutdown_connections 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 -- 172.21.15.27:0/3646843597 wait complete. 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 Processor -- start 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.975+0000 7f192d97d640 1 -- start start 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f192d97d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 0x7f192819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f192d97d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1928106c70 0x7f192819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f192d97d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1928109780 0x7f19281a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f192d97d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f192810f5f0 con 0x7f1928106c70 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f192d97d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f192810f470 con 0x7f1928104280 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f192d97d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f192810f770 con 0x7f1928109780 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f19267fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1928106c70 0x7f192819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 0x7f192819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 0x7f192819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40952/0 (socket says 172.21.15.27:40952) 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 -- 172.21.15.27:0/1802303448 learned_addr learned my addr 172.21.15.27:0/1802303448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f19277fe640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1928109780 0x7f19281a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 -- 172.21.15.27:0/1802303448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1928109780 msgr2=0x7f19281a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1928109780 0x7f19281a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 -- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1928106c70 msgr2=0x7f192819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1928106c70 0x7f192819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 -- 172.21.15.27:0/1802303448 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19281a1e10 con 0x7f1928104280 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.976+0000 7f1926ffd640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 0x7f192819a2d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1918006010 tx=0x7f1918004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.977+0000 7f192c97b640 1 -- 172.21.15.27:0/1802303448 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191803f030 con 0x7f1928104280 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.977+0000 7f192d97d640 1 -- 172.21.15.27:0/1802303448 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19281a20a0 con 0x7f1928104280 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.977+0000 7f192c97b640 1 -- 172.21.15.27:0/1802303448 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1918003aa0 con 0x7f1928104280 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.977+0000 7f192c97b640 1 -- 172.21.15.27:0/1802303448 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918035070 con 0x7f1928104280 2024-04-25T13:47:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.977+0000 7f192d97d640 1 -- 172.21.15.27:0/1802303448 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19281a2590 con 0x7f1928104280 2024-04-25T13:47:59.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.979+0000 7f192c97b640 1 -- 172.21.15.27:0/1802303448 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1918042030 con 0x7f1928104280 2024-04-25T13:47:59.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.979+0000 7f193482e740 1 Processor -- start 2024-04-25T13:47:59.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f193482e740 1 -- start start 2024-04-25T13:47:59.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f193482e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d0c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f193482e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d0cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f193482e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 0x55dec8d0da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f193482e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dec8c70d80 con 0x55dec89496b0 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f193482e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55dec8c2aa20 con 0x55dec89414d0 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f193482e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55dec8c6cd40 con 0x55dec896a390 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19277fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d0c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19267fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 0x55dec8d0da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d0cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19277fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d0c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33592/0 (socket says 172.21.15.27:33592) 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19277fe640 1 -- 172.21.15.27:0/1849531653 learned_addr learned my addr 172.21.15.27:0/1849531653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f192c97b640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f19100afaf0 0x7f19100b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19277fe640 1 -- 172.21.15.27:0/1849531653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 msgr2=0x55dec8d0cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19277fe640 1 --2- 172.21.15.27:0/1849531653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d0cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19277fe640 1 -- 172.21.15.27:0/1849531653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 msgr2=0x55dec8d0da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19277fe640 1 --2- 172.21.15.27:0/1849531653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 0x55dec8d0da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.980+0000 7f19277fe640 1 -- 172.21.15.27:0/1849531653 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec8c35c30 con 0x55dec896a390 2024-04-25T13:47:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.981+0000 7f19277fe640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f19100afaf0 0x7f19100b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.981+0000 7f192c97b640 1 -- 172.21.15.27:0/1802303448 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f19180ff220 con 0x7f1928104280 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.981+0000 7f19277fe640 1 --2- 172.21.15.27:0/1849531653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d0c810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1920003b10 tx=0x7f1920009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd64f744e9bb9f00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.981+0000 7f190dffb640 1 -- 172.21.15.27:0/1849531653 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920024070 con 0x55dec896a390 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.981+0000 7f19277fe640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f19100afaf0 0x7f19100b2000 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1920021000 tx=0x7f192001e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.981+0000 7f190dffb640 1 -- 172.21.15.27:0/1849531653 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19200132d0 con 0x55dec896a390 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.981+0000 7f190dffb640 1 -- 172.21.15.27:0/1849531653 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192001e910 con 0x55dec896a390 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 -- 172.21.15.27:0/1849531653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 msgr2=0x55dec8d0c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 --2- 172.21.15.27:0/1849531653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d0c810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1920003b10 tx=0x7f1920009f90 comp rx=0 tx=0).stop 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 -- 172.21.15.27:0/1849531653 shutdown_connections 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 --2- 172.21.15.27:0/1849531653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 0x55dec8d0da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 --2- 172.21.15.27:0/1849531653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d0cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 --2- 172.21.15.27:0/1849531653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d0c810 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 -- 172.21.15.27:0/1849531653 >> 172.21.15.27:0/1849531653 conn(0x55dec895e4e0 msgr2=0x55dec8d08020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 -- 172.21.15.27:0/1849531653 shutdown_connections 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 -- 172.21.15.27:0/1849531653 wait complete. 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 -- start start 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.982+0000 7f193482e740 1 Processor -- start 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f193482e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89414d0 0x55dec8d165a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f193482e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d16ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f193482e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec896a390 0x55dec8d1da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f193482e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dec8d0e530 con 0x55dec89414d0 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f193482e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55dec8bf7dc0 con 0x55dec896a390 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f193482e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55dec8d0e6b0 con 0x55dec89496b0 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f193482e740 10 client.0 _ll_register_callbacks cb 0x55dec8c13600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89414d0 0x55dec8d165a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f19267fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d16ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89414d0 0x55dec8d165a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60848/0 (socket says 172.21.15.27:60848) 2024-04-25T13:47:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f19267fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d16ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33604/0 (socket says 172.21.15.27:33604) 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 -- 172.21.15.27:0/2712956572 learned_addr learned my addr 172.21.15.27:0/2712956572 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f19277fe640 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec896a390 0x55dec8d1da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 -- 172.21.15.27:0/2712956572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 msgr2=0x55dec8d16ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d16ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 -- 172.21.15.27:0/2712956572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec896a390 msgr2=0x55dec8d1da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec896a390 0x55dec8d1da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 -- 172.21.15.27:0/2712956572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec89823f0 con 0x55dec89414d0 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1926ffd640 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89414d0 0x55dec8d165a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1918031500 tx=0x7f1918031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ac8f8d095a5f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.983+0000 7f1906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.984+0000 7f1906ffd640 1 -- 172.21.15.27:0/2712956572 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918049050 con 0x55dec89414d0 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.984+0000 7f1906ffd640 1 -- 172.21.15.27:0/2712956572 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1918003c50 con 0x55dec89414d0 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.984+0000 7f193482e740 1 -- 172.21.15.27:0/2712956572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19200132d0 con 0x55dec89414d0 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.984+0000 7f1906ffd640 1 -- 172.21.15.27:0/2712956572 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f19180c70d0 con 0x55dec89414d0 2024-04-25T13:47:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.984+0000 7f193482e740 10 client.16791 fetch_fsmap learned FSMap version 19 2024-04-25T13:47:59.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.984+0000 7f193482e740 1 -- 172.21.15.27:0/2712956572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55dec8a4c7d0 con 0x55dec89414d0 2024-04-25T13:47:59.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.985+0000 7f1906ffd640 1 -- 172.21.15.27:0/2712956572 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19200132d0 con 0x55dec89414d0 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.986+0000 7f1906ffd640 1 -- 172.21.15.27:0/2712956572 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f19180c3a10 con 0x55dec89414d0 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.986+0000 7f193482e740 10 client.16791 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.986+0000 7f193482e740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.986+0000 7f193482e740 20 client.16791 populate_metadata read hostname 'smithi027' 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.986+0000 7f193482e740 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dec8928f60 0x55dec8d24440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.986+0000 7f193482e740 4 client.16791 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.986+0000 7f193482e740 1 -- 172.21.15.27:0/2712956572 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dec8c7e990 con 0x55dec8928f60 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.986+0000 7f19267fc640 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dec8928f60 0x55dec8d24440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.987+0000 7f19267fc640 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dec8928f60 0x55dec8d24440 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4d7b6dee8f26446 server_cookie=2d67ab722cb396d3 in_seq=0 out_seq=0 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.987+0000 7f1906ffd640 10 client.16791 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f1906ffd640 1 -- 172.21.15.27:0/2712956572 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dec8c7e990 con 0x55dec8928f60 2024-04-25T13:47:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f1906ffd640 10 client.16791 handle_command_reply: tid=0 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 client.16791 shutdown 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 -- 172.21.15.27:0/2712956572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89414d0 msgr2=0x55dec8d165a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89414d0 0x55dec8d165a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1918031500 tx=0x7f1918031400 comp rx=0 tx=0).stop 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 -- 172.21.15.27:0/2712956572 shutdown_connections 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dec8928f60 0x55dec8d24440 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec896a390 0x55dec8d1da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f1906ffd640 0 client.16791 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d16ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 --2- 172.21.15.27:0/2712956572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89414d0 0x55dec8d165a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 -- 172.21.15.27:0/2712956572 >> 172.21.15.27:0/2712956572 conn(0x55dec895e4e0 msgr2=0x55dec8d0af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 -- 172.21.15.27:0/2712956572 shutdown_connections 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 1 -- 172.21.15.27:0/2712956572 wait complete. 2024-04-25T13:47:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:47:59.988+0000 7f193482e740 20 client.16791 trim_cache size 0 max 16384 2024-04-25T13:48:00.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.004+0000 7f193482e740 1 Processor -- start 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.005+0000 7f193482e740 1 -- start start 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.005+0000 7f193482e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d08160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f193482e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d1d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f193482e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec896a390 0x55dec8d16170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d08160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d08160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40984/0 (socket says 172.21.15.27:40984) 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19267fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d1d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19267fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d1d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33612/0 (socket says 172.21.15.27:33612) 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19267fc640 1 -- 172.21.15.27:0/3050236156 learned_addr learned my addr 172.21.15.27:0/3050236156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f193482e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19281a91c0 con 0x55dec896a390 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f193482e740 1 -- 172.21.15.27:0/3050236156 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19281aba40 con 0x55dec89414d0 2024-04-25T13:48:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f193482e740 1 -- 172.21.15.27:0/3050236156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f19281a9830 con 0x55dec89496b0 2024-04-25T13:48:00.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19277fe640 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec896a390 0x55dec8d16170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:00.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19267fc640 1 -- 172.21.15.27:0/3050236156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 msgr2=0x55dec8d08160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19267fc640 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d08160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19267fc640 1 -- 172.21.15.27:0/3050236156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec896a390 msgr2=0x55dec8d16170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19267fc640 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec896a390 0x55dec8d16170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19267fc640 1 -- 172.21.15.27:0/3050236156 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec8d07800 con 0x55dec89496b0 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f1926ffd640 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d08160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.006+0000 7f19277fe640 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec896a390 0x55dec8d16170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.007+0000 7f19267fc640 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d1d9d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55dec8d08db0 tx=0x7f191c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e7439e8c7702ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.007+0000 7f190d7fa640 1 -- 172.21.15.27:0/3050236156 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c0200f0 con 0x55dec89496b0 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.007+0000 7f190d7fa640 1 -- 172.21.15.27:0/3050236156 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f191c020740 con 0x55dec89496b0 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.007+0000 7f190d7fa640 1 -- 172.21.15.27:0/3050236156 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c010aa0 con 0x55dec89496b0 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 -- 172.21.15.27:0/3050236156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 msgr2=0x55dec8d1d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d1d9d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55dec8d08db0 tx=0x7f191c005f90 comp rx=0 tx=0).stop 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 -- 172.21.15.27:0/3050236156 shutdown_connections 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec896a390 0x55dec8d16170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec89496b0 0x55dec8d1d9d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 --2- 172.21.15.27:0/3050236156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d08160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 -- 172.21.15.27:0/3050236156 >> 172.21.15.27:0/3050236156 conn(0x55dec895e4e0 msgr2=0x55dec8d07a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 -- 172.21.15.27:0/3050236156 shutdown_connections 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 -- 172.21.15.27:0/3050236156 wait complete. 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.008+0000 7f193482e740 1 -- start start 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f193482e740 1 Processor -- start 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f193482e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d1b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f193482e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 0x55dec8d1b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f193482e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d1c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f193482e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19281a9f00 con 0x55dec89496b0 2024-04-25T13:48:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f193482e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19281ab370 con 0x55dec89414d0 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f193482e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f19281aa5d0 con 0x55dec896a390 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f193482e740 10 client.0 _ll_register_callbacks cb 0x7f1918005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f19267fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 0x55dec8d1b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d1b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f19277fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d1c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d1b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40986/0 (socket says 172.21.15.27:40986) 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 -- 172.21.15.27:0/4062368405 learned_addr learned my addr 172.21.15.27:0/4062368405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 -- 172.21.15.27:0/4062368405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 msgr2=0x55dec8d1c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d1c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 -- 172.21.15.27:0/4062368405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 msgr2=0x55dec8d1b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 0x55dec8d1b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 -- 172.21.15.27:0/4062368405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec8d1c970 con 0x55dec89414d0 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1926ffd640 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d1b000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f19180c7710 tx=0x7f191802a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9936501394af88d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.009+0000 7f1906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f1906ffd640 1 -- 172.21.15.27:0/4062368405 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19180e6060 con 0x55dec89414d0 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 1 -- 172.21.15.27:0/4062368405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f191c020740 con 0x55dec89414d0 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f1906ffd640 1 -- 172.21.15.27:0/4062368405 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f19180035f0 con 0x55dec89414d0 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f1906ffd640 1 -- 172.21.15.27:0/4062368405 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f19180a7050 con 0x55dec89414d0 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 10 client.26513 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 1 -- 172.21.15.27:0/4062368405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55dec8d1dbf0 con 0x55dec89414d0 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f1906ffd640 1 -- 172.21.15.27:0/4062368405 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f191c020740 con 0x55dec89414d0 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f1906ffd640 1 -- 172.21.15.27:0/4062368405 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f19180dcb20 con 0x55dec89414d0 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 10 client.26513 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 10 client.26513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 20 client.26513 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dec88fe3a0 0x55dec8d8ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 4 client.26513 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:48:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.010+0000 7f193482e740 1 -- 172.21.15.27:0/4062368405 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dec8c78800 con 0x55dec88fe3a0 2024-04-25T13:48:00.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.011+0000 7f19267fc640 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dec88fe3a0 0x55dec8d8ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:00.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.011+0000 7f19267fc640 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dec88fe3a0 0x55dec8d8ec70 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af70429d288c5987 server_cookie=359580977e056c8e in_seq=0 out_seq=0 2024-04-25T13:48:00.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.011+0000 7f1906ffd640 10 client.26513 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.012+0000 7f1906ffd640 1 -- 172.21.15.27:0/4062368405 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dec8c78800 con 0x55dec88fe3a0 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.012+0000 7f1906ffd640 10 client.26513 handle_command_reply: tid=0 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.012+0000 7f193482e740 1 client.26513 shutdown 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 -- 172.21.15.27:0/4062368405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 msgr2=0x55dec8d1b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d1b000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f19180c7710 tx=0x7f191802a3f0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 -- 172.21.15.27:0/4062368405 shutdown_connections 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dec88fe3a0 0x55dec8d8ec70 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dec896a390 0x55dec8d1c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dec89496b0 0x55dec8d1b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 --2- 172.21.15.27:0/4062368405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dec89414d0 0x55dec8d1b000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 -- 172.21.15.27:0/4062368405 >> 172.21.15.27:0/4062368405 conn(0x55dec895e4e0 msgr2=0x55dec8d16170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f1906ffd640 0 client.26513 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 -- 172.21.15.27:0/4062368405 shutdown_connections 2024-04-25T13:48:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.013+0000 7f193482e740 1 -- 172.21.15.27:0/4062368405 wait complete. 2024-04-25T13:48:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f193482e740 20 client.26513 trim_cache size 0 max 16384 2024-04-25T13:48:00.017 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 -- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f19100afaf0 msgr2=0x7f19100b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f19100afaf0 0x7f19100b2000 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1920021000 tx=0x7f192001e000 comp rx=0 tx=0).stop 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 -- 172.21.15.27:0/1802303448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 msgr2=0x7f192819a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 0x7f192819a2d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1918006010 tx=0x7f1918004270 comp rx=0 tx=0).stop 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 -- 172.21.15.27:0/1802303448 shutdown_connections 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f19100afaf0 0x7f19100b2000 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1928109780 0x7f19281a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1928106c70 0x7f192819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 --2- 172.21.15.27:0/1802303448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1928104280 0x7f192819a2d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 -- 172.21.15.27:0/1802303448 >> 172.21.15.27:0/1802303448 conn(0x7f19280fdbe0 msgr2=0x7f1928104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 -- 172.21.15.27:0/1802303448 shutdown_connections 2024-04-25T13:48:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:00.014+0000 7f192d97d640 1 -- 172.21.15.27:0/1802303448 wait complete. 2024-04-25T13:48:00.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:00 smithi122 ceph-mon[32843]: pgmap v562: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 586 op/s 2024-04-25T13:48:00.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:00 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2454603278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:48:00.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:00 smithi045 ceph-mon[32731]: pgmap v562: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 586 op/s 2024-04-25T13:48:00.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:00 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2454603278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:48:00.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:00 smithi027 ceph-mon[29140]: pgmap v562: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 586 op/s 2024-04-25T13:48:00.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:00 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2454603278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:48:01.036 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:48:01.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:48:01.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:48:01.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 -- 172.21.15.27:0/2590083468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc0108a10 msgr2=0x7f2cc010ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2590083468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc0108a10 0x7f2cc010ae30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009ca0 tx=0x7f2cb802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 -- 172.21.15.27:0/2590083468 shutdown_connections 2024-04-25T13:48:01.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2590083468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc0108a10 0x7f2cc010ae30 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2590083468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc01043c0 0x7f2cc0102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2590083468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc0103ee0 0x7f2cc01027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 -- 172.21.15.27:0/2590083468 >> 172.21.15.27:0/2590083468 conn(0x7f2cc00fd340 msgr2=0x7f2cc00ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 -- 172.21.15.27:0/2590083468 shutdown_connections 2024-04-25T13:48:01.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 -- 172.21.15.27:0/2590083468 wait complete. 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.201+0000 7f2cc8492640 1 Processor -- start 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc8492640 1 -- start start 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc8492640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc0103ee0 0x7f2cc0199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc8492640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc0108a10 0x7f2cc019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc8492640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc019f260 0x7f2cc01a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc8492640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc010e570 con 0x7f2cc019f260 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc8492640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc010e3f0 con 0x7f2cc0103ee0 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc8492640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc010e6f0 con 0x7f2cc0108a10 2024-04-25T13:48:01.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc019f260 0x7f2cc01a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc019f260 0x7f2cc01a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60894/0 (socket says 172.21.15.27:60894) 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc622e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc0103ee0 0x7f2cc0199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 -- 172.21.15.27:0/2326483005 learned_addr learned my addr 172.21.15.27:0/2326483005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc5a2d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc0108a10 0x7f2cc019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 -- 172.21.15.27:0/2326483005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc0108a10 msgr2=0x7f2cc019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc0108a10 0x7f2cc019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 -- 172.21.15.27:0/2326483005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc0103ee0 msgr2=0x7f2cc0199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc0103ee0 0x7f2cc0199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 -- 172.21.15.27:0/2326483005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc01a1cf0 con 0x7f2cc019f260 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc622e640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc0103ee0 0x7f2cc0199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc0108a10 0x7f2cc019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.202+0000 7f2cc6a2f640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc019f260 0x7f2cc01a1680 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009430 tx=0x7f2cb80316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.203+0000 7f2cb77fe640 1 -- 172.21.15.27:0/2326483005 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8003b70 con 0x7f2cc019f260 2024-04-25T13:48:01.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.203+0000 7f2cb77fe640 1 -- 172.21.15.27:0/2326483005 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb8003d10 con 0x7f2cc019f260 2024-04-25T13:48:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.203+0000 7f2cc8492640 1 -- 172.21.15.27:0/2326483005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc01a1f80 con 0x7f2cc019f260 2024-04-25T13:48:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.203+0000 7f2cb77fe640 1 -- 172.21.15.27:0/2326483005 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb802fb40 con 0x7f2cc019f260 2024-04-25T13:48:01.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.203+0000 7f2cc8492640 1 -- 172.21.15.27:0/2326483005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc01a23e0 con 0x7f2cc019f260 2024-04-25T13:48:01.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2ccf411740 1 Processor -- start 2024-04-25T13:48:01.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2cb77fe640 1 -- 172.21.15.27:0/2326483005 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2cb8004610 con 0x7f2cc019f260 2024-04-25T13:48:01.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2ccf411740 1 -- start start 2024-04-25T13:48:01.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1fd410 0x55903f5a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1dc730 0x55903f5a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2ccf411740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55903f503b00 con 0x55903f1fd410 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2ccf411740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55903f49f920 con 0x55903f1d4550 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.206+0000 7f2ccf411740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55903f4ffac0 con 0x55903f1dc730 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1fd410 0x55903f5a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1fd410 0x55903f5a28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60910/0 (socket says 172.21.15.27:60910) 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/1048676306 learned_addr learned my addr 172.21.15.27:0/1048676306 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1dc730 0x55903f5a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc622e640 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/1048676306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1dc730 msgr2=0x55903f5a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1dc730 0x55903f5a9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/1048676306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 msgr2=0x55903f5a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/1048676306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55903f2dd7e0 con 0x55903f1fd410 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc622e640 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cb77fe640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2cac0afb70 0x7f2cac0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc622e640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2cac0afb70 0x7f2cac0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1fd410 0x55903f5a28b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0013c20 tx=0x7f2cb0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42010a9d1cbcb2af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.207+0000 7f2cb77fe640 1 -- 172.21.15.27:0/2326483005 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2cb8100070 con 0x7f2cc019f260 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.208+0000 7f2cb4bfe640 1 -- 172.21.15.27:0/1048676306 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0013dd0 con 0x55903f1fd410 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.208+0000 7f2cc622e640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2cac0afb70 0x7f2cac0b2080 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55903f5a3c90 tx=0x7f2cbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.208+0000 7f2cb4bfe640 1 -- 172.21.15.27:0/1048676306 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb0013120 con 0x55903f1fd410 2024-04-25T13:48:01.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.208+0000 7f2cb4bfe640 1 -- 172.21.15.27:0/1048676306 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0003850 con 0x55903f1fd410 2024-04-25T13:48:01.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 -- 172.21.15.27:0/1048676306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1fd410 msgr2=0x55903f5a28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1fd410 0x55903f5a28b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0013c20 tx=0x7f2cb0009b30 comp rx=0 tx=0).stop 2024-04-25T13:48:01.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 -- 172.21.15.27:0/1048676306 shutdown_connections 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1dc730 0x55903f5a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1048676306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1fd410 0x55903f5a28b0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 -- 172.21.15.27:0/1048676306 >> 172.21.15.27:0/1048676306 conn(0x55903f1f1560 msgr2=0x55903f599e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 -- 172.21.15.27:0/1048676306 shutdown_connections 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 -- 172.21.15.27:0/1048676306 wait complete. 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 -- start start 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.209+0000 7f2ccf411740 1 Processor -- start 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1d4550 0x55903f5ac7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1dc730 0x55903f5acce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1fd410 0x55903f5b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ccf411740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55903f5aa720 con 0x55903f1d4550 2024-04-25T13:48:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ccf411740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55903f49d5d0 con 0x55903f1dc730 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ccf411740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55903f5aa8a0 con 0x55903f1fd410 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ccf411740 10 client.0 _ll_register_callbacks cb 0x55903f48b080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1d4550 0x55903f5ac7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1d4550 0x55903f5ac7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60922/0 (socket says 172.21.15.27:60922) 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 -- 172.21.15.27:0/1345997667 learned_addr learned my addr 172.21.15.27:0/1345997667 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc6a2f640 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1fd410 0x55903f5b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1dc730 0x55903f5acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 -- 172.21.15.27:0/1345997667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1fd410 msgr2=0x55903f5b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1fd410 0x55903f5b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 -- 172.21.15.27:0/1345997667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1dc730 msgr2=0x55903f5acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1dc730 0x55903f5acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 -- 172.21.15.27:0/1345997667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55903f33df70 con 0x55903f1d4550 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc6a2f640 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1fd410 0x55903f5b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2cc622e640 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1d4550 0x55903f5ac7a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc004600 tx=0x7f2cbc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baeaf5fd43cea0b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/1345997667 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc00f830 con 0x55903f1d4550 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/1345997667 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2cbc00f9d0 con 0x55903f1d4550 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.210+0000 7f2ccf411740 1 -- 172.21.15.27:0/1345997667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cb0013120 con 0x55903f1d4550 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.211+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/1345997667 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2cbc00fbd0 con 0x55903f1d4550 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.211+0000 7f2ccf411740 10 client.16809 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.211+0000 7f2ccf411740 1 -- 172.21.15.27:0/1345997667 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55903f1d5170 con 0x55903f1d4550 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.211+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/1345997667 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb0013120 con 0x55903f1d4550 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.211+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/1345997667 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2cbc027070 con 0x55903f1d4550 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.212+0000 7f2ccf411740 10 client.16809 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.212+0000 7f2ccf411740 10 client.16809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.212+0000 7f2ccf411740 20 client.16809 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.212+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55903f1bc1f0 0x55903f5ba200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.212+0000 7f2ccf411740 4 client.16809 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.212+0000 7f2ccf411740 1 -- 172.21.15.27:0/1345997667 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55903f511710 con 0x55903f1bc1f0 2024-04-25T13:48:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.212+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55903f1bc1f0 0x55903f5ba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.213+0000 7f2ca9ffb640 10 client.16809 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.213+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55903f1bc1f0 0x55903f5ba200 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9308393bb5aca1b5 server_cookie=a3a0fbcbad64a05b in_seq=0 out_seq=0 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.213+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/1345997667 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55903f511710 con 0x55903f1bc1f0 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.213+0000 7f2ca9ffb640 10 client.16809 handle_command_reply: tid=0 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.213+0000 7f2ccf411740 1 client.16809 shutdown 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.213+0000 7f2ccf411740 1 -- 172.21.15.27:0/1345997667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1d4550 msgr2=0x55903f5ac7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.213+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1d4550 0x55903f5ac7a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc004600 tx=0x7f2cbc004730 comp rx=0 tx=0).stop 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 1 -- 172.21.15.27:0/1345997667 shutdown_connections 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55903f1bc1f0 0x55903f5ba200 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1fd410 0x55903f5b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1dc730 0x55903f5acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ca9ffb640 0 client.16809 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 1 --2- 172.21.15.27:0/1345997667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1d4550 0x55903f5ac7a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 1 -- 172.21.15.27:0/1345997667 >> 172.21.15.27:0/1345997667 conn(0x55903f1f1560 msgr2=0x55903f598ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 1 -- 172.21.15.27:0/1345997667 shutdown_connections 2024-04-25T13:48:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 1 -- 172.21.15.27:0/1345997667 wait complete. 2024-04-25T13:48:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.214+0000 7f2ccf411740 20 client.16809 trim_cache size 0 max 16384 2024-04-25T13:48:01.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2ccf411740 1 Processor -- start 2024-04-25T13:48:01.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2ccf411740 1 -- start start 2024-04-25T13:48:01.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1f1560 0x55903f5ab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1bc1f0 0x55903f5abd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2ccf411740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01a6f50 con 0x55903f1bc1f0 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2ccf411740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01a97d0 con 0x55903f1d4550 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2ccf411740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01a75c0 con 0x55903f1f1560 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.236+0000 7f2cc5a2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1bc1f0 0x55903f5abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc5a2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1bc1f0 0x55903f5abd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60934/0 (socket says 172.21.15.27:60934) 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/2315089711 learned_addr learned my addr 172.21.15.27:0/2315089711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc622e640 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1f1560 0x55903f5ab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc6a2f640 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/2315089711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1f1560 msgr2=0x55903f5ab7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1f1560 0x55903f5ab7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/2315089711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 msgr2=0x55903f5bfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5bfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/2315089711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55903f5a5cd0 con 0x55903f1bc1f0 2024-04-25T13:48:01.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc622e640 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1f1560 0x55903f5ab7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:01.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1bc1f0 0x55903f5abd30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000fc60 tx=0x7f2cb0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8db78cdde45b411 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cabfff640 1 -- 172.21.15.27:0/2315089711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0005330 con 0x55903f1bc1f0 2024-04-25T13:48:01.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cabfff640 1 -- 172.21.15.27:0/2315089711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb002d070 con 0x55903f1bc1f0 2024-04-25T13:48:01.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.237+0000 7f2cabfff640 1 -- 172.21.15.27:0/2315089711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0005730 con 0x55903f1bc1f0 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 -- 172.21.15.27:0/2315089711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1bc1f0 msgr2=0x55903f5abd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1bc1f0 0x55903f5abd30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000fc60 tx=0x7f2cb0005f90 comp rx=0 tx=0).stop 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 -- 172.21.15.27:0/2315089711 shutdown_connections 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1d4550 0x55903f5bfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1bc1f0 0x55903f5abd30 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 --2- 172.21.15.27:0/2315089711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1f1560 0x55903f5ab7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 -- 172.21.15.27:0/2315089711 >> 172.21.15.27:0/2315089711 conn(0x55903f5aceb0 msgr2=0x55903f59d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 -- 172.21.15.27:0/2315089711 shutdown_connections 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.238+0000 7f2ccf411740 1 -- 172.21.15.27:0/2315089711 wait complete. 2024-04-25T13:48:01.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.239+0000 7f2ccf411740 1 -- start start 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.239+0000 7f2ccf411740 1 Processor -- start 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.239+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1bc1f0 0x55903f5b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1d4550 0x55903f5b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2ccf411740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1f1560 0x55903f6205a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2ccf411740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01a7c90 con 0x55903f1f1560 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2ccf411740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01a9100 con 0x55903f1bc1f0 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2ccf411740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc01a8360 con 0x55903f1d4550 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2ccf411740 10 client.0 _ll_register_callbacks cb 0x7f2cbc0043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc6a2f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1f1560 0x55903f6205a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc6a2f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1f1560 0x55903f6205a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60940/0 (socket says 172.21.15.27:60940) 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc6a2f640 1 -- 172.21.15.27:0/3080582770 learned_addr learned my addr 172.21.15.27:0/3080582770 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1d4550 0x55903f5b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc622e640 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1bc1f0 0x55903f5b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/3080582770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1bc1f0 msgr2=0x55903f5b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1bc1f0 0x55903f5b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/3080582770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1f1560 msgr2=0x55903f6205a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1f1560 0x55903f6205a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc5a2d640 1 -- 172.21.15.27:0/3080582770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55903f59e070 con 0x55903f1d4550 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.240+0000 7f2cc6a2f640 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1f1560 0x55903f6205a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2cc5a2d640 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1d4550 0x55903f5b3930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0000c00 tx=0x7f2cb00137f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=401b29daee463103 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2ca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/3080582770 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0038060 con 0x55903f1d4550 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/3080582770 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2cb00285a0 con 0x55903f1d4550 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2ccf411740 1 -- 172.21.15.27:0/3080582770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cb002d070 con 0x55903f1d4550 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/3080582770 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2cb0005130 con 0x55903f1d4550 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2ccf411740 10 client.26569 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2ccf411740 1 -- 172.21.15.27:0/3080582770 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55903f59f330 con 0x55903f1d4550 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.241+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/3080582770 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cb002d070 con 0x55903f1d4550 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/3080582770 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2cb0003bc0 con 0x55903f1d4550 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2ccf411740 10 client.26569 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:01.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2ccf411740 10 client.26569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2ccf411740 20 client.26569 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2ccf411740 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55903f1dc730 0x55903f5a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2ccf411740 4 client.26569 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:48:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2ccf411740 1 -- 172.21.15.27:0/3080582770 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55903f50b580 con 0x55903f1dc730 2024-04-25T13:48:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2cc622e640 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55903f1dc730 0x55903f5a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2ca9ffb640 10 client.26569 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.242+0000 7f2cc622e640 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55903f1dc730 0x55903f5a1a20 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=120e2540f5275bf6 server_cookie=699ce6dad17e3f3c in_seq=0 out_seq=0 2024-04-25T13:48:01.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.244+0000 7f2ca9ffb640 1 -- 172.21.15.27:0/3080582770 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55903f50b580 con 0x55903f1dc730 2024-04-25T13:48:01.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.244+0000 7f2ca9ffb640 10 client.26569 handle_command_reply: tid=0 2024-04-25T13:48:01.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.244+0000 7f2ccf411740 1 client.26569 shutdown 2024-04-25T13:48:01.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 -- 172.21.15.27:0/3080582770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1d4550 msgr2=0x55903f5b3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1d4550 0x55903f5b3930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0000c00 tx=0x7f2cb00137f0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 -- 172.21.15.27:0/3080582770 shutdown_connections 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55903f1dc730 0x55903f5a1a20 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55903f1f1560 0x55903f6205a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55903f1d4550 0x55903f5b3930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 --2- 172.21.15.27:0/3080582770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55903f1bc1f0 0x55903f5b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 -- 172.21.15.27:0/3080582770 >> 172.21.15.27:0/3080582770 conn(0x55903f5aceb0 msgr2=0x55903f59d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ca9ffb640 0 client.26569 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 -- 172.21.15.27:0/3080582770 shutdown_connections 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 1 -- 172.21.15.27:0/3080582770 wait complete. 2024-04-25T13:48:01.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.245+0000 7f2ccf411740 20 client.26569 trim_cache size 0 max 16384 2024-04-25T13:48:01.248 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:48:01.248 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:48:01.248 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "5fa84817-93c3-46bf-ae38-9dcfc3764b5d", 2024-04-25T13:48:01.248 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:48:01.248 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:48:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 -- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2cac0afb70 msgr2=0x7f2cac0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2cac0afb70 0x7f2cac0b2080 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55903f5a3c90 tx=0x7f2cbc005f90 comp rx=0 tx=0).stop 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 -- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc019f260 msgr2=0x7f2cc01a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc019f260 0x7f2cc01a1680 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009430 tx=0x7f2cb80316f0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 -- 172.21.15.27:0/2326483005 shutdown_connections 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2cac0afb70 0x7f2cac0b2080 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc019f260 0x7f2cc01a1680 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc0108a10 0x7f2cc019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 --2- 172.21.15.27:0/2326483005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc0103ee0 0x7f2cc0199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 -- 172.21.15.27:0/2326483005 >> 172.21.15.27:0/2326483005 conn(0x7f2cc00fd340 msgr2=0x7f2cc0101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 -- 172.21.15.27:0/2326483005 shutdown_connections 2024-04-25T13:48:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.246+0000 7f2cc8492640 1 -- 172.21.15.27:0/2326483005 wait complete. 2024-04-25T13:48:01.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:48:01.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:48:01.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.428+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3020057165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c103c20 msgr2=0x7f2e7c104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.428+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3020057165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c103c20 0x7f2e7c104030 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009900 tx=0x7f2e6802a410 comp rx=0 tx=0).stop 2024-04-25T13:48:01.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.430+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3020057165 shutdown_connections 2024-04-25T13:48:01.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.430+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3020057165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2e7c108920 0x7f2e7c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.430+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3020057165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c103c20 0x7f2e7c104030 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.430+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3020057165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7c1029e0 0x7f2e7c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.430+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3020057165 >> 172.21.15.27:0/3020057165 conn(0x7f2e7c0fdbd0 msgr2=0x7f2e7c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.430+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3020057165 shutdown_connections 2024-04-25T13:48:01.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.430+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3020057165 wait complete. 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.431+0000 7f2e82f7a640 1 Processor -- start 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.431+0000 7f2e82f7a640 1 -- start start 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.431+0000 7f2e82f7a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c1029e0 0x7f2e7c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.431+0000 7f2e82f7a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7c103c20 0x7f2e7c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.431+0000 7f2e82f7a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2e7c108920 0x7f2e7c1a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.431+0000 7f2e82f7a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10e530 con 0x7f2e7c1029e0 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.431+0000 7f2e82f7a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10e3b0 con 0x7f2e7c103c20 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.431+0000 7f2e82f7a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10e6b0 con 0x7f2e7c108920 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e80d16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c1029e0 0x7f2e7c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7c103c20 0x7f2e7c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7c103c20 0x7f2e7c19cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:41068/0 (socket says 172.21.15.27:41068) 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 -- 172.21.15.27:0/3505486070 learned_addr learned my addr 172.21.15.27:0/3505486070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e80d16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c1029e0 0x7f2e7c19c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60952/0 (socket says 172.21.15.27:60952) 2024-04-25T13:48:01.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e81517640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2e7c108920 0x7f2e7c1a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 -- 172.21.15.27:0/3505486070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2e7c108920 msgr2=0x7f2e7c1a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2e7c108920 0x7f2e7c1a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 -- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c1029e0 msgr2=0x7f2e7c19c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c1029e0 0x7f2e7c19c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 -- 172.21.15.27:0/3505486070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7c1a42a0 con 0x7f2e7c103c20 2024-04-25T13:48:01.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7c103c20 0x7f2e7c19cca0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2e68031440 tx=0x7f2e68003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.432+0000 7f2e79ffb640 1 -- 172.21.15.27:0/3505486070 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6803f070 con 0x7f2e7c103c20 2024-04-25T13:48:01.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.433+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3505486070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7c1a4530 con 0x7f2e7c103c20 2024-04-25T13:48:01.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.433+0000 7f2e79ffb640 1 -- 172.21.15.27:0/3505486070 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e68031c50 con 0x7f2e7c103c20 2024-04-25T13:48:01.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.433+0000 7f2e79ffb640 1 -- 172.21.15.27:0/3505486070 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68035030 con 0x7f2e7c103c20 2024-04-25T13:48:01.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.433+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3505486070 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7c1a4a70 con 0x7f2e7c103c20 2024-04-25T13:48:01.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.436+0000 7f2e79ffb640 1 -- 172.21.15.27:0/3505486070 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2e68042070 con 0x7f2e7c103c20 2024-04-25T13:48:01.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.436+0000 7f2e89e22740 1 Processor -- start 2024-04-25T13:48:01.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.436+0000 7f2e89e22740 1 -- start start 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343aac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e89e22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e3430ed80 con 0x564e33fdf4d0 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e89e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564e342c88b0 con 0x564e33fe76b0 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e89e22740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564e3430ad40 con 0x564e34008390 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e81517640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e81517640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343a9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33698/0 (socket says 172.21.15.27:33698) 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e81517640 1 -- 172.21.15.27:0/595964239 learned_addr learned my addr 172.21.15.27:0/595964239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e80d16640 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e79ffb640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2e6c0afaf0 0x7f2e6c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e7bfff640 1 -- 172.21.15.27:0/595964239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 msgr2=0x564e343a9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343a9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e7bfff640 1 -- 172.21.15.27:0/595964239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 msgr2=0x564e343a9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e7bfff640 1 -- 172.21.15.27:0/595964239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e34299a40 con 0x564e33fe76b0 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e81517640 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343a9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e81517640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2e6c0afaf0 0x7f2e6c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.437+0000 7f2e79ffb640 1 -- 172.21.15.27:0/3505486070 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2e680ff2a0 con 0x7f2e7c103c20 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.438+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343aac70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2e680fce20 tx=0x7f2e680c3720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52461539a553758 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.438+0000 7f2e66ffd640 1 -- 172.21.15.27:0/595964239 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68049020 con 0x564e33fe76b0 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.438+0000 7f2e66ffd640 1 -- 172.21.15.27:0/595964239 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e68031e00 con 0x564e33fe76b0 2024-04-25T13:48:01.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.438+0000 7f2e66ffd640 1 -- 172.21.15.27:0/595964239 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e680c7630 con 0x564e33fe76b0 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.439+0000 7f2e81517640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2e6c0afaf0 0x7f2e6c0b2000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x564e343aaab0 tx=0x7f2e74004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.439+0000 7f2e89e22740 1 -- 172.21.15.27:0/595964239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 msgr2=0x564e343aac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.439+0000 7f2e89e22740 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343aac70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2e680fce20 tx=0x7f2e680c3720 comp rx=0 tx=0).stop 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.439+0000 7f2e89e22740 1 -- 172.21.15.27:0/595964239 shutdown_connections 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.439+0000 7f2e89e22740 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343aac70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.439+0000 7f2e89e22740 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.439+0000 7f2e89e22740 1 --2- 172.21.15.27:0/595964239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343a9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.439+0000 7f2e89e22740 1 -- 172.21.15.27:0/595964239 >> 172.21.15.27:0/595964239 conn(0x564e33ffc4e0 msgr2=0x564e343a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.440+0000 7f2e89e22740 1 -- 172.21.15.27:0/595964239 shutdown_connections 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.440+0000 7f2e89e22740 1 -- 172.21.15.27:0/595964239 wait complete. 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.440+0000 7f2e89e22740 1 -- start start 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.440+0000 7f2e89e22740 1 Processor -- start 2024-04-25T13:48:01.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fdf4d0 0x564e343b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e343b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343baa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e89e22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564e343ab540 con 0x564e33fe76b0 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e89e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564e34295d10 con 0x564e33fdf4d0 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e89e22740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564e343ab6c0 con 0x564e34008390 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e89e22740 10 client.0 _ll_register_callbacks cb 0x564e342c2210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e343b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e80d16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fdf4d0 0x564e343b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e81517640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343baa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e343b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60964/0 (socket says 172.21.15.27:60964) 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e7bfff640 1 -- 172.21.15.27:0/2679712296 learned_addr learned my addr 172.21.15.27:0/2679712296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e80d16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fdf4d0 0x564e343b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:41094/0 (socket says 172.21.15.27:41094) 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e80d16640 1 -- 172.21.15.27:0/2679712296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 msgr2=0x564e343baa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e80d16640 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343baa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e80d16640 1 -- 172.21.15.27:0/2679712296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 msgr2=0x564e343b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e80d16640 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e343b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.441+0000 7f2e80d16640 1 -- 172.21.15.27:0/2679712296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e340ea7d0 con 0x564e33fdf4d0 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e80d16640 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fdf4d0 0x564e343b3640 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x564e343a9780 tx=0x7f2e70012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daf75e0360a0689b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e5bfff640 1 -- 172.21.15.27:0/2679712296 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70004050 con 0x564e33fdf4d0 2024-04-25T13:48:01.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e5bfff640 1 -- 172.21.15.27:0/2679712296 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2e700041f0 con 0x564e33fdf4d0 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e89e22740 1 -- 172.21.15.27:0/2679712296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564e3431c990 con 0x564e33fdf4d0 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e5bfff640 1 -- 172.21.15.27:0/2679712296 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2e700043f0 con 0x564e33fdf4d0 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e89e22740 10 client.26543 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e89e22740 1 -- 172.21.15.27:0/2679712296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564e340203f0 con 0x564e33fdf4d0 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e5bfff640 1 -- 172.21.15.27:0/2679712296 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564e3431c990 con 0x564e33fdf4d0 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e5bfff640 1 -- 172.21.15.27:0/2679712296 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2e70003ab0 con 0x564e33fdf4d0 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e89e22740 10 client.26543 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.442+0000 7f2e89e22740 10 client.26543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.443+0000 7f2e89e22740 20 client.26543 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.443+0000 7f2e89e22740 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564e33fc6f60 0x564e343c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.443+0000 7f2e89e22740 4 client.26543 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.443+0000 7f2e89e22740 1 -- 172.21.15.27:0/2679712296 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e34316800 con 0x564e33fc6f60 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.443+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564e33fc6f60 0x564e343c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.443+0000 7f2e5bfff640 10 client.26543 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.443+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564e33fc6f60 0x564e343c0fd0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e08eee130d2842f8 server_cookie=68e4676832f1cfe6 in_seq=0 out_seq=0 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e5bfff640 1 -- 172.21.15.27:0/2679712296 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564e34316800 con 0x564e33fc6f60 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e5bfff640 10 client.26543 handle_command_reply: tid=0 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 client.26543 shutdown 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 -- 172.21.15.27:0/2679712296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fdf4d0 msgr2=0x564e343b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fdf4d0 0x564e343b3640 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x564e343a9780 tx=0x7f2e70012b60 comp rx=0 tx=0).stop 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 -- 172.21.15.27:0/2679712296 shutdown_connections 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564e33fc6f60 0x564e343c0fd0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e34008390 0x564e343baa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e5bfff640 0 client.26543 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e343b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 --2- 172.21.15.27:0/2679712296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fdf4d0 0x564e343b3640 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 -- 172.21.15.27:0/2679712296 >> 172.21.15.27:0/2679712296 conn(0x564e33ffc4e0 msgr2=0x564e343a8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 -- 172.21.15.27:0/2679712296 shutdown_connections 2024-04-25T13:48:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 1 -- 172.21.15.27:0/2679712296 wait complete. 2024-04-25T13:48:01.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.444+0000 7f2e89e22740 20 client.26543 trim_cache size 0 max 16384 2024-04-25T13:48:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.466+0000 7f2e89e22740 1 Processor -- start 2024-04-25T13:48:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.466+0000 7f2e89e22740 1 -- start start 2024-04-25T13:48:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.466+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fc6f60 0x564e343b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.466+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.466+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343ba990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.466+0000 7f2e89e22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c1ab410 con 0x564e33fdf4d0 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.466+0000 7f2e89e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c1adc90 con 0x564e33fe76b0 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.466+0000 7f2e89e22740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c1aba80 con 0x564e33fc6f60 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e80d16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fc6f60 0x564e343b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60980/0 (socket says 172.21.15.27:60980) 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e7bfff640 1 -- 172.21.15.27:0/4012083864 learned_addr learned my addr 172.21.15.27:0/4012083864 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e81517640 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343ba990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e80d16640 1 -- 172.21.15.27:0/4012083864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 msgr2=0x564e343ba990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e80d16640 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343ba990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e80d16640 1 -- 172.21.15.27:0/4012083864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 msgr2=0x564e343a8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e80d16640 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e80d16640 1 -- 172.21.15.27:0/4012083864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e33fe00f0 con 0x564e33fc6f60 2024-04-25T13:48:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e80d16640 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fc6f60 0x564e343b22d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2e70003f20 tx=0x7f2e70010760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe1bc3d1a953ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.467+0000 7f2e667fc640 1 -- 172.21.15.27:0/4012083864 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70004000 con 0x564e33fc6f60 2024-04-25T13:48:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e667fc640 1 -- 172.21.15.27:0/4012083864 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e700041a0 con 0x564e33fc6f60 2024-04-25T13:48:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e667fc640 1 -- 172.21.15.27:0/4012083864 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7000f790 con 0x564e33fc6f60 2024-04-25T13:48:01.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 -- 172.21.15.27:0/4012083864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fc6f60 msgr2=0x564e343b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fc6f60 0x564e343b22d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2e70003f20 tx=0x7f2e70010760 comp rx=0 tx=0).stop 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 -- 172.21.15.27:0/4012083864 shutdown_connections 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fe76b0 0x564e343ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fdf4d0 0x564e343a8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 --2- 172.21.15.27:0/4012083864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fc6f60 0x564e343b22d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 -- 172.21.15.27:0/4012083864 >> 172.21.15.27:0/4012083864 conn(0x564e33ffc4e0 msgr2=0x564e343a8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 -- 172.21.15.27:0/4012083864 shutdown_connections 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.468+0000 7f2e89e22740 1 -- 172.21.15.27:0/4012083864 wait complete. 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.469+0000 7f2e89e22740 1 -- start start 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.469+0000 7f2e89e22740 1 Processor -- start 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.469+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fc6f60 0x564e343b8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.469+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fdf4d0 0x564e343b8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.469+0000 7f2e89e22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e34427330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e80d16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fc6f60 0x564e343b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fdf4d0 0x564e343b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fdf4d0 0x564e343b8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33716/0 (socket says 172.21.15.27:33716) 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 -- 172.21.15.27:0/852838869 learned_addr learned my addr 172.21.15.27:0/852838869 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c1ac150 con 0x564e33fe76b0 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c1ad5c0 con 0x564e33fc6f60 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c1ac820 con 0x564e33fdf4d0 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e81517640 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e34427330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e89e22740 10 client.0 _ll_register_callbacks cb 0x7f2e70009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 -- 172.21.15.27:0/852838869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fc6f60 msgr2=0x564e343b8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fc6f60 0x564e343b8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 -- 172.21.15.27:0/852838869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 msgr2=0x564e34427330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e34427330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 -- 172.21.15.27:0/852838869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e342cf4b0 con 0x564e33fdf4d0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e7bfff640 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fdf4d0 0x564e343b8850 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2e6802a9d0 tx=0x7f2e6802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ef545f615b3ce93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.470+0000 7f2e5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e5bfff640 1 -- 172.21.15.27:0/852838869 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e680e0070 con 0x564e33fdf4d0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e5bfff640 1 -- 172.21.15.27:0/852838869 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2e680c3af0 con 0x564e33fdf4d0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e700041a0 con 0x564e33fdf4d0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e5bfff640 1 -- 172.21.15.27:0/852838869 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2e680d5050 con 0x564e33fdf4d0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 10 client.26587 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564e34428970 con 0x564e33fdf4d0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e5bfff640 1 -- 172.21.15.27:0/852838869 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e700041a0 con 0x564e33fdf4d0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e5bfff640 1 -- 172.21.15.27:0/852838869 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2e680c31d0 con 0x564e33fdf4d0 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 10 client.26587 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 10 client.26587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:01.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 20 client.26587 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564e34008390 0x564e3442b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 4 client.26587 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:48:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.471+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564e342914a0 con 0x564e34008390 2024-04-25T13:48:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.472+0000 7f2e80d16640 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564e34008390 0x564e3442b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.472+0000 7f2e5bfff640 10 client.26587 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.472+0000 7f2e80d16640 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564e34008390 0x564e3442b000 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87ee96f6623a1dbb server_cookie=cb7a8901058a7e7f in_seq=0 out_seq=0 2024-04-25T13:48:01.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.472+0000 7f2e5bfff640 1 -- 172.21.15.27:0/852838869 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564e342914a0 con 0x564e34008390 2024-04-25T13:48:01.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.472+0000 7f2e5bfff640 10 client.26587 handle_command_reply: tid=0 2024-04-25T13:48:01.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.472+0000 7f2e89e22740 1 client.26587 shutdown 2024-04-25T13:48:01.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fdf4d0 msgr2=0x564e343b8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fdf4d0 0x564e343b8850 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2e6802a9d0 tx=0x7f2e6802a410 comp rx=0 tx=0).stop 2024-04-25T13:48:01.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 shutdown_connections 2024-04-25T13:48:01.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564e34008390 0x564e3442b000 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564e33fe76b0 0x564e34427330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e33fdf4d0 0x564e343b8850 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 --2- 172.21.15.27:0/852838869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564e33fc6f60 0x564e343b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e5bfff640 0 client.26587 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 >> 172.21.15.27:0/852838869 conn(0x564e33ffc4e0 msgr2=0x564e343a39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 shutdown_connections 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 1 -- 172.21.15.27:0/852838869 wait complete. 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e89e22740 20 client.26587 trim_cache size 0 max 16384 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:48:01.477 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:48:01.478 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:48:01.478 INFO:teuthology.orchestra.run.smithi027.stdout: "5fa84817-93c3-46bf-ae38-9dcfc3764b5d": { 2024-04-25T13:48:01.478 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:48:01.478 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "5fa84817-93c3-46bf-ae38-9dcfc3764b5d", 2024-04-25T13:48:01.478 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:48:01.478 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:48:01.478 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:48:01.478 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2e6c0afaf0 msgr2=0x7f2e6c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2e6c0afaf0 0x7f2e6c0b2000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x564e343aaab0 tx=0x7f2e74004a40 comp rx=0 tx=0).stop 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3505486070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7c103c20 msgr2=0x7f2e7c19cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7c103c20 0x7f2e7c19cca0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2e68031440 tx=0x7f2e68003c10 comp rx=0 tx=0).stop 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3505486070 shutdown_connections 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2e6c0afaf0 0x7f2e6c0b2000 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2e7c108920 0x7f2e7c1a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e7c103c20 0x7f2e7c19cca0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 --2- 172.21.15.27:0/3505486070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e7c1029e0 0x7f2e7c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.473+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3505486070 >> 172.21.15.27:0/3505486070 conn(0x7f2e7c0fdbd0 msgr2=0x7f2e7c0ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.474+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3505486070 shutdown_connections 2024-04-25T13:48:01.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:01.474+0000 7f2e82f7a640 1 -- 172.21.15.27:0/3505486070 wait complete. 2024-04-25T13:48:01.487 INFO:tasks.cephfs.filesystem:scrub status for tag:5fa84817-93c3-46bf-ae38-9dcfc3764b5d - {'path': '/', 'tag': '5fa84817-93c3-46bf-ae38-9dcfc3764b5d', 'options': 'recursive,force'} 2024-04-25T13:48:02.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:02 smithi122 ceph-mon[32843]: pgmap v563: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.3 MiB/s wr, 716 op/s 2024-04-25T13:48:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:02 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:48:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:02 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:02 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:48:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:02 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:02 smithi045 ceph-mon[32731]: pgmap v563: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.3 MiB/s wr, 716 op/s 2024-04-25T13:48:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:02 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:48:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:02 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:02 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:48:02.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:02 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:02.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:02 smithi027 ceph-mon[29140]: pgmap v563: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.3 MiB/s wr, 716 op/s 2024-04-25T13:48:02.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:02 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:48:02.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:02 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:02.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:02 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:48:02.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:02 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 415.0 s, 2249.0 tps, lat 44.576 ms stddev 50.862 2024-04-25T13:48:03.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:03 smithi122 ceph-mon[32843]: pgmap v564: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.4 MiB/s wr, 750 op/s 2024-04-25T13:48:03.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:03 smithi045 ceph-mon[32731]: pgmap v564: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.4 MiB/s wr, 750 op/s 2024-04-25T13:48:03.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:03 smithi027 ceph-mon[29140]: pgmap v564: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.4 MiB/s wr, 750 op/s 2024-04-25T13:48:04.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:04 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:48:04.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:04 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:48:04.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:04 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:48:05.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:05 smithi122 ceph-mon[32843]: pgmap v565: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 5.8 MiB/s wr, 624 op/s 2024-04-25T13:48:05.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:05 smithi045 ceph-mon[32731]: pgmap v565: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 5.8 MiB/s wr, 624 op/s 2024-04-25T13:48:05.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:05 smithi027 ceph-mon[29140]: pgmap v565: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 5.8 MiB/s wr, 624 op/s 2024-04-25T13:48:07.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:07 smithi027 ceph-mon[29140]: pgmap v566: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.4 MiB/s wr, 851 op/s 2024-04-25T13:48:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 420.0 s, 2788.4 tps, lat 35.640 ms stddev 43.054 2024-04-25T13:48:08.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:07 smithi122 ceph-mon[32843]: pgmap v566: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.4 MiB/s wr, 851 op/s 2024-04-25T13:48:08.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:07 smithi045 ceph-mon[32731]: pgmap v566: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.4 MiB/s wr, 851 op/s 2024-04-25T13:48:09.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:09 smithi027 ceph-mon[29140]: pgmap v567: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 5.8 MiB/s wr, 659 op/s 2024-04-25T13:48:10.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:09 smithi122 ceph-mon[32843]: pgmap v567: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 5.8 MiB/s wr, 659 op/s 2024-04-25T13:48:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:09 smithi045 ceph-mon[32731]: pgmap v567: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 5.8 MiB/s wr, 659 op/s 2024-04-25T13:48:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.870+0000 7fd6927f3640 1 -- 172.21.15.27:0/3581971320 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c1035b0 msgr2=0x7fd68c103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.870+0000 7fd6927f3640 1 --2- 172.21.15.27:0/3581971320 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c1035b0 0x7fd68c103ab0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd684009ca0 tx=0x7fd68402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.871+0000 7fd6927f3640 1 -- 172.21.15.27:0/3581971320 shutdown_connections 2024-04-25T13:48:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.871+0000 7fd6927f3640 1 --2- 172.21.15.27:0/3581971320 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c1035b0 0x7fd68c103ab0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.871+0000 7fd6927f3640 1 --2- 172.21.15.27:0/3581971320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd68c102a50 0x7fd68c102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.871+0000 7fd6927f3640 1 --2- 172.21.15.27:0/3581971320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd68c102070 0x7fd68c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.871+0000 7fd6927f3640 1 -- 172.21.15.27:0/3581971320 >> 172.21.15.27:0/3581971320 conn(0x7fd68c0fdc60 msgr2=0x7fd68c1000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.871+0000 7fd6927f3640 1 -- 172.21.15.27:0/3581971320 shutdown_connections 2024-04-25T13:48:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.871+0000 7fd6927f3640 1 -- 172.21.15.27:0/3581971320 wait complete. 2024-04-25T13:48:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.872+0000 7fd6927f3640 1 Processor -- start 2024-04-25T13:48:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.872+0000 7fd6927f3640 1 -- start start 2024-04-25T13:48:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.872+0000 7fd6927f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd68c102070 0x7fd68c19c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.872+0000 7fd6927f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c102a50 0x7fd68c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.872+0000 7fd6927f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd68c1035b0 0x7fd68c1a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.872+0000 7fd68b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c102a50 0x7fd68c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd68b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c102a50 0x7fd68c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57794/0 (socket says 172.21.15.27:57794) 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd68b7fe640 1 -- 172.21.15.27:0/2308872421 learned_addr learned my addr 172.21.15.27:0/2308872421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd690d90640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd68c1035b0 0x7fd68c1a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd68bfff640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd68c102070 0x7fd68c19c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c111830 con 0x7fd68c1035b0 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1116b0 con 0x7fd68c102070 2024-04-25T13:48:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1119b0 con 0x7fd68c102a50 2024-04-25T13:48:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd68b7fe640 1 -- 172.21.15.27:0/2308872421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd68c102070 msgr2=0x7fd68c19c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd68c102070 0x7fd68c19c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd68b7fe640 1 -- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd68c1035b0 msgr2=0x7fd68c1a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd68c1035b0 0x7fd68c1a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.873+0000 7fd68b7fe640 1 -- 172.21.15.27:0/2308872421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68c1a3ee0 con 0x7fd68c102a50 2024-04-25T13:48:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.874+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c102a50 0x7fd68c19c8e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd68000e9f0 tx=0x7fd68000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.874+0000 7fd6897fa640 1 -- 172.21.15.27:0/2308872421 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68000cd80 con 0x7fd68c102a50 2024-04-25T13:48:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.874+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68c1a41d0 con 0x7fd68c102a50 2024-04-25T13:48:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.874+0000 7fd6897fa640 1 -- 172.21.15.27:0/2308872421 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6800108e0 con 0x7fd68c102a50 2024-04-25T13:48:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.874+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd68c1a4770 con 0x7fd68c102a50 2024-04-25T13:48:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.874+0000 7fd6897fa640 1 -- 172.21.15.27:0/2308872421 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd680018980 con 0x7fd68c102a50 2024-04-25T13:48:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.877+0000 7fd6897fa640 1 -- 172.21.15.27:0/2308872421 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd680010430 con 0x7fd68c102a50 2024-04-25T13:48:11.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.879+0000 7fd6897fa640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd6740afb60 0x7fd6740b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.879+0000 7fd68bfff640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd6740afb60 0x7fd6740b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.879+0000 7fd6897fa640 1 -- 172.21.15.27:0/2308872421 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd6800def20 con 0x7fd68c102a50 2024-04-25T13:48:11.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.879+0000 7fd699780740 1 Processor -- start 2024-04-25T13:48:11.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.879+0000 7fd699780740 1 -- start start 2024-04-25T13:48:11.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.879+0000 7fd68bfff640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd6740afb60 0x7fd6740b2070 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd68c06bbd0 tx=0x7fd67c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:11.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.879+0000 7fd699780740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 0x555e5fb72180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd699780740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb72700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd699780740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 0x555e5fb79600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd699780740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555e5facf460 con 0x555e5f7a84d0 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd699780740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555e5fa89590 con 0x555e5f7c91b0 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd699780740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555e5facb420 con 0x555e5f7a02f0 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb72700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 0x555e5fb72180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb72700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57806/0 (socket says 172.21.15.27:57806) 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 -- 172.21.15.27:0/2892762282 learned_addr learned my addr 172.21.15.27:0/2892762282 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd690d90640 1 --2- 172.21.15.27:0/2892762282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 0x555e5fb79600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 -- 172.21.15.27:0/2892762282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 msgr2=0x555e5fb72180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/2892762282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 0x555e5fb72180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 -- 172.21.15.27:0/2892762282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 msgr2=0x555e5fb79600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/2892762282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 0x555e5fb79600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 -- 172.21.15.27:0/2892762282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e5fa960d0 con 0x555e5f7a02f0 2024-04-25T13:48:11.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.880+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/2892762282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb72700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd6800a46a0 tx=0x7fd6800107e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=929845354b70c3e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:11.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.881+0000 7fd6727fc640 1 -- 172.21.15.27:0/2892762282 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800a7430 con 0x555e5f7a02f0 2024-04-25T13:48:11.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.881+0000 7fd6727fc640 1 -- 172.21.15.27:0/2892762282 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6800ab7e0 con 0x555e5f7a02f0 2024-04-25T13:48:11.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.881+0000 7fd6727fc640 1 -- 172.21.15.27:0/2892762282 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800aee50 con 0x555e5f7a02f0 2024-04-25T13:48:11.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 -- 172.21.15.27:0/2892762282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 msgr2=0x555e5fb72700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 --2- 172.21.15.27:0/2892762282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb72700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd6800a46a0 tx=0x7fd6800107e0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 -- 172.21.15.27:0/2892762282 shutdown_connections 2024-04-25T13:48:11.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 --2- 172.21.15.27:0/2892762282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 0x555e5fb79600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 --2- 172.21.15.27:0/2892762282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb72700 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 --2- 172.21.15.27:0/2892762282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 0x555e5fb72180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 -- 172.21.15.27:0/2892762282 >> 172.21.15.27:0/2892762282 conn(0x555e5f7bd300 msgr2=0x555e5fb65300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 -- 172.21.15.27:0/2892762282 shutdown_connections 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.882+0000 7fd699780740 1 -- 172.21.15.27:0/2892762282 wait complete. 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.883+0000 7fd699780740 1 -- start start 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.883+0000 7fd699780740 1 Processor -- start 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd699780740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd699780740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 0x555e5fb7c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd699780740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 0x555e5fb833e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd699780740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555e5fb7a050 con 0x555e5f7a84d0 2024-04-25T13:48:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd699780740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555e5fa56b60 con 0x555e5f7c91b0 2024-04-25T13:48:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd699780740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555e5fb7a1d0 con 0x555e5f7a02f0 2024-04-25T13:48:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd699780740 10 client.0 _ll_register_callbacks cb 0x555e5fa7b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57812/0 (socket says 172.21.15.27:57812) 2024-04-25T13:48:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68bfff640 1 -- 172.21.15.27:0/1316491068 learned_addr learned my addr 172.21.15.27:0/1316491068 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 0x555e5fb7c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68bfff640 1 -- 172.21.15.27:0/1316491068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 msgr2=0x555e5fb833e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68bfff640 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 0x555e5fb833e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68bfff640 1 -- 172.21.15.27:0/1316491068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 msgr2=0x555e5fb7c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68bfff640 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 0x555e5fb7c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.884+0000 7fd68bfff640 1 -- 172.21.15.27:0/1316491068 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e5f7e1210 con 0x555e5f7a02f0 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.885+0000 7fd68bfff640 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7bf40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00ed70 tx=0x7fd67c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f1d2ac5eb5210cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.885+0000 7fd66b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.886+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1316491068 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c02b060 con 0x555e5f7a02f0 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.886+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1316491068 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd67c01db20 con 0x555e5f7a02f0 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.886+0000 7fd699780740 1 -- 172.21.15.27:0/1316491068 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6800ab7e0 con 0x555e5f7a02f0 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.886+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1316491068 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd67c01dd20 con 0x555e5f7a02f0 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.886+0000 7fd699780740 10 client.26599 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.886+0000 7fd699780740 1 -- 172.21.15.27:0/1316491068 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555e5f8ad620 con 0x555e5f7a02f0 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1316491068 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67c0268c0 con 0x555e5f7a02f0 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1316491068 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fd67c01c550 con 0x555e5f7a02f0 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd699780740 10 client.26599 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd699780740 10 client.26599 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd699780740 20 client.26599 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd699780740 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e5f787d80 0x555e5fb89a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd699780740 4 client.26599 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd699780740 1 -- 172.21.15.27:0/1316491068 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd6800ab7e0 con 0x555e5f787d80 2024-04-25T13:48:11.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.887+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e5f787d80 0x555e5fb89a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.888+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e5f787d80 0x555e5fb89a60 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c1888e3ad249c61 server_cookie=5bf810ce864fec20 in_seq=0 out_seq=0 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.888+0000 7fd66b7fe640 10 client.26599 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.889+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1316491068 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fd6800ab7e0 con 0x555e5f787d80 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.889+0000 7fd66b7fe640 10 client.26599 handle_command_reply: tid=0 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.889+0000 7fd699780740 1 client.26599 shutdown 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 -- 172.21.15.27:0/1316491068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 msgr2=0x555e5fb7bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7bf40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00ed70 tx=0x7fd67c004730 comp rx=0 tx=0).stop 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 -- 172.21.15.27:0/1316491068 shutdown_connections 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e5f787d80 0x555e5fb89a60 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7c91b0 0x555e5fb833e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7a84d0 0x555e5fb7c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 --2- 172.21.15.27:0/1316491068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7bf40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 -- 172.21.15.27:0/1316491068 >> 172.21.15.27:0/1316491068 conn(0x555e5f7bd300 msgr2=0x555e5fb64250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd66b7fe640 0 client.26599 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 -- 172.21.15.27:0/1316491068 shutdown_connections 2024-04-25T13:48:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 1 -- 172.21.15.27:0/1316491068 wait complete. 2024-04-25T13:48:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.890+0000 7fd699780740 20 client.26599 trim_cache size 0 max 16384 2024-04-25T13:48:11.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:11 smithi027 ceph-mon[29140]: pgmap v568: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.1 MiB/s wr, 755 op/s 2024-04-25T13:48:11.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.917+0000 7fd699780740 1 Processor -- start 2024-04-25T13:48:11.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.917+0000 7fd699780740 1 -- start start 2024-04-25T13:48:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.917+0000 7fd699780740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.917+0000 7fd699780740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 0x555e5fb83370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.918+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.918+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57816/0 (socket says 172.21.15.27:57816) 2024-04-25T13:48:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.918+0000 7fd68bfff640 1 -- 172.21.15.27:0/2296500697 learned_addr learned my addr 172.21.15.27:0/2296500697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.918+0000 7fd699780740 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 0x555e5fb68810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.918+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 0x555e5fb83370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.918 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.918+0000 7fd699780740 1 -- 172.21.15.27:0/2296500697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1a7190 con 0x555e5f7c91b0 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.918+0000 7fd699780740 1 -- 172.21.15.27:0/2296500697 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1a9a10 con 0x555e5f7a84d0 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.918+0000 7fd699780740 1 -- 172.21.15.27:0/2296500697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1a7800 con 0x555e5f7a02f0 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.919+0000 7fd690d90640 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 0x555e5fb68810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.919+0000 7fd68bfff640 1 -- 172.21.15.27:0/2296500697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 msgr2=0x555e5fb83370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.919+0000 7fd68bfff640 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 0x555e5fb83370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.919+0000 7fd68bfff640 1 -- 172.21.15.27:0/2296500697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 msgr2=0x555e5fb68810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.919+0000 7fd68bfff640 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 0x555e5fb68810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.919+0000 7fd68bfff640 1 -- 172.21.15.27:0/2296500697 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e5f787fb0 con 0x555e5f7a02f0 2024-04-25T13:48:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.920+0000 7fd68bfff640 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7afa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd67c01c920 tx=0x7fd67c009660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2ad7ff6cea4d4cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.920+0000 7fd671ffb640 1 -- 172.21.15.27:0/2296500697 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c036070 con 0x555e5f7a02f0 2024-04-25T13:48:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.920+0000 7fd671ffb640 1 -- 172.21.15.27:0/2296500697 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67c01db20 con 0x555e5f7a02f0 2024-04-25T13:48:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.921+0000 7fd671ffb640 1 -- 172.21.15.27:0/2296500697 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c004460 con 0x555e5f7a02f0 2024-04-25T13:48:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 -- 172.21.15.27:0/2296500697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 msgr2=0x555e5fb7afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7afa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd67c01c920 tx=0x7fd67c009660 comp rx=0 tx=0).stop 2024-04-25T13:48:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 -- 172.21.15.27:0/2296500697 shutdown_connections 2024-04-25T13:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 0x555e5fb68810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 0x555e5fb83370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 --2- 172.21.15.27:0/2296500697 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb7afa0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 -- 172.21.15.27:0/2296500697 >> 172.21.15.27:0/2296500697 conn(0x555e5f7bd300 msgr2=0x555e5fb68100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 -- 172.21.15.27:0/2296500697 shutdown_connections 2024-04-25T13:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.922+0000 7fd699780740 1 -- 172.21.15.27:0/2296500697 wait complete. 2024-04-25T13:48:11.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.923+0000 7fd699780740 1 -- start start 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.923+0000 7fd699780740 1 Processor -- start 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd699780740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb80640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd699780740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 0x555e5fb80b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd699780740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 0x555e5fb81ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb80640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd699780740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1a7ed0 con 0x555e5f7c91b0 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd699780740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1a9340 con 0x555e5f7a84d0 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd699780740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1a85a0 con 0x555e5f7a02f0 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd699780740 10 client.0 _ll_register_callbacks cb 0x7fd67c00e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb80640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57818/0 (socket says 172.21.15.27:57818) 2024-04-25T13:48:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd68bfff640 1 -- 172.21.15.27:0/1597134162 learned_addr learned my addr 172.21.15.27:0/1597134162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.924+0000 7fd690d90640 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 0x555e5fb81ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.925+0000 7fd68bfff640 1 -- 172.21.15.27:0/1597134162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 msgr2=0x555e5fb80b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.925+0000 7fd68bfff640 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 0x555e5fb80b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.925+0000 7fd68bfff640 1 -- 172.21.15.27:0/1597134162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 msgr2=0x555e5fb81ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.925+0000 7fd68bfff640 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 0x555e5fb81ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.925+0000 7fd68bfff640 1 -- 172.21.15.27:0/1597134162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e5fb82070 con 0x555e5f7a02f0 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.925+0000 7fd68bfff640 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb80640 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd67c01ca90 tx=0x7fd67c01c6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2004f6e1cbc565d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.925+0000 7fd66b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.925+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1597134162 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c036040 con 0x555e5f7a02f0 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.926+0000 7fd699780740 1 -- 172.21.15.27:0/1597134162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd67c01db20 con 0x555e5f7a02f0 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.926+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1597134162 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd67c009030 con 0x555e5f7a02f0 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.926+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1597134162 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd67c00e920 con 0x555e5f7a02f0 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.926+0000 7fd699780740 10 client.26605 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.926+0000 7fd699780740 1 -- 172.21.15.27:0/1597134162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555e5fb83300 con 0x555e5f7a02f0 2024-04-25T13:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.926+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1597134162 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67c03f070 con 0x555e5f7a02f0 2024-04-25T13:48:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.927+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1597134162 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fd67c046460 con 0x555e5f7a02f0 2024-04-25T13:48:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.928+0000 7fd699780740 10 client.26605 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.928+0000 7fd699780740 10 client.26605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.928+0000 7fd699780740 20 client.26605 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.928+0000 7fd699780740 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e5f75d1c0 0x555e5fbf40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.928+0000 7fd699780740 4 client.26605 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:48:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.928+0000 7fd699780740 1 -- 172.21.15.27:0/1597134162 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd67c01db20 con 0x555e5f75d1c0 2024-04-25T13:48:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.928+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e5f75d1c0 0x555e5fbf40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.929+0000 7fd66b7fe640 10 client.26605 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.929+0000 7fd68b7fe640 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e5f75d1c0 0x555e5fbf40d0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47a11d44a68dd35f server_cookie=db1bdc2b97bb99f6 in_seq=0 out_seq=0 2024-04-25T13:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.952+0000 7fd66b7fe640 1 -- 172.21.15.27:0/1597134162 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd67c01db20 con 0x555e5f75d1c0 2024-04-25T13:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.952+0000 7fd66b7fe640 10 client.26605 handle_command_reply: tid=0 2024-04-25T13:48:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.952+0000 7fd699780740 1 client.26605 shutdown 2024-04-25T13:48:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 -- 172.21.15.27:0/1597134162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 msgr2=0x555e5fb80640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb80640 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd67c01ca90 tx=0x7fd67c01c6b0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 -- 172.21.15.27:0/1597134162 shutdown_connections 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e5f75d1c0 0x555e5fbf40d0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e5f7c91b0 0x555e5fb81ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e5f7a84d0 0x555e5fb80b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 --2- 172.21.15.27:0/1597134162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e5f7a02f0 0x555e5fb80640 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 -- 172.21.15.27:0/1597134162 >> 172.21.15.27:0/1597134162 conn(0x555e5f7bd300 msgr2=0x555e5fb64250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd66b7fe640 0 client.26605 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 -- 172.21.15.27:0/1597134162 shutdown_connections 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.953+0000 7fd699780740 1 -- 172.21.15.27:0/1597134162 wait complete. 2024-04-25T13:48:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.954+0000 7fd699780740 20 client.26605 trim_cache size 0 max 16384 2024-04-25T13:48:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:48:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:48:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:48:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:48:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd6740afb60 msgr2=0x7fd6740b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd6740afb60 0x7fd6740b2070 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd68c06bbd0 tx=0x7fd67c005f90 comp rx=0 tx=0).stop 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c102a50 msgr2=0x7fd68c19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c102a50 0x7fd68c19c8e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd68000e9f0 tx=0x7fd68000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 shutdown_connections 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd6740afb60 0x7fd6740b2070 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd68c1035b0 0x7fd68c1a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd68c102a50 0x7fd68c19c8e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 --2- 172.21.15.27:0/2308872421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd68c102070 0x7fd68c19c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 >> 172.21.15.27:0/2308872421 conn(0x7fd68c0fdc60 msgr2=0x7fd68c0ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 shutdown_connections 2024-04-25T13:48:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:11.955+0000 7fd6927f3640 1 -- 172.21.15.27:0/2308872421 wait complete. 2024-04-25T13:48:12.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:11 smithi122 ceph-mon[32843]: pgmap v568: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.1 MiB/s wr, 755 op/s 2024-04-25T13:48:12.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:11 smithi045 ceph-mon[32731]: pgmap v568: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.1 MiB/s wr, 755 op/s 2024-04-25T13:48:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 425.0 s, 2797.6 tps, lat 35.823 ms stddev 42.279 2024-04-25T13:48:13.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:13 smithi027 ceph-mon[29140]: pgmap v569: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 7.4 MiB/s wr, 747 op/s 2024-04-25T13:48:14.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:13 smithi122 ceph-mon[32843]: pgmap v569: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 7.4 MiB/s wr, 747 op/s 2024-04-25T13:48:14.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:13 smithi045 ceph-mon[32731]: pgmap v569: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 7.4 MiB/s wr, 747 op/s 2024-04-25T13:48:15.878 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:15 smithi045 ceph-mon[32731]: pgmap v570: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.4 MiB/s wr, 679 op/s 2024-04-25T13:48:15.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:15 smithi027 ceph-mon[29140]: pgmap v570: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.4 MiB/s wr, 679 op/s 2024-04-25T13:48:16.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:15 smithi122 ceph-mon[32843]: pgmap v570: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.4 MiB/s wr, 679 op/s 2024-04-25T13:48:17.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:17 smithi027 ceph-mon[29140]: pgmap v571: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.7 MiB/s wr, 865 op/s 2024-04-25T13:48:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 430.0 s, 2667.2 tps, lat 37.490 ms stddev 47.150 2024-04-25T13:48:18.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:17 smithi122 ceph-mon[32843]: pgmap v571: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.7 MiB/s wr, 865 op/s 2024-04-25T13:48:18.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:17 smithi045 ceph-mon[32731]: pgmap v571: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.7 MiB/s wr, 865 op/s 2024-04-25T13:48:18.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:18 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:48:19.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:18 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:48:19.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:18 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:48:19.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: pgmap v572: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.0 MiB/s wr, 630 op/s 2024-04-25T13:48:19.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:19.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:19.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:19.901 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-04-25T13:48:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: Unable to set osd_memory_target on smithi045 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T13:48:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:19.902 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: Adjusting osd_memory_target on smithi122 to 176.1M 2024-04-25T13:48:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: Unable to set osd_memory_target on smithi122 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-25T13:48:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:48:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:48:19.903 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:19 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:48:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: pgmap v572: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.0 MiB/s wr, 630 op/s 2024-04-25T13:48:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: Unable to set osd_memory_target on smithi045 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: Adjusting osd_memory_target on smithi122 to 176.1M 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: Unable to set osd_memory_target on smithi122 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:48:20.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:19 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: pgmap v572: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.0 MiB/s wr, 630 op/s 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: Unable to set osd_memory_target on smithi045 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T13:48:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T13:48:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: Adjusting osd_memory_target on smithi122 to 176.1M 2024-04-25T13:48:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: Unable to set osd_memory_target on smithi122 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-25T13:48:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:48:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:48:20.079 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:19 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:48:22.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:21 smithi122 ceph-mon[32843]: pgmap v573: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.0 MiB/s wr, 729 op/s 2024-04-25T13:48:22.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:21 smithi045 ceph-mon[32731]: pgmap v573: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.0 MiB/s wr, 729 op/s 2024-04-25T13:48:22.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:21 smithi027 ceph-mon[29140]: pgmap v573: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.0 MiB/s wr, 729 op/s 2024-04-25T13:48:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 435.0 s, 1988.8 tps, lat 49.913 ms stddev 56.348 2024-04-25T13:48:24.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:23 smithi122 ceph-mon[32843]: pgmap v574: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 723 op/s 2024-04-25T13:48:24.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:23 smithi045 ceph-mon[32731]: pgmap v574: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 723 op/s 2024-04-25T13:48:24.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:23 smithi027 ceph-mon[29140]: pgmap v574: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.7 MiB/s wr, 723 op/s 2024-04-25T13:48:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.103+0000 7ff491dba640 1 -- 172.21.15.27:0/1738831220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff48c106ff0 msgr2=0x7ff48c107400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.103+0000 7ff491dba640 1 --2- 172.21.15.27:0/1738831220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff48c106ff0 0x7ff48c107400 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff474009900 tx=0x7ff47402a410 comp rx=0 tx=0).stop 2024-04-25T13:48:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.103+0000 7ff491dba640 1 -- 172.21.15.27:0/1738831220 shutdown_connections 2024-04-25T13:48:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.103+0000 7ff491dba640 1 --2- 172.21.15.27:0/1738831220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff48c10bcf0 0x7ff48c10e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.103+0000 7ff491dba640 1 --2- 172.21.15.27:0/1738831220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff48c106ff0 0x7ff48c107400 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.103+0000 7ff491dba640 1 --2- 172.21.15.27:0/1738831220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c105db0 0x7ff48c106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.103+0000 7ff491dba640 1 -- 172.21.15.27:0/1738831220 >> 172.21.15.27:0/1738831220 conn(0x7ff48c100f80 msgr2=0x7ff48c103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.104+0000 7ff491dba640 1 -- 172.21.15.27:0/1738831220 shutdown_connections 2024-04-25T13:48:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.104+0000 7ff491dba640 1 -- 172.21.15.27:0/1738831220 wait complete. 2024-04-25T13:48:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.105+0000 7ff491dba640 1 Processor -- start 2024-04-25T13:48:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.105+0000 7ff491dba640 1 -- start start 2024-04-25T13:48:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.105+0000 7ff491dba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff48c105db0 0x7ff48c1a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.105+0000 7ff491dba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff48c106ff0 0x7ff48c1a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.105+0000 7ff491dba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c10bcf0 0x7ff48c1a7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.105+0000 7ff491dba640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1118e0 con 0x7ff48c105db0 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.105+0000 7ff491dba640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c111760 con 0x7ff48c10bcf0 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.105+0000 7ff491dba640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c111a60 con 0x7ff48c106ff0 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c10bcf0 0x7ff48c1a7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c10bcf0 0x7ff48c1a7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:36134/0 (socket says 172.21.15.27:36134) 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48bfff640 1 -- 172.21.15.27:0/3548519588 learned_addr learned my addr 172.21.15.27:0/3548519588 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48b7fe640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff48c105db0 0x7ff48c1a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48affd640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff48c106ff0 0x7ff48c1a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48b7fe640 1 -- 172.21.15.27:0/3548519588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff48c106ff0 msgr2=0x7ff48c1a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48b7fe640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff48c106ff0 0x7ff48c1a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48b7fe640 1 -- 172.21.15.27:0/3548519588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c10bcf0 msgr2=0x7ff48c1a7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48b7fe640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c10bcf0 0x7ff48c1a7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.106+0000 7ff48b7fe640 1 -- 172.21.15.27:0/3548519588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c1a8380 con 0x7ff48c105db0 2024-04-25T13:48:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.107+0000 7ff48b7fe640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff48c105db0 0x7ff48c1a08b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff48c105680 tx=0x7ff47c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.107+0000 7ff488ff9640 1 -- 172.21.15.27:0/3548519588 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c00cc40 con 0x7ff48c105db0 2024-04-25T13:48:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.107+0000 7ff488ff9640 1 -- 172.21.15.27:0/3548519588 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c00cde0 con 0x7ff48c105db0 2024-04-25T13:48:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.107+0000 7ff488ff9640 1 -- 172.21.15.27:0/3548519588 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c010430 con 0x7ff48c105db0 2024-04-25T13:48:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.107+0000 7ff491dba640 1 -- 172.21.15.27:0/3548519588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48c1a8670 con 0x7ff48c105db0 2024-04-25T13:48:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.107+0000 7ff491dba640 1 -- 172.21.15.27:0/3548519588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48c1a8b40 con 0x7ff48c105db0 2024-04-25T13:48:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff498c60740 1 Processor -- start 2024-04-25T13:48:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff498c60740 1 -- start start 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff498c60740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920941b0 0x55e792434970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff498c60740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e79206b2f0 0x55e792434ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff498c60740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 0x55e79243bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff498c60740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e79239a460 con 0x55e79206b2f0 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff498c60740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e792354590 con 0x55e7920734d0 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff498c60740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e792396420 con 0x55e7920941b0 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920941b0 0x55e792434970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920941b0 0x55e792434970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57474/0 (socket says 172.21.15.27:57474) 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff48bfff640 1 -- 172.21.15.27:0/88162242 learned_addr learned my addr 172.21.15.27:0/88162242 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.111+0000 7ff48affd640 1 --2- 172.21.15.27:0/88162242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 0x55e79243bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.112+0000 7ff48bfff640 1 -- 172.21.15.27:0/88162242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 msgr2=0x55e79243bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.112+0000 7ff48bfff640 1 --2- 172.21.15.27:0/88162242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 0x55e79243bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.112+0000 7ff48bfff640 1 -- 172.21.15.27:0/88162242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e79206b2f0 msgr2=0x55e792434ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:48:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.112+0000 7ff48bfff640 1 --2- 172.21.15.27:0/88162242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e79206b2f0 0x55e792434ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.112+0000 7ff48bfff640 1 -- 172.21.15.27:0/88162242 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7923610d0 con 0x55e7920941b0 2024-04-25T13:48:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.112+0000 7ff488ff9640 1 -- 172.21.15.27:0/3548519588 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff47c0105d0 con 0x7ff48c105db0 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.113+0000 7ff48bfff640 1 --2- 172.21.15.27:0/88162242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920941b0 0x55e792434970 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff484011570 tx=0x7ff484011a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fa0cb362c7865d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.113+0000 7ff488ff9640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4780afb60 0x7ff4780b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.113+0000 7ff488ff9640 1 -- 172.21.15.27:0/3548519588 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff47c0deec0 con 0x7ff48c105db0 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.113+0000 7ff481ffb640 1 -- 172.21.15.27:0/88162242 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484004b20 con 0x55e7920941b0 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.114+0000 7ff481ffb640 1 -- 172.21.15.27:0/88162242 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff484004cc0 con 0x55e7920941b0 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.114+0000 7ff481ffb640 1 -- 172.21.15.27:0/88162242 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484005490 con 0x55e7920941b0 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.114+0000 7ff48b7fe640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4780afb60 0x7ff4780b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 -- 172.21.15.27:0/88162242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920941b0 msgr2=0x55e792434970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff48b7fe640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4780afb60 0x7ff4780b2070 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0dc9f0 tx=0x7ff47c00cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 --2- 172.21.15.27:0/88162242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920941b0 0x55e792434970 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff484011570 tx=0x7ff484011a30 comp rx=0 tx=0).stop 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 -- 172.21.15.27:0/88162242 shutdown_connections 2024-04-25T13:48:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 --2- 172.21.15.27:0/88162242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 0x55e79243bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 --2- 172.21.15.27:0/88162242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e79206b2f0 0x55e792434ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 --2- 172.21.15.27:0/88162242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920941b0 0x55e792434970 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 -- 172.21.15.27:0/88162242 >> 172.21.15.27:0/88162242 conn(0x55e792088300 msgr2=0x55e792430300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 -- 172.21.15.27:0/88162242 shutdown_connections 2024-04-25T13:48:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 -- 172.21.15.27:0/88162242 wait complete. 2024-04-25T13:48:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.115+0000 7ff498c60740 1 -- start start 2024-04-25T13:48:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.116+0000 7ff498c60740 1 Processor -- start 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.116+0000 7ff498c60740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e79243e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff498c60740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7920734d0 0x55e79243eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff498c60740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920941b0 0x55e792445a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff498c60740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e79243c890 con 0x55e7920734d0 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff498c60740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e792321b60 con 0x55e7920941b0 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff498c60740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e79243ca10 con 0x55e79206b2f0 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e79243e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e79243e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57490/0 (socket says 172.21.15.27:57490) 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 -- 172.21.15.27:0/2252428263 learned_addr learned my addr 172.21.15.27:0/2252428263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff498c60740 10 client.0 _ll_register_callbacks cb 0x55e7923462b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920941b0 0x55e792445a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48affd640 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7920734d0 0x55e79243eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 -- 172.21.15.27:0/2252428263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920941b0 msgr2=0x55e792445a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920941b0 0x55e792445a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 -- 172.21.15.27:0/2252428263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7920734d0 msgr2=0x55e79243eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7920734d0 0x55e79243eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 -- 172.21.15.27:0/2252428263 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7920ac210 con 0x55e79206b2f0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48affd640 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7920734d0 0x55e79243eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e79243e5d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff48401b040 tx=0x7ff484004e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e185f7b6be7fb7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff46affd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff46affd640 1 -- 172.21.15.27:0/2252428263 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48402b020 con 0x55e79206b2f0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.117+0000 7ff46affd640 1 -- 172.21.15.27:0/2252428263 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff484004ac0 con 0x55e79206b2f0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.118+0000 7ff498c60740 1 -- 172.21.15.27:0/2252428263 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff484004cc0 con 0x55e79206b2f0 2024-04-25T13:48:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.118+0000 7ff46affd640 1 -- 172.21.15.27:0/2252428263 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff4840053a0 con 0x55e79206b2f0 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.118+0000 7ff498c60740 10 client.26617 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.118+0000 7ff498c60740 1 -- 172.21.15.27:0/2252428263 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e792178620 con 0x55e79206b2f0 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.118+0000 7ff46affd640 1 -- 172.21.15.27:0/2252428263 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff484004cc0 con 0x55e79206b2f0 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.118+0000 7ff46affd640 1 -- 172.21.15.27:0/2252428263 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff484027970 con 0x55e79206b2f0 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.118+0000 7ff498c60740 10 client.26617 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.118+0000 7ff498c60740 10 client.26617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.119+0000 7ff498c60740 20 client.26617 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.119+0000 7ff498c60740 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e792052d80 0x55e79244c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.119+0000 7ff498c60740 4 client.26617 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.119+0000 7ff498c60740 1 -- 172.21.15.27:0/2252428263 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7923a8070 con 0x55e792052d80 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.120+0000 7ff48affd640 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e792052d80 0x55e79244c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.120+0000 7ff48affd640 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e792052d80 0x55e79244c070 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b780850e4f3fab1c server_cookie=5f24e061df7c6a85 in_seq=0 out_seq=0 2024-04-25T13:48:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.120+0000 7ff46affd640 10 client.26617 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.121+0000 7ff46affd640 1 -- 172.21.15.27:0/2252428263 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e7923a8070 con 0x55e792052d80 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.121+0000 7ff46affd640 10 client.26617 handle_command_reply: tid=0 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.121+0000 7ff498c60740 1 client.26617 shutdown 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 -- 172.21.15.27:0/2252428263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 msgr2=0x55e79243e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e79243e5d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff48401b040 tx=0x7ff484004e70 comp rx=0 tx=0).stop 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 -- 172.21.15.27:0/2252428263 shutdown_connections 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e792052d80 0x55e79244c070 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff46affd640 0 client.26617 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920941b0 0x55e792445a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e7920734d0 0x55e79243eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 --2- 172.21.15.27:0/2252428263 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e79243e5d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 -- 172.21.15.27:0/2252428263 >> 172.21.15.27:0/2252428263 conn(0x55e792088300 msgr2=0x55e79242f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.122+0000 7ff498c60740 1 -- 172.21.15.27:0/2252428263 shutdown_connections 2024-04-25T13:48:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.123+0000 7ff498c60740 1 -- 172.21.15.27:0/2252428263 wait complete. 2024-04-25T13:48:25.125 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.123+0000 7ff498c60740 20 client.26617 trim_cache size 0 max 16384 2024-04-25T13:48:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.145+0000 7ff498c60740 1 Processor -- start 2024-04-25T13:48:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.145+0000 7ff498c60740 1 -- start start 2024-04-25T13:48:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.145+0000 7ff498c60740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e79242fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.145+0000 7ff498c60740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e79206b2f0 0x55e792445a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.145+0000 7ff498c60740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920734d0 0x55e7924300e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.145+0000 7ff498c60740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1af780 con 0x55e792052d80 2024-04-25T13:48:25.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.145+0000 7ff498c60740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1b2000 con 0x55e79206b2f0 2024-04-25T13:48:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.145+0000 7ff498c60740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1afdf0 con 0x55e7920734d0 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.146+0000 7ff48affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e79206b2f0 0x55e792445a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.146+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e79242fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.146+0000 7ff48bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e79242fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39700/0 (socket says 172.21.15.27:39700) 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.146+0000 7ff48bfff640 1 -- 172.21.15.27:0/2353806980 learned_addr learned my addr 172.21.15.27:0/2353806980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.146+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2353806980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920734d0 0x55e7924300e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff48bfff640 1 -- 172.21.15.27:0/2353806980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e79206b2f0 msgr2=0x55e792445a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2353806980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e79206b2f0 0x55e792445a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff48bfff640 1 -- 172.21.15.27:0/2353806980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 msgr2=0x55e79242fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2353806980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e79242fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff48bfff640 1 -- 172.21.15.27:0/2353806980 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e79243bd90 con 0x55e7920734d0 2024-04-25T13:48:25.147 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2353806980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920734d0 0x55e7924300e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff484005d70 tx=0x7ff48400f8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4731d43e9404a531 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:25.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff4817fa640 1 -- 172.21.15.27:0/2353806980 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484041020 con 0x55e7920734d0 2024-04-25T13:48:25.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff4817fa640 1 -- 172.21.15.27:0/2353806980 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff484004fa0 con 0x55e7920734d0 2024-04-25T13:48:25.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.147+0000 7ff4817fa640 1 -- 172.21.15.27:0/2353806980 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48402b020 con 0x55e7920734d0 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.148+0000 7ff498c60740 1 -- 172.21.15.27:0/2353806980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920734d0 msgr2=0x55e7924300e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.149+0000 7ff498c60740 1 --2- 172.21.15.27:0/2353806980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920734d0 0x55e7924300e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff484005d70 tx=0x7ff48400f8d0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.149+0000 7ff498c60740 1 -- 172.21.15.27:0/2353806980 shutdown_connections 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.149+0000 7ff498c60740 1 --2- 172.21.15.27:0/2353806980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e7920734d0 0x55e7924300e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.149+0000 7ff498c60740 1 --2- 172.21.15.27:0/2353806980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e79206b2f0 0x55e792445a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.149+0000 7ff498c60740 1 --2- 172.21.15.27:0/2353806980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e79242fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.149+0000 7ff498c60740 1 -- 172.21.15.27:0/2353806980 >> 172.21.15.27:0/2353806980 conn(0x55e792088300 msgr2=0x55e7924389d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.149+0000 7ff498c60740 1 -- 172.21.15.27:0/2353806980 shutdown_connections 2024-04-25T13:48:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.149+0000 7ff498c60740 1 -- 172.21.15.27:0/2353806980 wait complete. 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 1 -- start start 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 1 Processor -- start 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e792440920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e792440e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 0x55e7924b2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1b04c0 con 0x55e792052d80 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1b1930 con 0x55e7920734d0 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1b0b90 con 0x55e79206b2f0 2024-04-25T13:48:25.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff498c60740 10 client.0 _ll_register_callbacks cb 0x7ff48400fa30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff48affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e792440e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff48affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e792440e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57514/0 (socket says 172.21.15.27:57514) 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff48affd640 1 -- 172.21.15.27:0/2308265782 learned_addr learned my addr 172.21.15.27:0/2308265782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e792440920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.150+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 0x55e7924b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.151+0000 7ff48affd640 1 -- 172.21.15.27:0/2308265782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 msgr2=0x55e7924b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.151+0000 7ff48affd640 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 0x55e7924b2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.151+0000 7ff48affd640 1 -- 172.21.15.27:0/2308265782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 msgr2=0x55e792440920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.151+0000 7ff48affd640 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e792440920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.151+0000 7ff48affd640 1 -- 172.21.15.27:0/2308265782 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7924414a0 con 0x55e79206b2f0 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.151+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e792440920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.151+0000 7ff48affd640 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e792440e60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff47402a9d0 tx=0x7ff474009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a56a83975237062 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:25.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.151+0000 7ff46affd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.152+0000 7ff46affd640 1 -- 172.21.15.27:0/2308265782 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47404a070 con 0x55e79206b2f0 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.152+0000 7ff46affd640 1 -- 172.21.15.27:0/2308265782 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff474046720 con 0x55e79206b2f0 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.152+0000 7ff498c60740 1 -- 172.21.15.27:0/2308265782 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff484004fa0 con 0x55e79206b2f0 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff46affd640 1 -- 172.21.15.27:0/2308265782 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff474054050 con 0x55e79206b2f0 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff498c60740 10 client.26623 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff498c60740 1 -- 172.21.15.27:0/2308265782 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e7924b3ad0 con 0x55e79206b2f0 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff46affd640 1 -- 172.21.15.27:0/2308265782 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff474046920 con 0x55e79206b2f0 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff46affd640 1 -- 172.21.15.27:0/2308265782 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff4740317a0 con 0x55e79206b2f0 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff498c60740 10 client.26623 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff498c60740 10 client.26623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff498c60740 20 client.26623 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff498c60740 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e7920941b0 0x55e7924b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:25.155 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff498c60740 4 client.26623 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:48:25.155 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff498c60740 1 -- 172.21.15.27:0/2308265782 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff484004fa0 con 0x55e7920941b0 2024-04-25T13:48:25.155 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.153+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e7920941b0 0x55e7924b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:25.155 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.154+0000 7ff46affd640 10 client.26623 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:25.155 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.155+0000 7ff48bfff640 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e7920941b0 0x55e7924b64a0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4c675d12fabbe37 server_cookie=1c659442303b4181 in_seq=0 out_seq=0 2024-04-25T13:48:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.177+0000 7ff46affd640 1 -- 172.21.15.27:0/2308265782 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff484004fa0 con 0x55e7920941b0 2024-04-25T13:48:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.177+0000 7ff46affd640 10 client.26623 handle_command_reply: tid=0 2024-04-25T13:48:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.177+0000 7ff498c60740 1 client.26623 shutdown 2024-04-25T13:48:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 -- 172.21.15.27:0/2308265782 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 msgr2=0x55e792440e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e792440e60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff47402a9d0 tx=0x7ff474009880 comp rx=0 tx=0).stop 2024-04-25T13:48:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 -- 172.21.15.27:0/2308265782 shutdown_connections 2024-04-25T13:48:25.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e7920941b0 0x55e7924b64a0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7920734d0 0x55e7924b2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e79206b2f0 0x55e792440e60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 --2- 172.21.15.27:0/2308265782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e792052d80 0x55e792440920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff46affd640 0 client.26623 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 -- 172.21.15.27:0/2308265782 >> 172.21.15.27:0/2308265782 conn(0x55e792088300 msgr2=0x55e79242fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 -- 172.21.15.27:0/2308265782 shutdown_connections 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 1 -- 172.21.15.27:0/2308265782 wait complete. 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.178+0000 7ff498c60740 20 client.26623 trim_cache size 0 max 16384 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:48:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:48:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:48:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:48:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 -- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4780afb60 msgr2=0x7ff4780b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4780afb60 0x7ff4780b2070 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0dc9f0 tx=0x7ff47c00cf90 comp rx=0 tx=0).stop 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 -- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff48c105db0 msgr2=0x7ff48c1a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff48c105db0 0x7ff48c1a08b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff48c105680 tx=0x7ff47c00ee50 comp rx=0 tx=0).stop 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 -- 172.21.15.27:0/3548519588 shutdown_connections 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4780afb60 0x7ff4780b2070 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff48c10bcf0 0x7ff48c1a7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff48c106ff0 0x7ff48c1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 --2- 172.21.15.27:0/3548519588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff48c105db0 0x7ff48c1a08b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 -- 172.21.15.27:0/3548519588 >> 172.21.15.27:0/3548519588 conn(0x7ff48c100f80 msgr2=0x7ff48c107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 -- 172.21.15.27:0/3548519588 shutdown_connections 2024-04-25T13:48:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:25.179+0000 7ff491dba640 1 -- 172.21.15.27:0/3548519588 wait complete. 2024-04-25T13:48:26.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:25 smithi122 ceph-mon[32843]: pgmap v575: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-25T13:48:26.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:25 smithi045 ceph-mon[32731]: pgmap v575: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-25T13:48:26.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:25 smithi027 ceph-mon[29140]: pgmap v575: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-25T13:48:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 440.0 s, 2495.0 tps, lat 40.189 ms stddev 44.255 2024-04-25T13:48:28.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:27 smithi122 ceph-mon[32843]: pgmap v576: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.0 MiB/s wr, 835 op/s 2024-04-25T13:48:28.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:27 smithi045 ceph-mon[32731]: pgmap v576: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.0 MiB/s wr, 835 op/s 2024-04-25T13:48:28.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:27 smithi027 ceph-mon[29140]: pgmap v576: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.0 MiB/s wr, 835 op/s 2024-04-25T13:48:30.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:29 smithi122 ceph-mon[32843]: pgmap v577: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 624 op/s 2024-04-25T13:48:30.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:29 smithi045 ceph-mon[32731]: pgmap v577: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 624 op/s 2024-04-25T13:48:30.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:29 smithi027 ceph-mon[29140]: pgmap v577: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 624 op/s 2024-04-25T13:48:31.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:48:31.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.641+0000 7fbb20f64640 1 -- 172.21.15.27:0/1936655355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c109780 msgr2=0x7fbb1c10bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.641+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1936655355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c109780 0x7fbb1c10bc90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009860 tx=0x7fbb1402a340 comp rx=0 tx=0).stop 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.641+0000 7fbb20f64640 1 -- 172.21.15.27:0/1936655355 shutdown_connections 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.641+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1936655355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c109780 0x7fbb1c10bc90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.641+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1936655355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb1c106c10 0x7fbb1c1090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.641+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1936655355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb1c101e30 0x7fbb1c102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.642+0000 7fbb20f64640 1 -- 172.21.15.27:0/1936655355 >> 172.21.15.27:0/1936655355 conn(0x7fbb1c0fd340 msgr2=0x7fbb1c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.642+0000 7fbb20f64640 1 -- 172.21.15.27:0/1936655355 shutdown_connections 2024-04-25T13:48:31.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.642+0000 7fbb20f64640 1 -- 172.21.15.27:0/1936655355 wait complete. 2024-04-25T13:48:31.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.643+0000 7fbb20f64640 1 Processor -- start 2024-04-25T13:48:31.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.643+0000 7fbb20f64640 1 -- start start 2024-04-25T13:48:31.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.643+0000 7fbb20f64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c101e30 0x7fbb1c1a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.643+0000 7fbb20f64640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb1c106c10 0x7fbb1c1a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.643+0000 7fbb20f64640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb1c109780 0x7fbb1c1a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb20f64640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c1042d0 con 0x7fbb1c106c10 2024-04-25T13:48:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb20f64640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c104150 con 0x7fbb1c101e30 2024-04-25T13:48:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb20f64640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c104450 con 0x7fbb1c109780 2024-04-25T13:48:31.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c101e30 0x7fbb1c1a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c101e30 0x7fbb1c1a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:36198/0 (socket says 172.21.15.27:36198) 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1a59c640 1 -- 172.21.15.27:0/1969321846 learned_addr learned my addr 172.21.15.27:0/1969321846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1ad9d640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb1c109780 0x7fbb1c1a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb1c106c10 0x7fbb1c1a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1a59c640 1 -- 172.21.15.27:0/1969321846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb1c109780 msgr2=0x7fbb1c1a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb1c109780 0x7fbb1c1a7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1a59c640 1 -- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb1c106c10 msgr2=0x7fbb1c1a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb1c106c10 0x7fbb1c1a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.644+0000 7fbb1a59c640 1 -- 172.21.15.27:0/1969321846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c1a8110 con 0x7fbb1c101e30 2024-04-25T13:48:31.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.645+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c101e30 0x7fbb1c1a05d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00c3a0 tx=0x7fbb0c00c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:31.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.645+0000 7fbb0b7fe640 1 -- 172.21.15.27:0/1969321846 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c01b070 con 0x7fbb1c101e30 2024-04-25T13:48:31.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.645+0000 7fbb0b7fe640 1 -- 172.21.15.27:0/1969321846 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb0c00bae0 con 0x7fbb1c101e30 2024-04-25T13:48:31.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.645+0000 7fbb0b7fe640 1 -- 172.21.15.27:0/1969321846 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c016bf0 con 0x7fbb1c101e30 2024-04-25T13:48:31.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.645+0000 7fbb1ad9d640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb1c109780 0x7fbb1c1a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:31.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.645+0000 7fbb20f64640 1 -- 172.21.15.27:0/1969321846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1c1a8400 con 0x7fbb1c101e30 2024-04-25T13:48:31.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.646+0000 7fbb20f64640 1 -- 172.21.15.27:0/1969321846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1c1a8900 con 0x7fbb1c101e30 2024-04-25T13:48:31.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.649+0000 7fbb27e13740 1 Processor -- start 2024-04-25T13:48:31.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.651+0000 7fbb0b7fe640 1 -- 172.21.15.27:0/1969321846 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbb0c01e070 con 0x7fbb1c101e30 2024-04-25T13:48:31.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.652+0000 7fbb0b7fe640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb040afcc0 0x7fbb040b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.652+0000 7fbb0b7fe640 1 -- 172.21.15.27:0/1969321846 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbb0c0deae0 con 0x7fbb1c101e30 2024-04-25T13:48:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.654+0000 7fbb27e13740 1 -- start start 2024-04-25T13:48:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.654+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 0x5628fe43ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.654+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe43efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.654+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe07c6b0 0x5628fe43fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.654+0000 7fbb27e13740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5628fe3a3d80 con 0x5628fe07c6b0 2024-04-25T13:48:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.654+0000 7fbb27e13740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5628fe35d8b0 con 0x5628fe09d390 2024-04-25T13:48:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.654+0000 7fbb27e13740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5628fe39fd40 con 0x5628fe0744d0 2024-04-25T13:48:31.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.658+0000 7fbb19d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe43efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.658+0000 7fbb1ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe07c6b0 0x5628fe43fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.658+0000 7fbb1ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe07c6b0 0x5628fe43fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39746/0 (socket says 172.21.15.27:39746) 2024-04-25T13:48:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.658+0000 7fbb1ad9d640 1 -- 172.21.15.27:0/2924428108 learned_addr learned my addr 172.21.15.27:0/2924428108 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.658+0000 7fbb1ad9d640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb040afcc0 0x7fbb040b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.658+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/2924428108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 0x5628fe43ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.661+0000 7fbb19d9b640 1 -- 172.21.15.27:0/2924428108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 msgr2=0x5628fe43ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.661+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/2924428108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 0x5628fe43ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.661+0000 7fbb19d9b640 1 -- 172.21.15.27:0/2924428108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe07c6b0 msgr2=0x5628fe43fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:31.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.661+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/2924428108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe07c6b0 0x5628fe43fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.661+0000 7fbb19d9b640 1 -- 172.21.15.27:0/2924428108 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628fe32ea40 con 0x5628fe0744d0 2024-04-25T13:48:31.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.666+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/2924428108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe43efe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbb10005d00 tx=0x7fbb100107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f1b7e376f8b40e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.668+0000 7fbb1ad9d640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb040afcc0 0x7fbb040b21d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbb14005eb0 tx=0x7fbb14005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.670+0000 7fbb08ff9640 1 -- 172.21.15.27:0/2924428108 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1001c480 con 0x5628fe0744d0 2024-04-25T13:48:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.670+0000 7fbb08ff9640 1 -- 172.21.15.27:0/2924428108 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb1001cad0 con 0x5628fe0744d0 2024-04-25T13:48:31.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.670+0000 7fbb08ff9640 1 -- 172.21.15.27:0/2924428108 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10011b50 con 0x5628fe0744d0 2024-04-25T13:48:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.672+0000 7fbb27e13740 1 -- 172.21.15.27:0/2924428108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 msgr2=0x5628fe43efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.672+0000 7fbb27e13740 1 --2- 172.21.15.27:0/2924428108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe43efe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbb10005d00 tx=0x7fbb100107a0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.676+0000 7fbb27e13740 1 -- 172.21.15.27:0/2924428108 shutdown_connections 2024-04-25T13:48:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.676+0000 7fbb27e13740 1 --2- 172.21.15.27:0/2924428108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe07c6b0 0x5628fe43fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.676+0000 7fbb27e13740 1 --2- 172.21.15.27:0/2924428108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe43efe0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.676+0000 7fbb27e13740 1 --2- 172.21.15.27:0/2924428108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 0x5628fe43ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.676+0000 7fbb27e13740 1 -- 172.21.15.27:0/2924428108 >> 172.21.15.27:0/2924428108 conn(0x5628fe0914e0 msgr2=0x5628fe43a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:31.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.681+0000 7fbb27e13740 1 -- 172.21.15.27:0/2924428108 shutdown_connections 2024-04-25T13:48:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.685+0000 7fbb27e13740 1 -- 172.21.15.27:0/2924428108 wait complete. 2024-04-25T13:48:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.685+0000 7fbb27e13740 1 -- start start 2024-04-25T13:48:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.685+0000 7fbb27e13740 1 Processor -- start 2024-04-25T13:48:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.686+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 0x5628fe4599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.686+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe07c6b0 0x5628fe459ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.686+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 0x5628fe460e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.687+0000 7fbb19d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe07c6b0 0x5628fe459ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.687+0000 7fbb19d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe07c6b0 0x5628fe459ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57552/0 (socket says 172.21.15.27:57552) 2024-04-25T13:48:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.687+0000 7fbb19d9b640 1 -- 172.21.15.27:0/1533920864 learned_addr learned my addr 172.21.15.27:0/1533920864 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.687+0000 7fbb1ad9d640 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 0x5628fe460e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.687+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5628fe440780 con 0x5628fe0744d0 2024-04-25T13:48:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.687+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5628fe32ad10 con 0x5628fe09d390 2024-04-25T13:48:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.687+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5628fe440900 con 0x5628fe07c6b0 2024-04-25T13:48:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.687+0000 7fbb27e13740 10 client.0 _ll_register_callbacks cb 0x5628fe357210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:31.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.688+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 0x5628fe4599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.688+0000 7fbb19d9b640 1 -- 172.21.15.27:0/1533920864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 msgr2=0x5628fe460e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.688+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 0x5628fe460e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.688+0000 7fbb19d9b640 1 -- 172.21.15.27:0/1533920864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 msgr2=0x5628fe4599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.688+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 0x5628fe4599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.688+0000 7fbb19d9b640 1 -- 172.21.15.27:0/1533920864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628fe17f7d0 con 0x5628fe07c6b0 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.689+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe07c6b0 0x5628fe459ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000bf50 tx=0x7fbb1000bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92a4dcb1409901c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.689+0000 7fbb01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.689+0000 7fbb01ffb640 1 -- 172.21.15.27:0/1533920864 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10011890 con 0x5628fe07c6b0 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.689+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb1001cad0 con 0x5628fe07c6b0 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.689+0000 7fbb01ffb640 1 -- 172.21.15.27:0/1533920864 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbb10011a30 con 0x5628fe07c6b0 2024-04-25T13:48:31.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.690+0000 7fbb01ffb640 1 -- 172.21.15.27:0/1533920864 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fbb10011c30 con 0x5628fe07c6b0 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb27e13740 10 client.26635 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5628fe0b53f0 con 0x5628fe07c6b0 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb01ffb640 1 -- 172.21.15.27:0/1533920864 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb1001cad0 con 0x5628fe07c6b0 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb01ffb640 1 -- 172.21.15.27:0/1533920864 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fbb1002d030 con 0x5628fe07c6b0 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb27e13740 10 client.26635 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb27e13740 10 client.26635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb27e13740 20 client.26635 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb27e13740 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628fe05bf60 0x5628fe466f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb27e13740 4 client.26635 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.691+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628fe3b1990 con 0x5628fe05bf60 2024-04-25T13:48:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.692+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628fe05bf60 0x5628fe466f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.692+0000 7fbb01ffb640 10 client.26635 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.693+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628fe05bf60 0x5628fe466f60 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70805c87b475d14f server_cookie=83119f8be7bed2cf in_seq=0 out_seq=0 2024-04-25T13:48:31.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.695+0000 7fbb01ffb640 1 -- 172.21.15.27:0/1533920864 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5628fe3b1990 con 0x5628fe05bf60 2024-04-25T13:48:31.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.695+0000 7fbb01ffb640 10 client.26635 handle_command_reply: tid=0 2024-04-25T13:48:31.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.695+0000 7fbb27e13740 1 client.26635 shutdown 2024-04-25T13:48:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe07c6b0 msgr2=0x5628fe459ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe07c6b0 0x5628fe459ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000bf50 tx=0x7fbb1000bf80 comp rx=0 tx=0).stop 2024-04-25T13:48:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 shutdown_connections 2024-04-25T13:48:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628fe05bf60 0x5628fe466f60 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe09d390 0x5628fe460e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe07c6b0 0x5628fe459ee0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 --2- 172.21.15.27:0/1533920864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 0x5628fe4599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 >> 172.21.15.27:0/1533920864 conn(0x5628fe0914e0 msgr2=0x5628fe43d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:31.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb01ffb640 0 client.26635 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:31.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 shutdown_connections 2024-04-25T13:48:31.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 1 -- 172.21.15.27:0/1533920864 wait complete. 2024-04-25T13:48:31.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.696+0000 7fbb27e13740 20 client.26635 trim_cache size 0 max 16384 2024-04-25T13:48:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.721+0000 7fbb27e13740 1 Processor -- start 2024-04-25T13:48:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.721+0000 7fbb27e13740 1 -- start start 2024-04-25T13:48:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 0x5628fe458690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe460d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe09d390 0x5628fe4612a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb27e13740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c072fb0 con 0x5628fe0744d0 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb27e13740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c1a9620 con 0x5628fe07c6b0 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb27e13740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c073620 con 0x5628fe09d390 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe460d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe460d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:36224/0 (socket says 172.21.15.27:36224) 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb19d9b640 1 -- 172.21.15.27:0/805663913 learned_addr learned my addr 172.21.15.27:0/805663913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 0x5628fe458690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb1ad9d640 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe09d390 0x5628fe4612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb19d9b640 1 -- 172.21.15.27:0/805663913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe09d390 msgr2=0x5628fe4612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe09d390 0x5628fe4612a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb19d9b640 1 -- 172.21.15.27:0/805663913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 msgr2=0x5628fe458690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 0x5628fe458690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb19d9b640 1 -- 172.21.15.27:0/805663913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628fe0750f0 con 0x5628fe07c6b0 2024-04-25T13:48:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.722+0000 7fbb1ad9d640 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe09d390 0x5628fe4612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:48:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.723+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe460d60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000c010 tx=0x7fbb100107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd187a094f89861d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.723+0000 7fbb03fff640 1 -- 172.21.15.27:0/805663913 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10005850 con 0x5628fe07c6b0 2024-04-25T13:48:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.723+0000 7fbb03fff640 1 -- 172.21.15.27:0/805663913 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb10011620 con 0x5628fe07c6b0 2024-04-25T13:48:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.723+0000 7fbb03fff640 1 -- 172.21.15.27:0/805663913 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10004fb0 con 0x5628fe07c6b0 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.724+0000 7fbb27e13740 1 -- 172.21.15.27:0/805663913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 msgr2=0x5628fe460d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.724+0000 7fbb27e13740 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe460d60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000c010 tx=0x7fbb100107a0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.725+0000 7fbb27e13740 1 -- 172.21.15.27:0/805663913 shutdown_connections 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.725+0000 7fbb27e13740 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe09d390 0x5628fe4612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.725+0000 7fbb27e13740 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe460d60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.725+0000 7fbb27e13740 1 --2- 172.21.15.27:0/805663913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe0744d0 0x5628fe458690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.725+0000 7fbb27e13740 1 -- 172.21.15.27:0/805663913 >> 172.21.15.27:0/805663913 conn(0x5628fe0914e0 msgr2=0x5628fe457f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.726+0000 7fbb27e13740 1 -- 172.21.15.27:0/805663913 shutdown_connections 2024-04-25T13:48:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.726+0000 7fbb27e13740 1 -- 172.21.15.27:0/805663913 wait complete. 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.726+0000 7fbb27e13740 1 -- start start 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.726+0000 7fbb27e13740 1 Processor -- start 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.726+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe461610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe45e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb27e13740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe09d390 0x5628fe45e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe45e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe461610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb27e13740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c073cf0 con 0x5628fe09d390 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe09d390 0x5628fe45e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe461610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57578/0 (socket says 172.21.15.27:57578) 2024-04-25T13:48:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1a59c640 1 -- 172.21.15.27:0/3737495643 learned_addr learned my addr 172.21.15.27:0/3737495643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe45e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:36226/0 (socket says 172.21.15.27:36226) 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c1a8f50 con 0x5628fe07c6b0 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c0743c0 con 0x5628fe0744d0 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb27e13740 10 client.0 _ll_register_callbacks cb 0x7fbb0c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1a59c640 1 -- 172.21.15.27:0/3737495643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 msgr2=0x5628fe45e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe45e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1a59c640 1 -- 172.21.15.27:0/3737495643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe09d390 msgr2=0x5628fe45e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe09d390 0x5628fe45e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb1a59c640 1 -- 172.21.15.27:0/3737495643 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628fe3644b0 con 0x5628fe0744d0 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.727+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe45e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb1a59c640 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe461610 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c009250 tx=0x7fbb0c0a6a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc29e18472855fb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb01ffb640 1 -- 172.21.15.27:0/3737495643 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c0afdf0 con 0x5628fe0744d0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb01ffb640 1 -- 172.21.15.27:0/3737495643 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbb0c0a32d0 con 0x5628fe0744d0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb10011620 con 0x5628fe0744d0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb01ffb640 1 -- 172.21.15.27:0/3737495643 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fbb0c00b630 con 0x5628fe0744d0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb27e13740 10 client.26641 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5628fe4ce3f0 con 0x5628fe0744d0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.728+0000 7fbb01ffb640 1 -- 172.21.15.27:0/3737495643 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb10011620 con 0x5628fe0744d0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb01ffb640 1 -- 172.21.15.27:0/3737495643 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fbb0c0b6ce0 con 0x5628fe0744d0 2024-04-25T13:48:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb27e13740 10 client.26641 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb27e13740 10 client.26641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb27e13740 20 client.26641 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb27e13740 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628fe0313a0 0x5628fe4d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb27e13740 4 client.26641 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5628fe3ab800 con 0x5628fe0313a0 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628fe0313a0 0x5628fe4d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb19d9b640 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628fe0313a0 0x5628fe4d0e10 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=324f0ab9fd56feab server_cookie=366a0127f9401fd in_seq=0 out_seq=0 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.729+0000 7fbb01ffb640 10 client.26641 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb01ffb640 1 -- 172.21.15.27:0/3737495643 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5628fe3ab800 con 0x5628fe0313a0 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb01ffb640 10 client.26641 handle_command_reply: tid=0 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 client.26641 shutdown 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 msgr2=0x5628fe461610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe461610 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c009250 tx=0x7fbb0c0a6a50 comp rx=0 tx=0).stop 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 shutdown_connections 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628fe0313a0 0x5628fe4d0e10 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb01ffb640 0 client.26641 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628fe09d390 0x5628fe45e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628fe07c6b0 0x5628fe45e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 --2- 172.21.15.27:0/3737495643 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628fe0744d0 0x5628fe461610 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 >> 172.21.15.27:0/3737495643 conn(0x5628fe0914e0 msgr2=0x5628fe439c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 shutdown_connections 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 1 -- 172.21.15.27:0/3737495643 wait complete. 2024-04-25T13:48:31.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.730+0000 7fbb27e13740 20 client.26641 trim_cache size 0 max 16384 2024-04-25T13:48:31.737 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:48:31.737 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:48:31.737 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:48:31.737 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 -- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb040afcc0 msgr2=0x7fbb040b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb040afcc0 0x7fbb040b21d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fbb14005eb0 tx=0x7fbb14005d40 comp rx=0 tx=0).stop 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 -- 172.21.15.27:0/1969321846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c101e30 msgr2=0x7fbb1c1a05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c101e30 0x7fbb1c1a05d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00c3a0 tx=0x7fbb0c00c860 comp rx=0 tx=0).stop 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 -- 172.21.15.27:0/1969321846 shutdown_connections 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbb040afcc0 0x7fbb040b21d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb1c109780 0x7fbb1c1a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb1c106c10 0x7fbb1c1a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 --2- 172.21.15.27:0/1969321846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1c101e30 0x7fbb1c1a05d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 -- 172.21.15.27:0/1969321846 >> 172.21.15.27:0/1969321846 conn(0x7fbb1c0fd340 msgr2=0x7fbb1c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:31.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 -- 172.21.15.27:0/1969321846 shutdown_connections 2024-04-25T13:48:31.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.731+0000 7fbb20f64640 1 -- 172.21.15.27:0/1969321846 wait complete. 2024-04-25T13:48:31.745 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:48:31.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:48:31.775 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:31 smithi027 ceph-mon[29140]: pgmap v578: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 MiB/s wr, 675 op/s 2024-04-25T13:48:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.918+0000 7f63f0e9f640 1 -- 172.21.15.27:0/4051588718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ec10a420 msgr2=0x7f63ec10add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.918+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/4051588718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ec10a420 0x7f63ec10add0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009860 tx=0x7f63d402a360 comp rx=0 tx=0).stop 2024-04-25T13:48:31.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.920+0000 7f63f0e9f640 1 -- 172.21.15.27:0/4051588718 shutdown_connections 2024-04-25T13:48:31.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.920+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/4051588718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ec110030 0x7f63ec110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.920+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/4051588718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ec10b310 0x7f63ec10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.920+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/4051588718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ec10a420 0x7f63ec10add0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.920+0000 7f63f0e9f640 1 -- 172.21.15.27:0/4051588718 >> 172.21.15.27:0/4051588718 conn(0x7f63ec0704e0 msgr2=0x7f63ec072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:31.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.923+0000 7f63f0e9f640 1 -- 172.21.15.27:0/4051588718 shutdown_connections 2024-04-25T13:48:31.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.924+0000 7f63f0e9f640 1 -- 172.21.15.27:0/4051588718 wait complete. 2024-04-25T13:48:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.925+0000 7f63f0e9f640 1 Processor -- start 2024-04-25T13:48:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.925+0000 7f63f0e9f640 1 -- start start 2024-04-25T13:48:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.925+0000 7f63f0e9f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ec10a420 0x7f63ec1a90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.925+0000 7f63f0e9f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ec10b310 0x7f63ec1a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.925+0000 7f63f0e9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ec110030 0x7f63ec1b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.925+0000 7f63f0e9f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec11e450 con 0x7f63ec10b310 2024-04-25T13:48:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.925+0000 7f63f0e9f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec11e2d0 con 0x7f63ec110030 2024-04-25T13:48:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.925+0000 7f63f0e9f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec11e5d0 con 0x7f63ec10a420 2024-04-25T13:48:31.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ea59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ec10a420 0x7f63ec1a90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ea59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ec10a420 0x7f63ec1a90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57606/0 (socket says 172.21.15.27:57606) 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ea59c640 1 -- 172.21.15.27:0/857566794 learned_addr learned my addr 172.21.15.27:0/857566794 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63e9d9b640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ec10b310 0x7f63ec1a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ead9d640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ec110030 0x7f63ec1b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ea59c640 1 -- 172.21.15.27:0/857566794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ec110030 msgr2=0x7f63ec1b0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ea59c640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ec110030 0x7f63ec1b0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ea59c640 1 -- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ec10b310 msgr2=0x7f63ec1a95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ea59c640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ec10b310 0x7f63ec1a95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:31.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.926+0000 7f63ea59c640 1 -- 172.21.15.27:0/857566794 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ec1b0bf0 con 0x7f63ec10a420 2024-04-25T13:48:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.927+0000 7f63ea59c640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ec10a420 0x7f63ec1a90b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009780 tx=0x7f63d4032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.927+0000 7f63e37fe640 1 -- 172.21.15.27:0/857566794 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4037050 con 0x7f63ec10a420 2024-04-25T13:48:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.927+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ec1b0e80 con 0x7f63ec10a420 2024-04-25T13:48:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.927+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ec1b12e0 con 0x7f63ec10a420 2024-04-25T13:48:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.927+0000 7f63e37fe640 1 -- 172.21.15.27:0/857566794 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d4038040 con 0x7f63ec10a420 2024-04-25T13:48:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.927+0000 7f63e37fe640 1 -- 172.21.15.27:0/857566794 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4003b90 con 0x7f63ec10a420 2024-04-25T13:48:31.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.930+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63cc0052c0 con 0x7f63ec10a420 2024-04-25T13:48:31.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.930+0000 7f63e37fe640 1 -- 172.21.15.27:0/857566794 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f63d4045030 con 0x7f63ec10a420 2024-04-25T13:48:31.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.931+0000 7f63e37fe640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63d80afbf0 0x7f63d80b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:31.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.932+0000 7f63e37fe640 1 -- 172.21.15.27:0/857566794 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f63d40fe8c0 con 0x7f63ec10a420 2024-04-25T13:48:31.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.932+0000 7f63e9d9b640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63d80afbf0 0x7f63d80b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:31.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.933+0000 7f63e9d9b640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63d80afbf0 0x7f63d80b2100 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f63ec1aa150 tx=0x7f63dc0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:31.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:31.939+0000 7f63e37fe640 1 -- 172.21.15.27:0/857566794 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63d40c7560 con 0x7f63ec10a420 2024-04-25T13:48:32.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:31 smithi122 ceph-mon[32843]: pgmap v578: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 MiB/s wr, 675 op/s 2024-04-25T13:48:32.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:31 smithi045 ceph-mon[32731]: pgmap v578: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 MiB/s wr, 675 op/s 2024-04-25T13:48:32.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.278+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63cc0055b0 con 0x7f63ec10a420 2024-04-25T13:48:32.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.282+0000 7f63e37fe640 1 -- 172.21.15.27:0/857566794 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f63d40cb1b0 con 0x7f63ec10a420 2024-04-25T13:48:32.282 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:48:32.282 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:48:32.282 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63d80afbf0 msgr2=0x7f63d80b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63d80afbf0 0x7f63d80b2100 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f63ec1aa150 tx=0x7f63dc0095c0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ec10a420 msgr2=0x7f63ec1a90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ec10a420 0x7f63ec1a90b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009780 tx=0x7f63d4032d50 comp rx=0 tx=0).stop 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 shutdown_connections 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63d80afbf0 0x7f63d80b2100 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63ec110030 0x7f63ec1b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63ec10b310 0x7f63ec1a95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 --2- 172.21.15.27:0/857566794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63ec10a420 0x7f63ec1a90b0 secure :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009780 tx=0x7f63d4032d50 comp rx=0 tx=0).stop 2024-04-25T13:48:32.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.286+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 >> 172.21.15.27:0/857566794 conn(0x7f63ec0704e0 msgr2=0x7f63ec10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:32.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.289+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 shutdown_connections 2024-04-25T13:48:32.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.290+0000 7f63f0e9f640 1 -- 172.21.15.27:0/857566794 wait complete. 2024-04-25T13:48:32.310 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:48:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.519+0000 7f894118d640 1 -- 172.21.15.27:0/605940708 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934036070 con 0x7f893c10b190 2024-04-25T13:48:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.520+0000 7f8943191640 1 -- 172.21.15.27:0/605940708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 msgr2=0x7f893c10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.520+0000 7f8943191640 1 --2- 172.21.15.27:0/605940708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 0x7f893c10b5a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8934009be0 tx=0x7f893402a6e0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.520+0000 7f8943191640 1 -- 172.21.15.27:0/605940708 shutdown_connections 2024-04-25T13:48:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.520+0000 7f8943191640 1 --2- 172.21.15.27:0/605940708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f893c10fe90 0x7f893c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.520+0000 7f8943191640 1 --2- 172.21.15.27:0/605940708 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 0x7f893c10b5a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.520+0000 7f8943191640 1 --2- 172.21.15.27:0/605940708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f893c10a340 0x7f893c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.520+0000 7f8943191640 1 -- 172.21.15.27:0/605940708 >> 172.21.15.27:0/605940708 conn(0x7f893c070320 msgr2=0x7f893c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.520+0000 7f8943191640 1 -- 172.21.15.27:0/605940708 shutdown_connections 2024-04-25T13:48:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.521+0000 7f8943191640 1 -- 172.21.15.27:0/605940708 wait complete. 2024-04-25T13:48:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f8943191640 1 Processor -- start 2024-04-25T13:48:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f8943191640 1 -- start start 2024-04-25T13:48:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f8943191640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f893c10a340 0x7f893c1a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f8943191640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 0x7f893c1a9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f8943191640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f893c10fe90 0x7f893c1b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f8943191640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f893c11e300 con 0x7f893c10a340 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f8943191640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f893c11e180 con 0x7f893c10fe90 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f8943191640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f893c11e480 con 0x7f893c10b190 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f894198e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 0x7f893c1a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f894198e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 0x7f893c1a9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57618/0 (socket says 172.21.15.27:57618) 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.522+0000 7f894218f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f893c10a340 0x7f893c1a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f894198e640 1 -- 172.21.15.27:0/3388713267 learned_addr learned my addr 172.21.15.27:0/3388713267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f8942990640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f893c10fe90 0x7f893c1b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f894198e640 1 -- 172.21.15.27:0/3388713267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f893c10fe90 msgr2=0x7f893c1b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f894198e640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f893c10fe90 0x7f893c1b03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f894198e640 1 -- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f893c10a340 msgr2=0x7f893c1a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f894198e640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f893c10a340 0x7f893c1a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f894198e640 1 -- 172.21.15.27:0/3388713267 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893c1b0a10 con 0x7f893c10b190 2024-04-25T13:48:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f894218f640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f893c10a340 0x7f893c1a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:48:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.523+0000 7f894198e640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 0x7f893c1a9410 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8934034040 tx=0x7f8934030650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.524+0000 7f89337fe640 1 -- 172.21.15.27:0/3388713267 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934041020 con 0x7f893c10b190 2024-04-25T13:48:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.524+0000 7f8943191640 1 -- 172.21.15.27:0/3388713267 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f893c1b0ca0 con 0x7f893c10b190 2024-04-25T13:48:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.524+0000 7f8943191640 1 -- 172.21.15.27:0/3388713267 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f893c1b1190 con 0x7f893c10b190 2024-04-25T13:48:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.524+0000 7f89337fe640 1 -- 172.21.15.27:0/3388713267 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89340318f0 con 0x7f893c10b190 2024-04-25T13:48:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.524+0000 7f89337fe640 1 -- 172.21.15.27:0/3388713267 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934036040 con 0x7f893c10b190 2024-04-25T13:48:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.527+0000 7f894b381740 1 Processor -- start 2024-04-25T13:48:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894b381740 1 -- start start 2024-04-25T13:48:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894b381740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997f5390 0x55f899b979e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894b381740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997cc4d0 0x55f899b97f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894b381740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997d46b0 0x55f899b98bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894b381740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f899afbd80 con 0x55f8997d46b0 2024-04-25T13:48:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894b381740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f899ab5a20 con 0x55f8997f5390 2024-04-25T13:48:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894b381740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f899af7d40 con 0x55f8997cc4d0 2024-04-25T13:48:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894218f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997cc4d0 0x55f899b97f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894218f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997cc4d0 0x55f899b97f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57634/0 (socket says 172.21.15.27:57634) 2024-04-25T13:48:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f894218f640 1 -- 172.21.15.27:0/628130316 learned_addr learned my addr 172.21.15.27:0/628130316 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.528+0000 7f8942990640 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997f5390 0x55f899b979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.529+0000 7f894218f640 1 -- 172.21.15.27:0/628130316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997f5390 msgr2=0x55f899b979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.529+0000 7f89337fe640 1 -- 172.21.15.27:0/3388713267 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8934044020 con 0x7f893c10b190 2024-04-25T13:48:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.529+0000 7f894218f640 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997f5390 0x55f899b979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.529+0000 7f894218f640 1 -- 172.21.15.27:0/628130316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997d46b0 msgr2=0x55f899b98bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:48:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.529+0000 7f894218f640 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997d46b0 0x55f899b98bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.529+0000 7f894218f640 1 -- 172.21.15.27:0/628130316 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f899ac0c30 con 0x55f8997cc4d0 2024-04-25T13:48:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.529+0000 7f8942990640 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997f5390 0x55f899b979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.530+0000 7f89337fe640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89280afc30 0x7f89280b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.530+0000 7f89337fe640 1 -- 172.21.15.27:0/3388713267 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f89340ff640 con 0x7f893c10b190 2024-04-25T13:48:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.531+0000 7f894218f640 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997cc4d0 0x55f899b97f60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f892c0077b0 tx=0x7f892c007c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=897aef6fa6119d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.531+0000 7f8942990640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89280afc30 0x7f89280b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.532+0000 7f8942990640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89280afc30 0x7f89280b2140 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x55f899b98a30 tx=0x7f893800b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.534+0000 7f8930bfe640 1 -- 172.21.15.27:0/628130316 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c00d950 con 0x55f8997cc4d0 2024-04-25T13:48:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.534+0000 7f8930bfe640 1 -- 172.21.15.27:0/628130316 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f892c018070 con 0x55f8997cc4d0 2024-04-25T13:48:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.534+0000 7f8930bfe640 1 -- 172.21.15.27:0/628130316 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c0123e0 con 0x55f8997cc4d0 2024-04-25T13:48:32.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.535+0000 7f894b381740 1 -- 172.21.15.27:0/628130316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997cc4d0 msgr2=0x55f899b97f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.535+0000 7f894b381740 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997cc4d0 0x55f899b97f60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f892c0077b0 tx=0x7f892c007c70 comp rx=0 tx=0).stop 2024-04-25T13:48:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.535+0000 7f894b381740 1 -- 172.21.15.27:0/628130316 shutdown_connections 2024-04-25T13:48:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.535+0000 7f894b381740 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997d46b0 0x55f899b98bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.535+0000 7f894b381740 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997cc4d0 0x55f899b97f60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.535+0000 7f894b381740 1 --2- 172.21.15.27:0/628130316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997f5390 0x55f899b979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.535+0000 7f894b381740 1 -- 172.21.15.27:0/628130316 >> 172.21.15.27:0/628130316 conn(0x55f8997e94e0 msgr2=0x55f899b931f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.536+0000 7f894b381740 1 -- 172.21.15.27:0/628130316 shutdown_connections 2024-04-25T13:48:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.536+0000 7f894b381740 1 -- 172.21.15.27:0/628130316 wait complete. 2024-04-25T13:48:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.536+0000 7f894b381740 1 -- start start 2024-04-25T13:48:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894b381740 1 Processor -- start 2024-04-25T13:48:32.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894b381740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997cc4d0 0x55f899ba1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894b381740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 0x55f899ba1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894b381740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997f5390 0x55f899ba8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894b381740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f899b99750 con 0x55f8997f5390 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894b381740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f899a82dc0 con 0x55f8997cc4d0 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894b381740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f899b998d0 con 0x55f8997d46b0 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894b381740 10 client.0 _ll_register_callbacks cb 0x55f899a9e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894218f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997cc4d0 0x55f899ba1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894218f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997cc4d0 0x55f899ba1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:36288/0 (socket says 172.21.15.27:36288) 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f894218f640 1 -- 172.21.15.27:0/575560205 learned_addr learned my addr 172.21.15.27:0/575560205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.537+0000 7f8942990640 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997f5390 0x55f899ba8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f894198e640 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 0x55f899ba1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f894218f640 1 -- 172.21.15.27:0/575560205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 msgr2=0x55f899ba1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f894218f640 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 0x55f899ba1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f894218f640 1 -- 172.21.15.27:0/575560205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997f5390 msgr2=0x55f899ba8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f894218f640 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997f5390 0x55f899ba8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f894218f640 1 -- 172.21.15.27:0/575560205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f89980d3f0 con 0x55f8997cc4d0 2024-04-25T13:48:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f894218f640 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997cc4d0 0x55f899ba1810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f892c006da0 tx=0x7f892c0136b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6bf40f39188191e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f8925ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:48:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f8925ffb640 1 -- 172.21.15.27:0/575560205 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c00dd60 con 0x55f8997cc4d0 2024-04-25T13:48:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f894b381740 1 -- 172.21.15.27:0/575560205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f892c018070 con 0x55f8997cc4d0 2024-04-25T13:48:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.538+0000 7f8925ffb640 1 -- 172.21.15.27:0/575560205 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f892c01c3f0 con 0x55f8997cc4d0 2024-04-25T13:48:32.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.539+0000 7f8925ffb640 1 -- 172.21.15.27:0/575560205 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f892c01c5f0 con 0x55f8997cc4d0 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.539+0000 7f894b381740 10 client.26612 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.539+0000 7f894b381740 1 -- 172.21.15.27:0/575560205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f8998d77d0 con 0x55f8997cc4d0 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.539+0000 7f8925ffb640 1 -- 172.21.15.27:0/575560205 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f892c018070 con 0x55f8997cc4d0 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.540+0000 7f8925ffb640 1 -- 172.21.15.27:0/575560205 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f892c02f660 con 0x55f8997cc4d0 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.540+0000 7f894b381740 10 client.26612 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.540+0000 7f894b381740 10 client.26612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.540+0000 7f894b381740 20 client.26612 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.540+0000 7f894b381740 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8997b3f60 0x55f899baf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.540+0000 7f894b381740 4 client.26612 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.540+0000 7f894b381740 1 -- 172.21.15.27:0/575560205 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f899b09990 con 0x55f8997b3f60 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.540+0000 7f894198e640 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8997b3f60 0x55f899baf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.541+0000 7f8925ffb640 10 client.26612 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.541+0000 7f894198e640 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8997b3f60 0x55f899baf240 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bbb5cd7e4abbc89 server_cookie=87fdb5d52364e59b in_seq=0 out_seq=0 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.541+0000 7f8925ffb640 1 -- 172.21.15.27:0/575560205 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f899b09990 con 0x55f8997b3f60 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.541+0000 7f8925ffb640 10 client.26612 handle_command_reply: tid=0 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.542+0000 7f894b381740 1 client.26612 shutdown 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 -- 172.21.15.27:0/575560205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997cc4d0 msgr2=0x55f899ba1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997cc4d0 0x55f899ba1810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f892c006da0 tx=0x7f892c0136b0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 -- 172.21.15.27:0/575560205 shutdown_connections 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8997b3f60 0x55f899baf240 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997f5390 0x55f899ba8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 0x55f899ba1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 --2- 172.21.15.27:0/575560205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997cc4d0 0x55f899ba1810 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 -- 172.21.15.27:0/575560205 >> 172.21.15.27:0/575560205 conn(0x55f8997e94e0 msgr2=0x55f899b96100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f8925ffb640 0 client.26612 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 -- 172.21.15.27:0/575560205 shutdown_connections 2024-04-25T13:48:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 1 -- 172.21.15.27:0/575560205 wait complete. 2024-04-25T13:48:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.543+0000 7f894b381740 20 client.26612 trim_cache size 0 max 16384 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894b381740 1 Processor -- start 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894b381740 1 -- start start 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894b381740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997e94e0 0x55f899b93330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894b381740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894b381740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997d46b0 0x55f899ba13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894b381740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f892000c370 con 0x55f8997cc4d0 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894b381740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f892000ebf0 con 0x55f8997d46b0 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894b381740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f892000c9e0 con 0x55f8997e94e0 2024-04-25T13:48:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f8942990640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997d46b0 0x55f899ba13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894218f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997e94e0 0x55f899b93330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f894198e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.561+0000 7f8942990640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997d46b0 0x55f899ba13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:36290/0 (socket says 172.21.15.27:36290) 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f8942990640 1 -- 172.21.15.27:0/1257590379 learned_addr learned my addr 172.21.15.27:0/1257590379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f894218f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997e94e0 0x55f899b93330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:57644/0 (socket says 172.21.15.27:57644) 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f894218f640 1 -- 172.21.15.27:0/1257590379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997d46b0 msgr2=0x55f899ba13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f894218f640 1 --2- 172.21.15.27:0/1257590379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997d46b0 0x55f899ba13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f894218f640 1 -- 172.21.15.27:0/1257590379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 msgr2=0x55f899ba8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f894218f640 1 --2- 172.21.15.27:0/1257590379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f894218f640 1 -- 172.21.15.27:0/1257590379 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f899b9ce70 con 0x55f8997e94e0 2024-04-25T13:48:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f894218f640 1 --2- 172.21.15.27:0/1257590379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997e94e0 0x55f899b93330 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f892c006e10 tx=0x7f892c007240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df0d4884c5f67abb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:32.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f8927fff640 1 -- 172.21.15.27:0/1257590379 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c02fc70 con 0x55f8997e94e0 2024-04-25T13:48:32.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f8927fff640 1 -- 172.21.15.27:0/1257590379 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f892c018040 con 0x55f8997e94e0 2024-04-25T13:48:32.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.562+0000 7f8927fff640 1 -- 172.21.15.27:0/1257590379 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c00ddc0 con 0x55f8997e94e0 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 -- 172.21.15.27:0/1257590379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997e94e0 msgr2=0x55f899b93330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 --2- 172.21.15.27:0/1257590379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997e94e0 0x55f899b93330 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f892c006e10 tx=0x7f892c007240 comp rx=0 tx=0).stop 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 -- 172.21.15.27:0/1257590379 shutdown_connections 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 --2- 172.21.15.27:0/1257590379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997d46b0 0x55f899ba13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 --2- 172.21.15.27:0/1257590379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 --2- 172.21.15.27:0/1257590379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997e94e0 0x55f899b93330 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 -- 172.21.15.27:0/1257590379 >> 172.21.15.27:0/1257590379 conn(0x55f899b8e4b0 msgr2=0x55f899b92c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 -- 172.21.15.27:0/1257590379 shutdown_connections 2024-04-25T13:48:32.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.563+0000 7f894b381740 1 -- 172.21.15.27:0/1257590379 wait complete. 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.564+0000 7f894b381740 1 -- start start 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.564+0000 7f894b381740 1 Processor -- start 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.564+0000 7f894b381740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.564+0000 7f894b381740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 0x55f899ba56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.564+0000 7f894b381740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997e94e0 0x55f899ba5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.564+0000 7f894b381740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f892000d0b0 con 0x55f8997cc4d0 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.564+0000 7f894b381740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f892000e520 con 0x55f8997e94e0 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.564+0000 7f894b381740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f892000d780 con 0x55f8997d46b0 2024-04-25T13:48:32.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894b381740 10 client.0 _ll_register_callbacks cb 0x7f893800b6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39838/0 (socket says 172.21.15.27:39838) 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 -- 172.21.15.27:0/2104613473 learned_addr learned my addr 172.21.15.27:0/2104613473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894198e640 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 0x55f899ba56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f8942990640 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997e94e0 0x55f899ba5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 -- 172.21.15.27:0/2104613473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 msgr2=0x55f899ba56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 0x55f899ba56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 -- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997e94e0 msgr2=0x55f899ba5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997e94e0 0x55f899ba5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 -- 172.21.15.27:0/2104613473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f899ba9110 con 0x55f8997cc4d0 2024-04-25T13:48:32.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f8942990640 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997e94e0 0x55f899ba5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894218f640 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8b60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f892c02fa80 tx=0x7f892c0070b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39098c4c5ff85b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f8925ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f8925ffb640 1 -- 172.21.15.27:0/2104613473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c03e030 con 0x55f8997cc4d0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f8925ffb640 1 -- 172.21.15.27:0/2104613473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f892c01c3f0 con 0x55f8997cc4d0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.565+0000 7f894b381740 1 -- 172.21.15.27:0/2104613473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f892c018040 con 0x55f8997cc4d0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f8925ffb640 1 -- 172.21.15.27:0/2104613473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f892c007630 con 0x55f8997cc4d0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894b381740 10 client.16902 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894b381740 1 -- 172.21.15.27:0/2104613473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f899c165e0 con 0x55f8997cc4d0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f8925ffb640 1 -- 172.21.15.27:0/2104613473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f892c018040 con 0x55f8997cc4d0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f8925ffb640 1 -- 172.21.15.27:0/2104613473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f892c01ccb0 con 0x55f8997cc4d0 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894b381740 10 client.16902 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:32.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894b381740 10 client.16902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894b381740 20 client.16902 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894b381740 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8997f5390 0x55f899c18fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894b381740 4 client.16902 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894b381740 1 -- 172.21.15.27:0/2104613473 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f899b03800 con 0x55f8997f5390 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.566+0000 7f894198e640 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8997f5390 0x55f899c18fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.567+0000 7f894198e640 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8997f5390 0x55f899c18fc0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b3440839a1f964f server_cookie=90c1874060297a2c in_seq=0 out_seq=0 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.567+0000 7f8925ffb640 10 client.16902 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.567+0000 7f8925ffb640 1 -- 172.21.15.27:0/2104613473 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f899b03800 con 0x55f8997f5390 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.567+0000 7f8925ffb640 10 client.16902 handle_command_reply: tid=0 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.568+0000 7f894b381740 1 client.16902 shutdown 2024-04-25T13:48:32.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.568+0000 7f894b381740 1 -- 172.21.15.27:0/2104613473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 msgr2=0x55f899ba8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.568+0000 7f894b381740 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8b60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f892c02fa80 tx=0x7f892c0070b0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.569+0000 7f894b381740 1 -- 172.21.15.27:0/2104613473 shutdown_connections 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.569+0000 7f894b381740 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8997f5390 0x55f899c18fc0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.569+0000 7f894b381740 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8997e94e0 0x55f899ba5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.569+0000 7f894b381740 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8997d46b0 0x55f899ba56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.569+0000 7f894b381740 1 --2- 172.21.15.27:0/2104613473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8997cc4d0 0x55f899ba8b60 secure :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f892c02fa80 tx=0x7f892c0070b0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.569+0000 7f894b381740 1 -- 172.21.15.27:0/2104613473 >> 172.21.15.27:0/2104613473 conn(0x55f899b8e4b0 msgr2=0x55f899b95ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.569+0000 7f8925ffb640 0 client.16902 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.570+0000 7f894b381740 1 -- 172.21.15.27:0/2104613473 shutdown_connections 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.570+0000 7f894b381740 1 -- 172.21.15.27:0/2104613473 wait complete. 2024-04-25T13:48:32.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.570+0000 7f894b381740 20 client.16902 trim_cache size 0 max 16384 2024-04-25T13:48:32.576 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:48:32.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 -- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89280afc30 msgr2=0x7f89280b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89280afc30 0x7f89280b2140 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x55f899b98a30 tx=0x7f893800b2e0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 -- 172.21.15.27:0/3388713267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 msgr2=0x7f893c1a9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 0x7f893c1a9410 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8934034040 tx=0x7f8934030650 comp rx=0 tx=0).stop 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 -- 172.21.15.27:0/3388713267 shutdown_connections 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89280afc30 0x7f89280b2140 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f893c10fe90 0x7f893c1b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f893c10b190 0x7f893c1a9410 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 --2- 172.21.15.27:0/3388713267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f893c10a340 0x7f893c1a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 -- 172.21.15.27:0/3388713267 >> 172.21.15.27:0/3388713267 conn(0x7f893c070320 msgr2=0x7f893c10be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 -- 172.21.15.27:0/3388713267 shutdown_connections 2024-04-25T13:48:32.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:32.571+0000 7f8943191640 1 -- 172.21.15.27:0/3388713267 wait complete. 2024-04-25T13:48:32.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:32 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/857566794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:48:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 445.0 s, 1789.0 tps, lat 56.038 ms stddev 61.118 2024-04-25T13:48:33.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:32 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/857566794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:48:33.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:32 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/857566794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:48:33.584 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:48:33.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:48:33.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.737+0000 7ff66b41a640 1 -- 172.21.15.27:0/3462963234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664103f50 msgr2=0x7ff664102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.737+0000 7ff66b41a640 1 --2- 172.21.15.27:0/3462963234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664103f50 0x7ff664102840 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff660009860 tx=0x7ff66002a340 comp rx=0 tx=0).stop 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.737+0000 7ff66b41a640 1 -- 172.21.15.27:0/3462963234 shutdown_connections 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.737+0000 7ff66b41a640 1 --2- 172.21.15.27:0/3462963234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff664108a90 0x7ff66410aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.737+0000 7ff66b41a640 1 --2- 172.21.15.27:0/3462963234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6641043f0 0x7ff664102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.737+0000 7ff66b41a640 1 --2- 172.21.15.27:0/3462963234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664103f50 0x7ff664102840 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.737+0000 7ff66b41a640 1 -- 172.21.15.27:0/3462963234 >> 172.21.15.27:0/3462963234 conn(0x7ff6640fd340 msgr2=0x7ff6640ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.738+0000 7ff66b41a640 1 -- 172.21.15.27:0/3462963234 shutdown_connections 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.738+0000 7ff66b41a640 1 -- 172.21.15.27:0/3462963234 wait complete. 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.738+0000 7ff66b41a640 1 Processor -- start 2024-04-25T13:48:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.738+0000 7ff66b41a640 1 -- start start 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff66b41a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff664103f50 0x7ff6640ff480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff66b41a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6641043f0 0x7ff6640fdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff66b41a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664108a90 0x7ff6640fe060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff66b41a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff66410e500 con 0x7ff664108a90 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff66b41a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff66410e380 con 0x7ff6641043f0 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff66b41a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff66410e680 con 0x7ff664103f50 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6699b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664108a90 0x7ff6640fe060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6699b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664108a90 0x7ff6640fe060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43186/0 (socket says 172.21.15.27:43186) 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6699b7640 1 -- 172.21.15.27:0/4250070714 learned_addr learned my addr 172.21.15.27:0/4250070714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6689b5640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6641043f0 0x7ff6640fdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6691b6640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff664103f50 0x7ff6640ff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6691b6640 1 -- 172.21.15.27:0/4250070714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6641043f0 msgr2=0x7ff6640fdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6691b6640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6641043f0 0x7ff6640fdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6691b6640 1 -- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664108a90 msgr2=0x7ff6640fe060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6691b6640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664108a90 0x7ff6640fe060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.739+0000 7ff6691b6640 1 -- 172.21.15.27:0/4250070714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6640fe790 con 0x7ff664103f50 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.740+0000 7ff6699b7640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664108a90 0x7ff6640fe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.740+0000 7ff6691b6640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff664103f50 0x7ff6640ff480 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff66002a860 tx=0x7ff660032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.741+0000 7ff65a7fc640 1 -- 172.21.15.27:0/4250070714 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660040020 con 0x7ff664103f50 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.741+0000 7ff65a7fc640 1 -- 172.21.15.27:0/4250070714 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff660030e10 con 0x7ff664103f50 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.741+0000 7ff65a7fc640 1 -- 172.21.15.27:0/4250070714 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660037040 con 0x7ff664103f50 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.741+0000 7ff66b41a640 1 -- 172.21.15.27:0/4250070714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6641a42b0 con 0x7ff664103f50 2024-04-25T13:48:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.741+0000 7ff66b41a640 1 -- 172.21.15.27:0/4250070714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6641a4660 con 0x7ff664103f50 2024-04-25T13:48:33.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff65a7fc640 1 -- 172.21.15.27:0/4250070714 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff660043020 con 0x7ff664103f50 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff6723a1740 1 Processor -- start 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff6723a1740 1 -- start start 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d5303410 0x55b3d56a44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52e2730 0x55b3d56ab920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff6723a1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b3d5609b00 con 0x55b3d52da550 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff6723a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b3d55a5920 con 0x55b3d52e2730 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.744+0000 7ff6723a1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b3d5605ac0 con 0x55b3d5303410 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6691b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6691b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56a4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43192/0 (socket says 172.21.15.27:43192) 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6691b6640 1 -- 172.21.15.27:0/3507340039 learned_addr learned my addr 172.21.15.27:0/3507340039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6689b5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52e2730 0x55b3d56ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6699b7640 1 --2- 172.21.15.27:0/3507340039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d5303410 0x55b3d56a44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff65a7fc640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6500afc40 0x7ff6500b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6689b5640 1 -- 172.21.15.27:0/3507340039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d5303410 msgr2=0x55b3d56a44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6689b5640 1 --2- 172.21.15.27:0/3507340039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d5303410 0x55b3d56a44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6689b5640 1 -- 172.21.15.27:0/3507340039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 msgr2=0x55b3d56a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6689b5640 1 --2- 172.21.15.27:0/3507340039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6689b5640 1 -- 172.21.15.27:0/3507340039 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d53e37e0 con 0x55b3d52e2730 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff65a7fc640 1 -- 172.21.15.27:0/4250070714 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff6600fe8c0 con 0x7ff664103f50 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6699b7640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6500afc40 0x7ff6500b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff6689b5640 1 --2- 172.21.15.27:0/3507340039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52e2730 0x55b3d56ab920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff654009530 tx=0x7ff654011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca1f3616e5b6c777 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.745+0000 7ff64f7fe640 1 -- 172.21.15.27:0/3507340039 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654004220 con 0x55b3d52e2730 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff64f7fe640 1 -- 172.21.15.27:0/3507340039 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6540043c0 con 0x55b3d52e2730 2024-04-25T13:48:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff64f7fe640 1 -- 172.21.15.27:0/3507340039 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654019630 con 0x55b3d52e2730 2024-04-25T13:48:33.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff6723a1740 1 -- 172.21.15.27:0/3507340039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52e2730 msgr2=0x55b3d56ab920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff6723a1740 1 --2- 172.21.15.27:0/3507340039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52e2730 0x55b3d56ab920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff654009530 tx=0x7ff654011810 comp rx=0 tx=0).stop 2024-04-25T13:48:33.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff6699b7640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6500afc40 0x7ff6500b2150 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x55b3d56a5670 tx=0x7ff65c004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff6723a1740 1 -- 172.21.15.27:0/3507340039 shutdown_connections 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff6723a1740 1 --2- 172.21.15.27:0/3507340039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52e2730 0x55b3d56ab920 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff6723a1740 1 --2- 172.21.15.27:0/3507340039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff6723a1740 1 --2- 172.21.15.27:0/3507340039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d5303410 0x55b3d56a44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.746+0000 7ff6723a1740 1 -- 172.21.15.27:0/3507340039 >> 172.21.15.27:0/3507340039 conn(0x55b3d52f7560 msgr2=0x55b3d569fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.748+0000 7ff6723a1740 1 -- 172.21.15.27:0/3507340039 shutdown_connections 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.748+0000 7ff6723a1740 1 -- 172.21.15.27:0/3507340039 wait complete. 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.748+0000 7ff6723a1740 1 -- start start 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.748+0000 7ff6723a1740 1 Processor -- start 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52e2730 0x55b3d56ae820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d5303410 0x55b3d56b5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6723a1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b3d56ac410 con 0x55b3d5303410 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6723a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b3d55a35d0 con 0x55b3d52da550 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6723a1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b3d56ac590 con 0x55b3d52e2730 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6723a1740 10 client.0 _ll_register_callbacks cb 0x55b3d5591080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6691b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6699b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d5303410 0x55b3d56b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6691b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52112/0 (socket says 172.21.15.27:52112) 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6691b6640 1 -- 172.21.15.27:0/4254126366 learned_addr learned my addr 172.21.15.27:0/4254126366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6689b5640 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52e2730 0x55b3d56ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6691b6640 1 -- 172.21.15.27:0/4254126366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52e2730 msgr2=0x55b3d56ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6691b6640 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52e2730 0x55b3d56ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6691b6640 1 -- 172.21.15.27:0/4254126366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d5303410 msgr2=0x55b3d56b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6691b6640 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d5303410 0x55b3d56b5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.749+0000 7ff6691b6640 1 -- 172.21.15.27:0/4254126366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3d5443f70 con 0x55b3d52da550 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff6691b6640 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56ae2e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff6600c40c0 tx=0x7ff6600042a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=922b1f9651a3867b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff64cff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff64cff9640 1 -- 172.21.15.27:0/4254126366 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600c6c20 con 0x55b3d52da550 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff64cff9640 1 -- 172.21.15.27:0/4254126366 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff6600c60b0 con 0x55b3d52da550 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff6723a1740 1 -- 172.21.15.27:0/4254126366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6540043c0 con 0x55b3d52da550 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff64cff9640 1 -- 172.21.15.27:0/4254126366 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff6600c65c0 con 0x55b3d52da550 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff6723a1740 10 client.26627 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff6723a1740 1 -- 172.21.15.27:0/4254126366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b3d52db170 con 0x55b3d52da550 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff64cff9640 1 -- 172.21.15.27:0/4254126366 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6540043c0 con 0x55b3d52da550 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff64cff9640 1 -- 172.21.15.27:0/4254126366 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff6600de030 con 0x55b3d52da550 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.750+0000 7ff6723a1740 10 client.26627 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.751+0000 7ff6723a1740 10 client.26627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.751+0000 7ff6723a1740 20 client.26627 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.751+0000 7ff6723a1740 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3d52c21f0 0x55b3d56bbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.751+0000 7ff6723a1740 4 client.26627 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.751+0000 7ff6723a1740 1 -- 172.21.15.27:0/4254126366 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3d5617710 con 0x55b3d52c21f0 2024-04-25T13:48:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.751+0000 7ff6689b5640 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3d52c21f0 0x55b3d56bbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.752+0000 7ff64cff9640 10 client.26627 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.752+0000 7ff6689b5640 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3d52c21f0 0x55b3d56bbab0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd5fc49e3f4fc75a server_cookie=19ea363fd776cabc in_seq=0 out_seq=0 2024-04-25T13:48:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.753+0000 7ff64cff9640 1 -- 172.21.15.27:0/4254126366 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b3d5617710 con 0x55b3d52c21f0 2024-04-25T13:48:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.753+0000 7ff64cff9640 10 client.26627 handle_command_reply: tid=0 2024-04-25T13:48:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.753+0000 7ff6723a1740 1 client.26627 shutdown 2024-04-25T13:48:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.753+0000 7ff6723a1740 1 -- 172.21.15.27:0/4254126366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 msgr2=0x55b3d56ae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.753+0000 7ff6723a1740 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56ae2e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff6600c40c0 tx=0x7ff6600042a0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 1 -- 172.21.15.27:0/4254126366 shutdown_connections 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3d52c21f0 0x55b3d56bbab0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff64cff9640 0 client.26627 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d5303410 0x55b3d56b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52e2730 0x55b3d56ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 1 --2- 172.21.15.27:0/4254126366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56ae2e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 1 -- 172.21.15.27:0/4254126366 >> 172.21.15.27:0/4254126366 conn(0x55b3d52f7560 msgr2=0x55b3d569eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 1 -- 172.21.15.27:0/4254126366 shutdown_connections 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 1 -- 172.21.15.27:0/4254126366 wait complete. 2024-04-25T13:48:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.754+0000 7ff6723a1740 20 client.26627 trim_cache size 0 max 16384 2024-04-25T13:48:33.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.795+0000 7ff6723a1740 1 Processor -- start 2024-04-25T13:48:33.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.796+0000 7ff6723a1740 1 -- start start 2024-04-25T13:48:33.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.797+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 0x55b3d56c0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.797+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52c21f0 0x55b3d56c0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.797+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.797+0000 7ff6723a1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6641ab310 con 0x55b3d52da550 2024-04-25T13:48:33.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.797+0000 7ff6723a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6641adb90 con 0x55b3d52c21f0 2024-04-25T13:48:33.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.797+0000 7ff6723a1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff6641ab980 con 0x55b3d52f7560 2024-04-25T13:48:33.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.801+0000 7ff6699b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.801+0000 7ff6699b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56c0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43210/0 (socket says 172.21.15.27:43210) 2024-04-25T13:48:33.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.801+0000 7ff6699b7640 1 -- 172.21.15.27:0/227066369 learned_addr learned my addr 172.21.15.27:0/227066369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:33.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.802+0000 7ff6691b6640 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 0x55b3d56c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.803+0000 7ff6689b5640 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52c21f0 0x55b3d56c0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.806+0000 7ff6689b5640 1 -- 172.21.15.27:0/227066369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 msgr2=0x55b3d56c0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:33.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.806+0000 7ff6689b5640 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 0x55b3d56c0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.806+0000 7ff6689b5640 1 -- 172.21.15.27:0/227066369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 msgr2=0x55b3d56c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:33.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.806+0000 7ff6689b5640 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.806+0000 7ff6689b5640 1 -- 172.21.15.27:0/227066369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d56a90f0 con 0x55b3d52c21f0 2024-04-25T13:48:33.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.809+0000 7ff6689b5640 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52c21f0 0x55b3d56c0740 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff65401b040 tx=0x7ff654005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada497ef6b563909 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:33.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.812+0000 7ff64effd640 1 -- 172.21.15.27:0/227066369 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654004180 con 0x55b3d52c21f0 2024-04-25T13:48:33.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.812+0000 7ff64effd640 1 -- 172.21.15.27:0/227066369 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6540054f0 con 0x55b3d52c21f0 2024-04-25T13:48:33.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.812+0000 7ff64effd640 1 -- 172.21.15.27:0/227066369 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff654003980 con 0x55b3d52c21f0 2024-04-25T13:48:33.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.813+0000 7ff6723a1740 1 -- 172.21.15.27:0/227066369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52c21f0 msgr2=0x55b3d56c0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.813+0000 7ff6723a1740 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52c21f0 0x55b3d56c0740 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff65401b040 tx=0x7ff654005a20 comp rx=0 tx=0).stop 2024-04-25T13:48:33.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.814+0000 7ff6723a1740 1 -- 172.21.15.27:0/227066369 shutdown_connections 2024-04-25T13:48:33.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.814+0000 7ff6723a1740 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52da550 0x55b3d56c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.814+0000 7ff6723a1740 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52c21f0 0x55b3d56c0740 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.814+0000 7ff6723a1740 1 --2- 172.21.15.27:0/227066369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 0x55b3d56c0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.814+0000 7ff6723a1740 1 -- 172.21.15.27:0/227066369 >> 172.21.15.27:0/227066369 conn(0x55b3d56bbda0 msgr2=0x55b3d56ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:33.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.821+0000 7ff6723a1740 1 -- 172.21.15.27:0/227066369 shutdown_connections 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.822+0000 7ff6723a1740 1 -- 172.21.15.27:0/227066369 wait complete. 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.822+0000 7ff6723a1740 1 -- start start 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.822+0000 7ff6723a1740 1 Processor -- start 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.825+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52c21f0 0x55b3d56b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.825+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.825+0000 7ff6723a1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 0x55b3d56b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.825+0000 7ff6723a1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff6641ac050 con 0x55b3d52c21f0 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.825+0000 7ff6723a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6641ad4c0 con 0x55b3d52da550 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.825+0000 7ff6723a1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff6641ac720 con 0x55b3d52f7560 2024-04-25T13:48:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.825+0000 7ff6723a1740 10 client.0 _ll_register_callbacks cb 0x7ff6600097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.826+0000 7ff6691b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52c21f0 0x55b3d56b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.826+0000 7ff6691b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52c21f0 0x55b3d56b29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43216/0 (socket says 172.21.15.27:43216) 2024-04-25T13:48:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.826+0000 7ff6691b6640 1 -- 172.21.15.27:0/2903071657 learned_addr learned my addr 172.21.15.27:0/2903071657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.827+0000 7ff6691b6640 1 -- 172.21.15.27:0/2903071657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 msgr2=0x55b3d56b3e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:48:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.827+0000 7ff6691b6640 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 0x55b3d56b3e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.827+0000 7ff6691b6640 1 -- 172.21.15.27:0/2903071657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 msgr2=0x55b3d56b2f10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.827+0000 7ff6691b6640 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56b2f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.827+0000 7ff6691b6640 1 -- 172.21.15.27:0/2903071657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3d56b4400 con 0x55b3d52c21f0 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.827+0000 7ff6691b6640 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52c21f0 0x55b3d56b29d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff6600c33f0 tx=0x7ff6600c27c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0361ae01f682d48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.827+0000 7ff64cff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.827+0000 7ff64cff9640 1 -- 172.21.15.27:0/2903071657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600c67a0 con 0x55b3d52c21f0 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.828+0000 7ff6723a1740 1 -- 172.21.15.27:0/2903071657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6540054f0 con 0x55b3d52c21f0 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.828+0000 7ff64cff9640 1 -- 172.21.15.27:0/2903071657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff660030750 con 0x55b3d52c21f0 2024-04-25T13:48:33.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.828+0000 7ff64cff9640 1 -- 172.21.15.27:0/2903071657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff6600d7050 con 0x55b3d52c21f0 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.828+0000 7ff6723a1740 10 client.16920 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.828+0000 7ff6723a1740 1 -- 172.21.15.27:0/2903071657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b3d56b4400 con 0x55b3d52c21f0 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.829+0000 7ff64cff9640 1 -- 172.21.15.27:0/2903071657 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6540054f0 con 0x55b3d52c21f0 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.829+0000 7ff64cff9640 1 -- 172.21.15.27:0/2903071657 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff6600de070 con 0x55b3d52c21f0 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.829+0000 7ff6723a1740 10 client.16920 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.829+0000 7ff6723a1740 10 client.16920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.829+0000 7ff6723a1740 20 client.16920 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.829+0000 7ff6723a1740 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3d52e2730 0x55b3d5724870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.829+0000 7ff6723a1740 4 client.16920 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:48:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.829+0000 7ff6723a1740 1 -- 172.21.15.27:0/2903071657 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b3d5611580 con 0x55b3d52e2730 2024-04-25T13:48:33.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.834+0000 7ff6689b5640 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3d52e2730 0x55b3d5724870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:33.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.835+0000 7ff6689b5640 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3d52e2730 0x55b3d5724870 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78e185285855fb72 server_cookie=9fabab30803bbcd8 in_seq=0 out_seq=0 2024-04-25T13:48:33.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.835+0000 7ff64cff9640 10 client.16920 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:33.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.837+0000 7ff64cff9640 1 -- 172.21.15.27:0/2903071657 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b3d5611580 con 0x55b3d52e2730 2024-04-25T13:48:33.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.837+0000 7ff64cff9640 10 client.16920 handle_command_reply: tid=0 2024-04-25T13:48:33.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.837+0000 7ff6723a1740 1 client.16920 shutdown 2024-04-25T13:48:33.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.839+0000 7ff6723a1740 1 -- 172.21.15.27:0/2903071657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52c21f0 msgr2=0x55b3d56b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.839+0000 7ff6723a1740 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52c21f0 0x55b3d56b29d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff6600c33f0 tx=0x7ff6600c27c0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.843+0000 7ff6723a1740 1 -- 172.21.15.27:0/2903071657 shutdown_connections 2024-04-25T13:48:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.843+0000 7ff6723a1740 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b3d52e2730 0x55b3d5724870 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.843+0000 7ff6723a1740 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b3d52f7560 0x55b3d56b3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.843+0000 7ff6723a1740 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b3d52da550 0x55b3d56b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.843+0000 7ff6723a1740 1 --2- 172.21.15.27:0/2903071657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b3d52c21f0 0x55b3d56b29d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.843+0000 7ff6723a1740 1 -- 172.21.15.27:0/2903071657 >> 172.21.15.27:0/2903071657 conn(0x55b3d56bbda0 msgr2=0x55b3d56a2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.843+0000 7ff64cff9640 0 client.16920 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.844+0000 7ff6723a1740 1 -- 172.21.15.27:0/2903071657 shutdown_connections 2024-04-25T13:48:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.844+0000 7ff6723a1740 1 -- 172.21.15.27:0/2903071657 wait complete. 2024-04-25T13:48:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.844+0000 7ff6723a1740 20 client.16920 trim_cache size 0 max 16384 2024-04-25T13:48:33.847 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:48:33.847 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:48:33.847 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "5cbccd9c-c7e2-41d3-95ae-4ac034d2a41c", 2024-04-25T13:48:33.847 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:48:33.848 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:48:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.850+0000 7ff66b41a640 1 -- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6500afc40 msgr2=0x7ff6500b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.850+0000 7ff66b41a640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6500afc40 0x7ff6500b2150 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x55b3d56a5670 tx=0x7ff65c004ab0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.850+0000 7ff66b41a640 1 -- 172.21.15.27:0/4250070714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff664103f50 msgr2=0x7ff6640ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.850+0000 7ff66b41a640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff664103f50 0x7ff6640ff480 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff66002a860 tx=0x7ff660032d50 comp rx=0 tx=0).stop 2024-04-25T13:48:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.850+0000 7ff66b41a640 1 -- 172.21.15.27:0/4250070714 shutdown_connections 2024-04-25T13:48:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.850+0000 7ff66b41a640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6500afc40 0x7ff6500b2150 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.850+0000 7ff66b41a640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff664108a90 0x7ff6640fe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.850+0000 7ff66b41a640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6641043f0 0x7ff6640fdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.851+0000 7ff66b41a640 1 --2- 172.21.15.27:0/4250070714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff664103f50 0x7ff6640ff480 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.851+0000 7ff66b41a640 1 -- 172.21.15.27:0/4250070714 >> 172.21.15.27:0/4250070714 conn(0x7ff6640fd340 msgr2=0x7ff664101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.853+0000 7ff66b41a640 1 -- 172.21.15.27:0/4250070714 shutdown_connections 2024-04-25T13:48:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:33.853+0000 7ff66b41a640 1 -- 172.21.15.27:0/4250070714 wait complete. 2024-04-25T13:48:33.868 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:33 smithi027 ceph-mon[29140]: pgmap v579: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 6.8 MiB/s wr, 695 op/s 2024-04-25T13:48:33.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:48:33.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:48:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.044+0000 7f90a379e640 1 -- 172.21.15.27:0/1711262491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 msgr2=0x7f909c104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.044+0000 7f90a379e640 1 --2- 172.21.15.27:0/1711262491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 0x7f909c104800 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f909800dc90 tx=0x7f909802e370 comp rx=0 tx=0).stop 2024-04-25T13:48:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.044+0000 7f90a379e640 1 -- 172.21.15.27:0/1711262491 shutdown_connections 2024-04-25T13:48:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.044+0000 7f90a379e640 1 --2- 172.21.15.27:0/1711262491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c109a40 0x7f909c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.044+0000 7f90a379e640 1 --2- 172.21.15.27:0/1711262491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c104d40 0x7f909c105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.044+0000 7f90a379e640 1 --2- 172.21.15.27:0/1711262491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 0x7f909c104800 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.044+0000 7f90a379e640 1 -- 172.21.15.27:0/1711262491 >> 172.21.15.27:0/1711262491 conn(0x7f909c0fecf0 msgr2=0x7f909c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.044+0000 7f90a379e640 1 -- 172.21.15.27:0/1711262491 shutdown_connections 2024-04-25T13:48:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.045+0000 7f90a379e640 1 -- 172.21.15.27:0/1711262491 wait complete. 2024-04-25T13:48:34.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.045+0000 7f90a379e640 1 Processor -- start 2024-04-25T13:48:34.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.045+0000 7f90a379e640 1 -- start start 2024-04-25T13:48:34.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a379e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 0x7f909c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a379e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c104d40 0x7f909c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a379e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c109a40 0x7f909c19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a153a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 0x7f909c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a0d39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c104d40 0x7f909c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a153a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 0x7f909c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47016/0 (socket says 172.21.15.27:47016) 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a0d39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c104d40 0x7f909c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52154/0 (socket says 172.21.15.27:52154) 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a153a640 1 -- 172.21.15.27:0/2778533825 learned_addr learned my addr 172.21.15.27:0/2778533825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f909c10d450 con 0x7f909c109a40 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f909c10d2d0 con 0x7f909c104d40 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f909c10d5d0 con 0x7f909c103b00 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.046+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c109a40 0x7f909c19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.047+0000 7f90a153a640 1 -- 172.21.15.27:0/2778533825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c104d40 msgr2=0x7f909c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.047+0000 7f90a153a640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c104d40 0x7f909c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.047+0000 7f90a153a640 1 -- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c109a40 msgr2=0x7f909c19f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.047+0000 7f90a153a640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c109a40 0x7f909c19f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.047+0000 7f90a153a640 1 -- 172.21.15.27:0/2778533825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c19fb40 con 0x7f909c103b00 2024-04-25T13:48:34.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.047+0000 7f90a153a640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 0x7f909c198000 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9098004970 tx=0x7f90980049a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:34.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.048+0000 7f90927fc640 1 -- 172.21.15.27:0/2778533825 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909800bb10 con 0x7f909c103b00 2024-04-25T13:48:34.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.048+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909c19fdd0 con 0x7f909c103b00 2024-04-25T13:48:34.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.048+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f909c1a0280 con 0x7f909c103b00 2024-04-25T13:48:34.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.048+0000 7f90927fc640 1 -- 172.21.15.27:0/2778533825 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f909800bcb0 con 0x7f909c103b00 2024-04-25T13:48:34.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.048+0000 7f90927fc640 1 -- 172.21.15.27:0/2778533825 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90980054d0 con 0x7f909c103b00 2024-04-25T13:48:34.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.051+0000 7f90927fc640 1 -- 172.21.15.27:0/2778533825 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9098046020 con 0x7f909c103b00 2024-04-25T13:48:34.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.052+0000 7f90aa64b740 1 Processor -- start 2024-04-25T13:48:34.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.052+0000 7f90927fc640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90880afd10 0x7f90880b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.052+0000 7f90aa64b740 1 -- start start 2024-04-25T13:48:34.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.052+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90880afd10 0x7f90880b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.052+0000 7f90927fc640 1 -- 172.21.15.27:0/2778533825 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9098102490 con 0x7f909c103b00 2024-04-25T13:48:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90880afd10 0x7f90880b2220 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9094005fd0 tx=0x7f9094009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde89a390 0x559ddec3ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 0x559ddec3bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec3cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90aa64b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559ddeba0d80 con 0x559dde8714d0 2024-04-25T13:48:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90aa64b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559ddeb5a8b0 con 0x559dde8796b0 2024-04-25T13:48:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90aa64b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559ddeb9cd40 con 0x559dde89a390 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec3cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec3cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52162/0 (socket says 172.21.15.27:52162) 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 -- 172.21.15.27:0/1742048998 learned_addr learned my addr 172.21.15.27:0/1742048998 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a0d39640 1 --2- 172.21.15.27:0/1742048998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 0x559ddec3bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 -- 172.21.15.27:0/1742048998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde89a390 msgr2=0x559ddec3ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/1742048998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde89a390 0x559ddec3ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 -- 172.21.15.27:0/1742048998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 msgr2=0x559ddec3bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/1742048998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 0x559ddec3bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.053+0000 7f90a1d3b640 1 -- 172.21.15.27:0/1742048998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ddeb2ba40 con 0x559dde8796b0 2024-04-25T13:48:34.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.054+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/1742048998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec3cc70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f90940115b0 tx=0x7f9094012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85346d5bd51fc320 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.054+0000 7f90877fe640 1 -- 172.21.15.27:0/1742048998 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9094022070 con 0x559dde8796b0 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.055+0000 7f90877fe640 1 -- 172.21.15.27:0/1742048998 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9094010c80 con 0x559dde8796b0 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.055+0000 7f90877fe640 1 -- 172.21.15.27:0/1742048998 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909401ecb0 con 0x559dde8796b0 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 -- 172.21.15.27:0/1742048998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 msgr2=0x559ddec3cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 --2- 172.21.15.27:0/1742048998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec3cc70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f90940115b0 tx=0x7f9094012000 comp rx=0 tx=0).stop 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 -- 172.21.15.27:0/1742048998 shutdown_connections 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 --2- 172.21.15.27:0/1742048998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec3cc70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 --2- 172.21.15.27:0/1742048998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 0x559ddec3bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 --2- 172.21.15.27:0/1742048998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde89a390 0x559ddec3ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 -- 172.21.15.27:0/1742048998 >> 172.21.15.27:0/1742048998 conn(0x559dde88e4e0 msgr2=0x559ddec37270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 -- 172.21.15.27:0/1742048998 shutdown_connections 2024-04-25T13:48:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.056+0000 7f90aa64b740 1 -- 172.21.15.27:0/1742048998 wait complete. 2024-04-25T13:48:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 1 -- start start 2024-04-25T13:48:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 1 Processor -- start 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8714d0 0x559ddec459e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 0x559ddec45f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde89a390 0x559ddec4ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559ddec3d780 con 0x559dde89a390 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559ddeb27d10 con 0x559dde8714d0 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559ddec3d900 con 0x559dde8796b0 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90aa64b740 10 client.0 _ll_register_callbacks cb 0x559ddeb54210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90a1d3b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde89a390 0x559ddec4ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90a153a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8714d0 0x559ddec459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90a1d3b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde89a390 0x559ddec4ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43246/0 (socket says 172.21.15.27:43246) 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90a1d3b640 1 -- 172.21.15.27:0/3988460336 learned_addr learned my addr 172.21.15.27:0/3988460336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.057+0000 7f90a0d39640 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 0x559ddec45f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90a153a640 1 -- 172.21.15.27:0/3988460336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 msgr2=0x559ddec45f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90a153a640 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 0x559ddec45f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90a153a640 1 -- 172.21.15.27:0/3988460336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde89a390 msgr2=0x559ddec4ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90a153a640 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde89a390 0x559ddec4ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90a153a640 1 -- 172.21.15.27:0/3988460336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dde97c7d0 con 0x559dde8714d0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde89a390 0x559ddec4ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90a153a640 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8714d0 0x559ddec459e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x559ddec3cab0 tx=0x7f90980ca2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2320dd2d78aedefc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f9084ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f9084ff9640 1 -- 172.21.15.27:0/3988460336 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90980ca4b0 con 0x559dde8714d0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f9084ff9640 1 -- 172.21.15.27:0/3988460336 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f90980050f0 con 0x559dde8714d0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90aa64b740 1 -- 172.21.15.27:0/3988460336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9094010c80 con 0x559dde8714d0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f9084ff9640 1 -- 172.21.15.27:0/3988460336 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f90980cb0a0 con 0x559dde8714d0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90aa64b740 10 client.26642 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.058+0000 7f90aa64b740 1 -- 172.21.15.27:0/3988460336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559dde8b23f0 con 0x559dde8714d0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f9084ff9640 1 -- 172.21.15.27:0/3988460336 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9094010c80 con 0x559dde8714d0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f9084ff9640 1 -- 172.21.15.27:0/3988460336 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f90980ed0d0 con 0x559dde8714d0 2024-04-25T13:48:34.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f90aa64b740 10 client.26642 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f90aa64b740 10 client.26642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f90aa64b740 20 client.26642 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f90aa64b740 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559dde858f60 0x559ddec53230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f90aa64b740 4 client.26642 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f90aa64b740 1 -- 172.21.15.27:0/3988460336 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ddebae990 con 0x559dde858f60 2024-04-25T13:48:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.059+0000 7f90a0d39640 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559dde858f60 0x559ddec53230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.060+0000 7f90a0d39640 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559dde858f60 0x559ddec53230 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cf4c1065d87e7cc server_cookie=ca5cee818cec95ff in_seq=0 out_seq=0 2024-04-25T13:48:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.060+0000 7f9084ff9640 10 client.26642 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:34.063 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:33 smithi122 ceph-mon[32843]: pgmap v579: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 6.8 MiB/s wr, 695 op/s 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.061+0000 7f9084ff9640 1 -- 172.21.15.27:0/3988460336 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559ddebae990 con 0x559dde858f60 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.061+0000 7f9084ff9640 10 client.26642 handle_command_reply: tid=0 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.061+0000 7f90aa64b740 1 client.26642 shutdown 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.061+0000 7f90aa64b740 1 -- 172.21.15.27:0/3988460336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8714d0 msgr2=0x559ddec459e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.061+0000 7f90aa64b740 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8714d0 0x559ddec459e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x559ddec3cab0 tx=0x7f90980ca2f0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 1 -- 172.21.15.27:0/3988460336 shutdown_connections 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559dde858f60 0x559ddec53230 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f9084ff9640 0 client.26642 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde89a390 0x559ddec4ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 0x559ddec45f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 1 --2- 172.21.15.27:0/3988460336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8714d0 0x559ddec459e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 1 -- 172.21.15.27:0/3988460336 >> 172.21.15.27:0/3988460336 conn(0x559dde88e4e0 msgr2=0x559ddec363f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 1 -- 172.21.15.27:0/3988460336 shutdown_connections 2024-04-25T13:48:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 1 -- 172.21.15.27:0/3988460336 wait complete. 2024-04-25T13:48:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.062+0000 7f90aa64b740 20 client.26642 trim_cache size 0 max 16384 2024-04-25T13:48:34.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:33 smithi045 ceph-mon[32731]: pgmap v579: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 6.8 MiB/s wr, 695 op/s 2024-04-25T13:48:34.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.087+0000 7f90aa64b740 1 Processor -- start 2024-04-25T13:48:34.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.087+0000 7f90aa64b740 1 -- start start 2024-04-25T13:48:34.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.089+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde858f60 0x559ddec359c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.089+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8714d0 0x559ddec35f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.089+0000 7f90a153a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde858f60 0x559ddec359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.089+0000 7f90a153a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde858f60 0x559ddec359c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43254/0 (socket says 172.21.15.27:43254) 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.089+0000 7f90a153a640 1 -- 172.21.15.27:0/2167587442 learned_addr learned my addr 172.21.15.27:0/2167587442 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec36440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90aa64b740 1 -- 172.21.15.27:0/2167587442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f908000c340 con 0x559dde858f60 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec36440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90a0d39640 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8714d0 0x559ddec35f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90aa64b740 1 -- 172.21.15.27:0/2167587442 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f908000ebc0 con 0x559dde8796b0 2024-04-25T13:48:34.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90aa64b740 1 -- 172.21.15.27:0/2167587442 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f908000c9b0 con 0x559dde8714d0 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90a153a640 1 -- 172.21.15.27:0/2167587442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8714d0 msgr2=0x559ddec35f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90a153a640 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8714d0 0x559ddec35f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90a153a640 1 -- 172.21.15.27:0/2167587442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 msgr2=0x559ddec36440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90a153a640 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec36440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.090+0000 7f90a153a640 1 -- 172.21.15.27:0/2167587442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dde8720f0 con 0x559dde858f60 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.091+0000 7f90a153a640 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde858f60 0x559ddec359c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f90980ca6a0 tx=0x7f909802e370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ace7747a6ba1907 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.091+0000 7f9086ffd640 1 -- 172.21.15.27:0/2167587442 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90980ca110 con 0x559dde858f60 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.091+0000 7f9086ffd640 1 -- 172.21.15.27:0/2167587442 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90980ca4b0 con 0x559dde858f60 2024-04-25T13:48:34.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.091+0000 7f9086ffd640 1 -- 172.21.15.27:0/2167587442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90980ced50 con 0x559dde858f60 2024-04-25T13:48:34.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 -- 172.21.15.27:0/2167587442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde858f60 msgr2=0x559ddec359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde858f60 0x559ddec359c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f90980ca6a0 tx=0x7f909802e370 comp rx=0 tx=0).stop 2024-04-25T13:48:34.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 -- 172.21.15.27:0/2167587442 shutdown_connections 2024-04-25T13:48:34.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde8796b0 0x559ddec36440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8714d0 0x559ddec35f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2167587442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde858f60 0x559ddec359c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 -- 172.21.15.27:0/2167587442 >> 172.21.15.27:0/2167587442 conn(0x559dde88e4e0 msgr2=0x559ddec36ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:34.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 -- 172.21.15.27:0/2167587442 shutdown_connections 2024-04-25T13:48:34.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.092+0000 7f90aa64b740 1 -- 172.21.15.27:0/2167587442 wait complete. 2024-04-25T13:48:34.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.093+0000 7f90aa64b740 1 -- start start 2024-04-25T13:48:34.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.093+0000 7f90aa64b740 1 Processor -- start 2024-04-25T13:48:34.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.094+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde858f60 0x559ddec499a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.094+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 0x559ddec49ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90aa64b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 0x559ddec4ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90aa64b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f908000d080 con 0x559dde8714d0 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90aa64b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f908000e4f0 con 0x559dde858f60 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90a153a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde858f60 0x559ddec499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90a153a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde858f60 0x559ddec499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52198/0 (socket says 172.21.15.27:52198) 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90a153a640 1 -- 172.21.15.27:0/2701783766 learned_addr learned my addr 172.21.15.27:0/2701783766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90a1d3b640 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 0x559ddec4ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90a0d39640 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 0x559ddec49ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f908000d750 con 0x559dde8796b0 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90aa64b740 10 client.0 _ll_register_callbacks cb 0x7f90980c6880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:34.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90a153a640 1 -- 172.21.15.27:0/2701783766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 msgr2=0x559ddec4ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.095+0000 7f90a153a640 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 0x559ddec4ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f90a153a640 1 -- 172.21.15.27:0/2701783766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 msgr2=0x559ddec49ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f90a153a640 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 0x559ddec49ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f90a153a640 1 -- 172.21.15.27:0/2701783766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ddeb614b0 con 0x559dde858f60 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f90a153a640 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde858f60 0x559ddec499a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f90980c7c10 tx=0x7f90980ca420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f8560df3d2175a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f9084ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f9084ff9640 1 -- 172.21.15.27:0/2701783766 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90980ca110 con 0x559dde858f60 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f9084ff9640 1 -- 172.21.15.27:0/2701783766 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f90980d3580 con 0x559dde858f60 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90980ca4b0 con 0x559dde858f60 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f9084ff9640 1 -- 172.21.15.27:0/2701783766 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f90980d3800 con 0x559dde858f60 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f90aa64b740 10 client.26648 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559ddeb614b0 con 0x559dde858f60 2024-04-25T13:48:34.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.096+0000 7f9084ff9640 1 -- 172.21.15.27:0/2701783766 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90980ced50 con 0x559dde858f60 2024-04-25T13:48:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f9084ff9640 1 -- 172.21.15.27:0/2701783766 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f90980e7070 con 0x559dde858f60 2024-04-25T13:48:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f90aa64b740 10 client.26648 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f90aa64b740 10 client.26648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f90aa64b740 20 client.26648 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:34.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559dde89a390 0x559ddecbd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f90aa64b740 4 client.26648 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f90980ca4b0 con 0x559dde89a390 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f90a0d39640 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559dde89a390 0x559ddecbd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f90a0d39640 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559dde89a390 0x559ddecbd5f0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9ec04d204b3a69a server_cookie=3f41be356d572026 in_seq=0 out_seq=0 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.097+0000 7f9084ff9640 10 client.26648 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f9084ff9640 1 -- 172.21.15.27:0/2701783766 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f90980ca4b0 con 0x559dde89a390 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f9084ff9640 10 client.26648 handle_command_reply: tid=0 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 client.26648 shutdown 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde858f60 msgr2=0x559ddec499a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde858f60 0x559ddec499a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f90980c7c10 tx=0x7f90980ca420 comp rx=0 tx=0).stop 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 shutdown_connections 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559dde89a390 0x559ddecbd5f0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559dde8796b0 0x559ddec4ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f9084ff9640 0 client.26648 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:34.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559dde8714d0 0x559ddec49ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 --2- 172.21.15.27:0/2701783766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559dde858f60 0x559ddec499a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 >> 172.21.15.27:0/2701783766 conn(0x559dde88e4e0 msgr2=0x559ddec44030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 shutdown_connections 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 1 -- 172.21.15.27:0/2701783766 wait complete. 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90aa64b740 20 client.26648 trim_cache size 0 max 16384 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stdout: "5cbccd9c-c7e2-41d3-95ae-4ac034d2a41c": { 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:48:34.101 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "5cbccd9c-c7e2-41d3-95ae-4ac034d2a41c", 2024-04-25T13:48:34.102 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:48:34.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:48:34.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:48:34.102 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90880afd10 msgr2=0x7f90880b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90a379e640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90880afd10 0x7f90880b2220 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9094005fd0 tx=0x7f9094009520 comp rx=0 tx=0).stop 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 msgr2=0x7f909c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.098+0000 7f90a379e640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 0x7f909c198000 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9098004970 tx=0x7f90980049a0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.099+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 shutdown_connections 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.099+0000 7f90a379e640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f90880afd10 0x7f90880b2220 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.099+0000 7f90a379e640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909c109a40 0x7f909c19f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.099+0000 7f90a379e640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f909c104d40 0x7f909c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.099+0000 7f90a379e640 1 --2- 172.21.15.27:0/2778533825 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f909c103b00 0x7f909c198000 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.099+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 >> 172.21.15.27:0/2778533825 conn(0x7f909c0fecf0 msgr2=0x7f909c105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.099+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 shutdown_connections 2024-04-25T13:48:34.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:48:34.099+0000 7f90a379e640 1 -- 172.21.15.27:0/2778533825 wait complete. 2024-04-25T13:48:34.112 INFO:tasks.cephfs.filesystem:scrub status for tag:5cbccd9c-c7e2-41d3-95ae-4ac034d2a41c - {'path': '/', 'tag': '5cbccd9c-c7e2-41d3-95ae-4ac034d2a41c', 'options': 'recursive,force'} 2024-04-25T13:48:35.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:34 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:48:35.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:34 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:35.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:34 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:48:35.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:34 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:48:35.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:34 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:35.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:34 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:48:35.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:34 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:48:35.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:34 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:35.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:34 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:48:36.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:35 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:36.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:35 smithi122 ceph-mon[32843]: pgmap v580: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 623 op/s 2024-04-25T13:48:36.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:35 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:36.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:35 smithi045 ceph-mon[32731]: pgmap v580: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 623 op/s 2024-04-25T13:48:36.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:35 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:48:36.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:35 smithi027 ceph-mon[29140]: pgmap v580: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 623 op/s 2024-04-25T13:48:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 450.0 s, 1898.4 tps, lat 52.747 ms stddev 56.302 2024-04-25T13:48:38.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:37 smithi122 ceph-mon[32843]: pgmap v581: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.9 MiB/s wr, 896 op/s 2024-04-25T13:48:38.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:37 smithi045 ceph-mon[32731]: pgmap v581: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.9 MiB/s wr, 896 op/s 2024-04-25T13:48:38.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:37 smithi027 ceph-mon[29140]: pgmap v581: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.9 MiB/s wr, 896 op/s 2024-04-25T13:48:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 -- 172.21.15.27:0/2148681742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810e530 msgr2=0x7fae8810f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 --2- 172.21.15.27:0/2148681742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810e530 0x7fae8810f230 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fae80009ca0 tx=0x7fae8002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 -- 172.21.15.27:0/2148681742 shutdown_connections 2024-04-25T13:48:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 --2- 172.21.15.27:0/2148681742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae88114470 0x7fae88116890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 --2- 172.21.15.27:0/2148681742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae8810f770 0x7fae8810fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 --2- 172.21.15.27:0/2148681742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810e530 0x7fae8810f230 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 -- 172.21.15.27:0/2148681742 >> 172.21.15.27:0/2148681742 conn(0x7fae8806e5b0 msgr2=0x7fae8810bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 -- 172.21.15.27:0/2148681742 shutdown_connections 2024-04-25T13:48:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.329+0000 7fae8f86e640 1 -- 172.21.15.27:0/2148681742 wait complete. 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8f86e640 1 Processor -- start 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8f86e640 1 -- start start 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8f86e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae8810e530 0x7fae881a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8f86e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810f770 0x7fae881a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8f86e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae88114470 0x7fae881ac120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8f86e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae8811a080 con 0x7fae88114470 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8f86e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae88119f00 con 0x7fae8810f770 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8f86e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fae8811a200 con 0x7fae8810e530 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8de0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae88114470 0x7fae881ac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8ce09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810f770 0x7fae881a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8ce09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810f770 0x7fae881a5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52206/0 (socket says 172.21.15.27:52206) 2024-04-25T13:48:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8de0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae88114470 0x7fae881ac120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43278/0 (socket says 172.21.15.27:43278) 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8ce09640 1 -- 172.21.15.27:0/3488405569 learned_addr learned my addr 172.21.15.27:0/3488405569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.330+0000 7fae8d60a640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae8810e530 0x7fae881a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8ce09640 1 -- 172.21.15.27:0/3488405569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae8810e530 msgr2=0x7fae881a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8ce09640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae8810e530 0x7fae881a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8ce09640 1 -- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae88114470 msgr2=0x7fae881ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8ce09640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae88114470 0x7fae881ac120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8ce09640 1 -- 172.21.15.27:0/3488405569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae881ac7c0 con 0x7fae8810f770 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8d60a640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae8810e530 0x7fae881a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8de0b640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae88114470 0x7fae881ac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8ce09640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810f770 0x7fae881a5240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fae8400ea60 tx=0x7fae8400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae7e7fc640 1 -- 172.21.15.27:0/3488405569 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8400cde0 con 0x7fae8810f770 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.331+0000 7fae8f86e640 1 -- 172.21.15.27:0/3488405569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae881acab0 con 0x7fae8810f770 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.332+0000 7fae8f86e640 1 -- 172.21.15.27:0/3488405569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae881ad020 con 0x7fae8810f770 2024-04-25T13:48:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.332+0000 7fae7e7fc640 1 -- 172.21.15.27:0/3488405569 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae840035f0 con 0x7fae8810f770 2024-04-25T13:48:38.334 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.332+0000 7fae7e7fc640 1 -- 172.21.15.27:0/3488405569 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8400cde0 con 0x7fae8810f770 2024-04-25T13:48:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.335+0000 7fae96804740 1 Processor -- start 2024-04-25T13:48:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae96804740 1 -- start start 2024-04-25T13:48:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae96804740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4dab1b0 0x5604c514b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae96804740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c514bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae96804740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 0x5604c5152df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae96804740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5604c50b1460 con 0x5604c4dab1b0 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae96804740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5604c506b590 con 0x5604c4d822f0 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae96804740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5604c50ad420 con 0x5604c4d8a4d0 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae8de0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4dab1b0 0x5604c514b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae8de0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4dab1b0 0x5604c514b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43280/0 (socket says 172.21.15.27:43280) 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae8de0b640 1 -- 172.21.15.27:0/2322740733 learned_addr learned my addr 172.21.15.27:0/2322740733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae8d60a640 1 --2- 172.21.15.27:0/2322740733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c514bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae7e7fc640 1 -- 172.21.15.27:0/3488405569 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fae84004000 con 0x7fae8810f770 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae8de0b640 1 -- 172.21.15.27:0/2322740733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 msgr2=0x5604c5152df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.336+0000 7fae8de0b640 1 --2- 172.21.15.27:0/2322740733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 0x5604c5152df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.337+0000 7fae8de0b640 1 -- 172.21.15.27:0/2322740733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 msgr2=0x5604c514bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.337+0000 7fae8de0b640 1 --2- 172.21.15.27:0/2322740733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c514bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.337+0000 7fae8de0b640 1 -- 172.21.15.27:0/2322740733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604c50780d0 con 0x5604c4dab1b0 2024-04-25T13:48:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.337+0000 7fae8de0b640 1 --2- 172.21.15.27:0/2322740733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4dab1b0 0x5604c514b970 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fae78011470 tx=0x7fae78011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb2c9c948f2dd06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.337+0000 7fae737fe640 1 -- 172.21.15.27:0/2322740733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78004020 con 0x5604c4dab1b0 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.337+0000 7fae737fe640 1 -- 172.21.15.27:0/2322740733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae780041c0 con 0x5604c4dab1b0 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.337+0000 7fae737fe640 1 -- 172.21.15.27:0/2322740733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7800f8b0 con 0x5604c4dab1b0 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.337+0000 7fae7e7fc640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae740afcd0 0x7fae740b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.338+0000 7fae7e7fc640 1 -- 172.21.15.27:0/3488405569 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fae840dea90 con 0x7fae8810f770 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.338+0000 7fae8d60a640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae740afcd0 0x7fae740b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.338+0000 7fae96804740 1 -- 172.21.15.27:0/2322740733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4dab1b0 msgr2=0x5604c514b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.338+0000 7fae96804740 1 --2- 172.21.15.27:0/2322740733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4dab1b0 0x5604c514b970 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fae78011470 tx=0x7fae78011930 comp rx=0 tx=0).stop 2024-04-25T13:48:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.338+0000 7fae96804740 1 -- 172.21.15.27:0/2322740733 shutdown_connections 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae96804740 1 --2- 172.21.15.27:0/2322740733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 0x5604c5152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae96804740 1 --2- 172.21.15.27:0/2322740733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c514bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae96804740 1 --2- 172.21.15.27:0/2322740733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4dab1b0 0x5604c514b970 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae96804740 1 -- 172.21.15.27:0/2322740733 >> 172.21.15.27:0/2322740733 conn(0x5604c4d9f300 msgr2=0x5604c5147300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae96804740 1 -- 172.21.15.27:0/2322740733 shutdown_connections 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae96804740 1 -- 172.21.15.27:0/2322740733 wait complete. 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae8d60a640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae740afcd0 0x7fae740b21e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fae80000f80 tx=0x7fae80005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae96804740 1 -- start start 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.339+0000 7fae96804740 1 Processor -- start 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae96804740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d822f0 0x5604c5155840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae96804740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 0x5604c5155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae96804740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4dab1b0 0x5604c515cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae96804740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5604c5153840 con 0x5604c4d822f0 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae96804740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5604c5038b60 con 0x5604c4dab1b0 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae96804740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5604c51539c0 con 0x5604c4d8a4d0 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae8d60a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d822f0 0x5604c5155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae96804740 10 client.0 _ll_register_callbacks cb 0x5604c505d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae8d60a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d822f0 0x5604c5155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43286/0 (socket says 172.21.15.27:43286) 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae8d60a640 1 -- 172.21.15.27:0/811618744 learned_addr learned my addr 172.21.15.27:0/811618744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae8de0b640 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4dab1b0 0x5604c515cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae8ce09640 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 0x5604c5155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.340+0000 7fae8d60a640 1 -- 172.21.15.27:0/811618744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 msgr2=0x5604c5155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae8d60a640 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 0x5604c5155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae8d60a640 1 -- 172.21.15.27:0/811618744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4dab1b0 msgr2=0x5604c515cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae8d60a640 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4dab1b0 0x5604c515cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae8d60a640 1 -- 172.21.15.27:0/811618744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604c4dc3210 con 0x5604c4d822f0 2024-04-25T13:48:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae8d60a640 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d822f0 0x5604c5155840 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fae80031b50 tx=0x7fae8002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f642c25b593f64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:48:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae70ff9640 1 -- 172.21.15.27:0/811618744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8002f610 con 0x5604c4d822f0 2024-04-25T13:48:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae96804740 1 -- 172.21.15.27:0/811618744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae780041c0 con 0x5604c4d822f0 2024-04-25T13:48:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae70ff9640 1 -- 172.21.15.27:0/811618744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fae8002f7b0 con 0x5604c4d822f0 2024-04-25T13:48:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae70ff9640 1 -- 172.21.15.27:0/811618744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fae8002f9b0 con 0x5604c4d822f0 2024-04-25T13:48:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae96804740 10 client.16950 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.341+0000 7fae96804740 1 -- 172.21.15.27:0/811618744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5604c4e8f620 con 0x5604c4d822f0 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.342+0000 7fae70ff9640 1 -- 172.21.15.27:0/811618744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae780041c0 con 0x5604c4d822f0 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.342+0000 7fae70ff9640 1 -- 172.21.15.27:0/811618744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fae80052070 con 0x5604c4d822f0 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.342+0000 7fae96804740 10 client.16950 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.342+0000 7fae96804740 10 client.16950 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.342+0000 7fae96804740 20 client.16950 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.342+0000 7fae96804740 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604c4d69d80 0x5604c51634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.342+0000 7fae96804740 4 client.16950 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.343+0000 7fae96804740 1 -- 172.21.15.27:0/811618744 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604c50bf070 con 0x5604c4d69d80 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.345+0000 7fae8ce09640 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604c4d69d80 0x5604c51634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.345+0000 7fae70ff9640 10 client.16950 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.345+0000 7fae8ce09640 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604c4d69d80 0x5604c51634e0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=399593392f93b97d server_cookie=a3486e40435d532d in_seq=0 out_seq=0 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae70ff9640 1 -- 172.21.15.27:0/811618744 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5604c50bf070 con 0x5604c4d69d80 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae70ff9640 10 client.16950 handle_command_reply: tid=0 2024-04-25T13:48:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 client.16950 shutdown 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 -- 172.21.15.27:0/811618744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d822f0 msgr2=0x5604c5155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d822f0 0x5604c5155840 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fae80031b50 tx=0x7fae8002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 -- 172.21.15.27:0/811618744 shutdown_connections 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604c4d69d80 0x5604c51634e0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4dab1b0 0x5604c515cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae70ff9640 0 client.16950 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d8a4d0 0x5604c5155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 --2- 172.21.15.27:0/811618744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d822f0 0x5604c5155840 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.346+0000 7fae96804740 1 -- 172.21.15.27:0/811618744 >> 172.21.15.27:0/811618744 conn(0x5604c4d9f300 msgr2=0x5604c5146250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.347+0000 7fae96804740 1 -- 172.21.15.27:0/811618744 shutdown_connections 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.347+0000 7fae96804740 1 -- 172.21.15.27:0/811618744 wait complete. 2024-04-25T13:48:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.347+0000 7fae96804740 20 client.16950 trim_cache size 0 max 16384 2024-04-25T13:48:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.371+0000 7fae96804740 1 Processor -- start 2024-04-25T13:48:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.371+0000 7fae96804740 1 -- start start 2024-04-25T13:48:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.371+0000 7fae96804740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d69d80 0x5604c51472b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.371+0000 7fae96804740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d822f0 0x5604c515cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.371+0000 7fae96804740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c5155410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.371+0000 7fae96804740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae881b3bf0 con 0x5604c4d8a4d0 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.371+0000 7fae96804740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae881b6470 con 0x5604c4d69d80 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.371+0000 7fae96804740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fae881b4260 con 0x5604c4d822f0 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8de0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c5155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8ce09640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d822f0 0x5604c515cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8de0b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c5155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43296/0 (socket says 172.21.15.27:43296) 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8de0b640 1 -- 172.21.15.27:0/1630370212 learned_addr learned my addr 172.21.15.27:0/1630370212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8ce09640 1 -- 172.21.15.27:0/1630370212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d69d80 msgr2=0x5604c51472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8d60a640 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d69d80 0x5604c51472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8ce09640 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d69d80 0x5604c51472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8ce09640 1 -- 172.21.15.27:0/1630370212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 msgr2=0x5604c5155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8ce09640 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c5155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8ce09640 1 -- 172.21.15.27:0/1630370212 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604c5153720 con 0x5604c4d822f0 2024-04-25T13:48:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8ce09640 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d822f0 0x5604c515cc00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fae840dc5c0 tx=0x7fae84003db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=235a458169acd5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:38.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae8d60a640 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d69d80 0x5604c51472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:48:38.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae72ffd640 1 -- 172.21.15.27:0/1630370212 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae840ab350 con 0x5604c4d822f0 2024-04-25T13:48:38.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae72ffd640 1 -- 172.21.15.27:0/1630370212 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae840afe00 con 0x5604c4d822f0 2024-04-25T13:48:38.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.372+0000 7fae72ffd640 1 -- 172.21.15.27:0/1630370212 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae840a2910 con 0x5604c4d822f0 2024-04-25T13:48:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.374+0000 7fae96804740 1 -- 172.21.15.27:0/1630370212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d822f0 msgr2=0x5604c515cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.374+0000 7fae96804740 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d822f0 0x5604c515cc00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fae840dc5c0 tx=0x7fae84003db0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.374+0000 7fae96804740 1 -- 172.21.15.27:0/1630370212 shutdown_connections 2024-04-25T13:48:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.374+0000 7fae96804740 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c5155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.374+0000 7fae96804740 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d822f0 0x5604c515cc00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.374+0000 7fae96804740 1 --2- 172.21.15.27:0/1630370212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d69d80 0x5604c51472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.374+0000 7fae96804740 1 -- 172.21.15.27:0/1630370212 >> 172.21.15.27:0/1630370212 conn(0x5604c4d9f300 msgr2=0x5604c5146ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.375+0000 7fae96804740 1 -- 172.21.15.27:0/1630370212 shutdown_connections 2024-04-25T13:48:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.375+0000 7fae96804740 1 -- 172.21.15.27:0/1630370212 wait complete. 2024-04-25T13:48:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.375+0000 7fae96804740 1 -- start start 2024-04-25T13:48:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.375+0000 7fae96804740 1 Processor -- start 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae96804740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d69d80 0x5604c515a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae96804740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c515aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae96804740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c515b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae96804740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fae881b4930 con 0x5604c4d8a4d0 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae96804740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fae881b5da0 con 0x5604c4d822f0 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae96804740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fae881b5000 con 0x5604c4d69d80 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae8ce09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c515aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae8ce09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c515aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52240/0 (socket says 172.21.15.27:52240) 2024-04-25T13:48:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae8ce09640 1 -- 172.21.15.27:0/1770029011 learned_addr learned my addr 172.21.15.27:0/1770029011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae96804740 10 client.0 _ll_register_callbacks cb 0x7fae78005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.376+0000 7fae8de0b640 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c515b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae8d60a640 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d69d80 0x5604c515a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae8ce09640 1 -- 172.21.15.27:0/1770029011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d69d80 msgr2=0x5604c515a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae8ce09640 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d69d80 0x5604c515a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae8ce09640 1 -- 172.21.15.27:0/1770029011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 msgr2=0x5604c515b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae8ce09640 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c515b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae8ce09640 1 -- 172.21.15.27:0/1770029011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604c515bf60 con 0x5604c4d822f0 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae8ce09640 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c515aae0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fae840dc930 tx=0x7fae840a3460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f06a9c5a8cd74a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae70ff9640 1 -- 172.21.15.27:0/1770029011 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae84087060 con 0x5604c4d822f0 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae70ff9640 1 -- 172.21.15.27:0/1770029011 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fae840a6200 con 0x5604c4d822f0 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae96804740 1 -- 172.21.15.27:0/1770029011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae840afe00 con 0x5604c4d822f0 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae70ff9640 1 -- 172.21.15.27:0/1770029011 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fae840ab350 con 0x5604c4d822f0 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae96804740 10 client.26663 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.377+0000 7fae96804740 1 -- 172.21.15.27:0/1770029011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5604c51cb890 con 0x5604c4d822f0 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae70ff9640 1 -- 172.21.15.27:0/1770029011 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae840afe00 con 0x5604c4d822f0 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae70ff9640 1 -- 172.21.15.27:0/1770029011 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fae840ac4e0 con 0x5604c4d822f0 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae96804740 10 client.26663 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae96804740 10 client.26663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae96804740 20 client.26663 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae96804740 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604c4dab1b0 0x5604c51ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae96804740 4 client.26663 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae96804740 1 -- 172.21.15.27:0/1770029011 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5604c50b8ee0 con 0x5604c4dab1b0 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.378+0000 7fae8d60a640 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604c4dab1b0 0x5604c51ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.379+0000 7fae8d60a640 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604c4dab1b0 0x5604c51ce1b0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22ecd91a4b8fb6a3 server_cookie=a077c85eae2af89b in_seq=0 out_seq=0 2024-04-25T13:48:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.379+0000 7fae70ff9640 10 client.26663 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae70ff9640 1 -- 172.21.15.27:0/1770029011 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5604c50b8ee0 con 0x5604c4dab1b0 2024-04-25T13:48:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae70ff9640 10 client.26663 handle_command_reply: tid=0 2024-04-25T13:48:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 client.26663 shutdown 2024-04-25T13:48:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 -- 172.21.15.27:0/1770029011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 msgr2=0x5604c515aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c515aae0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fae840dc930 tx=0x7fae840a3460 comp rx=0 tx=0).stop 2024-04-25T13:48:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 -- 172.21.15.27:0/1770029011 shutdown_connections 2024-04-25T13:48:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5604c4dab1b0 0x5604c51ce1b0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5604c4d8a4d0 0x5604c515b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c4d822f0 0x5604c515aae0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 --2- 172.21.15.27:0/1770029011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5604c4d69d80 0x5604c515a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae96804740 1 -- 172.21.15.27:0/1770029011 >> 172.21.15.27:0/1770029011 conn(0x5604c4d9f300 msgr2=0x5604c5155380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.395+0000 7fae70ff9640 0 client.26663 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.397+0000 7fae96804740 1 -- 172.21.15.27:0/1770029011 shutdown_connections 2024-04-25T13:48:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.397+0000 7fae96804740 1 -- 172.21.15.27:0/1770029011 wait complete. 2024-04-25T13:48:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.398+0000 7fae96804740 20 client.26663 trim_cache size 0 max 16384 2024-04-25T13:48:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:48:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:48:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:48:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:48:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.398+0000 7fae8f86e640 1 -- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae740afcd0 msgr2=0x7fae740b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.398+0000 7fae8f86e640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae740afcd0 0x7fae740b21e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fae80000f80 tx=0x7fae80005fd0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 -- 172.21.15.27:0/3488405569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810f770 msgr2=0x7fae881a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810f770 0x7fae881a5240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fae8400ea60 tx=0x7fae8400ef20 comp rx=0 tx=0).stop 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 -- 172.21.15.27:0/3488405569 shutdown_connections 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fae740afcd0 0x7fae740b21e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae88114470 0x7fae881ac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae8810f770 0x7fae881a5240 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 --2- 172.21.15.27:0/3488405569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae8810e530 0x7fae881a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 -- 172.21.15.27:0/3488405569 >> 172.21.15.27:0/3488405569 conn(0x7fae8806e5b0 msgr2=0x7fae88110450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 -- 172.21.15.27:0/3488405569 shutdown_connections 2024-04-25T13:48:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:38.399+0000 7fae8f86e640 1 -- 172.21.15.27:0/3488405569 wait complete. 2024-04-25T13:48:40.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:39 smithi122 ceph-mon[32843]: pgmap v582: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.1 MiB/s wr, 637 op/s 2024-04-25T13:48:40.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:39 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:48:40.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:39 smithi045 ceph-mon[32731]: pgmap v582: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.1 MiB/s wr, 637 op/s 2024-04-25T13:48:40.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:39 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:48:40.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:39 smithi027 ceph-mon[29140]: pgmap v582: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.1 MiB/s wr, 637 op/s 2024-04-25T13:48:40.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:39 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:48:42.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:41 smithi122 ceph-mon[32843]: pgmap v583: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.0 MiB/s wr, 706 op/s 2024-04-25T13:48:42.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:41 smithi045 ceph-mon[32731]: pgmap v583: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.0 MiB/s wr, 706 op/s 2024-04-25T13:48:42.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:41 smithi027 ceph-mon[29140]: pgmap v583: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.0 MiB/s wr, 706 op/s 2024-04-25T13:48:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 455.0 s, 2183.4 tps, lat 45.600 ms stddev 50.565 2024-04-25T13:48:44.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:43 smithi122 ceph-mon[32843]: pgmap v584: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.2 MiB/s wr, 763 op/s 2024-04-25T13:48:44.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:43 smithi045 ceph-mon[32731]: pgmap v584: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.2 MiB/s wr, 763 op/s 2024-04-25T13:48:44.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:43 smithi027 ceph-mon[29140]: pgmap v584: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.2 MiB/s wr, 763 op/s 2024-04-25T13:48:46.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:45 smithi122 ceph-mon[32843]: pgmap v585: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.9 MiB/s wr, 641 op/s 2024-04-25T13:48:46.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:45 smithi045 ceph-mon[32731]: pgmap v585: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.9 MiB/s wr, 641 op/s 2024-04-25T13:48:46.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:45 smithi027 ceph-mon[29140]: pgmap v585: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.9 MiB/s wr, 641 op/s 2024-04-25T13:48:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 460.0 s, 1514.2 tps, lat 66.081 ms stddev 71.415 2024-04-25T13:48:48.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:47 smithi122 ceph-mon[32843]: pgmap v586: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.7 MiB/s wr, 823 op/s 2024-04-25T13:48:48.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:47 smithi045 ceph-mon[32731]: pgmap v586: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.7 MiB/s wr, 823 op/s 2024-04-25T13:48:48.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:47 smithi027 ceph-mon[29140]: pgmap v586: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.7 MiB/s wr, 823 op/s 2024-04-25T13:48:50.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:49 smithi122 ceph-mon[32843]: pgmap v587: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.6 MiB/s wr, 539 op/s 2024-04-25T13:48:50.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:49 smithi045 ceph-mon[32731]: pgmap v587: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.6 MiB/s wr, 539 op/s 2024-04-25T13:48:50.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:49 smithi027 ceph-mon[29140]: pgmap v587: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.6 MiB/s wr, 539 op/s 2024-04-25T13:48:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.551+0000 7f071aebd640 1 -- 172.21.15.27:0/3936629662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07141076f0 msgr2=0x7f0714107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.551 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.551+0000 7f071aebd640 1 --2- 172.21.15.27:0/3936629662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07141076f0 0x7f0714107b00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0704009860 tx=0x7f070402a360 comp rx=0 tx=0).stop 2024-04-25T13:48:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.559+0000 7f071aebd640 1 -- 172.21.15.27:0/3936629662 shutdown_connections 2024-04-25T13:48:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.559+0000 7f071aebd640 1 --2- 172.21.15.27:0/3936629662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07141076f0 0x7f0714107b00 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.559+0000 7f071aebd640 1 --2- 172.21.15.27:0/3936629662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0714104d90 0x7f07141071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.559+0000 7f071aebd640 1 --2- 172.21.15.27:0/3936629662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0714102430 0x7f0714104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.559+0000 7f071aebd640 1 -- 172.21.15.27:0/3936629662 >> 172.21.15.27:0/3936629662 conn(0x7f07140fdc60 msgr2=0x7f07141000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.563+0000 7f071aebd640 1 -- 172.21.15.27:0/3936629662 shutdown_connections 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.563+0000 7f071aebd640 1 -- 172.21.15.27:0/3936629662 wait complete. 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.564+0000 7f071aebd640 1 Processor -- start 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.564+0000 7f071aebd640 1 -- start start 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.564+0000 7f071aebd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0714102430 0x7f07140743d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.564+0000 7f071aebd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0714104d90 0x7f0714072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.564+0000 7f071aebd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07141076f0 0x7f0714072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.564+0000 7f071aebd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f071406ce20 con 0x7f0714104d90 2024-04-25T13:48:51.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.564+0000 7f071aebd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f071406cca0 con 0x7f07141076f0 2024-04-25T13:48:51.566 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.564+0000 7f071aebd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f071406cfa0 con 0x7f0714102430 2024-04-25T13:48:51.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.566+0000 7f0718c59640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0714102430 0x7f07140743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.566+0000 7f0718c59640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0714102430 0x7f07140743d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43780/0 (socket says 172.21.15.27:43780) 2024-04-25T13:48:51.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.566+0000 7f0718c59640 1 -- 172.21.15.27:0/205328353 learned_addr learned my addr 172.21.15.27:0/205328353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:51.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.566+0000 7f0713fff640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0714104d90 0x7f0714072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.566+0000 7f0718c59640 1 -- 172.21.15.27:0/205328353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07141076f0 msgr2=0x7f0714072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.566+0000 7f0718c59640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07141076f0 0x7f0714072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.566+0000 7f0718c59640 1 -- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0714104d90 msgr2=0x7f0714072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.566+0000 7f0718c59640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0714104d90 0x7f0714072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.567+0000 7f0718c59640 1 -- 172.21.15.27:0/205328353 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07140734f0 con 0x7f0714102430 2024-04-25T13:48:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.567+0000 7f0713fff640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0714104d90 0x7f0714072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:48:51.568 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.567+0000 7f0718c59640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0714102430 0x7f07140743d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00c3a0 tx=0x7f06fc00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:51.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.569+0000 7f0711ffb640 1 -- 172.21.15.27:0/205328353 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc01b070 con 0x7f0714102430 2024-04-25T13:48:51.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.569+0000 7f0711ffb640 1 -- 172.21.15.27:0/205328353 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06fc00bae0 con 0x7f0714102430 2024-04-25T13:48:51.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.569+0000 7f0711ffb640 1 -- 172.21.15.27:0/205328353 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc016b90 con 0x7f0714102430 2024-04-25T13:48:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.572+0000 7f071aebd640 1 -- 172.21.15.27:0/205328353 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07140737e0 con 0x7f0714102430 2024-04-25T13:48:51.574 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.572+0000 7f071aebd640 1 -- 172.21.15.27:0/205328353 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071411b6a0 con 0x7f0714102430 2024-04-25T13:48:51.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.576+0000 7f0721d74740 1 Processor -- start 2024-04-25T13:48:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.580+0000 7f0721d74740 1 -- start start 2024-04-25T13:48:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.580+0000 7f0721d74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786da7b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.580+0000 7f0721d74740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6b22f0 0x55786da7bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.580+0000 7f0721d74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6ba4d0 0x55786da82df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.580+0000 7f0721d74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55786d9e1460 con 0x55786d6ba4d0 2024-04-25T13:48:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.580+0000 7f0721d74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55786d99b590 con 0x55786d6db1b0 2024-04-25T13:48:51.581 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.580+0000 7f0721d74740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55786d9dd420 con 0x55786d6b22f0 2024-04-25T13:48:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.580+0000 7f0711ffb640 1 -- 172.21.15.27:0/205328353 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f06fc00b630 con 0x7f0714102430 2024-04-25T13:48:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f071945a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786da7b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f071945a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786da7b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:50896/0 (socket says 172.21.15.27:50896) 2024-04-25T13:48:51.582 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f071945a640 1 -- 172.21.15.27:0/1782802890 learned_addr learned my addr 172.21.15.27:0/1782802890 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f071945a640 1 -- 172.21.15.27:0/1782802890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6b22f0 msgr2=0x55786da7bef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:48:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f071945a640 1 --2- 172.21.15.27:0/1782802890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6b22f0 0x55786da7bef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f071945a640 1 -- 172.21.15.27:0/1782802890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6ba4d0 msgr2=0x55786da82df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:48:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f071945a640 1 --2- 172.21.15.27:0/1782802890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6ba4d0 0x55786da82df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f071945a640 1 -- 172.21.15.27:0/1782802890 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55786d9a80d0 con 0x55786d6db1b0 2024-04-25T13:48:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.581+0000 7f0711ffb640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07000afcf0 0x7f07000b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.582+0000 7f0711ffb640 1 -- 172.21.15.27:0/205328353 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f06fc0def20 con 0x7f0714102430 2024-04-25T13:48:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.584+0000 7f0718c59640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07000afcf0 0x7f07000b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.586+0000 7f071945a640 1 --2- 172.21.15.27:0/1782802890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786da7b970 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0704030520 tx=0x7f0704005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e58dccdf6ad754b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:51.587 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.587+0000 7f070affd640 1 -- 172.21.15.27:0/1782802890 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704040020 con 0x55786d6db1b0 2024-04-25T13:48:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.587+0000 7f070affd640 1 -- 172.21.15.27:0/1782802890 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0704038040 con 0x55786d6db1b0 2024-04-25T13:48:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.588+0000 7f0721d74740 1 -- 172.21.15.27:0/1782802890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 msgr2=0x55786da7b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.588+0000 7f0721d74740 1 --2- 172.21.15.27:0/1782802890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786da7b970 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0704030520 tx=0x7f0704005e40 comp rx=0 tx=0).stop 2024-04-25T13:48:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.589+0000 7f0721d74740 1 -- 172.21.15.27:0/1782802890 shutdown_connections 2024-04-25T13:48:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.589+0000 7f0721d74740 1 --2- 172.21.15.27:0/1782802890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6ba4d0 0x55786da82df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.589+0000 7f0721d74740 1 --2- 172.21.15.27:0/1782802890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6b22f0 0x55786da7bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.589+0000 7f0721d74740 1 --2- 172.21.15.27:0/1782802890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786da7b970 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.589+0000 7f0721d74740 1 -- 172.21.15.27:0/1782802890 >> 172.21.15.27:0/1782802890 conn(0x55786d6cf300 msgr2=0x55786da77300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:51.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.591+0000 7f0718c59640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07000afcf0 0x7f07000b2200 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0a4640 tx=0x7f06fc016660 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:51.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.593+0000 7f0721d74740 1 -- 172.21.15.27:0/1782802890 shutdown_connections 2024-04-25T13:48:51.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.593+0000 7f0721d74740 1 -- 172.21.15.27:0/1782802890 wait complete. 2024-04-25T13:48:51.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.594+0000 7f0721d74740 1 -- start start 2024-04-25T13:48:51.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.594+0000 7f0721d74740 1 Processor -- start 2024-04-25T13:48:51.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f0721d74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6b22f0 0x55786da855f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.595 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f0721d74740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da85b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f0721d74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6db1b0 0x55786da8ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f0721d74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55786da83830 con 0x55786d6db1b0 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f0721d74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55786d968b60 con 0x55786d6b22f0 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f0721d74740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55786da839b0 con 0x55786d6ba4d0 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f0721d74740 10 client.0 _ll_register_callbacks cb 0x55786d98d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f071945a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6b22f0 0x55786da855f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f071945a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6b22f0 0x55786da855f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:50910/0 (socket says 172.21.15.27:50910) 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f071945a640 1 -- 172.21.15.27:0/3286104971 learned_addr learned my addr 172.21.15.27:0/3286104971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f071945a640 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6db1b0 0x55786da8ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.595+0000 7f0713fff640 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da85b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.596+0000 7f071945a640 1 -- 172.21.15.27:0/3286104971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 msgr2=0x55786da85b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:51.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.596+0000 7f071945a640 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da85b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.596+0000 7f071945a640 1 -- 172.21.15.27:0/3286104971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6db1b0 msgr2=0x55786da8ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:51.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.596+0000 7f071945a640 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6db1b0 0x55786da8ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.596+0000 7f071945a640 1 -- 172.21.15.27:0/3286104971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786d6f3210 con 0x55786d6b22f0 2024-04-25T13:48:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.598+0000 7f071945a640 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6b22f0 0x55786da855f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f07040043b0 tx=0x7f070402aae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11945f7c81c0a32c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.599+0000 7f06f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:48:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.599+0000 7f06f3fff640 1 -- 172.21.15.27:0/3286104971 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704040030 con 0x55786d6b22f0 2024-04-25T13:48:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.599+0000 7f06f3fff640 1 -- 172.21.15.27:0/3286104971 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f07040308d0 con 0x55786d6b22f0 2024-04-25T13:48:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.601+0000 7f0721d74740 1 -- 172.21.15.27:0/3286104971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0704038040 con 0x55786d6b22f0 2024-04-25T13:48:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.601+0000 7f06f3fff640 1 -- 172.21.15.27:0/3286104971 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0704051050 con 0x55786d6b22f0 2024-04-25T13:48:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.601+0000 7f0721d74740 10 client.26669 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.601+0000 7f0721d74740 1 -- 172.21.15.27:0/3286104971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55786d7bf620 con 0x55786d6b22f0 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f06f3fff640 1 -- 172.21.15.27:0/3286104971 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0704038040 con 0x55786d6b22f0 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f06f3fff640 1 -- 172.21.15.27:0/3286104971 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f07040038d0 con 0x55786d6b22f0 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f0721d74740 10 client.26669 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f0721d74740 10 client.26669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f0721d74740 20 client.26669 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f0721d74740 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55786d699d80 0x55786daa3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f0721d74740 4 client.26669 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f0721d74740 1 -- 172.21.15.27:0/3286104971 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55786d9ef070 con 0x55786d699d80 2024-04-25T13:48:51.602 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.602+0000 7f0713fff640 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55786d699d80 0x55786daa3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.604+0000 7f06f3fff640 10 client.26669 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:51.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.604+0000 7f0713fff640 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55786d699d80 0x55786daa3ef0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f885904a2f75a618 server_cookie=f6a4608968ee1360 in_seq=0 out_seq=0 2024-04-25T13:48:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.608+0000 7f06f3fff640 1 -- 172.21.15.27:0/3286104971 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55786d9ef070 con 0x55786d699d80 2024-04-25T13:48:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.608+0000 7f06f3fff640 10 client.26669 handle_command_reply: tid=0 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.608+0000 7f0721d74740 1 client.26669 shutdown 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.608+0000 7f0721d74740 1 -- 172.21.15.27:0/3286104971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6b22f0 msgr2=0x55786da855f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.608+0000 7f0721d74740 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6b22f0 0x55786da855f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f07040043b0 tx=0x7f070402aae0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 1 -- 172.21.15.27:0/3286104971 shutdown_connections 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55786d699d80 0x55786daa3ef0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6db1b0 0x55786da8ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da85b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 1 --2- 172.21.15.27:0/3286104971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6b22f0 0x55786da855f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 1 -- 172.21.15.27:0/3286104971 >> 172.21.15.27:0/3286104971 conn(0x55786d6cf300 msgr2=0x55786da76250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f06f3fff640 0 client.26669 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 1 -- 172.21.15.27:0/3286104971 shutdown_connections 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 1 -- 172.21.15.27:0/3286104971 wait complete. 2024-04-25T13:48:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.609+0000 7f0721d74740 20 client.26669 trim_cache size 0 max 16384 2024-04-25T13:48:51.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.633+0000 7f0721d74740 1 Processor -- start 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.633+0000 7f0721d74740 1 -- start start 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f0721d74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da844e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f0721d74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6ba4d0 0x55786da76ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f0721d74740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6db1b0 0x55786da770e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f0721d74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f071410eee0 con 0x55786d6b22f0 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f0721d74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f071411c290 con 0x55786d6ba4d0 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f0721d74740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f071410f550 con 0x55786d6db1b0 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da844e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da844e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47830/0 (socket says 172.21.15.27:47830) 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 -- 172.21.15.27:0/241246937 learned_addr learned my addr 172.21.15.27:0/241246937 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f0713fff640 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6ba4d0 0x55786da76ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6db1b0 0x55786da770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 -- 172.21.15.27:0/241246937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6db1b0 msgr2=0x55786da770e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6db1b0 0x55786da770e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 -- 172.21.15.27:0/241246937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6ba4d0 msgr2=0x55786da76ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6ba4d0 0x55786da76ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f071945a640 1 -- 172.21.15.27:0/241246937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55786da80420 con 0x55786d6b22f0 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.634+0000 7f0713fff640 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6ba4d0 0x55786da76ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.635+0000 7f071945a640 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da844e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0704000c00 tx=0x7f0704030340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f50705d16e2c03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.635+0000 7f070a7fc640 1 -- 172.21.15.27:0/241246937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704040070 con 0x55786d6b22f0 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.635+0000 7f070a7fc640 1 -- 172.21.15.27:0/241246937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f070404a020 con 0x55786d6b22f0 2024-04-25T13:48:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.635+0000 7f070a7fc640 1 -- 172.21.15.27:0/241246937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07040326a0 con 0x55786d6b22f0 2024-04-25T13:48:51.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 -- 172.21.15.27:0/241246937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 msgr2=0x55786da844e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da844e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0704000c00 tx=0x7f0704030340 comp rx=0 tx=0).stop 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 -- 172.21.15.27:0/241246937 shutdown_connections 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6db1b0 0x55786da770e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6ba4d0 0x55786da76ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 --2- 172.21.15.27:0/241246937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da844e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 -- 172.21.15.27:0/241246937 >> 172.21.15.27:0/241246937 conn(0x55786d6cf300 msgr2=0x55786da83dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 -- 172.21.15.27:0/241246937 shutdown_connections 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 -- 172.21.15.27:0/241246937 wait complete. 2024-04-25T13:48:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 -- start start 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.636+0000 7f0721d74740 1 Processor -- start 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0721d74740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da86d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0721d74740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da87270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0721d74740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786db0a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0721d74740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f071410fc20 con 0x55786d6b22f0 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0721d74740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f071411bbc0 con 0x55786d6db1b0 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0721d74740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f07141102f0 con 0x55786d6ba4d0 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0721d74740 10 client.0 _ll_register_callbacks cb 0x7f06fc009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f071945a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da86d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0713fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da87270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f071945a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da86d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47846/0 (socket says 172.21.15.27:47846) 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f071945a640 1 -- 172.21.15.27:0/1505163368 learned_addr learned my addr 172.21.15.27:0/1505163368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0713fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da87270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43804/0 (socket says 172.21.15.27:43804) 2024-04-25T13:48:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f071945a640 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786db0a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0713fff640 1 -- 172.21.15.27:0/1505163368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 msgr2=0x55786db0a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0713fff640 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786db0a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0713fff640 1 -- 172.21.15.27:0/1505163368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 msgr2=0x55786da86d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0713fff640 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da86d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f0713fff640 1 -- 172.21.15.27:0/1505163368 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786da8c240 con 0x55786d6ba4d0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.637+0000 7f071945a640 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da86d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0713fff640 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da87270 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x55786daa45a0 tx=0x7f070c018400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b32c2ff46752bf27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f06f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f06f3fff640 1 -- 172.21.15.27:0/1505163368 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c018ac0 con 0x55786d6ba4d0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f06f3fff640 1 -- 172.21.15.27:0/1505163368 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f070c013640 con 0x55786d6ba4d0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 1 -- 172.21.15.27:0/1505163368 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f070404a020 con 0x55786d6ba4d0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f06f3fff640 1 -- 172.21.15.27:0/1505163368 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f070c013860 con 0x55786d6ba4d0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 10 client.26725 fetch_fsmap learned FSMap version 19 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 1 -- 172.21.15.27:0/1505163368 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55786db0b650 con 0x55786d6ba4d0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f06f3fff640 1 -- 172.21.15.27:0/1505163368 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f070404a020 con 0x55786d6ba4d0 2024-04-25T13:48:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f06f3fff640 1 -- 172.21.15.27:0/1505163368 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f070c029030 con 0x55786d6ba4d0 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 10 client.26725 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 10 client.26725 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 20 client.26725 populate_metadata read hostname 'smithi027' 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55786d66f1c0 0x55786db0e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 4 client.26725 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.638+0000 7f0721d74740 1 -- 172.21.15.27:0/1505163368 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55786d9e8ee0 con 0x55786d66f1c0 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.639+0000 7f071945a640 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55786d66f1c0 0x55786db0e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.640+0000 7f071945a640 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55786d66f1c0 0x55786db0e040 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=673f12081e770076 server_cookie=3b45616fa2eb3a79 in_seq=0 out_seq=0 2024-04-25T13:48:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.640+0000 7f06f3fff640 10 client.26725 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:51.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f06f3fff640 1 -- 172.21.15.27:0/1505163368 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55786d9e8ee0 con 0x55786d66f1c0 2024-04-25T13:48:51.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f06f3fff640 10 client.26725 handle_command_reply: tid=0 2024-04-25T13:48:51.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 client.26725 shutdown 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 -- 172.21.15.27:0/1505163368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 msgr2=0x55786da87270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da87270 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x55786daa45a0 tx=0x7f070c018400 comp rx=0 tx=0).stop 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 -- 172.21.15.27:0/1505163368 shutdown_connections 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55786d66f1c0 0x55786db0e040 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55786d6db1b0 0x55786db0a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55786d6ba4d0 0x55786da87270 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 --2- 172.21.15.27:0/1505163368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55786d6b22f0 0x55786da86d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f06f3fff640 0 client.26725 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.652+0000 7f0721d74740 1 -- 172.21.15.27:0/1505163368 >> 172.21.15.27:0/1505163368 conn(0x55786d6cf300 msgr2=0x55786da76ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.653+0000 7f0721d74740 1 -- 172.21.15.27:0/1505163368 shutdown_connections 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.653+0000 7f0721d74740 1 -- 172.21.15.27:0/1505163368 wait complete. 2024-04-25T13:48:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.653+0000 7f0721d74740 20 client.26725 trim_cache size 0 max 16384 2024-04-25T13:48:51.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:48:51.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:48:51.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:48:51.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.654+0000 7f071aebd640 1 -- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07000afcf0 msgr2=0x7f07000b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.654+0000 7f071aebd640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07000afcf0 0x7f07000b2200 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0a4640 tx=0x7f06fc016660 comp rx=0 tx=0).stop 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.654+0000 7f071aebd640 1 -- 172.21.15.27:0/205328353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0714102430 msgr2=0x7f07140743d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.654+0000 7f071aebd640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0714102430 0x7f07140743d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00c3a0 tx=0x7f06fc00c860 comp rx=0 tx=0).stop 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.655+0000 7f071aebd640 1 -- 172.21.15.27:0/205328353 shutdown_connections 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.655+0000 7f071aebd640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f07000afcf0 0x7f07000b2200 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.655+0000 7f071aebd640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07141076f0 0x7f0714072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.655+0000 7f071aebd640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0714104d90 0x7f0714072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.655+0000 7f071aebd640 1 --2- 172.21.15.27:0/205328353 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0714102430 0x7f07140743d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.655+0000 7f071aebd640 1 -- 172.21.15.27:0/205328353 >> 172.21.15.27:0/205328353 conn(0x7f07140fdc60 msgr2=0x7f0714102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.655+0000 7f071aebd640 1 -- 172.21.15.27:0/205328353 shutdown_connections 2024-04-25T13:48:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:48:51.655+0000 7f071aebd640 1 -- 172.21.15.27:0/205328353 wait complete. 2024-04-25T13:48:52.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:51 smithi122 ceph-mon[32843]: pgmap v588: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.2 MiB/s wr, 654 op/s 2024-04-25T13:48:52.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:51 smithi045 ceph-mon[32731]: pgmap v588: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.2 MiB/s wr, 654 op/s 2024-04-25T13:48:52.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:51 smithi027 ceph-mon[29140]: pgmap v588: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.2 MiB/s wr, 654 op/s 2024-04-25T13:48:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 465.0 s, 1552.4 tps, lat 64.415 ms stddev 72.481 2024-04-25T13:48:54.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:53 smithi122 ceph-mon[32843]: pgmap v589: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.3 MiB/s wr, 695 op/s 2024-04-25T13:48:54.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:53 smithi045 ceph-mon[32731]: pgmap v589: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.3 MiB/s wr, 695 op/s 2024-04-25T13:48:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:53 smithi027 ceph-mon[29140]: pgmap v589: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.3 MiB/s wr, 695 op/s 2024-04-25T13:48:56.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:56 smithi122 ceph-mon[32843]: pgmap v590: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.3 MiB/s wr, 552 op/s 2024-04-25T13:48:56.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:55 smithi045 ceph-mon[32731]: pgmap v590: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.3 MiB/s wr, 552 op/s 2024-04-25T13:48:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:55 smithi027 ceph-mon[29140]: pgmap v590: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.3 MiB/s wr, 552 op/s 2024-04-25T13:48:57.916 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 470.0 s, 1922.4 tps, lat 51.989 ms stddev 58.560 2024-04-25T13:48:58.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:48:57 smithi122 ceph-mon[32843]: pgmap v591: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 MiB/s wr, 770 op/s 2024-04-25T13:48:58.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:48:57 smithi045 ceph-mon[32731]: pgmap v591: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 MiB/s wr, 770 op/s 2024-04-25T13:48:58.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:48:57 smithi027 ceph-mon[29140]: pgmap v591: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 MiB/s wr, 770 op/s 2024-04-25T13:49:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:00 smithi122 ceph-mon[32843]: pgmap v592: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.6 MiB/s wr, 569 op/s 2024-04-25T13:49:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:00 smithi045 ceph-mon[32731]: pgmap v592: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.6 MiB/s wr, 569 op/s 2024-04-25T13:49:00.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:00 smithi027 ceph-mon[29140]: pgmap v592: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.6 MiB/s wr, 569 op/s 2024-04-25T13:49:02.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:02 smithi122 ceph-mon[32843]: pgmap v593: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.7 MiB/s wr, 675 op/s 2024-04-25T13:49:02.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:02 smithi045 ceph-mon[32731]: pgmap v593: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.7 MiB/s wr, 675 op/s 2024-04-25T13:49:02.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:02 smithi027 ceph-mon[29140]: pgmap v593: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.7 MiB/s wr, 675 op/s 2024-04-25T13:49:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 475.0 s, 2206.6 tps, lat 45.174 ms stddev 50.776 2024-04-25T13:49:04.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:49:04.114 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:49:04.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.283+0000 7ff9f11b7640 1 -- 172.21.15.27:0/2217581132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ec10a420 msgr2=0x7ff9ec10add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.283+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/2217581132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ec10a420 0x7ff9ec10add0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc009860 tx=0x7ff9dc02a360 comp rx=0 tx=0).stop 2024-04-25T13:49:04.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.286+0000 7ff9f11b7640 1 -- 172.21.15.27:0/2217581132 shutdown_connections 2024-04-25T13:49:04.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.286+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/2217581132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ec110030 0x7ff9ec110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.286+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/2217581132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec10b310 0x7ff9ec10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.286+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/2217581132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ec10a420 0x7ff9ec10add0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.286+0000 7ff9f11b7640 1 -- 172.21.15.27:0/2217581132 >> 172.21.15.27:0/2217581132 conn(0x7ff9ec0704e0 msgr2=0x7ff9ec072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.293+0000 7ff9f11b7640 1 -- 172.21.15.27:0/2217581132 shutdown_connections 2024-04-25T13:49:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.296+0000 7ff9f11b7640 1 -- 172.21.15.27:0/2217581132 wait complete. 2024-04-25T13:49:04.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.297+0000 7ff9f11b7640 1 Processor -- start 2024-04-25T13:49:04.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.298+0000 7ff9f11b7640 1 -- start start 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.298+0000 7ff9f11b7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ec10a420 0x7ff9ec11b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.298+0000 7ff9f11b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec10b310 0x7ff9ec11bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.298+0000 7ff9f11b7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ec110030 0x7ff9ec119e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.298+0000 7ff9f11b7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec112380 con 0x7ff9ec10b310 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.298+0000 7ff9f11b7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec112200 con 0x7ff9ec10a420 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.298+0000 7ff9f11b7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec112500 con 0x7ff9ec110030 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.299+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ec10a420 0x7ff9ec11b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.299+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ec10a420 0x7ff9ec11b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39228/0 (socket says 172.21.15.27:39228) 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.299+0000 7ff9ead9d640 1 -- 172.21.15.27:0/488071203 learned_addr learned my addr 172.21.15.27:0/488071203 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.299+0000 7ff9ead9d640 1 -- 172.21.15.27:0/488071203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ec110030 msgr2=0x7ff9ec119e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.299+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ec110030 0x7ff9ec119e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.299+0000 7ff9ead9d640 1 -- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec10b310 msgr2=0x7ff9ec11bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.299+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec10b310 0x7ff9ec11bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.299+0000 7ff9ead9d640 1 -- 172.21.15.27:0/488071203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ec11a610 con 0x7ff9ec10a420 2024-04-25T13:49:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.300+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ec10a420 0x7ff9ec11b770 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc009780 tx=0x7ff9dc030d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.301+0000 7ff9d3fff640 1 -- 172.21.15.27:0/488071203 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc040030 con 0x7ff9ec10a420 2024-04-25T13:49:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.301+0000 7ff9f11b7640 1 -- 172.21.15.27:0/488071203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9ec11a8a0 con 0x7ff9ec10a420 2024-04-25T13:49:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.301+0000 7ff9f11b7640 1 -- 172.21.15.27:0/488071203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9ec1b9a10 con 0x7ff9ec10a420 2024-04-25T13:49:04.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.301+0000 7ff9d3fff640 1 -- 172.21.15.27:0/488071203 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc030450 con 0x7ff9ec10a420 2024-04-25T13:49:04.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.302+0000 7ff9d3fff640 1 -- 172.21.15.27:0/488071203 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc037040 con 0x7ff9ec10a420 2024-04-25T13:49:04.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:04 smithi122 ceph-mon[32843]: pgmap v594: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 MiB/s wr, 676 op/s 2024-04-25T13:49:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.310+0000 7ff9f806f740 1 Processor -- start 2024-04-25T13:49:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.311+0000 7ff9d3fff640 1 -- 172.21.15.27:0/488071203 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff9dc043070 con 0x7ff9ec10a420 2024-04-25T13:49:04.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.315+0000 7ff9d3fff640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff9d40afd00 0x7ff9d40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.315+0000 7ff9d3fff640 1 -- 172.21.15.27:0/488071203 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff9dc0fe860 con 0x7ff9ec10a420 2024-04-25T13:49:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.316+0000 7ff9eb59e640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff9d40afd00 0x7ff9d40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.317+0000 7ff9f806f740 1 -- start start 2024-04-25T13:49:04.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.317+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690dc4d0 0x5612694a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51264/0 (socket says 172.21.15.27:51264) 2024-04-25T13:49:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9ead9d640 1 -- 172.21.15.27:0/1057388394 learned_addr learned my addr 172.21.15.27:0/1057388394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690e46b0 0x5612694a7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9eb59e640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff9d40afd00 0x7ff9d40b2210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec11c350 tx=0x7ff9e4009710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9f806f740 1 -- 172.21.15.27:0/1057388394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56126940bd80 con 0x5612690e46b0 2024-04-25T13:49:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9f806f740 1 -- 172.21.15.27:0/1057388394 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5612693c58b0 con 0x5612690dc4d0 2024-04-25T13:49:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.318+0000 7ff9f806f740 1 -- 172.21.15.27:0/1057388394 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561269407d40 con 0x561269105390 2024-04-25T13:49:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.319+0000 7ff9eb59e640 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690e46b0 0x5612694a7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.319+0000 7ff9ea59c640 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690dc4d0 0x5612694a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.320+0000 7ff9ead9d640 1 -- 172.21.15.27:0/1057388394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690dc4d0 msgr2=0x5612694a6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.320+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690dc4d0 0x5612694a6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.320+0000 7ff9ead9d640 1 -- 172.21.15.27:0/1057388394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690e46b0 msgr2=0x5612694a7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.320+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690e46b0 0x5612694a7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.320+0000 7ff9ead9d640 1 -- 172.21.15.27:0/1057388394 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561269396a40 con 0x561269105390 2024-04-25T13:49:04.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.321+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a6a60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc030df0 tx=0x7ff9dc02a360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=376c933d14bb82c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.323+0000 7ff9d17fa640 1 -- 172.21.15.27:0/1057388394 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0c6d90 con 0x561269105390 2024-04-25T13:49:04.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.323+0000 7ff9d17fa640 1 -- 172.21.15.27:0/1057388394 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc0cb140 con 0x561269105390 2024-04-25T13:49:04.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.323+0000 7ff9d17fa640 1 -- 172.21.15.27:0/1057388394 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0cf760 con 0x561269105390 2024-04-25T13:49:04.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.326+0000 7ff9f806f740 1 -- 172.21.15.27:0/1057388394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 msgr2=0x5612694a6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.326+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a6a60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc030df0 tx=0x7ff9dc02a360 comp rx=0 tx=0).stop 2024-04-25T13:49:04.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.326+0000 7ff9f806f740 1 -- 172.21.15.27:0/1057388394 shutdown_connections 2024-04-25T13:49:04.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.326+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690e46b0 0x5612694a7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.326+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690dc4d0 0x5612694a6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.326+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1057388394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a6a60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.326+0000 7ff9f806f740 1 -- 172.21.15.27:0/1057388394 >> 172.21.15.27:0/1057388394 conn(0x5612690f94e0 msgr2=0x5612694a2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.327+0000 7ff9f806f740 1 -- 172.21.15.27:0/1057388394 shutdown_connections 2024-04-25T13:49:04.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.327+0000 7ff9f806f740 1 -- 172.21.15.27:0/1057388394 wait complete. 2024-04-25T13:49:04.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:04 smithi045 ceph-mon[32731]: pgmap v594: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 MiB/s wr, 676 op/s 2024-04-25T13:49:04.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.328+0000 7ff9f806f740 1 -- start start 2024-04-25T13:49:04.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.328+0000 7ff9f806f740 1 Processor -- start 2024-04-25T13:49:04.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 0x5612694b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9f806f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5612694a8540 con 0x5612690dc4d0 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9f806f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561269392d10 con 0x5612690e46b0 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9f806f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5612694a86c0 con 0x561269105390 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34718/0 (socket says 172.21.15.27:34718) 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9ead9d640 1 -- 172.21.15.27:0/3436031406 learned_addr learned my addr 172.21.15.27:0/3436031406 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9f806f740 10 client.0 _ll_register_callbacks cb 0x5612693bf210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9ea59c640 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 0x5612694b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.329+0000 7ff9eb59e640 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ead9d640 1 -- 172.21.15.27:0/3436031406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 msgr2=0x5612694b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694b79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ead9d640 1 -- 172.21.15.27:0/3436031406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 msgr2=0x5612694b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 0x5612694b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ead9d640 1 -- 172.21.15.27:0/3436031406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612691e77d0 con 0x5612690dc4d0 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9eb59e640 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b05a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0cfb50 tx=0x7ff9dc0cfc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc8c7781020cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ceff5640 1 -- 172.21.15.27:0/3436031406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0c6d90 con 0x5612690dc4d0 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ceff5640 1 -- 172.21.15.27:0/3436031406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff9dc0fc440 con 0x5612690dc4d0 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9f806f740 1 -- 172.21.15.27:0/3436031406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9dc0cb140 con 0x5612690dc4d0 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9ceff5640 1 -- 172.21.15.27:0/3436031406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff9dc0e93d0 con 0x5612690dc4d0 2024-04-25T13:49:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9f806f740 10 client.16977 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.330+0000 7ff9f806f740 1 -- 172.21.15.27:0/3436031406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56126911d3f0 con 0x5612690dc4d0 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.331+0000 7ff9ceff5640 1 -- 172.21.15.27:0/3436031406 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc0cb140 con 0x5612690dc4d0 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.331+0000 7ff9ceff5640 1 -- 172.21.15.27:0/3436031406 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff9dc0c6080 con 0x5612690dc4d0 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.331+0000 7ff9f806f740 10 client.16977 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.331+0000 7ff9f806f740 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.331+0000 7ff9f806f740 20 client.16977 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.331+0000 7ff9f806f740 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5612690c3f60 0x5612694bdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.331+0000 7ff9f806f740 4 client.16977 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.331+0000 7ff9f806f740 1 -- 172.21.15.27:0/3436031406 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561269419990 con 0x5612690c3f60 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.332+0000 7ff9ea59c640 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5612690c3f60 0x5612694bdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.332+0000 7ff9ceff5640 10 client.16977 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.332+0000 7ff9ea59c640 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5612690c3f60 0x5612694bdb60 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83b88fbd84e9d7ef server_cookie=4fd79b1668b0b080 in_seq=0 out_seq=0 2024-04-25T13:49:04.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9ceff5640 1 -- 172.21.15.27:0/3436031406 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561269419990 con 0x5612690c3f60 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9ceff5640 10 client.16977 handle_command_reply: tid=0 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 client.16977 shutdown 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 -- 172.21.15.27:0/3436031406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 msgr2=0x5612694b05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b05a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0cfb50 tx=0x7ff9dc0cfc60 comp rx=0 tx=0).stop 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 -- 172.21.15.27:0/3436031406 shutdown_connections 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5612690c3f60 0x5612694bdb60 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694b79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 0x5612694b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 --2- 172.21.15.27:0/3436031406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b05a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 -- 172.21.15.27:0/3436031406 >> 172.21.15.27:0/3436031406 conn(0x5612690f94e0 msgr2=0x5612694a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9ceff5640 0 client.16977 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 -- 172.21.15.27:0/3436031406 shutdown_connections 2024-04-25T13:49:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.333+0000 7ff9f806f740 1 -- 172.21.15.27:0/3436031406 wait complete. 2024-04-25T13:49:04.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.334+0000 7ff9f806f740 20 client.16977 trim_cache size 0 max 16384 2024-04-25T13:49:04.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9f806f740 1 Processor -- start 2024-04-25T13:49:04.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9f806f740 1 -- start start 2024-04-25T13:49:04.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 0x5612694a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9f806f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d800c380 con 0x5612690dc4d0 2024-04-25T13:49:04.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9f806f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d800ec00 con 0x5612690e46b0 2024-04-25T13:49:04.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9f806f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d800c9f0 con 0x561269105390 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694a1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34720/0 (socket says 172.21.15.27:34720) 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9ead9d640 1 -- 172.21.15.27:0/2296199247 learned_addr learned my addr 172.21.15.27:0/2296199247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9ea59c640 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 0x5612694a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.359+0000 7ff9eb59e640 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9ead9d640 1 -- 172.21.15.27:0/2296199247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 msgr2=0x5612694a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9ead9d640 1 -- 172.21.15.27:0/2296199247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 msgr2=0x5612694a6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 0x5612694a6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9ead9d640 1 -- 172.21.15.27:0/2296199247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612690dd0f0 con 0x5612690dc4d0 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694a1ca0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0c3f70 tx=0x7ff9dc02a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c0d9165ad5fb5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9d0ff9640 1 -- 172.21.15.27:0/2296199247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0c29b0 con 0x5612690dc4d0 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9d0ff9640 1 -- 172.21.15.27:0/2296199247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc0c6d90 con 0x5612690dc4d0 2024-04-25T13:49:04.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.360+0000 7ff9d0ff9640 1 -- 172.21.15.27:0/2296199247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0e9570 con 0x5612690dc4d0 2024-04-25T13:49:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 -- 172.21.15.27:0/2296199247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 msgr2=0x5612694a1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694a1ca0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0c3f70 tx=0x7ff9dc02a360 comp rx=0 tx=0).stop 2024-04-25T13:49:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 -- 172.21.15.27:0/2296199247 shutdown_connections 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561269105390 0x5612694a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5612690e46b0 0x5612694a6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 --2- 172.21.15.27:0/2296199247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694a1ca0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 -- 172.21.15.27:0/2296199247 >> 172.21.15.27:0/2296199247 conn(0x5612690f94e0 msgr2=0x56126949d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 -- 172.21.15.27:0/2296199247 shutdown_connections 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 -- 172.21.15.27:0/2296199247 wait complete. 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.361+0000 7ff9f806f740 1 -- start start 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9f806f740 1 Processor -- start 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5612690e46b0 0x5612694b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9f806f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561269105390 0x5612694b6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9f806f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d800d0c0 con 0x5612690dc4d0 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9f806f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d800e530 con 0x561269105390 2024-04-25T13:49:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9f806f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d800d790 con 0x5612690e46b0 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9f806f740 10 client.0 _ll_register_callbacks cb 0x7ff9e40095a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34732/0 (socket says 172.21.15.27:34732) 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9eb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561269105390 0x5612694b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 -- 172.21.15.27:0/1163440369 learned_addr learned my addr 172.21.15.27:0/1163440369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ea59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5612690e46b0 0x5612694b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 -- 172.21.15.27:0/1163440369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5612690e46b0 msgr2=0x5612694b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5612690e46b0 0x5612694b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 -- 172.21.15.27:0/1163440369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561269105390 msgr2=0x5612694b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561269105390 0x5612694b6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 -- 172.21.15.27:0/1163440369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612693cc4b0 con 0x5612690dc4d0 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ea59c640 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5612690e46b0 0x5612694b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9eb59e640 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561269105390 0x5612694b6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ead9d640 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b2eb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0c6d20 tx=0x7ff9dc0c2710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133184a2632fd3d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.362+0000 7ff9ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9ceff5640 1 -- 172.21.15.27:0/1163440369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0c6b60 con 0x5612690dc4d0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9ceff5640 1 -- 172.21.15.27:0/1163440369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff9dc0c32f0 con 0x5612690dc4d0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 1 -- 172.21.15.27:0/1163440369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9dc0c6d90 con 0x5612690dc4d0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9ceff5640 1 -- 172.21.15.27:0/1163440369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff9dc0c2bc0 con 0x5612690dc4d0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 10 client.16983 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 1 -- 172.21.15.27:0/1163440369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5612693cc4b0 con 0x5612690dc4d0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9ceff5640 1 -- 172.21.15.27:0/1163440369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc0c6d90 con 0x5612690dc4d0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9ceff5640 1 -- 172.21.15.27:0/1163440369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff9dc0c4ce0 con 0x5612690dc4d0 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 10 client.16983 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 20 client.16983 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5612690993a0 0x561269526860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 4 client.16983 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:49:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.363+0000 7ff9f806f740 1 -- 172.21.15.27:0/1163440369 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561269413800 con 0x5612690993a0 2024-04-25T13:49:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.364+0000 7ff9ea59c640 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5612690993a0 0x561269526860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.366+0000 7ff9ceff5640 10 client.16983 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.366+0000 7ff9ea59c640 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5612690993a0 0x561269526860 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=909da6027988aac server_cookie=479553549acfa3f3 in_seq=0 out_seq=0 2024-04-25T13:49:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9ceff5640 1 -- 172.21.15.27:0/1163440369 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561269413800 con 0x5612690993a0 2024-04-25T13:49:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9ceff5640 10 client.16983 handle_command_reply: tid=0 2024-04-25T13:49:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 client.16983 shutdown 2024-04-25T13:49:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 -- 172.21.15.27:0/1163440369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 msgr2=0x5612694b2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b2eb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0c6d20 tx=0x7ff9dc0c2710 comp rx=0 tx=0).stop 2024-04-25T13:49:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 -- 172.21.15.27:0/1163440369 shutdown_connections 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5612690993a0 0x561269526860 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561269105390 0x5612694b6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5612690e46b0 0x5612694b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 --2- 172.21.15.27:0/1163440369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5612690dc4d0 0x5612694b2eb0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9ceff5640 0 client.16983 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 -- 172.21.15.27:0/1163440369 >> 172.21.15.27:0/1163440369 conn(0x5612690f94e0 msgr2=0x5612694a6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 -- 172.21.15.27:0/1163440369 shutdown_connections 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 1 -- 172.21.15.27:0/1163440369 wait complete. 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.367+0000 7ff9f806f740 20 client.16983 trim_cache size 0 max 16384 2024-04-25T13:49:04.370 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:49:04.371 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:49:04.371 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:49:04.371 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:49:04.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 -- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff9d40afd00 msgr2=0x7ff9d40b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff9d40afd00 0x7ff9d40b2210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec11c350 tx=0x7ff9e4009710 comp rx=0 tx=0).stop 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 -- 172.21.15.27:0/488071203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ec10a420 msgr2=0x7ff9ec11b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ec10a420 0x7ff9ec11b770 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc009780 tx=0x7ff9dc030d80 comp rx=0 tx=0).stop 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 -- 172.21.15.27:0/488071203 shutdown_connections 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff9d40afd00 0x7ff9d40b2210 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9ec110030 0x7ff9ec119e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec10b310 0x7ff9ec11bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 --2- 172.21.15.27:0/488071203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9ec10a420 0x7ff9ec11b770 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 -- 172.21.15.27:0/488071203 >> 172.21.15.27:0/488071203 conn(0x7ff9ec0704e0 msgr2=0x7ff9ec072950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 -- 172.21.15.27:0/488071203 shutdown_connections 2024-04-25T13:49:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.368+0000 7ff9f11b7640 1 -- 172.21.15.27:0/488071203 wait complete. 2024-04-25T13:49:04.379 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:04 smithi027 ceph-mon[29140]: pgmap v594: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 MiB/s wr, 676 op/s 2024-04-25T13:49:04.381 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:49:04.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:49:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 -- 172.21.15.27:0/3519256957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0710108900 msgr2=0x7f071010ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 --2- 172.21.15.27:0/3519256957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0710108900 0x7f071010ad20 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0708009ca0 tx=0x7f070802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 -- 172.21.15.27:0/3519256957 shutdown_connections 2024-04-25T13:49:04.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 --2- 172.21.15.27:0/3519256957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0710108900 0x7f071010ad20 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 --2- 172.21.15.27:0/3519256957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0710103c00 0x7f0710104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 --2- 172.21.15.27:0/3519256957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07101029c0 0x7f07101036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 -- 172.21.15.27:0/3519256957 >> 172.21.15.27:0/3519256957 conn(0x7f07100fdbd0 msgr2=0x7f0710100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 -- 172.21.15.27:0/3519256957 shutdown_connections 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 -- 172.21.15.27:0/3519256957 wait complete. 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 Processor -- start 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.529+0000 7f07187a8640 1 -- start start 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f07187a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07101029c0 0x7f071006a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f07187a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0710103c00 0x7f071019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f07187a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0710108900 0x7f07101a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f07187a8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f071010e4d0 con 0x7f07101029c0 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f07187a8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f071010e350 con 0x7f0710103c00 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f07187a8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f071010e650 con 0x7f0710108900 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0716544640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07101029c0 0x7f071006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0715d43640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0710103c00 0x7f071019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0716544640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07101029c0 0x7f071006a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34760/0 (socket says 172.21.15.27:34760) 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0716544640 1 -- 172.21.15.27:0/3370343846 learned_addr learned my addr 172.21.15.27:0/3370343846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0716d45640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0710108900 0x7f07101a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0715d43640 1 -- 172.21.15.27:0/3370343846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0710108900 msgr2=0x7f07101a14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0715d43640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0710108900 0x7f07101a14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0715d43640 1 -- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07101029c0 msgr2=0x7f071006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0715d43640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07101029c0 0x7f071006a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0715d43640 1 -- 172.21.15.27:0/3370343846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07101a1cd0 con 0x7f0710103c00 2024-04-25T13:49:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.530+0000 7f0716544640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07101029c0 0x7f071006a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.531+0000 7f0715d43640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0710103c00 0x7f071019ef00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f070000e9f0 tx=0x7f070000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.532+0000 7f07077fe640 1 -- 172.21.15.27:0/3370343846 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070000ccc0 con 0x7f0710103c00 2024-04-25T13:49:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.532+0000 7f07077fe640 1 -- 172.21.15.27:0/3370343846 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07000108e0 con 0x7f0710103c00 2024-04-25T13:49:04.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.532+0000 7f07077fe640 1 -- 172.21.15.27:0/3370343846 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700018880 con 0x7f0710103c00 2024-04-25T13:49:04.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.533+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07101a1fc0 con 0x7f0710103c00 2024-04-25T13:49:04.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.533+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07101a2430 con 0x7f0710103c00 2024-04-25T13:49:04.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.534+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f071006af80 con 0x7f0710103c00 2024-04-25T13:49:04.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.535+0000 7f07077fe640 1 -- 172.21.15.27:0/3370343846 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0700010430 con 0x7f0710103c00 2024-04-25T13:49:04.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.536+0000 7f07077fe640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06fc0afc40 0x7f06fc0b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.536+0000 7f07077fe640 1 -- 172.21.15.27:0/3370343846 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f07000dee30 con 0x7f0710103c00 2024-04-25T13:49:04.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.536+0000 7f0716544640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06fc0afc40 0x7f06fc0b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.537+0000 7f0716544640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06fc0afc40 0x7f06fc0b2150 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f070c009fd0 tx=0x7f070c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.541+0000 7f07077fe640 1 -- 172.21.15.27:0/3370343846 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07000a73f0 con 0x7f0710103c00 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.809+0000 7f82979ce640 1 -- 172.21.15.27:0/2434153526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290108960 msgr2=0x7f829010ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.809+0000 7f82979ce640 1 --2- 172.21.15.27:0/2434153526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290108960 0x7f829010ad80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8288009be0 tx=0x7f828802a6f0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.809+0000 7f82979ce640 1 -- 172.21.15.27:0/2434153526 shutdown_connections 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.809+0000 7f82979ce640 1 --2- 172.21.15.27:0/2434153526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290108960 0x7f829010ad80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.809+0000 7f82979ce640 1 --2- 172.21.15.27:0/2434153526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8290103c60 0x7f8290104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.809+0000 7f82979ce640 1 --2- 172.21.15.27:0/2434153526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8290102a20 0x7f8290103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.809+0000 7f82979ce640 1 -- 172.21.15.27:0/2434153526 >> 172.21.15.27:0/2434153526 conn(0x7f82900fdc10 msgr2=0x7f82901000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.810+0000 7f82979ce640 1 -- 172.21.15.27:0/2434153526 shutdown_connections 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.810+0000 7f82979ce640 1 -- 172.21.15.27:0/2434153526 wait complete. 2024-04-25T13:49:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.810+0000 7f82979ce640 1 Processor -- start 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.810+0000 7f82979ce640 1 -- start start 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f82979ce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290102a20 0x7f8290198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f82979ce640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8290103c60 0x7f8290198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f82979ce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8290108960 0x7f829019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f82979ce640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f829010d4a0 con 0x7f8290108960 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f82979ce640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f829010d320 con 0x7f8290102a20 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f82979ce640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f829010d620 con 0x7f8290103c60 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f829576a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290102a20 0x7f8290198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f829576a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290102a20 0x7f8290198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39288/0 (socket says 172.21.15.27:39288) 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f829576a640 1 -- 172.21.15.27:0/4123850211 learned_addr learned my addr 172.21.15.27:0/4123850211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f8295f6b640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8290108960 0x7f829019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f8294f69640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8290103c60 0x7f8290198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f8294f69640 1 -- 172.21.15.27:0/4123850211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290102a20 msgr2=0x7f8290198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f8294f69640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290102a20 0x7f8290198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f8294f69640 1 -- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8290108960 msgr2=0x7f829019f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f8294f69640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8290108960 0x7f829019f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f8294f69640 1 -- 172.21.15.27:0/4123850211 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829019fb60 con 0x7f8290103c60 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.811+0000 7f8295f6b640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8290108960 0x7f829019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.812+0000 7f8294f69640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8290103c60 0x7f8290198560 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f828000b3f0 tx=0x7f828000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.812+0000 7f82867fc640 1 -- 172.21.15.27:0/4123850211 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280015020 con 0x7f8290103c60 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.812+0000 7f82867fc640 1 -- 172.21.15.27:0/4123850211 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8280011ce0 con 0x7f8290103c60 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.812+0000 7f82979ce640 1 -- 172.21.15.27:0/4123850211 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829019fe50 con 0x7f8290103c60 2024-04-25T13:49:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.812+0000 7f82867fc640 1 -- 172.21.15.27:0/4123850211 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280010d80 con 0x7f8290103c60 2024-04-25T13:49:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.813+0000 7f82979ce640 1 -- 172.21.15.27:0/4123850211 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82901a0340 con 0x7f8290103c60 2024-04-25T13:49:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.814+0000 7f82867fc640 1 -- 172.21.15.27:0/4123850211 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8280018020 con 0x7f8290103c60 2024-04-25T13:49:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.815+0000 7f82867fc640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f827c0b3ee0 0x7f827c0b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.815+0000 7f829e882740 1 Processor -- start 2024-04-25T13:49:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.815+0000 7f829e882740 1 -- start start 2024-04-25T13:49:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f82867fc640 1 -- 172.21.15.27:0/4123850211 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f82800de340 con 0x7f8290103c60 2024-04-25T13:49:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829e882740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65eb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829e882740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 0x5635e65eb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829e882740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829e882740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5635e6548460 con 0x5635e62421b0 2024-04-25T13:49:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829e882740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5635e6502590 con 0x5635e62192f0 2024-04-25T13:49:04.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829e882740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5635e6544420 con 0x5635e62214d0 2024-04-25T13:49:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829576a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8295f6b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8295f6b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51356/0 (socket says 172.21.15.27:51356) 2024-04-25T13:49:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829576a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65eb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34788/0 (socket says 172.21.15.27:34788) 2024-04-25T13:49:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8295f6b640 1 -- 172.21.15.27:0/3301040487 learned_addr learned my addr 172.21.15.27:0/3301040487 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8294f69640 1 --2- 172.21.15.27:0/3301040487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 0x5635e65eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f829576a640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f827c0b3ee0 0x7f827c0b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8295f6b640 1 -- 172.21.15.27:0/3301040487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 msgr2=0x5635e65eb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8295f6b640 1 --2- 172.21.15.27:0/3301040487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 0x5635e65eb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8295f6b640 1 -- 172.21.15.27:0/3301040487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 msgr2=0x5635e65eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8295f6b640 1 --2- 172.21.15.27:0/3301040487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65eb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.816+0000 7f8295f6b640 1 -- 172.21.15.27:0/3301040487 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e650f0d0 con 0x5635e62214d0 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.817+0000 7f8295f6b640 1 --2- 172.21.15.27:0/3301040487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f2600 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f828802ace0 tx=0x7f82880093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b03cee5a9f97f7a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.817+0000 7f829576a640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f827c0b3ee0 0x7f827c0b63f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x5635e65ec350 tx=0x7f828c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.817+0000 7f827b7fe640 1 -- 172.21.15.27:0/3301040487 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8288041030 con 0x5635e62214d0 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.817+0000 7f827b7fe640 1 -- 172.21.15.27:0/3301040487 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8288003e20 con 0x5635e62214d0 2024-04-25T13:49:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f827b7fe640 1 -- 172.21.15.27:0/3301040487 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8288036040 con 0x5635e62214d0 2024-04-25T13:49:04.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 -- 172.21.15.27:0/3301040487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 msgr2=0x5635e65f2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 --2- 172.21.15.27:0/3301040487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f2600 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f828802ace0 tx=0x7f82880093e0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 -- 172.21.15.27:0/3301040487 shutdown_connections 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 --2- 172.21.15.27:0/3301040487 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f2600 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 --2- 172.21.15.27:0/3301040487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 0x5635e65eb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 --2- 172.21.15.27:0/3301040487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 -- 172.21.15.27:0/3301040487 >> 172.21.15.27:0/3301040487 conn(0x5635e6236300 msgr2=0x5635e65de300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 -- 172.21.15.27:0/3301040487 shutdown_connections 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.818+0000 7f829e882740 1 -- 172.21.15.27:0/3301040487 wait complete. 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.819+0000 7f829e882740 1 -- start start 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.819+0000 7f829e882740 1 Processor -- start 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.819+0000 7f829e882740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 0x5635e65f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.819+0000 7f829e882740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.819+0000 7f829e882740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.819+0000 7f829e882740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5635e65f3090 con 0x5635e62421b0 2024-04-25T13:49:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.819+0000 7f829e882740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5635e64cfb60 con 0x5635e62192f0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.819+0000 7f829e882740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5635e65f3210 con 0x5635e62214d0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f829e882740 10 client.0 _ll_register_callbacks cb 0x5635e64f42b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51368/0 (socket says 172.21.15.27:51368) 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 -- 172.21.15.27:0/2582201279 learned_addr learned my addr 172.21.15.27:0/2582201279 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f829576a640 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 0x5635e65f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8295f6b640 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 -- 172.21.15.27:0/2582201279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 msgr2=0x5635e65f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 0x5635e65f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 -- 172.21.15.27:0/2582201279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 msgr2=0x5635e65fc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 -- 172.21.15.27:0/2582201279 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e625a210 con 0x5635e62214d0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8294f69640 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f5310 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f82800dbec0 tx=0x7f82800107e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa56694a95109e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8278ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8278ff9640 1 -- 172.21.15.27:0/2582201279 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82800c0060 con 0x5635e62214d0 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f8278ff9640 1 -- 172.21.15.27:0/2582201279 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f82800a2ae0 con 0x5635e62214d0 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.820+0000 7f829e882740 1 -- 172.21.15.27:0/2582201279 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8288003e20 con 0x5635e62214d0 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f8278ff9640 1 -- 172.21.15.27:0/2582201279 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f82800a6360 con 0x5635e62214d0 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829e882740 10 client.26758 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829e882740 1 -- 172.21.15.27:0/2582201279 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5635e6326620 con 0x5635e62214d0 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f8278ff9640 1 -- 172.21.15.27:0/2582201279 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8288003e20 con 0x5635e62214d0 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f8278ff9640 1 -- 172.21.15.27:0/2582201279 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f828008f0f0 con 0x5635e62214d0 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829e882740 10 client.26758 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829e882740 10 client.26758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829e882740 20 client.26758 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829e882740 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635e6200d80 0x5635e6602b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829e882740 4 client.26758 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829e882740 1 -- 172.21.15.27:0/2582201279 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635e6556070 con 0x5635e6200d80 2024-04-25T13:49:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.821+0000 7f829576a640 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635e6200d80 0x5635e6602b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.823+0000 7f829576a640 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635e6200d80 0x5635e6602b20 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6199817cc3a3cfd8 server_cookie=7a7df8f75ed263ef in_seq=0 out_seq=0 2024-04-25T13:49:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.823+0000 7f8278ff9640 10 client.26758 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.824+0000 7f8278ff9640 1 -- 172.21.15.27:0/2582201279 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5635e6556070 con 0x5635e6200d80 2024-04-25T13:49:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.824+0000 7f8278ff9640 10 client.26758 handle_command_reply: tid=0 2024-04-25T13:49:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.824+0000 7f829e882740 1 client.26758 shutdown 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.824+0000 7f829e882740 1 -- 172.21.15.27:0/2582201279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 msgr2=0x5635e65f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.824+0000 7f829e882740 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f5310 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f82800dbec0 tx=0x7f82800107e0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.824+0000 7f829e882740 1 -- 172.21.15.27:0/2582201279 shutdown_connections 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.824+0000 7f829e882740 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635e6200d80 0x5635e6602b20 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.825+0000 7f829e882740 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.825+0000 7f829e882740 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62214d0 0x5635e65f5310 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f82800dbec0 tx=0x7f82800107e0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.825+0000 7f829e882740 1 --2- 172.21.15.27:0/2582201279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e62192f0 0x5635e65f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.825+0000 7f829e882740 1 -- 172.21.15.27:0/2582201279 >> 172.21.15.27:0/2582201279 conn(0x5635e6236300 msgr2=0x5635e65dd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.825+0000 7f8278ff9640 0 client.26758 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.825+0000 7f829e882740 1 -- 172.21.15.27:0/2582201279 shutdown_connections 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.825+0000 7f829e882740 1 -- 172.21.15.27:0/2582201279 wait complete. 2024-04-25T13:49:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.825+0000 7f829e882740 20 client.26758 trim_cache size 0 max 16384 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829e882740 1 Processor -- start 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829e882740 1 -- start start 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829e882740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62192f0 0x5635e65ddcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829e882740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829e882740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65de200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829e882740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82901a7040 con 0x5635e62421b0 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829e882740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f82901a98c0 con 0x5635e61d61c0 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829e882740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f82901a76b0 con 0x5635e62192f0 2024-04-25T13:49:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f8294f69640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f8294f69640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34800/0 (socket says 172.21.15.27:34800) 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f829576a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62192f0 0x5635e65ddcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f8294f69640 1 -- 172.21.15.27:0/2287115153 learned_addr learned my addr 172.21.15.27:0/2287115153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.845+0000 7f8295f6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65de200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f8294f69640 1 -- 172.21.15.27:0/2287115153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62192f0 msgr2=0x5635e65ddcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f8294f69640 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62192f0 0x5635e65ddcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f8294f69640 1 -- 172.21.15.27:0/2287115153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 msgr2=0x5635e65de200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f8294f69640 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65de200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f8294f69640 1 -- 172.21.15.27:0/2287115153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e65d8f40 con 0x5635e62421b0 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f829576a640 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62192f0 0x5635e65ddcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f8295f6b640 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65de200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f8294f69640 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc230 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f828000bdb0 tx=0x7f8280010380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a72c0765db46e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f827affd640 1 -- 172.21.15.27:0/2287115153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82800a6850 con 0x5635e62421b0 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f827affd640 1 -- 172.21.15.27:0/2287115153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82800aac00 con 0x5635e62421b0 2024-04-25T13:49:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.846+0000 7f827affd640 1 -- 172.21.15.27:0/2287115153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828008f0f0 con 0x5635e62421b0 2024-04-25T13:49:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.847+0000 7f829e882740 1 -- 172.21.15.27:0/2287115153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 msgr2=0x5635e65fc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.847+0000 7f829e882740 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc230 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f828000bdb0 tx=0x7f8280010380 comp rx=0 tx=0).stop 2024-04-25T13:49:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.849+0000 7f829e882740 1 -- 172.21.15.27:0/2287115153 shutdown_connections 2024-04-25T13:49:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.849+0000 7f829e882740 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65de200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.849+0000 7f829e882740 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62421b0 0x5635e65fc230 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.849+0000 7f829e882740 1 --2- 172.21.15.27:0/2287115153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62192f0 0x5635e65ddcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.849+0000 7f829e882740 1 -- 172.21.15.27:0/2287115153 >> 172.21.15.27:0/2287115153 conn(0x5635e6236300 msgr2=0x5635e65ef210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.849+0000 7f829e882740 1 -- 172.21.15.27:0/2287115153 shutdown_connections 2024-04-25T13:49:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.849+0000 7f829e882740 1 -- 172.21.15.27:0/2287115153 wait complete. 2024-04-25T13:49:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.849+0000 7f829e882740 1 -- start start 2024-04-25T13:49:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.850+0000 7f829e882740 1 Processor -- start 2024-04-25T13:49:04.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.850+0000 7f829e882740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65f9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.850+0000 7f829e882740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62192f0 0x5635e65f9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.850+0000 7f829e882740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62421b0 0x5635e65fa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f829576a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65f9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f829e882740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f82901a7d80 con 0x5635e62192f0 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f829e882740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f82901a91f0 con 0x5635e61d61c0 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f829576a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65f9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39322/0 (socket says 172.21.15.27:39322) 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f829576a640 1 -- 172.21.15.27:0/1076390116 learned_addr learned my addr 172.21.15.27:0/1076390116 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f82901a8450 con 0x5635e62421b0 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8294f69640 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62192f0 0x5635e65f9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8295f6b640 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62421b0 0x5635e65fa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f829e882740 10 client.0 _ll_register_callbacks cb 0x7f8280010f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8295f6b640 1 -- 172.21.15.27:0/1076390116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 msgr2=0x5635e65f9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8295f6b640 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65f9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8295f6b640 1 -- 172.21.15.27:0/1076390116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62192f0 msgr2=0x5635e65f9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8295f6b640 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62192f0 0x5635e65f9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8295f6b640 1 -- 172.21.15.27:0/1076390116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e65fae30 con 0x5635e62421b0 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f829576a640 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8295f6b640 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62421b0 0x5635e65fa870 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8288009b30 tx=0x7f828802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12f3245165de8926 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.851+0000 7f8278ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f8278ff9640 1 -- 172.21.15.27:0/1076390116 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828804e020 con 0x5635e62421b0 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f8278ff9640 1 -- 172.21.15.27:0/1076390116 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8288003b70 con 0x5635e62421b0 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82800aac00 con 0x5635e62421b0 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f8278ff9640 1 -- 172.21.15.27:0/1076390116 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f8288036070 con 0x5635e62421b0 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f829e882740 10 client.26764 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5635e65fc0a0 con 0x5635e62421b0 2024-04-25T13:49:04.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f8278ff9640 1 -- 172.21.15.27:0/1076390116 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82800aac00 con 0x5635e62421b0 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f8278ff9640 1 -- 172.21.15.27:0/1076390116 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f828803d460 con 0x5635e62421b0 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f829e882740 10 client.26764 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.852+0000 7f829e882740 10 client.26764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.853+0000 7f829e882740 20 client.26764 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.853+0000 7f829e882740 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635e667bf30 0x5635e667e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.853+0000 7f829e882740 4 client.26764 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.853+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635e654fee0 con 0x5635e667bf30 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.853+0000 7f829576a640 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635e667bf30 0x5635e667e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.853+0000 7f8278ff9640 10 client.26764 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:04.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.853+0000 7f829576a640 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635e667bf30 0x5635e667e350 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad6049b1e38e227c server_cookie=cf5444228d3cbea1 in_seq=0 out_seq=0 2024-04-25T13:49:04.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.885+0000 7f8278ff9640 1 -- 172.21.15.27:0/1076390116 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635e654fee0 con 0x5635e667bf30 2024-04-25T13:49:04.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.885+0000 7f8278ff9640 10 client.26764 handle_command_reply: tid=0 2024-04-25T13:49:04.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.885+0000 7f829e882740 1 client.26764 shutdown 2024-04-25T13:49:04.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.885+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62421b0 msgr2=0x5635e65fa870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.885+0000 7f829e882740 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62421b0 0x5635e65fa870 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8288009b30 tx=0x7f828802a6f0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.885+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 shutdown_connections 2024-04-25T13:49:04.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.885+0000 7f829e882740 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5635e667bf30 0x5635e667e350 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f829e882740 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5635e62421b0 0x5635e65fa870 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f8278ff9640 0 client.26764 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:04.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f829e882740 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635e62192f0 0x5635e65f9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f829e882740 1 --2- 172.21.15.27:0/1076390116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e61d61c0 0x5635e65f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 >> 172.21.15.27:0/1076390116 conn(0x5635e6236300 msgr2=0x5635e65ddcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 shutdown_connections 2024-04-25T13:49:04.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f829e882740 1 -- 172.21.15.27:0/1076390116 wait complete. 2024-04-25T13:49:04.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f829e882740 20 client.26764 trim_cache size 0 max 16384 2024-04-25T13:49:04.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:49:04.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:49:04.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:49:04.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:49:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f82979ce640 1 -- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f827c0b3ee0 msgr2=0x7f827c0b63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.886+0000 7f82979ce640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f827c0b3ee0 0x7f827c0b63f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x5635e65ec350 tx=0x7f828c009260 comp rx=0 tx=0).stop 2024-04-25T13:49:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 -- 172.21.15.27:0/4123850211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8290103c60 msgr2=0x7f8290198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8290103c60 0x7f8290198560 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f828000b3f0 tx=0x7f828000b8b0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 -- 172.21.15.27:0/4123850211 shutdown_connections 2024-04-25T13:49:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f827c0b3ee0 0x7f827c0b63f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8290108960 0x7f829019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8290103c60 0x7f8290198560 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 --2- 172.21.15.27:0/4123850211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8290102a20 0x7f8290198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 -- 172.21.15.27:0/4123850211 >> 172.21.15.27:0/4123850211 conn(0x7f82900fdc10 msgr2=0x7f8290104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 -- 172.21.15.27:0/4123850211 shutdown_connections 2024-04-25T13:49:04.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:04.887+0000 7f82979ce640 1 -- 172.21.15.27:0/4123850211 wait complete. 2024-04-25T13:49:04.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.896+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0710107710 con 0x7f0710103c00 2024-04-25T13:49:04.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.898+0000 7f07077fe640 1 -- 172.21.15.27:0/3370343846 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f07000dce20 con 0x7f0710103c00 2024-04-25T13:49:04.898 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:49:04.899 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:49:04.899 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:49:04.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.909+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06fc0afc40 msgr2=0x7f06fc0b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.909+0000 7f07187a8640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06fc0afc40 0x7f06fc0b2150 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f070c009fd0 tx=0x7f070c005f90 comp rx=0 tx=0).stop 2024-04-25T13:49:04.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.909+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0710103c00 msgr2=0x7f071019ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:04.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.909+0000 7f07187a8640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0710103c00 0x7f071019ef00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f070000e9f0 tx=0x7f070000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.916+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 shutdown_connections 2024-04-25T13:49:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.916+0000 7f07187a8640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06fc0afc40 0x7f06fc0b2150 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.916+0000 7f07187a8640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0710108900 0x7f07101a14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.916+0000 7f07187a8640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0710103c00 0x7f071019ef00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.916+0000 7f07187a8640 1 --2- 172.21.15.27:0/3370343846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07101029c0 0x7f071006a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.916+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 >> 172.21.15.27:0/3370343846 conn(0x7f07100fdbd0 msgr2=0x7f07101048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:04.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.919+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 shutdown_connections 2024-04-25T13:49:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:04.921+0000 7f07187a8640 1 -- 172.21.15.27:0/3370343846 wait complete. 2024-04-25T13:49:04.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:49:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 -- 172.21.15.27:0/530829694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe950108930 msgr2=0x7fe95010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 --2- 172.21.15.27:0/530829694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe950108930 0x7fe95010ad50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe948009be0 tx=0x7fe94802a6c0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 -- 172.21.15.27:0/530829694 shutdown_connections 2024-04-25T13:49:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 --2- 172.21.15.27:0/530829694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe950108930 0x7fe95010ad50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 --2- 172.21.15.27:0/530829694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe950103c30 0x7fe950104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 --2- 172.21.15.27:0/530829694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9501029f0 0x7fe9501036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 -- 172.21.15.27:0/530829694 >> 172.21.15.27:0/530829694 conn(0x7fe9500fdc00 msgr2=0x7fe950100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 -- 172.21.15.27:0/530829694 shutdown_connections 2024-04-25T13:49:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.084+0000 7fe956bf8640 1 -- 172.21.15.27:0/530829694 wait complete. 2024-04-25T13:49:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe956bf8640 1 Processor -- start 2024-04-25T13:49:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe956bf8640 1 -- start start 2024-04-25T13:49:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe956bf8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9501029f0 0x7fe9501a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe956bf8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe950103c30 0x7fe9501a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe956bf8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe950108930 0x7fe9501a7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe956bf8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe95010ce30 con 0x7fe950103c30 2024-04-25T13:49:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe956bf8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe95010ccb0 con 0x7fe950108930 2024-04-25T13:49:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe956bf8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe95010cfb0 con 0x7fe9501029f0 2024-04-25T13:49:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe950103c30 0x7fe9501a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe954994640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9501029f0 0x7fe9501a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe955195640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe950108930 0x7fe9501a7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe954994640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9501029f0 0x7fe9501a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51410/0 (socket says 172.21.15.27:51410) 2024-04-25T13:49:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe954994640 1 -- 172.21.15.27:0/2438239788 learned_addr learned my addr 172.21.15.27:0/2438239788 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe950103c30 0x7fe9501a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34830/0 (socket says 172.21.15.27:34830) 2024-04-25T13:49:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe954994640 1 -- 172.21.15.27:0/2438239788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe950108930 msgr2=0x7fe9501a7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe954994640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe950108930 0x7fe9501a7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe954994640 1 -- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe950103c30 msgr2=0x7fe9501a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe954994640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe950103c30 0x7fe9501a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe954994640 1 -- 172.21.15.27:0/2438239788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9501a8320 con 0x7fe9501029f0 2024-04-25T13:49:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.085+0000 7fe94ffff640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe950103c30 0x7fe9501a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.086+0000 7fe954994640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9501029f0 0x7fe9501a07e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe94400e9f0 tx=0x7fe94400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.086+0000 7fe94dffb640 1 -- 172.21.15.27:0/2438239788 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94400ccc0 con 0x7fe9501029f0 2024-04-25T13:49:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.086+0000 7fe94dffb640 1 -- 172.21.15.27:0/2438239788 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9440035f0 con 0x7fe9501029f0 2024-04-25T13:49:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.086+0000 7fe956bf8640 1 -- 172.21.15.27:0/2438239788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9501a8610 con 0x7fe9501029f0 2024-04-25T13:49:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.086+0000 7fe94dffb640 1 -- 172.21.15.27:0/2438239788 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94400ce60 con 0x7fe9501029f0 2024-04-25T13:49:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.086+0000 7fe956bf8640 1 -- 172.21.15.27:0/2438239788 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9501a8bb0 con 0x7fe9501029f0 2024-04-25T13:49:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe95db89740 1 Processor -- start 2024-04-25T13:49:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe95db89740 1 -- start start 2024-04-25T13:49:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe94dffb640 1 -- 172.21.15.27:0/2438239788 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe944004020 con 0x7fe9501029f0 2024-04-25T13:49:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe95db89740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940d3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe95db89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 0x55c0940d3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe95db89740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d106b0 0x55c0940d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe95db89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c094037d80 con 0x55c093d084d0 2024-04-25T13:49:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe95db89740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c093ff1a20 con 0x55c093d106b0 2024-04-25T13:49:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.089+0000 7fe95db89740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55c094033d40 con 0x55c093d31390 2024-04-25T13:49:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe955195640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940d3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d106b0 0x55c0940d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe954994640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 0x55c0940d3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe955195640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940d3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51412/0 (socket says 172.21.15.27:51412) 2024-04-25T13:49:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe955195640 1 -- 172.21.15.27:0/1164852050 learned_addr learned my addr 172.21.15.27:0/1164852050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe94ffff640 1 -- 172.21.15.27:0/1164852050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 msgr2=0x55c0940d3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe94ffff640 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940d3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe94ffff640 1 -- 172.21.15.27:0/1164852050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 msgr2=0x55c0940d3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe94ffff640 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 0x55c0940d3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe94ffff640 1 -- 172.21.15.27:0/1164852050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c093ffcc30 con 0x55c093d106b0 2024-04-25T13:49:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe955195640 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940d3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe954994640 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 0x55c0940d3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe94dffb640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe9400afb70 0x7fe9400b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe94ffff640 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d106b0 0x55c0940d4a20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009530 tx=0x7fe93c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a2525cb11142a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.090+0000 7fe955195640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe9400afb70 0x7fe9400b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe94dffb640 1 -- 172.21.15.27:0/2438239788 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe9440dee40 con 0x7fe9501029f0 2024-04-25T13:49:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe93affd640 1 -- 172.21.15.27:0/1164852050 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c005020 con 0x55c093d106b0 2024-04-25T13:49:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe93affd640 1 -- 172.21.15.27:0/1164852050 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe93c0047a0 con 0x55c093d106b0 2024-04-25T13:49:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe93affd640 1 -- 172.21.15.27:0/1164852050 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c00f850 con 0x55c093d106b0 2024-04-25T13:49:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe95db89740 1 -- 172.21.15.27:0/1164852050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d106b0 msgr2=0x55c0940d4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe955195640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe9400afb70 0x7fe9400b2080 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe948009420 tx=0x7fe948030990 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe95db89740 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d106b0 0x55c0940d4a20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009530 tx=0x7fe93c0118d0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe95db89740 1 -- 172.21.15.27:0/1164852050 shutdown_connections 2024-04-25T13:49:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe95db89740 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d106b0 0x55c0940d4a20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe95db89740 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 0x55c0940d3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe95db89740 1 --2- 172.21.15.27:0/1164852050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940d3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.091+0000 7fe95db89740 1 -- 172.21.15.27:0/1164852050 >> 172.21.15.27:0/1164852050 conn(0x55c093d254e0 msgr2=0x55c0940cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.092+0000 7fe95db89740 1 -- 172.21.15.27:0/1164852050 shutdown_connections 2024-04-25T13:49:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.092+0000 7fe95db89740 1 -- 172.21.15.27:0/1164852050 wait complete. 2024-04-25T13:49:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.092+0000 7fe95db89740 1 -- start start 2024-04-25T13:49:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.092+0000 7fe95db89740 1 Processor -- start 2024-04-25T13:49:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe95db89740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe95db89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940ddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe95db89740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe95db89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c0940d5530 con 0x55c093d106b0 2024-04-25T13:49:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe95db89740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c093fbedc0 con 0x55c093d084d0 2024-04-25T13:49:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe95db89740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55c0940d56b0 con 0x55c093d31390 2024-04-25T13:49:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe954994640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe954994640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39372/0 (socket says 172.21.15.27:39372) 2024-04-25T13:49:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe954994640 1 -- 172.21.15.27:0/3804843551 learned_addr learned my addr 172.21.15.27:0/3804843551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe95db89740 10 client.0 _ll_register_callbacks cb 0x55c093fda600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.093+0000 7fe94ffff640 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940ddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.094+0000 7fe955195640 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.094+0000 7fe954994640 1 -- 172.21.15.27:0/3804843551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 msgr2=0x55c0940e4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.094+0000 7fe954994640 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940e4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.094+0000 7fe954994640 1 -- 172.21.15.27:0/3804843551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 msgr2=0x55c0940ddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.094+0000 7fe954994640 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940ddaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.094+0000 7fe954994640 1 -- 172.21.15.27:0/3804843551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c093d493f0 con 0x55c093d084d0 2024-04-25T13:49:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.095+0000 7fe92ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:49:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.095+0000 7fe954994640 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd5b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe9440a45c0 tx=0x7fe9440a32d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1c3798c69b7d23d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.095+0000 7fe92ffff640 1 -- 172.21.15.27:0/3804843551 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9440a6ac0 con 0x55c093d084d0 2024-04-25T13:49:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.095+0000 7fe95db89740 1 -- 172.21.15.27:0/3804843551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe93c0047a0 con 0x55c093d084d0 2024-04-25T13:49:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.095+0000 7fe92ffff640 1 -- 172.21.15.27:0/3804843551 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe9440a3940 con 0x55c093d084d0 2024-04-25T13:49:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.095+0000 7fe92ffff640 1 -- 172.21.15.27:0/3804843551 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe9440a6ac0 con 0x55c093d084d0 2024-04-25T13:49:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe95db89740 10 client.26729 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe95db89740 1 -- 172.21.15.27:0/3804843551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55c093e137d0 con 0x55c093d084d0 2024-04-25T13:49:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe92ffff640 1 -- 172.21.15.27:0/3804843551 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe93c0047a0 con 0x55c093d084d0 2024-04-25T13:49:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe92ffff640 1 -- 172.21.15.27:0/3804843551 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe9440e0030 con 0x55c093d084d0 2024-04-25T13:49:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe95db89740 10 client.26729 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe95db89740 10 client.26729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe95db89740 20 client.26729 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe95db89740 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c093ceff60 0x55c0940eb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe95db89740 4 client.26729 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe95db89740 1 -- 172.21.15.27:0/3804843551 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c094045990 con 0x55c093ceff60 2024-04-25T13:49:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.096+0000 7fe94ffff640 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c093ceff60 0x55c0940eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.097+0000 7fe94ffff640 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c093ceff60 0x55c0940eb240 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=871ba43e330b8a0d server_cookie=411d850a6d9ddc47 in_seq=0 out_seq=0 2024-04-25T13:49:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.097+0000 7fe92ffff640 10 client.26729 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.097+0000 7fe92ffff640 1 -- 172.21.15.27:0/3804843551 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c094045990 con 0x55c093ceff60 2024-04-25T13:49:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.097+0000 7fe92ffff640 10 client.26729 handle_command_reply: tid=0 2024-04-25T13:49:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.097+0000 7fe95db89740 1 client.26729 shutdown 2024-04-25T13:49:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 -- 172.21.15.27:0/3804843551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 msgr2=0x55c0940dd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd5b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe9440a45c0 tx=0x7fe9440a32d0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 -- 172.21.15.27:0/3804843551 shutdown_connections 2024-04-25T13:49:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c093ceff60 0x55c0940eb240 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe92ffff640 0 client.26729 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940ddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 --2- 172.21.15.27:0/3804843551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd5b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 -- 172.21.15.27:0/3804843551 >> 172.21.15.27:0/3804843551 conn(0x55c093d254e0 msgr2=0x55c0940ce040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:05.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 -- 172.21.15.27:0/3804843551 shutdown_connections 2024-04-25T13:49:05.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 1 -- 172.21.15.27:0/3804843551 wait complete. 2024-04-25T13:49:05.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.098+0000 7fe95db89740 20 client.26729 trim_cache size 0 max 16384 2024-04-25T13:49:05.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.125+0000 7fe95db89740 1 Processor -- start 2024-04-25T13:49:05.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.127+0000 7fe95db89740 1 -- start start 2024-04-25T13:49:05.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.127+0000 7fe95db89740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.127+0000 7fe95db89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940dd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.127+0000 7fe95db89740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940ef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.127+0000 7fe95db89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe9501af890 con 0x55c093d106b0 2024-04-25T13:49:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.127+0000 7fe95db89740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9501b2170 con 0x55c093d084d0 2024-04-25T13:49:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.127+0000 7fe95db89740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe9501aff60 con 0x55c093d31390 2024-04-25T13:49:05.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.128+0000 7fe955195640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940ef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.131+0000 7fe954994640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.131+0000 7fe954994640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39382/0 (socket says 172.21.15.27:39382) 2024-04-25T13:49:05.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.131+0000 7fe954994640 1 -- 172.21.15.27:0/1551357926 learned_addr learned my addr 172.21.15.27:0/1551357926 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.132+0000 7fe94ffff640 1 --2- 172.21.15.27:0/1551357926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940dd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.133+0000 7fe94ffff640 1 -- 172.21.15.27:0/1551357926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 msgr2=0x55c0940ef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.133+0000 7fe94ffff640 1 --2- 172.21.15.27:0/1551357926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940ef9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.133+0000 7fe94ffff640 1 -- 172.21.15.27:0/1551357926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 msgr2=0x55c0940dd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.133+0000 7fe94ffff640 1 --2- 172.21.15.27:0/1551357926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.133+0000 7fe94ffff640 1 -- 172.21.15.27:0/1551357926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0940d4490 con 0x55c093d106b0 2024-04-25T13:49:05.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.133+0000 7fe94ffff640 1 --2- 172.21.15.27:0/1551357926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940dd6c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe93c01b040 tx=0x7fe93c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51831ad726a3215b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:05.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.134+0000 7fe93a7fc640 1 -- 172.21.15.27:0/1551357926 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c01d040 con 0x55c093d106b0 2024-04-25T13:49:05.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.134+0000 7fe93a7fc640 1 -- 172.21.15.27:0/1551357926 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe93c004660 con 0x55c093d106b0 2024-04-25T13:49:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.135+0000 7fe95db89740 1 -- 172.21.15.27:0/1551357926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 msgr2=0x55c0940dd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.135+0000 7fe95db89740 1 --2- 172.21.15.27:0/1551357926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940dd6c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe93c01b040 tx=0x7fe93c005850 comp rx=0 tx=0).stop 2024-04-25T13:49:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.140+0000 7fe95db89740 1 -- 172.21.15.27:0/1551357926 shutdown_connections 2024-04-25T13:49:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.140+0000 7fe95db89740 1 --2- 172.21.15.27:0/1551357926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d31390 0x55c0940ef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.140+0000 7fe95db89740 1 --2- 172.21.15.27:0/1551357926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d106b0 0x55c0940dd6c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.140+0000 7fe95db89740 1 --2- 172.21.15.27:0/1551357926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d084d0 0x55c0940dd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.140+0000 7fe95db89740 1 -- 172.21.15.27:0/1551357926 >> 172.21.15.27:0/1551357926 conn(0x55c093d254e0 msgr2=0x55c0940d8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.145+0000 7fe95db89740 1 -- 172.21.15.27:0/1551357926 shutdown_connections 2024-04-25T13:49:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.145+0000 7fe95db89740 1 -- 172.21.15.27:0/1551357926 wait complete. 2024-04-25T13:49:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.146+0000 7fe95db89740 1 -- start start 2024-04-25T13:49:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.146+0000 7fe95db89740 1 Processor -- start 2024-04-25T13:49:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.147+0000 7fe95db89740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 0x55c0941500d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.147+0000 7fe95db89740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d106b0 0x55c094150610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.147+0000 7fe95db89740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d31390 0x55c094162580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.147+0000 7fe95db89740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe9501b0630 con 0x55c093d084d0 2024-04-25T13:49:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.147+0000 7fe95db89740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe9501b1aa0 con 0x55c093d31390 2024-04-25T13:49:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.147+0000 7fe95db89740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe9501b0d00 con 0x55c093d106b0 2024-04-25T13:49:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.147+0000 7fe95db89740 10 client.0 _ll_register_callbacks cb 0x7fe9440e2610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.147+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d106b0 0x55c094150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.148+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d106b0 0x55c094150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51432/0 (socket says 172.21.15.27:51432) 2024-04-25T13:49:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.148+0000 7fe94ffff640 1 -- 172.21.15.27:0/484925993 learned_addr learned my addr 172.21.15.27:0/484925993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.148+0000 7fe955195640 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d31390 0x55c094162580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.148+0000 7fe94ffff640 1 -- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d31390 msgr2=0x55c094162580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.148+0000 7fe94ffff640 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d31390 0x55c094162580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.148+0000 7fe94ffff640 1 -- 172.21.15.27:0/484925993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 msgr2=0x55c0941500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.148+0000 7fe94ffff640 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 0x55c0941500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.149+0000 7fe94ffff640 1 -- 172.21.15.27:0/484925993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c094162af0 con 0x55c093d106b0 2024-04-25T13:49:05.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.149+0000 7fe955195640 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d31390 0x55c094162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:49:05.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.149+0000 7fe94ffff640 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d106b0 0x55c094150610 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00a010 tx=0x7fe93c005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d79353e0b4acca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:05.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.150+0000 7fe92ffff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:05.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.152+0000 7fe92ffff640 1 -- 172.21.15.27:0/484925993 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c01d030 con 0x55c093d106b0 2024-04-25T13:49:05.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.152+0000 7fe95db89740 1 -- 172.21.15.27:0/484925993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe93c004660 con 0x55c093d106b0 2024-04-25T13:49:05.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.152+0000 7fe92ffff640 1 -- 172.21.15.27:0/484925993 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe93c005110 con 0x55c093d106b0 2024-04-25T13:49:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.155+0000 7fe92ffff640 1 -- 172.21.15.27:0/484925993 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe93c009c40 con 0x55c093d106b0 2024-04-25T13:49:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.155+0000 7fe95db89740 10 client.26782 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.155+0000 7fe95db89740 1 -- 172.21.15.27:0/484925993 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55c094163d50 con 0x55c093d106b0 2024-04-25T13:49:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.156+0000 7fe92ffff640 1 -- 172.21.15.27:0/484925993 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe93c004660 con 0x55c093d106b0 2024-04-25T13:49:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.157+0000 7fe92ffff640 1 -- 172.21.15.27:0/484925993 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe93c00f4e0 con 0x55c093d106b0 2024-04-25T13:49:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.158+0000 7fe95db89740 10 client.26782 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.158+0000 7fe95db89740 10 client.26782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.158+0000 7fe95db89740 20 client.26782 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.158+0000 7fe95db89740 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c093cc53a0 0x55c094166400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.158+0000 7fe95db89740 4 client.26782 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:49:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.158+0000 7fe95db89740 1 -- 172.21.15.27:0/484925993 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c09403f800 con 0x55c093cc53a0 2024-04-25T13:49:05.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.159+0000 7fe954994640 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c093cc53a0 0x55c094166400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:05.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.161+0000 7fe954994640 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c093cc53a0 0x55c094166400 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=226babe00c13f55b server_cookie=b4c691985de9e634 in_seq=0 out_seq=0 2024-04-25T13:49:05.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.162+0000 7fe92ffff640 10 client.26782 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.162+0000 7fe92ffff640 1 -- 172.21.15.27:0/484925993 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c09403f800 con 0x55c093cc53a0 2024-04-25T13:49:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.162+0000 7fe92ffff640 10 client.26782 handle_command_reply: tid=0 2024-04-25T13:49:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.162+0000 7fe95db89740 1 client.26782 shutdown 2024-04-25T13:49:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 -- 172.21.15.27:0/484925993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d106b0 msgr2=0x55c094150610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d106b0 0x55c094150610 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00a010 tx=0x7fe93c005970 comp rx=0 tx=0).stop 2024-04-25T13:49:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 -- 172.21.15.27:0/484925993 shutdown_connections 2024-04-25T13:49:05.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c093cc53a0 0x55c094166400 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c093d31390 0x55c094162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c093d106b0 0x55c094150610 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 --2- 172.21.15.27:0/484925993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c093d084d0 0x55c0941500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe92ffff640 0 client.26782 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 -- 172.21.15.27:0/484925993 >> 172.21.15.27:0/484925993 conn(0x55c093d254e0 msgr2=0x55c0940d1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 -- 172.21.15.27:0/484925993 shutdown_connections 2024-04-25T13:49:05.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.164+0000 7fe95db89740 1 -- 172.21.15.27:0/484925993 wait complete. 2024-04-25T13:49:05.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe95db89740 20 client.26782 trim_cache size 0 max 16384 2024-04-25T13:49:05.165 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 -- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe9400afb70 msgr2=0x7fe9400b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe9400afb70 0x7fe9400b2080 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe948009420 tx=0x7fe948030990 comp rx=0 tx=0).stop 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 -- 172.21.15.27:0/2438239788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9501029f0 msgr2=0x7fe9501a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9501029f0 0x7fe9501a07e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe94400e9f0 tx=0x7fe94400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 -- 172.21.15.27:0/2438239788 shutdown_connections 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe9400afb70 0x7fe9400b2080 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe950108930 0x7fe9501a7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe950103c30 0x7fe9501a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 --2- 172.21.15.27:0/2438239788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe9501029f0 0x7fe9501a07e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.165+0000 7fe956bf8640 1 -- 172.21.15.27:0/2438239788 >> 172.21.15.27:0/2438239788 conn(0x7fe9500fdc00 msgr2=0x7fe950100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.166+0000 7fe956bf8640 1 -- 172.21.15.27:0/2438239788 shutdown_connections 2024-04-25T13:49:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:05.166+0000 7fe956bf8640 1 -- 172.21.15.27:0/2438239788 wait complete. 2024-04-25T13:49:05.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:05 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3370343846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:49:05.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:05 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3370343846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:49:05.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:05 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3370343846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:49:06.181 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:49:06.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:49:06.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:49:06.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:06 smithi122 ceph-mon[32843]: pgmap v595: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.5 MiB/s wr, 580 op/s 2024-04-25T13:49:06.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:06 smithi045 ceph-mon[32731]: pgmap v595: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.5 MiB/s wr, 580 op/s 2024-04-25T13:49:06.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.353+0000 7f69fcf8f640 1 -- 172.21.15.27:0/3390568100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f81029f0 msgr2=0x7f69f81036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.353+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/3390568100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f81029f0 0x7f69f81036f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009900 tx=0x7f69e802a410 comp rx=0 tx=0).stop 2024-04-25T13:49:06.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 -- 172.21.15.27:0/3390568100 shutdown_connections 2024-04-25T13:49:06.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/3390568100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f8108930 0x7f69f810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/3390568100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f8103c30 0x7f69f8104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/3390568100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f81029f0 0x7f69f81036f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 -- 172.21.15.27:0/3390568100 >> 172.21.15.27:0/3390568100 conn(0x7f69f80fdc00 msgr2=0x7f69f8100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 -- 172.21.15.27:0/3390568100 shutdown_connections 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 -- 172.21.15.27:0/3390568100 wait complete. 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 Processor -- start 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.354+0000 7f69fcf8f640 1 -- start start 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69fcf8f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f81029f0 0x7f69f806bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69fcf8f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f8103c30 0x7f69f819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69fcf8f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8108930 0x7f69f81a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69fcf8f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69f810e4f0 con 0x7f69f8108930 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69fcf8f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69f810e370 con 0x7f69f8103c30 2024-04-25T13:49:06.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69fcf8f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f69f810e670 con 0x7f69f81029f0 2024-04-25T13:49:06.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69f6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8108930 0x7f69f81a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69f6d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8108930 0x7f69f81a1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34876/0 (socket says 172.21.15.27:34876) 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.355+0000 7f69f6d9d640 1 -- 172.21.15.27:0/2819738588 learned_addr learned my addr 172.21.15.27:0/2819738588 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.356+0000 7f69effff640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f8103c30 0x7f69f819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.356+0000 7f69f6d9d640 1 -- 172.21.15.27:0/2819738588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f81029f0 msgr2=0x7f69f806bcc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.356+0000 7f69f6d9d640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f81029f0 0x7f69f806bcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.356+0000 7f69f6d9d640 1 -- 172.21.15.27:0/2819738588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f8103c30 msgr2=0x7f69f819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.356+0000 7f69f6d9d640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f8103c30 0x7f69f819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.356+0000 7f69f6d9d640 1 -- 172.21.15.27:0/2819738588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f81a1d70 con 0x7f69f8108930 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.356+0000 7f69effff640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f8103c30 0x7f69f819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.357+0000 7f69f6d9d640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8108930 0x7f69f81a1550 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f69f000eb10 tx=0x7f69f000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.357+0000 7f69ef7fe640 1 -- 172.21.15.27:0/2819738588 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0004b20 con 0x7f69f8108930 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.357+0000 7f69ef7fe640 1 -- 172.21.15.27:0/2819738588 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69f0004cc0 con 0x7f69f8108930 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.357+0000 7f69fcf8f640 1 -- 172.21.15.27:0/2819738588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f81a2060 con 0x7f69f8108930 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.357+0000 7f69ef7fe640 1 -- 172.21.15.27:0/2819738588 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0004b20 con 0x7f69f8108930 2024-04-25T13:49:06.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.358+0000 7f69fcf8f640 1 -- 172.21.15.27:0/2819738588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f8199010 con 0x7f69f8108930 2024-04-25T13:49:06.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.360+0000 7f69ef7fe640 1 -- 172.21.15.27:0/2819738588 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f69f0010490 con 0x7f69f8108930 2024-04-25T13:49:06.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.360+0000 7f6a03e41740 1 Processor -- start 2024-04-25T13:49:06.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f6a03e41740 1 -- start start 2024-04-25T13:49:06.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b072a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b072aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0731920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f6a03e41740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5593b068fb00 con 0x5593b0368730 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f6a03e41740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5593b062b920 con 0x5593b0360550 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f6a03e41740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5593b068bac0 con 0x5593b0389410 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f69effff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0731920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f69f659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b072aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f69effff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0731920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34880/0 (socket says 172.21.15.27:34880) 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f69effff640 1 -- 172.21.15.27:0/1676185107 learned_addr learned my addr 172.21.15.27:0/1676185107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f69effff640 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b072a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.361+0000 7f69ef7fe640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69e40afcf0 0x7f69e40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69f659c640 1 -- 172.21.15.27:0/1676185107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 msgr2=0x5593b072a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69f659c640 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b072a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69f659c640 1 -- 172.21.15.27:0/1676185107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 msgr2=0x5593b0731920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69f659c640 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0731920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69f659c640 1 -- 172.21.15.27:0/1676185107 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593b04697e0 con 0x5593b0360550 2024-04-25T13:49:06.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69effff640 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b072a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:49:06.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69f6d9d640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69e40afcf0 0x7f69e40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69ef7fe640 1 -- 172.21.15.27:0/2819738588 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f69f00deee0 con 0x7f69f8108930 2024-04-25T13:49:06.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69f659c640 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b072aa20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f69e802a930 tx=0x7f69e8005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6ff72d31d9cf97c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69ecff9640 1 -- 172.21.15.27:0/1676185107 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e8035040 con 0x5593b0360550 2024-04-25T13:49:06.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69ecff9640 1 -- 172.21.15.27:0/1676185107 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e8032e20 con 0x5593b0360550 2024-04-25T13:49:06.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.362+0000 7f69ecff9640 1 -- 172.21.15.27:0/1676185107 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e8004a40 con 0x5593b0360550 2024-04-25T13:49:06.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.363+0000 7f6a03e41740 1 -- 172.21.15.27:0/1676185107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 msgr2=0x5593b072aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.363+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b072aa20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f69e802a930 tx=0x7f69e8005e50 comp rx=0 tx=0).stop 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.363+0000 7f69f6d9d640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69e40afcf0 0x7f69e40b2200 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f69f00a70f0 tx=0x7f69f009bd30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.363+0000 7f6a03e41740 1 -- 172.21.15.27:0/1676185107 shutdown_connections 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.363+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0731920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.363+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b072aa20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.363+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1676185107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b072a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.363+0000 7f6a03e41740 1 -- 172.21.15.27:0/1676185107 >> 172.21.15.27:0/1676185107 conn(0x5593b037d560 msgr2=0x5593b0725e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 -- 172.21.15.27:0/1676185107 shutdown_connections 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 -- 172.21.15.27:0/1676185107 wait complete. 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 -- start start 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 Processor -- start 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07342f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0734830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b073b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5593b0732310 con 0x5593b0368730 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5593b06295d0 con 0x5593b0360550 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.364+0000 7f6a03e41740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5593b0732490 con 0x5593b0389410 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f6a03e41740 10 client.0 _ll_register_callbacks cb 0x5593b0617080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b073b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b073b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51472/0 (socket says 172.21.15.27:51472) 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 -- 172.21.15.27:0/767903906 learned_addr learned my addr 172.21.15.27:0/767903906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69f659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0734830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 -- 172.21.15.27:0/767903906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 msgr2=0x5593b07342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07342f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 -- 172.21.15.27:0/767903906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 msgr2=0x5593b0734830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0734830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 -- 172.21.15.27:0/767903906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593b04c9f70 con 0x5593b0389410 2024-04-25T13:49:06.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69effff640 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b073b720 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f69e000f440 tx=0x7f69e0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bef341b9acfd2d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69ddffb640 1 -- 172.21.15.27:0/767903906 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e000fe80 con 0x5593b0389410 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f69ddffb640 1 -- 172.21.15.27:0/767903906 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f69e0005020 con 0x5593b0389410 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.365+0000 7f6a03e41740 1 -- 172.21.15.27:0/767903906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69e8032e20 con 0x5593b0389410 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f69ddffb640 1 -- 172.21.15.27:0/767903906 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f69e0005220 con 0x5593b0389410 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f6a03e41740 10 client.26791 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f6a03e41740 1 -- 172.21.15.27:0/767903906 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5593b0361170 con 0x5593b0389410 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f69ddffb640 1 -- 172.21.15.27:0/767903906 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e8032e20 con 0x5593b0389410 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f69ddffb640 1 -- 172.21.15.27:0/767903906 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f69e000e5d0 con 0x5593b0389410 2024-04-25T13:49:06.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f6a03e41740 10 client.26791 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:06.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f6a03e41740 10 client.26791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:06.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f6a03e41740 20 client.26791 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:06.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f6a03e41740 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5593b03481f0 0x5593b0741b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f6a03e41740 4 client.26791 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:06.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f6a03e41740 1 -- 172.21.15.27:0/767903906 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593b069d710 con 0x5593b03481f0 2024-04-25T13:49:06.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.366+0000 7f69f659c640 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5593b03481f0 0x5593b0741b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.376+0000 7f69f659c640 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5593b03481f0 0x5593b0741b80 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=148631699a139bb3 server_cookie=359e68df0f7527c2 in_seq=0 out_seq=0 2024-04-25T13:49:06.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.377+0000 7f69ddffb640 10 client.26791 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:06.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.378+0000 7f69ddffb640 1 -- 172.21.15.27:0/767903906 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5593b069d710 con 0x5593b03481f0 2024-04-25T13:49:06.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.378+0000 7f69ddffb640 10 client.26791 handle_command_reply: tid=0 2024-04-25T13:49:06.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.379+0000 7f6a03e41740 1 client.26791 shutdown 2024-04-25T13:49:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.381+0000 7f6a03e41740 1 -- 172.21.15.27:0/767903906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 msgr2=0x5593b073b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.381+0000 7f6a03e41740 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b073b720 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f69e000f440 tx=0x7f69e0004730 comp rx=0 tx=0).stop 2024-04-25T13:49:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.383+0000 7f6a03e41740 1 -- 172.21.15.27:0/767903906 shutdown_connections 2024-04-25T13:49:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.383+0000 7f6a03e41740 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5593b03481f0 0x5593b0741b80 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.383+0000 7f6a03e41740 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0389410 0x5593b073b720 secure :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f69e000f440 tx=0x7f69e0004730 comp rx=0 tx=0).stop 2024-04-25T13:49:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.383+0000 7f6a03e41740 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0734830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.383+0000 7f6a03e41740 1 --2- 172.21.15.27:0/767903906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.383+0000 7f6a03e41740 1 -- 172.21.15.27:0/767903906 >> 172.21.15.27:0/767903906 conn(0x5593b037d560 msgr2=0x5593b0724ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.384+0000 7f69ddffb640 0 client.26791 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.387+0000 7f6a03e41740 1 -- 172.21.15.27:0/767903906 shutdown_connections 2024-04-25T13:49:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.390+0000 7f6a03e41740 1 -- 172.21.15.27:0/767903906 wait complete. 2024-04-25T13:49:06.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.393+0000 7f6a03e41740 20 client.26791 trim_cache size 0 max 16384 2024-04-25T13:49:06.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:06 smithi027 ceph-mon[29140]: pgmap v595: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.5 MiB/s wr, 580 op/s 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f6a03e41740 1 Processor -- start 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f6a03e41740 1 -- start start 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0360550 0x5593b072ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0725d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b031d630 0x5593b073b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f6a03e41740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81ab360 con 0x5593b0368730 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f6a03e41740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81adbe0 con 0x5593b031d630 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f6a03e41740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81ab9d0 con 0x5593b0360550 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f69f659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0725d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f69effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0360550 0x5593b072ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f69effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0360550 0x5593b072ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51488/0 (socket says 172.21.15.27:51488) 2024-04-25T13:49:06.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f69effff640 1 -- 172.21.15.27:0/3810328723 learned_addr learned my addr 172.21.15.27:0/3810328723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f69f659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0725d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34892/0 (socket says 172.21.15.27:34892) 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.428+0000 7f69effff640 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b031d630 0x5593b073b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69effff640 1 -- 172.21.15.27:0/3810328723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b031d630 msgr2=0x5593b073b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69effff640 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b031d630 0x5593b073b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69effff640 1 -- 172.21.15.27:0/3810328723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 msgr2=0x5593b0725d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69effff640 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0725d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69effff640 1 -- 172.21.15.27:0/3810328723 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593b0348420 con 0x5593b0360550 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69f659c640 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0725d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69effff640 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0360550 0x5593b072ee40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f69e0001eb0 tx=0x7f69e0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52118e4c34e560bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69dffff640 1 -- 172.21.15.27:0/3810328723 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e000fe80 con 0x5593b0360550 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69dffff640 1 -- 172.21.15.27:0/3810328723 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e000e5d0 con 0x5593b0360550 2024-04-25T13:49:06.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.429+0000 7f69dffff640 1 -- 172.21.15.27:0/3810328723 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e0004c50 con 0x5593b0360550 2024-04-25T13:49:06.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.430+0000 7f6a03e41740 1 -- 172.21.15.27:0/3810328723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0360550 msgr2=0x5593b072ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.430+0000 7f6a03e41740 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0360550 0x5593b072ee40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f69e0001eb0 tx=0x7f69e0005f90 comp rx=0 tx=0).stop 2024-04-25T13:49:06.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.430+0000 7f6a03e41740 1 -- 172.21.15.27:0/3810328723 shutdown_connections 2024-04-25T13:49:06.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.430+0000 7f6a03e41740 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b031d630 0x5593b073b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.430+0000 7f6a03e41740 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b0368730 0x5593b0725d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.430+0000 7f6a03e41740 1 --2- 172.21.15.27:0/3810328723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0360550 0x5593b072ee40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.430+0000 7f6a03e41740 1 -- 172.21.15.27:0/3810328723 >> 172.21.15.27:0/3810328723 conn(0x5593b037d560 msgr2=0x5593b0725620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.431+0000 7f6a03e41740 1 -- 172.21.15.27:0/3810328723 shutdown_connections 2024-04-25T13:49:06.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.432+0000 7f6a03e41740 1 -- 172.21.15.27:0/3810328723 wait complete. 2024-04-25T13:49:06.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.432+0000 7f6a03e41740 1 -- start start 2024-04-25T13:49:06.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.433+0000 7f6a03e41740 1 Processor -- start 2024-04-25T13:49:06.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.433+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b031d630 0x5593b0738ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.433+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.433+0000 7f6a03e41740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0368730 0x5593b07b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f6a03e41740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81ac0a0 con 0x5593b031d630 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69effff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b031d630 0x5593b0738ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f6a03e41740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81ad510 con 0x5593b0360550 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69effff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b031d630 0x5593b0738ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34898/0 (socket says 172.21.15.27:34898) 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f6a03e41740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81ac770 con 0x5593b0368730 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69effff640 1 -- 172.21.15.27:0/1407842841 learned_addr learned my addr 172.21.15.27:0/1407842841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f6a03e41740 10 client.0 _ll_register_callbacks cb 0x7f69e0011d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69f659c640 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69effff640 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0368730 0x5593b07b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69f659c640 1 -- 172.21.15.27:0/1407842841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0368730 msgr2=0x5593b07b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69f659c640 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0368730 0x5593b07b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69f659c640 1 -- 172.21.15.27:0/1407842841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b031d630 msgr2=0x5593b0738ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69f659c640 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b031d630 0x5593b0738ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69f659c640 1 -- 172.21.15.27:0/1407842841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593b07b0c30 con 0x5593b0360550 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.434+0000 7f69effff640 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0368730 0x5593b07b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.435+0000 7f69f659c640 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07393e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f69e8006010 tx=0x7f69e802ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3c08c403d70f5d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.435+0000 7f69ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.435+0000 7f69ddffb640 1 -- 172.21.15.27:0/1407842841 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e8053070 con 0x5593b0360550 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.435+0000 7f69ddffb640 1 -- 172.21.15.27:0/1407842841 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f69e8004040 con 0x5593b0360550 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.435+0000 7f6a03e41740 1 -- 172.21.15.27:0/1407842841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69e000e5d0 con 0x5593b0360550 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.435+0000 7f69ddffb640 1 -- 172.21.15.27:0/1407842841 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f69e8035040 con 0x5593b0360550 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.435+0000 7f6a03e41740 10 client.26750 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:06.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.435+0000 7f6a03e41740 1 -- 172.21.15.27:0/1407842841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5593b07b1e40 con 0x5593b0360550 2024-04-25T13:49:06.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f69ddffb640 1 -- 172.21.15.27:0/1407842841 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e000e5d0 con 0x5593b0360550 2024-04-25T13:49:06.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f69ddffb640 1 -- 172.21.15.27:0/1407842841 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f69e804f3b0 con 0x5593b0360550 2024-04-25T13:49:06.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f6a03e41740 10 client.26750 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:06.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f6a03e41740 10 client.26750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:06.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f6a03e41740 20 client.26750 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:06.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5593b07b2480 0x5593b07b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f6a03e41740 4 client.26750 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f6a03e41740 1 -- 172.21.15.27:0/1407842841 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5593b0697580 con 0x5593b07b2480 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.436+0000 7f69effff640 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5593b07b2480 0x5593b07b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.437+0000 7f69ddffb640 10 client.26750 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.437+0000 7f69effff640 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5593b07b2480 0x5593b07b48a0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80b1d525b37c1ce4 server_cookie=42612567e63a111a in_seq=0 out_seq=0 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f69ddffb640 1 -- 172.21.15.27:0/1407842841 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5593b0697580 con 0x5593b07b2480 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f69ddffb640 10 client.26750 handle_command_reply: tid=0 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 client.26750 shutdown 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 -- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 msgr2=0x5593b07393e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07393e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f69e8006010 tx=0x7f69e802ab10 comp rx=0 tx=0).stop 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 -- 172.21.15.27:0/1407842841 shutdown_connections 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5593b07b2480 0x5593b07b48a0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5593b0368730 0x5593b07b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593b0360550 0x5593b07393e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 --2- 172.21.15.27:0/1407842841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5593b031d630 0x5593b0738ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f6a03e41740 1 -- 172.21.15.27:0/1407842841 >> 172.21.15.27:0/1407842841 conn(0x5593b037d560 msgr2=0x5593b07244c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.438+0000 7f69ddffb640 0 client.26750 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:06.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.439+0000 7f6a03e41740 1 -- 172.21.15.27:0/1407842841 shutdown_connections 2024-04-25T13:49:06.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.439+0000 7f6a03e41740 1 -- 172.21.15.27:0/1407842841 wait complete. 2024-04-25T13:49:06.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.439+0000 7f6a03e41740 20 client.26750 trim_cache size 0 max 16384 2024-04-25T13:49:06.441 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:49:06.441 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:49:06.441 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "965e0032-8c1d-44bc-be71-7f9fb0c454e8", 2024-04-25T13:49:06.441 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:49:06.441 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 -- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69e40afcf0 msgr2=0x7f69e40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69e40afcf0 0x7f69e40b2200 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f69f00a70f0 tx=0x7f69f009bd30 comp rx=0 tx=0).stop 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 -- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8108930 msgr2=0x7f69f81a1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8108930 0x7f69f81a1550 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f69f000eb10 tx=0x7f69f000c2d0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 -- 172.21.15.27:0/2819738588 shutdown_connections 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f69e40afcf0 0x7f69e40b2200 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69f8108930 0x7f69f81a1550 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69f8103c30 0x7f69f819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 --2- 172.21.15.27:0/2819738588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f81029f0 0x7f69f806bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 -- 172.21.15.27:0/2819738588 >> 172.21.15.27:0/2819738588 conn(0x7f69f80fdc00 msgr2=0x7f69f8104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 -- 172.21.15.27:0/2819738588 shutdown_connections 2024-04-25T13:49:06.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.440+0000 7f69fcf8f640 1 -- 172.21.15.27:0/2819738588 wait complete. 2024-04-25T13:49:06.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:49:06.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:49:06.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.640+0000 7f44f4f2d640 1 -- 172.21.15.27:0/2587197711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f0103c20 msgr2=0x7f44f0104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.640+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/2587197711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f0103c20 0x7f44f0104030 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009900 tx=0x7f44e002a440 comp rx=0 tx=0).stop 2024-04-25T13:49:06.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.643+0000 7f44f4f2d640 1 -- 172.21.15.27:0/2587197711 shutdown_connections 2024-04-25T13:49:06.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.643+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/2587197711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44f0069e90 0x7f44f006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.643+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/2587197711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f0103c20 0x7f44f0104030 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.643+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/2587197711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f01029e0 0x7f44f01036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.643+0000 7f44f4f2d640 1 -- 172.21.15.27:0/2587197711 >> 172.21.15.27:0/2587197711 conn(0x7f44f00fdbd0 msgr2=0x7f44f0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.644+0000 7f44f4f2d640 1 -- 172.21.15.27:0/2587197711 shutdown_connections 2024-04-25T13:49:06.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.644+0000 7f44f4f2d640 1 -- 172.21.15.27:0/2587197711 wait complete. 2024-04-25T13:49:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.646+0000 7f44f4f2d640 1 Processor -- start 2024-04-25T13:49:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.646+0000 7f44f4f2d640 1 -- start start 2024-04-25T13:49:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.646+0000 7f44f4f2d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44f0069e90 0x7f44f006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44f4f2d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f01029e0 0x7f44f006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44f4f2d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f0103c20 0x7f44f01b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44f4f2d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0107bd0 con 0x7f44f0103c20 2024-04-25T13:49:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44f4f2d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0107a50 con 0x7f44f0069e90 2024-04-25T13:49:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44ee59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44f0069e90 0x7f44f006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44ee59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44f0069e90 0x7f44f006d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39464/0 (socket says 172.21.15.27:39464) 2024-04-25T13:49:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44ee59c640 1 -- 172.21.15.27:0/1531521367 learned_addr learned my addr 172.21.15.27:0/1531521367 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0107d50 con 0x7f44f01029e0 2024-04-25T13:49:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44eed9d640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f0103c20 0x7f44f01b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.647+0000 7f44edd9b640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f01029e0 0x7f44f006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.648+0000 7f44ee59c640 1 -- 172.21.15.27:0/1531521367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f01029e0 msgr2=0x7f44f006d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.648+0000 7f44ee59c640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f01029e0 0x7f44f006d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.648+0000 7f44ee59c640 1 -- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f0103c20 msgr2=0x7f44f01b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.648+0000 7f44ee59c640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f0103c20 0x7f44f01b0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.648+0000 7f44ee59c640 1 -- 172.21.15.27:0/1531521367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f0074aa0 con 0x7f44f0069e90 2024-04-25T13:49:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.648+0000 7f44ee59c640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44f0069e90 0x7f44f006d4a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f44e800b9b0 tx=0x7f44e800be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.649+0000 7f44df7fe640 1 -- 172.21.15.27:0/1531521367 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8019070 con 0x7f44f0069e90 2024-04-25T13:49:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.649+0000 7f44df7fe640 1 -- 172.21.15.27:0/1531521367 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e800cba0 con 0x7f44f0069e90 2024-04-25T13:49:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.649+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f01b1310 con 0x7f44f0069e90 2024-04-25T13:49:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.649+0000 7f44df7fe640 1 -- 172.21.15.27:0/1531521367 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8019070 con 0x7f44f0069e90 2024-04-25T13:49:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.650+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f01b1860 con 0x7f44f0069e90 2024-04-25T13:49:06.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44fbddd740 1 Processor -- start 2024-04-25T13:49:06.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44fbddd740 1 -- start start 2024-04-25T13:49:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44df7fe640 1 -- 172.21.15.27:0/1531521367 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f44e801c070 con 0x7f44f0069e90 2024-04-25T13:49:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e80985390 0x559e80d26a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d26fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d27c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44fbddd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559e80c8bd80 con 0x559e8095c4d0 2024-04-25T13:49:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44fbddd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559e80c458b0 con 0x559e809646b0 2024-04-25T13:49:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.654+0000 7f44fbddd740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559e80c87d40 con 0x559e80985390 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e80985390 0x559e80d26a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e80985390 0x559e80d26a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51518/0 (socket says 172.21.15.27:51518) 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 -- 172.21.15.27:0/1577383143 learned_addr learned my addr 172.21.15.27:0/1577383143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44edd9b640 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d27c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44ee59c640 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d26fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 -- 172.21.15.27:0/1577383143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 msgr2=0x559e80d27c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d27c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 -- 172.21.15.27:0/1577383143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 msgr2=0x559e80d26fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d26fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 -- 172.21.15.27:0/1577383143 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e80c16a40 con 0x559e80985390 2024-04-25T13:49:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44df7fe640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44d80afdc0 0x7f44d80b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.655+0000 7f44eed9d640 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e80985390 0x559e80d26a60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f44e40099c0 tx=0x7f44e4009e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=615b21ae46957267 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.656+0000 7f44df7fe640 1 -- 172.21.15.27:0/1531521367 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f44e80ddde0 con 0x7f44f0069e90 2024-04-25T13:49:06.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.656+0000 7f44dcff9640 1 -- 172.21.15.27:0/1577383143 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4017070 con 0x559e80985390 2024-04-25T13:49:06.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.656+0000 7f44ee59c640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44d80afdc0 0x7f44d80b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.656+0000 7f44dcff9640 1 -- 172.21.15.27:0/1577383143 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e4013470 con 0x559e80985390 2024-04-25T13:49:06.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.656+0000 7f44dcff9640 1 -- 172.21.15.27:0/1577383143 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4012610 con 0x559e80985390 2024-04-25T13:49:06.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.657+0000 7f44fbddd740 1 -- 172.21.15.27:0/1577383143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e80985390 msgr2=0x559e80d26a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.657+0000 7f44ee59c640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44d80afdc0 0x7f44d80b22d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f44e80a23c0 tx=0x7f44e8014c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.657+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e80985390 0x559e80d26a60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f44e40099c0 tx=0x7f44e4009e80 comp rx=0 tx=0).stop 2024-04-25T13:49:06.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.657+0000 7f44fbddd740 1 -- 172.21.15.27:0/1577383143 shutdown_connections 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.657+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d27c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.657+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d26fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.657+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1577383143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e80985390 0x559e80d26a60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.657+0000 7f44fbddd740 1 -- 172.21.15.27:0/1577383143 >> 172.21.15.27:0/1577383143 conn(0x559e809794e0 msgr2=0x559e80d22270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.658+0000 7f44fbddd740 1 -- 172.21.15.27:0/1577383143 shutdown_connections 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.658+0000 7f44fbddd740 1 -- 172.21.15.27:0/1577383143 wait complete. 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.658+0000 7f44fbddd740 1 -- start start 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.658+0000 7f44fbddd740 1 Processor -- start 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d30900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d30e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d37da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44fbddd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559e80d28780 con 0x559e80985390 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44fbddd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559e80c12d10 con 0x559e809646b0 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44fbddd740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559e80d28900 con 0x559e8095c4d0 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44fbddd740 10 client.0 _ll_register_callbacks cb 0x559e80c3f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44eed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d30900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44edd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d30e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44edd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d30e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39480/0 (socket says 172.21.15.27:39480) 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44edd9b640 1 -- 172.21.15.27:0/983326836 learned_addr learned my addr 172.21.15.27:0/983326836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.659+0000 7f44eed9d640 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d37da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44edd9b640 1 -- 172.21.15.27:0/983326836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 msgr2=0x559e80d30900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44edd9b640 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d30900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44edd9b640 1 -- 172.21.15.27:0/983326836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 msgr2=0x559e80d37da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44edd9b640 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d37da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44edd9b640 1 -- 172.21.15.27:0/983326836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e80a677d0 con 0x559e809646b0 2024-04-25T13:49:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44eed9d640 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d37da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:49:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44edd9b640 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d30e40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009850 tx=0x7f44e00317d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a5073aa067332d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:49:06.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44d5ffb640 1 -- 172.21.15.27:0/983326836 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0035070 con 0x559e809646b0 2024-04-25T13:49:06.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44d5ffb640 1 -- 172.21.15.27:0/983326836 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f44e0031a50 con 0x559e809646b0 2024-04-25T13:49:06.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.660+0000 7f44fbddd740 1 -- 172.21.15.27:0/983326836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e4013470 con 0x559e809646b0 2024-04-25T13:49:06.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44d5ffb640 1 -- 172.21.15.27:0/983326836 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f44e0031c50 con 0x559e809646b0 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44fbddd740 10 client.26759 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44fbddd740 1 -- 172.21.15.27:0/983326836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559e8099d3f0 con 0x559e809646b0 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44d5ffb640 1 -- 172.21.15.27:0/983326836 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e4013470 con 0x559e809646b0 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44d5ffb640 1 -- 172.21.15.27:0/983326836 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f44e0046480 con 0x559e809646b0 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44fbddd740 10 client.26759 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44fbddd740 10 client.26759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44fbddd740 20 client.26759 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44fbddd740 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559e80943f60 0x559e80d3e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44fbddd740 4 client.26759 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.661+0000 7f44fbddd740 1 -- 172.21.15.27:0/983326836 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e80c99990 con 0x559e80943f60 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.662+0000 7f44eed9d640 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559e80943f60 0x559e80d3e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.662+0000 7f44d5ffb640 10 client.26759 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.662+0000 7f44eed9d640 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559e80943f60 0x559e80d3e6e0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9756c4eaa5db93b6 server_cookie=ca0e34fd2d3c27e1 in_seq=0 out_seq=0 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44d5ffb640 1 -- 172.21.15.27:0/983326836 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559e80c99990 con 0x559e80943f60 2024-04-25T13:49:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44d5ffb640 10 client.26759 handle_command_reply: tid=0 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 client.26759 shutdown 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 -- 172.21.15.27:0/983326836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 msgr2=0x559e80d30e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d30e40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009850 tx=0x7f44e00317d0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 -- 172.21.15.27:0/983326836 shutdown_connections 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559e80943f60 0x559e80d3e6e0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d37da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d30e40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 --2- 172.21.15.27:0/983326836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d30900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 -- 172.21.15.27:0/983326836 >> 172.21.15.27:0/983326836 conn(0x559e809794e0 msgr2=0x559e80d213f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44d5ffb640 0 client.26759 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 -- 172.21.15.27:0/983326836 shutdown_connections 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 1 -- 172.21.15.27:0/983326836 wait complete. 2024-04-25T13:49:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.664+0000 7f44fbddd740 20 client.26759 trim_cache size 0 max 16384 2024-04-25T13:49:06.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.692+0000 7f44fbddd740 1 Processor -- start 2024-04-25T13:49:06.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.692+0000 7f44fbddd740 1 -- start start 2024-04-25T13:49:06.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d2ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e809646b0 0x559e80d2f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e80985390 0x559e80d42ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44fbddd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01b7ff0 con 0x559e8095c4d0 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44eed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d2ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44fbddd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01ba870 con 0x559e80985390 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44fbddd740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01b8660 con 0x559e809646b0 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44eed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d2ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34940/0 (socket says 172.21.15.27:34940) 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44eed9d640 1 -- 172.21.15.27:0/440600657 learned_addr learned my addr 172.21.15.27:0/440600657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44edd9b640 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e809646b0 0x559e80d2f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44eed9d640 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e80985390 0x559e80d42ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44edd9b640 1 -- 172.21.15.27:0/440600657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e80985390 msgr2=0x559e80d42ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44edd9b640 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e80985390 0x559e80d42ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44edd9b640 1 -- 172.21.15.27:0/440600657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 msgr2=0x559e80d2ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44edd9b640 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d2ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44edd9b640 1 -- 172.21.15.27:0/440600657 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e8095d0f0 con 0x559e809646b0 2024-04-25T13:49:06.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44eed9d640 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d2ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:06.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.693+0000 7f44edd9b640 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e809646b0 0x559e80d2f4b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f44e0031440 tx=0x7f44e0005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=402e69e54d9c13a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.694+0000 7f44d7fff640 1 -- 172.21.15.27:0/440600657 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0035050 con 0x559e809646b0 2024-04-25T13:49:06.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.694+0000 7f44d7fff640 1 -- 172.21.15.27:0/440600657 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e0046460 con 0x559e809646b0 2024-04-25T13:49:06.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.694+0000 7f44d7fff640 1 -- 172.21.15.27:0/440600657 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0003750 con 0x559e809646b0 2024-04-25T13:49:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.695+0000 7f44fbddd740 1 -- 172.21.15.27:0/440600657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e809646b0 msgr2=0x559e80d2f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.695+0000 7f44fbddd740 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e809646b0 0x559e80d2f4b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f44e0031440 tx=0x7f44e0005a20 comp rx=0 tx=0).stop 2024-04-25T13:49:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.695+0000 7f44fbddd740 1 -- 172.21.15.27:0/440600657 shutdown_connections 2024-04-25T13:49:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.695+0000 7f44fbddd740 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e80985390 0x559e80d42ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.695+0000 7f44fbddd740 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e809646b0 0x559e80d2f4b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.695+0000 7f44fbddd740 1 --2- 172.21.15.27:0/440600657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e8095c4d0 0x559e80d2ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.695+0000 7f44fbddd740 1 -- 172.21.15.27:0/440600657 >> 172.21.15.27:0/440600657 conn(0x559e809794e0 msgr2=0x559e80d2b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.696+0000 7f44fbddd740 1 -- 172.21.15.27:0/440600657 shutdown_connections 2024-04-25T13:49:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.696+0000 7f44fbddd740 1 -- 172.21.15.27:0/440600657 wait complete. 2024-04-25T13:49:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.697+0000 7f44fbddd740 1 -- start start 2024-04-25T13:49:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.697+0000 7f44fbddd740 1 Processor -- start 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d37280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d33e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44fbddd740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d34340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44fbddd740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01b8d30 con 0x559e80985390 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44fbddd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01ba1a0 con 0x559e809646b0 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44fbddd740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01b9400 con 0x559e8095c4d0 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44fbddd740 10 client.0 _ll_register_callbacks cb 0x7f44e0004100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d37280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44edd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d33e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d37280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51544/0 (socket says 172.21.15.27:51544) 2024-04-25T13:49:06.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44edd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d33e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39502/0 (socket says 172.21.15.27:39502) 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 -- 172.21.15.27:0/1269979791 learned_addr learned my addr 172.21.15.27:0/1269979791 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d34340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 -- 172.21.15.27:0/1269979791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 msgr2=0x559e80d33e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d33e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 -- 172.21.15.27:0/1269979791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 msgr2=0x559e80d34340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d34340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.698+0000 7f44eed9d640 1 -- 172.21.15.27:0/1269979791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e80c4c4b0 con 0x559e8095c4d0 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44eed9d640 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d37280 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f44e40099c0 tx=0x7f44e4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1c0ae9b7b0fd590 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44d5ffb640 1 -- 172.21.15.27:0/1269979791 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e402ae60 con 0x559e8095c4d0 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44fbddd740 1 -- 172.21.15.27:0/1269979791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e0046460 con 0x559e8095c4d0 2024-04-25T13:49:06.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44d5ffb640 1 -- 172.21.15.27:0/1269979791 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f44e402a2e0 con 0x559e8095c4d0 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44d5ffb640 1 -- 172.21.15.27:0/1269979791 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f44e4017040 con 0x559e8095c4d0 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44fbddd740 10 client.26812 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44fbddd740 1 -- 172.21.15.27:0/1269979791 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559e80da5ae0 con 0x559e8095c4d0 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44d5ffb640 1 -- 172.21.15.27:0/1269979791 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e0046460 con 0x559e8095c4d0 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.699+0000 7f44d5ffb640 1 -- 172.21.15.27:0/1269979791 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f44e401c750 con 0x559e8095c4d0 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44fbddd740 10 client.26812 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44fbddd740 10 client.26812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44fbddd740 20 client.26812 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559e809193a0 0x559e80da81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44fbddd740 4 client.26812 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44fbddd740 1 -- 172.21.15.27:0/1269979791 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559e80c93800 con 0x559e809193a0 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44edd9b640 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559e809193a0 0x559e80da81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44d5ffb640 10 client.26812 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.700+0000 7f44edd9b640 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559e809193a0 0x559e80da81c0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37e90981693ac095 server_cookie=9c236046bc80953a in_seq=0 out_seq=0 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44d5ffb640 1 -- 172.21.15.27:0/1269979791 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559e80c93800 con 0x559e809193a0 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44d5ffb640 10 client.26812 handle_command_reply: tid=0 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 client.26812 shutdown 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 -- 172.21.15.27:0/1269979791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 msgr2=0x559e80d37280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d37280 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f44e40099c0 tx=0x7f44e4008040 comp rx=0 tx=0).stop 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 -- 172.21.15.27:0/1269979791 shutdown_connections 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559e809193a0 0x559e80da81c0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559e80985390 0x559e80d34340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44d5ffb640 0 client.26812 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559e809646b0 0x559e80d33e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 --2- 172.21.15.27:0/1269979791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e8095c4d0 0x559e80d37280 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 -- 172.21.15.27:0/1269979791 >> 172.21.15.27:0/1269979791 conn(0x559e809794e0 msgr2=0x559e80d2ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 -- 172.21.15.27:0/1269979791 shutdown_connections 2024-04-25T13:49:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 1 -- 172.21.15.27:0/1269979791 wait complete. 2024-04-25T13:49:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.701+0000 7f44fbddd740 20 client.26812 trim_cache size 0 max 16384 2024-04-25T13:49:06.710 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:49:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:49:06.710 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:49:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "965e0032-8c1d-44bc-be71-7f9fb0c454e8": { 2024-04-25T13:49:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:49:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "965e0032-8c1d-44bc-be71-7f9fb0c454e8", 2024-04-25T13:49:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:49:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:49:06.711 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:49:06.711 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:49:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44d80afdc0 msgr2=0x7f44d80b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44d80afdc0 0x7f44d80b22d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f44e80a23c0 tx=0x7f44e8014c90 comp rx=0 tx=0).stop 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44f0069e90 msgr2=0x7f44f006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44f0069e90 0x7f44f006d4a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f44e800b9b0 tx=0x7f44e800be70 comp rx=0 tx=0).stop 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 shutdown_connections 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44d80afdc0 0x7f44d80b22d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44f0103c20 0x7f44f01b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f44f01029e0 0x7f44f006d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 --2- 172.21.15.27:0/1531521367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44f0069e90 0x7f44f006d4a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 >> 172.21.15.27:0/1531521367 conn(0x7f44f00fdbd0 msgr2=0x7f44f00ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 shutdown_connections 2024-04-25T13:49:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:06.702+0000 7f44f4f2d640 1 -- 172.21.15.27:0/1531521367 wait complete. 2024-04-25T13:49:06.715 INFO:tasks.cephfs.filesystem:scrub status for tag:965e0032-8c1d-44bc-be71-7f9fb0c454e8 - {'path': '/', 'tag': '965e0032-8c1d-44bc-be71-7f9fb0c454e8', 'options': 'recursive,force'} 2024-04-25T13:49:07.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:07 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:49:07.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:07 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:07.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:07 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:49:07.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:07 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:07.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:07 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:49:07.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:07 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:07.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:07 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:49:07.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:07 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:07.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:07 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:49:07.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:07 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:07.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:07 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:49:07.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:07 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 480.0 s, 1600.2 tps, lat 63.122 ms stddev 70.483 2024-04-25T13:49:08.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:08 smithi122 ceph-mon[32843]: pgmap v596: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 773 op/s 2024-04-25T13:49:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:08 smithi045 ceph-mon[32731]: pgmap v596: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 773 op/s 2024-04-25T13:49:08.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:08 smithi027 ceph-mon[29140]: pgmap v596: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 773 op/s 2024-04-25T13:49:09.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:09 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:49:09.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:09 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:49:09.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:09 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:49:10.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:10 smithi122 ceph-mon[32843]: pgmap v597: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.1 MiB/s wr, 558 op/s 2024-04-25T13:49:10.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:10 smithi045 ceph-mon[32731]: pgmap v597: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.1 MiB/s wr, 558 op/s 2024-04-25T13:49:10.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:10 smithi027 ceph-mon[29140]: pgmap v597: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.1 MiB/s wr, 558 op/s 2024-04-25T13:49:12.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:12 smithi122 ceph-mon[32843]: pgmap v598: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.1 MiB/s wr, 697 op/s 2024-04-25T13:49:12.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:12 smithi045 ceph-mon[32731]: pgmap v598: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.1 MiB/s wr, 697 op/s 2024-04-25T13:49:12.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:12 smithi027 ceph-mon[29140]: pgmap v598: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.1 MiB/s wr, 697 op/s 2024-04-25T13:49:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 485.0 s, 2218.2 tps, lat 45.076 ms stddev 49.253 2024-04-25T13:49:14.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:14 smithi122 ceph-mon[32843]: pgmap v599: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.2 MiB/s wr, 737 op/s 2024-04-25T13:49:14.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:14 smithi045 ceph-mon[32731]: pgmap v599: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.2 MiB/s wr, 737 op/s 2024-04-25T13:49:14.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:14 smithi027 ceph-mon[29140]: pgmap v599: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.2 MiB/s wr, 737 op/s 2024-04-25T13:49:16.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:16 smithi122 ceph-mon[32843]: pgmap v600: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.4 MiB/s wr, 644 op/s 2024-04-25T13:49:16.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:16 smithi045 ceph-mon[32731]: pgmap v600: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.4 MiB/s wr, 644 op/s 2024-04-25T13:49:16.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:16 smithi027 ceph-mon[29140]: pgmap v600: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.4 MiB/s wr, 644 op/s 2024-04-25T13:49:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 490.0 s, 2510.8 tps, lat 39.547 ms stddev 43.618 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 -- 172.21.15.27:0/2969782245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018109a80 msgr2=0x7f101810bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 --2- 172.21.15.27:0/2969782245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018109a80 0x7f101810bea0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1010009ca0 tx=0x7f101002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 -- 172.21.15.27:0/2969782245 shutdown_connections 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 --2- 172.21.15.27:0/2969782245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018109a80 0x7f101810bea0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 --2- 172.21.15.27:0/2969782245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1018104d80 0x7f1018105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 --2- 172.21.15.27:0/2969782245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1018103b40 0x7f1018104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 -- 172.21.15.27:0/2969782245 >> 172.21.15.27:0/2969782245 conn(0x7f10180fed30 msgr2=0x7f10181011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 -- 172.21.15.27:0/2969782245 shutdown_connections 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 -- 172.21.15.27:0/2969782245 wait complete. 2024-04-25T13:49:18.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 Processor -- start 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.038+0000 7f101ed0d640 1 -- start start 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101ed0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1018103b40 0x7f101819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101ed0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018104d80 0x7f101819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101ed0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1018109a80 0x7f10181a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101ed0d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f101810f650 con 0x7f1018104d80 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101ed0d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f101810f4d0 con 0x7f1018109a80 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101ed0d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f101810f7d0 con 0x7f1018103b40 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018104d80 0x7f101819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018104d80 0x7f101819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55444/0 (socket says 172.21.15.27:55444) 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 -- 172.21.15.27:0/1654970230 learned_addr learned my addr 172.21.15.27:0/1654970230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101caa9640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1018103b40 0x7f101819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101d2aa640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1018109a80 0x7f10181a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 -- 172.21.15.27:0/1654970230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1018103b40 msgr2=0x7f101819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1018103b40 0x7f101819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 -- 172.21.15.27:0/1654970230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1018109a80 msgr2=0x7f10181a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1018109a80 0x7f10181a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 -- 172.21.15.27:0/1654970230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10181a1d00 con 0x7f1018104d80 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f101caa9640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1018103b40 0x7f101819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.039+0000 7f1017fff640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018104d80 0x7f101819a700 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f100400e9f0 tx=0x7f100400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.040+0000 7f1015ffb640 1 -- 172.21.15.27:0/1654970230 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100400cd80 con 0x7f1018104d80 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.040+0000 7f1015ffb640 1 -- 172.21.15.27:0/1654970230 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10040108e0 con 0x7f1018104d80 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.040+0000 7f101ed0d640 1 -- 172.21.15.27:0/1654970230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10181a1ff0 con 0x7f1018104d80 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.040+0000 7f1015ffb640 1 -- 172.21.15.27:0/1654970230 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004018980 con 0x7f1018104d80 2024-04-25T13:49:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.040+0000 7f101ed0d640 1 -- 172.21.15.27:0/1654970230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10181a2460 con 0x7f1018104d80 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.044+0000 7f1025bc4740 1 Processor -- start 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.044+0000 7f1025bc4740 1 -- start start 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.044+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f781b0 0x56117231cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.044+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117231d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.044+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f574d0 0x561172324200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.044+0000 7f1025bc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56117227e460 con 0x561171f4f2f0 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.044+0000 7f1025bc4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561172238590 con 0x561171f574d0 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.044+0000 7f1025bc4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56117227a420 con 0x561171f781b0 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f781b0 0x56117231cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f781b0 0x56117231cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50166/0 (socket says 172.21.15.27:50166) 2024-04-25T13:49:18.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 -- 172.21.15.27:0/4258482007 learned_addr learned my addr 172.21.15.27:0/4258482007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101caa9640 1 --2- 172.21.15.27:0/4258482007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117231d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 -- 172.21.15.27:0/4258482007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f574d0 msgr2=0x561172324200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:49:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 --2- 172.21.15.27:0/4258482007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f574d0 0x561172324200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 -- 172.21.15.27:0/4258482007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 msgr2=0x56117231d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 --2- 172.21.15.27:0/4258482007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117231d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 -- 172.21.15.27:0/4258482007 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611722450d0 con 0x561171f781b0 2024-04-25T13:49:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f1015ffb640 1 -- 172.21.15.27:0/1654970230 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1004010430 con 0x7f1018104d80 2024-04-25T13:49:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.045+0000 7f101d2aa640 1 --2- 172.21.15.27:0/4258482007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f781b0 0x56117231cd80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1010035000 tx=0x7f101002f410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9274482a568cfd8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.046+0000 7f1002ffd640 1 -- 172.21.15.27:0/4258482007 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010031650 con 0x561171f781b0 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.046+0000 7f1015ffb640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10080afae0 0x7f10080b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.046+0000 7f1002ffd640 1 -- 172.21.15.27:0/4258482007 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1010036030 con 0x561171f781b0 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.046+0000 7f1002ffd640 1 -- 172.21.15.27:0/4258482007 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010004680 con 0x561171f781b0 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.046+0000 7f101caa9640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10080afae0 0x7f10080b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.046+0000 7f1015ffb640 1 -- 172.21.15.27:0/1654970230 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f10040deec0 con 0x7f1018104d80 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 -- 172.21.15.27:0/4258482007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f781b0 msgr2=0x56117231cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4258482007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f781b0 0x56117231cd80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1010035000 tx=0x7f101002f410 comp rx=0 tx=0).stop 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 -- 172.21.15.27:0/4258482007 shutdown_connections 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4258482007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f574d0 0x561172324200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4258482007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117231d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4258482007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f781b0 0x56117231cd80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 -- 172.21.15.27:0/4258482007 >> 172.21.15.27:0/4258482007 conn(0x561171f6c300 msgr2=0x561172314300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:18.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f101caa9640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10080afae0 0x7f10080b1ff0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x56117231e160 tx=0x7f100c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:18.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 -- 172.21.15.27:0/4258482007 shutdown_connections 2024-04-25T13:49:18.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.047+0000 7f1025bc4740 1 -- 172.21.15.27:0/4258482007 wait complete. 2024-04-25T13:49:18.049 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.048+0000 7f1025bc4740 1 -- start start 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.048+0000 7f1025bc4740 1 Processor -- start 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172326960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x561172326ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x56117232de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f1025bc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561172324a70 con 0x561171f574d0 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f1025bc4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561172205b60 con 0x561171f781b0 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f1025bc4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561172324bf0 con 0x561171f4f2f0 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f101caa9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172326960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f101caa9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172326960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50170/0 (socket says 172.21.15.27:50170) 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f101caa9640 1 -- 172.21.15.27:0/3493755999 learned_addr learned my addr 172.21.15.27:0/3493755999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f1025bc4740 10 client.0 _ll_register_callbacks cb 0x56117222a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.049+0000 7f101d2aa640 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x56117232de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f1017fff640 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x561172326ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f101caa9640 1 -- 172.21.15.27:0/3493755999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 msgr2=0x56117232de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f101caa9640 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x56117232de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f101caa9640 1 -- 172.21.15.27:0/3493755999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 msgr2=0x561172326ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f101caa9640 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x561172326ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f101caa9640 1 -- 172.21.15.27:0/3493755999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561171f90210 con 0x561171f4f2f0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f1017fff640 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x561172326ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f101caa9640 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172326960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f100c0045f0 tx=0x7f100c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c703ed5489ae00b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f0ff7fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f0ff7fff640 1 -- 172.21.15.27:0/3493755999 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c00f830 con 0x561171f4f2f0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f0ff7fff640 1 -- 172.21.15.27:0/3493755999 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f100c00f9d0 con 0x561171f4f2f0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f1025bc4740 1 -- 172.21.15.27:0/3493755999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1010036030 con 0x561171f4f2f0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f0ff7fff640 1 -- 172.21.15.27:0/3493755999 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f100c00fbd0 con 0x561171f4f2f0 2024-04-25T13:49:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f1025bc4740 10 client.26824 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.050+0000 7f1025bc4740 1 -- 172.21.15.27:0/3493755999 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56117205c620 con 0x561171f4f2f0 2024-04-25T13:49:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.051+0000 7f0ff7fff640 1 -- 172.21.15.27:0/3493755999 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f100c00fd70 con 0x561171f4f2f0 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.052+0000 7f0ff7fff640 1 -- 172.21.15.27:0/3493755999 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f100c027070 con 0x561171f4f2f0 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.052+0000 7f1025bc4740 10 client.26824 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.052+0000 7f1025bc4740 10 client.26824 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.052+0000 7f1025bc4740 20 client.26824 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.052+0000 7f1025bc4740 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561171f36d80 0x5611723346e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.052+0000 7f1025bc4740 4 client.26824 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.052+0000 7f1025bc4740 1 -- 172.21.15.27:0/3493755999 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1010036030 con 0x561171f36d80 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.052+0000 7f1017fff640 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561171f36d80 0x5611723346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.053+0000 7f1017fff640 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561171f36d80 0x5611723346e0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d432cf95d6ed2d83 server_cookie=1f3e17e04e2b2a18 in_seq=0 out_seq=0 2024-04-25T13:49:18.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.054+0000 7f0ff7fff640 10 client.26824 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.054+0000 7f0ff7fff640 1 -- 172.21.15.27:0/3493755999 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f1010036030 con 0x561171f36d80 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.054+0000 7f0ff7fff640 10 client.26824 handle_command_reply: tid=0 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.054+0000 7f1025bc4740 1 client.26824 shutdown 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.054+0000 7f1025bc4740 1 -- 172.21.15.27:0/3493755999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 msgr2=0x561172326960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.054+0000 7f1025bc4740 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172326960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f100c0045f0 tx=0x7f100c004730 comp rx=0 tx=0).stop 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 1 -- 172.21.15.27:0/3493755999 shutdown_connections 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561171f36d80 0x5611723346e0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x56117232de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x561172326ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 1 --2- 172.21.15.27:0/3493755999 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172326960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f0ff7fff640 0 client.26824 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 1 -- 172.21.15.27:0/3493755999 >> 172.21.15.27:0/3493755999 conn(0x561171f6c300 msgr2=0x561172313250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 1 -- 172.21.15.27:0/3493755999 shutdown_connections 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 1 -- 172.21.15.27:0/3493755999 wait complete. 2024-04-25T13:49:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.055+0000 7f1025bc4740 20 client.26824 trim_cache size 0 max 16384 2024-04-25T13:49:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.074+0000 7f1025bc4740 1 Processor -- start 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.074+0000 7f1025bc4740 1 -- start start 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.074+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172320f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x56117232dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x5611723214a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1025bc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f10181a6fa0 con 0x561171f574d0 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1025bc4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f10181a9820 con 0x561171f781b0 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1025bc4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f10181a7610 con 0x561171f4f2f0 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x56117232dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x56117232dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55472/0 (socket says 172.21.15.27:55472) 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 -- 172.21.15.27:0/2064043932 learned_addr learned my addr 172.21.15.27:0/2064043932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f101caa9640 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172320f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f101d2aa640 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x5611723214a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 -- 172.21.15.27:0/2064043932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 msgr2=0x561172320f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172320f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 -- 172.21.15.27:0/2064043932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 msgr2=0x5611723214a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x5611723214a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 -- 172.21.15.27:0/2064043932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561172320a40 con 0x561171f574d0 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f101d2aa640 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x5611723214a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f101caa9640 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172320f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f1017fff640 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x56117232dd90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f10040a46f0 tx=0x7f10040dc8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc94a98fb5c0d451 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.075+0000 7f10027fc640 1 -- 172.21.15.27:0/2064043932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10040c0070 con 0x561171f574d0 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.076+0000 7f10027fc640 1 -- 172.21.15.27:0/2064043932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10040b0230 con 0x561171f574d0 2024-04-25T13:49:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.076+0000 7f10027fc640 1 -- 172.21.15.27:0/2064043932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004018d80 con 0x561171f574d0 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.077+0000 7f1025bc4740 1 -- 172.21.15.27:0/2064043932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 msgr2=0x56117232dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.077+0000 7f1025bc4740 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x56117232dd90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f10040a46f0 tx=0x7f10040dc8c0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.078+0000 7f1025bc4740 1 -- 172.21.15.27:0/2064043932 shutdown_connections 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.078+0000 7f1025bc4740 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x5611723214a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.078+0000 7f1025bc4740 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f574d0 0x56117232dd90 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.078+0000 7f1025bc4740 1 --2- 172.21.15.27:0/2064043932 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f4f2f0 0x561172320f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.078+0000 7f1025bc4740 1 -- 172.21.15.27:0/2064043932 >> 172.21.15.27:0/2064043932 conn(0x561171f6c300 msgr2=0x561172326900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.078+0000 7f1025bc4740 1 -- 172.21.15.27:0/2064043932 shutdown_connections 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.078+0000 7f1025bc4740 1 -- 172.21.15.27:0/2064043932 wait complete. 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.078+0000 7f1025bc4740 1 -- start start 2024-04-25T13:49:18.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1025bc4740 1 Processor -- start 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117232ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f574d0 0x56117232b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1025bc4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x56117239aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1025bc4740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f10181a7ce0 con 0x561171f4f2f0 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1025bc4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f10181a9150 con 0x561171f781b0 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1025bc4740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f10181a83b0 con 0x561171f574d0 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1025bc4740 10 client.0 _ll_register_callbacks cb 0x7f100c0044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f101caa9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117232ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f1017fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f574d0 0x56117232b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f101caa9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117232ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55486/0 (socket says 172.21.15.27:55486) 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f101caa9640 1 -- 172.21.15.27:0/4048339713 learned_addr learned my addr 172.21.15.27:0/4048339713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.079+0000 7f101d2aa640 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x56117239aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f101caa9640 1 -- 172.21.15.27:0/4048339713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f574d0 msgr2=0x56117232b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f101caa9640 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f574d0 0x56117232b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f101caa9640 1 -- 172.21.15.27:0/4048339713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 msgr2=0x56117239aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f101caa9640 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x56117239aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f101caa9640 1 -- 172.21.15.27:0/4048339713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56117232e020 con 0x561171f4f2f0 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f101caa9640 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117232ab20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f100c0045f0 tx=0x7f100c029880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d221f23a9cc75c71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f0ff7fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f0ff7fff640 1 -- 172.21.15.27:0/4048339713 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c00fd70 con 0x561171f4f2f0 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f0ff7fff640 1 -- 172.21.15.27:0/4048339713 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f100c018040 con 0x561171f4f2f0 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f1025bc4740 1 -- 172.21.15.27:0/4048339713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10040b0230 con 0x561171f4f2f0 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f0ff7fff640 1 -- 172.21.15.27:0/4048339713 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f100c003e50 con 0x561171f4f2f0 2024-04-25T13:49:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f1025bc4740 10 client.17067 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f1025bc4740 1 -- 172.21.15.27:0/4048339713 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56117239c010 con 0x561171f4f2f0 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.080+0000 7f0ff7fff640 1 -- 172.21.15.27:0/4048339713 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10040b0230 con 0x561171f4f2f0 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f0ff7fff640 1 -- 172.21.15.27:0/4048339713 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f100c027040 con 0x561171f4f2f0 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f1025bc4740 10 client.17067 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f1025bc4740 10 client.17067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f1025bc4740 20 client.17067 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561171f0c1c0 0x56117239eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f1025bc4740 4 client.17067 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f1025bc4740 1 -- 172.21.15.27:0/4048339713 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56117228c070 con 0x561171f0c1c0 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f1017fff640 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561171f0c1c0 0x56117239eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f0ff7fff640 10 client.17067 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.081+0000 7f1017fff640 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561171f0c1c0 0x56117239eab0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d52eedb12335b353 server_cookie=3d1fd096f2f052a5 in_seq=0 out_seq=0 2024-04-25T13:49:18.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.104+0000 7f0ff7fff640 1 -- 172.21.15.27:0/4048339713 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56117228c070 con 0x561171f0c1c0 2024-04-25T13:49:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.104+0000 7f0ff7fff640 10 client.17067 handle_command_reply: tid=0 2024-04-25T13:49:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 client.17067 shutdown 2024-04-25T13:49:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 -- 172.21.15.27:0/4048339713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 msgr2=0x56117232ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117232ab20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f100c0045f0 tx=0x7f100c029880 comp rx=0 tx=0).stop 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 -- 172.21.15.27:0/4048339713 shutdown_connections 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561171f0c1c0 0x56117239eab0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561171f781b0 0x56117239aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561171f574d0 0x56117232b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 --2- 172.21.15.27:0/4048339713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561171f4f2f0 0x56117232ab20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 -- 172.21.15.27:0/4048339713 >> 172.21.15.27:0/4048339713 conn(0x561171f6c300 msgr2=0x561172326900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f0ff7fff640 0 client.17067 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.105+0000 7f1025bc4740 1 -- 172.21.15.27:0/4048339713 shutdown_connections 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.106+0000 7f1025bc4740 1 -- 172.21.15.27:0/4048339713 wait complete. 2024-04-25T13:49:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.106+0000 7f1025bc4740 20 client.17067 trim_cache size 0 max 16384 2024-04-25T13:49:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:49:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:49:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:49:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.106+0000 7f101ed0d640 1 -- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10080afae0 msgr2=0x7f10080b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.106+0000 7f101ed0d640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10080afae0 0x7f10080b1ff0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x56117231e160 tx=0x7f100c005f90 comp rx=0 tx=0).stop 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.106+0000 7f101ed0d640 1 -- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018104d80 msgr2=0x7f101819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.106+0000 7f101ed0d640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018104d80 0x7f101819a700 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f100400e9f0 tx=0x7f100400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.107+0000 7f101ed0d640 1 -- 172.21.15.27:0/1654970230 shutdown_connections 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.107+0000 7f101ed0d640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f10080afae0 0x7f10080b1ff0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.107+0000 7f101ed0d640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1018109a80 0x7f10181a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.107+0000 7f101ed0d640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1018104d80 0x7f101819a700 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.107+0000 7f101ed0d640 1 --2- 172.21.15.27:0/1654970230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1018103b40 0x7f101819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.107+0000 7f101ed0d640 1 -- 172.21.15.27:0/1654970230 >> 172.21.15.27:0/1654970230 conn(0x7f10180fed30 msgr2=0x7f1018105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.107+0000 7f101ed0d640 1 -- 172.21.15.27:0/1654970230 shutdown_connections 2024-04-25T13:49:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:18.107+0000 7f101ed0d640 1 -- 172.21.15.27:0/1654970230 wait complete. 2024-04-25T13:49:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:18 smithi045 ceph-mon[32731]: pgmap v601: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.1 MiB/s wr, 825 op/s 2024-04-25T13:49:18.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:18 smithi027 ceph-mon[29140]: pgmap v601: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.1 MiB/s wr, 825 op/s 2024-04-25T13:49:18.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:18 smithi122 ceph-mon[32843]: pgmap v601: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.1 MiB/s wr, 825 op/s 2024-04-25T13:49:20.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:20 smithi045 ceph-mon[32731]: pgmap v602: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 MiB/s wr, 616 op/s 2024-04-25T13:49:20.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:20 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:49:20.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:20 smithi027 ceph-mon[29140]: pgmap v602: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 MiB/s wr, 616 op/s 2024-04-25T13:49:20.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:20 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:49:20.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:20 smithi122 ceph-mon[32843]: pgmap v602: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 MiB/s wr, 616 op/s 2024-04-25T13:49:20.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:20 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:49:21.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:49:21.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:49:21.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:49:21.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:49:21.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:49:21.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:49:21.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:49:21.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:49:21.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:49:22.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:22 smithi045 ceph-mon[32731]: pgmap v603: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 7.1 MiB/s wr, 757 op/s 2024-04-25T13:49:22.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:22 smithi027 ceph-mon[29140]: pgmap v603: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 7.1 MiB/s wr, 757 op/s 2024-04-25T13:49:22.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:22 smithi122 ceph-mon[32843]: pgmap v603: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 7.1 MiB/s wr, 757 op/s 2024-04-25T13:49:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 495.0 s, 1732.4 tps, lat 57.643 ms stddev 61.784 2024-04-25T13:49:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:24 smithi045 ceph-mon[32731]: pgmap v604: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.8 MiB/s wr, 708 op/s 2024-04-25T13:49:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:24 smithi027 ceph-mon[29140]: pgmap v604: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.8 MiB/s wr, 708 op/s 2024-04-25T13:49:24.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:24 smithi122 ceph-mon[32843]: pgmap v604: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.8 MiB/s wr, 708 op/s 2024-04-25T13:49:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:26 smithi045 ceph-mon[32731]: pgmap v605: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.4 MiB/s wr, 624 op/s 2024-04-25T13:49:26.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:26 smithi027 ceph-mon[29140]: pgmap v605: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.4 MiB/s wr, 624 op/s 2024-04-25T13:49:26.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:26 smithi122 ceph-mon[32843]: pgmap v605: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.4 MiB/s wr, 624 op/s 2024-04-25T13:49:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 500.0 s, 1484.2 tps, lat 67.833 ms stddev 67.733 2024-04-25T13:49:28.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:28 smithi027 ceph-mon[29140]: pgmap v606: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.5 MiB/s wr, 758 op/s 2024-04-25T13:49:28.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:28 smithi122 ceph-mon[32843]: pgmap v606: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.5 MiB/s wr, 758 op/s 2024-04-25T13:49:28.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:28 smithi045 ceph-mon[32731]: pgmap v606: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.5 MiB/s wr, 758 op/s 2024-04-25T13:49:30.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:30 smithi027 ceph-mon[29140]: pgmap v607: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 566 op/s 2024-04-25T13:49:30.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:30 smithi122 ceph-mon[32843]: pgmap v607: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 566 op/s 2024-04-25T13:49:30.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:30 smithi045 ceph-mon[32731]: pgmap v607: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 566 op/s 2024-04-25T13:49:31.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 -- 172.21.15.27:0/2258617581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c103400 msgr2=0x7f035c103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 --2- 172.21.15.27:0/2258617581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c103400 0x7f035c103810 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0350009900 tx=0x7f035002a410 comp rx=0 tx=0).stop 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 -- 172.21.15.27:0/2258617581 shutdown_connections 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 --2- 172.21.15.27:0/2258617581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f035c108100 0x7f035c10a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 --2- 172.21.15.27:0/2258617581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c103400 0x7f035c103810 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 --2- 172.21.15.27:0/2258617581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f035c1021c0 0x7f035c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 -- 172.21.15.27:0/2258617581 >> 172.21.15.27:0/2258617581 conn(0x7f035c0fd390 msgr2=0x7f035c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 -- 172.21.15.27:0/2258617581 shutdown_connections 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.254+0000 7f0361984640 1 -- 172.21.15.27:0/2258617581 wait complete. 2024-04-25T13:49:31.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.255+0000 7f0361984640 1 Processor -- start 2024-04-25T13:49:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.255+0000 7f0361984640 1 -- start start 2024-04-25T13:49:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.255+0000 7f0361984640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f035c1021c0 0x7f035c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f0361984640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f035c103400 0x7f035c19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f0361984640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c108100 0x7f035c1a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f0361984640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f035c10f660 con 0x7f035c108100 2024-04-25T13:49:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f0361984640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f035c10f4e0 con 0x7f035c1021c0 2024-04-25T13:49:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f0361984640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f035c10f7e0 con 0x7f035c103400 2024-04-25T13:49:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f035c103400 0x7f035c19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f035c103400 0x7f035c19c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51338/0 (socket says 172.21.15.27:51338) 2024-04-25T13:49:31.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f035a7fc640 1 -- 172.21.15.27:0/919975662 learned_addr learned my addr 172.21.15.27:0/919975662 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.256+0000 7f035b7fe640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c108100 0x7f035c1a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.257+0000 7f035b7fe640 1 -- 172.21.15.27:0/919975662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f035c103400 msgr2=0x7f035c19c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.257+0000 7f035b7fe640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f035c103400 0x7f035c19c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.257+0000 7f035b7fe640 1 -- 172.21.15.27:0/919975662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f035c1021c0 msgr2=0x7f035c19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.257+0000 7f035b7fe640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f035c1021c0 0x7f035c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.257+0000 7f035b7fe640 1 -- 172.21.15.27:0/919975662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035c1a3f50 con 0x7f035c108100 2024-04-25T13:49:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.257+0000 7f035b7fe640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c108100 0x7f035c1a38e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0354009fd0 tx=0x7f035400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:31.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.258+0000 7f0360982640 1 -- 172.21.15.27:0/919975662 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035400ce20 con 0x7f035c108100 2024-04-25T13:49:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.258+0000 7f0360982640 1 -- 172.21.15.27:0/919975662 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0354005020 con 0x7f035c108100 2024-04-25T13:49:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.258+0000 7f0360982640 1 -- 172.21.15.27:0/919975662 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354010690 con 0x7f035c108100 2024-04-25T13:49:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.258+0000 7f0361984640 1 -- 172.21.15.27:0/919975662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035c1a4240 con 0x7f035c108100 2024-04-25T13:49:31.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.258+0000 7f0361984640 1 -- 172.21.15.27:0/919975662 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f035c1a46e0 con 0x7f035c108100 2024-04-25T13:49:31.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.261+0000 7f036883b740 1 Processor -- start 2024-04-25T13:49:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f0360982640 1 -- 172.21.15.27:0/919975662 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f03540054d0 con 0x7f035c108100 2024-04-25T13:49:31.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f036883b740 1 -- start start 2024-04-25T13:49:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f036883b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x560559403180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f036883b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590312f0 0x560559403700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f036883b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f036883b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560559360460 con 0x5605590312f0 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f036883b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56055931a590 con 0x56055905a1b0 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f036883b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56055935c420 con 0x5605590394d0 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f035affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590312f0 0x560559403700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f035affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590312f0 0x560559403700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47076/0 (socket says 172.21.15.27:47076) 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.262+0000 7f035affd640 1 -- 172.21.15.27:0/4154215609 learned_addr learned my addr 172.21.15.27:0/4154215609 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035a7fc640 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x560559403180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f0360982640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03440afb60 0x7f03440b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035affd640 1 -- 172.21.15.27:0/4154215609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 msgr2=0x56055940a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035affd640 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035affd640 1 -- 172.21.15.27:0/4154215609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 msgr2=0x560559403180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:31.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035affd640 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x560559403180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035affd640 1 -- 172.21.15.27:0/4154215609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605593270d0 con 0x5605590312f0 2024-04-25T13:49:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035a7fc640 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x560559403180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:49:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035b7fe640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03440afb60 0x7f03440b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f0360982640 1 -- 172.21.15.27:0/919975662 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f03540df410 con 0x7f035c108100 2024-04-25T13:49:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f035affd640 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590312f0 0x560559403700 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f034c009bc0 tx=0x7f034c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fac819a2e304ac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.263+0000 7f0341ffb640 1 -- 172.21.15.27:0/4154215609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c005020 con 0x5605590312f0 2024-04-25T13:49:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.264+0000 7f0341ffb640 1 -- 172.21.15.27:0/4154215609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f034c0047a0 con 0x5605590312f0 2024-04-25T13:49:31.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.264+0000 7f0341ffb640 1 -- 172.21.15.27:0/4154215609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c010690 con 0x5605590312f0 2024-04-25T13:49:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f035b7fe640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03440afb60 0x7f03440b2070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f03540a4b90 tx=0x7f0354072140 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 -- 172.21.15.27:0/4154215609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590312f0 msgr2=0x560559403700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590312f0 0x560559403700 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f034c009bc0 tx=0x7f034c00e720 comp rx=0 tx=0).stop 2024-04-25T13:49:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 -- 172.21.15.27:0/4154215609 shutdown_connections 2024-04-25T13:49:31.270 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590312f0 0x560559403700 secure :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f034c009bc0 tx=0x7f034c00e720 comp rx=0 tx=0).stop 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 --2- 172.21.15.27:0/4154215609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x560559403180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 -- 172.21.15.27:0/4154215609 >> 172.21.15.27:0/4154215609 conn(0x56055904e300 msgr2=0x5605593feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 -- 172.21.15.27:0/4154215609 shutdown_connections 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.265+0000 7f036883b740 1 -- 172.21.15.27:0/4154215609 wait complete. 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.266+0000 7f036883b740 1 -- start start 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.266+0000 7f036883b740 1 Processor -- start 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.266+0000 7f036883b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.266+0000 7f036883b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x56055940d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.266+0000 7f036883b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559414490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.266+0000 7f036883b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56055940b050 con 0x560559018d80 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.271 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51358/0 (socket says 172.21.15.27:51358) 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 -- 172.21.15.27:0/1323700375 learned_addr learned my addr 172.21.15.27:0/1323700375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5605592e7b60 con 0x56055905a1b0 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56055940b1d0 con 0x5605590394d0 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035affd640 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x56055940d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559414490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f036883b740 10 client.0 _ll_register_callbacks cb 0x56055930c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 -- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 msgr2=0x56055940d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x56055940d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 -- 172.21.15.27:0/1323700375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 msgr2=0x560559414490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559414490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 -- 172.21.15.27:0/1323700375 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560559072210 con 0x5605590394d0 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035affd640 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x56055940d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:31.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f034c0047a0 con 0x5605590394d0 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f035a7fc640 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940cff0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0350006010 tx=0x7f035002a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c38b21f6b89c4dd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.267+0000 7f033affd640 10 client.26842 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f033affd640 1 -- 172.21.15.27:0/1323700375 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350051050 con 0x5605590394d0 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f033affd640 1 -- 172.21.15.27:0/1323700375 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f03500035f0 con 0x5605590394d0 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f033affd640 1 -- 172.21.15.27:0/1323700375 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0350035040 con 0x5605590394d0 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f036883b740 10 client.26842 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x56055913e620 con 0x5605590394d0 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f033affd640 1 -- 172.21.15.27:0/1323700375 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f0350054020 con 0x5605590394d0 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f036883b740 10 client.26842 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f036883b740 10 client.26842 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f036883b740 20 client.26842 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.268+0000 7f036883b740 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560558fee1c0 0x56055941abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.269+0000 7f036883b740 4 client.26842 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.269+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56055936e070 con 0x560558fee1c0 2024-04-25T13:49:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.269+0000 7f035affd640 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560558fee1c0 0x56055941abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.269+0000 7f033affd640 10 client.26842 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.269+0000 7f035affd640 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560558fee1c0 0x56055941abf0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=beb9d1bbb80ca52c server_cookie=faf583492d2a634e in_seq=0 out_seq=0 2024-04-25T13:49:31.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f033affd640 1 -- 172.21.15.27:0/1323700375 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56055936e070 con 0x560558fee1c0 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f033affd640 10 client.26842 handle_command_reply: tid=0 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f036883b740 1 client.26842 shutdown 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 msgr2=0x56055940cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f036883b740 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940cff0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0350006010 tx=0x7f035002a410 comp rx=0 tx=0).stop 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 shutdown_connections 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f036883b740 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560558fee1c0 0x56055941abf0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f036883b740 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559414490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f036883b740 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56055905a1b0 0x56055940d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.271+0000 7f036883b740 1 --2- 172.21.15.27:0/1323700375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5605590394d0 0x56055940cff0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.272+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 >> 172.21.15.27:0/1323700375 conn(0x56055904e300 msgr2=0x560559401a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.272+0000 7f033affd640 0 client.26842 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.273+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 shutdown_connections 2024-04-25T13:49:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.273+0000 7f036883b740 1 -- 172.21.15.27:0/1323700375 wait complete. 2024-04-25T13:49:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.273+0000 7f036883b740 20 client.26842 trim_cache size 0 max 16384 2024-04-25T13:49:31.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.293+0000 7f036883b740 1 Processor -- start 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.293+0000 7f036883b740 1 -- start start 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f036883b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560559018d80 0x56055940b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f036883b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590394d0 0x56055940bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f036883b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055905a1b0 0x5605593fe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f036883b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f035c1a6ef0 con 0x5605590394d0 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f036883b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f035c1a9750 con 0x560559018d80 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f036883b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f035c1a7540 con 0x56055905a1b0 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590394d0 0x56055940bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055905a1b0 0x5605593fe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590394d0 0x56055940bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47096/0 (socket says 172.21.15.27:47096) 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035affd640 1 -- 172.21.15.27:0/2307998055 learned_addr learned my addr 172.21.15.27:0/2307998055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055905a1b0 0x5605593fe4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51370/0 (socket says 172.21.15.27:51370) 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035a7fc640 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560559018d80 0x56055940b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035a7fc640 1 -- 172.21.15.27:0/2307998055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560559018d80 msgr2=0x56055940b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035a7fc640 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560559018d80 0x56055940b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035a7fc640 1 -- 172.21.15.27:0/2307998055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590394d0 msgr2=0x56055940bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035a7fc640 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590394d0 0x56055940bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035a7fc640 1 -- 172.21.15.27:0/2307998055 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055940dbf0 con 0x56055905a1b0 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.294+0000 7f035affd640 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590394d0 0x56055940bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.295+0000 7f035a7fc640 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055905a1b0 0x5605593fe4d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0350031440 tx=0x7f0350045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90e44959b3886089 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.295+0000 7f03417fa640 1 -- 172.21.15.27:0/2307998055 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350051040 con 0x56055905a1b0 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.295+0000 7f03417fa640 1 -- 172.21.15.27:0/2307998055 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f034c0047a0 con 0x56055905a1b0 2024-04-25T13:49:31.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.295+0000 7f03417fa640 1 -- 172.21.15.27:0/2307998055 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035003c480 con 0x56055905a1b0 2024-04-25T13:49:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.297+0000 7f036883b740 1 -- 172.21.15.27:0/2307998055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055905a1b0 msgr2=0x5605593fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.297+0000 7f036883b740 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055905a1b0 0x5605593fe4d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0350031440 tx=0x7f0350045000 comp rx=0 tx=0).stop 2024-04-25T13:49:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.298+0000 7f036883b740 1 -- 172.21.15.27:0/2307998055 shutdown_connections 2024-04-25T13:49:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.298+0000 7f036883b740 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055905a1b0 0x5605593fe4d0 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0350031440 tx=0x7f0350045000 comp rx=0 tx=0).stop 2024-04-25T13:49:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.298+0000 7f036883b740 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5605590394d0 0x56055940bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.298+0000 7f036883b740 1 --2- 172.21.15.27:0/2307998055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560559018d80 0x56055940b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.298+0000 7f036883b740 1 -- 172.21.15.27:0/2307998055 >> 172.21.15.27:0/2307998055 conn(0x56055904e300 msgr2=0x560559407470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:31.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.298+0000 7f036883b740 1 -- 172.21.15.27:0/2307998055 shutdown_connections 2024-04-25T13:49:31.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.299+0000 7f036883b740 1 -- 172.21.15.27:0/2307998055 wait complete. 2024-04-25T13:49:31.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.299+0000 7f036883b740 1 -- start start 2024-04-25T13:49:31.301 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f036883b740 1 Processor -- start 2024-04-25T13:49:31.301 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f036883b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559410c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f036883b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605590394d0 0x560559411150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f036883b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055947ef70 0x560559481390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f036883b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f035c1a7c10 con 0x560559018d80 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f036883b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f035c1a9080 con 0x5605590394d0 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f036883b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f035c1a82e0 con 0x56055947ef70 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f036883b740 10 client.0 _ll_register_callbacks cb 0x7f034c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559410c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f035a7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559410c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47100/0 (socket says 172.21.15.27:47100) 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f035a7fc640 1 -- 172.21.15.27:0/2996406514 learned_addr learned my addr 172.21.15.27:0/2996406514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.300+0000 7f035a7fc640 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055947ef70 0x560559481390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f035affd640 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605590394d0 0x560559411150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f035a7fc640 1 -- 172.21.15.27:0/2996406514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605590394d0 msgr2=0x560559411150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f035a7fc640 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605590394d0 0x560559411150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f035a7fc640 1 -- 172.21.15.27:0/2996406514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 msgr2=0x560559410c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f035a7fc640 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559410c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f035a7fc640 1 -- 172.21.15.27:0/2996406514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605594818d0 con 0x56055947ef70 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f035a7fc640 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055947ef70 0x560559481390 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0350003960 tx=0x7f03500056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18eaee846bf71b03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f033affd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f033affd640 1 -- 172.21.15.27:0/2996406514 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350051070 con 0x56055947ef70 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f033affd640 1 -- 172.21.15.27:0/2996406514 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f035003c690 con 0x56055947ef70 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.301+0000 7f036883b740 1 -- 172.21.15.27:0/2996406514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f034c0047a0 con 0x56055947ef70 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f033affd640 1 -- 172.21.15.27:0/2996406514 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f035003c890 con 0x56055947ef70 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f036883b740 10 client.26848 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f036883b740 1 -- 172.21.15.27:0/2996406514 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560559482b00 con 0x56055947ef70 2024-04-25T13:49:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f033affd640 1 -- 172.21.15.27:0/2996406514 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0350041030 con 0x56055947ef70 2024-04-25T13:49:31.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f033affd640 1 -- 172.21.15.27:0/2996406514 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f0350054070 con 0x56055947ef70 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f036883b740 10 client.26848 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f036883b740 10 client.26848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f036883b740 20 client.26848 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f036883b740 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5605594833a0 0x5605594857c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f036883b740 4 client.26848 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.302+0000 7f036883b740 1 -- 172.21.15.27:0/2996406514 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f034c0047a0 con 0x5605594833a0 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.303+0000 7f035affd640 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5605594833a0 0x5605594857c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.303+0000 7f033affd640 10 client.26848 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:31.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.303+0000 7f035affd640 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5605594833a0 0x5605594857c0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6f7fcbdaa06bfbc server_cookie=955d0ea24b65c73a in_seq=0 out_seq=0 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.357+0000 7f033affd640 1 -- 172.21.15.27:0/2996406514 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f034c0047a0 con 0x5605594833a0 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.357+0000 7f033affd640 10 client.26848 handle_command_reply: tid=0 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.357+0000 7f036883b740 1 client.26848 shutdown 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.357+0000 7f036883b740 1 -- 172.21.15.27:0/2996406514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055947ef70 msgr2=0x560559481390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.357+0000 7f036883b740 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055947ef70 0x560559481390 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0350003960 tx=0x7f03500056d0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 1 -- 172.21.15.27:0/2996406514 shutdown_connections 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5605594833a0 0x5605594857c0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56055947ef70 0x560559481390 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605590394d0 0x560559411150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f033affd640 0 client.26848 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 1 --2- 172.21.15.27:0/2996406514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560559018d80 0x560559410c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 1 -- 172.21.15.27:0/2996406514 >> 172.21.15.27:0/2996406514 conn(0x56055904e300 msgr2=0x560559407470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 1 -- 172.21.15.27:0/2996406514 shutdown_connections 2024-04-25T13:49:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 1 -- 172.21.15.27:0/2996406514 wait complete. 2024-04-25T13:49:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.358+0000 7f036883b740 20 client.26848 trim_cache size 0 max 16384 2024-04-25T13:49:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:49:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:49:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:49:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 -- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03440afb60 msgr2=0x7f03440b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03440afb60 0x7f03440b2070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f03540a4b90 tx=0x7f0354072140 comp rx=0 tx=0).stop 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 -- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c108100 msgr2=0x7f035c1a38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c108100 0x7f035c1a38e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0354009fd0 tx=0x7f035400ef10 comp rx=0 tx=0).stop 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 -- 172.21.15.27:0/919975662 shutdown_connections 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f03440afb60 0x7f03440b2070 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f035c108100 0x7f035c1a38e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f035c103400 0x7f035c19c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 --2- 172.21.15.27:0/919975662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f035c1021c0 0x7f035c19c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 -- 172.21.15.27:0/919975662 >> 172.21.15.27:0/919975662 conn(0x7f035c0fd390 msgr2=0x7f035c0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 -- 172.21.15.27:0/919975662 shutdown_connections 2024-04-25T13:49:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:31.359+0000 7f0361984640 1 -- 172.21.15.27:0/919975662 wait complete. 2024-04-25T13:49:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:32 smithi027 ceph-mon[29140]: pgmap v608: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 MiB/s wr, 673 op/s 2024-04-25T13:49:32.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:32 smithi122 ceph-mon[32843]: pgmap v608: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 MiB/s wr, 673 op/s 2024-04-25T13:49:32.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:32 smithi045 ceph-mon[32731]: pgmap v608: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 MiB/s wr, 673 op/s 2024-04-25T13:49:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 505.0 s, 1505.8 tps, lat 66.370 ms stddev 71.623 2024-04-25T13:49:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:34 smithi027 ceph-mon[29140]: pgmap v609: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-04-25T13:49:34.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:34 smithi122 ceph-mon[32843]: pgmap v609: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-04-25T13:49:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:34 smithi045 ceph-mon[32731]: pgmap v609: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-04-25T13:49:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:36 smithi027 ceph-mon[29140]: pgmap v610: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.6 MiB/s wr, 576 op/s 2024-04-25T13:49:36.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:36 smithi122 ceph-mon[32843]: pgmap v610: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.6 MiB/s wr, 576 op/s 2024-04-25T13:49:36.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:36 smithi045 ceph-mon[32731]: pgmap v610: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.6 MiB/s wr, 576 op/s 2024-04-25T13:49:36.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:49:36.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:49:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.892+0000 7f87db620640 1 -- 172.21.15.27:0/1650852443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 msgr2=0x7f87d4110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.892+0000 7f87db620640 1 --2- 172.21.15.27:0/1650852443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 0x7f87d4110440 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009be0 tx=0x7f87cc02a710 comp rx=0 tx=0).stop 2024-04-25T13:49:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.892+0000 7f87db620640 1 -- 172.21.15.27:0/1650852443 shutdown_connections 2024-04-25T13:49:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.892+0000 7f87db620640 1 --2- 172.21.15.27:0/1650852443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 0x7f87d4110440 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.892+0000 7f87db620640 1 --2- 172.21.15.27:0/1650852443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d410b310 0x7f87d410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.892+0000 7f87db620640 1 --2- 172.21.15.27:0/1650852443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87d410a420 0x7f87d410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.892+0000 7f87db620640 1 -- 172.21.15.27:0/1650852443 >> 172.21.15.27:0/1650852443 conn(0x7f87d40704e0 msgr2=0x7f87d4072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:36.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.895+0000 7f87db620640 1 -- 172.21.15.27:0/1650852443 shutdown_connections 2024-04-25T13:49:36.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.899+0000 7f87db620640 1 -- 172.21.15.27:0/1650852443 wait complete. 2024-04-25T13:49:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.900+0000 7f87db620640 1 Processor -- start 2024-04-25T13:49:36.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.900+0000 7f87db620640 1 -- start start 2024-04-25T13:49:36.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.901+0000 7f87db620640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87d410a420 0x7f87d41a9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.901+0000 7f87db620640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d410b310 0x7f87d41a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.901+0000 7f87db620640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 0x7f87d41b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.901+0000 7f87db620640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f87d411e4f0 con 0x7f87d410a420 2024-04-25T13:49:36.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.901+0000 7f87db620640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f87d411e370 con 0x7f87d410b310 2024-04-25T13:49:36.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.901+0000 7f87db620640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f87d411e670 con 0x7f87d4110030 2024-04-25T13:49:36.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.901+0000 7f87d9bbd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 0x7f87d41b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.905+0000 7f87d9bbd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 0x7f87d41b05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39248/0 (socket says 172.21.15.27:39248) 2024-04-25T13:49:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.905+0000 7f87d9bbd640 1 -- 172.21.15.27:0/3865976245 learned_addr learned my addr 172.21.15.27:0/3865976245 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:36.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.906+0000 7f87d93bc640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87d410a420 0x7f87d41a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.907+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d410b310 0x7f87d41a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.911+0000 7f87d9bbd640 1 -- 172.21.15.27:0/3865976245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d410b310 msgr2=0x7f87d41a9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.911+0000 7f87d9bbd640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d410b310 0x7f87d41a9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.911+0000 7f87d9bbd640 1 -- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87d410a420 msgr2=0x7f87d41a9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.911+0000 7f87d9bbd640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87d410a420 0x7f87d41a9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.911+0000 7f87d9bbd640 1 -- 172.21.15.27:0/3865976245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d41b0c10 con 0x7f87d4110030 2024-04-25T13:49:36.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.915+0000 7f87d9bbd640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 0x7f87d41b05a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f87cc02ad00 tx=0x7f87cc030d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.917+0000 7f87ca7fc640 1 -- 172.21.15.27:0/3865976245 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc041020 con 0x7f87d4110030 2024-04-25T13:49:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.917+0000 7f87ca7fc640 1 -- 172.21.15.27:0/3865976245 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87cc003e20 con 0x7f87d4110030 2024-04-25T13:49:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.917+0000 7f87ca7fc640 1 -- 172.21.15.27:0/3865976245 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc036040 con 0x7f87d4110030 2024-04-25T13:49:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.917+0000 7f87db620640 1 -- 172.21.15.27:0/3865976245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d41b0ea0 con 0x7f87d4110030 2024-04-25T13:49:36.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.917+0000 7f87db620640 1 -- 172.21.15.27:0/3865976245 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d41b12d0 con 0x7f87d4110030 2024-04-25T13:49:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.926+0000 7f87e25ad740 1 Processor -- start 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87e25ad740 1 -- start start 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced146a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd7c4d0 0x55bced146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd846b0 0x55bced147c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87e25ad740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bced0abd80 con 0x55bcecd846b0 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87e25ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55bced0658b0 con 0x55bcecda5390 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87e25ad740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55bced0a7d40 con 0x55bcecd7c4d0 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87d93bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd7c4d0 0x55bced146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87d93bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd7c4d0 0x55bced146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39258/0 (socket says 172.21.15.27:39258) 2024-04-25T13:49:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.930+0000 7f87d93bc640 1 -- 172.21.15.27:0/248026741 learned_addr learned my addr 172.21.15.27:0/248026741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.931+0000 7f87d93bc640 1 -- 172.21.15.27:0/248026741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 msgr2=0x55bced146a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.931+0000 7f87d93bc640 1 --2- 172.21.15.27:0/248026741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.931+0000 7f87d93bc640 1 -- 172.21.15.27:0/248026741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd846b0 msgr2=0x55bced147c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.931+0000 7f87d93bc640 1 --2- 172.21.15.27:0/248026741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd846b0 0x55bced147c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.931+0000 7f87d93bc640 1 -- 172.21.15.27:0/248026741 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bced036a40 con 0x55bcecd7c4d0 2024-04-25T13:49:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.931+0000 7f87ca7fc640 1 -- 172.21.15.27:0/3865976245 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f87cc044020 con 0x7f87d4110030 2024-04-25T13:49:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.932+0000 7f87d93bc640 1 --2- 172.21.15.27:0/248026741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd7c4d0 0x55bced146fe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009a40 tx=0x7f87d0009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d008dc242b397449 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:36.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.932+0000 7f87ca7fc640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87c00afd00 0x7f87c00b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.933+0000 7f87ca7fc640 1 -- 172.21.15.27:0/3865976245 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f87cc0fee40 con 0x7f87d4110030 2024-04-25T13:49:36.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.933+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87c00afd00 0x7f87c00b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.934+0000 7f87bf7fe640 1 -- 172.21.15.27:0/248026741 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0017070 con 0x55bcecd7c4d0 2024-04-25T13:49:36.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.934+0000 7f87bf7fe640 1 -- 172.21.15.27:0/248026741 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0013470 con 0x55bcecd7c4d0 2024-04-25T13:49:36.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.934+0000 7f87bf7fe640 1 -- 172.21.15.27:0/248026741 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0012610 con 0x55bcecd7c4d0 2024-04-25T13:49:36.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.934+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87c00afd00 0x7f87c00b2210 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55bced147ab0 tx=0x7f87c40098b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:36.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.935+0000 7f87e25ad740 1 -- 172.21.15.27:0/248026741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd7c4d0 msgr2=0x55bced146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.935+0000 7f87e25ad740 1 --2- 172.21.15.27:0/248026741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd7c4d0 0x55bced146fe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009a40 tx=0x7f87d0009f00 comp rx=0 tx=0).stop 2024-04-25T13:49:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.936+0000 7f87e25ad740 1 -- 172.21.15.27:0/248026741 shutdown_connections 2024-04-25T13:49:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.936+0000 7f87e25ad740 1 --2- 172.21.15.27:0/248026741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd846b0 0x55bced147c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.936+0000 7f87e25ad740 1 --2- 172.21.15.27:0/248026741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd7c4d0 0x55bced146fe0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.936+0000 7f87e25ad740 1 --2- 172.21.15.27:0/248026741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced146a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.936+0000 7f87e25ad740 1 -- 172.21.15.27:0/248026741 >> 172.21.15.27:0/248026741 conn(0x55bcecd994e0 msgr2=0x55bced142270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.937+0000 7f87e25ad740 1 -- 172.21.15.27:0/248026741 shutdown_connections 2024-04-25T13:49:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.937+0000 7f87e25ad740 1 -- 172.21.15.27:0/248026741 wait complete. 2024-04-25T13:49:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.937+0000 7f87e25ad740 1 -- start start 2024-04-25T13:49:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.937+0000 7f87e25ad740 1 Processor -- start 2024-04-25T13:49:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.938+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1507c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.938+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced150d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.938+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.938+0000 7f87e25ad740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bced148780 con 0x55bcecd7c4d0 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.938+0000 7f87e25ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55bced032d10 con 0x55bcecda5390 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.938+0000 7f87e25ad740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55bced148900 con 0x55bcecd846b0 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.939+0000 7f87d93bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.939+0000 7f87e25ad740 10 client.0 _ll_register_callbacks cb 0x55bced05f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.939+0000 7f87d93bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1507c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34620/0 (socket says 172.21.15.27:34620) 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.939+0000 7f87d93bc640 1 -- 172.21.15.27:0/2078365928 learned_addr learned my addr 172.21.15.27:0/2078365928 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.939+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.940+0000 7f87d8bbb640 1 -- 172.21.15.27:0/2078365928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 msgr2=0x55bced157bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:49:36.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.940+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced157bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.940+0000 7f87d8bbb640 1 -- 172.21.15.27:0/2078365928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 msgr2=0x55bced1507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:36.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.940+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.940+0000 7f87d8bbb640 1 -- 172.21.15.27:0/2078365928 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcece877d0 con 0x55bcecd846b0 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.941+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced150d00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f87c4005fd0 tx=0x7f87c4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d25a2870f0e6c59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.941+0000 7f87bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.941+0000 7f87bcff9640 1 -- 172.21.15.27:0/2078365928 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c400e9c0 con 0x55bcecd846b0 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.941+0000 7f87bcff9640 1 -- 172.21.15.27:0/2078365928 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f87c400eb60 con 0x55bcecd846b0 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.941+0000 7f87e25ad740 1 -- 172.21.15.27:0/2078365928 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d0013470 con 0x55bcecd846b0 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.941+0000 7f87bcff9640 1 -- 172.21.15.27:0/2078365928 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f87c400ed60 con 0x55bcecd846b0 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87e25ad740 10 client.26860 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87e25ad740 1 -- 172.21.15.27:0/2078365928 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55bcecdbd3f0 con 0x55bcecd846b0 2024-04-25T13:49:36.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87bcff9640 1 -- 172.21.15.27:0/2078365928 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0013470 con 0x55bcecd846b0 2024-04-25T13:49:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87bcff9640 1 -- 172.21.15.27:0/2078365928 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f87c401e420 con 0x55bcecd846b0 2024-04-25T13:49:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87e25ad740 10 client.26860 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87e25ad740 10 client.26860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87e25ad740 20 client.26860 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87e25ad740 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bcecd63f60 0x55bced15e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87e25ad740 4 client.26860 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.942+0000 7f87e25ad740 1 -- 172.21.15.27:0/2078365928 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bced0b9990 con 0x55bcecd63f60 2024-04-25T13:49:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.943+0000 7f87d93bc640 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bcecd63f60 0x55bced15e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.943+0000 7f87d93bc640 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bcecd63f60 0x55bced15e660 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69ee557514a147b9 server_cookie=8b3f126c06aaf198 in_seq=0 out_seq=0 2024-04-25T13:49:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.944+0000 7f87bcff9640 10 client.26860 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.944+0000 7f87bcff9640 1 -- 172.21.15.27:0/2078365928 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bced0b9990 con 0x55bcecd63f60 2024-04-25T13:49:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.944+0000 7f87bcff9640 10 client.26860 handle_command_reply: tid=0 2024-04-25T13:49:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.944+0000 7f87e25ad740 1 client.26860 shutdown 2024-04-25T13:49:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87e25ad740 1 -- 172.21.15.27:0/2078365928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 msgr2=0x55bced150d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87e25ad740 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced150d00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f87c4005fd0 tx=0x7f87c4009260 comp rx=0 tx=0).stop 2024-04-25T13:49:36.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87e25ad740 1 -- 172.21.15.27:0/2078365928 shutdown_connections 2024-04-25T13:49:36.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87e25ad740 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bcecd63f60 0x55bced15e660 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87e25ad740 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87e25ad740 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced150d00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87e25ad740 1 --2- 172.21.15.27:0/2078365928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87e25ad740 1 -- 172.21.15.27:0/2078365928 >> 172.21.15.27:0/2078365928 conn(0x55bcecd994e0 msgr2=0x55bced145180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.945+0000 7f87bcff9640 0 client.26860 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.946+0000 7f87e25ad740 1 -- 172.21.15.27:0/2078365928 shutdown_connections 2024-04-25T13:49:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.946+0000 7f87e25ad740 1 -- 172.21.15.27:0/2078365928 wait complete. 2024-04-25T13:49:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.947+0000 7f87e25ad740 20 client.26860 trim_cache size 0 max 16384 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.975+0000 7f87e25ad740 1 Processor -- start 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.975+0000 7f87e25ad740 1 -- start start 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.975+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced13d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.975+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced14b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced14b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87e25ad740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f87b800c460 con 0x55bcecd7c4d0 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87e25ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f87b800ece0 con 0x55bcecda5390 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87e25ad740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f87b800cad0 con 0x55bcecd846b0 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d93bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced13d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d8bbb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced14b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d93bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced13d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34626/0 (socket says 172.21.15.27:34626) 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d93bc640 1 -- 172.21.15.27:0/711975436 learned_addr learned my addr 172.21.15.27:0/711975436 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d9bbd640 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced14b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d8bbb640 1 -- 172.21.15.27:0/711975436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 msgr2=0x55bced14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced14b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d8bbb640 1 -- 172.21.15.27:0/711975436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 msgr2=0x55bced13d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced13d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d8bbb640 1 -- 172.21.15.27:0/711975436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcecd7d0f0 con 0x55bcecd846b0 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d93bc640 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced13d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.976+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced14b4b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f87c4000c00 tx=0x7f87c4005df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25f7b7c5bca5d9d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.977+0000 7f87beffd640 1 -- 172.21.15.27:0/711975436 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c4035070 con 0x55bcecd846b0 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.977+0000 7f87beffd640 1 -- 172.21.15.27:0/711975436 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87c401ea60 con 0x55bcecd846b0 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.977+0000 7f87beffd640 1 -- 172.21.15.27:0/711975436 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c4030970 con 0x55bcecd846b0 2024-04-25T13:49:36.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.977+0000 7f87e25ad740 1 -- 172.21.15.27:0/711975436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 msgr2=0x55bced14b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.977+0000 7f87e25ad740 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced14b4b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f87c4000c00 tx=0x7f87c4005df0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.977+0000 7f87e25ad740 1 -- 172.21.15.27:0/711975436 shutdown_connections 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.977+0000 7f87e25ad740 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecda5390 0x55bced14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecd846b0 0x55bced14b4b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 --2- 172.21.15.27:0/711975436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced13d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 -- 172.21.15.27:0/711975436 >> 172.21.15.27:0/711975436 conn(0x55bcecd994e0 msgr2=0x55bced1507c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 -- 172.21.15.27:0/711975436 shutdown_connections 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 -- 172.21.15.27:0/711975436 wait complete. 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 -- start start 2024-04-25T13:49:36.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 Processor -- start 2024-04-25T13:49:36.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1545b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.978+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecd846b0 0x55bced154af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87e25ad740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecda5390 0x55bced157a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87e25ad740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f87b800d1a0 con 0x55bcecd7c4d0 2024-04-25T13:49:36.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87e25ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f87b800e610 con 0x55bcecd846b0 2024-04-25T13:49:36.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d93bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87e25ad740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f87b800d870 con 0x55bcecda5390 2024-04-25T13:49:36.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d93bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34640/0 (socket says 172.21.15.27:34640) 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d9bbd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecda5390 0x55bced157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecd846b0 0x55bced154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d93bc640 1 -- 172.21.15.27:0/166451930 learned_addr learned my addr 172.21.15.27:0/166451930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87e25ad740 10 client.0 _ll_register_callbacks cb 0x7f87c4009750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d93bc640 1 -- 172.21.15.27:0/166451930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecda5390 msgr2=0x55bced157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d93bc640 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecda5390 0x55bced157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d93bc640 1 -- 172.21.15.27:0/166451930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecd846b0 msgr2=0x55bced154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d93bc640 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecd846b0 0x55bced154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.979+0000 7f87d93bc640 1 -- 172.21.15.27:0/166451930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bced06c4b0 con 0x55bcecd7c4d0 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.980+0000 7f87d93bc640 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1545b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x55bced15ee00 tx=0x7f87d00047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6a2eb4589a374 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.980+0000 7f87bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:49:36.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.980+0000 7f87bcff9640 1 -- 172.21.15.27:0/166451930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d002b070 con 0x55bcecd7c4d0 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.980+0000 7f87bcff9640 1 -- 172.21.15.27:0/166451930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f87d0025610 con 0x55bcecd7c4d0 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.981+0000 7f87e25ad740 1 -- 172.21.15.27:0/166451930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87c401ea60 con 0x55bcecd7c4d0 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.981+0000 7f87bcff9640 1 -- 172.21.15.27:0/166451930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f87d0017040 con 0x55bcecd7c4d0 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.981+0000 7f87e25ad740 10 client.17097 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.981+0000 7f87e25ad740 1 -- 172.21.15.27:0/166451930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55bced1c5190 con 0x55bcecd7c4d0 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87bcff9640 1 -- 172.21.15.27:0/166451930 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87c401ea60 con 0x55bcecd7c4d0 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87bcff9640 1 -- 172.21.15.27:0/166451930 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f87d00136d0 con 0x55bcecd7c4d0 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87e25ad740 10 client.17097 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87e25ad740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87e25ad740 20 client.17097 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87e25ad740 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bcecd393a0 0x55bced1c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87e25ad740 4 client.17097 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87e25ad740 1 -- 172.21.15.27:0/166451930 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bced0b3800 con 0x55bcecd393a0 2024-04-25T13:49:36.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.982+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bcecd393a0 0x55bced1c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87d8bbb640 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bcecd393a0 0x55bced1c78d0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68fdd4c63d42872c server_cookie=4fc5288a72bdeff5 in_seq=0 out_seq=0 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87bcff9640 10 client.17097 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87bcff9640 1 -- 172.21.15.27:0/166451930 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bced0b3800 con 0x55bcecd393a0 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87bcff9640 10 client.17097 handle_command_reply: tid=0 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 client.17097 shutdown 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 -- 172.21.15.27:0/166451930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 msgr2=0x55bced1545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1545b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x55bced15ee00 tx=0x7f87d00047d0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 -- 172.21.15.27:0/166451930 shutdown_connections 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bcecd393a0 0x55bced1c78d0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bcecda5390 0x55bced157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bcecd846b0 0x55bced154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 --2- 172.21.15.27:0/166451930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bcecd7c4d0 0x55bced1545b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87e25ad740 1 -- 172.21.15.27:0/166451930 >> 172.21.15.27:0/166451930 conn(0x55bcecd994e0 msgr2=0x55bced14b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:36.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.983+0000 7f87bcff9640 0 client.17097 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:36.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87e25ad740 1 -- 172.21.15.27:0/166451930 shutdown_connections 2024-04-25T13:49:36.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87e25ad740 1 -- 172.21.15.27:0/166451930 wait complete. 2024-04-25T13:49:36.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87e25ad740 20 client.17097 trim_cache size 0 max 16384 2024-04-25T13:49:36.988 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:49:36.988 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:49:36.988 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:49:36.988 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:49:36.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 -- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87c00afd00 msgr2=0x7f87c00b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87c00afd00 0x7f87c00b2210 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55bced147ab0 tx=0x7f87c40098b0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 -- 172.21.15.27:0/3865976245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 msgr2=0x7f87d41b05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:36.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 0x7f87d41b05a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f87cc02ad00 tx=0x7f87cc030d50 comp rx=0 tx=0).stop 2024-04-25T13:49:36.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 -- 172.21.15.27:0/3865976245 shutdown_connections 2024-04-25T13:49:36.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87c00afd00 0x7f87c00b2210 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87d4110030 0x7f87d41b05a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d410b310 0x7f87d41a9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 --2- 172.21.15.27:0/3865976245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87d410a420 0x7f87d41a9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:36.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 -- 172.21.15.27:0/3865976245 >> 172.21.15.27:0/3865976245 conn(0x7f87d40704e0 msgr2=0x7f87d410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:36.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 -- 172.21.15.27:0/3865976245 shutdown_connections 2024-04-25T13:49:36.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:36.984+0000 7f87db620640 1 -- 172.21.15.27:0/3865976245 wait complete. 2024-04-25T13:49:37.000 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:49:37.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:49:37.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 -- 172.21.15.27:0/2719437466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 msgr2=0x7f15bc1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/2719437466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 0x7f15bc1023f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009900 tx=0x7f15b802a410 comp rx=0 tx=0).stop 2024-04-25T13:49:37.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 -- 172.21.15.27:0/2719437466 shutdown_connections 2024-04-25T13:49:37.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/2719437466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15bc103520 0x7f15bc103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/2719437466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15bc1029c0 0x7f15bc102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/2719437466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 0x7f15bc1023f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 -- 172.21.15.27:0/2719437466 >> 172.21.15.27:0/2719437466 conn(0x7f15bc0fdbd0 msgr2=0x7f15bc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 -- 172.21.15.27:0/2719437466 shutdown_connections 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.163+0000 7f15c3bf3640 1 -- 172.21.15.27:0/2719437466 wait complete. 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c3bf3640 1 Processor -- start 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c3bf3640 1 -- start start 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c3bf3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 0x7f15bc19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c3bf3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15bc1029c0 0x7f15bc19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c3bf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15bc103520 0x7f15bc1a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c3bf3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1117b0 con 0x7f15bc101fe0 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c3bf3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc111630 con 0x7f15bc103520 2024-04-25T13:49:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c3bf3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc111930 con 0x7f15bc1029c0 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c198f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 0x7f15bc19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.164+0000 7f15c198f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 0x7f15bc19c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34666/0 (socket says 172.21.15.27:34666) 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c198f640 1 -- 172.21.15.27:0/1025784556 learned_addr learned my addr 172.21.15.27:0/1025784556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c2190640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15bc103520 0x7f15bc1a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c118e640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15bc1029c0 0x7f15bc19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c198f640 1 -- 172.21.15.27:0/1025784556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15bc1029c0 msgr2=0x7f15bc19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c198f640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15bc1029c0 0x7f15bc19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c198f640 1 -- 172.21.15.27:0/1025784556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15bc103520 msgr2=0x7f15bc1a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c198f640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15bc103520 0x7f15bc1a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c198f640 1 -- 172.21.15.27:0/1025784556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc1a3f20 con 0x7f15bc101fe0 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c2190640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15bc103520 0x7f15bc1a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c118e640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15bc1029c0 0x7f15bc19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15c198f640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 0x7f15bc19c3e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f15b8006010 tx=0x7f15b8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.165+0000 7f15b2ffd640 1 -- 172.21.15.27:0/1025784556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b803f030 con 0x7f15bc101fe0 2024-04-25T13:49:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.166+0000 7f15b2ffd640 1 -- 172.21.15.27:0/1025784556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15b8031910 con 0x7f15bc101fe0 2024-04-25T13:49:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.166+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15bc1a41b0 con 0x7f15bc101fe0 2024-04-25T13:49:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.166+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15bc1a46f0 con 0x7f15bc101fe0 2024-04-25T13:49:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.166+0000 7f15b2ffd640 1 -- 172.21.15.27:0/1025784556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b8035070 con 0x7f15bc101fe0 2024-04-25T13:49:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.167+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15bc10d440 con 0x7f15bc101fe0 2024-04-25T13:49:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.169+0000 7f15b2ffd640 1 -- 172.21.15.27:0/1025784556 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f15b8042030 con 0x7f15bc101fe0 2024-04-25T13:49:37.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.170+0000 7f15b2ffd640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f15a80afb70 0x7f15a80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.170+0000 7f15b2ffd640 1 -- 172.21.15.27:0/1025784556 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f15b80ff320 con 0x7f15bc101fe0 2024-04-25T13:49:37.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.170+0000 7f15c118e640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f15a80afb70 0x7f15a80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.171+0000 7f15c118e640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f15a80afb70 0x7f15a80b2080 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f15bc19d480 tx=0x7f15ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:37.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.175+0000 7f15b2ffd640 1 -- 172.21.15.27:0/1025784556 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f15b80c7880 con 0x7f15bc101fe0 2024-04-25T13:49:37.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.541+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15bc02db50 con 0x7f15bc101fe0 2024-04-25T13:49:37.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.543+0000 7f15b2ffd640 1 -- 172.21.15.27:0/1025784556 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f15b80cbb80 con 0x7f15bc101fe0 2024-04-25T13:49:37.545 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:49:37.545 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:49:37.545 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.549+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f15a80afb70 msgr2=0x7f15a80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.549+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f15a80afb70 0x7f15a80b2080 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f15bc19d480 tx=0x7f15ac005f90 comp rx=0 tx=0).stop 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.549+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 msgr2=0x7f15bc19c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.549+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 0x7f15bc19c3e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f15b8006010 tx=0x7f15b8004270 comp rx=0 tx=0).stop 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.550+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 shutdown_connections 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.550+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f15a80afb70 0x7f15a80b2080 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.550+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15bc103520 0x7f15bc1a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.550+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15bc1029c0 0x7f15bc19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.550+0000 7f15c3bf3640 1 --2- 172.21.15.27:0/1025784556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15bc101fe0 0x7f15bc19c3e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.550+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 >> 172.21.15.27:0/1025784556 conn(0x7f15bc0fdbd0 msgr2=0x7f15bc10a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:37.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.552+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 shutdown_connections 2024-04-25T13:49:37.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.553+0000 7f15c3bf3640 1 -- 172.21.15.27:0/1025784556 wait complete. 2024-04-25T13:49:37.570 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:49:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.750+0000 7fafb4f1c640 1 -- 172.21.15.27:0/4060578055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafb0107680 msgr2=0x7fafb0107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.750+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/4060578055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafb0107680 0x7fafb0107a90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009ca0 tx=0x7fafa802a8f0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.754+0000 7fafb4f1c640 1 -- 172.21.15.27:0/4060578055 shutdown_connections 2024-04-25T13:49:37.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.754+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/4060578055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafb0107680 0x7fafb0107a90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.754+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/4060578055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb0104d20 0x7fafb0107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.754+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/4060578055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb01023c0 0x7fafb01047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.754+0000 7fafb4f1c640 1 -- 172.21.15.27:0/4060578055 >> 172.21.15.27:0/4060578055 conn(0x7fafb00fdbd0 msgr2=0x7fafb0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:37.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.759+0000 7fafb4f1c640 1 -- 172.21.15.27:0/4060578055 shutdown_connections 2024-04-25T13:49:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.763+0000 7fafb4f1c640 1 -- 172.21.15.27:0/4060578055 wait complete. 2024-04-25T13:49:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.764+0000 7fafb4f1c640 1 Processor -- start 2024-04-25T13:49:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.768+0000 7fafb4f1c640 1 -- start start 2024-04-25T13:49:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.768+0000 7fafb4f1c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb01023c0 0x7fafb010ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.768+0000 7fafb4f1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafb0104d20 0x7fafb010e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.768+0000 7fafb4f1c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb0107680 0x7fafb010f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.768+0000 7fafb4f1c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0115c00 con 0x7fafb01023c0 2024-04-25T13:49:37.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.768+0000 7fafb4f1c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0115a80 con 0x7fafb0104d20 2024-04-25T13:49:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.768+0000 7fafb4f1c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0115d80 con 0x7fafb0107680 2024-04-25T13:49:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.770+0000 7fafaed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb0107680 0x7fafb010f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.770+0000 7fafaed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb0107680 0x7fafb010f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39326/0 (socket says 172.21.15.27:39326) 2024-04-25T13:49:37.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.770+0000 7fafaed9d640 1 -- 172.21.15.27:0/600308417 learned_addr learned my addr 172.21.15.27:0/600308417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.772+0000 7fafae59c640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb01023c0 0x7fafb010ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.772+0000 7fafaed9d640 1 -- 172.21.15.27:0/600308417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafb0104d20 msgr2=0x7fafb010e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.772+0000 7fafaed9d640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafb0104d20 0x7fafb010e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.772+0000 7fafaed9d640 1 -- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb01023c0 msgr2=0x7fafb010ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.772+0000 7fafaed9d640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb01023c0 0x7fafb010ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.772+0000 7fafaed9d640 1 -- 172.21.15.27:0/600308417 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb010f8e0 con 0x7fafb0107680 2024-04-25T13:49:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.773+0000 7fafaed9d640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb0107680 0x7fafb010f270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fafa8004af0 tx=0x7fafa8003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.778+0000 7faf9f7fe640 1 -- 172.21.15.27:0/600308417 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8004040 con 0x7fafb0107680 2024-04-25T13:49:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.778+0000 7faf9f7fe640 1 -- 172.21.15.27:0/600308417 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafa802d070 con 0x7fafb0107680 2024-04-25T13:49:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.778+0000 7faf9f7fe640 1 -- 172.21.15.27:0/600308417 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8030b80 con 0x7fafb0107680 2024-04-25T13:49:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.779+0000 7fafb4f1c640 1 -- 172.21.15.27:0/600308417 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb010fb70 con 0x7fafb0107680 2024-04-25T13:49:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.779+0000 7fafb4f1c640 1 -- 172.21.15.27:0/600308417 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb0110070 con 0x7fafb0107680 2024-04-25T13:49:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.784+0000 7fafbbdcc740 1 Processor -- start 2024-04-25T13:49:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.784+0000 7faf9f7fe640 1 -- 172.21.15.27:0/600308417 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fafa8030d20 con 0x7fafb0107680 2024-04-25T13:49:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7fafbbdcc740 1 -- start start 2024-04-25T13:49:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7faf9f7fe640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faf980afc80 0x7faf980b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c352a390 0x5573c38cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35096b0 0x5573c38cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5573c3830d80 con 0x5573c35014d0 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5573c37eaa20 con 0x5573c35096b0 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5573c382cd40 con 0x5573c352a390 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7faf9f7fe640 1 -- 172.21.15.27:0/600308417 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fafa8100280 con 0x7fafb0107680 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.786+0000 7fafadd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafadd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34684/0 (socket says 172.21.15.27:34684) 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafadd9b640 1 -- 172.21.15.27:0/306802187 learned_addr learned my addr 172.21.15.27:0/306802187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafaed9d640 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35096b0 0x5573c38cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafadd9b640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faf980afc80 0x7faf980b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafae59c640 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c352a390 0x5573c38cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafaed9d640 1 -- 172.21.15.27:0/306802187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c352a390 msgr2=0x5573c38cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafaed9d640 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c352a390 0x5573c38cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafaed9d640 1 -- 172.21.15.27:0/306802187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 msgr2=0x5573c38ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafaed9d640 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafaed9d640 1 -- 172.21.15.27:0/306802187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573c37f5c30 con 0x5573c35096b0 2024-04-25T13:49:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.787+0000 7fafaed9d640 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35096b0 0x5573c38cda20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fafa8031a00 tx=0x7fafa80305f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b74142c35e2b34e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.788+0000 7fafadd9b640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faf980afc80 0x7faf980b2190 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fafa4005d00 tx=0x7fafa400d680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.788+0000 7faf9cff9640 1 -- 172.21.15.27:0/306802187 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa80c8e90 con 0x5573c35096b0 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.788+0000 7faf9cff9640 1 -- 172.21.15.27:0/306802187 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafa8036030 con 0x5573c35096b0 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.788+0000 7faf9cff9640 1 -- 172.21.15.27:0/306802187 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa80c4a20 con 0x5573c35096b0 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 -- 172.21.15.27:0/306802187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35096b0 msgr2=0x5573c38cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35096b0 0x5573c38cda20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fafa8031a00 tx=0x7fafa80305f0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 -- 172.21.15.27:0/306802187 shutdown_connections 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35096b0 0x5573c38cda20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/306802187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c352a390 0x5573c38cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 -- 172.21.15.27:0/306802187 >> 172.21.15.27:0/306802187 conn(0x5573c351e4e0 msgr2=0x5573c38c8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 -- 172.21.15.27:0/306802187 shutdown_connections 2024-04-25T13:49:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.789+0000 7fafbbdcc740 1 -- 172.21.15.27:0/306802187 wait complete. 2024-04-25T13:49:37.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 1 -- start start 2024-04-25T13:49:37.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 1 Processor -- start 2024-04-25T13:49:37.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35014d0 0x5573c38d6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38d6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c352a390 0x5573c38dd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5573c38ce2f0 con 0x5573c352a390 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5573c37b7dc0 con 0x5573c35014d0 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5573c38ce470 con 0x5573c35096b0 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.790+0000 7fafbbdcc740 10 client.0 _ll_register_callbacks cb 0x5573c37d3600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafadd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38d6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafadd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38d6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39336/0 (socket says 172.21.15.27:39336) 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafadd9b640 1 -- 172.21.15.27:0/1819058653 learned_addr learned my addr 172.21.15.27:0/1819058653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafae59c640 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35014d0 0x5573c38d6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafaed9d640 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c352a390 0x5573c38dd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafae59c640 1 -- 172.21.15.27:0/1819058653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 msgr2=0x5573c38d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafae59c640 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38d6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafae59c640 1 -- 172.21.15.27:0/1819058653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c352a390 msgr2=0x5573c38dd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafae59c640 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c352a390 0x5573c38dd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafae59c640 1 -- 172.21.15.27:0/1819058653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573c35423f0 con 0x5573c35014d0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.791+0000 7fafaed9d640 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c352a390 0x5573c38dd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.792+0000 7fafae59c640 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35014d0 0x5573c38d6330 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5573c38cd860 tx=0x7fafa0011bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a42e5580ab282e3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.792+0000 7faf95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.792+0000 7faf95ffb640 1 -- 172.21.15.27:0/1819058653 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa000fde0 con 0x5573c35014d0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.792+0000 7fafbbdcc740 1 -- 172.21.15.27:0/1819058653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafa8036030 con 0x5573c35014d0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.792+0000 7faf95ffb640 1 -- 172.21.15.27:0/1819058653 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fafa0005020 con 0x5573c35014d0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.792+0000 7faf95ffb640 1 -- 172.21.15.27:0/1819058653 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fafa0005240 con 0x5573c35014d0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.792+0000 7fafbbdcc740 10 client.26822 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.792+0000 7fafbbdcc740 1 -- 172.21.15.27:0/1819058653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5573c360c7d0 con 0x5573c35014d0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7faf95ffb640 1 -- 172.21.15.27:0/1819058653 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafa8036030 con 0x5573c35014d0 2024-04-25T13:49:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7faf95ffb640 1 -- 172.21.15.27:0/1819058653 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fafa0018030 con 0x5573c35014d0 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7fafbbdcc740 10 client.26822 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7fafbbdcc740 10 client.26822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7fafbbdcc740 20 client.26822 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573c34e8f60 0x5573c38e45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7fafbbdcc740 4 client.26822 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7fafbbdcc740 1 -- 172.21.15.27:0/1819058653 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573c383e990 con 0x5573c34e8f60 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.793+0000 7fafadd9b640 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573c34e8f60 0x5573c38e45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.794+0000 7faf95ffb640 10 client.26822 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.794+0000 7fafadd9b640 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573c34e8f60 0x5573c38e45d0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30db511259e29c43 server_cookie=44575a28f464578d in_seq=0 out_seq=0 2024-04-25T13:49:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.794+0000 7faf95ffb640 1 -- 172.21.15.27:0/1819058653 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5573c383e990 con 0x5573c34e8f60 2024-04-25T13:49:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.794+0000 7faf95ffb640 10 client.26822 handle_command_reply: tid=0 2024-04-25T13:49:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.794+0000 7fafbbdcc740 1 client.26822 shutdown 2024-04-25T13:49:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 -- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35014d0 msgr2=0x5573c38d6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35014d0 0x5573c38d6330 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5573c38cd860 tx=0x7fafa0011bc0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 -- 172.21.15.27:0/1819058653 shutdown_connections 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573c34e8f60 0x5573c38e45d0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c352a390 0x5573c38dd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38d6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/1819058653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c35014d0 0x5573c38d6330 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 -- 172.21.15.27:0/1819058653 >> 172.21.15.27:0/1819058653 conn(0x5573c351e4e0 msgr2=0x5573c38c7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7faf95ffb640 0 client.26822 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 -- 172.21.15.27:0/1819058653 shutdown_connections 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 1 -- 172.21.15.27:0/1819058653 wait complete. 2024-04-25T13:49:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.795+0000 7fafbbdcc740 20 client.26822 trim_cache size 0 max 16384 2024-04-25T13:49:37.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.818+0000 7fafbbdcc740 1 Processor -- start 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafbbdcc740 1 -- start start 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38dd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38d4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0072200 con 0x5573c35014d0 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0074a80 con 0x5573c352a390 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0072870 con 0x5573c35096b0 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafae59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafae59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38c2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34696/0 (socket says 172.21.15.27:34696) 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafae59c640 1 -- 172.21.15.27:0/3846745042 learned_addr learned my addr 172.21.15.27:0/3846745042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafadd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38dd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafaed9d640 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38d4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafadd9b640 1 -- 172.21.15.27:0/3846745042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 msgr2=0x5573c38d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafadd9b640 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38d4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafadd9b640 1 -- 172.21.15.27:0/3846745042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 msgr2=0x5573c38c2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafadd9b640 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38c2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafadd9b640 1 -- 172.21.15.27:0/3846745042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573c38cc3d0 con 0x5573c35096b0 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.819+0000 7fafae59c640 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.820+0000 7fafadd9b640 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38dd6f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fafa400f730 tx=0x7fafa4005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e410677ad1b554fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.820+0000 7faf97fff640 1 -- 172.21.15.27:0/3846745042 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4005020 con 0x5573c35096b0 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.820+0000 7faf97fff640 1 -- 172.21.15.27:0/3846745042 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafa402a420 con 0x5573c35096b0 2024-04-25T13:49:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.820+0000 7faf97fff640 1 -- 172.21.15.27:0/3846745042 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4004a00 con 0x5573c35096b0 2024-04-25T13:49:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3846745042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 msgr2=0x5573c38dd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38dd6f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fafa400f730 tx=0x7fafa4005850 comp rx=0 tx=0).stop 2024-04-25T13:49:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3846745042 shutdown_connections 2024-04-25T13:49:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38d4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38dd6f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3846745042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3846745042 >> 172.21.15.27:0/3846745042 conn(0x5573c351e4e0 msgr2=0x5573c38cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3846745042 shutdown_connections 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3846745042 wait complete. 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 -- start start 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.821+0000 7fafbbdcc740 1 Processor -- start 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38dd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38d9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafbbdcc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38da350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0072f40 con 0x5573c35014d0 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafb00743b0 con 0x5573c352a390 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafbbdcc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fafb0073610 con 0x5573c35096b0 2024-04-25T13:49:37.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafbbdcc740 10 client.0 _ll_register_callbacks cb 0x7fafa000c690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafae59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38dd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafaed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38da350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafae59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38dd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34700/0 (socket says 172.21.15.27:34700) 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.822+0000 7fafae59c640 1 -- 172.21.15.27:0/3378677626 learned_addr learned my addr 172.21.15.27:0/3378677626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.823+0000 7fafae59c640 1 -- 172.21.15.27:0/3378677626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 msgr2=0x5573c38d9e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.823+0000 7fafae59c640 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38d9e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.823+0000 7fafae59c640 1 -- 172.21.15.27:0/3378677626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 msgr2=0x5573c38da350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.823+0000 7fafae59c640 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38da350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.823+0000 7fafae59c640 1 -- 172.21.15.27:0/3378677626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573c38ddc30 con 0x5573c35014d0 2024-04-25T13:49:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.823+0000 7fafaed9d640 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38da350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.824+0000 7fafae59c640 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38dd290 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fafa00194c0 tx=0x7fafa00055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73232b164738b65a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.824+0000 7faf95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:49:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.824+0000 7faf95ffb640 1 -- 172.21.15.27:0/3378677626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa000fdf0 con 0x5573c35014d0 2024-04-25T13:49:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.824+0000 7faf95ffb640 1 -- 172.21.15.27:0/3378677626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fafa0011040 con 0x5573c35014d0 2024-04-25T13:49:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.824+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3378677626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafa402a420 con 0x5573c35014d0 2024-04-25T13:49:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.824+0000 7faf95ffb640 1 -- 172.21.15.27:0/3378677626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fafa0011240 con 0x5573c35014d0 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.824+0000 7fafbbdcc740 10 client.17121 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.824+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3378677626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5573c38d1c40 con 0x5573c35014d0 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7faf95ffb640 1 -- 172.21.15.27:0/3378677626 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafa402a420 con 0x5573c35014d0 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7faf95ffb640 1 -- 172.21.15.27:0/3378677626 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fafa0018040 con 0x5573c35014d0 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7fafbbdcc740 10 client.17121 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7fafbbdcc740 10 client.17121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7fafbbdcc740 20 client.17121 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573c34be3a0 0x5573c394df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7fafbbdcc740 4 client.17121 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3378677626 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573c3838800 con 0x5573c34be3a0 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.825+0000 7fafadd9b640 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573c34be3a0 0x5573c394df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7faf95ffb640 10 client.17121 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafadd9b640 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573c34be3a0 0x5573c394df10 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40454c8b8a34cb59 server_cookie=f8747a206c2dbfe2 in_seq=0 out_seq=0 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7faf95ffb640 1 -- 172.21.15.27:0/3378677626 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573c3838800 con 0x5573c34be3a0 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7faf95ffb640 10 client.17121 handle_command_reply: tid=0 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 client.17121 shutdown 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3378677626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 msgr2=0x5573c38dd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38dd290 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fafa00194c0 tx=0x7fafa00055f0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3378677626 shutdown_connections 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573c34be3a0 0x5573c394df10 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573c352a390 0x5573c38da350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7faf95ffb640 0 client.17121 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573c35096b0 0x5573c38d9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 --2- 172.21.15.27:0/3378677626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573c35014d0 0x5573c38dd290 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3378677626 >> 172.21.15.27:0/3378677626 conn(0x5573c351e4e0 msgr2=0x5573c38c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3378677626 shutdown_connections 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.826+0000 7fafbbdcc740 1 -- 172.21.15.27:0/3378677626 wait complete. 2024-04-25T13:49:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafbbdcc740 20 client.17121 trim_cache size 0 max 16384 2024-04-25T13:49:37.831 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 -- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faf980afc80 msgr2=0x7faf980b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faf980afc80 0x7faf980b2190 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fafa4005d00 tx=0x7fafa400d680 comp rx=0 tx=0).stop 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 -- 172.21.15.27:0/600308417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb0107680 msgr2=0x7fafb010f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb0107680 0x7fafb010f270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fafa8004af0 tx=0x7fafa8003ba0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 -- 172.21.15.27:0/600308417 shutdown_connections 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faf980afc80 0x7faf980b2190 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fafb0107680 0x7fafb010f270 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafb0104d20 0x7fafb010e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 --2- 172.21.15.27:0/600308417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafb01023c0 0x7fafb010ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 -- 172.21.15.27:0/600308417 >> 172.21.15.27:0/600308417 conn(0x7fafb00fdbd0 msgr2=0x7fafb0103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 -- 172.21.15.27:0/600308417 shutdown_connections 2024-04-25T13:49:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:37.827+0000 7fafb4f1c640 1 -- 172.21.15.27:0/600308417 wait complete. 2024-04-25T13:49:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 510.0 s, 2122.2 tps, lat 47.141 ms stddev 50.096 2024-04-25T13:49:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:38 smithi027 ceph-mon[29140]: pgmap v611: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 666 op/s 2024-04-25T13:49:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:38 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1025784556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:49:38.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:38 smithi122 ceph-mon[32843]: pgmap v611: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 666 op/s 2024-04-25T13:49:38.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:38 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1025784556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:49:38.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:38 smithi045 ceph-mon[32731]: pgmap v611: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 666 op/s 2024-04-25T13:49:38.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:38 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1025784556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:49:38.841 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:49:38.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:49:38.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:49:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.038+0000 7f6b6df9d640 1 -- 172.21.15.27:0/4151047312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6806f0b0 msgr2=0x7f6b6806fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.038+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/4151047312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6806f0b0 0x7f6b6806fe00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009900 tx=0x7f6b5002a420 comp rx=0 tx=0).stop 2024-04-25T13:49:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.039+0000 7f6b6df9d640 1 -- 172.21.15.27:0/4151047312 shutdown_connections 2024-04-25T13:49:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.039+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/4151047312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68071040 0x7f6b68071450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.039+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/4151047312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b68070340 0x7f6b68070750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.039+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/4151047312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6806f0b0 0x7f6b6806fe00 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.039+0000 7f6b6df9d640 1 -- 172.21.15.27:0/4151047312 >> 172.21.15.27:0/4151047312 conn(0x7f6b6806e0d0 msgr2=0x7f6b6810bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.044+0000 7f6b6df9d640 1 -- 172.21.15.27:0/4151047312 shutdown_connections 2024-04-25T13:49:39.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.045+0000 7f6b6df9d640 1 -- 172.21.15.27:0/4151047312 wait complete. 2024-04-25T13:49:39.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.046+0000 7f6b6df9d640 1 Processor -- start 2024-04-25T13:49:39.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.046+0000 7f6b6df9d640 1 -- start start 2024-04-25T13:49:39.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.046+0000 7f6b6df9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b6806f0b0 0x7f6b6811b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.046+0000 7f6b6df9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68070340 0x7f6b68119e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.046+0000 7f6b6df9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b68071040 0x7f6b6811a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.046+0000 7f6b6df9d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6810e1a0 con 0x7f6b68071040 2024-04-25T13:49:39.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.046+0000 7f6b6df9d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6810e020 con 0x7f6b68070340 2024-04-25T13:49:39.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.046+0000 7f6b6df9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6810e320 con 0x7f6b6806f0b0 2024-04-25T13:49:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.047+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b6806f0b0 0x7f6b6811b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.047+0000 7f6b67fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68070340 0x7f6b68119e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.047+0000 7f6b67fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68070340 0x7f6b68119e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:44234/0 (socket says 172.21.15.27:44234) 2024-04-25T13:49:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.047+0000 7f6b67fff640 1 -- 172.21.15.27:0/1209726398 learned_addr learned my addr 172.21.15.27:0/1209726398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.047+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b68071040 0x7f6b6811a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.047+0000 7f6b67fff640 1 -- 172.21.15.27:0/1209726398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b6806f0b0 msgr2=0x7f6b6811b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.047+0000 7f6b67fff640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b6806f0b0 0x7f6b6811b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.047+0000 7f6b67fff640 1 -- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b68071040 msgr2=0x7f6b6811a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.048+0000 7f6b67fff640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b68071040 0x7f6b6811a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.048+0000 7f6b67fff640 1 -- 172.21.15.27:0/1209726398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6811a8a0 con 0x7f6b68070340 2024-04-25T13:49:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.048+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b68071040 0x7f6b6811a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:49:39.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.048+0000 7f6b67fff640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68070340 0x7f6b68119e20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000eab0 tx=0x7f6b6000ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.048+0000 7f6b65ffb640 1 -- 172.21.15.27:0/1209726398 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6000ce40 con 0x7f6b68070340 2024-04-25T13:49:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.048+0000 7f6b65ffb640 1 -- 172.21.15.27:0/1209726398 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b60011920 con 0x7f6b68070340 2024-04-25T13:49:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.048+0000 7f6b65ffb640 1 -- 172.21.15.27:0/1209726398 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6000ce40 con 0x7f6b68070340 2024-04-25T13:49:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.049+0000 7f6b6df9d640 1 -- 172.21.15.27:0/1209726398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6811ab90 con 0x7f6b68070340 2024-04-25T13:49:39.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.049+0000 7f6b6df9d640 1 -- 172.21.15.27:0/1209726398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b681b9970 con 0x7f6b68070340 2024-04-25T13:49:39.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.053+0000 7f6b76186740 1 Processor -- start 2024-04-25T13:49:39.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.053+0000 7f6b76186740 1 -- start start 2024-04-25T13:49:39.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.053+0000 7f6b76186740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46ea410 0x55f8b4a8b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.053+0000 7f6b76186740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a8bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b65ffb640 1 -- 172.21.15.27:0/1209726398 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6b60011470 con 0x7f6b68070340 2024-04-25T13:49:39.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b76186740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c9730 0x55f8b4a92ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b76186740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f8b49f0b00 con 0x55f8b46ea410 2024-04-25T13:49:39.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b76186740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f8b498c920 con 0x55f8b46c9730 2024-04-25T13:49:39.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b76186740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f8b49ecac0 con 0x55f8b46c1550 2024-04-25T13:49:39.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a8bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b67fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c9730 0x55f8b4a92ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a8bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39390/0 (socket says 172.21.15.27:39390) 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/2512813063 learned_addr learned my addr 172.21.15.27:0/2512813063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46ea410 0x55f8b4a8b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/2512813063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c9730 msgr2=0x55f8b4a92ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c9730 0x55f8b4a92ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/2512813063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46ea410 msgr2=0x55f8b4a8b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46ea410 0x55f8b4a8b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/2512813063 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8b47ca7e0 con 0x55f8b46c1550 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.054+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46ea410 0x55f8b4a8b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.055+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a8bbe0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6b50000c00 tx=0x7f6b50005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4a90fdbbbfff6b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.055+0000 7f6b5effd640 1 -- 172.21.15.27:0/2512813063 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50035040 con 0x55f8b46c1550 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.055+0000 7f6b5effd640 1 -- 172.21.15.27:0/2512813063 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b500054f0 con 0x55f8b46c1550 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.055+0000 7f6b5effd640 1 -- 172.21.15.27:0/2512813063 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50003850 con 0x55f8b46c1550 2024-04-25T13:49:39.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.055+0000 7f6b65ffb640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b540aff30 0x7f6b540b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.055+0000 7f6b65ffb640 1 -- 172.21.15.27:0/1209726398 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6b6000cc40 con 0x7f6b68070340 2024-04-25T13:49:39.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b540aff30 0x7f6b540b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 -- 172.21.15.27:0/2512813063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 msgr2=0x55f8b4a8bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b540aff30 0x7f6b540b2440 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55f8b4a8c830 tx=0x7f6b58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a8bbe0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6b50000c00 tx=0x7f6b50005f30 comp rx=0 tx=0).stop 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 -- 172.21.15.27:0/2512813063 shutdown_connections 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c9730 0x55f8b4a92ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a8bbe0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 --2- 172.21.15.27:0/2512813063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46ea410 0x55f8b4a8b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 -- 172.21.15.27:0/2512813063 >> 172.21.15.27:0/2512813063 conn(0x55f8b46de560 msgr2=0x55f8b4a86ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 -- 172.21.15.27:0/2512813063 shutdown_connections 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.056+0000 7f6b76186740 1 -- 172.21.15.27:0/2512813063 wait complete. 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.057+0000 7f6b76186740 1 -- start start 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.057+0000 7f6b76186740 1 Processor -- start 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.057+0000 7f6b76186740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a95450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b76186740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c9730 0x55f8b4a95990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a95450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a95450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39404/0 (socket says 172.21.15.27:39404) 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b76186740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46ea410 0x55f8b4a9c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/3806081441 learned_addr learned my addr 172.21.15.27:0/3806081441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f8b4a93520 con 0x55f8b46c9730 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b67fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c9730 0x55f8b4a95990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f8b498a5d0 con 0x55f8b46ea410 2024-04-25T13:49:39.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f8b4a936a0 con 0x55f8b46c1550 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b76186740 10 client.0 _ll_register_callbacks cb 0x55f8b4978080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.058+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46ea410 0x55f8b4a9c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.059+0000 7f6b6d79c640 1 -- 172.21.15.27:0/3806081441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 msgr2=0x55f8b4a95450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.059+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a95450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.059+0000 7f6b6d79c640 1 -- 172.21.15.27:0/3806081441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c9730 msgr2=0x55f8b4a95990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.059+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c9730 0x55f8b4a95990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.059+0000 7f6b6d79c640 1 -- 172.21.15.27:0/3806081441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8b482af70 con 0x55f8b46ea410 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b67fff640 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c9730 0x55f8b4a95990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46ea410 0x55f8b4a9c8f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b58012c10 tx=0x7f6b580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c0552d7b2f093f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b47fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b47fff640 1 -- 172.21.15.27:0/3806081441 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58023070 con 0x55f8b46ea410 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b47fff640 1 -- 172.21.15.27:0/3806081441 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6b58012c40 con 0x55f8b46ea410 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b500054f0 con 0x55f8b46ea410 2024-04-25T13:49:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b47fff640 1 -- 172.21.15.27:0/3806081441 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6b5802d050 con 0x55f8b46ea410 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b76186740 10 client.26840 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.060+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f8b46c2170 con 0x55f8b46ea410 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b47fff640 1 -- 172.21.15.27:0/3806081441 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b500054f0 con 0x55f8b46ea410 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b47fff640 1 -- 172.21.15.27:0/3806081441 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6b58003a00 con 0x55f8b46ea410 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b76186740 10 client.26840 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b76186740 10 client.26840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b76186740 20 client.26840 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b76186740 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8b46a91f0 0x55f8b4aa3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b76186740 4 client.26840 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8b49fe710 con 0x55f8b46a91f0 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.061+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8b46a91f0 0x55f8b4aa3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.062+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8b46a91f0 0x55f8b4aa3300 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d0a5821b0194c75 server_cookie=a9b2f3d720843c68 in_seq=0 out_seq=0 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.062+0000 7f6b47fff640 10 client.26840 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:39.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.062+0000 7f6b47fff640 1 -- 172.21.15.27:0/3806081441 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f8b49fe710 con 0x55f8b46a91f0 2024-04-25T13:49:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.062+0000 7f6b47fff640 10 client.26840 handle_command_reply: tid=0 2024-04-25T13:49:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.062+0000 7f6b76186740 1 client.26840 shutdown 2024-04-25T13:49:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46ea410 msgr2=0x55f8b4a9c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46ea410 0x55f8b4a9c8f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b58012c10 tx=0x7f6b580046c0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 shutdown_connections 2024-04-25T13:49:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8b46a91f0 0x55f8b4aa3300 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46ea410 0x55f8b4a9c8f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c9730 0x55f8b4a95990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b47fff640 0 client.26840 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:39.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 --2- 172.21.15.27:0/3806081441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c1550 0x55f8b4a95450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 >> 172.21.15.27:0/3806081441 conn(0x55f8b46de560 msgr2=0x55f8b4a85f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 shutdown_connections 2024-04-25T13:49:39.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.064+0000 7f6b76186740 1 -- 172.21.15.27:0/3806081441 wait complete. 2024-04-25T13:49:39.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.065+0000 7f6b76186740 20 client.26840 trim_cache size 0 max 16384 2024-04-25T13:49:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.088+0000 7f6b76186740 1 Processor -- start 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.088+0000 7f6b76186740 1 -- start start 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.088+0000 7f6b76186740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46de560 0x55f8b4a868e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.088+0000 7f6b76186740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c1550 0x55f8b4a86e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b76186740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 0x55f8b4a9c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46de560 0x55f8b4a868e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46de560 0x55f8b4a868e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:44270/0 (socket says 172.21.15.27:44270) 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/559474003 learned_addr learned my addr 172.21.15.27:0/559474003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b76186740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681c05b0 con 0x55f8b46c1550 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b76186740 1 -- 172.21.15.27:0/559474003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681c2e30 con 0x55f8b46de560 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b76186740 1 -- 172.21.15.27:0/559474003 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681c0c20 con 0x55f8b46c9730 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 0x55f8b4a9c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b67fff640 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c1550 0x55f8b4a86e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/559474003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 msgr2=0x55f8b4a9c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 0x55f8b4a9c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/559474003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c1550 msgr2=0x55f8b4a86e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.089+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c1550 0x55f8b4a86e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.090+0000 7f6b6cf9b640 1 -- 172.21.15.27:0/559474003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8b4a902b0 con 0x55f8b46de560 2024-04-25T13:49:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.090+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46de560 0x55f8b4a868e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6b50006010 tx=0x7f6b5002a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c6878790bda12f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.090+0000 7f6b5e7fc640 1 -- 172.21.15.27:0/559474003 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50035070 con 0x55f8b46de560 2024-04-25T13:49:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.090+0000 7f6b5e7fc640 1 -- 172.21.15.27:0/559474003 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b500518e0 con 0x55f8b46de560 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b5e7fc640 1 -- 172.21.15.27:0/559474003 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50003dc0 con 0x55f8b46de560 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 -- 172.21.15.27:0/559474003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46de560 msgr2=0x55f8b4a868e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46de560 0x55f8b4a868e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6b50006010 tx=0x7f6b5002a420 comp rx=0 tx=0).stop 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 -- 172.21.15.27:0/559474003 shutdown_connections 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 0x55f8b4a9c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46c1550 0x55f8b4a86e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 --2- 172.21.15.27:0/559474003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46de560 0x55f8b4a868e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 -- 172.21.15.27:0/559474003 >> 172.21.15.27:0/559474003 conn(0x55f8b4a95c20 msgr2=0x55f8b4a93aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 -- 172.21.15.27:0/559474003 shutdown_connections 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 -- 172.21.15.27:0/559474003 wait complete. 2024-04-25T13:49:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.091+0000 7f6b76186740 1 -- start start 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b76186740 1 Processor -- start 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b76186740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c1550 0x55f8b4a9a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b76186740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 0x55f8b4a9a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b76186740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46de560 0x55f8b4a9b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b76186740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681c12f0 con 0x55f8b46de560 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b76186740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681c2760 con 0x55f8b46c1550 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b76186740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6b681c19c0 con 0x55f8b46c9730 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b76186740 10 client.0 _ll_register_callbacks cb 0x7f6b580128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6d79c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46de560 0x55f8b4a9b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c1550 0x55f8b4a9a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6d79c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46de560 0x55f8b4a9b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34762/0 (socket says 172.21.15.27:34762) 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6cf9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c1550 0x55f8b4a9a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:44274/0 (socket says 172.21.15.27:44274) 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6d79c640 1 -- 172.21.15.27:0/1638375528 learned_addr learned my addr 172.21.15.27:0/1638375528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b67fff640 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 0x55f8b4a9a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6d79c640 1 -- 172.21.15.27:0/1638375528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 msgr2=0x55f8b4a9a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 0x55f8b4a9a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6d79c640 1 -- 172.21.15.27:0/1638375528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c1550 msgr2=0x55f8b4a9a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c1550 0x55f8b4a9a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.092+0000 7f6b6d79c640 1 -- 172.21.15.27:0/1638375528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8b4a9bd80 con 0x55f8b46de560 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b6d79c640 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46de560 0x55f8b4a9b7c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6b58011c40 tx=0x7f6b58031f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bdffb948c7d0ada server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b47fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b47fff640 1 -- 172.21.15.27:0/1638375528 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58023040 con 0x55f8b46de560 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b76186740 1 -- 172.21.15.27:0/1638375528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b500518e0 con 0x55f8b46de560 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b47fff640 1 -- 172.21.15.27:0/1638375528 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6b5801be30 con 0x55f8b46de560 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b47fff640 1 -- 172.21.15.27:0/1638375528 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6b58012ba0 con 0x55f8b46de560 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b76186740 10 client.17133 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b76186740 1 -- 172.21.15.27:0/1638375528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f8b4a9d0b0 con 0x55f8b46de560 2024-04-25T13:49:39.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b47fff640 1 -- 172.21.15.27:0/1638375528 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b500518e0 con 0x55f8b46de560 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b47fff640 1 -- 172.21.15.27:0/1638375528 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6b58003a00 con 0x55f8b46de560 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b76186740 10 client.17133 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b76186740 10 client.17133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.093+0000 7f6b76186740 20 client.17133 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.094+0000 7f6b76186740 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8b46ea410 0x55f8b4b0db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.094+0000 7f6b76186740 4 client.17133 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.094+0000 7f6b76186740 1 -- 172.21.15.27:0/1638375528 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f8b49f8580 con 0x55f8b46ea410 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.094+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8b46ea410 0x55f8b4b0db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.094+0000 7f6b47fff640 10 client.17133 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.094+0000 7f6b6cf9b640 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8b46ea410 0x55f8b4b0db40 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfd3cfec56a2f1d2 server_cookie=c64e928287b27d20 in_seq=0 out_seq=0 2024-04-25T13:49:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b47fff640 1 -- 172.21.15.27:0/1638375528 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f8b49f8580 con 0x55f8b46ea410 2024-04-25T13:49:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b47fff640 10 client.17133 handle_command_reply: tid=0 2024-04-25T13:49:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 client.17133 shutdown 2024-04-25T13:49:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 -- 172.21.15.27:0/1638375528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46de560 msgr2=0x55f8b4a9b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46de560 0x55f8b4a9b7c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6b58011c40 tx=0x7f6b58031f70 comp rx=0 tx=0).stop 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 -- 172.21.15.27:0/1638375528 shutdown_connections 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8b46ea410 0x55f8b4b0db40 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8b46de560 0x55f8b4a9b7c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8b46c9730 0x55f8b4a9a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 --2- 172.21.15.27:0/1638375528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8b46c1550 0x55f8b4a9a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.096+0000 7f6b76186740 1 -- 172.21.15.27:0/1638375528 >> 172.21.15.27:0/1638375528 conn(0x55f8b4a95c20 msgr2=0x55f8b4a868e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.097+0000 7f6b47fff640 0 client.17133 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.097+0000 7f6b76186740 1 -- 172.21.15.27:0/1638375528 shutdown_connections 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.097+0000 7f6b76186740 1 -- 172.21.15.27:0/1638375528 wait complete. 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.097+0000 7f6b76186740 20 client.17133 trim_cache size 0 max 16384 2024-04-25T13:49:39.100 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:49:39.101 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:49:39.101 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "807d4185-5141-4cc0-a290-da7cc01ac56d", 2024-04-25T13:49:39.101 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:49:39.101 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:49:39.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 -- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b540aff30 msgr2=0x7f6b540b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b540aff30 0x7f6b540b2440 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55f8b4a8c830 tx=0x7f6b58005f90 comp rx=0 tx=0).stop 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 -- 172.21.15.27:0/1209726398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68070340 msgr2=0x7f6b68119e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68070340 0x7f6b68119e20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000eab0 tx=0x7f6b6000ef70 comp rx=0 tx=0).stop 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 -- 172.21.15.27:0/1209726398 shutdown_connections 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6b540aff30 0x7f6b540b2440 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b68071040 0x7f6b6811a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b68070340 0x7f6b68119e20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 --2- 172.21.15.27:0/1209726398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b6806f0b0 0x7f6b6811b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 -- 172.21.15.27:0/1209726398 >> 172.21.15.27:0/1209726398 conn(0x7f6b6806e0d0 msgr2=0x7f6b68110b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 -- 172.21.15.27:0/1209726398 shutdown_connections 2024-04-25T13:49:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.098+0000 7f6b6df9d640 1 -- 172.21.15.27:0/1209726398 wait complete. 2024-04-25T13:49:39.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:49:39.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:49:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.301+0000 7fe1bbfff640 1 -- 172.21.15.27:0/453750519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 msgr2=0x7fe1bc101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.301+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/453750519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 0x7fe1bc101b60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0009900 tx=0x7fe1b002a410 comp rx=0 tx=0).stop 2024-04-25T13:49:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.302+0000 7fe1bbfff640 1 -- 172.21.15.27:0/453750519 shutdown_connections 2024-04-25T13:49:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.302+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/453750519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1bc1031c0 0x7fe1bc103660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.302+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/453750519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1bc102130 0x7fe1bc102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.302+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/453750519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 0x7fe1bc101b60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.302+0000 7fe1bbfff640 1 -- 172.21.15.27:0/453750519 >> 172.21.15.27:0/453750519 conn(0x7fe1bc0fd170 msgr2=0x7fe1bc0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.304+0000 7fe1bbfff640 1 -- 172.21.15.27:0/453750519 shutdown_connections 2024-04-25T13:49:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.304+0000 7fe1bbfff640 1 -- 172.21.15.27:0/453750519 wait complete. 2024-04-25T13:49:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.306+0000 7fe1bbfff640 1 Processor -- start 2024-04-25T13:49:39.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.307+0000 7fe1bbfff640 1 -- start start 2024-04-25T13:49:39.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.307+0000 7fe1bbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 0x7fe1bc19c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.308+0000 7fe1bbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1bc102130 0x7fe1bc19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.308+0000 7fe1baffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 0x7fe1bc19c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.308+0000 7fe1baffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 0x7fe1bc19c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39442/0 (socket says 172.21.15.27:39442) 2024-04-25T13:49:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.308+0000 7fe1baffd640 1 -- 172.21.15.27:0/750659845 learned_addr learned my addr 172.21.15.27:0/750659845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.308+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1bc102130 0x7fe1bc19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.308+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1bc1031c0 0x7fe1bc1a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.308+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc111780 con 0x7fe1bc102130 2024-04-25T13:49:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.309+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc111600 con 0x7fe1bc1031c0 2024-04-25T13:49:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.309+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc111900 con 0x7fe1bc069a20 2024-04-25T13:49:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.309+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1bc1031c0 0x7fe1bc1a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.309+0000 7fe1ba7fc640 1 -- 172.21.15.27:0/750659845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 msgr2=0x7fe1bc19c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.309+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 0x7fe1bc19c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.309+0000 7fe1ba7fc640 1 -- 172.21.15.27:0/750659845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1bc1031c0 msgr2=0x7fe1bc1a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.309+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1bc1031c0 0x7fe1bc1a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.309+0000 7fe1ba7fc640 1 -- 172.21.15.27:0/750659845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc1a3d50 con 0x7fe1bc102130 2024-04-25T13:49:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.310+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1bc102130 0x7fe1bc19c750 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00e9f0 tx=0x7fe1ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.310+0000 7fe1a3fff640 1 -- 172.21.15.27:0/750659845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac00cd80 con 0x7fe1bc102130 2024-04-25T13:49:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.310+0000 7fe1a3fff640 1 -- 172.21.15.27:0/750659845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1ac0035f0 con 0x7fe1bc102130 2024-04-25T13:49:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.310+0000 7fe1a3fff640 1 -- 172.21.15.27:0/750659845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac010670 con 0x7fe1bc102130 2024-04-25T13:49:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.310+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1bc1a3fe0 con 0x7fe1bc102130 2024-04-25T13:49:39.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.310+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1bc1a43f0 con 0x7fe1bc102130 2024-04-25T13:49:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.315+0000 7fe1c842a740 1 Processor -- start 2024-04-25T13:49:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.315+0000 7fe1c842a740 1 -- start start 2024-04-25T13:49:39.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.316+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8520c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.316+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe85211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.316+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8521e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.316+0000 7fe1c842a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557fe8485d80 con 0x557fe817f390 2024-04-25T13:49:39.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.316+0000 7fe1c842a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557fe843f8b0 con 0x557fe81564d0 2024-04-25T13:49:39.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.316+0000 7fe1c842a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557fe8481d40 con 0x557fe815e6b0 2024-04-25T13:49:39.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.316+0000 7fe1a3fff640 1 -- 172.21.15.27:0/750659845 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe1ac003ac0 con 0x7fe1bc102130 2024-04-25T13:49:39.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.317+0000 7fe1ba7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8521e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.317+0000 7fe1ba7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8521e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39454/0 (socket says 172.21.15.27:39454) 2024-04-25T13:49:39.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.317+0000 7fe1ba7fc640 1 -- 172.21.15.27:0/1360796175 learned_addr learned my addr 172.21.15.27:0/1360796175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.317+0000 7fe1a3fff640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe1a40afd00 0x7fe1a40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.318+0000 7fe1a3fff640 1 -- 172.21.15.27:0/750659845 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe1ac0decb0 con 0x7fe1bc102130 2024-04-25T13:49:39.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.318+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe1a40afd00 0x7fe1a40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.319+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8520c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.319+0000 7fe1baffd640 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe85211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.319+0000 7fe1bb7fe640 1 -- 172.21.15.27:0/1360796175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 msgr2=0x557fe8521e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.319+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8521e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.319+0000 7fe1bb7fe640 1 -- 172.21.15.27:0/1360796175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 msgr2=0x557fe85211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.319+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe85211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.319+0000 7fe1bb7fe640 1 -- 172.21.15.27:0/1360796175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fe8410a40 con 0x557fe817f390 2024-04-25T13:49:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.320+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8520c70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4005fd0 tx=0x7fe1b4012d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c0fd5bf770ad1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.320+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe1a40afd00 0x7fe1a40b2210 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4010740 tx=0x7fe1b4014040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.321+0000 7fe1a17fa640 1 -- 172.21.15.27:0/1360796175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b400e7b0 con 0x557fe817f390 2024-04-25T13:49:39.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.321+0000 7fe1a17fa640 1 -- 172.21.15.27:0/1360796175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b400ee00 con 0x557fe817f390 2024-04-25T13:49:39.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1a17fa640 1 -- 172.21.15.27:0/1360796175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b401fe40 con 0x557fe817f390 2024-04-25T13:49:39.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 -- 172.21.15.27:0/1360796175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 msgr2=0x557fe8520c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8520c70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4005fd0 tx=0x7fe1b4012d20 comp rx=0 tx=0).stop 2024-04-25T13:49:39.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 -- 172.21.15.27:0/1360796175 shutdown_connections 2024-04-25T13:49:39.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8521e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe85211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 --2- 172.21.15.27:0/1360796175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8520c70 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 -- 172.21.15.27:0/1360796175 >> 172.21.15.27:0/1360796175 conn(0x557fe81734e0 msgr2=0x557fe851c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 -- 172.21.15.27:0/1360796175 shutdown_connections 2024-04-25T13:49:39.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.322+0000 7fe1c842a740 1 -- 172.21.15.27:0/1360796175 wait complete. 2024-04-25T13:49:39.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.323+0000 7fe1c842a740 1 -- start start 2024-04-25T13:49:39.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.323+0000 7fe1c842a740 1 Processor -- start 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.323+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe81564d0 0x557fe8533310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.323+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 0x557fe8533850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.323+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe817f390 0x557fe853a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.324+0000 7fe1baffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe81564d0 0x557fe8533310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.324+0000 7fe1baffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe81564d0 0x557fe8533310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34808/0 (socket says 172.21.15.27:34808) 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.324+0000 7fe1baffd640 1 -- 172.21.15.27:0/4168234144 learned_addr learned my addr 172.21.15.27:0/4168234144 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.324+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557fe8522980 con 0x557fe81564d0 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.324+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557fe840cd10 con 0x557fe815e6b0 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.324+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557fe8522b00 con 0x557fe817f390 2024-04-25T13:49:39.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.324+0000 7fe1c842a740 10 client.0 _ll_register_callbacks cb 0x557fe8439210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:39.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.324+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe817f390 0x557fe853a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.325+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 0x557fe8533850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.325+0000 7fe1bb7fe640 1 -- 172.21.15.27:0/4168234144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 msgr2=0x557fe8533850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.325+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 0x557fe8533850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.325+0000 7fe1bb7fe640 1 -- 172.21.15.27:0/4168234144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe81564d0 msgr2=0x557fe8533310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.325+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe81564d0 0x557fe8533310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.325+0000 7fe1bb7fe640 1 -- 172.21.15.27:0/4168234144 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fe82617d0 con 0x557fe817f390 2024-04-25T13:49:39.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.325+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe817f390 0x557fe853a7b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x557fe8521cc0 tx=0x7fe1b4010fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9112b10b8002702d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.326+0000 7fe19eff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.326+0000 7fe19eff5640 1 -- 172.21.15.27:0/4168234144 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b403e070 con 0x557fe817f390 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.326+0000 7fe19eff5640 1 -- 172.21.15.27:0/4168234144 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe1b4009600 con 0x557fe817f390 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.326+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b400ee00 con 0x557fe817f390 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe19eff5640 1 -- 172.21.15.27:0/4168234144 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe1b4004c10 con 0x557fe817f390 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe1c842a740 10 client.26914 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557fe81973f0 con 0x557fe817f390 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe19eff5640 1 -- 172.21.15.27:0/4168234144 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b400ee00 con 0x557fe817f390 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe19eff5640 1 -- 172.21.15.27:0/4168234144 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe1b400f040 con 0x557fe817f390 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe1c842a740 10 client.26914 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe1c842a740 10 client.26914 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe1c842a740 20 client.26914 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:39.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe1c842a740 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fe813df60 0x557fe8530b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe1c842a740 4 client.26914 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:39.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.327+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fe8493990 con 0x557fe813df60 2024-04-25T13:49:39.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.328+0000 7fe1baffd640 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fe813df60 0x557fe8530b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.328+0000 7fe19eff5640 10 client.26914 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:39.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.329+0000 7fe1baffd640 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fe813df60 0x557fe8530b30 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e99a0eed5c4294f server_cookie=7c13ce1200305a5 in_seq=0 out_seq=0 2024-04-25T13:49:39.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.330+0000 7fe19eff5640 1 -- 172.21.15.27:0/4168234144 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557fe8493990 con 0x557fe813df60 2024-04-25T13:49:39.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.330+0000 7fe19eff5640 10 client.26914 handle_command_reply: tid=0 2024-04-25T13:49:39.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.330+0000 7fe1c842a740 1 client.26914 shutdown 2024-04-25T13:49:39.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe817f390 msgr2=0x557fe853a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe817f390 0x557fe853a7b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x557fe8521cc0 tx=0x7fe1b4010fd0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 shutdown_connections 2024-04-25T13:49:39.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fe813df60 0x557fe8530b30 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe817f390 0x557fe853a7b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 0x557fe8533850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 --2- 172.21.15.27:0/4168234144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe81564d0 0x557fe8533310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 >> 172.21.15.27:0/4168234144 conn(0x557fe81734e0 msgr2=0x557fe851f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe19eff5640 0 client.26914 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:39.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 shutdown_connections 2024-04-25T13:49:39.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 1 -- 172.21.15.27:0/4168234144 wait complete. 2024-04-25T13:49:39.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.332+0000 7fe1c842a740 20 client.26914 trim_cache size 0 max 16384 2024-04-25T13:49:39.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.358+0000 7fe1c842a740 1 Processor -- start 2024-04-25T13:49:39.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.358+0000 7fe1c842a740 1 -- start start 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe81564d0 0x557fe851beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 0x557fe851c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe852a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1c842a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a800c460 con 0x557fe817f390 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1baffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe81564d0 0x557fe851beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1baffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe81564d0 0x557fe851beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39462/0 (socket says 172.21.15.27:39462) 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1baffd640 1 -- 172.21.15.27:0/3925227718 learned_addr learned my addr 172.21.15.27:0/3925227718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 0x557fe851c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe852a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.359+0000 7fe1c842a740 1 -- 172.21.15.27:0/3925227718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a800ece0 con 0x557fe815e6b0 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1c842a740 1 -- 172.21.15.27:0/3925227718 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a800cad0 con 0x557fe81564d0 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1baffd640 1 -- 172.21.15.27:0/3925227718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 msgr2=0x557fe851c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1baffd640 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 0x557fe851c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1baffd640 1 -- 172.21.15.27:0/3925227718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 msgr2=0x557fe852a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1baffd640 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe852a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1baffd640 1 -- 172.21.15.27:0/3925227718 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fe81570f0 con 0x557fe81564d0 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1baffd640 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe81564d0 0x557fe851beb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe1b002aa00 tx=0x7fe1b0004560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4607053ad5b5c0d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1a0ff9640 1 -- 172.21.15.27:0/3925227718 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b005d070 con 0x557fe81564d0 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1a0ff9640 1 -- 172.21.15.27:0/3925227718 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b00579f0 con 0x557fe81564d0 2024-04-25T13:49:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.360+0000 7fe1a0ff9640 1 -- 172.21.15.27:0/3925227718 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0035040 con 0x557fe81564d0 2024-04-25T13:49:39.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 -- 172.21.15.27:0/3925227718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe81564d0 msgr2=0x557fe851beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe81564d0 0x557fe851beb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe1b002aa00 tx=0x7fe1b0004560 comp rx=0 tx=0).stop 2024-04-25T13:49:39.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 -- 172.21.15.27:0/3925227718 shutdown_connections 2024-04-25T13:49:39.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe852a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe815e6b0 0x557fe851c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 --2- 172.21.15.27:0/3925227718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe81564d0 0x557fe851beb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 -- 172.21.15.27:0/3925227718 >> 172.21.15.27:0/3925227718 conn(0x557fe81734e0 msgr2=0x557fe8525810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 -- 172.21.15.27:0/3925227718 shutdown_connections 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.361+0000 7fe1c842a740 1 -- 172.21.15.27:0/3925227718 wait complete. 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 1 -- start start 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 1 Processor -- start 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe8539a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8536590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8536ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1baffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe8539a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1baffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe8539a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:44324/0 (socket says 172.21.15.27:44324) 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1baffd640 1 -- 172.21.15.27:0/2006246878 learned_addr learned my addr 172.21.15.27:0/2006246878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a800d1a0 con 0x557fe817f390 2024-04-25T13:49:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a800e610 con 0x557fe81564d0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a800d870 con 0x557fe815e6b0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1c842a740 10 client.0 _ll_register_callbacks cb 0x7fe1b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8536590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.362+0000 7fe1bb7fe640 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8536ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1baffd640 1 -- 172.21.15.27:0/2006246878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 msgr2=0x557fe8536590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1baffd640 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8536590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1baffd640 1 -- 172.21.15.27:0/2006246878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 msgr2=0x557fe8536ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1baffd640 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8536ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1baffd640 1 -- 172.21.15.27:0/2006246878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fe84464b0 con 0x557fe81564d0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1baffd640 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe8539a10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0031440 tx=0x7fe1b00045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bced811825c19cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe19eff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe19eff5640 1 -- 172.21.15.27:0/2006246878 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0066060 con 0x557fe81564d0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe19eff5640 1 -- 172.21.15.27:0/2006246878 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe1b00037f0 con 0x557fe81564d0 2024-04-25T13:49:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b00579f0 con 0x557fe81564d0 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe19eff5640 1 -- 172.21.15.27:0/2006246878 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe1b005d040 con 0x557fe81564d0 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1c842a740 10 client.26864 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.363+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557fe859f600 con 0x557fe81564d0 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.364+0000 7fe19eff5640 1 -- 172.21.15.27:0/2006246878 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1b00579f0 con 0x557fe81564d0 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.364+0000 7fe19eff5640 1 -- 172.21.15.27:0/2006246878 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe1b00318a0 con 0x557fe81564d0 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.364+0000 7fe1c842a740 10 client.26864 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.364+0000 7fe1c842a740 10 client.26864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.364+0000 7fe1c842a740 20 client.26864 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.364+0000 7fe1c842a740 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fe81133a0 0x557fe85a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.365+0000 7fe1c842a740 4 client.26864 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:49:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.365+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557fe848d800 con 0x557fe81133a0 2024-04-25T13:49:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.365+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fe81133a0 0x557fe85a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1ba7fc640 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fe81133a0 0x557fe85a1f70 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23a824c4596b9487 server_cookie=42c2ab8a6a0e7ef7 in_seq=0 out_seq=0 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe19eff5640 10 client.26864 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe19eff5640 1 -- 172.21.15.27:0/2006246878 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557fe848d800 con 0x557fe81133a0 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe19eff5640 10 client.26864 handle_command_reply: tid=0 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 client.26864 shutdown 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 msgr2=0x557fe8539a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe8539a10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0031440 tx=0x7fe1b00045f0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 shutdown_connections 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557fe81133a0 0x557fe85a1f70 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557fe817f390 0x557fe8536ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557fe815e6b0 0x557fe8536590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 --2- 172.21.15.27:0/2006246878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557fe81564d0 0x557fe8539a10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 >> 172.21.15.27:0/2006246878 conn(0x557fe81734e0 msgr2=0x557fe852a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe19eff5640 0 client.26864 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 shutdown_connections 2024-04-25T13:49:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.366+0000 7fe1c842a740 1 -- 172.21.15.27:0/2006246878 wait complete. 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1c842a740 20 client.26864 trim_cache size 0 max 16384 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stdout: "807d4185-5141-4cc0-a290-da7cc01ac56d": { 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "807d4185-5141-4cc0-a290-da7cc01ac56d", 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:49:39.370 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:49:39.371 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:49:39.371 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:49:39.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe1a40afd00 msgr2=0x7fe1a40b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe1a40afd00 0x7fe1a40b2210 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4010740 tx=0x7fe1b4014040 comp rx=0 tx=0).stop 2024-04-25T13:49:39.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1bc102130 msgr2=0x7fe1bc19c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:39.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1bc102130 0x7fe1bc19c750 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00e9f0 tx=0x7fe1ac00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 shutdown_connections 2024-04-25T13:49:39.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe1a40afd00 0x7fe1a40b2210 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1bc1031c0 0x7fe1bc1a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1bc102130 0x7fe1bc19c750 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 --2- 172.21.15.27:0/750659845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1bc069a20 0x7fe1bc19c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:39.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.367+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 >> 172.21.15.27:0/750659845 conn(0x7fe1bc0fd170 msgr2=0x7fe1bc109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:39.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.368+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 shutdown_connections 2024-04-25T13:49:39.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:49:39.368+0000 7fe1bbfff640 1 -- 172.21.15.27:0/750659845 wait complete. 2024-04-25T13:49:39.382 INFO:tasks.cephfs.filesystem:scrub status for tag:807d4185-5141-4cc0-a290-da7cc01ac56d - {'path': '/', 'tag': '807d4185-5141-4cc0-a290-da7cc01ac56d', 'options': 'recursive,force'} 2024-04-25T13:49:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:40 smithi027 ceph-mon[29140]: pgmap v612: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-25T13:49:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:40 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:49:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:40 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:40 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:49:40.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:40 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:40.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:40 smithi122 ceph-mon[32843]: pgmap v612: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-25T13:49:40.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:40 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:49:40.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:40 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:40.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:40 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:49:40.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:40 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:40.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:40 smithi045 ceph-mon[32731]: pgmap v612: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-25T13:49:40.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:40 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:49:40.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:40 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:40.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:40 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:49:40.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:40 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:49:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:42 smithi027 ceph-mon[29140]: pgmap v613: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.5 MiB/s wr, 626 op/s 2024-04-25T13:49:42.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:42 smithi122 ceph-mon[32843]: pgmap v613: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.5 MiB/s wr, 626 op/s 2024-04-25T13:49:42.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:42 smithi045 ceph-mon[32731]: pgmap v613: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.5 MiB/s wr, 626 op/s 2024-04-25T13:49:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 515.0 s, 2262.2 tps, lat 43.854 ms stddev 49.309 2024-04-25T13:49:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:44 smithi027 ceph-mon[29140]: pgmap v614: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 6.4 MiB/s wr, 736 op/s 2024-04-25T13:49:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:44 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:49:44.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.508+0000 7f468c73f640 1 -- 172.21.15.27:0/2970066980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684108960 msgr2=0x7f468410ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.508 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.508+0000 7f468c73f640 1 --2- 172.21.15.27:0/2970066980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684108960 0x7f468410ad80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f467c009ca0 tx=0x7f467c02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.509+0000 7f468c73f640 1 -- 172.21.15.27:0/2970066980 shutdown_connections 2024-04-25T13:49:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.509+0000 7f468c73f640 1 --2- 172.21.15.27:0/2970066980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684108960 0x7f468410ad80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.509+0000 7f468c73f640 1 --2- 172.21.15.27:0/2970066980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4684103c60 0x7f4684104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.509+0000 7f468c73f640 1 --2- 172.21.15.27:0/2970066980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4684102a20 0x7f4684103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.509+0000 7f468c73f640 1 -- 172.21.15.27:0/2970066980 >> 172.21.15.27:0/2970066980 conn(0x7f46840fdc10 msgr2=0x7f46841000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.509+0000 7f468c73f640 1 -- 172.21.15.27:0/2970066980 shutdown_connections 2024-04-25T13:49:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.510+0000 7f468c73f640 1 -- 172.21.15.27:0/2970066980 wait complete. 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468c73f640 1 Processor -- start 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468c73f640 1 -- start start 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468c73f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4684102a20 0x7f4684113340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468c73f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684103c60 0x7f4684113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468c73f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4684108960 0x7f46841b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468c73f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f468410d340 con 0x7f4684103c60 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468c73f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f468410d1c0 con 0x7f4684108960 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468c73f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f468410d4c0 con 0x7f4684102a20 2024-04-25T13:49:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f4689cda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684103c60 0x7f4684113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468acdc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4684108960 0x7f46841b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f4689cda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684103c60 0x7f4684113880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49500/0 (socket says 172.21.15.27:49500) 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f4689cda640 1 -- 172.21.15.27:0/3881954846 learned_addr learned my addr 172.21.15.27:0/3881954846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.511+0000 7f468acdc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4684108960 0x7f46841b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:44272/0 (socket says 172.21.15.27:44272) 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.512+0000 7f468a4db640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4684102a20 0x7f4684113340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.512+0000 7f4689cda640 1 -- 172.21.15.27:0/3881954846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4684102a20 msgr2=0x7f4684113340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.512+0000 7f4689cda640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4684102a20 0x7f4684113340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.512+0000 7f4689cda640 1 -- 172.21.15.27:0/3881954846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4684108960 msgr2=0x7f46841b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.512+0000 7f4689cda640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4684108960 0x7f46841b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.512+0000 7f4689cda640 1 -- 172.21.15.27:0/3881954846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46841b0b30 con 0x7f4684103c60 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.512+0000 7f468a4db640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4684102a20 0x7f4684113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.512+0000 7f468acdc640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4684108960 0x7f46841b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.513+0000 7f4689cda640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684103c60 0x7f4684113880 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f467400ea60 tx=0x7f467400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.513+0000 7f467b7fe640 1 -- 172.21.15.27:0/3881954846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467400cde0 con 0x7f4684103c60 2024-04-25T13:49:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.513+0000 7f467b7fe640 1 -- 172.21.15.27:0/3881954846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46740035f0 con 0x7f4684103c60 2024-04-25T13:49:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.513+0000 7f468c73f640 1 -- 172.21.15.27:0/3881954846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46841b0e20 con 0x7f4684103c60 2024-04-25T13:49:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.513+0000 7f467b7fe640 1 -- 172.21.15.27:0/3881954846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4674010690 con 0x7f4684103c60 2024-04-25T13:49:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.513+0000 7f468c73f640 1 -- 172.21.15.27:0/3881954846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46841b12e0 con 0x7f4684103c60 2024-04-25T13:49:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.516+0000 7f46935f2740 1 Processor -- start 2024-04-25T13:49:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.516+0000 7f467b7fe640 1 -- 172.21.15.27:0/3881954846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4674003ac0 con 0x7f4684103c60 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f46935f2740 1 -- start start 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f46935f2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034ac970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f46935f2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034acef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f46935f2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f46935f2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564303412460 con 0x5643030eb4d0 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f46935f2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5643033cc590 con 0x5643030e32f0 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f46935f2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56430340e420 con 0x56430310c1b0 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f468a4db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f4689cda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f468a4db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034acef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:44284/0 (socket says 172.21.15.27:44284) 2024-04-25T13:49:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f468acdc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.517+0000 7f468a4db640 1 -- 172.21.15.27:0/2714398997 learned_addr learned my addr 172.21.15.27:0/2714398997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f468a4db640 1 -- 172.21.15.27:0/2714398997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 msgr2=0x5643034ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f468a4db640 1 --2- 172.21.15.27:0/2714398997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f468a4db640 1 -- 172.21.15.27:0/2714398997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 msgr2=0x5643034b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f468a4db640 1 --2- 172.21.15.27:0/2714398997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f468a4db640 1 -- 172.21.15.27:0/2714398997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643033d90d0 con 0x5643030e32f0 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f468a4db640 1 --2- 172.21.15.27:0/2714398997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034acef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4680009cc0 tx=0x7f4680010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1bd809d59d649a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f4678ff9640 1 -- 172.21.15.27:0/2714398997 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468001b480 con 0x5643030e32f0 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f467b7fe640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afbb0 0x7f46700b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f4678ff9640 1 -- 172.21.15.27:0/2714398997 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4680012900 con 0x5643030e32f0 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f4678ff9640 1 -- 172.21.15.27:0/2714398997 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46800118d0 con 0x5643030e32f0 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f468acdc640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afbb0 0x7f46700b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.518+0000 7f467b7fe640 1 -- 172.21.15.27:0/3881954846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f46740deea0 con 0x7f4684103c60 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f468acdc640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afbb0 0x7f46700b20c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x5643034adb40 tx=0x7f467c009460 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f46935f2740 1 -- 172.21.15.27:0/2714398997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 msgr2=0x5643034acef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f46935f2740 1 --2- 172.21.15.27:0/2714398997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034acef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4680009cc0 tx=0x7f4680010740 comp rx=0 tx=0).stop 2024-04-25T13:49:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f46935f2740 1 -- 172.21.15.27:0/2714398997 shutdown_connections 2024-04-25T13:49:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f46935f2740 1 --2- 172.21.15.27:0/2714398997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f46935f2740 1 --2- 172.21.15.27:0/2714398997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034acef0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f46935f2740 1 --2- 172.21.15.27:0/2714398997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.521 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f46935f2740 1 -- 172.21.15.27:0/2714398997 >> 172.21.15.27:0/2714398997 conn(0x564303100300 msgr2=0x5643034a8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.519+0000 7f46935f2740 1 -- 172.21.15.27:0/2714398997 shutdown_connections 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.520+0000 7f46935f2740 1 -- 172.21.15.27:0/2714398997 wait complete. 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.520+0000 7f46935f2740 1 -- start start 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.520+0000 7f46935f2740 1 Processor -- start 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f46935f2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030e32f0 0x5643034b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f46935f2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030eb4d0 0x5643034b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f46935f2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56430310c1b0 0x5643034bdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f468a4db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030e32f0 0x5643034b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f468a4db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030e32f0 0x5643034b6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49528/0 (socket says 172.21.15.27:49528) 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f468a4db640 1 -- 172.21.15.27:0/869350303 learned_addr learned my addr 172.21.15.27:0/869350303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f468acdc640 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56430310c1b0 0x5643034bdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f4689cda640 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030eb4d0 0x5643034b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5643034b4840 con 0x5643030e32f0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564303399b60 con 0x56430310c1b0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.521+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5643034b49c0 con 0x5643030eb4d0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.522+0000 7f46935f2740 10 client.0 _ll_register_callbacks cb 0x5643033be2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.522+0000 7f468acdc640 1 -- 172.21.15.27:0/869350303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030eb4d0 msgr2=0x5643034b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.522+0000 7f468acdc640 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030eb4d0 0x5643034b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.522+0000 7f468acdc640 1 -- 172.21.15.27:0/869350303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030e32f0 msgr2=0x5643034b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.522+0000 7f468acdc640 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030e32f0 0x5643034b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.522+0000 7f468acdc640 1 -- 172.21.15.27:0/869350303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564303124210 con 0x56430310c1b0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.522+0000 7f468acdc640 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56430310c1b0 0x5643034bdaf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f467c009bf0 tx=0x7f467c02a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f8a5542479f04e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.522+0000 7f466dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:49:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.523+0000 7f466dffb640 1 -- 172.21.15.27:0/869350303 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c004250 con 0x56430310c1b0 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.523+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4680012900 con 0x56430310c1b0 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.523+0000 7f466dffb640 1 -- 172.21.15.27:0/869350303 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f467c0043f0 con 0x56430310c1b0 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.523+0000 7f466dffb640 1 -- 172.21.15.27:0/869350303 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f467c0045f0 con 0x56430310c1b0 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.523+0000 7f46935f2740 10 client.26876 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.523+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5643031f0620 con 0x56430310c1b0 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f466dffb640 1 -- 172.21.15.27:0/869350303 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f467c036040 con 0x56430310c1b0 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f466dffb640 1 -- 172.21.15.27:0/869350303 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f467c052070 con 0x56430310c1b0 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f46935f2740 10 client.26876 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f46935f2740 10 client.26876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f46935f2740 20 client.26876 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f46935f2740 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5643030cad80 0x5643034c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f46935f2740 4 client.26876 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4680012900 con 0x5643030cad80 2024-04-25T13:49:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.524+0000 7f468a4db640 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5643030cad80 0x5643034c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.526+0000 7f466dffb640 10 client.26876 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.526+0000 7f468a4db640 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5643030cad80 0x5643034c44d0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd1688daf08f1393 server_cookie=b5cb46c811fd957a in_seq=0 out_seq=0 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f466dffb640 1 -- 172.21.15.27:0/869350303 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f4680012900 con 0x5643030cad80 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f466dffb640 10 client.26876 handle_command_reply: tid=0 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 client.26876 shutdown 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56430310c1b0 msgr2=0x5643034bdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56430310c1b0 0x5643034bdaf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f467c009bf0 tx=0x7f467c02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 shutdown_connections 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5643030cad80 0x5643034c44d0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56430310c1b0 0x5643034bdaf0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030eb4d0 0x5643034b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 --2- 172.21.15.27:0/869350303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030e32f0 0x5643034b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 >> 172.21.15.27:0/869350303 conn(0x564303100300 msgr2=0x5643034ab210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f466dffb640 0 client.26876 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 shutdown_connections 2024-04-25T13:49:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 1 -- 172.21.15.27:0/869350303 wait complete. 2024-04-25T13:49:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.527+0000 7f46935f2740 20 client.26876 trim_cache size 0 max 16384 2024-04-25T13:49:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f46935f2740 1 Processor -- start 2024-04-25T13:49:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f46935f2740 1 -- start start 2024-04-25T13:49:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f46935f2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f46935f2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f46935f2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034bda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f46935f2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f46841b8040 con 0x5643030eb4d0 2024-04-25T13:49:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f46935f2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46841ba8c0 con 0x5643030e32f0 2024-04-25T13:49:44.552 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f46935f2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f46841b86b0 con 0x56430310c1b0 2024-04-25T13:49:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f4689cda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f468acdc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034bda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f4689cda640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49530/0 (socket says 172.21.15.27:49530) 2024-04-25T13:49:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f4689cda640 1 -- 172.21.15.27:0/2620338623 learned_addr learned my addr 172.21.15.27:0/2620338623 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:44.553 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.552+0000 7f468a4db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.553+0000 7f468acdc640 1 -- 172.21.15.27:0/2620338623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 msgr2=0x5643034b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.553+0000 7f468acdc640 1 --2- 172.21.15.27:0/2620338623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.553+0000 7f468acdc640 1 -- 172.21.15.27:0/2620338623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 msgr2=0x5643034b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.553+0000 7f468acdc640 1 --2- 172.21.15.27:0/2620338623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.553+0000 7f468acdc640 1 -- 172.21.15.27:0/2620338623 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643034ad660 con 0x56430310c1b0 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.554+0000 7f468acdc640 1 --2- 172.21.15.27:0/2620338623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034bda10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f467c0306b0 tx=0x7f467c033000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8785c76f21bbc9a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.554+0000 7f466ffff640 1 -- 172.21.15.27:0/2620338623 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c004250 con 0x56430310c1b0 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.554+0000 7f466ffff640 1 -- 172.21.15.27:0/2620338623 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f467c0043f0 con 0x56430310c1b0 2024-04-25T13:49:44.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.554+0000 7f466ffff640 1 -- 172.21.15.27:0/2620338623 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c02f9d0 con 0x56430310c1b0 2024-04-25T13:49:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 -- 172.21.15.27:0/2620338623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 msgr2=0x5643034bda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 --2- 172.21.15.27:0/2620338623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034bda10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f467c0306b0 tx=0x7f467c033000 comp rx=0 tx=0).stop 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 -- 172.21.15.27:0/2620338623 shutdown_connections 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 --2- 172.21.15.27:0/2620338623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56430310c1b0 0x5643034bda10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 --2- 172.21.15.27:0/2620338623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5643030eb4d0 0x5643034b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 --2- 172.21.15.27:0/2620338623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030e32f0 0x5643034b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 -- 172.21.15.27:0/2620338623 >> 172.21.15.27:0/2620338623 conn(0x564303100300 msgr2=0x5643034ac8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 -- 172.21.15.27:0/2620338623 shutdown_connections 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.555+0000 7f46935f2740 1 -- 172.21.15.27:0/2620338623 wait complete. 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 1 -- start start 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 1 Processor -- start 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030e32f0 0x5643034bb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030eb4d0 0x5643034bb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56430310c1b0 0x5643034bc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f46841b8d80 con 0x56430310c1b0 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46841ba1f0 con 0x5643030eb4d0 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f46841b9450 con 0x5643030e32f0 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f46935f2740 10 client.0 _ll_register_callbacks cb 0x7f467c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468acdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56430310c1b0 0x5643034bc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468a4db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030e32f0 0x5643034bb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468acdc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56430310c1b0 0x5643034bc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49536/0 (socket says 172.21.15.27:49536) 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468acdc640 1 -- 172.21.15.27:0/4109274810 learned_addr learned my addr 172.21.15.27:0/4109274810 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f4689cda640 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030eb4d0 0x5643034bb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468a4db640 1 -- 172.21.15.27:0/4109274810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030eb4d0 msgr2=0x5643034bb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468a4db640 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030eb4d0 0x5643034bb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468a4db640 1 -- 172.21.15.27:0/4109274810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56430310c1b0 msgr2=0x5643034bc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468a4db640 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56430310c1b0 0x5643034bc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468a4db640 1 -- 172.21.15.27:0/4109274810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643034bdf50 con 0x5643030e32f0 2024-04-25T13:49:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.556+0000 7f468acdc640 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56430310c1b0 0x5643034bc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f468a4db640 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030e32f0 0x5643034bb140 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5643034c4c70 tx=0x7f4680010d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2d620fb64c951e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f466dffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f466dffb640 1 -- 172.21.15.27:0/4109274810 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680005040 con 0x5643030e32f0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f466dffb640 1 -- 172.21.15.27:0/4109274810 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f46800051e0 con 0x5643030e32f0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f46935f2740 1 -- 172.21.15.27:0/4109274810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f467c0043f0 con 0x5643030e32f0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f466dffb640 1 -- 172.21.15.27:0/4109274810 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f46800053e0 con 0x5643030e32f0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f46935f2740 10 client.26935 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f46935f2740 1 -- 172.21.15.27:0/4109274810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5643034b41b0 con 0x5643030e32f0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f466dffb640 1 -- 172.21.15.27:0/4109274810 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f467c0043f0 con 0x5643030e32f0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.557+0000 7f466dffb640 1 -- 172.21.15.27:0/4109274810 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f468000f040 con 0x5643030e32f0 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.558+0000 7f46935f2740 10 client.26935 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.558+0000 7f46935f2740 10 client.26935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.558+0000 7f46935f2740 20 client.26935 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.558+0000 7f46935f2740 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5643030a01c0 0x56430352eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:44.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.558+0000 7f46935f2740 4 client.26935 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:49:44.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.558+0000 7f46935f2740 1 -- 172.21.15.27:0/4109274810 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564303420070 con 0x5643030a01c0 2024-04-25T13:49:44.561 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:44 smithi122 ceph-mon[32843]: pgmap v614: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 6.4 MiB/s wr, 736 op/s 2024-04-25T13:49:44.561 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:44 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:49:44.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.559+0000 7f4689cda640 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5643030a01c0 0x56430352eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:44.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.559+0000 7f466dffb640 10 client.26935 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:44.562 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.559+0000 7f4689cda640 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5643030a01c0 0x56430352eaa0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=398b6ca37ae5d016 server_cookie=17525041dcdd52a7 in_seq=0 out_seq=0 2024-04-25T13:49:44.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:44 smithi045 ceph-mon[32731]: pgmap v614: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 6.4 MiB/s wr, 736 op/s 2024-04-25T13:49:44.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:44 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:49:44.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f466dffb640 1 -- 172.21.15.27:0/4109274810 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564303420070 con 0x5643030a01c0 2024-04-25T13:49:44.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f466dffb640 10 client.26935 handle_command_reply: tid=0 2024-04-25T13:49:44.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 client.26935 shutdown 2024-04-25T13:49:44.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 -- 172.21.15.27:0/4109274810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030e32f0 msgr2=0x5643034bb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030e32f0 0x5643034bb140 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5643034c4c70 tx=0x7f4680010d70 comp rx=0 tx=0).stop 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 -- 172.21.15.27:0/4109274810 shutdown_connections 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5643030a01c0 0x56430352eaa0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56430310c1b0 0x5643034bc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5643030eb4d0 0x5643034bb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f466dffb640 0 client.26935 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 --2- 172.21.15.27:0/4109274810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5643030e32f0 0x5643034bb140 secure :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5643034c4c70 tx=0x7f4680010d70 comp rx=0 tx=0).stop 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 -- 172.21.15.27:0/4109274810 >> 172.21.15.27:0/4109274810 conn(0x564303100300 msgr2=0x5643034b6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 -- 172.21.15.27:0/4109274810 shutdown_connections 2024-04-25T13:49:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 1 -- 172.21.15.27:0/4109274810 wait complete. 2024-04-25T13:49:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.649+0000 7f46935f2740 20 client.26935 trim_cache size 0 max 16384 2024-04-25T13:49:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:49:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:49:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:49:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:49:44.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.650+0000 7f468c73f640 1 -- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afbb0 msgr2=0x7f46700b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.651+0000 7f468c73f640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afbb0 0x7f46700b20c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x5643034adb40 tx=0x7f467c009460 comp rx=0 tx=0).stop 2024-04-25T13:49:44.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.651+0000 7f468c73f640 1 -- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684103c60 msgr2=0x7f4684113880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:44.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.651+0000 7f468c73f640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684103c60 0x7f4684113880 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f467400ea60 tx=0x7f467400ef20 comp rx=0 tx=0).stop 2024-04-25T13:49:44.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.653+0000 7f468c73f640 1 -- 172.21.15.27:0/3881954846 shutdown_connections 2024-04-25T13:49:44.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.654+0000 7f468c73f640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afbb0 0x7f46700b20c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.654+0000 7f468c73f640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4684108960 0x7f46841b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.654+0000 7f468c73f640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4684103c60 0x7f4684113880 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.654+0000 7f468c73f640 1 --2- 172.21.15.27:0/3881954846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4684102a20 0x7f4684113340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:44.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.654+0000 7f468c73f640 1 -- 172.21.15.27:0/3881954846 >> 172.21.15.27:0/3881954846 conn(0x7f46840fdc10 msgr2=0x7f4684104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:44.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.654+0000 7f468c73f640 1 -- 172.21.15.27:0/3881954846 shutdown_connections 2024-04-25T13:49:44.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:44.654+0000 7f468c73f640 1 -- 172.21.15.27:0/3881954846 wait complete. 2024-04-25T13:49:46.378 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:46 smithi045 ceph-mon[32731]: pgmap v615: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.3 MiB/s wr, 706 op/s 2024-04-25T13:49:46.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:46 smithi027 ceph-mon[29140]: pgmap v615: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.3 MiB/s wr, 706 op/s 2024-04-25T13:49:46.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:46 smithi122 ceph-mon[32843]: pgmap v615: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.3 MiB/s wr, 706 op/s 2024-04-25T13:49:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 520.0 s, 2830.8 tps, lat 35.577 ms stddev 41.349 2024-04-25T13:49:48.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:48 smithi027 ceph-mon[29140]: pgmap v616: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.7 MiB/s wr, 798 op/s 2024-04-25T13:49:48.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:48 smithi122 ceph-mon[32843]: pgmap v616: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.7 MiB/s wr, 798 op/s 2024-04-25T13:49:48.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:48 smithi045 ceph-mon[32731]: pgmap v616: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.7 MiB/s wr, 798 op/s 2024-04-25T13:49:50.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:50 smithi027 ceph-mon[29140]: pgmap v617: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-04-25T13:49:50.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:50 smithi122 ceph-mon[32843]: pgmap v617: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-04-25T13:49:50.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:50 smithi045 ceph-mon[32731]: pgmap v617: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 6.5 MiB/s wr, 720 op/s 2024-04-25T13:49:52.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:52 smithi027 ceph-mon[29140]: pgmap v618: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 6.1 MiB/s wr, 697 op/s 2024-04-25T13:49:52.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:52 smithi122 ceph-mon[32843]: pgmap v618: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 6.1 MiB/s wr, 697 op/s 2024-04-25T13:49:52.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:52 smithi045 ceph-mon[32731]: pgmap v618: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 6.1 MiB/s wr, 697 op/s 2024-04-25T13:49:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 525.0 s, 1772.8 tps, lat 55.907 ms stddev 60.969 2024-04-25T13:49:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:54 smithi027 ceph-mon[29140]: pgmap v619: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 7.7 MiB/s wr, 815 op/s 2024-04-25T13:49:54.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:54 smithi122 ceph-mon[32843]: pgmap v619: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 7.7 MiB/s wr, 815 op/s 2024-04-25T13:49:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:54 smithi045 ceph-mon[32731]: pgmap v619: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 7.7 MiB/s wr, 815 op/s 2024-04-25T13:49:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:56 smithi027 ceph-mon[29140]: pgmap v620: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.9 MiB/s wr, 608 op/s 2024-04-25T13:49:56.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:56 smithi122 ceph-mon[32843]: pgmap v620: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.9 MiB/s wr, 608 op/s 2024-04-25T13:49:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:56 smithi045 ceph-mon[32731]: pgmap v620: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.9 MiB/s wr, 608 op/s 2024-04-25T13:49:57.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.800+0000 7f6193b34640 1 -- 172.21.15.27:0/4178344526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c102a30 msgr2=0x7f618c103730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.800+0000 7f6193b34640 1 --2- 172.21.15.27:0/4178344526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c102a30 0x7f618c103730 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f618401acb0 tx=0x7f618403b920 comp rx=0 tx=0).stop 2024-04-25T13:49:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.802+0000 7f6193b34640 1 -- 172.21.15.27:0/4178344526 shutdown_connections 2024-04-25T13:49:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.802+0000 7f6193b34640 1 --2- 172.21.15.27:0/4178344526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618c108970 0x7f618c10ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.802+0000 7f6193b34640 1 --2- 172.21.15.27:0/4178344526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618c103c70 0x7f618c104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.802+0000 7f6193b34640 1 --2- 172.21.15.27:0/4178344526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c102a30 0x7f618c103730 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.802+0000 7f6193b34640 1 -- 172.21.15.27:0/4178344526 >> 172.21.15.27:0/4178344526 conn(0x7f618c0fdc20 msgr2=0x7f618c1000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:57.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.806+0000 7f6193b34640 1 -- 172.21.15.27:0/4178344526 shutdown_connections 2024-04-25T13:49:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.808+0000 7f6193b34640 1 -- 172.21.15.27:0/4178344526 wait complete. 2024-04-25T13:49:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.808+0000 7f6193b34640 1 Processor -- start 2024-04-25T13:49:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.812+0000 7f6193b34640 1 -- start start 2024-04-25T13:49:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.812+0000 7f6193b34640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618c102a30 0x7f618c0748d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.812+0000 7f6193b34640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618c103c70 0x7f618c072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.812+0000 7f6193b34640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c108970 0x7f618c0734b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.812+0000 7f6193b34640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f618c06ce20 con 0x7f618c108970 2024-04-25T13:49:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.812+0000 7f6193b34640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f618c06cca0 con 0x7f618c102a30 2024-04-25T13:49:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.812+0000 7f6193b34640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f618c06cfa0 con 0x7f618c103c70 2024-04-25T13:49:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.813+0000 7f61918d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618c102a30 0x7f618c0748d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.813+0000 7f61918d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618c102a30 0x7f618c0748d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49812/0 (socket says 172.21.15.27:49812) 2024-04-25T13:49:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.813+0000 7f61918d0640 1 -- 172.21.15.27:0/3742676428 learned_addr learned my addr 172.21.15.27:0/3742676428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.813+0000 7f61920d1640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c108970 0x7f618c0734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.814+0000 7f61920d1640 1 -- 172.21.15.27:0/3742676428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618c103c70 msgr2=0x7f618c072f70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:49:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.814+0000 7f61920d1640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618c103c70 0x7f618c072f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.814+0000 7f61920d1640 1 -- 172.21.15.27:0/3742676428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618c102a30 msgr2=0x7f618c0748d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.814+0000 7f61920d1640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618c102a30 0x7f618c0748d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.814+0000 7f61920d1640 1 -- 172.21.15.27:0/3742676428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c073a20 con 0x7f618c108970 2024-04-25T13:49:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.814+0000 7f61920d1640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c108970 0x7f618c0734b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f618800b3e0 tx=0x7f618800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.816+0000 7f6182ffd640 1 -- 172.21.15.27:0/3742676428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188015020 con 0x7f618c108970 2024-04-25T13:49:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.816+0000 7f6182ffd640 1 -- 172.21.15.27:0/3742676428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6188011ce0 con 0x7f618c108970 2024-04-25T13:49:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.817+0000 7f6182ffd640 1 -- 172.21.15.27:0/3742676428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188010e00 con 0x7f618c108970 2024-04-25T13:49:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.817+0000 7f6193b34640 1 -- 172.21.15.27:0/3742676428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c073d10 con 0x7f618c108970 2024-04-25T13:49:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.817+0000 7f6193b34640 1 -- 172.21.15.27:0/3742676428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c1a8db0 con 0x7f618c108970 2024-04-25T13:49:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.822+0000 7f619a9e8740 1 Processor -- start 2024-04-25T13:49:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.826+0000 7f6182ffd640 1 -- 172.21.15.27:0/3742676428 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6188018020 con 0x7f618c108970 2024-04-25T13:49:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.831+0000 7f619a9e8740 1 -- start start 2024-04-25T13:49:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.834+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb042e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.834+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb00652f0 0x559eb042eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.834+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb006d4d0 0x559eb0435df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.834+0000 7f619a9e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559eb0394460 con 0x559eb008e1b0 2024-04-25T13:49:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.834+0000 7f619a9e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559eb034e590 con 0x559eb00652f0 2024-04-25T13:49:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.834+0000 7f619a9e8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559eb0390420 con 0x559eb006d4d0 2024-04-25T13:49:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.834+0000 7f6182ffd640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f61780afcf0 0x7f61780b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.834+0000 7f6182ffd640 1 -- 172.21.15.27:0/3742676428 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f61880de1a0 con 0x7f618c108970 2024-04-25T13:49:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.836+0000 7f61920d1640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f61780afcf0 0x7f61780b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.836+0000 7f61918d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb00652f0 0x559eb042eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.836+0000 7f61918d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb00652f0 0x559eb042eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49814/0 (socket says 172.21.15.27:49814) 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.836+0000 7f61918d0640 1 -- 172.21.15.27:0/1581984554 learned_addr learned my addr 172.21.15.27:0/1581984554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.837+0000 7f61910cf640 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb006d4d0 0x559eb0435df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.837+0000 7f61910cf640 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb042e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.837+0000 7f61918d0640 1 -- 172.21.15.27:0/1581984554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb006d4d0 msgr2=0x559eb0435df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.837+0000 7f61918d0640 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb006d4d0 0x559eb0435df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.837+0000 7f61918d0640 1 -- 172.21.15.27:0/1581984554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 msgr2=0x559eb042e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.837+0000 7f61918d0640 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb042e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.837+0000 7f61918d0640 1 -- 172.21.15.27:0/1581984554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559eb035b0d0 con 0x559eb00652f0 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.837+0000 7f61918d0640 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb00652f0 0x559eb042eef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f61840406b0 tx=0x7f61840406e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22c4f7d5d7c28b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.838+0000 7f61920d1640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f61780afcf0 0x7f61780b2200 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6188009fd0 tx=0x7f6188010820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.838+0000 7f6177fff640 1 -- 172.21.15.27:0/1581984554 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184045030 con 0x559eb00652f0 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.838+0000 7f6177fff640 1 -- 172.21.15.27:0/1581984554 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6184003e20 con 0x559eb00652f0 2024-04-25T13:49:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.838+0000 7f6177fff640 1 -- 172.21.15.27:0/1581984554 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61840042d0 con 0x559eb00652f0 2024-04-25T13:49:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.839+0000 7f619a9e8740 1 -- 172.21.15.27:0/1581984554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb00652f0 msgr2=0x559eb042eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.839+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb00652f0 0x559eb042eef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f61840406b0 tx=0x7f61840406e0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.839+0000 7f619a9e8740 1 -- 172.21.15.27:0/1581984554 shutdown_connections 2024-04-25T13:49:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.839+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb006d4d0 0x559eb0435df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.839+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb00652f0 0x559eb042eef0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.839+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1581984554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb042e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.839+0000 7f619a9e8740 1 -- 172.21.15.27:0/1581984554 >> 172.21.15.27:0/1581984554 conn(0x559eb0082300 msgr2=0x559eb042a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.841+0000 7f619a9e8740 1 -- 172.21.15.27:0/1581984554 shutdown_connections 2024-04-25T13:49:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.841+0000 7f619a9e8740 1 -- 172.21.15.27:0/1581984554 wait complete. 2024-04-25T13:49:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.842+0000 7f619a9e8740 1 -- start start 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.842+0000 7f619a9e8740 1 Processor -- start 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0438640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb0438b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 0x559eb043fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f619a9e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559eb04367e0 con 0x559eb006d4d0 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f619a9e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559eb031bb60 con 0x559eb008e1b0 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f619a9e8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559eb0436960 con 0x559eb00652f0 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f619a9e8740 10 client.0 _ll_register_callbacks cb 0x559eb03402b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f61918d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb0438b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f61910cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0438640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f61910cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0438640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59168/0 (socket says 172.21.15.27:59168) 2024-04-25T13:49:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f61910cf640 1 -- 172.21.15.27:0/1857575345 learned_addr learned my addr 172.21.15.27:0/1857575345 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.843+0000 7f61910cf640 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 0x559eb043fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61910cf640 1 -- 172.21.15.27:0/1857575345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 msgr2=0x559eb043fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61910cf640 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 0x559eb043fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61910cf640 1 -- 172.21.15.27:0/1857575345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 msgr2=0x559eb0438b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61910cf640 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb0438b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61910cf640 1 -- 172.21.15.27:0/1857575345 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559eb00a6210 con 0x559eb00652f0 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61910cf640 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0438640 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f617c018c50 tx=0x7f617c006860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c1f277d326eb77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61757fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61757fa640 1 -- 172.21.15.27:0/1857575345 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c01e030 con 0x559eb00652f0 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f619a9e8740 1 -- 172.21.15.27:0/1857575345 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6184003e20 con 0x559eb00652f0 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61757fa640 1 -- 172.21.15.27:0/1857575345 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f617c016b80 con 0x559eb00652f0 2024-04-25T13:49:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.844+0000 7f61757fa640 1 -- 172.21.15.27:0/1857575345 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f617c011040 con 0x559eb00652f0 2024-04-25T13:49:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.847+0000 7f619a9e8740 10 client.26941 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.847+0000 7f619a9e8740 1 -- 172.21.15.27:0/1857575345 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559eb0172620 con 0x559eb00652f0 2024-04-25T13:49:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.848+0000 7f61757fa640 1 -- 172.21.15.27:0/1857575345 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6184003e20 con 0x559eb00652f0 2024-04-25T13:49:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.848+0000 7f61757fa640 1 -- 172.21.15.27:0/1857575345 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f617c014050 con 0x559eb00652f0 2024-04-25T13:49:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.848+0000 7f619a9e8740 10 client.26941 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.848+0000 7f619a9e8740 10 client.26941 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.848+0000 7f619a9e8740 20 client.26941 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.848+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559eb004cd80 0x559eb0446350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.848+0000 7f619a9e8740 4 client.26941 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:49:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.848+0000 7f619a9e8740 1 -- 172.21.15.27:0/1857575345 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559eb03a2070 con 0x559eb004cd80 2024-04-25T13:49:57.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.850+0000 7f61918d0640 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559eb004cd80 0x559eb0446350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.851+0000 7f61918d0640 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559eb004cd80 0x559eb0446350 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e487916f6e9fd548 server_cookie=b3db69b7c4887cde in_seq=0 out_seq=0 2024-04-25T13:49:57.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.852+0000 7f61757fa640 10 client.26941 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:57.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.852+0000 7f61757fa640 1 -- 172.21.15.27:0/1857575345 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559eb03a2070 con 0x559eb004cd80 2024-04-25T13:49:57.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.852+0000 7f61757fa640 10 client.26941 handle_command_reply: tid=0 2024-04-25T13:49:57.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.852+0000 7f619a9e8740 1 client.26941 shutdown 2024-04-25T13:49:57.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.852+0000 7f619a9e8740 1 -- 172.21.15.27:0/1857575345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 msgr2=0x559eb0438640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:57.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.852+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0438640 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f617c018c50 tx=0x7f617c006860 comp rx=0 tx=0).stop 2024-04-25T13:49:57.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 1 -- 172.21.15.27:0/1857575345 shutdown_connections 2024-04-25T13:49:57.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559eb004cd80 0x559eb0446350 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 0x559eb043fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb0438b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1857575345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0438640 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 1 -- 172.21.15.27:0/1857575345 >> 172.21.15.27:0/1857575345 conn(0x559eb0082300 msgr2=0x559eb042d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f61757fa640 0 client.26941 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 1 -- 172.21.15.27:0/1857575345 shutdown_connections 2024-04-25T13:49:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 1 -- 172.21.15.27:0/1857575345 wait complete. 2024-04-25T13:49:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.853+0000 7f619a9e8740 20 client.26941 trim_cache size 0 max 16384 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f619a9e8740 1 Processor -- start 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f619a9e8740 1 -- start start 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0429c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb043fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 0x559eb042a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f619a9e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1afa70 con 0x559eb006d4d0 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f619a9e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1b22f0 con 0x559eb008e1b0 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f619a9e8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1b00e0 con 0x559eb00652f0 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f61918d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb043fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f61918d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb043fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36418/0 (socket says 172.21.15.27:36418) 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f61918d0640 1 -- 172.21.15.27:0/2801756809 learned_addr learned my addr 172.21.15.27:0/2801756809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.876+0000 7f61910cf640 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0429c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61910cf640 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 0x559eb042a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61918d0640 1 -- 172.21.15.27:0/2801756809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 msgr2=0x559eb0429c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61918d0640 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0429c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61918d0640 1 -- 172.21.15.27:0/2801756809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 msgr2=0x559eb042a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61918d0640 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 0x559eb042a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61918d0640 1 -- 172.21.15.27:0/2801756809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559eb0435ee0 con 0x559eb006d4d0 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61918d0640 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb043fa70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f618401a630 tx=0x7f618403b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28f2a2bbff92702 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61777fe640 1 -- 172.21.15.27:0/2801756809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184045040 con 0x559eb006d4d0 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61777fe640 1 -- 172.21.15.27:0/2801756809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6184040e80 con 0x559eb006d4d0 2024-04-25T13:49:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.877+0000 7f61777fe640 1 -- 172.21.15.27:0/2801756809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61840043d0 con 0x559eb006d4d0 2024-04-25T13:49:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 -- 172.21.15.27:0/2801756809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 msgr2=0x559eb043fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb043fa70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f618401a630 tx=0x7f618403b920 comp rx=0 tx=0).stop 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 -- 172.21.15.27:0/2801756809 shutdown_connections 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb008e1b0 0x559eb042a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb006d4d0 0x559eb043fa70 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 --2- 172.21.15.27:0/2801756809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb0429c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 -- 172.21.15.27:0/2801756809 >> 172.21.15.27:0/2801756809 conn(0x559eb0082300 msgr2=0x559eb0432b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 -- 172.21.15.27:0/2801756809 shutdown_connections 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 -- 172.21.15.27:0/2801756809 wait complete. 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.879+0000 7f619a9e8740 1 -- start start 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f619a9e8740 1 Processor -- start 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb043ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb006d4d0 0x559eb043cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f619a9e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb04ac740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f619a9e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1b07b0 con 0x559eb008e1b0 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f619a9e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1b1c20 con 0x559eb006d4d0 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f619a9e8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1b0e80 con 0x559eb00652f0 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f61910cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb043ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f61918d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb006d4d0 0x559eb043cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f61910cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb04ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f61918d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb006d4d0 0x559eb043cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49844/0 (socket says 172.21.15.27:49844) 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f61918d0640 1 -- 172.21.15.27:0/1858351888 learned_addr learned my addr 172.21.15.27:0/1858351888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.880+0000 7f619a9e8740 10 client.0 _ll_register_callbacks cb 0x7f617c00d810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61910cf640 1 -- 172.21.15.27:0/1858351888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 msgr2=0x559eb043ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61910cf640 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb043ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61910cf640 1 -- 172.21.15.27:0/1858351888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb006d4d0 msgr2=0x559eb043cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61910cf640 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb006d4d0 0x559eb043cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61910cf640 1 -- 172.21.15.27:0/1858351888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559eb04acc80 con 0x559eb008e1b0 2024-04-25T13:49:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61918d0640 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb006d4d0 0x559eb043cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61910cf640 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb04ac740 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f617c008e70 tx=0x7f617c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880098031df7d0ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61757fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61757fa640 1 -- 172.21.15.27:0/1858351888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c01e070 con 0x559eb008e1b0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61757fa640 1 -- 172.21.15.27:0/1858351888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f617c016d50 con 0x559eb008e1b0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f619a9e8740 1 -- 172.21.15.27:0/1858351888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6184040e80 con 0x559eb008e1b0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f61757fa640 1 -- 172.21.15.27:0/1858351888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f617c004a60 con 0x559eb008e1b0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f619a9e8740 10 client.17187 fetch_fsmap learned FSMap version 19 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.881+0000 7f619a9e8740 1 -- 172.21.15.27:0/1858351888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559eb04adf00 con 0x559eb008e1b0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f61757fa640 1 -- 172.21.15.27:0/1858351888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6184040e80 con 0x559eb008e1b0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f61757fa640 1 -- 172.21.15.27:0/1858351888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f617c014040 con 0x559eb008e1b0 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f619a9e8740 10 client.17187 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f619a9e8740 10 client.17187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:49:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f619a9e8740 20 client.17187 populate_metadata read hostname 'smithi027' 2024-04-25T13:49:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559eb00221c0 0x559eb04b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:49:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f619a9e8740 4 client.17187 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:49:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f619a9e8740 1 -- 172.21.15.27:0/1858351888 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559eb039bee0 con 0x559eb00221c0 2024-04-25T13:49:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.882+0000 7f61918d0640 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559eb00221c0 0x559eb04b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:49:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.883+0000 7f61918d0640 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559eb00221c0 0x559eb04b08c0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc7a6ddea1a0b494 server_cookie=3d51963979759b51 in_seq=0 out_seq=0 2024-04-25T13:49:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.883+0000 7f61757fa640 10 client.17187 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.901+0000 7f61757fa640 1 -- 172.21.15.27:0/1858351888 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559eb039bee0 con 0x559eb00221c0 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.901+0000 7f61757fa640 10 client.17187 handle_command_reply: tid=0 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.901+0000 7f619a9e8740 1 client.17187 shutdown 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 -- 172.21.15.27:0/1858351888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 msgr2=0x559eb04ac740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb04ac740 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f617c008e70 tx=0x7f617c000bc0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 -- 172.21.15.27:0/1858351888 shutdown_connections 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559eb00221c0 0x559eb04b08c0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559eb008e1b0 0x559eb04ac740 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559eb006d4d0 0x559eb043cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f61757fa640 0 client.17187 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 --2- 172.21.15.27:0/1858351888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559eb00652f0 0x559eb043ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 -- 172.21.15.27:0/1858351888 >> 172.21.15.27:0/1858351888 conn(0x559eb0082300 msgr2=0x559eb042d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 -- 172.21.15.27:0/1858351888 shutdown_connections 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 1 -- 172.21.15.27:0/1858351888 wait complete. 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.902+0000 7f619a9e8740 20 client.17187 trim_cache size 0 max 16384 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:49:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:49:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 -- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f61780afcf0 msgr2=0x7f61780b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f61780afcf0 0x7f61780b2200 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6188009fd0 tx=0x7f6188010820 comp rx=0 tx=0).stop 2024-04-25T13:49:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 -- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c108970 msgr2=0x7f618c0734b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:49:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c108970 0x7f618c0734b0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f618800b3e0 tx=0x7f618800b8a0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 -- 172.21.15.27:0/3742676428 shutdown_connections 2024-04-25T13:49:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f61780afcf0 0x7f61780b2200 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f618c108970 0x7f618c0734b0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f618c103c70 0x7f618c072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 --2- 172.21.15.27:0/3742676428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618c102a30 0x7f618c0748d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:49:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 -- 172.21.15.27:0/3742676428 >> 172.21.15.27:0/3742676428 conn(0x7f618c0fdc20 msgr2=0x7f618c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:49:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 -- 172.21.15.27:0/3742676428 shutdown_connections 2024-04-25T13:49:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:49:57.903+0000 7f6193b34640 1 -- 172.21.15.27:0/3742676428 wait complete. 2024-04-25T13:49:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 530.0 s, 1538.4 tps, lat 65.308 ms stddev 70.048 2024-04-25T13:49:58.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:49:58 smithi027 ceph-mon[29140]: pgmap v621: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.5 MiB/s wr, 770 op/s 2024-04-25T13:49:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:49:58 smithi122 ceph-mon[32843]: pgmap v621: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.5 MiB/s wr, 770 op/s 2024-04-25T13:49:58.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:49:58 smithi045 ceph-mon[32731]: pgmap v621: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.5 MiB/s wr, 770 op/s 2024-04-25T13:50:00.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:00 smithi027 ceph-mon[29140]: pgmap v622: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.3 MiB/s wr, 613 op/s 2024-04-25T13:50:00.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:00 smithi027 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T13:50:00.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:00 smithi122 ceph-mon[32843]: pgmap v622: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.3 MiB/s wr, 613 op/s 2024-04-25T13:50:00.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:00 smithi122 ceph-mon[32843]: overall HEALTH_OK 2024-04-25T13:50:00.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:00 smithi045 ceph-mon[32731]: pgmap v622: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.3 MiB/s wr, 613 op/s 2024-04-25T13:50:00.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:00 smithi045 ceph-mon[32731]: overall HEALTH_OK 2024-04-25T13:50:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:02 smithi122 ceph-mon[32843]: pgmap v623: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.4 MiB/s wr, 647 op/s 2024-04-25T13:50:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:02 smithi045 ceph-mon[32731]: pgmap v623: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.4 MiB/s wr, 647 op/s 2024-04-25T13:50:02.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:02 smithi027 ceph-mon[29140]: pgmap v623: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.4 MiB/s wr, 647 op/s 2024-04-25T13:50:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 535.0 s, 1818.2 tps, lat 55.035 ms stddev 64.310 2024-04-25T13:50:04.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:04 smithi122 ceph-mon[32843]: pgmap v624: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.9 MiB/s wr, 725 op/s 2024-04-25T13:50:04.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:04 smithi045 ceph-mon[32731]: pgmap v624: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.9 MiB/s wr, 725 op/s 2024-04-25T13:50:04.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:04 smithi027 ceph-mon[29140]: pgmap v624: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.9 MiB/s wr, 725 op/s 2024-04-25T13:50:06.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:06 smithi122 ceph-mon[32843]: pgmap v625: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T13:50:06.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:06 smithi045 ceph-mon[32731]: pgmap v625: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T13:50:06.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:06 smithi027 ceph-mon[29140]: pgmap v625: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T13:50:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 540.0 s, 2558.8 tps, lat 39.360 ms stddev 43.062 2024-04-25T13:50:08.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:08 smithi122 ceph-mon[32843]: pgmap v626: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 727 op/s 2024-04-25T13:50:08.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:08 smithi045 ceph-mon[32731]: pgmap v626: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 727 op/s 2024-04-25T13:50:08.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:08 smithi027 ceph-mon[29140]: pgmap v626: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 727 op/s 2024-04-25T13:50:09.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:50:09.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:50:09.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.561+0000 7f9618814640 1 -- 172.21.15.27:0/921557400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96141027d0 msgr2=0x7f9614102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.561+0000 7f9618814640 1 --2- 172.21.15.27:0/921557400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96141027d0 0x7f9614102ca0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9608009860 tx=0x7f960802a380 comp rx=0 tx=0).stop 2024-04-25T13:50:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.566+0000 7f9618814640 1 -- 172.21.15.27:0/921557400 shutdown_connections 2024-04-25T13:50:09.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.566+0000 7f9618814640 1 --2- 172.21.15.27:0/921557400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9614107940 0x7f9614107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.566+0000 7f9618814640 1 --2- 172.21.15.27:0/921557400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96141027d0 0x7f9614102ca0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.566+0000 7f9618814640 1 --2- 172.21.15.27:0/921557400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9614101d90 0x7f96141021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.566+0000 7f9618814640 1 -- 172.21.15.27:0/921557400 >> 172.21.15.27:0/921557400 conn(0x7f96140fd380 msgr2=0x7f96140ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.570+0000 7f9618814640 1 -- 172.21.15.27:0/921557400 shutdown_connections 2024-04-25T13:50:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.572+0000 7f9618814640 1 -- 172.21.15.27:0/921557400 wait complete. 2024-04-25T13:50:09.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.573+0000 7f9618814640 1 Processor -- start 2024-04-25T13:50:09.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.577+0000 7f9618814640 1 -- start start 2024-04-25T13:50:09.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.577+0000 7f9618814640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9614101d90 0x7f9614112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.577+0000 7f9618814640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96141027d0 0x7f9614111280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.577+0000 7f9618814640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9614107940 0x7f96141117c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.577+0000 7f9618814640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9614109620 con 0x7f9614101d90 2024-04-25T13:50:09.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.577+0000 7f9618814640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96141094a0 con 0x7f96141027d0 2024-04-25T13:50:09.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.577+0000 7f9618814640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f96141097a0 con 0x7f9614107940 2024-04-25T13:50:09.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f961259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9614101d90 0x7f9614112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f961259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9614101d90 0x7f9614112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46380/0 (socket says 172.21.15.27:46380) 2024-04-25T13:50:09.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f961259c640 1 -- 172.21.15.27:0/3837701506 learned_addr learned my addr 172.21.15.27:0/3837701506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:09.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f9611d9b640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96141027d0 0x7f9614111280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f961259c640 1 -- 172.21.15.27:0/3837701506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9614107940 msgr2=0x7f96141117c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:50:09.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f961259c640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9614107940 0x7f96141117c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f961259c640 1 -- 172.21.15.27:0/3837701506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96141027d0 msgr2=0x7f9614111280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f961259c640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96141027d0 0x7f9614111280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.578+0000 7f961259c640 1 -- 172.21.15.27:0/3837701506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9614111d00 con 0x7f9614101d90 2024-04-25T13:50:09.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.579+0000 7f961259c640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9614101d90 0x7f9614112be0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f960400e9f0 tx=0x7f960400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:09.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.579+0000 7f96037fe640 1 -- 172.21.15.27:0/3837701506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960400cd80 con 0x7f9614101d90 2024-04-25T13:50:09.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.579+0000 7f9618814640 1 -- 172.21.15.27:0/3837701506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9614111ff0 con 0x7f9614101d90 2024-04-25T13:50:09.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.580+0000 7f9618814640 1 -- 172.21.15.27:0/3837701506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96141b13c0 con 0x7f9614101d90 2024-04-25T13:50:09.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.580+0000 7f96037fe640 1 -- 172.21.15.27:0/3837701506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96040108e0 con 0x7f9614101d90 2024-04-25T13:50:09.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.580+0000 7f96037fe640 1 -- 172.21.15.27:0/3837701506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9604018980 con 0x7f9614101d90 2024-04-25T13:50:09.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.584+0000 7f96037fe640 1 -- 172.21.15.27:0/3837701506 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9604018b20 con 0x7f9614101d90 2024-04-25T13:50:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.589+0000 7f96037fe640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95fc0afd00 0x7f95fc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.589+0000 7f96037fe640 1 -- 172.21.15.27:0/3837701506 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f96040dff40 con 0x7f9614101d90 2024-04-25T13:50:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.589+0000 7f9611d9b640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95fc0afd00 0x7f95fc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.590+0000 7f961f79c740 1 Processor -- start 2024-04-25T13:50:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.590+0000 7f961f79c740 1 -- start start 2024-04-25T13:50:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.590+0000 7f961f79c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749aaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.590+0000 7f961f79c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e04d0 0x5584749aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.590+0000 7f961f79c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749abc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.590+0000 7f961f79c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55847490fd80 con 0x558474609390 2024-04-25T13:50:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.590+0000 7f961f79c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5584748c98b0 con 0x5584745e04d0 2024-04-25T13:50:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.590+0000 7f961f79c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55847490bd40 con 0x5584745e86b0 2024-04-25T13:50:09.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.591+0000 7f9611d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e04d0 0x5584749aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.591+0000 7f961259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749aaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.591+0000 7f961259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749aaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46396/0 (socket says 172.21.15.27:46396) 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.591+0000 7f961259c640 1 -- 172.21.15.27:0/3032160993 learned_addr learned my addr 172.21.15.27:0/3032160993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.591+0000 7f9612d9d640 1 --2- 172.21.15.27:0/3032160993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.592+0000 7f9611d9b640 1 -- 172.21.15.27:0/3032160993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 msgr2=0x5584749abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.592+0000 7f9611d9b640 1 --2- 172.21.15.27:0/3032160993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.592+0000 7f9611d9b640 1 -- 172.21.15.27:0/3032160993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 msgr2=0x5584749aaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.592+0000 7f9611d9b640 1 --2- 172.21.15.27:0/3032160993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749aaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.592+0000 7f9611d9b640 1 -- 172.21.15.27:0/3032160993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55847489aa40 con 0x5584745e04d0 2024-04-25T13:50:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.593+0000 7f9611d9b640 1 --2- 172.21.15.27:0/3032160993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e04d0 0x5584749aafe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9608009780 tx=0x7f9608041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84a8ff5ed389806f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.593+0000 7f9611d9b640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95fc0afd00 0x7f95fc0b2210 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9614112580 tx=0x7f9608030bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.597+0000 7f9600ff9640 1 -- 172.21.15.27:0/3032160993 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9608037040 con 0x5584745e04d0 2024-04-25T13:50:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.597+0000 7f9600ff9640 1 -- 172.21.15.27:0/3032160993 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9608046070 con 0x5584745e04d0 2024-04-25T13:50:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.597+0000 7f9600ff9640 1 -- 172.21.15.27:0/3032160993 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96080326a0 con 0x5584745e04d0 2024-04-25T13:50:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.599+0000 7f961f79c740 1 -- 172.21.15.27:0/3032160993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e04d0 msgr2=0x5584749aafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.599+0000 7f961f79c740 1 --2- 172.21.15.27:0/3032160993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e04d0 0x5584749aafe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9608009780 tx=0x7f9608041040 comp rx=0 tx=0).stop 2024-04-25T13:50:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.599+0000 7f961f79c740 1 -- 172.21.15.27:0/3032160993 shutdown_connections 2024-04-25T13:50:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.599+0000 7f961f79c740 1 --2- 172.21.15.27:0/3032160993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.599+0000 7f961f79c740 1 --2- 172.21.15.27:0/3032160993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e04d0 0x5584749aafe0 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9608009780 tx=0x7f9608041040 comp rx=0 tx=0).stop 2024-04-25T13:50:09.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.599+0000 7f961f79c740 1 --2- 172.21.15.27:0/3032160993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749aaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.599+0000 7f961f79c740 1 -- 172.21.15.27:0/3032160993 >> 172.21.15.27:0/3032160993 conn(0x5584745fd4e0 msgr2=0x5584749a6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.605+0000 7f961f79c740 1 -- 172.21.15.27:0/3032160993 shutdown_connections 2024-04-25T13:50:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.606+0000 7f961f79c740 1 -- 172.21.15.27:0/3032160993 wait complete. 2024-04-25T13:50:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.606+0000 7f961f79c740 1 -- start start 2024-04-25T13:50:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.606+0000 7f961f79c740 1 Processor -- start 2024-04-25T13:50:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961f79c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e86b0 0x5584749c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961f79c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961f79c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745c7f60 0x5584749cced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961f79c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5584749ac6c0 con 0x558474609390 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961f79c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x558474896d10 con 0x5584745e86b0 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961f79c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5584749ac840 con 0x5584745c7f60 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961f79c740 10 client.0 _ll_register_callbacks cb 0x5584748c3210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f9612d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745c7f60 0x5584749cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e86b0 0x5584749c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e86b0 0x5584749c5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47912/0 (socket says 172.21.15.27:47912) 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f961259c640 1 -- 172.21.15.27:0/4285761090 learned_addr learned my addr 172.21.15.27:0/4285761090 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.607+0000 7f9611d9b640 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.608+0000 7f9612d9d640 1 -- 172.21.15.27:0/4285761090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e86b0 msgr2=0x5584749c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.608+0000 7f9612d9d640 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e86b0 0x5584749c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.608+0000 7f9612d9d640 1 -- 172.21.15.27:0/4285761090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 msgr2=0x5584749c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.608+0000 7f9612d9d640 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.608+0000 7f9612d9d640 1 -- 172.21.15.27:0/4285761090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584746eb7d0 con 0x5584745c7f60 2024-04-25T13:50:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.608+0000 7f9612d9d640 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745c7f60 0x5584749cced0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f960c005fd0 tx=0x7f960c00ff30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c33832e2b04bf5b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.609+0000 7f95f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.609+0000 7f95f9ffb640 1 -- 172.21.15.27:0/4285761090 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c00cbe0 con 0x5584745c7f60 2024-04-25T13:50:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.609+0000 7f961f79c740 1 -- 172.21.15.27:0/4285761090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9608046070 con 0x5584745c7f60 2024-04-25T13:50:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.609+0000 7f95f9ffb640 1 -- 172.21.15.27:0/4285761090 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f960c00cd80 con 0x5584745c7f60 2024-04-25T13:50:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.609+0000 7f95f9ffb640 1 -- 172.21.15.27:0/4285761090 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f960c003970 con 0x5584745c7f60 2024-04-25T13:50:09.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.612+0000 7f961f79c740 10 client.26956 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:09.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.612+0000 7f961f79c740 1 -- 172.21.15.27:0/4285761090 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5584746213f0 con 0x5584745c7f60 2024-04-25T13:50:09.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.613+0000 7f95f9ffb640 1 -- 172.21.15.27:0/4285761090 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9608046070 con 0x5584745c7f60 2024-04-25T13:50:09.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.613+0000 7f95f9ffb640 1 -- 172.21.15.27:0/4285761090 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f960c00b040 con 0x5584745c7f60 2024-04-25T13:50:09.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.613+0000 7f961f79c740 10 client.26956 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:09.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.613+0000 7f961f79c740 10 client.26956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:09.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.613+0000 7f961f79c740 20 client.26956 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.613+0000 7f961f79c740 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55847459d3a0 0x5584749b9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.613+0000 7f961f79c740 4 client.26956 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.613+0000 7f961f79c740 1 -- 172.21.15.27:0/4285761090 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55847491d990 con 0x55847459d3a0 2024-04-25T13:50:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.616+0000 7f961259c640 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55847459d3a0 0x5584749b9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.616+0000 7f95f9ffb640 10 client.26956 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.617+0000 7f961259c640 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55847459d3a0 0x5584749b9a00 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaaf74e41f8c38c5 server_cookie=8f3231001397886d in_seq=0 out_seq=0 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.620+0000 7f95f9ffb640 1 -- 172.21.15.27:0/4285761090 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55847491d990 con 0x55847459d3a0 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.620+0000 7f95f9ffb640 10 client.26956 handle_command_reply: tid=0 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.620+0000 7f961f79c740 1 client.26956 shutdown 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 -- 172.21.15.27:0/4285761090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745c7f60 msgr2=0x5584749cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745c7f60 0x5584749cced0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f960c005fd0 tx=0x7f960c00ff30 comp rx=0 tx=0).stop 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 -- 172.21.15.27:0/4285761090 shutdown_connections 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55847459d3a0 0x5584749b9a00 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745c7f60 0x5584749cced0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558474609390 0x5584749c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 --2- 172.21.15.27:0/4285761090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745e86b0 0x5584749c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 -- 172.21.15.27:0/4285761090 >> 172.21.15.27:0/4285761090 conn(0x5584745fd4e0 msgr2=0x5584749a9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f95f9ffb640 0 client.26956 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 -- 172.21.15.27:0/4285761090 shutdown_connections 2024-04-25T13:50:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 1 -- 172.21.15.27:0/4285761090 wait complete. 2024-04-25T13:50:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.622+0000 7f961f79c740 20 client.26956 trim_cache size 0 max 16384 2024-04-25T13:50:09.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.658+0000 7f961f79c740 1 Processor -- start 2024-04-25T13:50:09.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.658+0000 7f961f79c740 1 -- start start 2024-04-25T13:50:09.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.658+0000 7f961f79c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745c7f60 0x5584749a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.658+0000 7f961f79c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745e86b0 0x5584749a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f961f79c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558474609390 0x5584749cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f961259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745c7f60 0x5584749a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f961259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745c7f60 0x5584749a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47914/0 (socket says 172.21.15.27:47914) 2024-04-25T13:50:09.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f9611d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745e86b0 0x5584749a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f9611d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745e86b0 0x5584749a61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46418/0 (socket says 172.21.15.27:46418) 2024-04-25T13:50:09.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f961f79c740 1 -- 172.21.15.27:0/4006623042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f95f400c460 con 0x5584745e86b0 2024-04-25T13:50:09.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f961259c640 1 -- 172.21.15.27:0/4006623042 learned_addr learned my addr 172.21.15.27:0/4006623042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:09.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f961f79c740 1 -- 172.21.15.27:0/4006623042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f95f400ece0 con 0x5584745c7f60 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f961f79c740 1 -- 172.21.15.27:0/4006623042 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f95f400cad0 con 0x558474609390 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f9612d9d640 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558474609390 0x5584749cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f9611d9b640 1 -- 172.21.15.27:0/4006623042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558474609390 msgr2=0x5584749cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f9611d9b640 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558474609390 0x5584749cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f9611d9b640 1 -- 172.21.15.27:0/4006623042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745c7f60 msgr2=0x5584749a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f9611d9b640 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745c7f60 0x5584749a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f9611d9b640 1 -- 172.21.15.27:0/4006623042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584745e10f0 con 0x5584745e86b0 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.659+0000 7f961259c640 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745c7f60 0x5584749a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.660+0000 7f9611d9b640 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745e86b0 0x5584749a61e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x5584749ab6e0 tx=0x7f960802a380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed90b13fb6ade43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.660+0000 7f95fbfff640 1 -- 172.21.15.27:0/4006623042 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9608057070 con 0x5584745e86b0 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.660+0000 7f95fbfff640 1 -- 172.21.15.27:0/4006623042 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9608046070 con 0x5584745e86b0 2024-04-25T13:50:09.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.660+0000 7f95fbfff640 1 -- 172.21.15.27:0/4006623042 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9608037040 con 0x5584745e86b0 2024-04-25T13:50:09.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 -- 172.21.15.27:0/4006623042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745e86b0 msgr2=0x5584749a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745e86b0 0x5584749a61e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x5584749ab6e0 tx=0x7f960802a380 comp rx=0 tx=0).stop 2024-04-25T13:50:09.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 -- 172.21.15.27:0/4006623042 shutdown_connections 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558474609390 0x5584749cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745e86b0 0x5584749a61e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 --2- 172.21.15.27:0/4006623042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5584745c7f60 0x5584749a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 -- 172.21.15.27:0/4006623042 >> 172.21.15.27:0/4006623042 conn(0x5584745fd4e0 msgr2=0x5584749b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 -- 172.21.15.27:0/4006623042 shutdown_connections 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.661+0000 7f961f79c740 1 -- 172.21.15.27:0/4006623042 wait complete. 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 1 -- start start 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 1 Processor -- start 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745c7f60 0x5584749cac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558474609390 0x5584749cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f95f400d1a0 con 0x5584745c7f60 2024-04-25T13:50:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f95f400e610 con 0x558474609390 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f95f400d870 con 0x5584745e86b0 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.662+0000 7f961f79c740 10 client.0 _ll_register_callbacks cb 0x7f960c0099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9612d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558474609390 0x5584749cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f961259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745c7f60 0x5584749cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36076/0 (socket says 172.21.15.27:36076) 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 -- 172.21.15.27:0/1067708005 learned_addr learned my addr 172.21.15.27:0/1067708005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 -- 172.21.15.27:0/1067708005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558474609390 msgr2=0x5584749cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558474609390 0x5584749cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 -- 172.21.15.27:0/1067708005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745c7f60 msgr2=0x5584749cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745c7f60 0x5584749cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 -- 172.21.15.27:0/1067708005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584748d04b0 con 0x5584745e86b0 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f9611d9b640 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749cb190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9608000c00 tx=0x7f9608004ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2457a976dd0f4854 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:09.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.663+0000 7f95f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:09.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f95f9ffb640 1 -- 172.21.15.27:0/1067708005 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9608057070 con 0x5584745e86b0 2024-04-25T13:50:09.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f95f9ffb640 1 -- 172.21.15.27:0/1067708005 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9608032440 con 0x5584745e86b0 2024-04-25T13:50:09.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f961f79c740 1 -- 172.21.15.27:0/1067708005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9608046070 con 0x5584745e86b0 2024-04-25T13:50:09.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f95f9ffb640 1 -- 172.21.15.27:0/1067708005 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9608037040 con 0x5584745e86b0 2024-04-25T13:50:09.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f961f79c740 10 client.26962 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:09.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f961f79c740 1 -- 172.21.15.27:0/1067708005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5584749cd770 con 0x5584745e86b0 2024-04-25T13:50:09.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f95f9ffb640 1 -- 172.21.15.27:0/1067708005 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9608046070 con 0x5584745e86b0 2024-04-25T13:50:09.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f95f9ffb640 1 -- 172.21.15.27:0/1067708005 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f96080525d0 con 0x5584745e86b0 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f961f79c740 10 client.26962 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.664+0000 7f961f79c740 10 client.26962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.665+0000 7f961f79c740 20 client.26962 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.665+0000 7f961f79c740 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558474a4c660 0x558474a4ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.665+0000 7f961f79c740 4 client.26962 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.665+0000 7f961f79c740 1 -- 172.21.15.27:0/1067708005 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558474917800 con 0x558474a4c660 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.665+0000 7f961259c640 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558474a4c660 0x558474a4ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.665+0000 7f95f9ffb640 10 client.26962 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961259c640 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558474a4c660 0x558474a4ea80 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d29a7503eaafb86 server_cookie=e02d85d418a7e35e in_seq=0 out_seq=0 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f95f9ffb640 1 -- 172.21.15.27:0/1067708005 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558474917800 con 0x558474a4c660 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f95f9ffb640 10 client.26962 handle_command_reply: tid=0 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 client.26962 shutdown 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 -- 172.21.15.27:0/1067708005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 msgr2=0x5584749cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749cb190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9608000c00 tx=0x7f9608004ea0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 -- 172.21.15.27:0/1067708005 shutdown_connections 2024-04-25T13:50:09.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x558474a4c660 0x558474a4ea80 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558474609390 0x5584749cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f95f9ffb640 0 client.26962 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5584745e86b0 0x5584749cb190 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 --2- 172.21.15.27:0/1067708005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5584745c7f60 0x5584749cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.666+0000 7f961f79c740 1 -- 172.21.15.27:0/1067708005 >> 172.21.15.27:0/1067708005 conn(0x5584745fd4e0 msgr2=0x5584749b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.667+0000 7f961f79c740 1 -- 172.21.15.27:0/1067708005 shutdown_connections 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.667+0000 7f961f79c740 1 -- 172.21.15.27:0/1067708005 wait complete. 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.667+0000 7f961f79c740 20 client.26962 trim_cache size 0 max 16384 2024-04-25T13:50:09.670 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:50:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:50:09.671 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:50:09.671 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:50:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.667+0000 7f9618814640 1 -- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95fc0afd00 msgr2=0x7f95fc0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.667+0000 7f9618814640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95fc0afd00 0x7f95fc0b2210 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9614112580 tx=0x7f9608030bc0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.667+0000 7f9618814640 1 -- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9614101d90 msgr2=0x7f9614112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.667+0000 7f9618814640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9614101d90 0x7f9614112be0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f960400e9f0 tx=0x7f960400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.668+0000 7f9618814640 1 -- 172.21.15.27:0/3837701506 shutdown_connections 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.668+0000 7f9618814640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f95fc0afd00 0x7f95fc0b2210 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.668+0000 7f9618814640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9614107940 0x7f96141117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.668+0000 7f9618814640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96141027d0 0x7f9614111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.668+0000 7f9618814640 1 --2- 172.21.15.27:0/3837701506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9614101d90 0x7f9614112be0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.668+0000 7f9618814640 1 -- 172.21.15.27:0/3837701506 >> 172.21.15.27:0/3837701506 conn(0x7f96140fd380 msgr2=0x7f9614105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.668+0000 7f9618814640 1 -- 172.21.15.27:0/3837701506 shutdown_connections 2024-04-25T13:50:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.668+0000 7f9618814640 1 -- 172.21.15.27:0/3837701506 wait complete. 2024-04-25T13:50:09.684 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:50:09.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.851+0000 7ff719ffb640 1 -- 172.21.15.27:0/3645340203 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7100318a0 con 0x7ff71c103c00 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 -- 172.21.15.27:0/3645340203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c103c00 msgr2=0x7ff71c104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 --2- 172.21.15.27:0/3645340203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c103c00 0x7ff71c104010 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff710009900 tx=0x7ff71002a410 comp rx=0 tx=0).stop 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 -- 172.21.15.27:0/3645340203 shutdown_connections 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 --2- 172.21.15.27:0/3645340203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c108900 0x7ff71c10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 --2- 172.21.15.27:0/3645340203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c103c00 0x7ff71c104010 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 --2- 172.21.15.27:0/3645340203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff71c1029c0 0x7ff71c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 -- 172.21.15.27:0/3645340203 >> 172.21.15.27:0/3645340203 conn(0x7ff71c0fdbd0 msgr2=0x7ff71c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 -- 172.21.15.27:0/3645340203 shutdown_connections 2024-04-25T13:50:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.852+0000 7ff721896640 1 -- 172.21.15.27:0/3645340203 wait complete. 2024-04-25T13:50:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.853+0000 7ff721896640 1 Processor -- start 2024-04-25T13:50:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff721896640 1 -- start start 2024-04-25T13:50:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff721896640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff71c1029c0 0x7ff71c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff721896640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c108900 0x7ff71c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff721896640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c19d5b0 0x7ff71c19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff721896640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c10d420 con 0x7ff71c108900 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff721896640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c10d2a0 con 0x7ff71c19d5b0 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff721896640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c10d5a0 con 0x7ff71c1029c0 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff71affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff71c1029c0 0x7ff71c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff71affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff71c1029c0 0x7ff71c1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36092/0 (socket says 172.21.15.27:36092) 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff71affd640 1 -- 172.21.15.27:0/2859389229 learned_addr learned my addr 172.21.15.27:0/2859389229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.854+0000 7ff71b7fe640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c19d5b0 0x7ff71c19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.855+0000 7ff71a7fc640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c108900 0x7ff71c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.855+0000 7ff71affd640 1 -- 172.21.15.27:0/2859389229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c19d5b0 msgr2=0x7ff71c19d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.855+0000 7ff71affd640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c19d5b0 0x7ff71c19d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.855+0000 7ff71affd640 1 -- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c108900 msgr2=0x7ff71c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.855+0000 7ff71affd640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c108900 0x7ff71c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.855+0000 7ff71affd640 1 -- 172.21.15.27:0/2859389229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c19e0f0 con 0x7ff71c1029c0 2024-04-25T13:50:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.855+0000 7ff71affd640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff71c1029c0 0x7ff71c1980f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00e9f0 tx=0x7ff70c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.855+0000 7ff720894640 1 -- 172.21.15.27:0/2859389229 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c00ccc0 con 0x7ff71c1029c0 2024-04-25T13:50:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.856+0000 7ff720894640 1 -- 172.21.15.27:0/2859389229 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff70c0035f0 con 0x7ff71c1029c0 2024-04-25T13:50:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.856+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71c19e3e0 con 0x7ff71c1029c0 2024-04-25T13:50:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.856+0000 7ff720894640 1 -- 172.21.15.27:0/2859389229 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70c00ce60 con 0x7ff71c1029c0 2024-04-25T13:50:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.856+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff71c19e980 con 0x7ff71c1029c0 2024-04-25T13:50:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.857+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff71c19f400 con 0x7ff71c1029c0 2024-04-25T13:50:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.859+0000 7ff720894640 1 -- 172.21.15.27:0/2859389229 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff70c003aa0 con 0x7ff71c1029c0 2024-04-25T13:50:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.860+0000 7ff720894640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7040afb70 0x7ff7040b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.860+0000 7ff720894640 1 -- 172.21.15.27:0/2859389229 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff70c0defd0 con 0x7ff71c1029c0 2024-04-25T13:50:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.860+0000 7ff71a7fc640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7040afb70 0x7ff7040b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.862+0000 7ff71a7fc640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7040afb70 0x7ff7040b2080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff710005ab0 tx=0x7ff7100314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:09.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:09.866+0000 7ff720894640 1 -- 172.21.15.27:0/2859389229 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff70c076ff0 con 0x7ff71c1029c0 2024-04-25T13:50:10.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.283+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff71c107180 con 0x7ff71c1029c0 2024-04-25T13:50:10.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.284+0000 7ff720894640 1 -- 172.21.15.27:0/2859389229 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7ff70c0a7b80 con 0x7ff71c1029c0 2024-04-25T13:50:10.284 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:50:10.284 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:50:10.285 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:50:10.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7040afb70 msgr2=0x7ff7040b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7040afb70 0x7ff7040b2080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff710005ab0 tx=0x7ff7100314a0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff71c1029c0 msgr2=0x7ff71c1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff71c1029c0 0x7ff71c1980f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00e9f0 tx=0x7ff70c00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 shutdown_connections 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff7040afb70 0x7ff7040b2080 secure :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff710005ab0 tx=0x7ff7100314a0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff71c19d5b0 0x7ff71c19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71c108900 0x7ff71c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 --2- 172.21.15.27:0/2859389229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff71c1029c0 0x7ff71c1980f0 secure :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00e9f0 tx=0x7ff70c00eeb0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 >> 172.21.15.27:0/2859389229 conn(0x7ff71c0fdbd0 msgr2=0x7ff71c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:10.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.287+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 shutdown_connections 2024-04-25T13:50:10.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.288+0000 7ff721896640 1 -- 172.21.15.27:0/2859389229 wait complete. 2024-04-25T13:50:10.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.465+0000 7f88b38ca640 1 -- 172.21.15.27:0/622646178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 msgr2=0x7f88ac1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.465+0000 7f88b38ca640 1 --2- 172.21.15.27:0/622646178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 0x7f88ac1036e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f88a8009900 tx=0x7f88a802a410 comp rx=0 tx=0).stop 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.466+0000 7f88b38ca640 1 -- 172.21.15.27:0/622646178 shutdown_connections 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.466+0000 7f88b38ca640 1 --2- 172.21.15.27:0/622646178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88ac108920 0x7f88ac10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.466+0000 7f88b38ca640 1 --2- 172.21.15.27:0/622646178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88ac103c20 0x7f88ac104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.466+0000 7f88b38ca640 1 --2- 172.21.15.27:0/622646178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 0x7f88ac1036e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.466+0000 7f88b38ca640 1 -- 172.21.15.27:0/622646178 >> 172.21.15.27:0/622646178 conn(0x7f88ac0fdbd0 msgr2=0x7f88ac100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.466+0000 7f88b38ca640 1 -- 172.21.15.27:0/622646178 shutdown_connections 2024-04-25T13:50:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.466+0000 7f88b38ca640 1 -- 172.21.15.27:0/622646178 wait complete. 2024-04-25T13:50:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b38ca640 1 Processor -- start 2024-04-25T13:50:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b38ca640 1 -- start start 2024-04-25T13:50:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b38ca640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 0x7f88ac19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b38ca640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88ac103c20 0x7f88ac19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b38ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88ac108920 0x7f88ac1a3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b38ca640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10e4e0 con 0x7f88ac1029e0 2024-04-25T13:50:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b38ca640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10e360 con 0x7f88ac108920 2024-04-25T13:50:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b38ca640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10e660 con 0x7f88ac103c20 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b1666640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 0x7f88ac19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b0e65640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88ac103c20 0x7f88ac19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b1666640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 0x7f88ac19c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46462/0 (socket says 172.21.15.27:46462) 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b1666640 1 -- 172.21.15.27:0/3022720847 learned_addr learned my addr 172.21.15.27:0/3022720847 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.467+0000 7f88b1e67640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88ac108920 0x7f88ac1a3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.468+0000 7f88b1666640 1 -- 172.21.15.27:0/3022720847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88ac103c20 msgr2=0x7f88ac19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.468+0000 7f88b1666640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88ac103c20 0x7f88ac19cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.468+0000 7f88b1666640 1 -- 172.21.15.27:0/3022720847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88ac108920 msgr2=0x7f88ac1a3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.468+0000 7f88b1666640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88ac108920 0x7f88ac1a3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.468+0000 7f88b1666640 1 -- 172.21.15.27:0/3022720847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ac1a4310 con 0x7f88ac1029e0 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.468+0000 7f88b0e65640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88ac103c20 0x7f88ac19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.469+0000 7f88b1666640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 0x7f88ac19c7d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f88a8006010 tx=0x7f88a8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.469+0000 7f88a27fc640 1 -- 172.21.15.27:0/3022720847 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a803f030 con 0x7f88ac1029e0 2024-04-25T13:50:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.469+0000 7f88a27fc640 1 -- 172.21.15.27:0/3022720847 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a8031910 con 0x7f88ac1029e0 2024-04-25T13:50:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.469+0000 7f88a27fc640 1 -- 172.21.15.27:0/3022720847 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8035070 con 0x7f88ac1029e0 2024-04-25T13:50:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.470+0000 7f88b38ca640 1 -- 172.21.15.27:0/3022720847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ac1a45a0 con 0x7f88ac1029e0 2024-04-25T13:50:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.470+0000 7f88b38ca640 1 -- 172.21.15.27:0/3022720847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ac1a4a40 con 0x7f88ac1029e0 2024-04-25T13:50:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88ba77b740 1 Processor -- start 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88ba77b740 1 -- start start 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88a27fc640 1 -- 172.21.15.27:0/3022720847 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f88a8042030 con 0x7f88ac1029e0 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e737810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e737d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 0x556e7e738a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88ba77b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556e7e69bd80 con 0x556e7e3746b0 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88ba77b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556e7e655a20 con 0x556e7e36c4d0 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88ba77b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556e7e697d40 con 0x556e7e395390 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e737810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e737810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36126/0 (socket says 172.21.15.27:36126) 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1e67640 1 -- 172.21.15.27:0/1272199374 learned_addr learned my addr 172.21.15.27:0/1272199374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b0e65640 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 0x556e7e738a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1666640 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e737d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1e67640 1 -- 172.21.15.27:0/1272199374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 msgr2=0x556e7e737d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:10.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1e67640 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e737d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1e67640 1 -- 172.21.15.27:0/1272199374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 msgr2=0x556e7e738a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:10.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1e67640 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 0x556e7e738a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.473+0000 7f88b1e67640 1 -- 172.21.15.27:0/1272199374 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e7e660c30 con 0x556e7e395390 2024-04-25T13:50:10.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.474+0000 7f88a27fc640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88980afb70 0x7f88980b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.474+0000 7f88b1e67640 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e737810 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009fd0 tx=0x7f88a40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60456854d87510eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:10.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.474+0000 7f88b1666640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88980afb70 0x7f88980b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.474+0000 7f88a27fc640 1 -- 172.21.15.27:0/3022720847 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f88a80feee0 con 0x7f88ac1029e0 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.474+0000 7f88977fe640 1 -- 172.21.15.27:0/1272199374 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4004d20 con 0x556e7e395390 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.474+0000 7f88977fe640 1 -- 172.21.15.27:0/1272199374 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a4004050 con 0x556e7e395390 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.474+0000 7f88977fe640 1 -- 172.21.15.27:0/1272199374 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4005280 con 0x556e7e395390 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 -- 172.21.15.27:0/1272199374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 msgr2=0x556e7e737810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e737810 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009fd0 tx=0x7f88a40118d0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88b1666640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88980afb70 0x7f88980b2080 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f88a80c34c0 tx=0x7f88a80b0040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 -- 172.21.15.27:0/1272199374 shutdown_connections 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 0x556e7e738a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e737d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 --2- 172.21.15.27:0/1272199374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e737810 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 -- 172.21.15.27:0/1272199374 >> 172.21.15.27:0/1272199374 conn(0x556e7e3894e0 msgr2=0x556e7e733020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 -- 172.21.15.27:0/1272199374 shutdown_connections 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.475+0000 7f88ba77b740 1 -- 172.21.15.27:0/1272199374 wait complete. 2024-04-25T13:50:10.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.476+0000 7f88ba77b740 1 -- start start 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.476+0000 7f88ba77b740 1 Processor -- start 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e749d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 0x556e7e74a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e751220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88ba77b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556e7e739530 con 0x556e7e3746b0 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88ba77b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556e7e622dc0 con 0x556e7e36c4d0 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88ba77b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556e7e7396b0 con 0x556e7e395390 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88ba77b740 10 client.0 _ll_register_callbacks cb 0x556e7e63e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e749d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e751220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e749d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47972/0 (socket says 172.21.15.27:47972) 2024-04-25T13:50:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b1e67640 1 -- 172.21.15.27:0/3924941565 learned_addr learned my addr 172.21.15.27:0/3924941565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b0e65640 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 0x556e7e74a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b1e67640 1 -- 172.21.15.27:0/3924941565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 msgr2=0x556e7e751220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b1e67640 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e751220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b1e67640 1 -- 172.21.15.27:0/3924941565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 msgr2=0x556e7e74a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.477+0000 7f88b1e67640 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 0x556e7e74a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.478+0000 7f88b1e67640 1 -- 172.21.15.27:0/3924941565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e7e3ad3f0 con 0x556e7e36c4d0 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.478+0000 7f88b1e67640 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e749d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f88a4005fd0 tx=0x7f88a4003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=364cd12e813ffdf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.478+0000 7f8894ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.478+0000 7f8894ff9640 1 -- 172.21.15.27:0/3924941565 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4004c40 con 0x556e7e36c4d0 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.478+0000 7f88ba77b740 1 -- 172.21.15.27:0/3924941565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88a4004050 con 0x556e7e36c4d0 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.478+0000 7f8894ff9640 1 -- 172.21.15.27:0/3924941565 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f88a4004de0 con 0x556e7e36c4d0 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.479+0000 7f8894ff9640 1 -- 172.21.15.27:0/3924941565 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f88a4005030 con 0x556e7e36c4d0 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.479+0000 7f88ba77b740 10 client.26933 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.479+0000 7f88ba77b740 1 -- 172.21.15.27:0/3924941565 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556e7e4777d0 con 0x556e7e36c4d0 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.479+0000 7f8894ff9640 1 -- 172.21.15.27:0/3924941565 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a4004050 con 0x556e7e36c4d0 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.479+0000 7f8894ff9640 1 -- 172.21.15.27:0/3924941565 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f88a400f6e0 con 0x556e7e36c4d0 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.480+0000 7f88ba77b740 10 client.26933 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.480+0000 7f88ba77b740 10 client.26933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.480+0000 7f88ba77b740 20 client.26933 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.480+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556e7e353f60 0x556e7e7577e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.480+0000 7f88ba77b740 4 client.26933 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.480+0000 7f88ba77b740 1 -- 172.21.15.27:0/3924941565 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e7e6a9990 con 0x556e7e353f60 2024-04-25T13:50:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.481+0000 7f88b0e65640 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556e7e353f60 0x556e7e7577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.481+0000 7f8894ff9640 10 client.26933 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.481+0000 7f88b0e65640 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556e7e353f60 0x556e7e7577e0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ff3f764741f4929 server_cookie=2772f2bd39718d65 in_seq=0 out_seq=0 2024-04-25T13:50:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.482+0000 7f8894ff9640 1 -- 172.21.15.27:0/3924941565 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556e7e6a9990 con 0x556e7e353f60 2024-04-25T13:50:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.482+0000 7f8894ff9640 10 client.26933 handle_command_reply: tid=0 2024-04-25T13:50:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.482+0000 7f88ba77b740 1 client.26933 shutdown 2024-04-25T13:50:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.482+0000 7f88ba77b740 1 -- 172.21.15.27:0/3924941565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 msgr2=0x556e7e749d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.482+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e749d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f88a4005fd0 tx=0x7f88a4003f50 comp rx=0 tx=0).stop 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 1 -- 172.21.15.27:0/3924941565 shutdown_connections 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556e7e353f60 0x556e7e7577e0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f8894ff9640 0 client.26933 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e395390 0x556e7e751220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e3746b0 0x556e7e74a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3924941565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e36c4d0 0x556e7e749d80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 1 -- 172.21.15.27:0/3924941565 >> 172.21.15.27:0/3924941565 conn(0x556e7e3894e0 msgr2=0x556e7e735f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 1 -- 172.21.15.27:0/3924941565 shutdown_connections 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 1 -- 172.21.15.27:0/3924941565 wait complete. 2024-04-25T13:50:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.483+0000 7f88ba77b740 20 client.26933 trim_cache size 0 max 16384 2024-04-25T13:50:10.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.503+0000 7f88ba77b740 1 Processor -- start 2024-04-25T13:50:10.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.503+0000 7f88ba77b740 1 -- start start 2024-04-25T13:50:10.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e353f60 0x556e7e7365b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e36c4d0 0x556e7e7511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e3746b0 0x556e7e749360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88ba77b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1a6fd0 con 0x556e7e36c4d0 2024-04-25T13:50:10.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88ba77b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1a9850 con 0x556e7e353f60 2024-04-25T13:50:10.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88ba77b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1a7640 con 0x556e7e3746b0 2024-04-25T13:50:10.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e353f60 0x556e7e7365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88b0e65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e36c4d0 0x556e7e7511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e353f60 0x556e7e7365b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:47988/0 (socket says 172.21.15.27:47988) 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88b1e67640 1 -- 172.21.15.27:0/1613273653 learned_addr learned my addr 172.21.15.27:0/1613273653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.504+0000 7f88b1e67640 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e3746b0 0x556e7e749360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f88b1e67640 1 -- 172.21.15.27:0/1613273653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e3746b0 msgr2=0x556e7e749360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f88b1e67640 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e3746b0 0x556e7e749360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f88b1e67640 1 -- 172.21.15.27:0/1613273653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e36c4d0 msgr2=0x556e7e7511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f88b1e67640 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e36c4d0 0x556e7e7511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f88b1e67640 1 -- 172.21.15.27:0/1613273653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e7e757d10 con 0x556e7e353f60 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f88b0e65640 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e36c4d0 0x556e7e7511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f88b1e67640 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e353f60 0x556e7e7365b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f88a4005d70 tx=0x7f88a40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b269646580e43da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f8896ffd640 1 -- 172.21.15.27:0/1613273653 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a400fcf0 con 0x556e7e353f60 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f8896ffd640 1 -- 172.21.15.27:0/1613273653 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a402d070 con 0x556e7e353f60 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.505+0000 7f8896ffd640 1 -- 172.21.15.27:0/1613273653 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4004cc0 con 0x556e7e353f60 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.506+0000 7f88ba77b740 1 -- 172.21.15.27:0/1613273653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e353f60 msgr2=0x556e7e7365b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.506+0000 7f88ba77b740 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e353f60 0x556e7e7365b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f88a4005d70 tx=0x7f88a40118d0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.506+0000 7f88ba77b740 1 -- 172.21.15.27:0/1613273653 shutdown_connections 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.506+0000 7f88ba77b740 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e3746b0 0x556e7e749360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.506+0000 7f88ba77b740 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e36c4d0 0x556e7e7511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.507+0000 7f88ba77b740 1 --2- 172.21.15.27:0/1613273653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e353f60 0x556e7e7365b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.507+0000 7f88ba77b740 1 -- 172.21.15.27:0/1613273653 >> 172.21.15.27:0/1613273653 conn(0x556e7e3894e0 msgr2=0x556e7e735ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:10.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.507+0000 7f88ba77b740 1 -- 172.21.15.27:0/1613273653 shutdown_connections 2024-04-25T13:50:10.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.507+0000 7f88ba77b740 1 -- 172.21.15.27:0/1613273653 wait complete. 2024-04-25T13:50:10.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.507+0000 7f88ba77b740 1 -- start start 2024-04-25T13:50:10.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.507+0000 7f88ba77b740 1 Processor -- start 2024-04-25T13:50:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.508+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e353f60 0x556e7e74e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e36c4d0 0x556e7e74e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88ba77b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e3746b0 0x556e7e74f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88ba77b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1a7d10 con 0x556e7e353f60 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88ba77b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1a9180 con 0x556e7e3746b0 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e353f60 0x556e7e74e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88b1e67640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e353f60 0x556e7e74e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46498/0 (socket says 172.21.15.27:46498) 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88b1e67640 1 -- 172.21.15.27:0/3930305895 learned_addr learned my addr 172.21.15.27:0/3930305895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac1a83e0 con 0x556e7e36c4d0 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.509+0000 7f88ba77b740 10 client.0 _ll_register_callbacks cb 0x7f889c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.510+0000 7f88b0e65640 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e36c4d0 0x556e7e74e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.510+0000 7f88b1e67640 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e3746b0 0x556e7e74f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.510+0000 7f88b0e65640 1 -- 172.21.15.27:0/3930305895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e3746b0 msgr2=0x556e7e74f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.510+0000 7f88b0e65640 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e3746b0 0x556e7e74f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.510+0000 7f88b0e65640 1 -- 172.21.15.27:0/3930305895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e353f60 msgr2=0x556e7e74e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.510+0000 7f88b0e65640 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e353f60 0x556e7e74e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.510+0000 7f88b0e65640 1 -- 172.21.15.27:0/3930305895 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e7e74fd00 con 0x556e7e36c4d0 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.510+0000 7f88b1e67640 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e353f60 0x556e7e74e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f88b0e65640 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e36c4d0 0x556e7e74e810 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x556e7e757ee0 tx=0x7f889c010ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13b842cffd741601 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f8894ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f8894ff9640 1 -- 172.21.15.27:0/3930305895 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c00fbe0 con 0x556e7e36c4d0 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88a402d070 con 0x556e7e36c4d0 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f8894ff9640 1 -- 172.21.15.27:0/3930305895 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f889c00fd80 con 0x556e7e36c4d0 2024-04-25T13:50:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f8894ff9640 1 -- 172.21.15.27:0/3930305895 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f889c0200f0 con 0x556e7e36c4d0 2024-04-25T13:50:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f88ba77b740 10 client.26986 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556e7e751050 con 0x556e7e36c4d0 2024-04-25T13:50:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.511+0000 7f8894ff9640 1 -- 172.21.15.27:0/3930305895 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a402d070 con 0x556e7e36c4d0 2024-04-25T13:50:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.512+0000 7f8894ff9640 1 -- 172.21.15.27:0/3930305895 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f889c02b5d0 con 0x556e7e36c4d0 2024-04-25T13:50:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.512+0000 7f88ba77b740 10 client.26986 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.512+0000 7f88ba77b740 10 client.26986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.512+0000 7f88ba77b740 20 client.26986 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.512+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556e7e395390 0x556e7e7c1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.512+0000 7f88ba77b740 4 client.26986 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.512+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e7e748e40 con 0x556e7e395390 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.512+0000 7f88b1e67640 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556e7e395390 0x556e7e7c1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f8894ff9640 10 client.26986 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88b1e67640 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556e7e395390 0x556e7e7c1ea0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3bd43402e22ec3d server_cookie=b74779fb0e177cbe in_seq=0 out_seq=0 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f8894ff9640 1 -- 172.21.15.27:0/3930305895 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e7e748e40 con 0x556e7e395390 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f8894ff9640 10 client.26986 handle_command_reply: tid=0 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 client.26986 shutdown 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e36c4d0 msgr2=0x556e7e74e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e36c4d0 0x556e7e74e810 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x556e7e757ee0 tx=0x7f889c010ea0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 shutdown_connections 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556e7e395390 0x556e7e7c1ea0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f8894ff9640 0 client.26986 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556e7e3746b0 0x556e7e74f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556e7e36c4d0 0x556e7e74e810 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 --2- 172.21.15.27:0/3930305895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556e7e353f60 0x556e7e74e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 >> 172.21.15.27:0/3930305895 conn(0x556e7e3894e0 msgr2=0x556e7e735ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 shutdown_connections 2024-04-25T13:50:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 1 -- 172.21.15.27:0/3930305895 wait complete. 2024-04-25T13:50:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.513+0000 7f88ba77b740 20 client.26986 trim_cache size 0 max 16384 2024-04-25T13:50:10.521 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:50:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 -- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88980afb70 msgr2=0x7f88980b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88980afb70 0x7f88980b2080 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f88a80c34c0 tx=0x7f88a80b0040 comp rx=0 tx=0).stop 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 -- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 msgr2=0x7f88ac19c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 0x7f88ac19c7d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f88a8006010 tx=0x7f88a8004270 comp rx=0 tx=0).stop 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 -- 172.21.15.27:0/3022720847 shutdown_connections 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88980afb70 0x7f88980b2080 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88ac108920 0x7f88ac1a3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f88ac103c20 0x7f88ac19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 --2- 172.21.15.27:0/3022720847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ac1029e0 0x7f88ac19c7d0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 -- 172.21.15.27:0/3022720847 >> 172.21.15.27:0/3022720847 conn(0x7f88ac0fdbd0 msgr2=0x7f88ac0ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 -- 172.21.15.27:0/3022720847 shutdown_connections 2024-04-25T13:50:10.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:10.514+0000 7f88b38ca640 1 -- 172.21.15.27:0/3022720847 wait complete. 2024-04-25T13:50:10.526 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:10 smithi027 ceph-mon[29140]: pgmap v627: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 MiB/s wr, 523 op/s 2024-04-25T13:50:10.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:10 smithi122 ceph-mon[32843]: pgmap v627: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 MiB/s wr, 523 op/s 2024-04-25T13:50:10.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:10 smithi045 ceph-mon[32731]: pgmap v627: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 MiB/s wr, 523 op/s 2024-04-25T13:50:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.060+0000 7f468cd0d640 1 -- 172.21.15.27:0/2232781328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 msgr2=0x7f4688110480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.060+0000 7f468cd0d640 1 --2- 172.21.15.27:0/2232781328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 0x7f4688110480 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f467c009900 tx=0x7f467c02a440 comp rx=0 tx=0).stop 2024-04-25T13:50:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.065+0000 7f468cd0d640 1 -- 172.21.15.27:0/2232781328 shutdown_connections 2024-04-25T13:50:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.065+0000 7f468cd0d640 1 --2- 172.21.15.27:0/2232781328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 0x7f4688110480 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.065+0000 7f468cd0d640 1 --2- 172.21.15.27:0/2232781328 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468810b350 0x7f468810b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.065+0000 7f468cd0d640 1 --2- 172.21.15.27:0/2232781328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468810a420 0x7f468810ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.065+0000 7f468cd0d640 1 -- 172.21.15.27:0/2232781328 >> 172.21.15.27:0/2232781328 conn(0x7f4688070540 msgr2=0x7f46880729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.070+0000 7f468cd0d640 1 -- 172.21.15.27:0/2232781328 shutdown_connections 2024-04-25T13:50:11.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.075+0000 7f468cd0d640 1 -- 172.21.15.27:0/2232781328 wait complete. 2024-04-25T13:50:11.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.078+0000 7f468cd0d640 1 Processor -- start 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.082+0000 7f468cd0d640 1 -- start start 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.082+0000 7f468cd0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468810a420 0x7f46881a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.082+0000 7f468cd0d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468810b350 0x7f46881a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.082+0000 7f468cd0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 0x7f46881a7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.082+0000 7f468cd0d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4688113d00 con 0x7f4688110070 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.082+0000 7f468cd0d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4688113b80 con 0x7f468810a420 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.082+0000 7f468cd0d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4688113e80 con 0x7f468810b350 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.083+0000 7f468659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468810a420 0x7f46881a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.084+0000 7f4686d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 0x7f46881a7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.084+0000 7f4686d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 0x7f46881a7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46520/0 (socket says 172.21.15.27:46520) 2024-04-25T13:50:11.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.084+0000 7f4686d9d640 1 -- 172.21.15.27:0/4021218792 learned_addr learned my addr 172.21.15.27:0/4021218792 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.085+0000 7f4686d9d640 1 -- 172.21.15.27:0/4021218792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468810b350 msgr2=0x7f46881a0cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:50:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.085+0000 7f4686d9d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468810b350 0x7f46881a0cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.085+0000 7f4686d9d640 1 -- 172.21.15.27:0/4021218792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468810a420 msgr2=0x7f46881a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.085+0000 7f4686d9d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468810a420 0x7f46881a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.085+0000 7f4686d9d640 1 -- 172.21.15.27:0/4021218792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46881a82d0 con 0x7f4688110070 2024-04-25T13:50:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.085+0000 7f4686d9d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 0x7f46881a7c60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f467c0317d0 tx=0x7f467c004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.087+0000 7f46777fe640 1 -- 172.21.15.27:0/4021218792 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c040020 con 0x7f4688110070 2024-04-25T13:50:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.087+0000 7f46777fe640 1 -- 172.21.15.27:0/4021218792 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f467c0035f0 con 0x7f4688110070 2024-04-25T13:50:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.087+0000 7f46777fe640 1 -- 172.21.15.27:0/4021218792 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c035070 con 0x7f4688110070 2024-04-25T13:50:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.089+0000 7f468cd0d640 1 -- 172.21.15.27:0/4021218792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46881a8560 con 0x7f4688110070 2024-04-25T13:50:11.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.089+0000 7f468cd0d640 1 -- 172.21.15.27:0/4021218792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46881a8a00 con 0x7f4688110070 2024-04-25T13:50:11.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.092+0000 7f4693bc3740 1 Processor -- start 2024-04-25T13:50:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.094+0000 7f4693bc3740 1 -- start start 2024-04-25T13:50:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.095+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d064970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.095+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d064ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.095+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d06bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.095+0000 7f4693bc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56479cfca460 con 0x56479ccc41b0 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.095+0000 7f4693bc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56479cf84590 con 0x56479cca34d0 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.095+0000 7f4693bc3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56479cfc6420 con 0x56479cc9b2f0 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.095+0000 7f46777fe640 1 -- 172.21.15.27:0/4021218792 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f467c043020 con 0x7f4688110070 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.096+0000 7f46777fe640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afcf0 0x7f46700b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.097+0000 7f468659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d064ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.097+0000 7f4686d9d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afcf0 0x7f46700b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.097+0000 7f46777fe640 1 -- 172.21.15.27:0/4021218792 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f467c0ffb80 con 0x7f4688110070 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.098+0000 7f4685d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d064970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.098+0000 7f4685d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d064970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46536/0 (socket says 172.21.15.27:46536) 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.098+0000 7f4685d9b640 1 -- 172.21.15.27:0/193308427 learned_addr learned my addr 172.21.15.27:0/193308427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.098+0000 7f4685d9b640 1 --2- 172.21.15.27:0/193308427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d06bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.099+0000 7f4686d9d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afcf0 0x7f46700b2200 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f467c0c7d30 tx=0x7f467c02a440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.100+0000 7f4685d9b640 1 -- 172.21.15.27:0/193308427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 msgr2=0x56479d064ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.100+0000 7f4685d9b640 1 --2- 172.21.15.27:0/193308427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d064ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.100+0000 7f4685d9b640 1 -- 172.21.15.27:0/193308427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 msgr2=0x56479d064970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.100+0000 7f4685d9b640 1 --2- 172.21.15.27:0/193308427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d064970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.100+0000 7f4685d9b640 1 -- 172.21.15.27:0/193308427 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56479cf910d0 con 0x56479cca34d0 2024-04-25T13:50:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.101+0000 7f4685d9b640 1 --2- 172.21.15.27:0/193308427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d06bdf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x56479d065b40 tx=0x7f4678004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd49b3d8ae597b34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.101+0000 7f4674ff9640 1 -- 172.21.15.27:0/193308427 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678005020 con 0x56479cca34d0 2024-04-25T13:50:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.102+0000 7f4674ff9640 1 -- 172.21.15.27:0/193308427 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46780047a0 con 0x56479cca34d0 2024-04-25T13:50:11.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.103+0000 7f4693bc3740 1 -- 172.21.15.27:0/193308427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 msgr2=0x56479d06bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.103+0000 7f4693bc3740 1 --2- 172.21.15.27:0/193308427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d06bdf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x56479d065b40 tx=0x7f4678004730 comp rx=0 tx=0).stop 2024-04-25T13:50:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.105+0000 7f4693bc3740 1 -- 172.21.15.27:0/193308427 shutdown_connections 2024-04-25T13:50:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.105+0000 7f4693bc3740 1 --2- 172.21.15.27:0/193308427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d06bdf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.105+0000 7f4693bc3740 1 --2- 172.21.15.27:0/193308427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d064ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.105+0000 7f4693bc3740 1 --2- 172.21.15.27:0/193308427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d064970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.105+0000 7f4693bc3740 1 -- 172.21.15.27:0/193308427 >> 172.21.15.27:0/193308427 conn(0x56479ccb8300 msgr2=0x56479d060300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.111+0000 7f4693bc3740 1 -- 172.21.15.27:0/193308427 shutdown_connections 2024-04-25T13:50:11.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.113+0000 7f4693bc3740 1 -- 172.21.15.27:0/193308427 wait complete. 2024-04-25T13:50:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.113+0000 7f4693bc3740 1 -- start start 2024-04-25T13:50:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.113+0000 7f4693bc3740 1 Processor -- start 2024-04-25T13:50:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cc9b2f0 0x56479d06e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cca34d0 0x56479d06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.113 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d075af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4693bc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56479d06c890 con 0x56479ccc41b0 2024-04-25T13:50:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4693bc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56479cf51b60 con 0x56479cc9b2f0 2024-04-25T13:50:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4693bc3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56479d06ca10 con 0x56479cca34d0 2024-04-25T13:50:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4693bc3740 10 client.0 _ll_register_callbacks cb 0x56479cf762b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:11.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f468659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cca34d0 0x56479d06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f468659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cca34d0 0x56479d06eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36208/0 (socket says 172.21.15.27:36208) 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f468659c640 1 -- 172.21.15.27:0/3672834503 learned_addr learned my addr 172.21.15.27:0/3672834503 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4685d9b640 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cc9b2f0 0x56479d06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.114+0000 7f4685d9b640 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d075af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.115+0000 7f468659c640 1 -- 172.21.15.27:0/3672834503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cc9b2f0 msgr2=0x56479d06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.115+0000 7f468659c640 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cc9b2f0 0x56479d06e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.115+0000 7f468659c640 1 -- 172.21.15.27:0/3672834503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 msgr2=0x56479d075af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.115+0000 7f468659c640 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d075af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.115+0000 7f468659c640 1 -- 172.21.15.27:0/3672834503 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56479ccdc210 con 0x56479cca34d0 2024-04-25T13:50:11.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.115+0000 7f468659c640 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cca34d0 0x56479d06eb90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4680021a80 tx=0x7f4680021f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=635a6a6bfc99b560 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.115+0000 7f466dffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:11.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.116+0000 7f466dffb640 1 -- 172.21.15.27:0/3672834503 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680020c20 con 0x56479cca34d0 2024-04-25T13:50:11.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.116+0000 7f466dffb640 1 -- 172.21.15.27:0/3672834503 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4680020dc0 con 0x56479cca34d0 2024-04-25T13:50:11.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.116+0000 7f4693bc3740 1 -- 172.21.15.27:0/3672834503 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46780047a0 con 0x56479cca34d0 2024-04-25T13:50:11.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f466dffb640 1 -- 172.21.15.27:0/3672834503 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4680004b20 con 0x56479cca34d0 2024-04-25T13:50:11.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f4693bc3740 10 client.26995 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:11.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f4693bc3740 1 -- 172.21.15.27:0/3672834503 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56479cda8620 con 0x56479cca34d0 2024-04-25T13:50:11.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f466dffb640 1 -- 172.21.15.27:0/3672834503 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46780047a0 con 0x56479cca34d0 2024-04-25T13:50:11.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f466dffb640 1 -- 172.21.15.27:0/3672834503 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f468001f040 con 0x56479cca34d0 2024-04-25T13:50:11.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f4693bc3740 10 client.26995 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f4693bc3740 10 client.26995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f4693bc3740 20 client.26995 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56479cc82d80 0x56479d07c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f4693bc3740 4 client.26995 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.117+0000 7f4693bc3740 1 -- 172.21.15.27:0/3672834503 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56479cfd8070 con 0x56479cc82d80 2024-04-25T13:50:11.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.118+0000 7f4685d9b640 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56479cc82d80 0x56479d07c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.118+0000 7f466dffb640 10 client.26995 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.118+0000 7f4685d9b640 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56479cc82d80 0x56479d07c430 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee7fe31d623385ce server_cookie=21f5656c63d63b59 in_seq=0 out_seq=0 2024-04-25T13:50:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.119+0000 7f466dffb640 1 -- 172.21.15.27:0/3672834503 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56479cfd8070 con 0x56479cc82d80 2024-04-25T13:50:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.119+0000 7f466dffb640 10 client.26995 handle_command_reply: tid=0 2024-04-25T13:50:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.119+0000 7f4693bc3740 1 client.26995 shutdown 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.119+0000 7f4693bc3740 1 -- 172.21.15.27:0/3672834503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cca34d0 msgr2=0x56479d06eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.119+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cca34d0 0x56479d06eb90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4680021a80 tx=0x7f4680021f40 comp rx=0 tx=0).stop 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 1 -- 172.21.15.27:0/3672834503 shutdown_connections 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56479cc82d80 0x56479d07c430 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d075af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cca34d0 0x56479d06eb90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f466dffb640 0 client.26995 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3672834503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cc9b2f0 0x56479d06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 1 -- 172.21.15.27:0/3672834503 >> 172.21.15.27:0/3672834503 conn(0x56479ccb8300 msgr2=0x56479d063210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 1 -- 172.21.15.27:0/3672834503 shutdown_connections 2024-04-25T13:50:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 1 -- 172.21.15.27:0/3672834503 wait complete. 2024-04-25T13:50:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.120+0000 7f4693bc3740 20 client.26995 trim_cache size 0 max 16384 2024-04-25T13:50:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4693bc3740 1 Processor -- start 2024-04-25T13:50:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4693bc3740 1 -- start start 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d068a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d068f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d075a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4693bc3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f46881af760 con 0x56479ccc41b0 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4693bc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46881b1fe0 con 0x56479cca34d0 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4693bc3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f46881afdd0 con 0x56479cc9b2f0 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4685d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4685d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d068a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36218/0 (socket says 172.21.15.27:36218) 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f468659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4685d9b640 1 -- 172.21.15.27:0/3643616596 learned_addr learned my addr 172.21.15.27:0/3643616596 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.143+0000 7f4685d9b640 1 --2- 172.21.15.27:0/3643616596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f4685d9b640 1 -- 172.21.15.27:0/3643616596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 msgr2=0x56479d068f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f4685d9b640 1 --2- 172.21.15.27:0/3643616596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d068f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f4685d9b640 1 -- 172.21.15.27:0/3643616596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 msgr2=0x56479d075a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f4685d9b640 1 --2- 172.21.15.27:0/3643616596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d075a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f4685d9b640 1 -- 172.21.15.27:0/3643616596 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56479d075f50 con 0x56479cc9b2f0 2024-04-25T13:50:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f4685d9b640 1 --2- 172.21.15.27:0/3643616596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d068a30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4678006010 tx=0x7f467801f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e44b40567fd47fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f466ffff640 1 -- 172.21.15.27:0/3643616596 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678026040 con 0x56479cc9b2f0 2024-04-25T13:50:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f466ffff640 1 -- 172.21.15.27:0/3643616596 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4678004850 con 0x56479cc9b2f0 2024-04-25T13:50:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.144+0000 7f466ffff640 1 -- 172.21.15.27:0/3643616596 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46780055b0 con 0x56479cc9b2f0 2024-04-25T13:50:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 -- 172.21.15.27:0/3643616596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 msgr2=0x56479d068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3643616596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d068a30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4678006010 tx=0x7f467801f040 comp rx=0 tx=0).stop 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 -- 172.21.15.27:0/3643616596 shutdown_connections 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3643616596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d075a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3643616596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 --2- 172.21.15.27:0/3643616596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d068a30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 -- 172.21.15.27:0/3643616596 >> 172.21.15.27:0/3643616596 conn(0x56479ccb8300 msgr2=0x56479d0648f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 -- 172.21.15.27:0/3643616596 shutdown_connections 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 -- 172.21.15.27:0/3643616596 wait complete. 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.146+0000 7f4693bc3740 1 -- start start 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 1 Processor -- start 2024-04-25T13:50:11.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d072d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d074200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d072d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d072d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36232/0 (socket says 172.21.15.27:36232) 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 -- 172.21.15.27:0/1902810230 learned_addr learned my addr 172.21.15.27:0/1902810230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f46881b04a0 con 0x56479ccc41b0 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f468659c640 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46881b1910 con 0x56479cca34d0 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f46881b0b70 con 0x56479cc9b2f0 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d074200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 10 client.0 _ll_register_callbacks cb 0x7f468001a530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:11.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 -- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 msgr2=0x56479d0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d0732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 -- 172.21.15.27:0/1902810230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 msgr2=0x56479d074200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d074200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4685d9b640 1 -- 172.21.15.27:0/1902810230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56479d0747c0 con 0x56479cc9b2f0 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.147+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56479cfd1ee0 con 0x56479cc9b2f0 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f468659c640 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f4685d9b640 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d072d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x56479d076990 tx=0x7f4678004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32287abf3683eef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f466dffb640 10 client.27001 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f466dffb640 1 -- 172.21.15.27:0/1902810230 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46780107d0 con 0x56479cc9b2f0 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f466dffb640 1 -- 172.21.15.27:0/1902810230 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4678010970 con 0x56479cc9b2f0 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f466dffb640 1 -- 172.21.15.27:0/1902810230 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4678010bb0 con 0x56479cc9b2f0 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f4693bc3740 10 client.27001 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:11.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x56479d0759a0 con 0x56479cc9b2f0 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f466dffb640 1 -- 172.21.15.27:0/1902810230 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f467802f480 con 0x56479cc9b2f0 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f4693bc3740 10 client.27001 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f4693bc3740 10 client.27001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.148+0000 7f4693bc3740 20 client.27001 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.149+0000 7f4693bc3740 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56479cc581c0 0x56479d0e68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.149+0000 7f4693bc3740 4 client.27001 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.149+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56479cf4d5f0 con 0x56479cc581c0 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.149+0000 7f468659c640 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56479cc581c0 0x56479d0e68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.149+0000 7f468659c640 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56479cc581c0 0x56479d0e68b0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=439526415bb19db5 server_cookie=e3bbfc2206600390 in_seq=0 out_seq=0 2024-04-25T13:50:11.153 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.149+0000 7f466dffb640 10 client.27001 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f466dffb640 1 -- 172.21.15.27:0/1902810230 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56479cf4d5f0 con 0x56479cc581c0 2024-04-25T13:50:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f466dffb640 10 client.27001 handle_command_reply: tid=0 2024-04-25T13:50:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 client.27001 shutdown 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 msgr2=0x56479d072d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d072d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x56479d076990 tx=0x7f4678004730 comp rx=0 tx=0).stop 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 shutdown_connections 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56479cc581c0 0x56479d0e68b0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56479ccc41b0 0x56479d074200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f466dffb640 0 client.27001 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56479cca34d0 0x56479d0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 --2- 172.21.15.27:0/1902810230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56479cc9b2f0 0x56479d072d90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.176+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 >> 172.21.15.27:0/1902810230 conn(0x56479ccb8300 msgr2=0x56479d0648f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.177 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 shutdown_connections 2024-04-25T13:50:11.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f4693bc3740 1 -- 172.21.15.27:0/1902810230 wait complete. 2024-04-25T13:50:11.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f4693bc3740 20 client.27001 trim_cache size 0 max 16384 2024-04-25T13:50:11.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:50:11.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:50:11.178 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:50:11.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:50:11.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 -- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afcf0 msgr2=0x7f46700b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afcf0 0x7f46700b2200 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f467c0c7d30 tx=0x7f467c02a440 comp rx=0 tx=0).stop 2024-04-25T13:50:11.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 -- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 msgr2=0x7f46881a7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 0x7f46881a7c60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f467c0317d0 tx=0x7f467c004e80 comp rx=0 tx=0).stop 2024-04-25T13:50:11.179 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 -- 172.21.15.27:0/4021218792 shutdown_connections 2024-04-25T13:50:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f46700afcf0 0x7f46700b2200 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4688110070 0x7f46881a7c60 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f468810b350 0x7f46881a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 --2- 172.21.15.27:0/4021218792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f468810a420 0x7f46881a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.177+0000 7f468cd0d640 1 -- 172.21.15.27:0/4021218792 >> 172.21.15.27:0/4021218792 conn(0x7f4688070540 msgr2=0x7f468810e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.178+0000 7f468cd0d640 1 -- 172.21.15.27:0/4021218792 shutdown_connections 2024-04-25T13:50:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:11.178+0000 7f468cd0d640 1 -- 172.21.15.27:0/4021218792 wait complete. 2024-04-25T13:50:11.528 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:50:11.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:50:11.529 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:50:11.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:11 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2859389229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:50:11.560 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:11 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2859389229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:50:11.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:11 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2859389229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:50:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.710+0000 7fc357620640 1 -- 172.21.15.27:0/2931687936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3501029e0 msgr2=0x7fc3501036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.710+0000 7fc357620640 1 --2- 172.21.15.27:0/2931687936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3501029e0 0x7fc3501036e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc34c009900 tx=0x7fc34c02a410 comp rx=0 tx=0).stop 2024-04-25T13:50:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.711+0000 7fc357620640 1 -- 172.21.15.27:0/2931687936 shutdown_connections 2024-04-25T13:50:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.711+0000 7fc357620640 1 --2- 172.21.15.27:0/2931687936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc350108920 0x7fc35010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.711+0000 7fc357620640 1 --2- 172.21.15.27:0/2931687936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc350103c20 0x7fc350104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.711+0000 7fc357620640 1 --2- 172.21.15.27:0/2931687936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3501029e0 0x7fc3501036e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.711+0000 7fc357620640 1 -- 172.21.15.27:0/2931687936 >> 172.21.15.27:0/2931687936 conn(0x7fc3500fdbd0 msgr2=0x7fc350100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.711+0000 7fc357620640 1 -- 172.21.15.27:0/2931687936 shutdown_connections 2024-04-25T13:50:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.711+0000 7fc357620640 1 -- 172.21.15.27:0/2931687936 wait complete. 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc357620640 1 Processor -- start 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc357620640 1 -- start start 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc357620640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3501029e0 0x7fc35019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc357620640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc350103c20 0x7fc35019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc357620640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc350108920 0x7fc3501a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc357620640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc35010e4e0 con 0x7fc350108920 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc357620640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc35010e360 con 0x7fc3501029e0 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc357620640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc35010e660 con 0x7fc350103c20 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc3553bc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3501029e0 0x7fc35019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc3553bc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3501029e0 0x7fc35019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48072/0 (socket says 172.21.15.27:48072) 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc354bbb640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc350103c20 0x7fc35019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc3553bc640 1 -- 172.21.15.27:0/787883410 learned_addr learned my addr 172.21.15.27:0/787883410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.712+0000 7fc355bbd640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc350108920 0x7fc3501a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc354bbb640 1 -- 172.21.15.27:0/787883410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3501029e0 msgr2=0x7fc35019c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc354bbb640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3501029e0 0x7fc35019c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc354bbb640 1 -- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc350108920 msgr2=0x7fc3501a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc354bbb640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc350108920 0x7fc3501a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc354bbb640 1 -- 172.21.15.27:0/787883410 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3501a4000 con 0x7fc350103c20 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc354bbb640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc350103c20 0x7fc35019ca00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc34000e9f0 tx=0x7fc34000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc3467fc640 1 -- 172.21.15.27:0/787883410 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34000ccc0 con 0x7fc350103c20 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc357620640 1 -- 172.21.15.27:0/787883410 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3501a42f0 con 0x7fc350103c20 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc3467fc640 1 -- 172.21.15.27:0/787883410 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3400108e0 con 0x7fc350103c20 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc3467fc640 1 -- 172.21.15.27:0/787883410 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340018880 con 0x7fc350103c20 2024-04-25T13:50:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.713+0000 7fc357620640 1 -- 172.21.15.27:0/787883410 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3501a4890 con 0x7fc350103c20 2024-04-25T13:50:11.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.717+0000 7fc3467fc640 1 -- 172.21.15.27:0/787883410 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc340010430 con 0x7fc350103c20 2024-04-25T13:50:11.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.718+0000 7fc3467fc640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc33c0afb70 0x7fc33c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.718+0000 7fc35e5ac740 1 Processor -- start 2024-04-25T13:50:11.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc35e5ac740 1 -- start start 2024-04-25T13:50:11.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc3467fc640 1 -- 172.21.15.27:0/787883410 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc3400def00 con 0x7fc350103c20 2024-04-25T13:50:11.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bcdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bd5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5595c4b2ab00 con 0x5595c4824410 2024-04-25T13:50:11.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5595c4ac6920 con 0x5595c47fb550 2024-04-25T13:50:11.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5595c4b26ac0 con 0x5595c4803730 2024-04-25T13:50:11.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.719+0000 7fc3553bc640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc33c0afb70 0x7fc33c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48088/0 (socket says 172.21.15.27:48088) 2024-04-25T13:50:11.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 -- 172.21.15.27:0/944256080 learned_addr learned my addr 172.21.15.27:0/944256080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc3553bc640 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bcdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 -- 172.21.15.27:0/944256080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 msgr2=0x5595c4bd5130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bd5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 -- 172.21.15.27:0/944256080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 msgr2=0x5595c4bcdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bcdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 -- 172.21.15.27:0/944256080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595c49047e0 con 0x5595c47fb550 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc3553bc640 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bcdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.720+0000 7fc354bbb640 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bce230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc3400a2d60 tx=0x7fc34000ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87a13b57b5fb4475 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.721+0000 7fc33b7fe640 1 -- 172.21.15.27:0/944256080 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3400b0210 con 0x5595c47fb550 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.721+0000 7fc33b7fe640 1 -- 172.21.15.27:0/944256080 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc340010d90 con 0x5595c47fb550 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.721+0000 7fc33b7fe640 1 -- 172.21.15.27:0/944256080 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3400aed20 con 0x5595c47fb550 2024-04-25T13:50:11.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 -- 172.21.15.27:0/944256080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 msgr2=0x5595c4bce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bce230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc3400a2d60 tx=0x7fc34000ce60 comp rx=0 tx=0).stop 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc3553bc640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc33c0afb70 0x7fc33c0b2080 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc34c031500 tx=0x7fc34c009560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 -- 172.21.15.27:0/944256080 shutdown_connections 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bd5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bce230 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/944256080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bcdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 -- 172.21.15.27:0/944256080 >> 172.21.15.27:0/944256080 conn(0x5595c4818560 msgr2=0x5595c4bc9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 -- 172.21.15.27:0/944256080 shutdown_connections 2024-04-25T13:50:11.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 -- 172.21.15.27:0/944256080 wait complete. 2024-04-25T13:50:11.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 -- start start 2024-04-25T13:50:11.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.722+0000 7fc35e5ac740 1 Processor -- start 2024-04-25T13:50:11.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.723+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bd7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.723+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bd8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.723+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bdf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.723+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5595c4bd5bd0 con 0x5595c4824410 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.723+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5595c4ac45d0 con 0x5595c47fb550 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.723+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5595c4bd5d50 con 0x5595c4803730 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.723+0000 7fc35e5ac740 10 client.0 _ll_register_callbacks cb 0x5595c4ab2080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.724+0000 7fc355bbd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bdf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.724+0000 7fc355bbd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bdf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46590/0 (socket says 172.21.15.27:46590) 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.724+0000 7fc355bbd640 1 -- 172.21.15.27:0/483764008 learned_addr learned my addr 172.21.15.27:0/483764008 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.724+0000 7fc354bbb640 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bd8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.724+0000 7fc3553bc640 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bd7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc354bbb640 1 -- 172.21.15.27:0/483764008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 msgr2=0x5595c4bd7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc354bbb640 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bd7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc354bbb640 1 -- 172.21.15.27:0/483764008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 msgr2=0x5595c4bdf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc354bbb640 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bdf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc354bbb640 1 -- 172.21.15.27:0/483764008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595c4964f70 con 0x5595c4803730 2024-04-25T13:50:11.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc354bbb640 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bd8160 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc3400a7160 tx=0x7fc3400a75e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f856656dfee46caa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc338ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc338ff9640 1 -- 172.21.15.27:0/483764008 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3400b0210 con 0x5595c4803730 2024-04-25T13:50:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.725+0000 7fc338ff9640 1 -- 172.21.15.27:0/483764008 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc3400aecf0 con 0x5595c4803730 2024-04-25T13:50:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.726+0000 7fc35e5ac740 1 -- 172.21.15.27:0/483764008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc340010d90 con 0x5595c4803730 2024-04-25T13:50:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.726+0000 7fc338ff9640 1 -- 172.21.15.27:0/483764008 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc3400a2dd0 con 0x5595c4803730 2024-04-25T13:50:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.726+0000 7fc35e5ac740 10 client.27010 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.726+0000 7fc35e5ac740 1 -- 172.21.15.27:0/483764008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5595c47fc170 con 0x5595c4803730 2024-04-25T13:50:11.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.727+0000 7fc338ff9640 1 -- 172.21.15.27:0/483764008 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc340010d90 con 0x5595c4803730 2024-04-25T13:50:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.728+0000 7fc338ff9640 1 -- 172.21.15.27:0/483764008 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc340082aa0 con 0x5595c4803730 2024-04-25T13:50:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.728+0000 7fc35e5ac740 10 client.27010 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.728+0000 7fc35e5ac740 10 client.27010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.728+0000 7fc35e5ac740 20 client.27010 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.728+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5595c47e31f0 0x5595c4be5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.728+0000 7fc35e5ac740 4 client.27010 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:11.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.728+0000 7fc35e5ac740 1 -- 172.21.15.27:0/483764008 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595c4b38710 con 0x5595c47e31f0 2024-04-25T13:50:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.729+0000 7fc3553bc640 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5595c47e31f0 0x5595c4be5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.729+0000 7fc338ff9640 10 client.27010 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:11.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.729+0000 7fc3553bc640 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5595c47e31f0 0x5595c4be5610 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3079d25fd9dea4c6 server_cookie=94138edecfc5cce5 in_seq=0 out_seq=0 2024-04-25T13:50:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.730+0000 7fc338ff9640 1 -- 172.21.15.27:0/483764008 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5595c4b38710 con 0x5595c47e31f0 2024-04-25T13:50:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.730+0000 7fc338ff9640 10 client.27010 handle_command_reply: tid=0 2024-04-25T13:50:11.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.731+0000 7fc35e5ac740 1 client.27010 shutdown 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 -- 172.21.15.27:0/483764008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 msgr2=0x5595c4bd8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bd8160 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc3400a7160 tx=0x7fc3400a75e0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 -- 172.21.15.27:0/483764008 shutdown_connections 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5595c47e31f0 0x5595c4be5610 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4824410 0x5595c4bdf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bd8160 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/483764008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c47fb550 0x5595c4bd7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 -- 172.21.15.27:0/483764008 >> 172.21.15.27:0/483764008 conn(0x5595c4818560 msgr2=0x5595c4bbfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc338ff9640 0 client.27010 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:11.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 -- 172.21.15.27:0/483764008 shutdown_connections 2024-04-25T13:50:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 1 -- 172.21.15.27:0/483764008 wait complete. 2024-04-25T13:50:11.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.732+0000 7fc35e5ac740 20 client.27010 trim_cache size 0 max 16384 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.762+0000 7fc35e5ac740 1 Processor -- start 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.762+0000 7fc35e5ac740 1 -- start start 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c47fb550 0x5595c4bc9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bdef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 0x5595c4bc9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc3501a7000 con 0x5595c47fb550 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc3501a97d0 con 0x5595c4824410 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc35e5ac740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc3501a75c0 con 0x5595c4803730 2024-04-25T13:50:11.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc3553bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c47fb550 0x5595c4bc9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc3553bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c47fb550 0x5595c4bc9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46596/0 (socket says 172.21.15.27:46596) 2024-04-25T13:50:11.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc3553bc640 1 -- 172.21.15.27:0/1235119672 learned_addr learned my addr 172.21.15.27:0/1235119672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.763+0000 7fc355bbd640 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 0x5595c4bc9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc354bbb640 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bdef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc3553bc640 1 -- 172.21.15.27:0/1235119672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 msgr2=0x5595c4bdef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc3553bc640 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bdef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc3553bc640 1 -- 172.21.15.27:0/1235119672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 msgr2=0x5595c4bc9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc3553bc640 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 0x5595c4bc9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc3553bc640 1 -- 172.21.15.27:0/1235119672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595c4bd2a70 con 0x5595c47fb550 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc354bbb640 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bdef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc3553bc640 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c47fb550 0x5595c4bc9000 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc34c005ab0 tx=0x7fc34c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7136345af6776fad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc33affd640 1 -- 172.21.15.27:0/1235119672 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c031dd0 con 0x5595c47fb550 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc33affd640 1 -- 172.21.15.27:0/1235119672 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc34c056020 con 0x5595c47fb550 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.764+0000 7fc33affd640 1 -- 172.21.15.27:0/1235119672 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c032040 con 0x5595c47fb550 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1235119672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c47fb550 msgr2=0x5595c4bc9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c47fb550 0x5595c4bc9000 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc34c005ab0 tx=0x7fc34c03e000 comp rx=0 tx=0).stop 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1235119672 shutdown_connections 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 0x5595c4bc9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c4803730 0x5595c4bdef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1235119672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c47fb550 0x5595c4bc9000 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1235119672 >> 172.21.15.27:0/1235119672 conn(0x5595c4818560 msgr2=0x5595c4bd6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1235119672 shutdown_connections 2024-04-25T13:50:11.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.765+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1235119672 wait complete. 2024-04-25T13:50:11.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.766+0000 7fc35e5ac740 1 -- start start 2024-04-25T13:50:11.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.766+0000 7fc35e5ac740 1 Processor -- start 2024-04-25T13:50:11.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c47fb550 0x5595c4bdbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4803730 0x5595c4bdc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc3553bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c47fb550 0x5595c4bdbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc3553bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c47fb550 0x5595c4bdbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36278/0 (socket says 172.21.15.27:36278) 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc35e5ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 0x5595c4c5c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc3553bc640 1 -- 172.21.15.27:0/1145373327 learned_addr learned my addr 172.21.15.27:0/1145373327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc3501a7c90 con 0x5595c4803730 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc3501a9100 con 0x5595c4824410 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc355bbd640 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 0x5595c4c5c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc354bbb640 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4803730 0x5595c4bdc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.767+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc3501a8360 con 0x5595c47fb550 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.768+0000 7fc35e5ac740 10 client.0 _ll_register_callbacks cb 0x7fc340010720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.768+0000 7fc3553bc640 1 -- 172.21.15.27:0/1145373327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 msgr2=0x5595c4c5c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.768+0000 7fc3553bc640 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 0x5595c4c5c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.768+0000 7fc3553bc640 1 -- 172.21.15.27:0/1145373327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4803730 msgr2=0x5595c4bdc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.768+0000 7fc3553bc640 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4803730 0x5595c4bdc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.768+0000 7fc3553bc640 1 -- 172.21.15.27:0/1145373327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595c4bdf060 con 0x5595c47fb550 2024-04-25T13:50:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.768+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc34c056020 con 0x5595c47fb550 2024-04-25T13:50:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.768+0000 7fc3553bc640 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c47fb550 0x5595c4bdbb60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc34c032fd0 tx=0x7fc34c004780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67eb738906303371 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc338ff9640 10 client.27013 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc338ff9640 1 -- 172.21.15.27:0/1145373327 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c004e40 con 0x5595c47fb550 2024-04-25T13:50:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc338ff9640 1 -- 172.21.15.27:0/1145373327 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc34c005030 con 0x5595c47fb550 2024-04-25T13:50:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc338ff9640 1 -- 172.21.15.27:0/1145373327 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc34c005230 con 0x5595c47fb550 2024-04-25T13:50:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc35e5ac740 10 client.27013 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:11.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x5595c4c5dbb0 con 0x5595c47fb550 2024-04-25T13:50:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc338ff9640 1 -- 172.21.15.27:0/1145373327 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc34c0053d0 con 0x5595c47fb550 2024-04-25T13:50:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc35e5ac740 10 client.27013 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc35e5ac740 10 client.27013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc35e5ac740 20 client.27013 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5595c47b8630 0x5595c4c60440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:11.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc35e5ac740 4 client.27013 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:50:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.769+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5595c4b32580 con 0x5595c47b8630 2024-04-25T13:50:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.770+0000 7fc354bbb640 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5595c47b8630 0x5595c4c60440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.770+0000 7fc338ff9640 10 client.27013 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.770+0000 7fc354bbb640 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5595c47b8630 0x5595c4c60440 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e531a231265bb265 server_cookie=2c60f3137069eef5 in_seq=0 out_seq=0 2024-04-25T13:50:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.771+0000 7fc338ff9640 1 -- 172.21.15.27:0/1145373327 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5595c4b32580 con 0x5595c47b8630 2024-04-25T13:50:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc338ff9640 10 client.27013 handle_command_reply: tid=0 2024-04-25T13:50:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 client.27013 shutdown 2024-04-25T13:50:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c47fb550 msgr2=0x5595c4bdbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c47fb550 0x5595c4bdbb60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc34c032fd0 tx=0x7fc34c004780 comp rx=0 tx=0).stop 2024-04-25T13:50:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 shutdown_connections 2024-04-25T13:50:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5595c47b8630 0x5595c4c60440 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595c4824410 0x5595c4c5c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5595c4803730 0x5595c4bdc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 --2- 172.21.15.27:0/1145373327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5595c47fb550 0x5595c4bdbb60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 >> 172.21.15.27:0/1145373327 conn(0x5595c4818560 msgr2=0x5595c4bc8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.772+0000 7fc338ff9640 0 client.27013 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.773+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 shutdown_connections 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.773+0000 7fc35e5ac740 1 -- 172.21.15.27:0/1145373327 wait complete. 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.773+0000 7fc35e5ac740 20 client.27013 trim_cache size 0 max 16384 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:50:11.774 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "fbff23e9-e638-49a2-a120-0e98bae00eda", 2024-04-25T13:50:11.775 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:50:11.775 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:50:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 -- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc33c0afb70 msgr2=0x7fc33c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc33c0afb70 0x7fc33c0b2080 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc34c031500 tx=0x7fc34c009560 comp rx=0 tx=0).stop 2024-04-25T13:50:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 -- 172.21.15.27:0/787883410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc350103c20 msgr2=0x7fc35019ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc350103c20 0x7fc35019ca00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc34000e9f0 tx=0x7fc34000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 -- 172.21.15.27:0/787883410 shutdown_connections 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc33c0afb70 0x7fc33c0b2080 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc350108920 0x7fc3501a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc350103c20 0x7fc35019ca00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 --2- 172.21.15.27:0/787883410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3501029e0 0x7fc35019c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 -- 172.21.15.27:0/787883410 >> 172.21.15.27:0/787883410 conn(0x7fc3500fdbd0 msgr2=0x7fc3500ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 -- 172.21.15.27:0/787883410 shutdown_connections 2024-04-25T13:50:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:11.774+0000 7fc357620640 1 -- 172.21.15.27:0/787883410 wait complete. 2024-04-25T13:50:11.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:50:11.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:50:12.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 -- 172.21.15.27:0/814813123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd810a520 msgr2=0x7fcfd810b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 --2- 172.21.15.27:0/814813123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd810a520 0x7fcfd810b220 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009ca0 tx=0x7fcfd002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 -- 172.21.15.27:0/814813123 shutdown_connections 2024-04-25T13:50:12.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 --2- 172.21.15.27:0/814813123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfd8110460 0x7fcfd8112880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 --2- 172.21.15.27:0/814813123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd810b760 0x7fcfd810bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 --2- 172.21.15.27:0/814813123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd810a520 0x7fcfd810b220 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 -- 172.21.15.27:0/814813123 >> 172.21.15.27:0/814813123 conn(0x7fcfd8070940 msgr2=0x7fcfd8072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:12.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 -- 172.21.15.27:0/814813123 shutdown_connections 2024-04-25T13:50:12.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.022+0000 7fcfddb72640 1 -- 172.21.15.27:0/814813123 wait complete. 2024-04-25T13:50:12.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.023+0000 7fcfddb72640 1 Processor -- start 2024-04-25T13:50:12.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.023+0000 7fcfddb72640 1 -- start start 2024-04-25T13:50:12.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.023+0000 7fcfddb72640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd810a520 0x7fcfd81a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.023+0000 7fcfddb72640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd810b760 0x7fcfd81a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.023+0000 7fcfddb72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfd8110460 0x7fcfd81ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.024+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd810b760 0x7fcfd81a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.024+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd810b760 0x7fcfd81a5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36310/0 (socket says 172.21.15.27:36310) 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.024+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/3864449027 learned_addr learned my addr 172.21.15.27:0/3864449027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.024+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8114de0 con 0x7fcfd810a520 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.024+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8114c60 con 0x7fcfd8110460 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.024+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8114f60 con 0x7fcfd810b760 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.024+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfd8110460 0x7fcfd81ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/3864449027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfd8110460 msgr2=0x7fcfd81ac100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfd8110460 0x7fcfd81ac100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd810a520 msgr2=0x7fcfd81a4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd810a520 0x7fcfd81a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/3864449027 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd81ac770 con 0x7fcfd810b760 2024-04-25T13:50:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfd8110460 0x7fcfd81ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:50:12.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd810b760 0x7fcfd81a5170 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c600 tx=0x7fcfcc00cac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:12.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd4ff9640 1 -- 172.21.15.27:0/3864449027 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc015e20 con 0x7fcfd810b760 2024-04-25T13:50:12.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd81177f0 con 0x7fcfd810b760 2024-04-25T13:50:12.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd4ff9640 1 -- 172.21.15.27:0/3864449027 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfcc0035f0 con 0x7fcfd810b760 2024-04-25T13:50:12.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfd4ff9640 1 -- 172.21.15.27:0/3864449027 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc00b750 con 0x7fcfd810b760 2024-04-25T13:50:12.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.025+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd8117d90 con 0x7fcfd810b760 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.028+0000 7fcfd4ff9640 1 -- 172.21.15.27:0/3864449027 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcfcc004000 con 0x7fcfd810b760 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.028+0000 7fcfe4a28740 1 Processor -- start 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.028+0000 7fcfe4a28740 1 -- start start 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031c39a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b03186f4d0 0x55b031c39fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b031b9ed80 con 0x55b03186f4d0 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b031b588b0 con 0x55b031898390 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b031b9ad40 con 0x55b0318776b0 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfd7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031c39a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfd7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031c39a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:48142/0 (socket says 172.21.15.27:48142) 2024-04-25T13:50:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfd7fff640 1 -- 172.21.15.27:0/4036322837 learned_addr learned my addr 172.21.15.27:0/4036322837 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:12.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.029+0000 7fcfd7fff640 1 -- 172.21.15.27:0/4036322837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 msgr2=0x55b031c3ac70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:50:12.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.030+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/4036322837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.030+0000 7fcfd7fff640 1 -- 172.21.15.27:0/4036322837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b03186f4d0 msgr2=0x55b031c39fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:12.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.030+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/4036322837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b03186f4d0 0x55b031c39fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.030+0000 7fcfd7fff640 1 -- 172.21.15.27:0/4036322837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b031b29a40 con 0x55b031898390 2024-04-25T13:50:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.030+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/4036322837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031c39a60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009bb0 tx=0x7fcfc8010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1852f9ff7e16788 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.030+0000 7fcfbdffb640 1 -- 172.21.15.27:0/4036322837 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8004020 con 0x55b031898390 2024-04-25T13:50:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.030+0000 7fcfbdffb640 1 -- 172.21.15.27:0/4036322837 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc80041c0 con 0x55b031898390 2024-04-25T13:50:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.030+0000 7fcfbdffb640 1 -- 172.21.15.27:0/4036322837 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80126b0 con 0x55b031898390 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.031+0000 7fcfd4ff9640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfc00afb70 0x7fcfc00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfd4ff9640 1 -- 172.21.15.27:0/3864449027 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcfcc0e6d20 con 0x7fcfd810b760 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfe4a28740 1 -- 172.21.15.27:0/4036322837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 msgr2=0x55b031c39a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/4036322837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031c39a60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009bb0 tx=0x7fcfc8010740 comp rx=0 tx=0).stop 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfe4a28740 1 -- 172.21.15.27:0/4036322837 shutdown_connections 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/4036322837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/4036322837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b03186f4d0 0x55b031c39fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/4036322837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031c39a60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfe4a28740 1 -- 172.21.15.27:0/4036322837 >> 172.21.15.27:0/4036322837 conn(0x55b03188c4e0 msgr2=0x55b031c35270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:12.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfc00afb70 0x7fcfc00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.032+0000 7fcfe4a28740 1 -- 172.21.15.27:0/4036322837 shutdown_connections 2024-04-25T13:50:12.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.033+0000 7fcfe4a28740 1 -- 172.21.15.27:0/4036322837 wait complete. 2024-04-25T13:50:12.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.033+0000 7fcfe4a28740 1 -- start start 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.033+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfc00afb70 0x7fcfc00b2080 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcfd002ade0 tx=0x7fcfd0031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.033+0000 7fcfe4a28740 1 Processor -- start 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 0x55b031c43a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c43f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c4aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b031c3b780 con 0x55b031898390 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b031b25d10 con 0x55b03186f4d0 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b031c3b900 con 0x55b0318776b0 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfe4a28740 10 client.0 _ll_register_callbacks cb 0x55b031b52210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c43f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c4aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c43f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36326/0 (socket says 172.21.15.27:36326) 2024-04-25T13:50:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c4aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46628/0 (socket says 172.21.15.27:46628) 2024-04-25T13:50:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd7fff640 1 -- 172.21.15.27:0/1481147402 learned_addr learned my addr 172.21.15.27:0/1481147402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 0x55b031c43a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/1481147402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 msgr2=0x55b031c43a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 0x55b031c43a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/1481147402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 msgr2=0x55b031c4aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c4aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.034+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/1481147402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b03197a7d0 con 0x55b0318776b0 2024-04-25T13:50:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.036+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c43f50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0ac4d0 tx=0x7fcfcc0aeeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aeafe67081da2a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.036+0000 7fcfb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.036+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/1481147402 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0b3610 con 0x55b0318776b0 2024-04-25T13:50:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.036+0000 7fcfe4a28740 1 -- 172.21.15.27:0/1481147402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfc80041c0 con 0x55b0318776b0 2024-04-25T13:50:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.036+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/1481147402 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcfcc0037a0 con 0x55b0318776b0 2024-04-25T13:50:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/1481147402 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc0b8030 con 0x55b0318776b0 2024-04-25T13:50:12.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfe4a28740 10 client.27022 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfe4a28740 1 -- 172.21.15.27:0/1481147402 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b0318b03f0 con 0x55b0318776b0 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/1481147402 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc80041c0 con 0x55b0318776b0 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/1481147402 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcfcc0aad60 con 0x55b0318776b0 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfe4a28740 10 client.27022 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfe4a28740 10 client.27022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfe4a28740 20 client.27022 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b031856f60 0x55b031c51810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfe4a28740 4 client.27022 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.037+0000 7fcfe4a28740 1 -- 172.21.15.27:0/1481147402 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b031bac990 con 0x55b031856f60 2024-04-25T13:50:12.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.038+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b031856f60 0x55b031c51810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.039+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b031856f60 0x55b031c51810 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab4886319fa4dbb1 server_cookie=e79055c1cb9e8e18 in_seq=0 out_seq=0 2024-04-25T13:50:12.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.039+0000 7fcfb6ffd640 10 client.27022 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:12.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.040+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/1481147402 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b031bac990 con 0x55b031856f60 2024-04-25T13:50:12.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.040+0000 7fcfb6ffd640 10 client.27022 handle_command_reply: tid=0 2024-04-25T13:50:12.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.040+0000 7fcfe4a28740 1 client.27022 shutdown 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfe4a28740 1 -- 172.21.15.27:0/1481147402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 msgr2=0x55b031c43f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c43f50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0ac4d0 tx=0x7fcfcc0aeeb0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfe4a28740 1 -- 172.21.15.27:0/1481147402 shutdown_connections 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b031856f60 0x55b031c51810 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c4aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c43f50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/1481147402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 0x55b031c43a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfe4a28740 1 -- 172.21.15.27:0/1481147402 >> 172.21.15.27:0/1481147402 conn(0x55b03188c4e0 msgr2=0x55b031c343f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.041+0000 7fcfb6ffd640 0 client.27022 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.042+0000 7fcfe4a28740 1 -- 172.21.15.27:0/1481147402 shutdown_connections 2024-04-25T13:50:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.042+0000 7fcfe4a28740 1 -- 172.21.15.27:0/1481147402 wait complete. 2024-04-25T13:50:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.042+0000 7fcfe4a28740 20 client.27022 trim_cache size 0 max 16384 2024-04-25T13:50:12.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.072+0000 7fcfe4a28740 1 Processor -- start 2024-04-25T13:50:12.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.072+0000 7fcfe4a28740 1 -- start start 2024-04-25T13:50:12.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.072+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 0x55b031c3e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.072+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.072+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c42050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.072+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81af980 con 0x55b031898390 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.072+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81b2200 con 0x55b03186f4d0 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.072+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81afff0 con 0x55b0318776b0 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.073+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.073+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:36336/0 (socket says 172.21.15.27:36336) 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.073+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/848977088 learned_addr learned my addr 172.21.15.27:0/848977088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.076+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c42050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.076+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 0x55b031c3e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.079+0000 7fcfd7fff640 1 -- 172.21.15.27:0/848977088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 msgr2=0x55b031c3ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:12.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.079+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.079+0000 7fcfd7fff640 1 -- 172.21.15.27:0/848977088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 msgr2=0x55b031c3e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:12.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.079+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 0x55b031c3e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.079+0000 7fcfd7fff640 1 -- 172.21.15.27:0/848977088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b031c440f0 con 0x55b031898390 2024-04-25T13:50:12.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.079+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:12.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.080+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c42050 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009460 tx=0x7fcfc80099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc46f59cbc3524b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:12.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.081+0000 7fcfbd7fa640 1 -- 172.21.15.27:0/848977088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802a070 con 0x55b031898390 2024-04-25T13:50:12.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.081+0000 7fcfbd7fa640 1 -- 172.21.15.27:0/848977088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc8005730 con 0x55b031898390 2024-04-25T13:50:12.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.083+0000 7fcfe4a28740 1 -- 172.21.15.27:0/848977088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 msgr2=0x55b031c42050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.083+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c42050 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009460 tx=0x7fcfc80099b0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.087+0000 7fcfe4a28740 1 -- 172.21.15.27:0/848977088 shutdown_connections 2024-04-25T13:50:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.087+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b031898390 0x55b031c42050 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.087+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0318776b0 0x55b031c3ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.087+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/848977088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b03186f4d0 0x55b031c3e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.087+0000 7fcfe4a28740 1 -- 172.21.15.27:0/848977088 >> 172.21.15.27:0/848977088 conn(0x55b03188c4e0 msgr2=0x55b031c300e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:12.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.091+0000 7fcfe4a28740 1 -- 172.21.15.27:0/848977088 shutdown_connections 2024-04-25T13:50:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.094+0000 7fcfe4a28740 1 -- 172.21.15.27:0/848977088 wait complete. 2024-04-25T13:50:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.095+0000 7fcfe4a28740 1 -- start start 2024-04-25T13:50:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.095+0000 7fcfe4a28740 1 Processor -- start 2024-04-25T13:50:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.096+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b03186f4d0 0x55b031c473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.096+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0318776b0 0x55b031c478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.096+0000 7fcfe4a28740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031cb7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.096+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81b06c0 con 0x55b0318776b0 2024-04-25T13:50:12.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.096+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81b1b30 con 0x55b031898390 2024-04-25T13:50:12.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.096+0000 7fcfe4a28740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81b0d90 con 0x55b03186f4d0 2024-04-25T13:50:12.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.096+0000 7fcfe4a28740 10 client.0 _ll_register_callbacks cb 0x7fcfcc004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:12.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.101+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0318776b0 0x55b031c478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.101+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0318776b0 0x55b031c478e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46650/0 (socket says 172.21.15.27:46650) 2024-04-25T13:50:12.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.101+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/3298463095 learned_addr learned my addr 172.21.15.27:0/3298463095 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:12.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.101+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031cb7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.103+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b03186f4d0 0x55b031c473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.107+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/3298463095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b03186f4d0 msgr2=0x55b031c473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:12.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.111+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b03186f4d0 0x55b031c473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.111+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 msgr2=0x55b031cb7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:12.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.111+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031cb7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.111+0000 7fcfd6ffd640 1 -- 172.21.15.27:0/3298463095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0318700f0 con 0x55b0318776b0 2024-04-25T13:50:12.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.112+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b03186f4d0 0x55b031c473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:12.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.114+0000 7fcfd7fff640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031cb7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:50:12.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.115+0000 7fcfd6ffd640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0318776b0 0x55b031c478e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0aef60 tx=0x7fcfcc0039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a37b178140581a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:12.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.115+0000 7fcfb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:50:12.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.119+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/3298463095 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0aebb0 con 0x55b0318776b0 2024-04-25T13:50:12.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.119+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/3298463095 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcfcc0aade0 con 0x55b0318776b0 2024-04-25T13:50:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.120+0000 7fcfe4a28740 1 -- 172.21.15.27:0/3298463095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b031ba6800 con 0x55b0318776b0 2024-04-25T13:50:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.122+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/3298463095 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc0ce050 con 0x55b0318776b0 2024-04-25T13:50:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.122+0000 7fcfe4a28740 10 client.17277 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.122+0000 7fcfe4a28740 1 -- 172.21.15.27:0/3298463095 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b031b5f4b0 con 0x55b0318776b0 2024-04-25T13:50:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.123+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/3298463095 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b031ba6800 con 0x55b0318776b0 2024-04-25T13:50:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.123+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/3298463095 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcfcc0a99f0 con 0x55b0318776b0 2024-04-25T13:50:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.123+0000 7fcfe4a28740 10 client.17277 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.123+0000 7fcfe4a28740 10 client.17277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.123+0000 7fcfe4a28740 20 client.17277 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.123+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b03182c3a0 0x55b031cbb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.123+0000 7fcfe4a28740 4 client.17277 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:50:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.123+0000 7fcfe4a28740 1 -- 172.21.15.27:0/3298463095 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b031b214a0 con 0x55b03182c3a0 2024-04-25T13:50:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.124+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b03182c3a0 0x55b031cbb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.124+0000 7fcfd77fe640 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b03182c3a0 0x55b031cbb8e0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77b4724df0a8b794 server_cookie=c7126ed67a0f2a31 in_seq=0 out_seq=0 2024-04-25T13:50:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.124+0000 7fcfb6ffd640 10 client.17277 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.125+0000 7fcfb6ffd640 1 -- 172.21.15.27:0/3298463095 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55b031b214a0 con 0x55b03182c3a0 2024-04-25T13:50:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.125+0000 7fcfb6ffd640 10 client.17277 handle_command_reply: tid=0 2024-04-25T13:50:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.125+0000 7fcfe4a28740 1 client.17277 shutdown 2024-04-25T13:50:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.131+0000 7fcfe4a28740 1 -- 172.21.15.27:0/3298463095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0318776b0 msgr2=0x55b031c478e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.131+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0318776b0 0x55b031c478e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0aef60 tx=0x7fcfcc0039d0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.132+0000 7fcfe4a28740 1 -- 172.21.15.27:0/3298463095 shutdown_connections 2024-04-25T13:50:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.132+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b03182c3a0 0x55b031cbb8e0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.132+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b031898390 0x55b031cb7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.132+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0318776b0 0x55b031c478e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.132+0000 7fcfe4a28740 1 --2- 172.21.15.27:0/3298463095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b03186f4d0 0x55b031c473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.132+0000 7fcfe4a28740 1 -- 172.21.15.27:0/3298463095 >> 172.21.15.27:0/3298463095 conn(0x55b03188c4e0 msgr2=0x55b031c39a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.132+0000 7fcfb6ffd640 0 client.17277 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.134+0000 7fcfe4a28740 1 -- 172.21.15.27:0/3298463095 shutdown_connections 2024-04-25T13:50:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.136+0000 7fcfe4a28740 1 -- 172.21.15.27:0/3298463095 wait complete. 2024-04-25T13:50:12.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.136+0000 7fcfe4a28740 20 client.17277 trim_cache size 0 max 16384 2024-04-25T13:50:12.136 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:50:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (400 inodes in the stack)", 2024-04-25T13:50:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:50:12.137 INFO:teuthology.orchestra.run.smithi027.stdout: "fbff23e9-e638-49a2-a120-0e98bae00eda": { 2024-04-25T13:50:12.137 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:50:12.137 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "fbff23e9-e638-49a2-a120-0e98bae00eda", 2024-04-25T13:50:12.137 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:50:12.137 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:50:12.137 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:50:12.137 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:50:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.137+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfc00afb70 msgr2=0x7fcfc00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.137+0000 7fcfddb72640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfc00afb70 0x7fcfc00b2080 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcfd002ade0 tx=0x7fcfd0031af0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.137+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd810b760 msgr2=0x7fcfd81a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.137+0000 7fcfddb72640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd810b760 0x7fcfd81a5170 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c600 tx=0x7fcfcc00cac0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.142+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 shutdown_connections 2024-04-25T13:50:12.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.142+0000 7fcfddb72640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfc00afb70 0x7fcfc00b2080 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.142+0000 7fcfddb72640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfd8110460 0x7fcfd81ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.142+0000 7fcfddb72640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd810b760 0x7fcfd81a5170 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.142+0000 7fcfddb72640 1 --2- 172.21.15.27:0/3864449027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfd810a520 0x7fcfd81a4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.142+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 >> 172.21.15.27:0/3864449027 conn(0x7fcfd8070940 msgr2=0x7fcfd8072d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.145+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 shutdown_connections 2024-04-25T13:50:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:12.145+0000 7fcfddb72640 1 -- 172.21.15.27:0/3864449027 wait complete. 2024-04-25T13:50:12.169 INFO:tasks.cephfs.filesystem:scrub status for tag:fbff23e9-e638-49a2-a120-0e98bae00eda - {'path': '/', 'tag': 'fbff23e9-e638-49a2-a120-0e98bae00eda', 'options': 'recursive,force'} 2024-04-25T13:50:12.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:12 smithi122 ceph-mon[32843]: pgmap v628: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-04-25T13:50:12.593 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:12 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:50:12.593 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:12 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:12.593 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:12 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:50:12.594 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:12 smithi045 ceph-mon[32731]: pgmap v628: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-04-25T13:50:12.594 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:12 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:50:12.594 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:12 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:12.594 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:12 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:50:12.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:12 smithi027 ceph-mon[29140]: pgmap v628: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-04-25T13:50:12.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:12 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:50:12.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:12 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:12.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:12 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:50:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 545.0 s, 1660.0 tps, lat 60.071 ms stddev 70.379 2024-04-25T13:50:13.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:13 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:13.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:13 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:13.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:13 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:14.459 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:14 smithi122 ceph-mon[32843]: pgmap v629: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 716 op/s 2024-04-25T13:50:14.459 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:14 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:50:14.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:14 smithi045 ceph-mon[32731]: pgmap v629: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 716 op/s 2024-04-25T13:50:14.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:14 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:50:14.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:14 smithi027 ceph-mon[29140]: pgmap v629: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 716 op/s 2024-04-25T13:50:14.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:14 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:50:16.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:16 smithi122 ceph-mon[32843]: pgmap v630: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 11 MiB/s wr, 654 op/s 2024-04-25T13:50:16.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:16 smithi045 ceph-mon[32731]: pgmap v630: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 11 MiB/s wr, 654 op/s 2024-04-25T13:50:16.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:16 smithi027 ceph-mon[29140]: pgmap v630: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 11 MiB/s wr, 654 op/s 2024-04-25T13:50:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 550.0 s, 1899.4 tps, lat 52.159 ms stddev 55.935 2024-04-25T13:50:18.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:18 smithi122 ceph-mon[32843]: pgmap v631: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 17 MiB/s wr, 925 op/s 2024-04-25T13:50:18.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:18 smithi045 ceph-mon[32731]: pgmap v631: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 17 MiB/s wr, 925 op/s 2024-04-25T13:50:18.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:18 smithi027 ceph-mon[29140]: pgmap v631: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 17 MiB/s wr, 925 op/s 2024-04-25T13:50:20.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:20 smithi122 ceph-mon[32843]: pgmap v632: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 16 MiB/s wr, 767 op/s 2024-04-25T13:50:20.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:20 smithi045 ceph-mon[32731]: pgmap v632: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 16 MiB/s wr, 767 op/s 2024-04-25T13:50:20.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:20 smithi027 ceph-mon[29140]: pgmap v632: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 16 MiB/s wr, 767 op/s 2024-04-25T13:50:21.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:21 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:50:21.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:21 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:50:21.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:21 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:50:22.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:22 smithi122 ceph-mon[32843]: pgmap v633: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 18 MiB/s wr, 804 op/s 2024-04-25T13:50:22.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:22 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:50:22.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:22 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:50:22.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:22 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:50:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:22 smithi045 ceph-mon[32731]: pgmap v633: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 18 MiB/s wr, 804 op/s 2024-04-25T13:50:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:22 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:50:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:22 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:50:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:22 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:50:22.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:22 smithi027 ceph-mon[29140]: pgmap v633: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 18 MiB/s wr, 804 op/s 2024-04-25T13:50:22.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:22 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:50:22.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:22 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:50:22.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:22 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:50:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 555.0 s, 2001.0 tps, lat 50.401 ms stddev 58.771 2024-04-25T13:50:24.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3669d9b640 1 -- 172.21.15.27:0/566363626 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660035020 con 0x7f366c102d70 2024-04-25T13:50:24.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 -- 172.21.15.27:0/566363626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 msgr2=0x7f366c103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 --2- 172.21.15.27:0/566363626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 0x7f366c103180 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3660009930 tx=0x7f366002a460 comp rx=0 tx=0).stop 2024-04-25T13:50:24.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 -- 172.21.15.27:0/566363626 shutdown_connections 2024-04-25T13:50:24.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 --2- 172.21.15.27:0/566363626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366c1036c0 0x7f366c103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 --2- 172.21.15.27:0/566363626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 0x7f366c103180 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 --2- 172.21.15.27:0/566363626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c102420 0x7f366c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 -- 172.21.15.27:0/566363626 >> 172.21.15.27:0/566363626 conn(0x7f366c0fdc30 msgr2=0x7f366c1000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:24.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 -- 172.21.15.27:0/566363626 shutdown_connections 2024-04-25T13:50:24.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.343+0000 7f3671092640 1 -- 172.21.15.27:0/566363626 wait complete. 2024-04-25T13:50:24.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.344+0000 7f3671092640 1 Processor -- start 2024-04-25T13:50:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f3671092640 1 -- start start 2024-04-25T13:50:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f3671092640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c102420 0x7f366c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f3671092640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 0x7f366c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f3671092640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366c1036c0 0x7f366c1a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f3671092640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f366c111940 con 0x7f366c102420 2024-04-25T13:50:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f3671092640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f366c1117c0 con 0x7f366c1036c0 2024-04-25T13:50:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f3671092640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f366c111ac0 con 0x7f366c102d70 2024-04-25T13:50:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 0x7f366c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 0x7f366c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39704/0 (socket says 172.21.15.27:39704) 2024-04-25T13:50:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366a59c640 1 -- 172.21.15.27:0/602171842 learned_addr learned my addr 172.21.15.27:0/602171842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366b59e640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366c1036c0 0x7f366c1a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366a59c640 1 -- 172.21.15.27:0/602171842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366c1036c0 msgr2=0x7f366c1a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366a59c640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366c1036c0 0x7f366c1a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366a59c640 1 -- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c102420 msgr2=0x7f366c19c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366a59c640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c102420 0x7f366c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.345+0000 7f366a59c640 1 -- 172.21.15.27:0/602171842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f366c1a4020 con 0x7f366c102d70 2024-04-25T13:50:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.346+0000 7f366a59c640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 0x7f366c19ca20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3660005d90 tx=0x7f3660004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.346+0000 7f3653fff640 1 -- 172.21.15.27:0/602171842 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660040070 con 0x7f366c102d70 2024-04-25T13:50:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.346+0000 7f3653fff640 1 -- 172.21.15.27:0/602171842 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3660004d20 con 0x7f366c102d70 2024-04-25T13:50:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.346+0000 7f3653fff640 1 -- 172.21.15.27:0/602171842 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660035020 con 0x7f366c102d70 2024-04-25T13:50:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.346+0000 7f3671092640 1 -- 172.21.15.27:0/602171842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f366c104210 con 0x7f366c102d70 2024-04-25T13:50:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.346+0000 7f3671092640 1 -- 172.21.15.27:0/602171842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f366c104730 con 0x7f366c102d70 2024-04-25T13:50:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.350+0000 7f3678018740 1 Processor -- start 2024-04-25T13:50:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.351+0000 7f3678018740 1 -- start start 2024-04-25T13:50:24.351 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.351+0000 7f3678018740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50ac1b0 0x5632a544c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.351 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.351+0000 7f3678018740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a50832f0 0x5632a544cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.351 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.351+0000 7f3678018740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a5453df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.351+0000 7f3678018740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5632a53b2460 con 0x5632a50ac1b0 2024-04-25T13:50:24.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.351+0000 7f3678018740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5632a536c590 con 0x5632a508b4d0 2024-04-25T13:50:24.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.351+0000 7f3678018740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5632a53ae420 con 0x5632a50832f0 2024-04-25T13:50:24.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50ac1b0 0x5632a544c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50ac1b0 0x5632a544c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41762/0 (socket says 172.21.15.27:41762) 2024-04-25T13:50:24.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366b59e640 1 -- 172.21.15.27:0/2482853625 learned_addr learned my addr 172.21.15.27:0/2482853625 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:24.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366a59c640 1 --2- 172.21.15.27:0/2482853625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a5453df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366a59c640 1 -- 172.21.15.27:0/2482853625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a50832f0 msgr2=0x5632a544cef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366a59c640 1 --2- 172.21.15.27:0/2482853625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a50832f0 0x5632a544cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366a59c640 1 -- 172.21.15.27:0/2482853625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50ac1b0 msgr2=0x5632a544c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366a59c640 1 --2- 172.21.15.27:0/2482853625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50ac1b0 0x5632a544c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.352+0000 7f366a59c640 1 -- 172.21.15.27:0/2482853625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632a53790d0 con 0x5632a508b4d0 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.353+0000 7f366a59c640 1 --2- 172.21.15.27:0/2482853625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a5453df0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f36600fcff0 tx=0x7f36601055a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=390026208d8e64a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.353+0000 7f3653fff640 1 -- 172.21.15.27:0/602171842 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3660043070 con 0x7f366c102d70 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.353+0000 7f36517fa640 1 -- 172.21.15.27:0/2482853625 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36600ffb80 con 0x5632a508b4d0 2024-04-25T13:50:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.353+0000 7f36517fa640 1 -- 172.21.15.27:0/2482853625 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366010c030 con 0x5632a508b4d0 2024-04-25T13:50:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.354+0000 7f3653fff640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36540afbe0 0x7f36540b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.354+0000 7f36517fa640 1 -- 172.21.15.27:0/2482853625 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660111780 con 0x5632a508b4d0 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.354+0000 7f3653fff640 1 -- 172.21.15.27:0/602171842 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f36600ff550 con 0x7f366c102d70 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.354+0000 7f3678018740 1 -- 172.21.15.27:0/2482853625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 msgr2=0x5632a5453df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.354+0000 7f3678018740 1 --2- 172.21.15.27:0/2482853625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a5453df0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f36600fcff0 tx=0x7f36601055a0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.355+0000 7f366b59e640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36540afbe0 0x7f36540b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.355+0000 7f3678018740 1 -- 172.21.15.27:0/2482853625 shutdown_connections 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.355+0000 7f3678018740 1 --2- 172.21.15.27:0/2482853625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a5453df0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.355+0000 7f3678018740 1 --2- 172.21.15.27:0/2482853625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a50832f0 0x5632a544cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.355+0000 7f3678018740 1 --2- 172.21.15.27:0/2482853625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50ac1b0 0x5632a544c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.355+0000 7f3678018740 1 -- 172.21.15.27:0/2482853625 >> 172.21.15.27:0/2482853625 conn(0x5632a50a0300 msgr2=0x5632a5448300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.355+0000 7f3678018740 1 -- 172.21.15.27:0/2482853625 shutdown_connections 2024-04-25T13:50:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.355+0000 7f3678018740 1 -- 172.21.15.27:0/2482853625 wait complete. 2024-04-25T13:50:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.356+0000 7f3678018740 1 -- start start 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.356+0000 7f366b59e640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36540afbe0 0x7f36540b20f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x5632a544db40 tx=0x7f3664015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.356+0000 7f3678018740 1 Processor -- start 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.356+0000 7f3678018740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5456370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.356+0000 7f3678018740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a508b4d0 0x5632a54568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.356+0000 7f3678018740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a50ac1b0 0x5632a545d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f3678018740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5632a5454600 con 0x5632a50832f0 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5456370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5456370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41772/0 (socket says 172.21.15.27:41772) 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f366ad9d640 1 -- 172.21.15.27:0/3656642162 learned_addr learned my addr 172.21.15.27:0/3656642162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f366a59c640 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a508b4d0 0x5632a54568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f366b59e640 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a50ac1b0 0x5632a545d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5632a5339b60 con 0x5632a50ac1b0 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5632a5454780 con 0x5632a508b4d0 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.357+0000 7f3678018740 10 client.0 _ll_register_callbacks cb 0x5632a535e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f366b59e640 1 -- 172.21.15.27:0/3656642162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a508b4d0 msgr2=0x5632a54568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f366b59e640 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a508b4d0 0x5632a54568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f366b59e640 1 -- 172.21.15.27:0/3656642162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 msgr2=0x5632a5456370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f366b59e640 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5456370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f366b59e640 1 -- 172.21.15.27:0/3656642162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632a50c4210 con 0x5632a50ac1b0 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f366ad9d640 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5456370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f366b59e640 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a50ac1b0 0x5632a545d810 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3664007c20 tx=0x7f3664004a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32a033e69cdeed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f364eff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:50:24.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f364eff5640 1 -- 172.21.15.27:0/3656642162 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366402e020 con 0x5632a50ac1b0 2024-04-25T13:50:24.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.358+0000 7f364eff5640 1 -- 172.21.15.27:0/3656642162 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f366402a3f0 con 0x5632a50ac1b0 2024-04-25T13:50:24.360 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.359+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f366010c030 con 0x5632a50ac1b0 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f364eff5640 1 -- 172.21.15.27:0/3656642162 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f3664026040 con 0x5632a50ac1b0 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f3678018740 10 client.26993 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5632a5190620 con 0x5632a50ac1b0 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f364eff5640 1 -- 172.21.15.27:0/3656642162 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366402aaa0 con 0x5632a50ac1b0 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f364eff5640 1 -- 172.21.15.27:0/3656642162 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f366403b020 con 0x5632a50ac1b0 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f3678018740 10 client.26993 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f3678018740 10 client.26993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f3678018740 20 client.26993 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f3678018740 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5632a506ad80 0x5632a54640c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f3678018740 4 client.26993 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.360+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f366010c030 con 0x5632a506ad80 2024-04-25T13:50:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.361+0000 7f366ad9d640 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5632a506ad80 0x5632a54640c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.362+0000 7f366ad9d640 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5632a506ad80 0x5632a54640c0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a3658f5e33e02d6 server_cookie=466f6e5fe1b214f3 in_seq=0 out_seq=0 2024-04-25T13:50:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.362+0000 7f364eff5640 10 client.26993 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.363+0000 7f364eff5640 1 -- 172.21.15.27:0/3656642162 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f366010c030 con 0x5632a506ad80 2024-04-25T13:50:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.363+0000 7f364eff5640 10 client.26993 handle_command_reply: tid=0 2024-04-25T13:50:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.363+0000 7f3678018740 1 client.26993 shutdown 2024-04-25T13:50:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.364+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a50ac1b0 msgr2=0x5632a545d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.364 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.364+0000 7f3678018740 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a50ac1b0 0x5632a545d810 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3664007c20 tx=0x7f3664004a60 comp rx=0 tx=0).stop 2024-04-25T13:50:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.365+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 shutdown_connections 2024-04-25T13:50:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.365+0000 7f3678018740 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5632a506ad80 0x5632a54640c0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.365+0000 7f3678018740 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a50ac1b0 0x5632a545d810 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.365+0000 7f364eff5640 0 client.26993 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.365+0000 7f3678018740 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a508b4d0 0x5632a54568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.365+0000 7f3678018740 1 --2- 172.21.15.27:0/3656642162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5456370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.365+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 >> 172.21.15.27:0/3656642162 conn(0x5632a50a0300 msgr2=0x5632a5447250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.365+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 shutdown_connections 2024-04-25T13:50:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.366+0000 7f3678018740 1 -- 172.21.15.27:0/3656642162 wait complete. 2024-04-25T13:50:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.366+0000 7f3678018740 20 client.26993 trim_cache size 0 max 16384 2024-04-25T13:50:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.388+0000 7f3678018740 1 Processor -- start 2024-04-25T13:50:24.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.388+0000 7f3678018740 1 -- start start 2024-04-25T13:50:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.388+0000 7f3678018740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54509d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.389+0000 7f3678018740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5450f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.389+0000 7f3678018740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.389+0000 7f3678018740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f366c1a71b0 con 0x5632a50832f0 2024-04-25T13:50:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.389+0000 7f3678018740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f366c1a9a10 con 0x5632a508b4d0 2024-04-25T13:50:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.389+0000 7f3678018740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f366c1a7800 con 0x5632a506ad80 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.389+0000 7f366b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.389+0000 7f366b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:58960/0 (socket says 172.21.15.27:58960) 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.389+0000 7f366b59e640 1 -- 172.21.15.27:0/3558376797 learned_addr learned my addr 172.21.15.27:0/3558376797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.390+0000 7f366ad9d640 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.390+0000 7f366a59c640 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5450f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.390+0000 7f366b59e640 1 -- 172.21.15.27:0/3558376797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 msgr2=0x5632a54509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.390+0000 7f366b59e640 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.390+0000 7f366b59e640 1 -- 172.21.15.27:0/3558376797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 msgr2=0x5632a5450f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.390+0000 7f366b59e640 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5450f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.390+0000 7f366b59e640 1 -- 172.21.15.27:0/3558376797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632a545dc70 con 0x5632a508b4d0 2024-04-25T13:50:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.390+0000 7f366b59e640 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545d730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3664015010 tx=0x7f3664020880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9be1de4da438e80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.391+0000 7f3650ff9640 1 -- 172.21.15.27:0/3558376797 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366402e040 con 0x5632a508b4d0 2024-04-25T13:50:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.391+0000 7f3650ff9640 1 -- 172.21.15.27:0/3558376797 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366401ec80 con 0x5632a508b4d0 2024-04-25T13:50:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.392+0000 7f3650ff9640 1 -- 172.21.15.27:0/3558376797 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664046c10 con 0x5632a508b4d0 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 -- 172.21.15.27:0/3558376797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 msgr2=0x5632a545d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545d730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3664015010 tx=0x7f3664020880 comp rx=0 tx=0).stop 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 -- 172.21.15.27:0/3558376797 shutdown_connections 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545d730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5450f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 --2- 172.21.15.27:0/3558376797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 -- 172.21.15.27:0/3558376797 >> 172.21.15.27:0/3558376797 conn(0x5632a50a0300 msgr2=0x5632a544c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 -- 172.21.15.27:0/3558376797 shutdown_connections 2024-04-25T13:50:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 -- 172.21.15.27:0/3558376797 wait complete. 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.393+0000 7f3678018740 1 -- start start 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f3678018740 1 Processor -- start 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f3678018740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54589c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f3678018740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5458f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f3678018740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f3678018740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f366c1a7ed0 con 0x5632a50832f0 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f3678018740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f366c1a9340 con 0x5632a508b4d0 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f3678018740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f366c1a85a0 con 0x5632a506ad80 2024-04-25T13:50:24.396 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f3678018740 10 client.0 _ll_register_callbacks cb 0x7f3664020b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:39730/0 (socket says 172.21.15.27:39730) 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366ad9d640 1 -- 172.21.15.27:0/4126817992 learned_addr learned my addr 172.21.15.27:0/4126817992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366a59c640 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5458f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366b59e640 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366ad9d640 1 -- 172.21.15.27:0/4126817992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 msgr2=0x5632a545be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366ad9d640 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366ad9d640 1 -- 172.21.15.27:0/4126817992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 msgr2=0x5632a5458f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366ad9d640 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5458f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366ad9d640 1 -- 172.21.15.27:0/4126817992 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632a545c400 con 0x5632a506ad80 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.394+0000 7f366a59c640 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5458f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.395+0000 7f366ad9d640 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54589c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5632a5464800 tx=0x7f365c0044d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d71f84314fedbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:24.397 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.395+0000 7f364eff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.395+0000 7f364eff5640 1 -- 172.21.15.27:0/4126817992 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365c01a590 con 0x5632a506ad80 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.395+0000 7f364eff5640 1 -- 172.21.15.27:0/4126817992 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f365c01a730 con 0x5632a506ad80 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.395+0000 7f3678018740 1 -- 172.21.15.27:0/4126817992 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f366401ec80 con 0x5632a506ad80 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.395+0000 7f364eff5640 1 -- 172.21.15.27:0/4126817992 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f365c02a070 con 0x5632a506ad80 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.395+0000 7f3678018740 10 client.27043 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.395+0000 7f3678018740 1 -- 172.21.15.27:0/4126817992 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5632a545d760 con 0x5632a506ad80 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f364eff5640 1 -- 172.21.15.27:0/4126817992 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f366401ec80 con 0x5632a506ad80 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f364eff5640 1 -- 172.21.15.27:0/4126817992 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f365c01a070 con 0x5632a506ad80 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f3678018740 10 client.27043 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f3678018740 10 client.27043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f3678018740 20 client.27043 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f3678018740 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5632a50ac1b0 0x5632a54ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f3678018740 4 client.27043 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:50:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f3678018740 1 -- 172.21.15.27:0/4126817992 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632a53c0070 con 0x5632a50ac1b0 2024-04-25T13:50:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.396+0000 7f366a59c640 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5632a50ac1b0 0x5632a54ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.397+0000 7f364eff5640 10 client.27043 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.397+0000 7f366a59c640 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5632a50ac1b0 0x5632a54ce990 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83da66ee5fcfdd96 server_cookie=18adce4992cd98b6 in_seq=0 out_seq=0 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f364eff5640 1 -- 172.21.15.27:0/4126817992 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632a53c0070 con 0x5632a50ac1b0 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f364eff5640 10 client.27043 handle_command_reply: tid=0 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 client.27043 shutdown 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 -- 172.21.15.27:0/4126817992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 msgr2=0x5632a54589c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54589c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5632a5464800 tx=0x7f365c0044d0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 -- 172.21.15.27:0/4126817992 shutdown_connections 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5632a50ac1b0 0x5632a54ce990 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5632a508b4d0 0x5632a545be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5632a50832f0 0x5632a5458f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 --2- 172.21.15.27:0/4126817992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5632a506ad80 0x5632a54589c0 secure :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5632a5464800 tx=0x7f365c0044d0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 -- 172.21.15.27:0/4126817992 >> 172.21.15.27:0/4126817992 conn(0x5632a50a0300 msgr2=0x5632a544c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f364eff5640 0 client.27043 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.415+0000 7f3678018740 1 -- 172.21.15.27:0/4126817992 shutdown_connections 2024-04-25T13:50:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3678018740 1 -- 172.21.15.27:0/4126817992 wait complete. 2024-04-25T13:50:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3678018740 20 client.27043 trim_cache size 0 max 16384 2024-04-25T13:50:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:50:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:50:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:50:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:50:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 -- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36540afbe0 msgr2=0x7f36540b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36540afbe0 0x7f36540b20f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x5632a544db40 tx=0x7f3664015040 comp rx=0 tx=0).stop 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 -- 172.21.15.27:0/602171842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 msgr2=0x7f366c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 0x7f366c19ca20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3660005d90 tx=0x7f3660004570 comp rx=0 tx=0).stop 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 -- 172.21.15.27:0/602171842 shutdown_connections 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36540afbe0 0x7f36540b20f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f366c1036c0 0x7f366c1a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f366c102d70 0x7f366c19ca20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 --2- 172.21.15.27:0/602171842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f366c102420 0x7f366c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.416+0000 7f3671092640 1 -- 172.21.15.27:0/602171842 >> 172.21.15.27:0/602171842 conn(0x7f366c0fdc30 msgr2=0x7f366c0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.417+0000 7f3671092640 1 -- 172.21.15.27:0/602171842 shutdown_connections 2024-04-25T13:50:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:24.417+0000 7f3671092640 1 -- 172.21.15.27:0/602171842 wait complete. 2024-04-25T13:50:24.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:24 smithi122 ceph-mon[32843]: pgmap v634: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 21 MiB/s wr, 927 op/s 2024-04-25T13:50:24.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:24 smithi045 ceph-mon[32731]: pgmap v634: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 21 MiB/s wr, 927 op/s 2024-04-25T13:50:24.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:24 smithi027 ceph-mon[29140]: pgmap v634: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 21 MiB/s wr, 927 op/s 2024-04-25T13:50:26.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:26 smithi122 ceph-mon[32843]: pgmap v635: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 18 MiB/s wr, 760 op/s 2024-04-25T13:50:26.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:26 smithi045 ceph-mon[32731]: pgmap v635: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 18 MiB/s wr, 760 op/s 2024-04-25T13:50:26.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:26 smithi027 ceph-mon[29140]: pgmap v635: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 18 MiB/s wr, 760 op/s 2024-04-25T13:50:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 560.0 s, 1995.0 tps, lat 50.233 ms stddev 58.513 2024-04-25T13:50:28.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:28 smithi122 ceph-mon[32843]: pgmap v636: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 21 MiB/s wr, 926 op/s 2024-04-25T13:50:28.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:28 smithi045 ceph-mon[32731]: pgmap v636: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 21 MiB/s wr, 926 op/s 2024-04-25T13:50:28.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:28 smithi027 ceph-mon[29140]: pgmap v636: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 21 MiB/s wr, 926 op/s 2024-04-25T13:50:30.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:30 smithi122 ceph-mon[32843]: pgmap v637: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 678 op/s 2024-04-25T13:50:30.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:30 smithi045 ceph-mon[32731]: pgmap v637: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 678 op/s 2024-04-25T13:50:30.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:30 smithi027 ceph-mon[29140]: pgmap v637: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 678 op/s 2024-04-25T13:50:32.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:32 smithi122 ceph-mon[32843]: pgmap v638: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 16 MiB/s wr, 671 op/s 2024-04-25T13:50:32.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:32 smithi045 ceph-mon[32731]: pgmap v638: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 16 MiB/s wr, 671 op/s 2024-04-25T13:50:32.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:32 smithi027 ceph-mon[29140]: pgmap v638: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 16 MiB/s wr, 671 op/s 2024-04-25T13:50:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 565.0 s, 1924.6 tps, lat 51.656 ms stddev 60.860 2024-04-25T13:50:34.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:34 smithi122 ceph-mon[32843]: pgmap v639: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 820 op/s 2024-04-25T13:50:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:34 smithi045 ceph-mon[32731]: pgmap v639: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 820 op/s 2024-04-25T13:50:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:34 smithi027 ceph-mon[29140]: pgmap v639: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 19 MiB/s wr, 820 op/s 2024-04-25T13:50:36.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:36 smithi122 ceph-mon[32843]: pgmap v640: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 14 MiB/s wr, 620 op/s 2024-04-25T13:50:36.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:36 smithi045 ceph-mon[32731]: pgmap v640: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 14 MiB/s wr, 620 op/s 2024-04-25T13:50:36.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:36 smithi027 ceph-mon[29140]: pgmap v640: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 14 MiB/s wr, 620 op/s 2024-04-25T13:50:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 -- 172.21.15.27:0/956688203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 msgr2=0x7f52e81051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 --2- 172.21.15.27:0/956688203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 0x7f52e81051a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009900 tx=0x7f52d802a410 comp rx=0 tx=0).stop 2024-04-25T13:50:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 -- 172.21.15.27:0/956688203 shutdown_connections 2024-04-25T13:50:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 --2- 172.21.15.27:0/956688203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52e8109a90 0x7f52e810beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 --2- 172.21.15.27:0/956688203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 0x7f52e81051a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 --2- 172.21.15.27:0/956688203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e8103b50 0x7f52e8104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.550 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 -- 172.21.15.27:0/956688203 >> 172.21.15.27:0/956688203 conn(0x7f52e80fed20 msgr2=0x7f52e81011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:37.551 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 -- 172.21.15.27:0/956688203 shutdown_connections 2024-04-25T13:50:37.551 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.550+0000 7f52f07a8640 1 -- 172.21.15.27:0/956688203 wait complete. 2024-04-25T13:50:37.551 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.552+0000 7f52f07a8640 1 Processor -- start 2024-04-25T13:50:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52f07a8640 1 -- start start 2024-04-25T13:50:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52f07a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52e8103b50 0x7f52e8071a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52f07a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 0x7f52e8071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52f07a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e8109a90 0x7f52e81ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52f07a8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52e810f6a0 con 0x7f52e8104d90 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52f07a8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52e810f520 con 0x7f52e8109a90 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52f07a8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f52e810f820 con 0x7f52e8103b50 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52edd43640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 0x7f52e8071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52ee544640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52e8103b50 0x7f52e8071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52eed45640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e8109a90 0x7f52e81ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52edd43640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 0x7f52e8071fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33570/0 (socket says 172.21.15.27:33570) 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52ee544640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52e8103b50 0x7f52e8071a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:52314/0 (socket says 172.21.15.27:52314) 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52ee544640 1 -- 172.21.15.27:0/240170956 learned_addr learned my addr 172.21.15.27:0/240170956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.553+0000 7f52eed45640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e8109a90 0x7f52e81ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38168/0 (socket says 172.21.15.27:38168) 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52eed45640 1 -- 172.21.15.27:0/240170956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52e8103b50 msgr2=0x7f52e8071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52eed45640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52e8103b50 0x7f52e8071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52eed45640 1 -- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 msgr2=0x7f52e8071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52eed45640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 0x7f52e8071fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52eed45640 1 -- 172.21.15.27:0/240170956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e81ac740 con 0x7f52e8109a90 2024-04-25T13:50:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52ee544640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52e8103b50 0x7f52e8071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52eed45640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e8109a90 0x7f52e81ac1b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009fd0 tx=0x7f52e000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52df7fe640 1 -- 172.21.15.27:0/240170956 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e000ce20 con 0x7f52e8109a90 2024-04-25T13:50:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52df7fe640 1 -- 172.21.15.27:0/240170956 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e0005020 con 0x7f52e8109a90 2024-04-25T13:50:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52df7fe640 1 -- 172.21.15.27:0/240170956 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0010690 con 0x7f52e8109a90 2024-04-25T13:50:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52f07a8640 1 -- 172.21.15.27:0/240170956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e81aca30 con 0x7f52e8109a90 2024-04-25T13:50:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.554+0000 7f52f07a8640 1 -- 172.21.15.27:0/240170956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e81acef0 con 0x7f52e8109a90 2024-04-25T13:50:37.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.557+0000 7f52df7fe640 1 -- 172.21.15.27:0/240170956 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f52e0010830 con 0x7f52e8109a90 2024-04-25T13:50:37.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.557+0000 7f52f765a740 1 Processor -- start 2024-04-25T13:50:37.557 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52f765a740 1 -- start start 2024-04-25T13:50:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52f765a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11361b0 0x5568f14d6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52f765a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f110d2f0 0x5568f14d6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52f765a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11154d0 0x5568f14dddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52f765a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5568f143c460 con 0x5568f110d2f0 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52f765a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5568f13f6590 con 0x5568f11361b0 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52f765a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5568f1438420 con 0x5568f11154d0 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52ee544640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f110d2f0 0x5568f14d6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52df7fe640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52d40afb60 0x7f52d40b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52ee544640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f110d2f0 0x5568f14d6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33574/0 (socket says 172.21.15.27:33574) 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52ee544640 1 -- 172.21.15.27:0/1746503372 learned_addr learned my addr 172.21.15.27:0/1746503372 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52edd43640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11361b0 0x5568f14d6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52edd43640 1 --2- 172.21.15.27:0/1746503372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11154d0 0x5568f14dddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52df7fe640 1 -- 172.21.15.27:0/240170956 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f52e00df250 con 0x7f52e8109a90 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52ee544640 1 -- 172.21.15.27:0/1746503372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11154d0 msgr2=0x5568f14dddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52ee544640 1 --2- 172.21.15.27:0/1746503372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11154d0 0x5568f14dddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52ee544640 1 -- 172.21.15.27:0/1746503372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11361b0 msgr2=0x5568f14d6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52ee544640 1 --2- 172.21.15.27:0/1746503372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11361b0 0x5568f14d6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.558+0000 7f52ee544640 1 -- 172.21.15.27:0/1746503372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f14030d0 con 0x5568f110d2f0 2024-04-25T13:50:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.559+0000 7f52eed45640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52d40afb60 0x7f52d40b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.559+0000 7f52ee544640 1 --2- 172.21.15.27:0/1746503372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f110d2f0 0x5568f14d6ef0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f52e40115c0 tx=0x7f52e4011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b53c0baa1d77fde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.559+0000 7f52eed45640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52d40afb60 0x7f52d40b2070 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f52e00a7460 tx=0x7f52e0014000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.559+0000 7f52dcff9640 1 -- 172.21.15.27:0/1746503372 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4005020 con 0x5568f110d2f0 2024-04-25T13:50:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.559+0000 7f52dcff9640 1 -- 172.21.15.27:0/1746503372 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e40047a0 con 0x5568f110d2f0 2024-04-25T13:50:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.559+0000 7f52dcff9640 1 -- 172.21.15.27:0/1746503372 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e400f8e0 con 0x5568f110d2f0 2024-04-25T13:50:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.560+0000 7f52f765a740 1 -- 172.21.15.27:0/1746503372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f110d2f0 msgr2=0x5568f14d6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.560+0000 7f52f765a740 1 --2- 172.21.15.27:0/1746503372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f110d2f0 0x5568f14d6ef0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f52e40115c0 tx=0x7f52e4011a80 comp rx=0 tx=0).stop 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.561+0000 7f52f765a740 1 -- 172.21.15.27:0/1746503372 shutdown_connections 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.561+0000 7f52f765a740 1 --2- 172.21.15.27:0/1746503372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11154d0 0x5568f14dddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.561+0000 7f52f765a740 1 --2- 172.21.15.27:0/1746503372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f110d2f0 0x5568f14d6ef0 secure :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f52e40115c0 tx=0x7f52e4011a80 comp rx=0 tx=0).stop 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.561+0000 7f52f765a740 1 --2- 172.21.15.27:0/1746503372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11361b0 0x5568f14d6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.562+0000 7f52f765a740 1 -- 172.21.15.27:0/1746503372 >> 172.21.15.27:0/1746503372 conn(0x5568f112a300 msgr2=0x5568f14d2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.562+0000 7f52f765a740 1 -- 172.21.15.27:0/1746503372 shutdown_connections 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.562+0000 7f52f765a740 1 -- 172.21.15.27:0/1746503372 wait complete. 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 1 -- start start 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 1 Processor -- start 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f110d2f0 0x5568f14e06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 0x5568f14e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5568f14de840 con 0x5568f11361b0 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5568f13c3b60 con 0x5568f11154d0 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5568f14de9c0 con 0x5568f110d2f0 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.563+0000 7f52f765a740 10 client.0 _ll_register_callbacks cb 0x5568f13e82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f110d2f0 0x5568f14e06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f110d2f0 0x5568f14e06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:52332/0 (socket says 172.21.15.27:52332) 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 -- 172.21.15.27:0/1215354689 learned_addr learned my addr 172.21.15.27:0/1215354689 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 0x5568f14e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52ee544640 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 -- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 msgr2=0x5568f14e0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 -- 172.21.15.27:0/1215354689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 msgr2=0x5568f14e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 0x5568f14e7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52edd43640 1 -- 172.21.15.27:0/1215354689 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f114e210 con 0x5568f110d2f0 2024-04-25T13:50:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.564+0000 7f52ee544640 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:50:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.581+0000 7f52edd43640 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f110d2f0 0x5568f14e06a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f52d80096c0 tx=0x7f52d802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37a93cd7310e9930 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.581+0000 7f52d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:37.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.584+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1215354689 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8035040 con 0x5568f110d2f0 2024-04-25T13:50:37.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.584+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1215354689 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f52d8032c30 con 0x5568f110d2f0 2024-04-25T13:50:37.586 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.585+0000 7f52f765a740 1 -- 172.21.15.27:0/1215354689 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e40047a0 con 0x5568f110d2f0 2024-04-25T13:50:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.589+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1215354689 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f52d8032e30 con 0x5568f110d2f0 2024-04-25T13:50:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.590+0000 7f52f765a740 10 client.27055 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.590+0000 7f52f765a740 1 -- 172.21.15.27:0/1215354689 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5568f121a620 con 0x5568f110d2f0 2024-04-25T13:50:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.594+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1215354689 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e40047a0 con 0x5568f110d2f0 2024-04-25T13:50:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.594+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1215354689 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f52d803d4f0 con 0x5568f110d2f0 2024-04-25T13:50:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.594+0000 7f52f765a740 10 client.27055 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.594+0000 7f52f765a740 10 client.27055 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.594+0000 7f52f765a740 20 client.27055 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.594+0000 7f52f765a740 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5568f10f4d80 0x5568f14ee280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.594+0000 7f52f765a740 4 client.27055 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.594+0000 7f52f765a740 1 -- 172.21.15.27:0/1215354689 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568f144a070 con 0x5568f10f4d80 2024-04-25T13:50:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.595+0000 7f52ee544640 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5568f10f4d80 0x5568f14ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.596+0000 7f52ee544640 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5568f10f4d80 0x5568f14ee280 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85e84eac156b169e server_cookie=fb79593a49c08e7f in_seq=0 out_seq=0 2024-04-25T13:50:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.598+0000 7f52d1ffb640 10 client.27055 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.598+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1215354689 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5568f144a070 con 0x5568f10f4d80 2024-04-25T13:50:37.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.598+0000 7f52d1ffb640 10 client.27055 handle_command_reply: tid=0 2024-04-25T13:50:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.598+0000 7f52f765a740 1 client.27055 shutdown 2024-04-25T13:50:37.600 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.600+0000 7f52f765a740 1 -- 172.21.15.27:0/1215354689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f110d2f0 msgr2=0x5568f14e06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.600 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.600+0000 7f52f765a740 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f110d2f0 0x5568f14e06a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f52d80096c0 tx=0x7f52d802a410 comp rx=0 tx=0).stop 2024-04-25T13:50:37.600 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.600+0000 7f52f765a740 1 -- 172.21.15.27:0/1215354689 shutdown_connections 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.600+0000 7f52f765a740 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5568f10f4d80 0x5568f14ee280 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.600+0000 7f52f765a740 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 0x5568f14e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.600+0000 7f52f765a740 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.600+0000 7f52f765a740 1 --2- 172.21.15.27:0/1215354689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f110d2f0 0x5568f14e06a0 secure :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f52d80096c0 tx=0x7f52d802a410 comp rx=0 tx=0).stop 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.600+0000 7f52f765a740 1 -- 172.21.15.27:0/1215354689 >> 172.21.15.27:0/1215354689 conn(0x5568f112a300 msgr2=0x5568f14d5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.601+0000 7f52d1ffb640 0 client.27055 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.601+0000 7f52f765a740 1 -- 172.21.15.27:0/1215354689 shutdown_connections 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.601+0000 7f52f765a740 1 -- 172.21.15.27:0/1215354689 wait complete. 2024-04-25T13:50:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.601+0000 7f52f765a740 20 client.27055 trim_cache size 0 max 16384 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.627+0000 7f52f765a740 1 Processor -- start 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.627+0000 7f52f765a740 1 -- start start 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52f765a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52f765a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11361b0 0x5568f14e7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52f765a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f10ca1c0 0x5568f14d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52f765a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52e81b3d10 con 0x5568f10ca1c0 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52f765a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52e81b6590 con 0x5568f11154d0 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52f765a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f52e81b4380 con 0x5568f11361b0 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52edd43640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f10ca1c0 0x5568f14d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52ee544640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11361b0 0x5568f14e7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52edd43640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f10ca1c0 0x5568f14d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33598/0 (socket says 172.21.15.27:33598) 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52edd43640 1 -- 172.21.15.27:0/1684947669 learned_addr learned my addr 172.21.15.27:0/1684947669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.628+0000 7f52edd43640 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52ee544640 1 -- 172.21.15.27:0/1684947669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 msgr2=0x5568f14d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52ee544640 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52ee544640 1 -- 172.21.15.27:0/1684947669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f10ca1c0 msgr2=0x5568f14d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52ee544640 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f10ca1c0 0x5568f14d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52ee544640 1 -- 172.21.15.27:0/1684947669 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f14db550 con 0x5568f11361b0 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52edd43640 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f10ca1c0 0x5568f14d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52ee544640 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11361b0 0x5568f14e7ad0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x5568f14d3090 tx=0x7f52e4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17ede916290c3e34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52d3fff640 1 -- 172.21.15.27:0/1684947669 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4031070 con 0x5568f11361b0 2024-04-25T13:50:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52d3fff640 1 -- 172.21.15.27:0/1684947669 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e4029030 con 0x5568f11361b0 2024-04-25T13:50:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.629+0000 7f52d3fff640 1 -- 172.21.15.27:0/1684947669 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e401d040 con 0x5568f11361b0 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 -- 172.21.15.27:0/1684947669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11361b0 msgr2=0x5568f14e7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11361b0 0x5568f14e7ad0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x5568f14d3090 tx=0x7f52e4009310 comp rx=0 tx=0).stop 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 -- 172.21.15.27:0/1684947669 shutdown_connections 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f10ca1c0 0x5568f14d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f11361b0 0x5568f14e7ad0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 --2- 172.21.15.27:0/1684947669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 -- 172.21.15.27:0/1684947669 >> 172.21.15.27:0/1684947669 conn(0x5568f112a300 msgr2=0x5568f14d1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 -- 172.21.15.27:0/1684947669 shutdown_connections 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.630+0000 7f52f765a740 1 -- 172.21.15.27:0/1684947669 wait complete. 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 1 -- start start 2024-04-25T13:50:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 1 Processor -- start 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f10ca1c0 0x5568f14e5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 0x5568f14e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f52e81b4a50 con 0x5568f11361b0 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52e81b5ec0 con 0x5568f11154d0 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f52e81b5120 con 0x5568f10ca1c0 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52f765a740 10 client.0 _ll_register_callbacks cb 0x7f52d80045a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52ee544640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52edd43640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f10ca1c0 0x5568f14e5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52ee544640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38210/0 (socket says 172.21.15.27:38210) 2024-04-25T13:50:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52ee544640 1 -- 172.21.15.27:0/1655652718 learned_addr learned my addr 172.21.15.27:0/1655652718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52edd43640 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 0x5568f14e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52ee544640 1 -- 172.21.15.27:0/1655652718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f10ca1c0 msgr2=0x5568f14e5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52ee544640 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f10ca1c0 0x5568f14e5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52ee544640 1 -- 172.21.15.27:0/1655652718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 msgr2=0x5568f14e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52ee544640 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 0x5568f14e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52ee544640 1 -- 172.21.15.27:0/1655652718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f14d28a0 con 0x5568f11154d0 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.631+0000 7f52edd43640 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f10ca1c0 0x5568f14e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.632+0000 7f52ee544640 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e5b50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f52e401b040 tx=0x7f52e4009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2710bdeddc081a85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.632+0000 7f52d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:50:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.633+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1655652718 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e401d030 con 0x5568f11154d0 2024-04-25T13:50:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.633+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1655652718 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f52e4005bf0 con 0x5568f11154d0 2024-04-25T13:50:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.633+0000 7f52f765a740 1 -- 172.21.15.27:0/1655652718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e4029030 con 0x5568f11154d0 2024-04-25T13:50:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1655652718 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f52e4031070 con 0x5568f11154d0 2024-04-25T13:50:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52f765a740 10 client.27017 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52f765a740 1 -- 172.21.15.27:0/1655652718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5568f1556720 con 0x5568f11154d0 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1655652718 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e40054e0 con 0x5568f11154d0 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1655652718 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f52e400f660 con 0x5568f11154d0 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52f765a740 10 client.27017 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52f765a740 10 client.27017 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52f765a740 20 client.27017 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.634+0000 7f52f765a740 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5568f1556de0 0x5568f1559200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.635+0000 7f52f765a740 4 client.27017 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.635+0000 7f52f765a740 1 -- 172.21.15.27:0/1655652718 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f52e4029030 con 0x5568f1556de0 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.635+0000 7f52edd43640 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5568f1556de0 0x5568f1559200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.635+0000 7f52d1ffb640 10 client.27017 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.635+0000 7f52edd43640 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5568f1556de0 0x5568f1559200 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fbdeb801c6a9fec server_cookie=87f9698aef1cfa33 in_seq=0 out_seq=0 2024-04-25T13:50:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52d1ffb640 1 -- 172.21.15.27:0/1655652718 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f52e4029030 con 0x5568f1556de0 2024-04-25T13:50:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52d1ffb640 10 client.27017 handle_command_reply: tid=0 2024-04-25T13:50:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 client.27017 shutdown 2024-04-25T13:50:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 -- 172.21.15.27:0/1655652718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 msgr2=0x5568f14e5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.653 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e5b50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f52e401b040 tx=0x7f52e4009c30 comp rx=0 tx=0).stop 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 -- 172.21.15.27:0/1655652718 shutdown_connections 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5568f1556de0 0x5568f1559200 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5568f11361b0 0x5568f14e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52d1ffb640 0 client.27017 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568f11154d0 0x5568f14e5b50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 --2- 172.21.15.27:0/1655652718 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5568f10ca1c0 0x5568f14e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 -- 172.21.15.27:0/1655652718 >> 172.21.15.27:0/1655652718 conn(0x5568f112a300 msgr2=0x5568f14e0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 -- 172.21.15.27:0/1655652718 shutdown_connections 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 1 -- 172.21.15.27:0/1655652718 wait complete. 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.653+0000 7f52f765a740 20 client.27017 trim_cache size 0 max 16384 2024-04-25T13:50:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:50:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:50:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:50:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:50:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 -- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52d40afb60 msgr2=0x7f52d40b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52d40afb60 0x7f52d40b2070 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f52e00a7460 tx=0x7f52e0014000 comp rx=0 tx=0).stop 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 -- 172.21.15.27:0/240170956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e8109a90 msgr2=0x7f52e81ac1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e8109a90 0x7f52e81ac1b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009fd0 tx=0x7f52e000ef10 comp rx=0 tx=0).stop 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 -- 172.21.15.27:0/240170956 shutdown_connections 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f52d40afb60 0x7f52d40b2070 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52e8109a90 0x7f52e81ac1b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52e8104d90 0x7f52e8071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 --2- 172.21.15.27:0/240170956 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52e8103b50 0x7f52e8071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 -- 172.21.15.27:0/240170956 >> 172.21.15.27:0/240170956 conn(0x7f52e80fed20 msgr2=0x7f52e8105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 -- 172.21.15.27:0/240170956 shutdown_connections 2024-04-25T13:50:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:37.654+0000 7f52f07a8640 1 -- 172.21.15.27:0/240170956 wait complete. 2024-04-25T13:50:37.916 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 570.0 s, 1960.0 tps, lat 51.120 ms stddev 59.623 2024-04-25T13:50:38.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:38 smithi122 ceph-mon[32843]: pgmap v641: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 MiB/s wr, 752 op/s 2024-04-25T13:50:38.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:38 smithi045 ceph-mon[32731]: pgmap v641: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 MiB/s wr, 752 op/s 2024-04-25T13:50:38.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:38 smithi027 ceph-mon[29140]: pgmap v641: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 MiB/s wr, 752 op/s 2024-04-25T13:50:40.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:40 smithi122 ceph-mon[32843]: pgmap v642: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 MiB/s wr, 564 op/s 2024-04-25T13:50:40.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:40 smithi045 ceph-mon[32731]: pgmap v642: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 MiB/s wr, 564 op/s 2024-04-25T13:50:40.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:40 smithi027 ceph-mon[29140]: pgmap v642: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 MiB/s wr, 564 op/s 2024-04-25T13:50:42.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:50:42.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:50:42.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 -- 172.21.15.27:0/33332920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36141029f0 msgr2=0x7f36141036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 --2- 172.21.15.27:0/33332920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36141029f0 0x7f36141036f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3604009900 tx=0x7f360402a410 comp rx=0 tx=0).stop 2024-04-25T13:50:42.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 -- 172.21.15.27:0/33332920 shutdown_connections 2024-04-25T13:50:42.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 --2- 172.21.15.27:0/33332920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3614108930 0x7f361410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 --2- 172.21.15.27:0/33332920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3614103c30 0x7f3614104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 --2- 172.21.15.27:0/33332920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36141029f0 0x7f36141036f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 -- 172.21.15.27:0/33332920 >> 172.21.15.27:0/33332920 conn(0x7f36140fdc00 msgr2=0x7f3614100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:42.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 -- 172.21.15.27:0/33332920 shutdown_connections 2024-04-25T13:50:42.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.330+0000 7f3618dd5640 1 -- 172.21.15.27:0/33332920 wait complete. 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3618dd5640 1 Processor -- start 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3618dd5640 1 -- start start 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3618dd5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36141029f0 0x7f361419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3618dd5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3614103c30 0x7f361419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3618dd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3614108930 0x7f36141a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3618dd5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f361410f670 con 0x7f3614108930 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3618dd5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f361410f4f0 con 0x7f36141029f0 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3618dd5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f361410f7f0 con 0x7f3614103c30 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f361259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36141029f0 0x7f361419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f3612d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3614108930 0x7f36141a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f361259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36141029f0 0x7f361419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38224/0 (socket says 172.21.15.27:38224) 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.331+0000 7f361259c640 1 -- 172.21.15.27:0/4248312476 learned_addr learned my addr 172.21.15.27:0/4248312476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.332+0000 7f360bfff640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3614103c30 0x7f361419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.332+0000 7f361259c640 1 -- 172.21.15.27:0/4248312476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3614103c30 msgr2=0x7f361419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.332+0000 7f361259c640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3614103c30 0x7f361419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.332+0000 7f361259c640 1 -- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3614108930 msgr2=0x7f36141a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.332+0000 7f361259c640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3614108930 0x7f36141a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.332+0000 7f361259c640 1 -- 172.21.15.27:0/4248312476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36141a1e40 con 0x7f36141029f0 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.332+0000 7f3612d9d640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3614108930 0x7f36141a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.332+0000 7f360bfff640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3614103c30 0x7f361419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.333+0000 7f361259c640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36141029f0 0x7f361419a300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f360402a930 tx=0x7f3604031a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.333+0000 7f360b7fe640 1 -- 172.21.15.27:0/4248312476 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604048070 con 0x7f36141029f0 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.333+0000 7f360b7fe640 1 -- 172.21.15.27:0/4248312476 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f360403ab20 con 0x7f36141029f0 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.333+0000 7f360b7fe640 1 -- 172.21.15.27:0/4248312476 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604035050 con 0x7f36141029f0 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.333+0000 7f3618dd5640 1 -- 172.21.15.27:0/4248312476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361410b440 con 0x7f36141029f0 2024-04-25T13:50:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.333+0000 7f3618dd5640 1 -- 172.21.15.27:0/4248312476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361410b960 con 0x7f36141029f0 2024-04-25T13:50:42.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.336+0000 7f360b7fe640 1 -- 172.21.15.27:0/4248312476 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f360404b070 con 0x7f36141029f0 2024-04-25T13:50:42.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.337+0000 7f360b7fe640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36000afb70 0x7f36000b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.337+0000 7f360b7fe640 1 -- 172.21.15.27:0/4248312476 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3604102960 con 0x7f36141029f0 2024-04-25T13:50:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.337+0000 7f360bfff640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36000afb70 0x7f36000b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.337+0000 7f361f922740 1 Processor -- start 2024-04-25T13:50:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.337+0000 7f361f922740 1 -- start start 2024-04-25T13:50:42.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.337+0000 7f360bfff640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36000afb70 0x7f36000b2080 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f361419b3a0 tx=0x7f35fc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361f922740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca99390 0x55aa3ce3aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361f922740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca704d0 0x55aa3ce3afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361f922740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 0x55aa3ce3bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361f922740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aa3cd9fd80 con 0x55aa3ca704d0 2024-04-25T13:50:42.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361f922740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55aa3cd598b0 con 0x55aa3ca99390 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361f922740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55aa3cd9bd40 con 0x55aa3ca786b0 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca99390 0x55aa3ce3aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f360bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca704d0 0x55aa3ce3afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca99390 0x55aa3ce3aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38226/0 (socket says 172.21.15.27:38226) 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f360bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca704d0 0x55aa3ce3afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33638/0 (socket says 172.21.15.27:33638) 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f361259c640 1 -- 172.21.15.27:0/709999061 learned_addr learned my addr 172.21.15.27:0/709999061 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.338+0000 7f3612d9d640 1 --2- 172.21.15.27:0/709999061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 0x55aa3ce3bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.339+0000 7f360bfff640 1 -- 172.21.15.27:0/709999061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 msgr2=0x55aa3ce3bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:42.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.339+0000 7f360bfff640 1 --2- 172.21.15.27:0/709999061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 0x55aa3ce3bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.339+0000 7f360bfff640 1 -- 172.21.15.27:0/709999061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca99390 msgr2=0x55aa3ce3aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:42.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.339+0000 7f360bfff640 1 --2- 172.21.15.27:0/709999061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca99390 0x55aa3ce3aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.339+0000 7f360bfff640 1 -- 172.21.15.27:0/709999061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa3cd2aa40 con 0x55aa3ca704d0 2024-04-25T13:50:42.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.340+0000 7f360bfff640 1 --2- 172.21.15.27:0/709999061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca704d0 0x55aa3ce3afe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00bfd0 tx=0x7f35fc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36af944427a0a4b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:42.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.340+0000 7f3608ff9640 1 -- 172.21.15.27:0/709999061 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc004220 con 0x55aa3ca704d0 2024-04-25T13:50:42.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.340+0000 7f3608ff9640 1 -- 172.21.15.27:0/709999061 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35fc0043c0 con 0x55aa3ca704d0 2024-04-25T13:50:42.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.340+0000 7f3608ff9640 1 -- 172.21.15.27:0/709999061 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc01cb70 con 0x55aa3ca704d0 2024-04-25T13:50:42.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.342+0000 7f361f922740 1 -- 172.21.15.27:0/709999061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca704d0 msgr2=0x55aa3ce3afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.342+0000 7f361f922740 1 --2- 172.21.15.27:0/709999061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca704d0 0x55aa3ce3afe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00bfd0 tx=0x7f35fc013000 comp rx=0 tx=0).stop 2024-04-25T13:50:42.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.342+0000 7f361f922740 1 -- 172.21.15.27:0/709999061 shutdown_connections 2024-04-25T13:50:42.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.342+0000 7f361f922740 1 --2- 172.21.15.27:0/709999061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 0x55aa3ce3bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.342+0000 7f361f922740 1 --2- 172.21.15.27:0/709999061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca704d0 0x55aa3ce3afe0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.342+0000 7f361f922740 1 --2- 172.21.15.27:0/709999061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca99390 0x55aa3ce3aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.342+0000 7f361f922740 1 -- 172.21.15.27:0/709999061 >> 172.21.15.27:0/709999061 conn(0x55aa3ca8d4e0 msgr2=0x55aa3ce36270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:42.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.342+0000 7f361f922740 1 -- 172.21.15.27:0/709999061 shutdown_connections 2024-04-25T13:50:42.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.343+0000 7f361f922740 1 -- 172.21.15.27:0/709999061 wait complete. 2024-04-25T13:50:42.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.343+0000 7f361f922740 1 -- start start 2024-04-25T13:50:42.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.343+0000 7f361f922740 1 Processor -- start 2024-04-25T13:50:42.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.343+0000 7f361f922740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce44780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361f922740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361f922740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca99390 0x55aa3ce4bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce44780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce44780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38234/0 (socket says 172.21.15.27:38234) 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f360bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f360bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33640/0 (socket says 172.21.15.27:33640) 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361259c640 1 -- 172.21.15.27:0/2938251840 learned_addr learned my addr 172.21.15.27:0/2938251840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aa3ce3c780 con 0x55aa3ca786b0 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55aa3cd26d10 con 0x55aa3ca704d0 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55aa3ce3c900 con 0x55aa3ca99390 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361f922740 10 client.0 _ll_register_callbacks cb 0x55aa3cd53210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:42.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f3612d9d640 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca99390 0x55aa3ce4bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361259c640 1 -- 172.21.15.27:0/2938251840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca99390 msgr2=0x55aa3ce4bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361259c640 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca99390 0x55aa3ce4bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361259c640 1 -- 172.21.15.27:0/2938251840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca786b0 msgr2=0x55aa3ce44cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361259c640 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.344+0000 7f361259c640 1 -- 172.21.15.27:0/2938251840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa3cb7b7d0 con 0x55aa3ca704d0 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.345+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35fc0043c0 con 0x55aa3ca704d0 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.345+0000 7f3612d9d640 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca99390 0x55aa3ce4bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.345+0000 7f361259c640 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce44780 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3604100800 tx=0x7f36040037d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5210a80abecf8ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.345+0000 7f35f9ffb640 10 client.27026 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.345+0000 7f35f9ffb640 1 -- 172.21.15.27:0/2938251840 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36040d3cd0 con 0x55aa3ca704d0 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.345+0000 7f35f9ffb640 1 -- 172.21.15.27:0/2938251840 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f36040cb180 con 0x55aa3ca704d0 2024-04-25T13:50:42.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.345+0000 7f35f9ffb640 1 -- 172.21.15.27:0/2938251840 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f36040cb380 con 0x55aa3ca704d0 2024-04-25T13:50:42.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f361f922740 10 client.27026 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:42.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=19}) -- 0x55aa3cab13f0 con 0x55aa3ca704d0 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f35f9ffb640 1 -- 172.21.15.27:0/2938251840 <== mon.2 v2:172.21.15.45:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f36040d3840 con 0x55aa3ca704d0 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f361f922740 10 client.27026 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f361f922740 10 client.27026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f361f922740 20 client.27026 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f361f922740 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa3ca57f60 0x55aa3ce51ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f361f922740 4 client.27026 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.346+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa3cdad990 con 0x55aa3ca57f60 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.347+0000 7f360bfff640 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa3ca57f60 0x55aa3ce51ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.347+0000 7f35f9ffb640 10 client.27026 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.347+0000 7f360bfff640 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa3ca57f60 0x55aa3ce51ec0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4eb6e07555ba59ce server_cookie=edd44854e0642da9 in_seq=0 out_seq=0 2024-04-25T13:50:42.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.348+0000 7f35f9ffb640 1 -- 172.21.15.27:0/2938251840 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55aa3cdad990 con 0x55aa3ca57f60 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.348+0000 7f35f9ffb640 10 client.27026 handle_command_reply: tid=0 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.348+0000 7f361f922740 1 client.27026 shutdown 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.348+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 msgr2=0x55aa3ce44780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.348+0000 7f361f922740 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce44780 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3604100800 tx=0x7f36040037d0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.348+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 shutdown_connections 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.348+0000 7f361f922740 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa3ca57f60 0x55aa3ce51ec0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.349+0000 7f35f9ffb640 0 client.27026 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.349+0000 7f361f922740 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca99390 0x55aa3ce4bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.349+0000 7f361f922740 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.349+0000 7f361f922740 1 --2- 172.21.15.27:0/2938251840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce44780 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.349+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 >> 172.21.15.27:0/2938251840 conn(0x55aa3ca8d4e0 msgr2=0x55aa3ce353f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:42.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.349+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 shutdown_connections 2024-04-25T13:50:42.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.349+0000 7f361f922740 1 -- 172.21.15.27:0/2938251840 wait complete. 2024-04-25T13:50:42.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.349+0000 7f361f922740 20 client.27026 trim_cache size 0 max 16384 2024-04-25T13:50:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361f922740 1 Processor -- start 2024-04-25T13:50:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361f922740 1 -- start start 2024-04-25T13:50:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361f922740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce43610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361f922740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca704d0 0x55aa3ce4bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361f922740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361f922740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f36141a7110 con 0x55aa3ca57f60 2024-04-25T13:50:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361f922740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f36141a9990 con 0x55aa3ca786b0 2024-04-25T13:50:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361f922740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f36141a7780 con 0x55aa3ca704d0 2024-04-25T13:50:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce43610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f3612d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce43610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33652/0 (socket says 172.21.15.27:33652) 2024-04-25T13:50:42.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f361259c640 1 -- 172.21.15.27:0/623523340 learned_addr learned my addr 172.21.15.27:0/623523340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.375+0000 7f360bfff640 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca704d0 0x55aa3ce4bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.376+0000 7f3612d9d640 1 -- 172.21.15.27:0/623523340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca704d0 msgr2=0x55aa3ce4bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.376+0000 7f3612d9d640 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca704d0 0x55aa3ce4bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.376+0000 7f3612d9d640 1 -- 172.21.15.27:0/623523340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 msgr2=0x55aa3ce43610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.376+0000 7f3612d9d640 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce43610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.376+0000 7f3612d9d640 1 -- 172.21.15.27:0/623523340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa3ca710f0 con 0x55aa3ca786b0 2024-04-25T13:50:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.376+0000 7f360bfff640 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca704d0 0x55aa3ce4bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:42.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.377+0000 7f3612d9d640 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44350 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55aa3ce31b40 tx=0x7f360c009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44435606307d3c5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.378+0000 7f35fbfff640 1 -- 172.21.15.27:0/623523340 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c01b070 con 0x55aa3ca786b0 2024-04-25T13:50:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.378+0000 7f35fbfff640 1 -- 172.21.15.27:0/623523340 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f360c007c70 con 0x55aa3ca786b0 2024-04-25T13:50:42.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.378+0000 7f35fbfff640 1 -- 172.21.15.27:0/623523340 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c016610 con 0x55aa3ca786b0 2024-04-25T13:50:42.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 -- 172.21.15.27:0/623523340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca786b0 msgr2=0x55aa3ce44350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44350 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55aa3ce31b40 tx=0x7f360c009650 comp rx=0 tx=0).stop 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 -- 172.21.15.27:0/623523340 shutdown_connections 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca786b0 0x55aa3ce44350 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca704d0 0x55aa3ce4bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 --2- 172.21.15.27:0/623523340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce43610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 -- 172.21.15.27:0/623523340 >> 172.21.15.27:0/623523340 conn(0x55aa3ca8d4e0 msgr2=0x55aa3ce42f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 -- 172.21.15.27:0/623523340 shutdown_connections 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.379+0000 7f361f922740 1 -- 172.21.15.27:0/623523340 wait complete. 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.380+0000 7f361f922740 1 -- start start 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.380+0000 7f361f922740 1 Processor -- start 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.380+0000 7f361f922740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce48bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.380+0000 7f361f922740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce49130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.380+0000 7f361f922740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 0x55aa3cec0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f360bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce49130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f360bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce49130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38248/0 (socket says 172.21.15.27:38248) 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f361259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce48bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f361259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce48bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33660/0 (socket says 172.21.15.27:33660) 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f360bfff640 1 -- 172.21.15.27:0/1418992841 learned_addr learned my addr 172.21.15.27:0/1418992841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f36141a7e50 con 0x55aa3ca57f60 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f36141a92c0 con 0x55aa3ca704d0 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f36141a8520 con 0x55aa3ca786b0 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f361f922740 10 client.0 _ll_register_callbacks cb 0x7f36041003d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:42.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f3612d9d640 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 0x55aa3cec0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f360bfff640 1 -- 172.21.15.27:0/1418992841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 msgr2=0x55aa3cec0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f360bfff640 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 0x55aa3cec0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f360bfff640 1 -- 172.21.15.27:0/1418992841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 msgr2=0x55aa3ce48bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f360bfff640 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce48bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.381+0000 7f360bfff640 1 -- 172.21.15.27:0/1418992841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa3cd604b0 con 0x55aa3ca704d0 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f360bfff640 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce49130 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55aa3ce3a780 tx=0x7f35fc01cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f99acef2621be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f35f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f35f9ffb640 1 -- 172.21.15.27:0/1418992841 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc044060 con 0x55aa3ca704d0 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f360c007c70 con 0x55aa3ca704d0 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f35f9ffb640 1 -- 172.21.15.27:0/1418992841 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f35fc037130 con 0x55aa3ca704d0 2024-04-25T13:50:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f35f9ffb640 1 -- 172.21.15.27:0/1418992841 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f35fc01c720 con 0x55aa3ca704d0 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f361f922740 10 client.27032 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55aa3cec2030 con 0x55aa3ca704d0 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f35f9ffb640 1 -- 172.21.15.27:0/1418992841 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f360c007c70 con 0x55aa3ca704d0 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f35f9ffb640 1 -- 172.21.15.27:0/1418992841 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f35fc03e030 con 0x55aa3ca704d0 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f361f922740 10 client.27032 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f361f922740 10 client.27032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f361f922740 20 client.27032 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.382+0000 7f361f922740 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa3ca99390 0x55aa3cec4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.383+0000 7f361f922740 4 client.27032 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.383+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aa3cda7800 con 0x55aa3ca99390 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.383+0000 7f361259c640 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa3ca99390 0x55aa3cec4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.383+0000 7f35f9ffb640 10 client.27032 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361259c640 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa3ca99390 0x55aa3cec4810 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e34c79355624e39b server_cookie=15f41fa2c4c1e64a in_seq=0 out_seq=0 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f35f9ffb640 1 -- 172.21.15.27:0/1418992841 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55aa3cda7800 con 0x55aa3ca99390 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f35f9ffb640 10 client.27032 handle_command_reply: tid=0 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 client.27032 shutdown 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 msgr2=0x55aa3ce49130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce49130 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55aa3ce3a780 tx=0x7f35fc01cc40 comp rx=0 tx=0).stop 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 shutdown_connections 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa3ca99390 0x55aa3cec4810 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa3ca786b0 0x55aa3cec0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3ca704d0 0x55aa3ce49130 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f35f9ffb640 0 client.27032 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 --2- 172.21.15.27:0/1418992841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa3ca57f60 0x55aa3ce48bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 >> 172.21.15.27:0/1418992841 conn(0x55aa3ca8d4e0 msgr2=0x55aa3ce390f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 shutdown_connections 2024-04-25T13:50:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 1 -- 172.21.15.27:0/1418992841 wait complete. 2024-04-25T13:50:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.384+0000 7f361f922740 20 client.27032 trim_cache size 0 max 16384 2024-04-25T13:50:42.389 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:50:42.389 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:50:42.389 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:50:42.389 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:50:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 -- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36000afb70 msgr2=0x7f36000b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36000afb70 0x7f36000b2080 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f361419b3a0 tx=0x7f35fc00b390 comp rx=0 tx=0).stop 2024-04-25T13:50:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 -- 172.21.15.27:0/4248312476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36141029f0 msgr2=0x7f361419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36141029f0 0x7f361419a300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f360402a930 tx=0x7f3604031a50 comp rx=0 tx=0).stop 2024-04-25T13:50:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 -- 172.21.15.27:0/4248312476 shutdown_connections 2024-04-25T13:50:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36000afb70 0x7f36000b2080 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3614108930 0x7f36141a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3614103c30 0x7f361419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 --2- 172.21.15.27:0/4248312476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f36141029f0 0x7f361419a300 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 -- 172.21.15.27:0/4248312476 >> 172.21.15.27:0/4248312476 conn(0x7f36140fdc00 msgr2=0x7f3614104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 -- 172.21.15.27:0/4248312476 shutdown_connections 2024-04-25T13:50:42.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.385+0000 7f3618dd5640 1 -- 172.21.15.27:0/4248312476 wait complete. 2024-04-25T13:50:42.399 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:50:42.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:50:42.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:42 smithi122 ceph-mon[32843]: pgmap v643: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 MiB/s wr, 578 op/s 2024-04-25T13:50:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.568+0000 7f71706f2640 1 -- 172.21.15.27:0/3449042269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168107910 msgr2=0x7f7168107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.568+0000 7f71706f2640 1 --2- 172.21.15.27:0/3449042269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168107910 0x7f7168107d20 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7160009be0 tx=0x7f716002a6e0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.569+0000 7f71706f2640 1 -- 172.21.15.27:0/3449042269 shutdown_connections 2024-04-25T13:50:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.569+0000 7f71706f2640 1 --2- 172.21.15.27:0/3449042269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168107910 0x7f7168107d20 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.569+0000 7f71706f2640 1 --2- 172.21.15.27:0/3449042269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71681027a0 0x7f7168102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.569+0000 7f71706f2640 1 --2- 172.21.15.27:0/3449042269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7168101d60 0x7f71681021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.569+0000 7f71706f2640 1 -- 172.21.15.27:0/3449042269 >> 172.21.15.27:0/3449042269 conn(0x7f71680fd330 msgr2=0x7f71680ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.569+0000 7f71706f2640 1 -- 172.21.15.27:0/3449042269 shutdown_connections 2024-04-25T13:50:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.569+0000 7f71706f2640 1 -- 172.21.15.27:0/3449042269 wait complete. 2024-04-25T13:50:42.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f71706f2640 1 Processor -- start 2024-04-25T13:50:42.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f71706f2640 1 -- start start 2024-04-25T13:50:42.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f71706f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168101d60 0x7f7168197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f71706f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681027a0 0x7f71681984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f71706f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7168107910 0x7f716819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f71706f2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f716810b540 con 0x7f7168101d60 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f71706f2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f716810b3c0 con 0x7f7168107910 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f71706f2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f716810b6c0 con 0x7f71681027a0 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f716e48e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168101d60 0x7f7168197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f716ec8f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7168107910 0x7f716819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f716e48e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168101d60 0x7f7168197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33686/0 (socket says 172.21.15.27:33686) 2024-04-25T13:50:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f716e48e640 1 -- 172.21.15.27:0/1022821752 learned_addr learned my addr 172.21.15.27:0/1022821752 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.570+0000 7f716dc8d640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681027a0 0x7f71681984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f716dc8d640 1 -- 172.21.15.27:0/1022821752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7168107910 msgr2=0x7f716819f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f716dc8d640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7168107910 0x7f716819f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f716dc8d640 1 -- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168101d60 msgr2=0x7f7168197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f716dc8d640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168101d60 0x7f7168197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f716dc8d640 1 -- 172.21.15.27:0/1022821752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716819fae0 con 0x7f71681027a0 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f716e48e640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168101d60 0x7f7168197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f716ec8f640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7168107910 0x7f716819f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f716dc8d640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681027a0 0x7f71681984e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7164005fd0 tx=0x7f716401d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f715f7fe640 1 -- 172.21.15.27:0/1022821752 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716401fe00 con 0x7f71681027a0 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f715f7fe640 1 -- 172.21.15.27:0/1022821752 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71640035f0 con 0x7f71681027a0 2024-04-25T13:50:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716819fdd0 con 0x7f71681027a0 2024-04-25T13:50:42.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f715f7fe640 1 -- 172.21.15.27:0/1022821752 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640216a0 con 0x7f71681027a0 2024-04-25T13:50:42.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.571+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71681a02b0 con 0x7f71681027a0 2024-04-25T13:50:42.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.573+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71681a0570 con 0x7f71681027a0 2024-04-25T13:50:42.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.574+0000 7f715f7fe640 1 -- 172.21.15.27:0/1022821752 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7164004020 con 0x7f71681027a0 2024-04-25T13:50:42.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.575+0000 7f715f7fe640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71500afaf0 0x7f71500b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:42.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.575+0000 7f716e48e640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71500afaf0 0x7f71500b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:42.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.575+0000 7f715f7fe640 1 -- 172.21.15.27:0/1022821752 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f71640efe80 con 0x7f71681027a0 2024-04-25T13:50:42.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.576+0000 7f716e48e640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71500afaf0 0x7f71500b2000 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7158009e00 tx=0x7f7158009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:42.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:42 smithi045 ceph-mon[32731]: pgmap v643: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 MiB/s wr, 578 op/s 2024-04-25T13:50:42.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.580+0000 7f715f7fe640 1 -- 172.21.15.27:0/1022821752 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f716401a520 con 0x7f71681027a0 2024-04-25T13:50:42.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:42 smithi027 ceph-mon[29140]: pgmap v643: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 MiB/s wr, 578 op/s 2024-04-25T13:50:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 575.0 s, 2214.0 tps, lat 44.994 ms stddev 55.505 2024-04-25T13:50:42.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.935+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71681a07e0 con 0x7f71681027a0 2024-04-25T13:50:42.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.936+0000 7f715f7fe640 1 -- 172.21.15.27:0/1022821752 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f71640b8a30 con 0x7f71681027a0 2024-04-25T13:50:42.936 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:50:42.936 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:50:42.936 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71500afaf0 msgr2=0x7f71500b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71500afaf0 0x7f71500b2000 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7158009e00 tx=0x7f7158009190 comp rx=0 tx=0).stop 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681027a0 msgr2=0x7f71681984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681027a0 0x7f71681984e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7164005fd0 tx=0x7f716401d6b0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 shutdown_connections 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71500afaf0 0x7f71500b2000 secure :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7158009e00 tx=0x7f7158009190 comp rx=0 tx=0).stop 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7168107910 0x7f716819f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681027a0 0x7f71681984e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 --2- 172.21.15.27:0/1022821752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7168101d60 0x7f7168197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 >> 172.21.15.27:0/1022821752 conn(0x7f71680fd330 msgr2=0x7f71680ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 shutdown_connections 2024-04-25T13:50:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:42.939+0000 7f71706f2640 1 -- 172.21.15.27:0/1022821752 wait complete. 2024-04-25T13:50:42.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:50:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.156+0000 7f06e534d640 1 -- 172.21.15.27:0/1831945907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010a420 msgr2=0x7f06e010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.156+0000 7f06e534d640 1 --2- 172.21.15.27:0/1831945907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010a420 0x7f06e010add0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009ca0 tx=0x7f06d802a8f0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.158+0000 7f06e534d640 1 -- 172.21.15.27:0/1831945907 shutdown_connections 2024-04-25T13:50:43.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.158+0000 7f06e534d640 1 --2- 172.21.15.27:0/1831945907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06e0110030 0x7f06e0110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.158+0000 7f06e534d640 1 --2- 172.21.15.27:0/1831945907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06e010b310 0x7f06e010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.158+0000 7f06e534d640 1 --2- 172.21.15.27:0/1831945907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010a420 0x7f06e010add0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.158+0000 7f06e534d640 1 -- 172.21.15.27:0/1831945907 >> 172.21.15.27:0/1831945907 conn(0x7f06e00704e0 msgr2=0x7f06e0072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:43.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.166+0000 7f06e534d640 1 -- 172.21.15.27:0/1831945907 shutdown_connections 2024-04-25T13:50:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.170+0000 7f06e534d640 1 -- 172.21.15.27:0/1831945907 wait complete. 2024-04-25T13:50:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.170+0000 7f06e534d640 1 Processor -- start 2024-04-25T13:50:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.176+0000 7f06e534d640 1 -- start start 2024-04-25T13:50:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.176+0000 7f06e534d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06e010a420 0x7f06e01a90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.176+0000 7f06e534d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010b310 0x7f06e01a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.176+0000 7f06e534d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06e0110030 0x7f06e01b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.176+0000 7f06e534d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f06e011e4e0 con 0x7f06e010b310 2024-04-25T13:50:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.176+0000 7f06e534d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f06e011e360 con 0x7f06e0110030 2024-04-25T13:50:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.176+0000 7f06e534d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f06e011e660 con 0x7f06e010a420 2024-04-25T13:50:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.178+0000 7f06de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010b310 0x7f06e01a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.178+0000 7f06de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010b310 0x7f06e01a95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33704/0 (socket says 172.21.15.27:33704) 2024-04-25T13:50:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.178+0000 7f06de7fc640 1 -- 172.21.15.27:0/3846995190 learned_addr learned my addr 172.21.15.27:0/3846995190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.179+0000 7f06de7fc640 1 -- 172.21.15.27:0/3846995190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06e010a420 msgr2=0x7f06e01a90b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:50:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.179+0000 7f06de7fc640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06e010a420 0x7f06e01a90b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.179+0000 7f06de7fc640 1 -- 172.21.15.27:0/3846995190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06e0110030 msgr2=0x7f06e01b0580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:50:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.179+0000 7f06de7fc640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06e0110030 0x7f06e01b0580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.179+0000 7f06de7fc640 1 -- 172.21.15.27:0/3846995190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e0111c90 con 0x7f06e010b310 2024-04-25T13:50:43.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.180+0000 7f06de7fc640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010b310 0x7f06e01a95f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f06d400c890 tx=0x7f06d400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:43.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.184+0000 7f06c7fff640 1 -- 172.21.15.27:0/3846995190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4004220 con 0x7f06e010b310 2024-04-25T13:50:43.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.184+0000 7f06e534d640 1 -- 172.21.15.27:0/3846995190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e0111f80 con 0x7f06e010b310 2024-04-25T13:50:43.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.184+0000 7f06e534d640 1 -- 172.21.15.27:0/3846995190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e01124d0 con 0x7f06e010b310 2024-04-25T13:50:43.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.184+0000 7f06c7fff640 1 -- 172.21.15.27:0/3846995190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d40043c0 con 0x7f06e010b310 2024-04-25T13:50:43.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.184+0000 7f06c7fff640 1 -- 172.21.15.27:0/3846995190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4004a00 con 0x7f06e010b310 2024-04-25T13:50:43.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.191+0000 7f06c7fff640 1 -- 172.21.15.27:0/3846995190 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f06d4004c60 con 0x7f06e010b310 2024-04-25T13:50:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.192+0000 7f06c7fff640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06c80afd00 0x7f06c80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.192+0000 7f06ec1ff740 1 Processor -- start 2024-04-25T13:50:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.192+0000 7f06c7fff640 1 -- 172.21.15.27:0/3846995190 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f06d40df880 con 0x7f06e010b310 2024-04-25T13:50:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.195+0000 7f06deffd640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06c80afd00 0x7f06c80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.199+0000 7f06ec1ff740 1 -- start start 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c336f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c336fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c3370a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06deffd640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06c80afd00 0x7f06c80b2210 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f06d8004af0 tx=0x7f06d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5561c32d3d80 con 0x5561c2fa44d0 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5561c328da20 con 0x5561c2fcd390 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5561c32cfd40 con 0x5561c2fac6b0 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06deffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c336f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06deffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c336f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38298/0 (socket says 172.21.15.27:38298) 2024-04-25T13:50:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.201+0000 7f06deffd640 1 -- 172.21.15.27:0/1467811246 learned_addr learned my addr 172.21.15.27:0/1467811246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:43.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.202+0000 7f06de7fc640 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c336fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.202+0000 7f06df7fe640 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c3370a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.202+0000 7f06deffd640 1 -- 172.21.15.27:0/1467811246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 msgr2=0x5561c3370a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:43.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.202+0000 7f06deffd640 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c3370a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.202+0000 7f06deffd640 1 -- 172.21.15.27:0/1467811246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 msgr2=0x5561c336fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:43.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.202+0000 7f06deffd640 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c336fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.202+0000 7f06deffd640 1 -- 172.21.15.27:0/1467811246 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561c3298c30 con 0x5561c2fcd390 2024-04-25T13:50:43.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.203+0000 7f06deffd640 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c336f810 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f06d8035000 tx=0x7f06d80095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3efa37083e7e05db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:43.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.203+0000 7f06c57fa640 1 -- 172.21.15.27:0/1467811246 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d804b070 con 0x5561c2fcd390 2024-04-25T13:50:43.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.203+0000 7f06c57fa640 1 -- 172.21.15.27:0/1467811246 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d8036040 con 0x5561c2fcd390 2024-04-25T13:50:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.204+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1467811246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 msgr2=0x5561c336f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.204+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c336f810 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f06d8035000 tx=0x7f06d80095c0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.205+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1467811246 shutdown_connections 2024-04-25T13:50:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.205+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c3370a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.205+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c336fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.205+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1467811246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c336f810 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.205+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1467811246 >> 172.21.15.27:0/1467811246 conn(0x5561c2fc14e0 msgr2=0x5561c336b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:43.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.209+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1467811246 shutdown_connections 2024-04-25T13:50:43.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.212+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1467811246 wait complete. 2024-04-25T13:50:43.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.212+0000 7f06ec1ff740 1 -- start start 2024-04-25T13:50:43.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.212+0000 7f06ec1ff740 1 Processor -- start 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.213+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.213+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c3379aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.213+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c3380a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.213+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5561c3371520 con 0x5561c2fa44d0 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.213+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5561c325adc0 con 0x5561c2fcd390 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.213+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5561c33716a0 con 0x5561c2fac6b0 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.213+0000 7f06ec1ff740 10 client.0 _ll_register_callbacks cb 0x5561c3276600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.215+0000 7f06deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.215+0000 7f06deffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33728/0 (socket says 172.21.15.27:33728) 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.215+0000 7f06deffd640 1 -- 172.21.15.27:0/1406333949 learned_addr learned my addr 172.21.15.27:0/1406333949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:43.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.215+0000 7f06df7fe640 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c3380a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.215+0000 7f06df7fe640 1 -- 172.21.15.27:0/1406333949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 msgr2=0x5561c3379aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:43.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.215+0000 7f06df7fe640 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c3379aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.215+0000 7f06df7fe640 1 -- 172.21.15.27:0/1406333949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 msgr2=0x5561c3379560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:43.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.216+0000 7f06df7fe640 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.216+0000 7f06df7fe640 1 -- 172.21.15.27:0/1406333949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561c2fe53f0 con 0x5561c2fcd390 2024-04-25T13:50:43.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.216+0000 7f06df7fe640 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c3380a00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f06d00113e0 tx=0x7f06d00118a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b171ada3f6b9e09c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:43.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.216+0000 7f06c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:50:43.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.220+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1406333949 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0012920 con 0x5561c2fcd390 2024-04-25T13:50:43.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.220+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1406333949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d8036040 con 0x5561c2fcd390 2024-04-25T13:50:43.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.220+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1406333949 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f06d0012ac0 con 0x5561c2fcd390 2024-04-25T13:50:43.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.222+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1406333949 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f06d0012cc0 con 0x5561c2fcd390 2024-04-25T13:50:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.227+0000 7f06ec1ff740 10 client.27044 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:43.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.227+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1406333949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5561c30af7d0 con 0x5561c2fcd390 2024-04-25T13:50:43.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.229+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1406333949 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d8036040 con 0x5561c2fcd390 2024-04-25T13:50:43.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.230+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1406333949 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f06d000f4e0 con 0x5561c2fcd390 2024-04-25T13:50:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.232+0000 7f06ec1ff740 10 client.27044 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.232+0000 7f06ec1ff740 10 client.27044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:43.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.232+0000 7f06ec1ff740 20 client.27044 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.232+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5561c2f8bf60 0x5561c33a9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.232+0000 7f06ec1ff740 4 client.27044 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.232+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1406333949 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561c32e1990 con 0x5561c2f8bf60 2024-04-25T13:50:43.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.235+0000 7f06deffd640 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5561c2f8bf60 0x5561c33a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.242+0000 7f06deffd640 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5561c2f8bf60 0x5561c33a9320 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285c31d458268ffa server_cookie=dfa7c3f5ffc8fd03 in_seq=0 out_seq=0 2024-04-25T13:50:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.243+0000 7f06c2ff5640 10 client.27044 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.248+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1406333949 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5561c32e1990 con 0x5561c2f8bf60 2024-04-25T13:50:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.248+0000 7f06c2ff5640 10 client.27044 handle_command_reply: tid=0 2024-04-25T13:50:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.248+0000 7f06ec1ff740 1 client.27044 shutdown 2024-04-25T13:50:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.248+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 msgr2=0x5561c3380a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.248+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c3380a00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f06d00113e0 tx=0x7f06d00118a0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.249+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1406333949 shutdown_connections 2024-04-25T13:50:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.249+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5561c2f8bf60 0x5561c33a9320 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.249+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fcd390 0x5561c3380a00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.249+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c3379aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.249+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1406333949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.249+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1406333949 >> 172.21.15.27:0/1406333949 conn(0x5561c2fc14e0 msgr2=0x5561c336a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.249+0000 7f06c2ff5640 0 client.27044 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:43.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.254+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1406333949 shutdown_connections 2024-04-25T13:50:43.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.255+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1406333949 wait complete. 2024-04-25T13:50:43.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.255+0000 7f06ec1ff740 20 client.27044 trim_cache size 0 max 16384 2024-04-25T13:50:43.275 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:43 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1022821752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:50:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.284+0000 7f06ec1ff740 1 Processor -- start 2024-04-25T13:50:43.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.284+0000 7f06ec1ff740 1 -- start start 2024-04-25T13:50:43.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.284+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.284+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fac6b0 0x5561c3380920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.284+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fcd390 0x5561c3379aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.284+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc00c4d0 con 0x5561c2fac6b0 2024-04-25T13:50:43.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.284+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc00ed50 con 0x5561c2fa44d0 2024-04-25T13:50:43.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.284+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc00cb40 con 0x5561c2fcd390 2024-04-25T13:50:43.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.285+0000 7f06deffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fac6b0 0x5561c3380920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06de7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fac6b0 0x5561c3380920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33730/0 (socket says 172.21.15.27:33730) 2024-04-25T13:50:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06de7fc640 1 -- 172.21.15.27:0/293663629 learned_addr learned my addr 172.21.15.27:0/293663629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06df7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fcd390 0x5561c3379aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06de7fc640 1 -- 172.21.15.27:0/293663629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fcd390 msgr2=0x5561c3379aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:43.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06de7fc640 1 --2- 172.21.15.27:0/293663629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fcd390 0x5561c3379aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06de7fc640 1 -- 172.21.15.27:0/293663629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 msgr2=0x5561c3379560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06de7fc640 1 --2- 172.21.15.27:0/293663629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06de7fc640 1 -- 172.21.15.27:0/293663629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561c3380e60 con 0x5561c2fac6b0 2024-04-25T13:50:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.286+0000 7f06df7fe640 1 --2- 172.21.15.27:0/293663629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fcd390 0x5561c3379aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.287+0000 7f06de7fc640 1 --2- 172.21.15.27:0/293663629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fac6b0 0x5561c3380920 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f06d40055a0 tx=0x7f06d40df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca351a5b9ffa624 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.287+0000 7f06c4ff9640 1 -- 172.21.15.27:0/293663629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d401d030 con 0x5561c2fac6b0 2024-04-25T13:50:43.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.288+0000 7f06c4ff9640 1 -- 172.21.15.27:0/293663629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d40be030 con 0x5561c2fac6b0 2024-04-25T13:50:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.288+0000 7f06c4ff9640 1 -- 172.21.15.27:0/293663629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d40c23d0 con 0x5561c2fac6b0 2024-04-25T13:50:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.289+0000 7f06ec1ff740 1 -- 172.21.15.27:0/293663629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fac6b0 msgr2=0x5561c3380920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:43.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.289+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/293663629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fac6b0 0x5561c3380920 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f06d40055a0 tx=0x7f06d40df040 comp rx=0 tx=0).stop 2024-04-25T13:50:43.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.291+0000 7f06ec1ff740 1 -- 172.21.15.27:0/293663629 shutdown_connections 2024-04-25T13:50:43.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.291+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/293663629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fcd390 0x5561c3379aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.291+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/293663629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fac6b0 0x5561c3380920 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.291+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/293663629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 0x5561c3379560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.291+0000 7f06ec1ff740 1 -- 172.21.15.27:0/293663629 >> 172.21.15.27:0/293663629 conn(0x5561c2fc14e0 msgr2=0x5561c3374290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:43.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.294+0000 7f06ec1ff740 1 -- 172.21.15.27:0/293663629 shutdown_connections 2024-04-25T13:50:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.294+0000 7f06ec1ff740 1 -- 172.21.15.27:0/293663629 wait complete. 2024-04-25T13:50:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.294+0000 7f06ec1ff740 1 -- start start 2024-04-25T13:50:43.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.295+0000 7f06ec1ff740 1 Processor -- start 2024-04-25T13:50:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.295+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 0x5561c337e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.295+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c337e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.295+0000 7f06ec1ff740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fcd390 0x5561c337f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.295+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc00d210 con 0x5561c2fcd390 2024-04-25T13:50:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.295+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc00e680 con 0x5561c2fa44d0 2024-04-25T13:50:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.295+0000 7f06ec1ff740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc00d8e0 con 0x5561c2fac6b0 2024-04-25T13:50:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.295+0000 7f06ec1ff740 10 client.0 _ll_register_callbacks cb 0x7f06d0009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:43.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.296+0000 7f06de7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c337e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.296+0000 7f06deffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 0x5561c337e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.296+0000 7f06de7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c337e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:52482/0 (socket says 172.21.15.27:52482) 2024-04-25T13:50:43.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.296+0000 7f06de7fc640 1 -- 172.21.15.27:0/1590623398 learned_addr learned my addr 172.21.15.27:0/1590623398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:43.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.297+0000 7f06de7fc640 1 -- 172.21.15.27:0/1590623398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 msgr2=0x5561c337e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.297+0000 7f06de7fc640 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 0x5561c337e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.297+0000 7f06de7fc640 1 -- 172.21.15.27:0/1590623398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fcd390 msgr2=0x5561c337f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.297+0000 7f06de7fc640 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fcd390 0x5561c337f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.297+0000 7f06de7fc640 1 -- 172.21.15.27:0/1590623398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561c337fbb0 con 0x5561c2fac6b0 2024-04-25T13:50:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.297+0000 7f06de7fc640 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c337e710 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f06d40dd3a0 tx=0x7f06d40ac0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=799058b3811f4353 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.297+0000 7f06c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.298+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1590623398 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d401d070 con 0x5561c2fac6b0 2024-04-25T13:50:43.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.298+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1590623398 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f06d40a6f90 con 0x5561c2fac6b0 2024-04-25T13:50:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.299+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1590623398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d40be030 con 0x5561c2fac6b0 2024-04-25T13:50:43.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.300+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1590623398 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f06d40a7d30 con 0x5561c2fac6b0 2024-04-25T13:50:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.302+0000 7f06ec1ff740 10 client.27094 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:43.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.302+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1590623398 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5561c3411740 con 0x5561c2fac6b0 2024-04-25T13:50:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.302+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1590623398 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d40be030 con 0x5561c2fac6b0 2024-04-25T13:50:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.303+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1590623398 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f06d40a6b00 con 0x5561c2fac6b0 2024-04-25T13:50:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.303+0000 7f06ec1ff740 10 client.27094 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.303+0000 7f06ec1ff740 10 client.27094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.303+0000 7f06ec1ff740 20 client.27094 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:43.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.303+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5561c2f613a0 0x5561c3414100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.303+0000 7f06ec1ff740 4 client.27094 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:50:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.303+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1590623398 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5561c32db800 con 0x5561c2f613a0 2024-04-25T13:50:43.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.303+0000 7f06deffd640 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5561c2f613a0 0x5561c3414100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:43.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.304+0000 7f06deffd640 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5561c2f613a0 0x5561c3414100 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a4aa498abcff6fa server_cookie=ee4d86980e4c7683 in_seq=0 out_seq=0 2024-04-25T13:50:43.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.304+0000 7f06c2ff5640 10 client.27094 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:43.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.304+0000 7f06c2ff5640 1 -- 172.21.15.27:0/1590623398 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5561c32db800 con 0x5561c2f613a0 2024-04-25T13:50:43.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.305+0000 7f06c2ff5640 10 client.27094 handle_command_reply: tid=0 2024-04-25T13:50:43.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.305+0000 7f06ec1ff740 1 client.27094 shutdown 2024-04-25T13:50:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.309+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1590623398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 msgr2=0x5561c337e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.309+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c337e710 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f06d40dd3a0 tx=0x7f06d40ac0e0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.310+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1590623398 shutdown_connections 2024-04-25T13:50:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.310+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5561c2f613a0 0x5561c3414100 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.310+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5561c2fcd390 0x5561c337f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.310+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5561c2fac6b0 0x5561c337e710 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.310+0000 7f06ec1ff740 1 --2- 172.21.15.27:0/1590623398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561c2fa44d0 0x5561c337e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.310+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1590623398 >> 172.21.15.27:0/1590623398 conn(0x5561c2fc14e0 msgr2=0x5561c3374290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.310+0000 7f06c2ff5640 0 client.27094 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.311+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1590623398 shutdown_connections 2024-04-25T13:50:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.311+0000 7f06ec1ff740 1 -- 172.21.15.27:0/1590623398 wait complete. 2024-04-25T13:50:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.312+0000 7f06ec1ff740 20 client.27094 trim_cache size 0 max 16384 2024-04-25T13:50:43.319 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:50:43.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.324+0000 7f06e534d640 1 -- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06c80afd00 msgr2=0x7f06c80b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:43.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.324+0000 7f06e534d640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06c80afd00 0x7f06c80b2210 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f06d8004af0 tx=0x7f06d8005fd0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.324+0000 7f06e534d640 1 -- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010b310 msgr2=0x7f06e01a95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:43.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.324+0000 7f06e534d640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010b310 0x7f06e01a95f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f06d400c890 tx=0x7f06d400cd50 comp rx=0 tx=0).stop 2024-04-25T13:50:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.326+0000 7f06e534d640 1 -- 172.21.15.27:0/3846995190 shutdown_connections 2024-04-25T13:50:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.326+0000 7f06e534d640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f06c80afd00 0x7f06c80b2210 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.326+0000 7f06e534d640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06e0110030 0x7f06e01b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.326+0000 7f06e534d640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06e010b310 0x7f06e01a95f0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.326+0000 7f06e534d640 1 --2- 172.21.15.27:0/3846995190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06e010a420 0x7f06e01a90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.326+0000 7f06e534d640 1 -- 172.21.15.27:0/3846995190 >> 172.21.15.27:0/3846995190 conn(0x7f06e00704e0 msgr2=0x7f06e010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:43.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.327+0000 7f06e534d640 1 -- 172.21.15.27:0/3846995190 shutdown_connections 2024-04-25T13:50:43.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:43.327+0000 7f06e534d640 1 -- 172.21.15.27:0/3846995190 wait complete. 2024-04-25T13:50:43.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:43 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1022821752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:50:43.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:43 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1022821752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:50:44.349 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:50:44.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:50:44.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.528+0000 7fa53418f640 1 -- 172.21.15.27:0/3780427489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c1029f0 msgr2=0x7fa52c1036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.528+0000 7fa53418f640 1 --2- 172.21.15.27:0/3780427489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c1029f0 0x7fa52c1036f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa528009900 tx=0x7fa52802a410 comp rx=0 tx=0).stop 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 -- 172.21.15.27:0/3780427489 shutdown_connections 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 --2- 172.21.15.27:0/3780427489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa52c108930 0x7fa52c10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 --2- 172.21.15.27:0/3780427489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52c103c30 0x7fa52c104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 --2- 172.21.15.27:0/3780427489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c1029f0 0x7fa52c1036f0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 -- 172.21.15.27:0/3780427489 >> 172.21.15.27:0/3780427489 conn(0x7fa52c0fdc00 msgr2=0x7fa52c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 -- 172.21.15.27:0/3780427489 shutdown_connections 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 -- 172.21.15.27:0/3780427489 wait complete. 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 Processor -- start 2024-04-25T13:50:44.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.529+0000 7fa53418f640 1 -- start start 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53418f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52c1029f0 0x7fa52c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53418f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa52c103c30 0x7fa52c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53418f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c108930 0x7fa52c1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53418f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10e4f0 con 0x7fa52c108930 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53418f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10e370 con 0x7fa52c1029f0 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53418f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10e670 con 0x7fa52c103c30 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c108930 0x7fa52c1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c108930 0x7fa52c1a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38778/0 (socket says 172.21.15.27:38778) 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 -- 172.21.15.27:0/3899750540 learned_addr learned my addr 172.21.15.27:0/3899750540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53172a640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa52c103c30 0x7fa52c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa531f2b640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52c1029f0 0x7fa52c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 -- 172.21.15.27:0/3899750540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa52c103c30 msgr2=0x7fa52c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa52c103c30 0x7fa52c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 -- 172.21.15.27:0/3899750540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52c1029f0 msgr2=0x7fa52c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52c1029f0 0x7fa52c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 -- 172.21.15.27:0/3899750540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52c1a1d80 con 0x7fa52c108930 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53172a640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa52c103c30 0x7fa52c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.530+0000 7fa53272c640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c108930 0x7fa52c1a1710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa52400eb10 tx=0x7fa52400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.531+0000 7fa522ffd640 1 -- 172.21.15.27:0/3899750540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524004b20 con 0x7fa52c108930 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.531+0000 7fa53418f640 1 -- 172.21.15.27:0/3899750540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52c1a2070 con 0x7fa52c108930 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.531+0000 7fa522ffd640 1 -- 172.21.15.27:0/3899750540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa524004cc0 con 0x7fa52c108930 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.531+0000 7fa522ffd640 1 -- 172.21.15.27:0/3899750540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524010690 con 0x7fa52c108930 2024-04-25T13:50:44.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.531+0000 7fa53418f640 1 -- 172.21.15.27:0/3899750540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa52c1a25c0 con 0x7fa52c108930 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.533+0000 7fa53b042740 1 Processor -- start 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.533+0000 7fa522ffd640 1 -- 172.21.15.27:0/3899750540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa524005020 con 0x7fa52c108930 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53b042740 1 -- start start 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53b042740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6014a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53b042740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc601a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53b042740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc608920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53b042740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5639bc566b00 con 0x5639bc260410 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53b042740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5639bc502920 con 0x5639bc23f730 2024-04-25T13:50:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53b042740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5639bc562ac0 con 0x5639bc237550 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53172a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6014a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53172a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6014a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38790/0 (socket says 172.21.15.27:38790) 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53172a640 1 -- 172.21.15.27:0/1732373621 learned_addr learned my addr 172.21.15.27:0/1732373621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa522ffd640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5180afaf0 0x7fa5180b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa531f2b640 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc601a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.534+0000 7fa53172a640 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc608920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa53272c640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5180afaf0 0x7fa5180b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa531f2b640 1 -- 172.21.15.27:0/1732373621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 msgr2=0x5639bc608920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa531f2b640 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc608920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa531f2b640 1 -- 172.21.15.27:0/1732373621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 msgr2=0x5639bc6014a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa531f2b640 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6014a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa531f2b640 1 -- 172.21.15.27:0/1732373621 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639bc3407e0 con 0x5639bc237550 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa53172a640 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc608920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa522ffd640 1 -- 172.21.15.27:0/3899750540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5240df140 con 0x7fa52c108930 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa531f2b640 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc601a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa52802a930 tx=0x7fa528000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6e2424385526a62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa53272c640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5180afaf0 0x7fa5180b2000 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa5240a75e0 tx=0x7fa5240c3040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa517fff640 1 -- 172.21.15.27:0/1732373621 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa528035030 con 0x5639bc237550 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa517fff640 1 -- 172.21.15.27:0/1732373621 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa528040070 con 0x5639bc237550 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.535+0000 7fa517fff640 1 -- 172.21.15.27:0/1732373621 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa528003870 con 0x5639bc237550 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.536+0000 7fa53b042740 1 -- 172.21.15.27:0/1732373621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 msgr2=0x5639bc601a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.536+0000 7fa53b042740 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc601a20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa52802a930 tx=0x7fa528000bc0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.536+0000 7fa53b042740 1 -- 172.21.15.27:0/1732373621 shutdown_connections 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.536+0000 7fa53b042740 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc608920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.537+0000 7fa53b042740 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc601a20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.537+0000 7fa53b042740 1 --2- 172.21.15.27:0/1732373621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.537+0000 7fa53b042740 1 -- 172.21.15.27:0/1732373621 >> 172.21.15.27:0/1732373621 conn(0x5639bc254560 msgr2=0x5639bc5fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.537+0000 7fa53b042740 1 -- 172.21.15.27:0/1732373621 shutdown_connections 2024-04-25T13:50:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.537+0000 7fa53b042740 1 -- 172.21.15.27:0/1732373621 wait complete. 2024-04-25T13:50:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.537+0000 7fa53b042740 1 -- start start 2024-04-25T13:50:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.537+0000 7fa53b042740 1 Processor -- start 2024-04-25T13:50:44.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.539+0000 7fa53b042740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc237550 0x5639bc60e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.539+0000 7fa53b042740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc23f730 0x5639bc60ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.539+0000 7fa53b042740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc260410 0x5639bc60b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa53b042740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5639bc609310 con 0x5639bc237550 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa53b042740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5639bc5005d0 con 0x5639bc260410 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa531f2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc23f730 0x5639bc60ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa53b042740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5639bc609490 con 0x5639bc23f730 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa53b042740 10 client.0 _ll_register_callbacks cb 0x5639bc4ee080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa53172a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc237550 0x5639bc60e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa53172a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc237550 0x5639bc60e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38806/0 (socket says 172.21.15.27:38806) 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa53172a640 1 -- 172.21.15.27:0/2748467190 learned_addr learned my addr 172.21.15.27:0/2748467190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.541+0000 7fa53172a640 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc260410 0x5639bc60b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.541+0000 7fa53172a640 1 -- 172.21.15.27:0/2748467190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc23f730 msgr2=0x5639bc60ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.540+0000 7fa531f2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc23f730 0x5639bc60ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46152/0 (socket says 172.21.15.27:46152) 2024-04-25T13:50:44.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.541+0000 7fa53172a640 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc23f730 0x5639bc60ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.541+0000 7fa53172a640 1 -- 172.21.15.27:0/2748467190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc260410 msgr2=0x5639bc60b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.541+0000 7fa53172a640 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc260410 0x5639bc60b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.541+0000 7fa53172a640 1 -- 172.21.15.27:0/2748467190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639bc3a0f70 con 0x5639bc237550 2024-04-25T13:50:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.541+0000 7fa531f2b640 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc23f730 0x5639bc60ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-25T13:50:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.550+0000 7fa53172a640 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc237550 0x5639bc60e6b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa51c014e60 tx=0x7fa51c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7373d87fa77ce8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.551+0000 7fa5157fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:50:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.555+0000 7fa5157fa640 1 -- 172.21.15.27:0/2748467190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c004250 con 0x5639bc237550 2024-04-25T13:50:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.555+0000 7fa5157fa640 1 -- 172.21.15.27:0/2748467190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa51c0043f0 con 0x5639bc237550 2024-04-25T13:50:44.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:44 smithi122 ceph-mon[32843]: pgmap v644: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 756 op/s 2024-04-25T13:50:44.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.558+0000 7fa53b042740 1 -- 172.21.15.27:0/2748467190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa528040070 con 0x5639bc237550 2024-04-25T13:50:44.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.563+0000 7fa5157fa640 1 -- 172.21.15.27:0/2748467190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa51c027050 con 0x5639bc237550 2024-04-25T13:50:44.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.564+0000 7fa53b042740 10 client.17361 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:44.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.564+0000 7fa53b042740 1 -- 172.21.15.27:0/2748467190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5639bc238170 con 0x5639bc237550 2024-04-25T13:50:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.566+0000 7fa5157fa640 1 -- 172.21.15.27:0/2748467190 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa528040070 con 0x5639bc237550 2024-04-25T13:50:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.567+0000 7fa5157fa640 1 -- 172.21.15.27:0/2748467190 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa51c021930 con 0x5639bc237550 2024-04-25T13:50:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.567+0000 7fa53b042740 10 client.17361 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.567+0000 7fa53b042740 10 client.17361 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.567+0000 7fa53b042740 20 client.17361 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.567+0000 7fa53b042740 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5639bc21f1f0 0x5639bc621960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.567+0000 7fa53b042740 4 client.17361 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.567+0000 7fa53b042740 1 -- 172.21.15.27:0/2748467190 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639bc574710 con 0x5639bc21f1f0 2024-04-25T13:50:44.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.569+0000 7fa531f2b640 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5639bc21f1f0 0x5639bc621960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.570+0000 7fa531f2b640 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5639bc21f1f0 0x5639bc621960 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6987e3248a850a8b server_cookie=d4cf8714b62035da in_seq=0 out_seq=0 2024-04-25T13:50:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.571+0000 7fa5157fa640 10 client.17361 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.572+0000 7fa5157fa640 1 -- 172.21.15.27:0/2748467190 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5639bc574710 con 0x5639bc21f1f0 2024-04-25T13:50:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.572+0000 7fa5157fa640 10 client.17361 handle_command_reply: tid=0 2024-04-25T13:50:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.572+0000 7fa53b042740 1 client.17361 shutdown 2024-04-25T13:50:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.573+0000 7fa53b042740 1 -- 172.21.15.27:0/2748467190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc237550 msgr2=0x5639bc60e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.573+0000 7fa53b042740 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc237550 0x5639bc60e6b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa51c014e60 tx=0x7fa51c005f90 comp rx=0 tx=0).stop 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 1 -- 172.21.15.27:0/2748467190 shutdown_connections 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5639bc21f1f0 0x5639bc621960 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc260410 0x5639bc60b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc23f730 0x5639bc60ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 1 --2- 172.21.15.27:0/2748467190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc237550 0x5639bc60e6b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 1 -- 172.21.15.27:0/2748467190 >> 172.21.15.27:0/2748467190 conn(0x5639bc254560 msgr2=0x5639bc5fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa5157fa640 0 client.17361 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 1 -- 172.21.15.27:0/2748467190 shutdown_connections 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 1 -- 172.21.15.27:0/2748467190 wait complete. 2024-04-25T13:50:44.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.574+0000 7fa53b042740 20 client.17361 trim_cache size 0 max 16384 2024-04-25T13:50:44.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:44 smithi045 ceph-mon[32731]: pgmap v644: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 756 op/s 2024-04-25T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.606+0000 7fa53b042740 1 Processor -- start 2024-04-25T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.606+0000 7fa53b042740 1 -- start start 2024-04-25T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.606+0000 7fa53b042740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc605790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.606+0000 7fa53b042740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc605cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53b042740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53b042740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1a7060 con 0x5639bc260410 2024-04-25T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53b042740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1a98e0 con 0x5639bc23f730 2024-04-25T13:50:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53b042740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1a76d0 con 0x5639bc237550 2024-04-25T13:50:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53172a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc605790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa531f2b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc605cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa531f2b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc605cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52122/0 (socket says 172.21.15.27:52122) 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53172a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc605790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46156/0 (socket says 172.21.15.27:46156) 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa531f2b640 1 -- 172.21.15.27:0/2954103506 learned_addr learned my addr 172.21.15.27:0/2954103506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53172a640 1 --2- 172.21.15.27:0/2954103506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53172a640 1 -- 172.21.15.27:0/2954103506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 msgr2=0x5639bc605cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53172a640 1 --2- 172.21.15.27:0/2954103506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc605cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53172a640 1 -- 172.21.15.27:0/2954103506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 msgr2=0x5639bc6098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53172a640 1 --2- 172.21.15.27:0/2954103506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.607+0000 7fa53172a640 1 -- 172.21.15.27:0/2954103506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639bc61f350 con 0x5639bc237550 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.608+0000 7fa53172a640 1 --2- 172.21.15.27:0/2954103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc605790 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa51c014e30 tx=0x7fa51c004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbe40fa223efcc75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.608+0000 7fa5177fe640 1 -- 172.21.15.27:0/2954103506 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c003ee0 con 0x5639bc237550 2024-04-25T13:50:44.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.608+0000 7fa5177fe640 1 -- 172.21.15.27:0/2954103506 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c005070 con 0x5639bc237550 2024-04-25T13:50:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.608+0000 7fa5177fe640 1 -- 172.21.15.27:0/2954103506 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c003850 con 0x5639bc237550 2024-04-25T13:50:44.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 -- 172.21.15.27:0/2954103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 msgr2=0x5639bc605790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 --2- 172.21.15.27:0/2954103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc605790 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa51c014e30 tx=0x7fa51c004610 comp rx=0 tx=0).stop 2024-04-25T13:50:44.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 -- 172.21.15.27:0/2954103506 shutdown_connections 2024-04-25T13:50:44.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 --2- 172.21.15.27:0/2954103506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 --2- 172.21.15.27:0/2954103506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc605cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 --2- 172.21.15.27:0/2954103506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc605790 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 -- 172.21.15.27:0/2954103506 >> 172.21.15.27:0/2954103506 conn(0x5639bc254560 msgr2=0x5639bc60e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 -- 172.21.15.27:0/2954103506 shutdown_connections 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.609+0000 7fa53b042740 1 -- 172.21.15.27:0/2954103506 wait complete. 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.610+0000 7fa53b042740 1 -- start start 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.610+0000 7fa53b042740 1 Processor -- start 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.610+0000 7fa53b042740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc633080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.610+0000 7fa53b042740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc62fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.610+0000 7fa53b042740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6301a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53b042740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1a7da0 con 0x5639bc260410 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53b042740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1a9210 con 0x5639bc23f730 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc633080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa531f2b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc62fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc633080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46172/0 (socket says 172.21.15.27:46172) 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 -- 172.21.15.27:0/1964894593 learned_addr learned my addr 172.21.15.27:0/1964894593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53b042740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c1a8470 con 0x5639bc237550 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53b042740 10 client.0 _ll_register_callbacks cb 0x7fa51c014c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 -- 172.21.15.27:0/1964894593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 msgr2=0x5639bc62fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc62fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 -- 172.21.15.27:0/1964894593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 msgr2=0x5639bc6301a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6301a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 -- 172.21.15.27:0/1964894593 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639bc687970 con 0x5639bc237550 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa53172a640 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc633080 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa51c014d40 tx=0x7fa51c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=163fbf698d60a978 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.611+0000 7fa5157fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa5157fa640 1 -- 172.21.15.27:0/1964894593 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0035f0 con 0x5639bc237550 2024-04-25T13:50:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa5157fa640 1 -- 172.21.15.27:0/1964894593 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa51c003790 con 0x5639bc237550 2024-04-25T13:50:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 1 -- 172.21.15.27:0/1964894593 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa51c005070 con 0x5639bc237550 2024-04-25T13:50:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa5157fa640 1 -- 172.21.15.27:0/1964894593 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa51c003990 con 0x5639bc237550 2024-04-25T13:50:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 10 client.27109 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:44.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 1 -- 172.21.15.27:0/1964894593 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5639bc688b60 con 0x5639bc237550 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa5157fa640 1 -- 172.21.15.27:0/1964894593 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c005070 con 0x5639bc237550 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa5157fa640 1 -- 172.21.15.27:0/1964894593 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa51c02b750 con 0x5639bc237550 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 10 client.27109 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 10 client.27109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 20 client.27109 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5639bc1f4630 0x5639bc68b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 4 client.27109 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.612+0000 7fa53b042740 1 -- 172.21.15.27:0/1964894593 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5639bc56e580 con 0x5639bc1f4630 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.613+0000 7fa531f2b640 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5639bc1f4630 0x5639bc68b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.613+0000 7fa5157fa640 10 client.27109 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:44.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.613+0000 7fa531f2b640 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5639bc1f4630 0x5639bc68b4d0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50f967c188315c17 server_cookie=71202714274d091 in_seq=0 out_seq=0 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.614+0000 7fa5157fa640 1 -- 172.21.15.27:0/1964894593 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5639bc56e580 con 0x5639bc1f4630 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.614+0000 7fa5157fa640 10 client.27109 handle_command_reply: tid=0 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.614+0000 7fa53b042740 1 client.27109 shutdown 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.614+0000 7fa53b042740 1 -- 172.21.15.27:0/1964894593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 msgr2=0x5639bc633080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.614+0000 7fa53b042740 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc633080 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa51c014d40 tx=0x7fa51c000bc0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 1 -- 172.21.15.27:0/1964894593 shutdown_connections 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5639bc1f4630 0x5639bc68b4d0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5639bc260410 0x5639bc6301a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5639bc23f730 0x5639bc62fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa5157fa640 0 client.27109 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 1 --2- 172.21.15.27:0/1964894593 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5639bc237550 0x5639bc633080 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 1 -- 172.21.15.27:0/1964894593 >> 172.21.15.27:0/1964894593 conn(0x5639bc254560 msgr2=0x5639bc605790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 1 -- 172.21.15.27:0/1964894593 shutdown_connections 2024-04-25T13:50:44.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 1 -- 172.21.15.27:0/1964894593 wait complete. 2024-04-25T13:50:44.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.615+0000 7fa53b042740 20 client.27109 trim_cache size 0 max 16384 2024-04-25T13:50:44.619 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:50:44.619 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:50:44.619 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "a42f0648-09e1-48d3-a2c8-b5db773dfc3e", 2024-04-25T13:50:44.619 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:50:44.619 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:50:44.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 -- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5180afaf0 msgr2=0x7fa5180b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5180afaf0 0x7fa5180b2000 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa5240a75e0 tx=0x7fa5240c3040 comp rx=0 tx=0).stop 2024-04-25T13:50:44.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 -- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c108930 msgr2=0x7fa52c1a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c108930 0x7fa52c1a1710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa52400eb10 tx=0x7fa52400c2d0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 -- 172.21.15.27:0/3899750540 shutdown_connections 2024-04-25T13:50:44.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5180afaf0 0x7fa5180b2000 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa52c108930 0x7fa52c1a1710 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa52c103c30 0x7fa52c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 --2- 172.21.15.27:0/3899750540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa52c1029f0 0x7fa52c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.616+0000 7fa53418f640 1 -- 172.21.15.27:0/3899750540 >> 172.21.15.27:0/3899750540 conn(0x7fa52c0fdc00 msgr2=0x7fa52c109ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.617+0000 7fa53418f640 1 -- 172.21.15.27:0/3899750540 shutdown_connections 2024-04-25T13:50:44.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.617+0000 7fa53418f640 1 -- 172.21.15.27:0/3899750540 wait complete. 2024-04-25T13:50:44.631 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:44 smithi027 ceph-mon[29140]: pgmap v644: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 756 op/s 2024-04-25T13:50:44.633 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:50:44.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:50:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 -- 172.21.15.27:0/3326914992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 msgr2=0x7ff67c10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 --2- 172.21.15.27:0/3326914992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 0x7ff67c10ad40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff674009ca0 tx=0x7ff67402a8b0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 -- 172.21.15.27:0/3326914992 shutdown_connections 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 --2- 172.21.15.27:0/3326914992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 0x7ff67c10ad40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 --2- 172.21.15.27:0/3326914992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c103c20 0x7ff67c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 --2- 172.21.15.27:0/3326914992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff67c1029e0 0x7ff67c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 -- 172.21.15.27:0/3326914992 >> 172.21.15.27:0/3326914992 conn(0x7ff67c0fdbd0 msgr2=0x7ff67c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 -- 172.21.15.27:0/3326914992 shutdown_connections 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.805+0000 7ff68240c640 1 -- 172.21.15.27:0/3326914992 wait complete. 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.806+0000 7ff68240c640 1 Processor -- start 2024-04-25T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.806+0000 7ff68240c640 1 -- start start 2024-04-25T13:50:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.806+0000 7ff68240c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff67c1029e0 0x7ff67c10d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff68240c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c103c20 0x7ff67c10b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff68240c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 0x7ff67c10be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff68240c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10f670 con 0x7ff67c103c20 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff68240c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10f4f0 con 0x7ff67c108920 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff68240c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10f7f0 con 0x7ff67c1029e0 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff67c1029e0 0x7ff67c10d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff67c1029e0 0x7ff67c10d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46186/0 (socket says 172.21.15.27:46186) 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67bfff640 1 -- 172.21.15.27:0/335886072 learned_addr learned my addr 172.21.15.27:0/335886072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff6809a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 0x7ff67c10be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c103c20 0x7ff67c10b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67bfff640 1 -- 172.21.15.27:0/335886072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 msgr2=0x7ff67c10be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67bfff640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 0x7ff67c10be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67bfff640 1 -- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c103c20 msgr2=0x7ff67c10b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67bfff640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c103c20 0x7ff67c10b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff67bfff640 1 -- 172.21.15.27:0/335886072 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c10c5c0 con 0x7ff67c1029e0 2024-04-25T13:50:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.807+0000 7ff6809a9640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 0x7ff67c10be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.808+0000 7ff67bfff640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff67c1029e0 0x7ff67c10d2b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00e980 tx=0x7ff66c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.808+0000 7ff6797fa640 1 -- 172.21.15.27:0/335886072 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c00cd80 con 0x7ff67c1029e0 2024-04-25T13:50:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.808+0000 7ff6797fa640 1 -- 172.21.15.27:0/335886072 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff66c0108e0 con 0x7ff67c1029e0 2024-04-25T13:50:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.808+0000 7ff68240c640 1 -- 172.21.15.27:0/335886072 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67c1a4530 con 0x7ff67c1029e0 2024-04-25T13:50:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.808+0000 7ff6797fa640 1 -- 172.21.15.27:0/335886072 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c00cd80 con 0x7ff67c1029e0 2024-04-25T13:50:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.809+0000 7ff68240c640 1 -- 172.21.15.27:0/335886072 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff67c1a49a0 con 0x7ff67c1029e0 2024-04-25T13:50:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.811+0000 7ff6797fa640 1 -- 172.21.15.27:0/335886072 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff66c01f020 con 0x7ff67c1029e0 2024-04-25T13:50:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.812+0000 7ff68939a740 1 Processor -- start 2024-04-25T13:50:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.812+0000 7ff68939a740 1 -- start start 2024-04-25T13:50:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.812+0000 7ff6797fa640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6640afcf0 0x7ff6640b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff68939a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b9d390 0x5585d2f3ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff68939a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b744d0 0x5585d2f3efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff6809a9640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6640afcf0 0x7ff6640b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff6797fa640 1 -- 172.21.15.27:0/335886072 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff66c0de840 con 0x7ff67c1029e0 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff68939a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b7c6b0 0x5585d2f3fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff68939a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5585d2ea3d80 con 0x5585d2b7c6b0 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff68939a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5585d2e5d8b0 con 0x5585d2b744d0 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff68939a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5585d2e9fd40 con 0x5585d2b9d390 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b9d390 0x5585d2f3ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b9d390 0x5585d2f3ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46192/0 (socket says 172.21.15.27:46192) 2024-04-25T13:50:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67bfff640 1 -- 172.21.15.27:0/1437924033 learned_addr learned my addr 172.21.15.27:0/1437924033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b744d0 0x5585d2f3efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67bfff640 1 -- 172.21.15.27:0/1437924033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b744d0 msgr2=0x5585d2f3efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67bfff640 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b744d0 0x5585d2f3efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff6809a9640 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b7c6b0 0x5585d2f3fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67bfff640 1 -- 172.21.15.27:0/1437924033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b7c6b0 msgr2=0x5585d2f3fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67bfff640 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b7c6b0 0x5585d2f3fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff67bfff640 1 -- 172.21.15.27:0/1437924033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585d2e2ea40 con 0x5585d2b9d390 2024-04-25T13:50:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.813+0000 7ff6809a9640 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b7c6b0 0x5585d2f3fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:50:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.814+0000 7ff67bfff640 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b9d390 0x5585d2f3ea60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0a6ce0 tx=0x7ff66c0a67d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf657d824292e0bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.814+0000 7ff6809a9640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6640afcf0 0x7ff6640b2200 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff674006010 tx=0x7ff674030cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.814+0000 7ff6627fc640 1 -- 172.21.15.27:0/1437924033 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c0a7450 con 0x5585d2b9d390 2024-04-25T13:50:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.814+0000 7ff6627fc640 1 -- 172.21.15.27:0/1437924033 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff66c0a6d50 con 0x5585d2b9d390 2024-04-25T13:50:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.814+0000 7ff6627fc640 1 -- 172.21.15.27:0/1437924033 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c0a2fe0 con 0x5585d2b9d390 2024-04-25T13:50:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.815+0000 7ff68939a740 1 -- 172.21.15.27:0/1437924033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b9d390 msgr2=0x5585d2f3ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.815+0000 7ff68939a740 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b9d390 0x5585d2f3ea60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0a6ce0 tx=0x7ff66c0a67d0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.816+0000 7ff68939a740 1 -- 172.21.15.27:0/1437924033 shutdown_connections 2024-04-25T13:50:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.816+0000 7ff68939a740 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b7c6b0 0x5585d2f3fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.816+0000 7ff68939a740 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b744d0 0x5585d2f3efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.816+0000 7ff68939a740 1 --2- 172.21.15.27:0/1437924033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b9d390 0x5585d2f3ea60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.816+0000 7ff68939a740 1 -- 172.21.15.27:0/1437924033 >> 172.21.15.27:0/1437924033 conn(0x5585d2b914e0 msgr2=0x5585d2f3a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.816+0000 7ff68939a740 1 -- 172.21.15.27:0/1437924033 shutdown_connections 2024-04-25T13:50:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.817+0000 7ff68939a740 1 -- 172.21.15.27:0/1437924033 wait complete. 2024-04-25T13:50:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.817+0000 7ff68939a740 1 -- start start 2024-04-25T13:50:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff68939a740 1 Processor -- start 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff68939a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b744d0 0x5585d2f511d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff68939a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f51710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff68939a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b9d390 0x5585d2f58600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff68939a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5585d2f40780 con 0x5585d2b744d0 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff68939a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5585d2e2ad10 con 0x5585d2b9d390 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff68939a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5585d2f40900 con 0x5585d2b7c6b0 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff67b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f51710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff68939a740 10 client.0 _ll_register_callbacks cb 0x5585d2e57210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff67b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f51710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46206/0 (socket says 172.21.15.27:46206) 2024-04-25T13:50:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff67b7fe640 1 -- 172.21.15.27:0/1531831576 learned_addr learned my addr 172.21.15.27:0/1531831576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.818+0000 7ff67bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b744d0 0x5585d2f511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff6809a9640 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b9d390 0x5585d2f58600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff67b7fe640 1 -- 172.21.15.27:0/1531831576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b9d390 msgr2=0x5585d2f58600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b9d390 0x5585d2f58600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff67b7fe640 1 -- 172.21.15.27:0/1531831576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b744d0 msgr2=0x5585d2f511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b744d0 0x5585d2f511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff67b7fe640 1 -- 172.21.15.27:0/1531831576 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585d2c7f7d0 con 0x5585d2b7c6b0 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f51710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff6700126e0 tx=0x7ff670012ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d18563c5d2b4914 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff65b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff65b7fe640 1 -- 172.21.15.27:0/1531831576 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670012190 con 0x5585d2b7c6b0 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff65b7fe640 1 -- 172.21.15.27:0/1531831576 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff67000f920 con 0x5585d2b7c6b0 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.819+0000 7ff68939a740 1 -- 172.21.15.27:0/1531831576 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff66c0a6d50 con 0x5585d2b7c6b0 2024-04-25T13:50:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff65b7fe640 1 -- 172.21.15.27:0/1531831576 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff670023070 con 0x5585d2b7c6b0 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff68939a740 10 client.27121 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff68939a740 1 -- 172.21.15.27:0/1531831576 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5585d2bb53f0 con 0x5585d2b7c6b0 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff65b7fe640 1 -- 172.21.15.27:0/1531831576 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff66c0a6d50 con 0x5585d2b7c6b0 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff65b7fe640 1 -- 172.21.15.27:0/1531831576 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff67001cc10 con 0x5585d2b7c6b0 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff68939a740 10 client.27121 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff68939a740 10 client.27121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff68939a740 20 client.27121 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff68939a740 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5585d2b5bf60 0x5585d2f5eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff68939a740 4 client.27121 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff68939a740 1 -- 172.21.15.27:0/1531831576 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585d2eb1990 con 0x5585d2b5bf60 2024-04-25T13:50:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.820+0000 7ff67bfff640 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5585d2b5bf60 0x5585d2f5eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.821+0000 7ff65b7fe640 10 client.27121 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.821+0000 7ff67bfff640 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5585d2b5bf60 0x5585d2f5eb10 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=853e0d6b44cff625 server_cookie=ac627b513994d66d in_seq=0 out_seq=0 2024-04-25T13:50:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.823+0000 7ff65b7fe640 1 -- 172.21.15.27:0/1531831576 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5585d2eb1990 con 0x5585d2b5bf60 2024-04-25T13:50:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.823+0000 7ff65b7fe640 10 client.27121 handle_command_reply: tid=0 2024-04-25T13:50:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.823+0000 7ff68939a740 1 client.27121 shutdown 2024-04-25T13:50:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.823+0000 7ff68939a740 1 -- 172.21.15.27:0/1531831576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 msgr2=0x5585d2f51710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.824+0000 7ff68939a740 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f51710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff6700126e0 tx=0x7ff670012ba0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.824+0000 7ff68939a740 1 -- 172.21.15.27:0/1531831576 shutdown_connections 2024-04-25T13:50:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.824+0000 7ff68939a740 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5585d2b5bf60 0x5585d2f5eb10 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.824+0000 7ff68939a740 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b9d390 0x5585d2f58600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.824+0000 7ff65b7fe640 0 client.27121 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.824+0000 7ff68939a740 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f51710 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.824+0000 7ff68939a740 1 --2- 172.21.15.27:0/1531831576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b744d0 0x5585d2f511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.824+0000 7ff68939a740 1 -- 172.21.15.27:0/1531831576 >> 172.21.15.27:0/1531831576 conn(0x5585d2b914e0 msgr2=0x5585d2f3d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.826+0000 7ff68939a740 1 -- 172.21.15.27:0/1531831576 shutdown_connections 2024-04-25T13:50:44.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.826+0000 7ff68939a740 1 -- 172.21.15.27:0/1531831576 wait complete. 2024-04-25T13:50:44.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.827+0000 7ff68939a740 20 client.27121 trim_cache size 0 max 16384 2024-04-25T13:50:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.856+0000 7ff68939a740 1 Processor -- start 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.856+0000 7ff68939a740 1 -- start start 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff68939a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b5bf60 0x5585d2f50500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff68939a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b744d0 0x5585d2f58520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff68939a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b7c6b0 0x5585d2f58a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff68939a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c1a6fe0 con 0x5585d2b5bf60 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff68939a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c1a9860 con 0x5585d2b7c6b0 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff68939a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c1a7650 con 0x5585d2b744d0 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff6809a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b7c6b0 0x5585d2f58a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b5bf60 0x5585d2f50500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b744d0 0x5585d2f58520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b744d0 0x5585d2f58520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46222/0 (socket says 172.21.15.27:46222) 2024-04-25T13:50:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67b7fe640 1 -- 172.21.15.27:0/2047692269 learned_addr learned my addr 172.21.15.27:0/2047692269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff6809a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b7c6b0 0x5585d2f58a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52190/0 (socket says 172.21.15.27:52190) 2024-04-25T13:50:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67b7fe640 1 -- 172.21.15.27:0/2047692269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b7c6b0 msgr2=0x5585d2f58a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/2047692269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b7c6b0 0x5585d2f58a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67b7fe640 1 -- 172.21.15.27:0/2047692269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b5bf60 msgr2=0x5585d2f50500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/2047692269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b5bf60 0x5585d2f50500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff67b7fe640 1 -- 172.21.15.27:0/2047692269 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585d2f43e40 con 0x5585d2b744d0 2024-04-25T13:50:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.857+0000 7ff6809a9640 1 --2- 172.21.15.27:0/2047692269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b7c6b0 0x5585d2f58a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.858+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/2047692269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b744d0 0x5585d2f58520 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff670010620 tx=0x7ff6700123a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c638d5446192cd78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.858+0000 7ff661ffb640 1 -- 172.21.15.27:0/2047692269 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67002d070 con 0x5585d2b744d0 2024-04-25T13:50:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.858+0000 7ff661ffb640 1 -- 172.21.15.27:0/2047692269 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff67000f920 con 0x5585d2b744d0 2024-04-25T13:50:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.858+0000 7ff661ffb640 1 -- 172.21.15.27:0/2047692269 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670005500 con 0x5585d2b744d0 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.859+0000 7ff68939a740 1 -- 172.21.15.27:0/2047692269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b744d0 msgr2=0x5585d2f58520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.859+0000 7ff68939a740 1 --2- 172.21.15.27:0/2047692269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b744d0 0x5585d2f58520 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff670010620 tx=0x7ff6700123a0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.860+0000 7ff68939a740 1 -- 172.21.15.27:0/2047692269 shutdown_connections 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.860+0000 7ff68939a740 1 --2- 172.21.15.27:0/2047692269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b7c6b0 0x5585d2f58a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.860+0000 7ff68939a740 1 --2- 172.21.15.27:0/2047692269 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b744d0 0x5585d2f58520 secure :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff670010620 tx=0x7ff6700123a0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.860+0000 7ff68939a740 1 --2- 172.21.15.27:0/2047692269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b5bf60 0x5585d2f50500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.860+0000 7ff68939a740 1 -- 172.21.15.27:0/2047692269 >> 172.21.15.27:0/2047692269 conn(0x5585d2b914e0 msgr2=0x5585d2f3ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.860+0000 7ff68939a740 1 -- 172.21.15.27:0/2047692269 shutdown_connections 2024-04-25T13:50:44.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.860+0000 7ff68939a740 1 -- 172.21.15.27:0/2047692269 wait complete. 2024-04-25T13:50:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.861+0000 7ff68939a740 1 -- start start 2024-04-25T13:50:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.861+0000 7ff68939a740 1 Processor -- start 2024-04-25T13:50:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff68939a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b5bf60 0x5585d2f55f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff68939a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f564a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff67bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b5bf60 0x5585d2f55f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff67bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b5bf60 0x5585d2f55f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52204/0 (socket says 172.21.15.27:52204) 2024-04-25T13:50:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff67bfff640 1 -- 172.21.15.27:0/3021955493 learned_addr learned my addr 172.21.15.27:0/3021955493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:44.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff68939a740 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b9d390 0x5585d2f573d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c1a7d20 con 0x5585d2b9d390 2024-04-25T13:50:44.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c1a9190 con 0x5585d2b5bf60 2024-04-25T13:50:44.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c1a83f0 con 0x5585d2b7c6b0 2024-04-25T13:50:44.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.862+0000 7ff6809a9640 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b9d390 0x5585d2f573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff67bfff640 1 -- 172.21.15.27:0/3021955493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 msgr2=0x5585d2f564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff67bfff640 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff67bfff640 1 -- 172.21.15.27:0/3021955493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b9d390 msgr2=0x5585d2f573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff67bfff640 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b9d390 0x5585d2f573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff67bfff640 1 -- 172.21.15.27:0/3021955493 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585d2b750f0 con 0x5585d2b5bf60 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff67bfff640 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b5bf60 0x5585d2f55f60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0a2dd0 tx=0x7ff66c00efc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79dcf90dcd53ed4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff65b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff68939a740 10 client.0 _ll_register_callbacks cb 0x7ff670009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3021955493 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c010d00 con 0x5585d2b5bf60 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.863+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3021955493 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff66c0a2fe0 con 0x5585d2b5bf60 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff67000f920 con 0x5585d2b5bf60 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3021955493 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff66c0c1340 con 0x5585d2b5bf60 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 10 client.27086 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=19}) -- 0x5585d2e644b0 con 0x5585d2b5bf60 2024-04-25T13:50:44.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3021955493 <== mon.2 v2:172.21.15.45:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff66c0af720 con 0x5585d2b5bf60 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 10 client.27086 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 10 client.27086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 20 client.27086 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5585d2b313a0 0x5585d2f58bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 4 client.27086 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.864+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7ff67000f920 con 0x5585d2b313a0 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.865+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5585d2b313a0 0x5585d2f58bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.865+0000 7ff65b7fe640 10 client.27086 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:44.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.865+0000 7ff67b7fe640 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5585d2b313a0 0x5585d2f58bd0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e923d721bb761576 server_cookie=565be11ffac3d26e in_seq=0 out_seq=0 2024-04-25T13:50:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.867+0000 7ff65b7fe640 1 -- 172.21.15.27:0/3021955493 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x7ff67000f920 con 0x5585d2b313a0 2024-04-25T13:50:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.867+0000 7ff65b7fe640 10 client.27086 handle_command_reply: tid=0 2024-04-25T13:50:44.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.867+0000 7ff68939a740 1 client.27086 shutdown 2024-04-25T13:50:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.869+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b5bf60 msgr2=0x5585d2f55f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.869+0000 7ff68939a740 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b5bf60 0x5585d2f55f60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0a2dd0 tx=0x7ff66c00efc0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 shutdown_connections 2024-04-25T13:50:44.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5585d2b313a0 0x5585d2f58bd0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5585d2b9d390 0x5585d2f573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5585d2b7c6b0 0x5585d2f564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 1 --2- 172.21.15.27:0/3021955493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5585d2b5bf60 0x5585d2f55f60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff65b7fe640 0 client.27086 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 >> 172.21.15.27:0/3021955493 conn(0x5585d2b914e0 msgr2=0x5585d2f39c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 shutdown_connections 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 1 -- 172.21.15.27:0/3021955493 wait complete. 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68939a740 20 client.27086 trim_cache size 0 max 16384 2024-04-25T13:50:44.871 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (541 inodes in the stack)", 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: "a42f0648-09e1-48d3-a2c8-b5db773dfc3e": { 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a42f0648-09e1-48d3-a2c8-b5db773dfc3e", 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:50:44.872 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68240c640 1 -- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6640afcf0 msgr2=0x7ff6640b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68240c640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6640afcf0 0x7ff6640b2200 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff674006010 tx=0x7ff674030cb0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68240c640 1 -- 172.21.15.27:0/335886072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff67c1029e0 msgr2=0x7ff67c10d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.870+0000 7ff68240c640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff67c1029e0 0x7ff67c10d2b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00e980 tx=0x7ff66c00ee40 comp rx=0 tx=0).stop 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.871+0000 7ff68240c640 1 -- 172.21.15.27:0/335886072 shutdown_connections 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.871+0000 7ff68240c640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff6640afcf0 0x7ff6640b2200 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.871+0000 7ff68240c640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff67c108920 0x7ff67c10be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.871+0000 7ff68240c640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c103c20 0x7ff67c10b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.871+0000 7ff68240c640 1 --2- 172.21.15.27:0/335886072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff67c1029e0 0x7ff67c10d2b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.871+0000 7ff68240c640 1 -- 172.21.15.27:0/335886072 >> 172.21.15.27:0/335886072 conn(0x7ff67c0fdbd0 msgr2=0x7ff67c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.871+0000 7ff68240c640 1 -- 172.21.15.27:0/335886072 shutdown_connections 2024-04-25T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:50:44.871+0000 7ff68240c640 1 -- 172.21.15.27:0/335886072 wait complete. 2024-04-25T13:50:44.888 INFO:tasks.cephfs.filesystem:scrub status for tag:a42f0648-09e1-48d3-a2c8-b5db773dfc3e - {'path': '/', 'tag': 'a42f0648-09e1-48d3-a2c8-b5db773dfc3e', 'options': 'recursive,force'} 2024-04-25T13:50:45.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:45 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:50:45.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:45 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:45.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:45 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:50:45.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:45 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:50:45.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:45 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:45.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:45 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:50:45.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:45 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:50:45.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:45 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:45.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:45 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:50:46.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:46 smithi122 ceph-mon[32843]: pgmap v645: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 MiB/s wr, 798 op/s 2024-04-25T13:50:46.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:46 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:46.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:46 smithi045 ceph-mon[32731]: pgmap v645: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 MiB/s wr, 798 op/s 2024-04-25T13:50:46.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:46 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:46.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:46 smithi027 ceph-mon[29140]: pgmap v645: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 MiB/s wr, 798 op/s 2024-04-25T13:50:46.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:46 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:50:47.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:47 smithi045 ceph-mon[32731]: pgmap v646: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 35 MiB/s wr, 1.65k op/s 2024-04-25T13:50:47.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:47 smithi027 ceph-mon[29140]: pgmap v646: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 35 MiB/s wr, 1.65k op/s 2024-04-25T13:50:47.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:47 smithi122 ceph-mon[32843]: pgmap v646: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 35 MiB/s wr, 1.65k op/s 2024-04-25T13:50:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 580.0 s, 1149.2 tps, lat 87.392 ms stddev 157.610 2024-04-25T13:50:50.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:49 smithi122 ceph-mon[32843]: pgmap v647: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 36 MiB/s wr, 1.66k op/s 2024-04-25T13:50:50.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:49 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:50:50.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:49 smithi045 ceph-mon[32731]: pgmap v647: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 36 MiB/s wr, 1.66k op/s 2024-04-25T13:50:50.078 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:49 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:50:50.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:49 smithi027 ceph-mon[29140]: pgmap v647: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 36 MiB/s wr, 1.66k op/s 2024-04-25T13:50:50.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:49 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:50:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 -- 172.21.15.27:0/2157709137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6610102a50 msgr2=0x7f6610103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 --2- 172.21.15.27:0/2157709137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6610102a50 0x7f6610103750 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6600009900 tx=0x7f660002a410 comp rx=0 tx=0).stop 2024-04-25T13:50:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 -- 172.21.15.27:0/2157709137 shutdown_connections 2024-04-25T13:50:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 --2- 172.21.15.27:0/2157709137 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6610108990 0x7f661010adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 --2- 172.21.15.27:0/2157709137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6610103c90 0x7f66101040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 --2- 172.21.15.27:0/2157709137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6610102a50 0x7f6610103750 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.811 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 -- 172.21.15.27:0/2157709137 >> 172.21.15.27:0/2157709137 conn(0x7f66100fdc60 msgr2=0x7f66101000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:50.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 -- 172.21.15.27:0/2157709137 shutdown_connections 2024-04-25T13:50:50.812 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.811+0000 7f6614849640 1 -- 172.21.15.27:0/2157709137 wait complete. 2024-04-25T13:50:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f6614849640 1 Processor -- start 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f6614849640 1 -- start start 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f6614849640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6610102a50 0x7f6610198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f6614849640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6610103c90 0x7f66101985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f6614849640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6610108990 0x7f661019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f6614849640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f661010d4b0 con 0x7f6610102a50 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f6614849640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f661010d330 con 0x7f6610108990 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f6614849640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f661010d630 con 0x7f6610103c90 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f660dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6610103c90 0x7f66101985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f660e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6610102a50 0x7f6610198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f660dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6610103c90 0x7f66101985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46250/0 (socket says 172.21.15.27:46250) 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f660dd9b640 1 -- 172.21.15.27:0/761054351 learned_addr learned my addr 172.21.15.27:0/761054351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.812+0000 7f660ed9d640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6610108990 0x7f661019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f660dd9b640 1 -- 172.21.15.27:0/761054351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6610108990 msgr2=0x7f661019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f660dd9b640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6610108990 0x7f661019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f660dd9b640 1 -- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6610102a50 msgr2=0x7f6610198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f660dd9b640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6610102a50 0x7f6610198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f660dd9b640 1 -- 172.21.15.27:0/761054351 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661019fba0 con 0x7f6610103c90 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f660dd9b640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6610103c90 0x7f66101985a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f660400e9f0 tx=0x7f660400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f65ff7fe640 1 -- 172.21.15.27:0/761054351 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660400cd80 con 0x7f6610103c90 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f65ff7fe640 1 -- 172.21.15.27:0/761054351 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66040108e0 con 0x7f6610103c90 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f6614849640 1 -- 172.21.15.27:0/761054351 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661019fe90 con 0x7f6610103c90 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f65ff7fe640 1 -- 172.21.15.27:0/761054351 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604018980 con 0x7f6610103c90 2024-04-25T13:50:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.813+0000 7f6614849640 1 -- 172.21.15.27:0/761054351 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66101a02f0 con 0x7f6610103c90 2024-04-25T13:50:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.817+0000 7f65ff7fe640 1 -- 172.21.15.27:0/761054351 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6604010430 con 0x7f6610103c90 2024-04-25T13:50:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.818+0000 7f65ff7fe640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65f80afb30 0x7f65f80b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.818+0000 7f65ff7fe640 1 -- 172.21.15.27:0/761054351 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f66040def00 con 0x7f6610103c90 2024-04-25T13:50:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.818+0000 7f660e59c640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65f80afb30 0x7f65f80b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f661b7d3740 1 Processor -- start 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f660e59c640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65f80afb30 0x7f65f80b2040 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f660002aa00 tx=0x7f6600000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f661b7d3740 1 -- start start 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d987a161b0 0x55d987db6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d9879ed2f0 0x55d987db6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dbddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f661b7d3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d987d1c460 con 0x55d9879ed2f0 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f661b7d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d987cd6590 con 0x55d987a161b0 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.819+0000 7f661b7d3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d987d18420 con 0x55d9879f54d0 2024-04-25T13:50:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d9879ed2f0 0x55d987db6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d9879ed2f0 0x55d987db6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38914/0 (socket says 172.21.15.27:38914) 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660dd9b640 1 -- 172.21.15.27:0/2477575250 learned_addr learned my addr 172.21.15.27:0/2477575250 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660ed9d640 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dbddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660e59c640 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d987a161b0 0x55d987db6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660ed9d640 1 -- 172.21.15.27:0/2477575250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d987a161b0 msgr2=0x55d987db6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660ed9d640 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d987a161b0 0x55d987db6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660ed9d640 1 -- 172.21.15.27:0/2477575250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d9879ed2f0 msgr2=0x55d987db6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660ed9d640 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d9879ed2f0 0x55d987db6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660ed9d640 1 -- 172.21.15.27:0/2477575250 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d987ce30d0 con 0x55d9879f54d0 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f660ed9d640 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dbddf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6608009f00 tx=0x7f6608009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b826c5d31288ffe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f65fcff9640 1 -- 172.21.15.27:0/2477575250 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6608021070 con 0x55d9879f54d0 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f65fcff9640 1 -- 172.21.15.27:0/2477575250 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6608003c30 con 0x55d9879f54d0 2024-04-25T13:50:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.820+0000 7f65fcff9640 1 -- 172.21.15.27:0/2477575250 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660801c900 con 0x55d9879f54d0 2024-04-25T13:50:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.822+0000 7f661b7d3740 1 -- 172.21.15.27:0/2477575250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 msgr2=0x55d987dbddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.822+0000 7f661b7d3740 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dbddf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6608009f00 tx=0x7f6608009500 comp rx=0 tx=0).stop 2024-04-25T13:50:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.822+0000 7f661b7d3740 1 -- 172.21.15.27:0/2477575250 shutdown_connections 2024-04-25T13:50:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.822+0000 7f661b7d3740 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dbddf0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.822+0000 7f661b7d3740 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d9879ed2f0 0x55d987db6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.822+0000 7f661b7d3740 1 --2- 172.21.15.27:0/2477575250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d987a161b0 0x55d987db6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.822+0000 7f661b7d3740 1 -- 172.21.15.27:0/2477575250 >> 172.21.15.27:0/2477575250 conn(0x55d987a0a300 msgr2=0x55d987db2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.823+0000 7f661b7d3740 1 -- 172.21.15.27:0/2477575250 shutdown_connections 2024-04-25T13:50:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.823+0000 7f661b7d3740 1 -- 172.21.15.27:0/2477575250 wait complete. 2024-04-25T13:50:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.823+0000 7f661b7d3740 1 -- start start 2024-04-25T13:50:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.823+0000 7f661b7d3740 1 Processor -- start 2024-04-25T13:50:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f661b7d3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55d987dbe840 con 0x55d987a161b0 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f660e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f660dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f660e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52232/0 (socket says 172.21.15.27:52232) 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f660dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46262/0 (socket says 172.21.15.27:46262) 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f660e59c640 1 -- 172.21.15.27:0/1657586864 learned_addr learned my addr 172.21.15.27:0/1657586864 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55d987ca3b60 con 0x55d9879ed2f0 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f660ed9d640 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55d987dbe9c0 con 0x55d9879f54d0 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.824+0000 7f661b7d3740 10 client.0 _ll_register_callbacks cb 0x55d987cc82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f660dd9b640 1 -- 172.21.15.27:0/1657586864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 msgr2=0x55d987dc06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f660dd9b640 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f660dd9b640 1 -- 172.21.15.27:0/1657586864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 msgr2=0x55d987dc7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f660dd9b640 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f660dd9b640 1 -- 172.21.15.27:0/1657586864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d987a2e210 con 0x55d9879f54d0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6608003c30 con 0x55d9879f54d0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f660dd9b640 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc0bf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f66040a3540 tx=0x7f66040a6cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9403e62747e9214a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f65f5ffb640 10 client.27139 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f65f5ffb640 1 -- 172.21.15.27:0/1657586864 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66040c1060 con 0x55d9879f54d0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f65f5ffb640 1 -- 172.21.15.27:0/1657586864 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f66040a6720 con 0x55d9879f54d0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f65f5ffb640 1 -- 172.21.15.27:0/1657586864 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f66040a6940 con 0x55d9879f54d0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f661b7d3740 10 client.27139 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.825+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x55d987afa620 con 0x55d9879f54d0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f65f5ffb640 1 -- 172.21.15.27:0/1657586864 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6604018b20 con 0x55d9879f54d0 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f661b7d3740 10 client.27139 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f661b7d3740 10 client.27139 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f661b7d3740 20 client.27139 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f661b7d3740 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d9879d4d80 0x55d987dce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f661b7d3740 4 client.27139 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:50:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d987d2a070 con 0x55d9879d4d80 2024-04-25T13:50:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f660e59c640 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d9879d4d80 0x55d987dce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f65f5ffb640 10 client.27139 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.826+0000 7f660e59c640 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d9879d4d80 0x55d987dce1e0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bc5ef4490a7f7e9 server_cookie=161bfbd8b5f7300f in_seq=0 out_seq=0 2024-04-25T13:50:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.827+0000 7f65f5ffb640 1 -- 172.21.15.27:0/1657586864 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d987d2a070 con 0x55d9879d4d80 2024-04-25T13:50:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.827+0000 7f65f5ffb640 10 client.27139 handle_command_reply: tid=0 2024-04-25T13:50:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.827+0000 7f661b7d3740 1 client.27139 shutdown 2024-04-25T13:50:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 msgr2=0x55d987dc0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc0bf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f66040a3540 tx=0x7f66040a6cf0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 shutdown_connections 2024-04-25T13:50:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d9879d4d80 0x55d987dce1e0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc0bf0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 --2- 172.21.15.27:0/1657586864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 >> 172.21.15.27:0/1657586864 conn(0x55d987a0a300 msgr2=0x55d987db1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f65f5ffb640 0 client.27139 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 shutdown_connections 2024-04-25T13:50:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 1 -- 172.21.15.27:0/1657586864 wait complete. 2024-04-25T13:50:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.829+0000 7f661b7d3740 20 client.27139 trim_cache size 0 max 16384 2024-04-25T13:50:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.850+0000 7f661b7d3740 1 Processor -- start 2024-04-25T13:50:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.850+0000 7f661b7d3740 1 -- start start 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dbab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dbb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f661b7d3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f66101a7050 con 0x55d987a161b0 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f661b7d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f66101a98d0 con 0x55d9879ed2f0 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f661b7d3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f66101a76c0 con 0x55d9879f54d0 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dbb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660dd9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dbb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38926/0 (socket says 172.21.15.27:38926) 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660ed9d640 1 -- 172.21.15.27:0/2469042413 learned_addr learned my addr 172.21.15.27:0/2469042413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660e59c640 1 --2- 172.21.15.27:0/2469042413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dbab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660dd9b640 1 -- 172.21.15.27:0/2469042413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 msgr2=0x55d987dbab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660dd9b640 1 --2- 172.21.15.27:0/2469042413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dbab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660dd9b640 1 -- 172.21.15.27:0/2469042413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 msgr2=0x55d987dbb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660dd9b640 1 --2- 172.21.15.27:0/2469042413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dbb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660dd9b640 1 -- 172.21.15.27:0/2469042413 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d987db7580 con 0x55d9879f54d0 2024-04-25T13:50:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.851+0000 7f660dd9b640 1 --2- 172.21.15.27:0/2469042413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc7a00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f66040dca80 tx=0x7f6604022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e85ad3a3a68293 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.852+0000 7f65f7fff640 1 -- 172.21.15.27:0/2469042413 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66040a7290 con 0x55d9879f54d0 2024-04-25T13:50:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.852+0000 7f65f7fff640 1 -- 172.21.15.27:0/2469042413 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6608003c30 con 0x55d9879f54d0 2024-04-25T13:50:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.852+0000 7f65f7fff640 1 -- 172.21.15.27:0/2469042413 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604018b20 con 0x55d9879f54d0 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.852+0000 7f661b7d3740 1 -- 172.21.15.27:0/2469042413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 msgr2=0x55d987dc7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.852+0000 7f661b7d3740 1 --2- 172.21.15.27:0/2469042413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc7a00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f66040dca80 tx=0x7f6604022000 comp rx=0 tx=0).stop 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.853+0000 7f661b7d3740 1 -- 172.21.15.27:0/2469042413 shutdown_connections 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.853+0000 7f661b7d3740 1 --2- 172.21.15.27:0/2469042413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dbb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.853+0000 7f661b7d3740 1 --2- 172.21.15.27:0/2469042413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc7a00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.853+0000 7f661b7d3740 1 --2- 172.21.15.27:0/2469042413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dbab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.853+0000 7f661b7d3740 1 -- 172.21.15.27:0/2469042413 >> 172.21.15.27:0/2469042413 conn(0x55d987a0a300 msgr2=0x55d987db68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.853+0000 7f661b7d3740 1 -- 172.21.15.27:0/2469042413 shutdown_connections 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.853+0000 7f661b7d3740 1 -- 172.21.15.27:0/2469042413 wait complete. 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.854+0000 7f661b7d3740 1 -- start start 2024-04-25T13:50:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.854+0000 7f661b7d3740 1 Processor -- start 2024-04-25T13:50:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f661b7d3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f661b7d3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f66101a7d90 con 0x55d987a161b0 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f661b7d3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f66101a9200 con 0x55d9879ed2f0 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f661b7d3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f66101a8460 con 0x55d9879f54d0 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f661b7d3740 10 client.0 _ll_register_callbacks cb 0x7f66040a6e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f660ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f660ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38940/0 (socket says 172.21.15.27:38940) 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f660ed9d640 1 -- 172.21.15.27:0/3171525515 learned_addr learned my addr 172.21.15.27:0/3171525515 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f660e59c640 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.855+0000 7f660dd9b640 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f660e59c640 1 -- 172.21.15.27:0/3171525515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 msgr2=0x55d987dc5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f660e59c640 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f660e59c640 1 -- 172.21.15.27:0/3171525515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 msgr2=0x55d987dc6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:50:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f660e59c640 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f660e59c640 1 -- 172.21.15.27:0/3171525515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d987dc7320 con 0x55d9879ed2f0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f660ed9d640 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f660e59c640 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc58f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55d987dce900 tx=0x7f660002a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7fca81e7912fe3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f65f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f65f5ffb640 1 -- 172.21.15.27:0/3171525515 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6600035040 con 0x55d9879ed2f0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f661b7d3740 1 -- 172.21.15.27:0/3171525515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6608003c30 con 0x55d9879ed2f0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.856+0000 7f65f5ffb640 1 -- 172.21.15.27:0/3171525515 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6600003af0 con 0x55d9879ed2f0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.857+0000 7f65f5ffb640 1 -- 172.21.15.27:0/3171525515 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6600003cf0 con 0x55d9879ed2f0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.857+0000 7f661b7d3740 10 client.27104 fetch_fsmap learned FSMap version 19 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.857+0000 7f661b7d3740 1 -- 172.21.15.27:0/3171525515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55d987e3f490 con 0x55d9879ed2f0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.857+0000 7f65f5ffb640 1 -- 172.21.15.27:0/3171525515 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6608003c30 con 0x55d9879ed2f0 2024-04-25T13:50:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.857+0000 7f65f5ffb640 1 -- 172.21.15.27:0/3171525515 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f66000319b0 con 0x55d9879ed2f0 2024-04-25T13:50:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f661b7d3740 10 client.27104 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:50:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f661b7d3740 10 client.27104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:50:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f661b7d3740 20 client.27104 populate_metadata read hostname 'smithi027' 2024-04-25T13:50:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f661b7d3740 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d9879aa1c0 0x55d987e41df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:50:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f661b7d3740 4 client.27104 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:50:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f661b7d3740 1 -- 172.21.15.27:0/3171525515 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d987d23ee0 con 0x55d9879aa1c0 2024-04-25T13:50:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f660dd9b640 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d9879aa1c0 0x55d987e41df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:50:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f65f5ffb640 10 client.27104 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.858+0000 7f660dd9b640 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d9879aa1c0 0x55d987e41df0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c99fbb0f050afee server_cookie=341dd41c542a9e7f in_seq=0 out_seq=0 2024-04-25T13:50:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.996+0000 7f65f5ffb640 1 -- 172.21.15.27:0/3171525515 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d987d23ee0 con 0x55d9879aa1c0 2024-04-25T13:50:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.996+0000 7f65f5ffb640 10 client.27104 handle_command_reply: tid=0 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.996+0000 7f661b7d3740 1 client.27104 shutdown 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 -- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 msgr2=0x55d987dc58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc58f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55d987dce900 tx=0x7f660002a410 comp rx=0 tx=0).stop 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 -- 172.21.15.27:0/3171525515 shutdown_connections 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55d9879aa1c0 0x55d987e41df0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d987a161b0 0x55d987dc6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55d9879f54d0 0x55d987dc5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f65f5ffb640 0 client.27104 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 --2- 172.21.15.27:0/3171525515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d9879ed2f0 0x55d987dc58f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 -- 172.21.15.27:0/3171525515 >> 172.21.15.27:0/3171525515 conn(0x55d987a0a300 msgr2=0x55d987dbed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 -- 172.21.15.27:0/3171525515 shutdown_connections 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 1 -- 172.21.15.27:0/3171525515 wait complete. 2024-04-25T13:50:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.997+0000 7f661b7d3740 20 client.27104 trim_cache size 0 max 16384 2024-04-25T13:50:50.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:50:50.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:50:50.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:50:50.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 -- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65f80afb30 msgr2=0x7f65f80b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65f80afb30 0x7f65f80b2040 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f660002aa00 tx=0x7f6600000bc0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 -- 172.21.15.27:0/761054351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6610103c90 msgr2=0x7f66101985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6610103c90 0x7f66101985a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f660400e9f0 tx=0x7f660400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 -- 172.21.15.27:0/761054351 shutdown_connections 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65f80afb30 0x7f65f80b2040 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6610108990 0x7f661019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6610103c90 0x7f66101985a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 --2- 172.21.15.27:0/761054351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6610102a50 0x7f6610198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 -- 172.21.15.27:0/761054351 >> 172.21.15.27:0/761054351 conn(0x7f66100fdc60 msgr2=0x7f6610104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:50:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 -- 172.21.15.27:0/761054351 shutdown_connections 2024-04-25T13:50:51.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:50:50.998+0000 7f6614849640 1 -- 172.21.15.27:0/761054351 wait complete. 2024-04-25T13:50:52.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:51 smithi122 ceph-mon[32843]: pgmap v648: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 37 MiB/s wr, 1.69k op/s 2024-04-25T13:50:52.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:51 smithi045 ceph-mon[32731]: pgmap v648: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 37 MiB/s wr, 1.69k op/s 2024-04-25T13:50:52.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:51 smithi027 ceph-mon[29140]: pgmap v648: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 37 MiB/s wr, 1.69k op/s 2024-04-25T13:50:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 585.0 s, 1707.6 tps, lat 58.352 ms stddev 69.063 2024-04-25T13:50:54.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:53 smithi122 ceph-mon[32843]: pgmap v649: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 39 MiB/s wr, 1.94k op/s 2024-04-25T13:50:54.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:53 smithi045 ceph-mon[32731]: pgmap v649: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 39 MiB/s wr, 1.94k op/s 2024-04-25T13:50:54.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:53 smithi027 ceph-mon[29140]: pgmap v649: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 39 MiB/s wr, 1.94k op/s 2024-04-25T13:50:56.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:55 smithi122 ceph-mon[32843]: pgmap v650: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 36 MiB/s wr, 1.83k op/s 2024-04-25T13:50:56.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:55 smithi045 ceph-mon[32731]: pgmap v650: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 36 MiB/s wr, 1.83k op/s 2024-04-25T13:50:56.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:55 smithi027 ceph-mon[29140]: pgmap v650: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 36 MiB/s wr, 1.83k op/s 2024-04-25T13:50:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 590.0 s, 1973.0 tps, lat 50.050 ms stddev 53.700 2024-04-25T13:50:58.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:57 smithi122 ceph-mon[32843]: pgmap v651: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 33 MiB/s wr, 1.75k op/s 2024-04-25T13:50:58.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:57 smithi045 ceph-mon[32731]: pgmap v651: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 33 MiB/s wr, 1.75k op/s 2024-04-25T13:50:58.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:57 smithi027 ceph-mon[29140]: pgmap v651: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 33 MiB/s wr, 1.75k op/s 2024-04-25T13:51:00.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:50:59 smithi122 ceph-mon[32843]: pgmap v652: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 918 op/s 2024-04-25T13:51:00.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:50:59 smithi045 ceph-mon[32731]: pgmap v652: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 918 op/s 2024-04-25T13:51:00.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:50:59 smithi027 ceph-mon[29140]: pgmap v652: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 918 op/s 2024-04-25T13:51:02.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:01 smithi122 ceph-mon[32843]: pgmap v653: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 786 op/s 2024-04-25T13:51:02.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:01 smithi045 ceph-mon[32731]: pgmap v653: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 786 op/s 2024-04-25T13:51:02.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:01 smithi027 ceph-mon[29140]: pgmap v653: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 786 op/s 2024-04-25T13:51:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 595.0 s, 2045.0 tps, lat 49.850 ms stddev 64.431 2024-04-25T13:51:04.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:03 smithi122 ceph-mon[32843]: pgmap v654: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 893 op/s 2024-04-25T13:51:04.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:03 smithi045 ceph-mon[32731]: pgmap v654: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 893 op/s 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 -- 172.21.15.27:0/1836574079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4138103c80 msgr2=0x7f4138104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 --2- 172.21.15.27:0/1836574079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4138103c80 0x7f4138104090 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f412c009900 tx=0x7f412c02a410 comp rx=0 tx=0).stop 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 -- 172.21.15.27:0/1836574079 shutdown_connections 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 --2- 172.21.15.27:0/1836574079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138108980 0x7f413810ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 --2- 172.21.15.27:0/1836574079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4138103c80 0x7f4138104090 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 --2- 172.21.15.27:0/1836574079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4138102a40 0x7f4138103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 -- 172.21.15.27:0/1836574079 >> 172.21.15.27:0/1836574079 conn(0x7f41380fdc30 msgr2=0x7f41381000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 -- 172.21.15.27:0/1836574079 shutdown_connections 2024-04-25T13:51:04.148 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.148+0000 7f413d714640 1 -- 172.21.15.27:0/1836574079 wait complete. 2024-04-25T13:51:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.149+0000 7f413d714640 1 Processor -- start 2024-04-25T13:51:04.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:03 smithi027 ceph-mon[29140]: pgmap v654: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 893 op/s 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.150+0000 7f413d714640 1 -- start start 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.150+0000 7f413d714640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4138102a40 0x7f413806b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f413d714640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138103c80 0x7f413806bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f413d714640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4138108980 0x7f41380738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f413d714640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f413810e570 con 0x7f4138108980 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f413d714640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f413810e3f0 con 0x7f4138102a40 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f413d714640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f413810e6f0 con 0x7f4138103c80 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41367fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138103c80 0x7f413806bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f4136ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4138102a40 0x7f413806b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41367fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138103c80 0x7f413806bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:48158/0 (socket says 172.21.15.27:48158) 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41367fc640 1 -- 172.21.15.27:0/400786526 learned_addr learned my addr 172.21.15.27:0/400786526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41377fe640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4138108980 0x7f41380738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f4136ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4138102a40 0x7f413806b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:55532/0 (socket says 172.21.15.27:55532) 2024-04-25T13:51:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41367fc640 1 -- 172.21.15.27:0/400786526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4138102a40 msgr2=0x7f413806b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41367fc640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4138102a40 0x7f413806b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41367fc640 1 -- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4138108980 msgr2=0x7f41380738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41367fc640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4138108980 0x7f41380738e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.151+0000 7f41367fc640 1 -- 172.21.15.27:0/400786526 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4138074100 con 0x7f4138103c80 2024-04-25T13:51:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.152+0000 7f41367fc640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138103c80 0x7f413806bd40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f412c006010 tx=0x7f412c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.152+0000 7f411ffff640 1 -- 172.21.15.27:0/400786526 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c03f030 con 0x7f4138103c80 2024-04-25T13:51:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.152+0000 7f413d714640 1 -- 172.21.15.27:0/400786526 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4138074390 con 0x7f4138103c80 2024-04-25T13:51:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.152+0000 7f411ffff640 1 -- 172.21.15.27:0/400786526 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f412c031910 con 0x7f4138103c80 2024-04-25T13:51:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.152+0000 7f413d714640 1 -- 172.21.15.27:0/400786526 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4138071850 con 0x7f4138103c80 2024-04-25T13:51:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.153+0000 7f411ffff640 1 -- 172.21.15.27:0/400786526 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c035070 con 0x7f4138103c80 2024-04-25T13:51:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f411ffff640 1 -- 172.21.15.27:0/400786526 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f412c042030 con 0x7f4138103c80 2024-04-25T13:51:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f41445c6740 1 Processor -- start 2024-04-25T13:51:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f41445c6740 1 -- start start 2024-04-25T13:51:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f411ffff640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41200afb60 0x7f41200b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f41445c6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3868970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f41445c6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 0x55b0f3868ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f41445c6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 0x55b0f386fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f41445c6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b0f37ce460 con 0x55b0f34c81b0 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f41445c6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b0f3788590 con 0x55b0f349f2f0 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.155+0000 7f41445c6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b0f37ca420 con 0x55b0f34a74d0 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f41377fe640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41200afb60 0x7f41200b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3868970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3868970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56024/0 (socket says 172.21.15.27:56024) 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f41367fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 0x55b0f3868ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 -- 172.21.15.27:0/3874667451 learned_addr learned my addr 172.21.15.27:0/3874667451 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f41377fe640 1 --2- 172.21.15.27:0/3874667451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 0x55b0f386fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 -- 172.21.15.27:0/3874667451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 msgr2=0x55b0f386fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 --2- 172.21.15.27:0/3874667451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 0x55b0f386fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 -- 172.21.15.27:0/3874667451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 msgr2=0x55b0f3868ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 --2- 172.21.15.27:0/3874667451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 0x55b0f3868ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 -- 172.21.15.27:0/3874667451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0f37950d0 con 0x55b0f34c81b0 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f411ffff640 1 -- 172.21.15.27:0/400786526 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f412c0ff070 con 0x7f4138103c80 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f41377fe640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41200afb60 0x7f41200b2070 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4130009fd0 tx=0x7f4130009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.156+0000 7f4136ffd640 1 --2- 172.21.15.27:0/3874667451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3868970 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4128009c60 tx=0x7f4128011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11410502c25c4f38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.157+0000 7f411d7fa640 1 -- 172.21.15.27:0/3874667451 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128004220 con 0x55b0f34c81b0 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.157+0000 7f411d7fa640 1 -- 172.21.15.27:0/3874667451 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41280043c0 con 0x55b0f34c81b0 2024-04-25T13:51:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.157+0000 7f411d7fa640 1 -- 172.21.15.27:0/3874667451 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128004d30 con 0x55b0f34c81b0 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.157+0000 7f41445c6740 1 -- 172.21.15.27:0/3874667451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 msgr2=0x55b0f3868970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.157+0000 7f41445c6740 1 --2- 172.21.15.27:0/3874667451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3868970 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4128009c60 tx=0x7f4128011810 comp rx=0 tx=0).stop 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 -- 172.21.15.27:0/3874667451 shutdown_connections 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 --2- 172.21.15.27:0/3874667451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 0x55b0f386fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 --2- 172.21.15.27:0/3874667451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 0x55b0f3868ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 --2- 172.21.15.27:0/3874667451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3868970 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 -- 172.21.15.27:0/3874667451 >> 172.21.15.27:0/3874667451 conn(0x55b0f34bc300 msgr2=0x55b0f3864300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 -- 172.21.15.27:0/3874667451 shutdown_connections 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 -- 172.21.15.27:0/3874667451 wait complete. 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 -- start start 2024-04-25T13:51:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.158+0000 7f41445c6740 1 Processor -- start 2024-04-25T13:51:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.159+0000 7f41445c6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 0x55b0f38725f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.159+0000 7f41445c6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f3872b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.159+0000 7f41445c6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3879a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.159+0000 7f41445c6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b0f3870840 con 0x55b0f34c81b0 2024-04-25T13:51:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.159+0000 7f41445c6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b0f3755b60 con 0x55b0f34a74d0 2024-04-25T13:51:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.159+0000 7f41445c6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b0f38709c0 con 0x55b0f349f2f0 2024-04-25T13:51:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.159+0000 7f41445c6740 10 client.0 _ll_register_callbacks cb 0x55b0f377a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.160+0000 7f41367fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f3872b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.160+0000 7f41367fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f3872b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:55554/0 (socket says 172.21.15.27:55554) 2024-04-25T13:51:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.160+0000 7f41367fc640 1 -- 172.21.15.27:0/2529373219 learned_addr learned my addr 172.21.15.27:0/2529373219 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f41367fc640 1 -- 172.21.15.27:0/2529373219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 msgr2=0x55b0f38725f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:51:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f41367fc640 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 0x55b0f38725f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f41367fc640 1 -- 172.21.15.27:0/2529373219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 msgr2=0x55b0f3879a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f41367fc640 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3879a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f41367fc640 1 -- 172.21.15.27:0/2529373219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0f34e0210 con 0x55b0f34a74d0 2024-04-25T13:51:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f41367fc640 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f3872b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f412c0fcb40 tx=0x7f412c009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=706bb8f91ecc9e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f411aff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:51:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f411aff5640 1 -- 172.21.15.27:0/2529373219 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c0c70e0 con 0x55b0f34a74d0 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.161+0000 7f411aff5640 1 -- 172.21.15.27:0/2529373219 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f412c0dc520 con 0x55b0f34a74d0 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.162+0000 7f41445c6740 1 -- 172.21.15.27:0/2529373219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41280043c0 con 0x55b0f34a74d0 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.162+0000 7f411aff5640 1 -- 172.21.15.27:0/2529373219 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f412c0e8050 con 0x55b0f34a74d0 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.162+0000 7f41445c6740 10 client.27116 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.162+0000 7f41445c6740 1 -- 172.21.15.27:0/2529373219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b0f35ac620 con 0x55b0f34a74d0 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.162+0000 7f411aff5640 1 -- 172.21.15.27:0/2529373219 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41280043c0 con 0x55b0f34a74d0 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.162+0000 7f411aff5640 1 -- 172.21.15.27:0/2529373219 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f412c1010f0 con 0x55b0f34a74d0 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.162+0000 7f41445c6740 10 client.27116 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.162+0000 7f41445c6740 10 client.27116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.163+0000 7f41445c6740 20 client.27116 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.163+0000 7f41445c6740 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b0f3486d80 0x55b0f3880360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.163+0000 7f41445c6740 4 client.27116 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.163+0000 7f41445c6740 1 -- 172.21.15.27:0/2529373219 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0f37dc070 con 0x55b0f3486d80 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.163+0000 7f4136ffd640 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b0f3486d80 0x55b0f3880360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.163+0000 7f411aff5640 10 client.27116 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.163+0000 7f4136ffd640 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b0f3486d80 0x55b0f3880360 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82059ff7f67e34f4 server_cookie=1044471d2420b7b8 in_seq=0 out_seq=0 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f411aff5640 1 -- 172.21.15.27:0/2529373219 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b0f37dc070 con 0x55b0f3486d80 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f411aff5640 10 client.27116 handle_command_reply: tid=0 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 client.27116 shutdown 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 -- 172.21.15.27:0/2529373219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 msgr2=0x55b0f3872b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f3872b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f412c0fcb40 tx=0x7f412c009620 comp rx=0 tx=0).stop 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 -- 172.21.15.27:0/2529373219 shutdown_connections 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b0f3486d80 0x55b0f3880360 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3879a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f3872b30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f411aff5640 0 client.27116 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 --2- 172.21.15.27:0/2529373219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 0x55b0f38725f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 -- 172.21.15.27:0/2529373219 >> 172.21.15.27:0/2529373219 conn(0x55b0f34bc300 msgr2=0x55b0f3863250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:04.168 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 -- 172.21.15.27:0/2529373219 shutdown_connections 2024-04-25T13:51:04.168 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.164+0000 7f41445c6740 1 -- 172.21.15.27:0/2529373219 wait complete. 2024-04-25T13:51:04.168 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.165+0000 7f41445c6740 20 client.27116 trim_cache size 0 max 16384 2024-04-25T13:51:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.186+0000 7f41445c6740 1 Processor -- start 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.186+0000 7f41445c6740 1 -- start start 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41445c6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 0x55b0f3863be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41445c6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f38799b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41445c6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3864120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41445c6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41381af780 con 0x55b0f34c81b0 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41445c6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b2000 con 0x55b0f34a74d0 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41445c6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f41381afdf0 con 0x55b0f349f2f0 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41367fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f38799b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3864120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f4136ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 0x55b0f3863be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41367fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f38799b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:55570/0 (socket says 172.21.15.27:55570) 2024-04-25T13:51:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41367fc640 1 -- 172.21.15.27:0/2166419725 learned_addr learned my addr 172.21.15.27:0/2166419725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:04.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41367fc640 1 -- 172.21.15.27:0/2166419725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 msgr2=0x55b0f3863be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:04.190 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41367fc640 1 --2- 172.21.15.27:0/2166419725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 0x55b0f3863be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41367fc640 1 -- 172.21.15.27:0/2166419725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 msgr2=0x55b0f3864120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41367fc640 1 --2- 172.21.15.27:0/2166419725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3864120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f41367fc640 1 -- 172.21.15.27:0/2166419725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0f38701b0 con 0x55b0f34a74d0 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.187+0000 7f4136ffd640 1 --2- 172.21.15.27:0/2166419725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 0x55b0f3863be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.188+0000 7f41367fc640 1 --2- 172.21.15.27:0/2166419725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f38799b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f412c0c7a70 tx=0x7f412c0c7af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=314526c4a0f27a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.188+0000 7f411cff9640 1 -- 172.21.15.27:0/2166419725 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c0d5070 con 0x55b0f34a74d0 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.188+0000 7f411cff9640 1 -- 172.21.15.27:0/2166419725 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f412c0d0380 con 0x55b0f34a74d0 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.188+0000 7f411cff9640 1 -- 172.21.15.27:0/2166419725 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c0cff40 con 0x55b0f34a74d0 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 -- 172.21.15.27:0/2166419725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 msgr2=0x55b0f38799b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 --2- 172.21.15.27:0/2166419725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f38799b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f412c0c7a70 tx=0x7f412c0c7af0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 -- 172.21.15.27:0/2166419725 shutdown_connections 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 --2- 172.21.15.27:0/2166419725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3864120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 --2- 172.21.15.27:0/2166419725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f34a74d0 0x55b0f38799b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 --2- 172.21.15.27:0/2166419725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f349f2f0 0x55b0f3863be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 -- 172.21.15.27:0/2166419725 >> 172.21.15.27:0/2166419725 conn(0x55b0f34bc300 msgr2=0x55b0f386ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 -- 172.21.15.27:0/2166419725 shutdown_connections 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 -- 172.21.15.27:0/2166419725 wait complete. 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 -- start start 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.189+0000 7f41445c6740 1 Processor -- start 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41445c6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 0x55b0f3876f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41445c6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 0x55b0f3877460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41445c6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3878340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41445c6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b04c0 con 0x55b0f34c81b0 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41445c6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b1930 con 0x55b0f349f2f0 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41445c6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f41381b0b90 con 0x55b0f34a74d0 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3878340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f4136ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 0x55b0f3876f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3878340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56064/0 (socket says 172.21.15.27:56064) 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 -- 172.21.15.27:0/1083920971 learned_addr learned my addr 172.21.15.27:0/1083920971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41367fc640 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 0x55b0f3877460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41445c6740 10 client.0 _ll_register_callbacks cb 0x7f412c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 -- 172.21.15.27:0/1083920971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 msgr2=0x55b0f3877460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 0x55b0f3877460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 -- 172.21.15.27:0/1083920971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 msgr2=0x55b0f3876f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 0x55b0f3876f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 -- 172.21.15.27:0/1083920971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0f3879ef0 con 0x55b0f34c81b0 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f41377fe640 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3878340 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f41300108c0 tx=0x7f4130012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b0347d07cbd966 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.190+0000 7f411aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.191+0000 7f411aff5640 1 -- 172.21.15.27:0/1083920971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130031060 con 0x55b0f34c81b0 2024-04-25T13:51:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.191+0000 7f411aff5640 1 -- 172.21.15.27:0/1083920971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4130019070 con 0x55b0f34c81b0 2024-04-25T13:51:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.191+0000 7f41445c6740 1 -- 172.21.15.27:0/1083920971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f412c0d0380 con 0x55b0f34c81b0 2024-04-25T13:51:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.191+0000 7f411aff5640 1 -- 172.21.15.27:0/1083920971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4130003b80 con 0x55b0f34c81b0 2024-04-25T13:51:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.191+0000 7f41445c6740 10 client.17415 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.191+0000 7f41445c6740 1 -- 172.21.15.27:0/1083920971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b0f3879a00 con 0x55b0f34c81b0 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.193+0000 7f411aff5640 1 -- 172.21.15.27:0/1083920971 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f412c0d0380 con 0x55b0f34c81b0 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f411aff5640 1 -- 172.21.15.27:0/1083920971 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f413001f040 con 0x55b0f34c81b0 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f41445c6740 10 client.17415 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f41445c6740 10 client.17415 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f41445c6740 20 client.17415 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f41445c6740 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b0f345c1c0 0x55b0f38eacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f41445c6740 4 client.17415 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f41445c6740 1 -- 172.21.15.27:0/1083920971 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b0f37d5ee0 con 0x55b0f345c1c0 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f4136ffd640 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b0f345c1c0 0x55b0f38eacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.194+0000 7f4136ffd640 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b0f345c1c0 0x55b0f38eacb0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46c3203ccde7962b server_cookie=5c0926e9c99062a5 in_seq=0 out_seq=0 2024-04-25T13:51:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.195+0000 7f411aff5640 10 client.17415 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.256+0000 7f411aff5640 1 -- 172.21.15.27:0/1083920971 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b0f37d5ee0 con 0x55b0f345c1c0 2024-04-25T13:51:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.256+0000 7f411aff5640 10 client.17415 handle_command_reply: tid=0 2024-04-25T13:51:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.256+0000 7f41445c6740 1 client.17415 shutdown 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.256+0000 7f41445c6740 1 -- 172.21.15.27:0/1083920971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 msgr2=0x55b0f3878340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.256+0000 7f41445c6740 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3878340 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f41300108c0 tx=0x7f4130012040 comp rx=0 tx=0).stop 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 1 -- 172.21.15.27:0/1083920971 shutdown_connections 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b0f345c1c0 0x55b0f38eacb0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f411aff5640 0 client.17415 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0f34c81b0 0x55b0f3878340 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b0f34a74d0 0x55b0f3877460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 1 --2- 172.21.15.27:0/1083920971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b0f349f2f0 0x55b0f3876f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 1 -- 172.21.15.27:0/1083920971 >> 172.21.15.27:0/1083920971 conn(0x55b0f34bc300 msgr2=0x55b0f3863be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 1 -- 172.21.15.27:0/1083920971 shutdown_connections 2024-04-25T13:51:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 1 -- 172.21.15.27:0/1083920971 wait complete. 2024-04-25T13:51:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.257+0000 7f41445c6740 20 client.17415 trim_cache size 0 max 16384 2024-04-25T13:51:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:51:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:51:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:51:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:51:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 -- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41200afb60 msgr2=0x7f41200b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41200afb60 0x7f41200b2070 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4130009fd0 tx=0x7f4130009500 comp rx=0 tx=0).stop 2024-04-25T13:51:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 -- 172.21.15.27:0/400786526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138103c80 msgr2=0x7f413806bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138103c80 0x7f413806bd40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f412c006010 tx=0x7f412c0042f0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 -- 172.21.15.27:0/400786526 shutdown_connections 2024-04-25T13:51:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f41200afb60 0x7f41200b2070 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4138108980 0x7f41380738e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4138103c80 0x7f413806bd40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 --2- 172.21.15.27:0/400786526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4138102a40 0x7f413806b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.258+0000 7f413d714640 1 -- 172.21.15.27:0/400786526 >> 172.21.15.27:0/400786526 conn(0x7f41380fdc30 msgr2=0x7f4138104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.259+0000 7f413d714640 1 -- 172.21.15.27:0/400786526 shutdown_connections 2024-04-25T13:51:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:04.259+0000 7f413d714640 1 -- 172.21.15.27:0/400786526 wait complete. 2024-04-25T13:51:06.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:05 smithi122 ceph-mon[32843]: pgmap v655: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.9 MiB/s wr, 664 op/s 2024-04-25T13:51:06.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:05 smithi045 ceph-mon[32731]: pgmap v655: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.9 MiB/s wr, 664 op/s 2024-04-25T13:51:06.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:05 smithi027 ceph-mon[29140]: pgmap v655: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.9 MiB/s wr, 664 op/s 2024-04-25T13:51:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 600.0 s, 1932.2 tps, lat 51.209 ms stddev 62.562 2024-04-25T13:51:08.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:07 smithi122 ceph-mon[32843]: pgmap v656: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.5 MiB/s wr, 714 op/s 2024-04-25T13:51:08.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:07 smithi045 ceph-mon[32731]: pgmap v656: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.5 MiB/s wr, 714 op/s 2024-04-25T13:51:08.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:07 smithi027 ceph-mon[29140]: pgmap v656: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.5 MiB/s wr, 714 op/s 2024-04-25T13:51:10.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:09 smithi122 ceph-mon[32843]: pgmap v657: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 580 op/s 2024-04-25T13:51:10.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:09 smithi045 ceph-mon[32731]: pgmap v657: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 580 op/s 2024-04-25T13:51:10.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:09 smithi027 ceph-mon[29140]: pgmap v657: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 580 op/s 2024-04-25T13:51:12.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:11 smithi122 ceph-mon[32843]: pgmap v658: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 653 op/s 2024-04-25T13:51:12.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:11 smithi045 ceph-mon[32731]: pgmap v658: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 653 op/s 2024-04-25T13:51:12.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:11 smithi027 ceph-mon[29140]: pgmap v658: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 653 op/s 2024-04-25T13:51:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 605.0 s, 1642.2 tps, lat 61.137 ms stddev 70.112 2024-04-25T13:51:14.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:13 smithi122 ceph-mon[32843]: pgmap v659: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 709 op/s 2024-04-25T13:51:14.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:13 smithi045 ceph-mon[32731]: pgmap v659: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 709 op/s 2024-04-25T13:51:14.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:13 smithi027 ceph-mon[29140]: pgmap v659: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.3 MiB/s wr, 709 op/s 2024-04-25T13:51:14.889 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:51:14.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f0cce640 1 -- 172.21.15.27:0/371021382 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4030d70 con 0x7fb5ec10ba70 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 -- 172.21.15.27:0/371021382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 msgr2=0x7fb5ec10deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/371021382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 0x7fb5ec10deb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009be0 tx=0x7fb5e402a6c0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 -- 172.21.15.27:0/371021382 shutdown_connections 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/371021382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 0x7fb5ec10deb0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/371021382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5ec0769f0 0x7fb5ec10b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/371021382 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5ec0760a0 0x7fb5ec0764b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 -- 172.21.15.27:0/371021382 >> 172.21.15.27:0/371021382 conn(0x7fb5ec0fed10 msgr2=0x7fb5ec1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 -- 172.21.15.27:0/371021382 shutdown_connections 2024-04-25T13:51:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.048+0000 7fb5f3f34640 1 -- 172.21.15.27:0/371021382 wait complete. 2024-04-25T13:51:15.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f3f34640 1 Processor -- start 2024-04-25T13:51:15.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f3f34640 1 -- start start 2024-04-25T13:51:15.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f3f34640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5ec0760a0 0x7fb5ec19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f3f34640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5ec0769f0 0x7fb5ec19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f3f34640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 0x7fb5ec1a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f3f34640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec111880 con 0x7fb5ec10ba70 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f3f34640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec111700 con 0x7fb5ec0769f0 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f3f34640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec111a00 con 0x7fb5ec0760a0 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f24d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 0x7fb5ec1a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f24d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 0x7fb5ec1a3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56982/0 (socket says 172.21.15.27:56982) 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f24d1640 1 -- 172.21.15.27:0/2334118454 learned_addr learned my addr 172.21.15.27:0/2334118454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5ec0760a0 0x7fb5ec19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5ec0769f0 0x7fb5ec19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f24d1640 1 -- 172.21.15.27:0/2334118454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5ec0760a0 msgr2=0x7fb5ec19c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f24d1640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5ec0760a0 0x7fb5ec19c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f24d1640 1 -- 172.21.15.27:0/2334118454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5ec0769f0 msgr2=0x7fb5ec19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f24d1640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5ec0769f0 0x7fb5ec19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.049+0000 7fb5f24d1640 1 -- 172.21.15.27:0/2334118454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ec1a3ed0 con 0x7fb5ec10ba70 2024-04-25T13:51:15.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.050+0000 7fb5f24d1640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 0x7fb5ec1a3910 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4034040 tx=0x7fb5e4032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.050+0000 7fb5e2ffd640 1 -- 172.21.15.27:0/2334118454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4041020 con 0x7fb5ec10ba70 2024-04-25T13:51:15.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.050+0000 7fb5f3f34640 1 -- 172.21.15.27:0/2334118454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ec1a4160 con 0x7fb5ec10ba70 2024-04-25T13:51:15.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.050+0000 7fb5f3f34640 1 -- 172.21.15.27:0/2334118454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ec1a46a0 con 0x7fb5ec10ba70 2024-04-25T13:51:15.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.050+0000 7fb5e2ffd640 1 -- 172.21.15.27:0/2334118454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e4003b70 con 0x7fb5ec10ba70 2024-04-25T13:51:15.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.050+0000 7fb5e2ffd640 1 -- 172.21.15.27:0/2334118454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4036040 con 0x7fb5ec10ba70 2024-04-25T13:51:15.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.053+0000 7fb5fade6740 1 Processor -- start 2024-04-25T13:51:15.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.054+0000 7fb5e2ffd640 1 -- 172.21.15.27:0/2334118454 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb5e4044020 con 0x7fb5ec10ba70 2024-04-25T13:51:15.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.054+0000 7fb5fade6740 1 -- start start 2024-04-25T13:51:15.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.054+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 0x562a9f0dfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.054+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0dffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.054+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.054+0000 7fb5fade6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a9f044d80 con 0x562a9ed3e390 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.054+0000 7fb5fade6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562a9effe8b0 con 0x562a9ed154d0 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.054+0000 7fb5fade6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562a9f040d40 con 0x562a9ed1d6b0 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f14cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f14cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 0x562a9f0dfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f14cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 0x562a9f0dfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56994/0 (socket says 172.21.15.27:56994) 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f1cd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0dffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f14cf640 1 -- 172.21.15.27:0/3015344940 learned_addr learned my addr 172.21.15.27:0/3015344940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5e2ffd640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afaf0 0x7fb5d80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f1cd0640 1 -- 172.21.15.27:0/3015344940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 msgr2=0x562a9f0e0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/3015344940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f1cd0640 1 -- 172.21.15.27:0/3015344940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 msgr2=0x562a9f0dfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/3015344940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 0x562a9f0dfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5e2ffd640 1 -- 172.21.15.27:0/2334118454 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5e40ff390 con 0x7fb5ec10ba70 2024-04-25T13:51:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f1cd0640 1 -- 172.21.15.27:0/3015344940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a9efcfa40 con 0x562a9ed154d0 2024-04-25T13:51:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.055+0000 7fb5f24d1640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afaf0 0x7fb5d80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.056+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/3015344940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0dffe0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8005fd0 tx=0x7fb5e8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c702fe240eed3f08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.056+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3015344940 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8004230 con 0x562a9ed154d0 2024-04-25T13:51:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.056+0000 7fb5f24d1640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afaf0 0x7fb5d80b2000 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb5e402ac80 tx=0x7fb5e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.056+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3015344940 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e8009e50 con 0x562a9ed154d0 2024-04-25T13:51:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.056+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3015344940 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8019640 con 0x562a9ed154d0 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.057+0000 7fb5fade6740 1 -- 172.21.15.27:0/3015344940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 msgr2=0x562a9f0dffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.057+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3015344940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0dffe0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8005fd0 tx=0x7fb5e8011810 comp rx=0 tx=0).stop 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.057+0000 7fb5fade6740 1 -- 172.21.15.27:0/3015344940 shutdown_connections 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.057+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3015344940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.057+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3015344940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0dffe0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.057+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3015344940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 0x562a9f0dfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.057+0000 7fb5fade6740 1 -- 172.21.15.27:0/3015344940 >> 172.21.15.27:0/3015344940 conn(0x562a9ed324e0 msgr2=0x562a9f0db270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.058+0000 7fb5fade6740 1 -- 172.21.15.27:0/3015344940 shutdown_connections 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.058+0000 7fb5fade6740 1 -- 172.21.15.27:0/3015344940 wait complete. 2024-04-25T13:51:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.058+0000 7fb5fade6740 1 -- start start 2024-04-25T13:51:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.058+0000 7fb5fade6740 1 Processor -- start 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0e98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 0x562a9f0f0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5fade6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562a9f0e1780 con 0x562a9ed3e390 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5fade6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562a9efcbd10 con 0x562a9ed154d0 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5fade6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562a9f0e1900 con 0x562a9ed1d6b0 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5fade6740 10 client.0 _ll_register_callbacks cb 0x562a9eff8210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5f1cd0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5f14cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0e98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.059+0000 7fb5f1cd0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37190/0 (socket says 172.21.15.27:37190) 2024-04-25T13:51:15.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.060+0000 7fb5f1cd0640 1 -- 172.21.15.27:0/1398113162 learned_addr learned my addr 172.21.15.27:0/1398113162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.060+0000 7fb5f1cd0640 1 -- 172.21.15.27:0/1398113162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 msgr2=0x562a9f0e98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.060+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0e98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.060+0000 7fb5f1cd0640 1 -- 172.21.15.27:0/1398113162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 msgr2=0x562a9f0f0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.060+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 0x562a9f0f0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.060+0000 7fb5f1cd0640 1 -- 172.21.15.27:0/1398113162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a9ee207d0 con 0x562a9ed1d6b0 2024-04-25T13:51:15.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.061+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e9df0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009640 tx=0x7fb5e8005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e34ec9e6061ac9c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.061+0000 7fb5d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:15.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.061+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0e98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:51:15.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.061+0000 7fb5d57fa640 1 -- 172.21.15.27:0/1398113162 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8028020 con 0x562a9ed1d6b0 2024-04-25T13:51:15.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.061+0000 7fb5d57fa640 1 -- 172.21.15.27:0/1398113162 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5e8005430 con 0x562a9ed1d6b0 2024-04-25T13:51:15.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.064+0000 7fb5fade6740 1 -- 172.21.15.27:0/1398113162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5e8009e50 con 0x562a9ed1d6b0 2024-04-25T13:51:15.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.071+0000 7fb5d57fa640 1 -- 172.21.15.27:0/1398113162 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb5e801f040 con 0x562a9ed1d6b0 2024-04-25T13:51:15.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.076+0000 7fb5fade6740 10 client.27172 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:15.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.076+0000 7fb5fade6740 1 -- 172.21.15.27:0/1398113162 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562a9ed563f0 con 0x562a9ed1d6b0 2024-04-25T13:51:15.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.082+0000 7fb5d57fa640 1 -- 172.21.15.27:0/1398113162 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e8009e50 con 0x562a9ed1d6b0 2024-04-25T13:51:15.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.082+0000 7fb5d57fa640 1 -- 172.21.15.27:0/1398113162 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb5e8035020 con 0x562a9ed1d6b0 2024-04-25T13:51:15.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.086+0000 7fb5fade6740 10 client.27172 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:15.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.086+0000 7fb5fade6740 10 client.27172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:15.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.086+0000 7fb5fade6740 20 client.27172 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:15.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.086+0000 7fb5fade6740 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a9ecfcf60 0x562a9f0f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.086+0000 7fb5fade6740 4 client.27172 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:15.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.086+0000 7fb5fade6740 1 -- 172.21.15.27:0/1398113162 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a9f052990 con 0x562a9ecfcf60 2024-04-25T13:51:15.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.091+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a9ecfcf60 0x562a9f0f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.111+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a9ecfcf60 0x562a9f0f70c0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37f78c66da68077b server_cookie=42277c99542d99ae in_seq=0 out_seq=0 2024-04-25T13:51:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.112+0000 7fb5d57fa640 10 client.27172 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.112+0000 7fb5d57fa640 1 -- 172.21.15.27:0/1398113162 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562a9f052990 con 0x562a9ecfcf60 2024-04-25T13:51:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.112+0000 7fb5d57fa640 10 client.27172 handle_command_reply: tid=0 2024-04-25T13:51:15.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.114+0000 7fb5fade6740 1 client.27172 shutdown 2024-04-25T13:51:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.121+0000 7fb5fade6740 1 -- 172.21.15.27:0/1398113162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 msgr2=0x562a9f0e9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.121+0000 7fb5fade6740 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e9df0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009640 tx=0x7fb5e8005040 comp rx=0 tx=0).stop 2024-04-25T13:51:15.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.125+0000 7fb5fade6740 1 -- 172.21.15.27:0/1398113162 shutdown_connections 2024-04-25T13:51:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.125+0000 7fb5fade6740 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a9ecfcf60 0x562a9f0f70c0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.126+0000 7fb5fade6740 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed3e390 0x562a9f0f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.126+0000 7fb5fade6740 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed1d6b0 0x562a9f0e9df0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.126+0000 7fb5fade6740 1 --2- 172.21.15.27:0/1398113162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed154d0 0x562a9f0e98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.126+0000 7fb5fade6740 1 -- 172.21.15.27:0/1398113162 >> 172.21.15.27:0/1398113162 conn(0x562a9ed324e0 msgr2=0x562a9f0de180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.127+0000 7fb5d57fa640 0 client.27172 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.127+0000 7fb5fade6740 1 -- 172.21.15.27:0/1398113162 shutdown_connections 2024-04-25T13:51:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.127+0000 7fb5fade6740 1 -- 172.21.15.27:0/1398113162 wait complete. 2024-04-25T13:51:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.128+0000 7fb5fade6740 20 client.27172 trim_cache size 0 max 16384 2024-04-25T13:51:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.164+0000 7fb5fade6740 1 Processor -- start 2024-04-25T13:51:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.164+0000 7fb5fade6740 1 -- start start 2024-04-25T13:51:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed154d0 0x562a9f0dac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed1d6b0 0x562a9f0db170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed3e390 0x562a9f0f0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5fade6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d000c450 con 0x562a9ed154d0 2024-04-25T13:51:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5fade6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d000ecd0 con 0x562a9ed1d6b0 2024-04-25T13:51:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5fade6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d000cac0 con 0x562a9ed3e390 2024-04-25T13:51:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5f14cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed3e390 0x562a9f0f0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5f14cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed154d0 0x562a9f0dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5f14cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed154d0 0x562a9f0dac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57014/0 (socket says 172.21.15.27:57014) 2024-04-25T13:51:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.165+0000 7fb5f14cf640 1 -- 172.21.15.27:0/4088909488 learned_addr learned my addr 172.21.15.27:0/4088909488 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed1d6b0 0x562a9f0db170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5f14cf640 1 -- 172.21.15.27:0/4088909488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed3e390 msgr2=0x562a9f0f0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed3e390 0x562a9f0f0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5f14cf640 1 -- 172.21.15.27:0/4088909488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed1d6b0 msgr2=0x562a9f0db170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed1d6b0 0x562a9f0db170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5f14cf640 1 -- 172.21.15.27:0/4088909488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a9ed160f0 con 0x562a9ed154d0 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed1d6b0 0x562a9f0db170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed154d0 0x562a9f0dac30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x562a9f0e0ab0 tx=0x7fb5dc00a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3461af0b21587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.166+0000 7fb5d77fe640 1 -- 172.21.15.27:0/4088909488 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc01f980 con 0x562a9ed154d0 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.167+0000 7fb5d77fe640 1 -- 172.21.15.27:0/4088909488 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5dc0054d0 con 0x562a9ed154d0 2024-04-25T13:51:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.167+0000 7fb5d77fe640 1 -- 172.21.15.27:0/4088909488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc00fc60 con 0x562a9ed154d0 2024-04-25T13:51:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.167+0000 7fb5fade6740 1 -- 172.21.15.27:0/4088909488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed154d0 msgr2=0x562a9f0dac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.167+0000 7fb5fade6740 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed154d0 0x562a9f0dac30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x562a9f0e0ab0 tx=0x7fb5dc00a2d0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.168+0000 7fb5fade6740 1 -- 172.21.15.27:0/4088909488 shutdown_connections 2024-04-25T13:51:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.168+0000 7fb5fade6740 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ed3e390 0x562a9f0f0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.168+0000 7fb5fade6740 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed1d6b0 0x562a9f0db170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.168+0000 7fb5fade6740 1 --2- 172.21.15.27:0/4088909488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed154d0 0x562a9f0dac30 secure :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0x562a9f0e0ab0 tx=0x7fb5dc00a2d0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.168+0000 7fb5fade6740 1 -- 172.21.15.27:0/4088909488 >> 172.21.15.27:0/4088909488 conn(0x562a9ed324e0 msgr2=0x562a9f0e45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.169+0000 7fb5fade6740 1 -- 172.21.15.27:0/4088909488 shutdown_connections 2024-04-25T13:51:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.169+0000 7fb5fade6740 1 -- 172.21.15.27:0/4088909488 wait complete. 2024-04-25T13:51:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.169+0000 7fb5fade6740 1 -- start start 2024-04-25T13:51:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.170+0000 7fb5fade6740 1 Processor -- start 2024-04-25T13:51:15.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.170+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed1d6b0 0x562a9f0ee280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.170+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed3e390 0x562a9f0ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5fade6740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ecd23a0 0x562a9f15d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f1cd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed3e390 0x562a9f0ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f1cd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed3e390 0x562a9f0ee7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45502/0 (socket says 172.21.15.27:45502) 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5fade6740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d000d190 con 0x562a9ed1d6b0 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f14cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ecd23a0 0x562a9f15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5fade6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d000e600 con 0x562a9ed3e390 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5fade6740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d000d860 con 0x562a9ecd23a0 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f1cd0640 1 -- 172.21.15.27:0/3625015522 learned_addr learned my addr 172.21.15.27:0/3625015522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed1d6b0 0x562a9f0ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5fade6740 10 client.0 _ll_register_callbacks cb 0x7fb5dc00aa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f14cf640 1 -- 172.21.15.27:0/3625015522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ecd23a0 msgr2=0x562a9f15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ecd23a0 0x562a9f15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f14cf640 1 -- 172.21.15.27:0/3625015522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed3e390 msgr2=0x562a9f0ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed3e390 0x562a9f0ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f14cf640 1 -- 172.21.15.27:0/3625015522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a9f0054b0 con 0x562a9ed1d6b0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.171+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed3e390 0x562a9f0ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5f14cf640 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed1d6b0 0x562a9f0ee280 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x562a9f0ea6c0 tx=0x7fb5dc0113d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e515c3185bcef9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5d57fa640 1 -- 172.21.15.27:0/3625015522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc030030 con 0x562a9ed1d6b0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5d57fa640 1 -- 172.21.15.27:0/3625015522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5dc01f980 con 0x562a9ed1d6b0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5fade6740 1 -- 172.21.15.27:0/3625015522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5dc0054d0 con 0x562a9ed1d6b0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5d57fa640 1 -- 172.21.15.27:0/3625015522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb5dc033040 con 0x562a9ed1d6b0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5fade6740 10 client.17430 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5fade6740 1 -- 172.21.15.27:0/3625015522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562a9f15e740 con 0x562a9ed1d6b0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.172+0000 7fb5d57fa640 1 -- 172.21.15.27:0/3625015522 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5dc0054d0 con 0x562a9ed1d6b0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.173+0000 7fb5d57fa640 1 -- 172.21.15.27:0/3625015522 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb5dc011a80 con 0x562a9ed1d6b0 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.173+0000 7fb5fade6740 10 client.17430 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.173+0000 7fb5fade6740 10 client.17430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.173+0000 7fb5fade6740 20 client.17430 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.173+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a9f15f080 0x562a9f1614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.173+0000 7fb5fade6740 4 client.17430 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:51:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.173+0000 7fb5fade6740 1 -- 172.21.15.27:0/3625015522 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562a9f04c800 con 0x562a9f15f080 2024-04-25T13:51:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.174+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a9f15f080 0x562a9f1614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.174+0000 7fb5d57fa640 10 client.17430 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:15.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.174+0000 7fb5f1cd0640 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a9f15f080 0x562a9f1614a0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41aeee23806e4abc server_cookie=bffa4e9b14971d90 in_seq=0 out_seq=0 2024-04-25T13:51:15.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5d57fa640 1 -- 172.21.15.27:0/3625015522 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562a9f04c800 con 0x562a9f15f080 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5d57fa640 10 client.17430 handle_command_reply: tid=0 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 client.17430 shutdown 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 -- 172.21.15.27:0/3625015522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed1d6b0 msgr2=0x562a9f0ee280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed1d6b0 0x562a9f0ee280 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x562a9f0ea6c0 tx=0x7fb5dc0113d0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 -- 172.21.15.27:0/3625015522 shutdown_connections 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562a9f15f080 0x562a9f1614a0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562a9ecd23a0 0x562a9f15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a9ed3e390 0x562a9f0ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 --2- 172.21.15.27:0/3625015522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562a9ed1d6b0 0x562a9f0ee280 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 -- 172.21.15.27:0/3625015522 >> 172.21.15.27:0/3625015522 conn(0x562a9ed324e0 msgr2=0x562a9f0dac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5d57fa640 0 client.17430 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 -- 172.21.15.27:0/3625015522 shutdown_connections 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.175+0000 7fb5fade6740 1 -- 172.21.15.27:0/3625015522 wait complete. 2024-04-25T13:51:15.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5fade6740 20 client.17430 trim_cache size 0 max 16384 2024-04-25T13:51:15.178 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:51:15.178 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:51:15.178 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:51:15.178 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 -- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afaf0 msgr2=0x7fb5d80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afaf0 0x7fb5d80b2000 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb5e402ac80 tx=0x7fb5e4009500 comp rx=0 tx=0).stop 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 -- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 msgr2=0x7fb5ec1a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 0x7fb5ec1a3910 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4034040 tx=0x7fb5e4032f60 comp rx=0 tx=0).stop 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 -- 172.21.15.27:0/2334118454 shutdown_connections 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afaf0 0x7fb5d80b2000 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5ec10ba70 0x7fb5ec1a3910 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5ec0769f0 0x7fb5ec19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 --2- 172.21.15.27:0/2334118454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5ec0760a0 0x7fb5ec19c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 -- 172.21.15.27:0/2334118454 >> 172.21.15.27:0/2334118454 conn(0x7fb5ec0fed10 msgr2=0x7fb5ec0ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 -- 172.21.15.27:0/2334118454 shutdown_connections 2024-04-25T13:51:15.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.176+0000 7fb5f3f34640 1 -- 172.21.15.27:0/2334118454 wait complete. 2024-04-25T13:51:15.193 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:51:15.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:51:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 -- 172.21.15.27:0/1988744169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 msgr2=0x7f5c3c10e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 --2- 172.21.15.27:0/1988744169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 0x7f5c3c10e060 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009ca0 tx=0x7f5c3402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 -- 172.21.15.27:0/1988744169 shutdown_connections 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 --2- 172.21.15.27:0/1988744169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 0x7f5c3c10e060 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 --2- 172.21.15.27:0/1988744169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c106f40 0x7f5c3c107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 --2- 172.21.15.27:0/1988744169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c3c105d00 0x7f5c3c106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 -- 172.21.15.27:0/1988744169 >> 172.21.15.27:0/1988744169 conn(0x7f5c3c100ef0 msgr2=0x7f5c3c1033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 -- 172.21.15.27:0/1988744169 shutdown_connections 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.359+0000 7f5c440a7640 1 -- 172.21.15.27:0/1988744169 wait complete. 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.360+0000 7f5c440a7640 1 Processor -- start 2024-04-25T13:51:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.360+0000 7f5c440a7640 1 -- start start 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.360+0000 7f5c440a7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c3c105d00 0x7f5c3c1a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.360+0000 7f5c440a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c106f40 0x7f5c3c1a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.360+0000 7f5c440a7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 0x7f5c3c1a7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.360+0000 7f5c440a7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c078950 con 0x7f5c3c10bc40 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.360+0000 7f5c440a7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c0787d0 con 0x7f5c3c106f40 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.360+0000 7f5c440a7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c078ad0 con 0x7f5c3c105d00 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c41642640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c106f40 0x7f5c3c1a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c41642640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c106f40 0x7f5c3c1a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45514/0 (socket says 172.21.15.27:45514) 2024-04-25T13:51:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c41642640 1 -- 172.21.15.27:0/101160835 learned_addr learned my addr 172.21.15.27:0/101160835 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c42644640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 0x7f5c3c1a7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c41e43640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c3c105d00 0x7f5c3c1a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c42644640 1 -- 172.21.15.27:0/101160835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c3c105d00 msgr2=0x7f5c3c1a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c42644640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c3c105d00 0x7f5c3c1a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c42644640 1 -- 172.21.15.27:0/101160835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c106f40 msgr2=0x7f5c3c1a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c42644640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c106f40 0x7f5c3c1a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.361+0000 7f5c42644640 1 -- 172.21.15.27:0/101160835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c1a8320 con 0x7f5c3c10bc40 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.362+0000 7f5c42644640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 0x7f5c3c1a7cb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f5c34033040 tx=0x7f5c34031680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.362+0000 7f5c32ffd640 1 -- 172.21.15.27:0/101160835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34003b70 con 0x7f5c3c10bc40 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.362+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3c1a85b0 con 0x7f5c3c10bc40 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.362+0000 7f5c32ffd640 1 -- 172.21.15.27:0/101160835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c34003d10 con 0x7f5c3c10bc40 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.362+0000 7f5c32ffd640 1 -- 172.21.15.27:0/101160835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3402f970 con 0x7f5c3c10bc40 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.362+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c3c1a8a90 con 0x7f5c3c10bc40 2024-04-25T13:51:15.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.363+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c3c07c150 con 0x7f5c3c10bc40 2024-04-25T13:51:15.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.366+0000 7f5c32ffd640 1 -- 172.21.15.27:0/101160835 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5c34004b40 con 0x7f5c3c10bc40 2024-04-25T13:51:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.367+0000 7f5c32ffd640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5c280afbb0 0x7f5c280b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.367+0000 7f5c41e43640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5c280afbb0 0x7f5c280b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.367+0000 7f5c32ffd640 1 -- 172.21.15.27:0/101160835 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f5c34100120 con 0x7f5c3c10bc40 2024-04-25T13:51:15.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.367+0000 7f5c41e43640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5c280afbb0 0x7f5c280b20c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c106900 tx=0x7f5c38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.372+0000 7f5c32ffd640 1 -- 172.21.15.27:0/101160835 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c340c8cd0 con 0x7f5c3c10bc40 2024-04-25T13:51:15.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.746+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c3c02db50 con 0x7f5c3c10bc40 2024-04-25T13:51:15.751 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:51:15.751 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:51:15.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.751+0000 7f5c32ffd640 1 -- 172.21.15.27:0/101160835 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f5c340c85d0 con 0x7f5c3c10bc40 2024-04-25T13:51:15.752 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:51:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.759+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5c280afbb0 msgr2=0x7f5c280b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.759+0000 7f5c440a7640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5c280afbb0 0x7f5c280b20c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c106900 tx=0x7f5c38005f90 comp rx=0 tx=0).stop 2024-04-25T13:51:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.759+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 msgr2=0x7f5c3c1a7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.759+0000 7f5c440a7640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 0x7f5c3c1a7cb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f5c34033040 tx=0x7f5c34031680 comp rx=0 tx=0).stop 2024-04-25T13:51:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.761+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 shutdown_connections 2024-04-25T13:51:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.761+0000 7f5c440a7640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f5c280afbb0 0x7f5c280b20c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.761+0000 7f5c440a7640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c3c10bc40 0x7f5c3c1a7cb0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.761+0000 7f5c440a7640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c3c106f40 0x7f5c3c1a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.761+0000 7f5c440a7640 1 --2- 172.21.15.27:0/101160835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c3c105d00 0x7f5c3c1a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.761+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 >> 172.21.15.27:0/101160835 conn(0x7f5c3c100ef0 msgr2=0x7f5c3c079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.765+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 shutdown_connections 2024-04-25T13:51:15.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.767+0000 7f5c440a7640 1 -- 172.21.15.27:0/101160835 wait complete. 2024-04-25T13:51:15.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:51:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.973+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/1029547032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db8110030 msgr2=0x7f1db8110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.973+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/1029547032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db8110030 0x7f1db8110440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1db0009ca0 tx=0x7f1db002a8e0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.974+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/1029547032 shutdown_connections 2024-04-25T13:51:15.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.974+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/1029547032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db8110030 0x7f1db8110440 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.974+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/1029547032 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db810b310 0x7f1db810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.974+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/1029547032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db810a420 0x7f1db810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.974+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/1029547032 >> 172.21.15.27:0/1029547032 conn(0x7f1db80704e0 msgr2=0x7f1db8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.975+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/1029547032 shutdown_connections 2024-04-25T13:51:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.975+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/1029547032 wait complete. 2024-04-25T13:51:15.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.976+0000 7f1dbd4b1640 1 Processor -- start 2024-04-25T13:51:15.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.976+0000 7f1dbd4b1640 1 -- start start 2024-04-25T13:51:15.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.976+0000 7f1dbd4b1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db810a420 0x7f1db811b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.976+0000 7f1dbd4b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db810b310 0x7f1db811a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.976+0000 7f1dbd4b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db8110030 0x7f1db811a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.976+0000 7f1dbd4b1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1db81123c0 con 0x7f1db810a420 2024-04-25T13:51:15.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.976+0000 7f1dbd4b1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1db8112240 con 0x7f1db8110030 2024-04-25T13:51:15.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.976+0000 7f1dbd4b1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1db8112540 con 0x7f1db810b310 2024-04-25T13:51:15.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db810b310 0x7f1db811a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db810b310 0x7f1db811a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37242/0 (socket says 172.21.15.27:37242) 2024-04-25T13:51:15.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db810a420 0x7f1db811b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db67fc640 1 -- 172.21.15.27:0/3259110608 learned_addr learned my addr 172.21.15.27:0/3259110608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db77fe640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db8110030 0x7f1db811a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db67fc640 1 -- 172.21.15.27:0/3259110608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db8110030 msgr2=0x7f1db811a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db67fc640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db8110030 0x7f1db811a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db67fc640 1 -- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db810a420 msgr2=0x7f1db811b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db67fc640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db810a420 0x7f1db811b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.977+0000 7f1db67fc640 1 -- 172.21.15.27:0/3259110608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db811aa80 con 0x7f1db810b310 2024-04-25T13:51:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.978+0000 7f1db67fc640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db810b310 0x7f1db811a000 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1da800b450 tx=0x7f1da800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.978+0000 7f1d9ffff640 1 -- 172.21.15.27:0/3259110608 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da8015020 con 0x7f1db810b310 2024-04-25T13:51:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.978+0000 7f1d9ffff640 1 -- 172.21.15.27:0/3259110608 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da8011d40 con 0x7f1db810b310 2024-04-25T13:51:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.978+0000 7f1d9ffff640 1 -- 172.21.15.27:0/3259110608 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da800f340 con 0x7f1db810b310 2024-04-25T13:51:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.978+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/3259110608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db811ad70 con 0x7f1db810b310 2024-04-25T13:51:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.978+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/3259110608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db81b9c60 con 0x7f1db810b310 2024-04-25T13:51:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.982+0000 7f1dc443a740 1 Processor -- start 2024-04-25T13:51:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.982+0000 7f1dc443a740 1 -- start start 2024-04-25T13:51:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x56368919b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x56368919bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688dd86b0 0x56368919ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1dc443a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5636890ffd80 con 0x563688df9390 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1dc443a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5636890b9a20 con 0x563688dd04d0 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1dc443a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5636890fbd40 con 0x563688dd86b0 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1d9ffff640 1 -- 172.21.15.27:0/3259110608 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1da8018020 con 0x7f1db810b310 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x56368919b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x56368919b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57068/0 (socket says 172.21.15.27:57068) 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db77fe640 1 -- 172.21.15.27:0/3866364113 learned_addr learned my addr 172.21.15.27:0/3866364113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db67fc640 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688dd86b0 0x56368919ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x56368919bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db6ffd640 1 -- 172.21.15.27:0/3866364113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688dd86b0 msgr2=0x56368919ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688dd86b0 0x56368919ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db6ffd640 1 -- 172.21.15.27:0/3866364113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 msgr2=0x56368919b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x56368919b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db6ffd640 1 -- 172.21.15.27:0/3866364113 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636890c4c30 con 0x563688dd04d0 2024-04-25T13:51:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.983+0000 7f1db77fe640 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x56368919b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.984+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x56368919bd90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00cb70 tx=0x7f1dac00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e337ffcad6184b3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.984+0000 7f1d9ffff640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1da00afd00 0x7f1da00b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.984+0000 7f1d9d7fa640 1 -- 172.21.15.27:0/3866364113 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac004220 con 0x563688dd04d0 2024-04-25T13:51:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.984+0000 7f1d9d7fa640 1 -- 172.21.15.27:0/3866364113 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dac0043c0 con 0x563688dd04d0 2024-04-25T13:51:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.984+0000 7f1d9d7fa640 1 -- 172.21.15.27:0/3866364113 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac004d00 con 0x563688dd04d0 2024-04-25T13:51:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.984+0000 7f1d9ffff640 1 -- 172.21.15.27:0/3259110608 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1da80de580 con 0x7f1db810b310 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.984+0000 7f1db77fe640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1da00afd00 0x7f1da00b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1db77fe640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1da00afd00 0x7f1da00b2210 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x56368919c860 tx=0x7f1db0031470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 -- 172.21.15.27:0/3866364113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 msgr2=0x56368919bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x56368919bd90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00cb70 tx=0x7f1dac00e720 comp rx=0 tx=0).stop 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 -- 172.21.15.27:0/3866364113 shutdown_connections 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688dd86b0 0x56368919ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x56368919bd90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 --2- 172.21.15.27:0/3866364113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x56368919b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 -- 172.21.15.27:0/3866364113 >> 172.21.15.27:0/3866364113 conn(0x563688ded4e0 msgr2=0x563689197020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 -- 172.21.15.27:0/3866364113 shutdown_connections 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.985+0000 7f1dc443a740 1 -- 172.21.15.27:0/3866364113 wait complete. 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 1 -- start start 2024-04-25T13:51:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 1 Processor -- start 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd86b0 0x5636891a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688df9390 0x5636891aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1db6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56368919d530 con 0x563688dd04d0 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563689086dc0 con 0x563688dd86b0 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56368919d6b0 con 0x563688df9390 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1db6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57082/0 (socket says 172.21.15.27:57082) 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1db67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd86b0 0x5636891a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1db6ffd640 1 -- 172.21.15.27:0/2131590177 learned_addr learned my addr 172.21.15.27:0/2131590177 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.986+0000 7f1dc443a740 10 client.0 _ll_register_callbacks cb 0x5636890a2600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1db77fe640 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688df9390 0x5636891aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1db67fc640 1 -- 172.21.15.27:0/2131590177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688df9390 msgr2=0x5636891aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1db67fc640 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688df9390 0x5636891aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1db67fc640 1 -- 172.21.15.27:0/2131590177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 msgr2=0x5636891a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1db67fc640 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1db67fc640 1 -- 172.21.15.27:0/2131590177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563688e113f0 con 0x563688dd86b0 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1db67fc640 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd86b0 0x5636891a5b30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1da80e1fd0 tx=0x7f1da8010770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2051420eec82cde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1d9aff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1d9aff5640 1 -- 172.21.15.27:0/2131590177 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da80a6380 con 0x563688dd86b0 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1d9aff5640 1 -- 172.21.15.27:0/2131590177 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1da80de2e0 con 0x563688dd86b0 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1dc443a740 1 -- 172.21.15.27:0/2131590177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dac0043c0 con 0x563688dd86b0 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.987+0000 7f1d9aff5640 1 -- 172.21.15.27:0/2131590177 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1da80a61a0 con 0x563688dd86b0 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1dc443a740 10 client.27149 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1dc443a740 1 -- 172.21.15.27:0/2131590177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563688edb7d0 con 0x563688dd86b0 2024-04-25T13:51:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1d9aff5640 1 -- 172.21.15.27:0/2131590177 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dac0043c0 con 0x563688dd86b0 2024-04-25T13:51:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1d9aff5640 1 -- 172.21.15.27:0/2131590177 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1da80c7ce0 con 0x563688dd86b0 2024-04-25T13:51:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1dc443a740 10 client.27149 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1dc443a740 10 client.27149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1dc443a740 20 client.27149 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563688db7f60 0x5636891b3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1dc443a740 4 client.27149 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.988+0000 7f1dc443a740 1 -- 172.21.15.27:0/2131590177 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56368910d990 con 0x563688db7f60 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.989+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563688db7f60 0x5636891b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.990+0000 7f1d9aff5640 10 client.27149 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.990+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563688db7f60 0x5636891b3230 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43b7a40f0c0a6565 server_cookie=3c1efc4fbc12a018 in_seq=0 out_seq=0 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.990+0000 7f1d9aff5640 1 -- 172.21.15.27:0/2131590177 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56368910d990 con 0x563688db7f60 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.990+0000 7f1d9aff5640 10 client.27149 handle_command_reply: tid=0 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 client.27149 shutdown 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 -- 172.21.15.27:0/2131590177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd86b0 msgr2=0x5636891a5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd86b0 0x5636891a5b30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1da80e1fd0 tx=0x7f1da8010770 comp rx=0 tx=0).stop 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 -- 172.21.15.27:0/2131590177 shutdown_connections 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563688db7f60 0x5636891b3230 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688df9390 0x5636891aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd86b0 0x5636891a5b30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2131590177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1dc443a740 1 -- 172.21.15.27:0/2131590177 >> 172.21.15.27:0/2131590177 conn(0x563688ded4e0 msgr2=0x563689199f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.991+0000 7f1d9aff5640 0 client.27149 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.992+0000 7f1dc443a740 1 -- 172.21.15.27:0/2131590177 shutdown_connections 2024-04-25T13:51:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.992+0000 7f1dc443a740 1 -- 172.21.15.27:0/2131590177 wait complete. 2024-04-25T13:51:15.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:15.992+0000 7f1dc443a740 20 client.27149 trim_cache size 0 max 16384 2024-04-25T13:51:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.012+0000 7f1dc443a740 1 Processor -- start 2024-04-25T13:51:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.012+0000 7f1dc443a740 1 -- start start 2024-04-25T13:51:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.012+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x563689196a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x563689196f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x563689199e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1dc443a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1db81c0770 con 0x563688df9390 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1db6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x563689196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1dc443a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1db81c2ff0 con 0x563688dd04d0 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1dc443a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1db81c0de0 con 0x563688d8d3a0 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1db77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x563689199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1db77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x563689199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37264/0 (socket says 172.21.15.27:37264) 2024-04-25T13:51:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.013+0000 7f1db77fe640 1 -- 172.21.15.27:0/2031994067 learned_addr learned my addr 172.21.15.27:0/2031994067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.014+0000 7f1db6ffd640 1 -- 172.21.15.27:0/2031994067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 msgr2=0x563689199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.014+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/2031994067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x563689199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.014+0000 7f1db6ffd640 1 -- 172.21.15.27:0/2031994067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 msgr2=0x563689196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.014+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/2031994067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x563689196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.014+0000 7f1db6ffd640 1 -- 172.21.15.27:0/2031994067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563688dd88e0 con 0x563688dd04d0 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.014+0000 7f1db77fe640 1 --2- 172.21.15.27:0/2031994067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x563689199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.015+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/2031994067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x563689196a50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0057c0 tx=0x7f1dac00c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c26efe9f00872f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.015+0000 7f1d9cff9640 1 -- 172.21.15.27:0/2031994067 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac02a110 con 0x563688dd04d0 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.015+0000 7f1d9cff9640 1 -- 172.21.15.27:0/2031994067 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dac02a760 con 0x563688dd04d0 2024-04-25T13:51:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.015+0000 7f1d9cff9640 1 -- 172.21.15.27:0/2031994067 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac01a9a0 con 0x563688dd04d0 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 -- 172.21.15.27:0/2031994067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 msgr2=0x563689196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2031994067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x563689196a50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0057c0 tx=0x7f1dac00c4b0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 -- 172.21.15.27:0/2031994067 shutdown_connections 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2031994067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x563689199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2031994067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688df9390 0x563689196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 --2- 172.21.15.27:0/2031994067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688dd04d0 0x563689196a50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 -- 172.21.15.27:0/2031994067 >> 172.21.15.27:0/2031994067 conn(0x563688ded4e0 msgr2=0x56368919b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 -- 172.21.15.27:0/2031994067 shutdown_connections 2024-04-25T13:51:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 -- 172.21.15.27:0/2031994067 wait complete. 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 -- start start 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.016+0000 7f1dc443a740 1 Processor -- start 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x5636891a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1dc443a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688df9390 0x563689219560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1dc443a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1db81c14b0 con 0x563688dd04d0 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1dc443a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1db81c2920 con 0x563688df9390 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1dc443a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1db81c1b80 con 0x563688d8d3a0 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1dc443a740 10 client.0 _ll_register_callbacks cb 0x7f1da80046e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:16.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57102/0 (socket says 172.21.15.27:57102) 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 -- 172.21.15.27:0/1965391162 learned_addr learned my addr 172.21.15.27:0/1965391162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x5636891a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db77fe640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688df9390 0x563689219560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 -- 172.21.15.27:0/1965391162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 msgr2=0x5636891a90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x5636891a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 -- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688df9390 msgr2=0x563689219560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688df9390 0x563689219560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 -- 172.21.15.27:0/1965391162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563689219ad0 con 0x563688dd04d0 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db77fe640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688df9390 0x563689219560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x5636891a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1db67fc640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a9600 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1da8009fd0 tx=0x7f1da80af8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798150a3cba70049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1d9aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:16.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.017+0000 7f1d9aff5640 1 -- 172.21.15.27:0/1965391162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da80a6380 con 0x563688dd04d0 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 1 -- 172.21.15.27:0/1965391162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dac02a760 con 0x563688dd04d0 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1d9aff5640 1 -- 172.21.15.27:0/1965391162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1da8011aa0 con 0x563688dd04d0 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1d9aff5640 1 -- 172.21.15.27:0/1965391162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1da80af1d0 con 0x563688dd04d0 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 10 client.17448 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 1 -- 172.21.15.27:0/1965391162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56368921add0 con 0x563688dd04d0 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1d9aff5640 1 -- 172.21.15.27:0/1965391162 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dac02a760 con 0x563688dd04d0 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1d9aff5640 1 -- 172.21.15.27:0/1965391162 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1da80c7ce0 con 0x563688dd04d0 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 10 client.17448 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 10 client.17448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 20 client.17448 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56368921b5f0 0x56368921da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 4 client.17448 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.018+0000 7f1dc443a740 1 -- 172.21.15.27:0/1965391162 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563689107800 con 0x56368921b5f0 2024-04-25T13:51:16.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.019+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56368921b5f0 0x56368921da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:16.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.019+0000 7f1d9aff5640 10 client.17448 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.020+0000 7f1db6ffd640 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56368921b5f0 0x56368921da10 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4a9d8ee231d454d server_cookie=b31ecf5d0b074cce in_seq=0 out_seq=0 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.020+0000 7f1d9aff5640 1 -- 172.21.15.27:0/1965391162 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563689107800 con 0x56368921b5f0 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1d9aff5640 10 client.17448 handle_command_reply: tid=0 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 client.17448 shutdown 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 -- 172.21.15.27:0/1965391162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 msgr2=0x5636891a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a9600 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1da8009fd0 tx=0x7f1da80af8e0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 -- 172.21.15.27:0/1965391162 shutdown_connections 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56368921b5f0 0x56368921da10 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563688df9390 0x563689219560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563688dd04d0 0x5636891a9600 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 --2- 172.21.15.27:0/1965391162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563688d8d3a0 0x5636891a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 -- 172.21.15.27:0/1965391162 >> 172.21.15.27:0/1965391162 conn(0x563688ded4e0 msgr2=0x563689196950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:16.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1d9aff5640 0 client.17448 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 -- 172.21.15.27:0/1965391162 shutdown_connections 2024-04-25T13:51:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 1 -- 172.21.15.27:0/1965391162 wait complete. 2024-04-25T13:51:16.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.021+0000 7f1dc443a740 20 client.17448 trim_cache size 0 max 16384 2024-04-25T13:51:16.025 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1da00afd00 msgr2=0x7f1da00b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1da00afd00 0x7f1da00b2210 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x56368919c860 tx=0x7f1db0031470 comp rx=0 tx=0).stop 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/3259110608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db810b310 msgr2=0x7f1db811a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db810b310 0x7f1db811a000 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1da800b450 tx=0x7f1da800b910 comp rx=0 tx=0).stop 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/3259110608 shutdown_connections 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1da00afd00 0x7f1da00b2210 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1db8110030 0x7f1db811a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1db810b310 0x7f1db811a000 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 --2- 172.21.15.27:0/3259110608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1db810a420 0x7f1db811b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/3259110608 >> 172.21.15.27:0/3259110608 conn(0x7f1db80704e0 msgr2=0x7f1db810e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/3259110608 shutdown_connections 2024-04-25T13:51:16.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:16.022+0000 7f1dbd4b1640 1 -- 172.21.15.27:0/3259110608 wait complete. 2024-04-25T13:51:16.036 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:15 smithi027 ceph-mon[29140]: pgmap v660: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 614 op/s 2024-04-25T13:51:16.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:15 smithi122 ceph-mon[32843]: pgmap v660: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 614 op/s 2024-04-25T13:51:16.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:15 smithi045 ceph-mon[32731]: pgmap v660: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 614 op/s 2024-04-25T13:51:16.979 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:16 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/101160835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:51:17.038 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:51:17.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:51:17.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:51:17.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:16 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/101160835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:51:17.080 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:16 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/101160835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:51:17.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.242+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3810596135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501029e0 msgr2=0x7fcc501036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.242+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3810596135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501029e0 0x7fcc501036e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00d910 tx=0x7fcc4c02c3b0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.244+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3810596135 shutdown_connections 2024-04-25T13:51:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.244+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3810596135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc50108920 0x7fcc5010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.244+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3810596135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc50103c20 0x7fcc50104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.244+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3810596135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc501029e0 0x7fcc501036e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.244+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3810596135 >> 172.21.15.27:0/3810596135 conn(0x7fcc500fdbd0 msgr2=0x7fcc50100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.246+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3810596135 shutdown_connections 2024-04-25T13:51:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.246+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3810596135 wait complete. 2024-04-25T13:51:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc57a0c640 1 Processor -- start 2024-04-25T13:51:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc57a0c640 1 -- start start 2024-04-25T13:51:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc57a0c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc501029e0 0x7fcc50198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc57a0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc50103c20 0x7fcc50198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc57a0c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc50108920 0x7fcc5019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc57a0c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5010d440 con 0x7fcc50108920 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc57a0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5010d2c0 con 0x7fcc50103c20 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc57a0c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5010d5c0 con 0x7fcc501029e0 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc54fa7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc50103c20 0x7fcc50198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.247+0000 7fcc557a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc501029e0 0x7fcc50198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc557a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc501029e0 0x7fcc50198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37290/0 (socket says 172.21.15.27:37290) 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc54fa7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc50103c20 0x7fcc50198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45590/0 (socket says 172.21.15.27:45590) 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc557a8640 1 -- 172.21.15.27:0/3749358001 learned_addr learned my addr 172.21.15.27:0/3749358001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc55fa9640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc50108920 0x7fcc5019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc557a8640 1 -- 172.21.15.27:0/3749358001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc50103c20 msgr2=0x7fcc50198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc557a8640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc50103c20 0x7fcc50198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc557a8640 1 -- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc50108920 msgr2=0x7fcc5019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc557a8640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc50108920 0x7fcc5019f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc557a8640 1 -- 172.21.15.27:0/3749358001 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5019fe90 con 0x7fcc501029e0 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc50103c20 0x7fcc50198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.248+0000 7fcc557a8640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc501029e0 0x7fcc50198350 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00d540 tx=0x7fcc4c0349c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.249+0000 7fcc467fc640 1 -- 172.21.15.27:0/3749358001 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c034d70 con 0x7fcc501029e0 2024-04-25T13:51:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.249+0000 7fcc467fc640 1 -- 172.21.15.27:0/3749358001 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc4c041070 con 0x7fcc501029e0 2024-04-25T13:51:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.249+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3749358001 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc501a0120 con 0x7fcc501029e0 2024-04-25T13:51:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.250+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3749358001 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc501a05a0 con 0x7fcc501029e0 2024-04-25T13:51:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.250+0000 7fcc467fc640 1 -- 172.21.15.27:0/3749358001 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c033620 con 0x7fcc501029e0 2024-04-25T13:51:17.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.253+0000 7fcc467fc640 1 -- 172.21.15.27:0/3749358001 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcc4c00c050 con 0x7fcc501029e0 2024-04-25T13:51:17.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc467fc640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcc3c0b4040 0x7fcc3c0b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc5e997740 1 Processor -- start 2024-04-25T13:51:17.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc5e997740 1 -- start start 2024-04-25T13:51:17.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def8b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def8ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def92920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc5e997740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5589deef0b00 con 0x5589debc9730 2024-04-25T13:51:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc5e997740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5589dee8c920 con 0x5589debea410 2024-04-25T13:51:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc5e997740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5589deeecac0 con 0x5589debc1550 2024-04-25T13:51:17.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.254+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcc3c0b4040 0x7fcc3c0b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc55fa9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def92920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc55fa9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def92920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57136/0 (socket says 172.21.15.27:57136) 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc55fa9640 1 -- 172.21.15.27:0/168157259 learned_addr learned my addr 172.21.15.27:0/168157259 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc557a8640 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def8b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def8ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc557a8640 1 -- 172.21.15.27:0/168157259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 msgr2=0x5589def8ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc557a8640 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def8ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc557a8640 1 -- 172.21.15.27:0/168157259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 msgr2=0x5589def92920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc557a8640 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def92920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc557a8640 1 -- 172.21.15.27:0/168157259 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589decca7e0 con 0x5589debea410 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc55fa9640 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def92920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc467fc640 1 -- 172.21.15.27:0/3749358001 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcc4c0feb50 con 0x7fcc501029e0 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc557a8640 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def8b4a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0c6f90 tx=0x7fcc4c03cd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7cb6df61a166b02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcc3c0b4040 0x7fcc3c0b6550 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcc48005c40 tx=0x7fcc48008d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.255+0000 7fcc3b7fe640 1 -- 172.21.15.27:0/168157259 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0c2ec0 con 0x5589debea410 2024-04-25T13:51:17.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.256+0000 7fcc3b7fe640 1 -- 172.21.15.27:0/168157259 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc4c0c6b10 con 0x5589debea410 2024-04-25T13:51:17.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.256+0000 7fcc3b7fe640 1 -- 172.21.15.27:0/168157259 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c08d980 con 0x5589debea410 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.256+0000 7fcc5e997740 1 -- 172.21.15.27:0/168157259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 msgr2=0x5589def8b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.256+0000 7fcc5e997740 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def8b4a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0c6f90 tx=0x7fcc4c03cd90 comp rx=0 tx=0).stop 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 -- 172.21.15.27:0/168157259 shutdown_connections 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def92920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def8ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 --2- 172.21.15.27:0/168157259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def8b4a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 -- 172.21.15.27:0/168157259 >> 172.21.15.27:0/168157259 conn(0x5589debde560 msgr2=0x5589def86e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 -- 172.21.15.27:0/168157259 shutdown_connections 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 -- 172.21.15.27:0/168157259 wait complete. 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 -- start start 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.257+0000 7fcc5e997740 1 Processor -- start 2024-04-25T13:51:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def94f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def954a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def9c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc5e997740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5589def93130 con 0x5589debc9730 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc5e997740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5589dee8a5d0 con 0x5589debea410 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc5e997740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5589def932b0 con 0x5589debc1550 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc5e997740 10 client.0 _ll_register_callbacks cb 0x5589dee78080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc557a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def94f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc557a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def94f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37316/0 (socket says 172.21.15.27:37316) 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc557a8640 1 -- 172.21.15.27:0/2288532000 learned_addr learned my addr 172.21.15.27:0/2288532000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc55fa9640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def9c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc557a8640 1 -- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 msgr2=0x5589def9c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.258+0000 7fcc557a8640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def9c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc557a8640 1 -- 172.21.15.27:0/2288532000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 msgr2=0x5589def954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc557a8640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def954a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc557a8640 1 -- 172.21.15.27:0/2288532000 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ded2af70 con 0x5589debc1550 2024-04-25T13:51:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc55fa9640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def9c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc557a8640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def94f60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0092a0 tx=0x7fcc4c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f09e8f90b15661d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc38ff9640 1 -- 172.21.15.27:0/2288532000 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0c2ec0 con 0x5589debc1550 2024-04-25T13:51:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.259+0000 7fcc38ff9640 1 -- 172.21.15.27:0/2288532000 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcc4c0c3290 con 0x5589debc1550 2024-04-25T13:51:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.260+0000 7fcc5e997740 1 -- 172.21.15.27:0/2288532000 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc4c0c6b10 con 0x5589debc1550 2024-04-25T13:51:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.261+0000 7fcc38ff9640 1 -- 172.21.15.27:0/2288532000 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcc4c0c7000 con 0x5589debc1550 2024-04-25T13:51:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.261+0000 7fcc5e997740 10 client.27211 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:17.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.261+0000 7fcc5e997740 1 -- 172.21.15.27:0/2288532000 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5589debc2170 con 0x5589debc1550 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.261+0000 7fcc38ff9640 1 -- 172.21.15.27:0/2288532000 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc4c0c6b10 con 0x5589debc1550 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.261+0000 7fcc38ff9640 1 -- 172.21.15.27:0/2288532000 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcc4c0cf9d0 con 0x5589debc1550 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.261+0000 7fcc5e997740 10 client.27211 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.261+0000 7fcc5e997740 10 client.27211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.262+0000 7fcc5e997740 20 client.27211 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.262+0000 7fcc5e997740 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5589deba91f0 0x5589defa2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.262+0000 7fcc5e997740 4 client.27211 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.262+0000 7fcc5e997740 1 -- 172.21.15.27:0/2288532000 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589deefe710 con 0x5589deba91f0 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.262+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5589deba91f0 0x5589defa2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.263+0000 7fcc38ff9640 10 client.27211 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.263+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5589deba91f0 0x5589defa2600 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18f360c0c0dd1609 server_cookie=ae2218aac87d623d in_seq=0 out_seq=0 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.263+0000 7fcc38ff9640 1 -- 172.21.15.27:0/2288532000 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5589deefe710 con 0x5589deba91f0 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.263+0000 7fcc38ff9640 10 client.27211 handle_command_reply: tid=0 2024-04-25T13:51:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.263+0000 7fcc5e997740 1 client.27211 shutdown 2024-04-25T13:51:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc5e997740 1 -- 172.21.15.27:0/2288532000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 msgr2=0x5589def94f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc5e997740 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def94f60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0092a0 tx=0x7fcc4c000bc0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc5e997740 1 -- 172.21.15.27:0/2288532000 shutdown_connections 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc5e997740 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5589deba91f0 0x5589defa2600 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc5e997740 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debea410 0x5589def9c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc38ff9640 0 client.27211 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc5e997740 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc9730 0x5589def954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc5e997740 1 --2- 172.21.15.27:0/2288532000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc1550 0x5589def94f60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.264+0000 7fcc5e997740 1 -- 172.21.15.27:0/2288532000 >> 172.21.15.27:0/2288532000 conn(0x5589debde560 msgr2=0x5589def85ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.265+0000 7fcc5e997740 1 -- 172.21.15.27:0/2288532000 shutdown_connections 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.265+0000 7fcc5e997740 1 -- 172.21.15.27:0/2288532000 wait complete. 2024-04-25T13:51:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.265+0000 7fcc5e997740 20 client.27211 trim_cache size 0 max 16384 2024-04-25T13:51:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.298+0000 7fcc5e997740 1 Processor -- start 2024-04-25T13:51:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.300+0000 7fcc5e997740 1 -- start start 2024-04-25T13:51:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589deba91f0 0x5589def93c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc1550 0x5589defa6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc9730 0x5589defa72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc5e997740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcc501a6f20 con 0x5589deba91f0 2024-04-25T13:51:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc5e997740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcc501a97a0 con 0x5589debc1550 2024-04-25T13:51:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc5e997740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcc501a7590 con 0x5589debc9730 2024-04-25T13:51:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc54fa7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc1550 0x5589defa6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc54fa7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc1550 0x5589defa6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45622/0 (socket says 172.21.15.27:45622) 2024-04-25T13:51:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc54fa7640 1 -- 172.21.15.27:0/1364015625 learned_addr learned my addr 172.21.15.27:0/1364015625 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc557a8640 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589deba91f0 0x5589def93c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc55fa9640 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc9730 0x5589defa72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc54fa7640 1 -- 172.21.15.27:0/1364015625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc9730 msgr2=0x5589defa72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc9730 0x5589defa72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc54fa7640 1 -- 172.21.15.27:0/1364015625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589deba91f0 msgr2=0x5589def93c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589deba91f0 0x5589def93c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.301+0000 7fcc54fa7640 1 -- 172.21.15.27:0/1364015625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589defa2b30 con 0x5589debc1550 2024-04-25T13:51:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.302+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc1550 0x5589defa6d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x5589def8c880 tx=0x7fcc48012f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b287f15f2bd226c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.302+0000 7fcc3affd640 1 -- 172.21.15.27:0/1364015625 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4802e070 con 0x5589debc1550 2024-04-25T13:51:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.302+0000 7fcc3affd640 1 -- 172.21.15.27:0/1364015625 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc48028780 con 0x5589debc1550 2024-04-25T13:51:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.302+0000 7fcc3affd640 1 -- 172.21.15.27:0/1364015625 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4801fb00 con 0x5589debc1550 2024-04-25T13:51:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 -- 172.21.15.27:0/1364015625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc1550 msgr2=0x5589defa6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc1550 0x5589defa6d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x5589def8c880 tx=0x7fcc48012f90 comp rx=0 tx=0).stop 2024-04-25T13:51:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 -- 172.21.15.27:0/1364015625 shutdown_connections 2024-04-25T13:51:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589debc9730 0x5589defa72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc1550 0x5589defa6d80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1364015625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589deba91f0 0x5589def93c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 -- 172.21.15.27:0/1364015625 >> 172.21.15.27:0/1364015625 conn(0x5589debde560 msgr2=0x5589def93530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 -- 172.21.15.27:0/1364015625 shutdown_connections 2024-04-25T13:51:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.304+0000 7fcc5e997740 1 -- 172.21.15.27:0/1364015625 wait complete. 2024-04-25T13:51:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.305+0000 7fcc5e997740 1 -- start start 2024-04-25T13:51:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.305+0000 7fcc5e997740 1 Processor -- start 2024-04-25T13:51:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.306+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589deba91f0 0x5589def99190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.306+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc1550 0x5589def996d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc5e997740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc9730 0x5589def9a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc5e997740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcc501a7c60 con 0x5589debc1550 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc5e997740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcc501a90d0 con 0x5589debc9730 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc5e997740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcc501a8330 con 0x5589deba91f0 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc55fa9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc9730 0x5589def9a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc55fa9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc9730 0x5589def9a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45634/0 (socket says 172.21.15.27:45634) 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc55fa9640 1 -- 172.21.15.27:0/1160432854 learned_addr learned my addr 172.21.15.27:0/1160432854 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc5e997740 10 client.0 _ll_register_callbacks cb 0x7fcc4c0d4c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc1550 0x5589def996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc55fa9640 1 -- 172.21.15.27:0/1160432854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589deba91f0 msgr2=0x5589def99190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc55fa9640 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589deba91f0 0x5589def99190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc55fa9640 1 -- 172.21.15.27:0/1160432854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc1550 msgr2=0x5589def996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.307+0000 7fcc55fa9640 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc1550 0x5589def996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.308+0000 7fcc55fa9640 1 -- 172.21.15.27:0/1160432854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589def90160 con 0x5589debc9730 2024-04-25T13:51:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.308+0000 7fcc54fa7640 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc1550 0x5589def996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:51:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.308+0000 7fcc55fa9640 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc9730 0x5589def9a5b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x5589def87bc0 tx=0x7fcc40010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c49b3fd046a3574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.308+0000 7fcc38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:51:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.308+0000 7fcc38ff9640 1 -- 172.21.15.27:0/1160432854 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40010e10 con 0x5589debc9730 2024-04-25T13:51:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.308+0000 7fcc38ff9640 1 -- 172.21.15.27:0/1160432854 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcc4000f5d0 con 0x5589debc9730 2024-04-25T13:51:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.309+0000 7fcc5e997740 1 -- 172.21.15.27:0/1160432854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc48028780 con 0x5589debc9730 2024-04-25T13:51:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.309+0000 7fcc38ff9640 1 -- 172.21.15.27:0/1160432854 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcc4000f810 con 0x5589debc9730 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.309+0000 7fcc5e997740 10 client.27167 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.309+0000 7fcc5e997740 1 -- 172.21.15.27:0/1160432854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5589def9bd40 con 0x5589debc9730 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc38ff9640 1 -- 172.21.15.27:0/1160432854 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc4000fe60 con 0x5589debc9730 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc38ff9640 1 -- 172.21.15.27:0/1160432854 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fcc4002c030 con 0x5589debc9730 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc5e997740 10 client.27167 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc5e997740 10 client.27167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc5e997740 20 client.27167 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5589debea410 0x5589df00d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc5e997740 4 client.27167 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc5e997740 1 -- 172.21.15.27:0/1160432854 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fcc48028780 con 0x5589debea410 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.310+0000 7fcc557a8640 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5589debea410 0x5589df00d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.311+0000 7fcc38ff9640 10 client.27167 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.311+0000 7fcc557a8640 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5589debea410 0x5589df00d0d0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4cabbc4a0b32ea6 server_cookie=4c28f2bcf80e365a in_seq=0 out_seq=0 2024-04-25T13:51:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.312+0000 7fcc38ff9640 1 -- 172.21.15.27:0/1160432854 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fcc48028780 con 0x5589debea410 2024-04-25T13:51:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.312+0000 7fcc38ff9640 10 client.27167 handle_command_reply: tid=0 2024-04-25T13:51:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.312+0000 7fcc5e997740 1 client.27167 shutdown 2024-04-25T13:51:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc5e997740 1 -- 172.21.15.27:0/1160432854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc9730 msgr2=0x5589def9a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc9730 0x5589def9a5b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x5589def87bc0 tx=0x7fcc40010800 comp rx=0 tx=0).stop 2024-04-25T13:51:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc5e997740 1 -- 172.21.15.27:0/1160432854 shutdown_connections 2024-04-25T13:51:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5589debea410 0x5589df00d0d0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5589debc9730 0x5589def9a5b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc38ff9640 0 client.27167 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5589debc1550 0x5589def996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc5e997740 1 --2- 172.21.15.27:0/1160432854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5589deba91f0 0x5589def99190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.313+0000 7fcc5e997740 1 -- 172.21.15.27:0/1160432854 >> 172.21.15.27:0/1160432854 conn(0x5589debde560 msgr2=0x5589def89ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.314+0000 7fcc5e997740 1 -- 172.21.15.27:0/1160432854 shutdown_connections 2024-04-25T13:51:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.314+0000 7fcc5e997740 1 -- 172.21.15.27:0/1160432854 wait complete. 2024-04-25T13:51:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.314+0000 7fcc5e997740 20 client.27167 trim_cache size 0 max 16384 2024-04-25T13:51:17.315 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:51:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:51:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "3b495791-be56-47ad-b864-7acea1e3e7c3", 2024-04-25T13:51:17.315 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:51:17.315 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcc3c0b4040 msgr2=0x7fcc3c0b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcc3c0b4040 0x7fcc3c0b6550 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcc48005c40 tx=0x7fcc48008d50 comp rx=0 tx=0).stop 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3749358001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc501029e0 msgr2=0x7fcc50198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc501029e0 0x7fcc50198350 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00d540 tx=0x7fcc4c0349c0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3749358001 shutdown_connections 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcc3c0b4040 0x7fcc3c0b6550 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc50108920 0x7fcc5019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc50103c20 0x7fcc50198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 --2- 172.21.15.27:0/3749358001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc501029e0 0x7fcc50198350 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3749358001 >> 172.21.15.27:0/3749358001 conn(0x7fcc500fdbd0 msgr2=0x7fcc50104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.315+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3749358001 shutdown_connections 2024-04-25T13:51:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.316+0000 7fcc57a0c640 1 -- 172.21.15.27:0/3749358001 wait complete. 2024-04-25T13:51:17.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:51:17.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:51:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.416+0000 7fef08f26640 1 -- 172.21.15.27:0/3246758279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef04110090 msgr2=0x7fef041124b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.416+0000 7fef08f26640 1 --2- 172.21.15.27:0/3246758279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef04110090 0x7fef041124b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7feefc009860 tx=0x7feefc02a340 comp rx=0 tx=0).stop 2024-04-25T13:51:17.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.431+0000 7fef08f26640 1 -- 172.21.15.27:0/3246758279 shutdown_connections 2024-04-25T13:51:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.431+0000 7fef08f26640 1 --2- 172.21.15.27:0/3246758279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef04110090 0x7fef041124b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.431+0000 7fef08f26640 1 --2- 172.21.15.27:0/3246758279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef0410b390 0x7fef0410b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.431+0000 7fef08f26640 1 --2- 172.21.15.27:0/3246758279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0410a150 0x7fef0410ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.431+0000 7fef08f26640 1 -- 172.21.15.27:0/3246758279 >> 172.21.15.27:0/3246758279 conn(0x7fef04078de0 msgr2=0x7fef041079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.443+0000 7fef08f26640 1 -- 172.21.15.27:0/3246758279 shutdown_connections 2024-04-25T13:51:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.447+0000 7fef08f26640 1 -- 172.21.15.27:0/3246758279 wait complete. 2024-04-25T13:51:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.448+0000 7fef08f26640 1 Processor -- start 2024-04-25T13:51:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.450+0000 7fef08f26640 1 -- start start 2024-04-25T13:51:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef08f26640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef0410a150 0x7fef04080870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef08f26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef0410b390 0x7fef04080db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef08f26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef04110090 0x7fef04081d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef08f26640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fef04115c70 con 0x7fef04110090 2024-04-25T13:51:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef08f26640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fef04115af0 con 0x7fef0410b390 2024-04-25T13:51:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef08f26640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fef04115df0 con 0x7fef0410a150 2024-04-25T13:51:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef04110090 0x7fef04081d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef04110090 0x7fef04081d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57172/0 (socket says 172.21.15.27:57172) 2024-04-25T13:51:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.451+0000 7fef02d9d640 1 -- 172.21.15.27:0/2633610846 learned_addr learned my addr 172.21.15.27:0/2633610846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.452+0000 7fef02d9d640 1 -- 172.21.15.27:0/2633610846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef0410a150 msgr2=0x7fef04080870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.452+0000 7fef02d9d640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef0410a150 0x7fef04080870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.452+0000 7fef02d9d640 1 -- 172.21.15.27:0/2633610846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef0410b390 msgr2=0x7fef04080db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:51:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.452+0000 7fef02d9d640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef0410b390 0x7fef04080db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.452+0000 7fef02d9d640 1 -- 172.21.15.27:0/2633610846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef04082380 con 0x7fef04110090 2024-04-25T13:51:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.453+0000 7fef02d9d640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef04110090 0x7fef04081d10 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feefc0303f0 tx=0x7feefc032df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.453+0000 7feef37fe640 1 -- 172.21.15.27:0/2633610846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc037040 con 0x7fef04110090 2024-04-25T13:51:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.453+0000 7feef37fe640 1 -- 172.21.15.27:0/2633610846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feefc041070 con 0x7fef04110090 2024-04-25T13:51:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.453+0000 7feef37fe640 1 -- 172.21.15.27:0/2633610846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc003850 con 0x7fef04110090 2024-04-25T13:51:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.454+0000 7fef08f26640 1 -- 172.21.15.27:0/2633610846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef04082610 con 0x7fef04110090 2024-04-25T13:51:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.454+0000 7fef08f26640 1 -- 172.21.15.27:0/2633610846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef04082a50 con 0x7fef04110090 2024-04-25T13:51:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.457+0000 7fef0fdd8740 1 Processor -- start 2024-04-25T13:51:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.458+0000 7feef37fe640 1 -- 172.21.15.27:0/2633610846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7feefc0039f0 con 0x7fef04110090 2024-04-25T13:51:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.458+0000 7fef0fdd8740 1 -- start start 2024-04-25T13:51:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.459+0000 7feef37fe640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feeec0afcf0 0x7feeec0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.460+0000 7feef37fe640 1 -- 172.21.15.27:0/2633610846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feefc0ff6d0 con 0x7fef04110090 2024-04-25T13:51:17.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.461+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 0x5580f4d94970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.461+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d94ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.461+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49d34d0 0x5580f4d9bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.461+0000 7fef0fdd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5580f4cfa460 con 0x5580f49d34d0 2024-04-25T13:51:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.461+0000 7fef0fdd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5580f4cb4590 con 0x5580f49cb2f0 2024-04-25T13:51:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.461+0000 7fef0fdd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5580f4cf6420 con 0x5580f49f41b0 2024-04-25T13:51:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.462+0000 7fef01d9b640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feeec0afcf0 0x7feeec0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.462+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d94ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.463+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d94ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45642/0 (socket says 172.21.15.27:45642) 2024-04-25T13:51:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.463+0000 7fef01d9b640 1 -- 172.21.15.27:0/4208866986 learned_addr learned my addr 172.21.15.27:0/4208866986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.463+0000 7fef01d9b640 1 -- 172.21.15.27:0/4208866986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 msgr2=0x5580f4d94970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:51:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.463+0000 7fef01d9b640 1 --2- 172.21.15.27:0/4208866986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 0x5580f4d94970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.463+0000 7fef01d9b640 1 -- 172.21.15.27:0/4208866986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49d34d0 msgr2=0x5580f4d9bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.463+0000 7fef01d9b640 1 --2- 172.21.15.27:0/4208866986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49d34d0 0x5580f4d9bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.463+0000 7fef01d9b640 1 -- 172.21.15.27:0/4208866986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580f4cc10d0 con 0x5580f49cb2f0 2024-04-25T13:51:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.464+0000 7fef01d9b640 1 --2- 172.21.15.27:0/4208866986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d94ef0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7feef4010de0 tx=0x7feef401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c80c1e0286df51e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.464+0000 7fef01d9b640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feeec0afcf0 0x7feeec0b2200 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7feef4009fd0 tx=0x7feef4009ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.466+0000 7feef0ff9640 1 -- 172.21.15.27:0/4208866986 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4021070 con 0x5580f49cb2f0 2024-04-25T13:51:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.466+0000 7feef0ff9640 1 -- 172.21.15.27:0/4208866986 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef400ebf0 con 0x5580f49cb2f0 2024-04-25T13:51:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.466+0000 7feef0ff9640 1 -- 172.21.15.27:0/4208866986 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef401ddb0 con 0x5580f49cb2f0 2024-04-25T13:51:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.467+0000 7fef0fdd8740 1 -- 172.21.15.27:0/4208866986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 msgr2=0x5580f4d94ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.467+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/4208866986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d94ef0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7feef4010de0 tx=0x7feef401a000 comp rx=0 tx=0).stop 2024-04-25T13:51:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.473+0000 7fef0fdd8740 1 -- 172.21.15.27:0/4208866986 shutdown_connections 2024-04-25T13:51:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.473+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/4208866986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49d34d0 0x5580f4d9bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.474+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/4208866986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d94ef0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.474+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/4208866986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 0x5580f4d94970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.474+0000 7fef0fdd8740 1 -- 172.21.15.27:0/4208866986 >> 172.21.15.27:0/4208866986 conn(0x5580f49e8300 msgr2=0x5580f4d90300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.474+0000 7fef0fdd8740 1 -- 172.21.15.27:0/4208866986 shutdown_connections 2024-04-25T13:51:17.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.478+0000 7fef0fdd8740 1 -- 172.21.15.27:0/4208866986 wait complete. 2024-04-25T13:51:17.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.478+0000 7fef0fdd8740 1 -- start start 2024-04-25T13:51:17.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.478+0000 7fef0fdd8740 1 Processor -- start 2024-04-25T13:51:17.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d9e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4d9ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4da5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef0fdd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5580f4d9c7e0 con 0x5580f49f41b0 2024-04-25T13:51:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef0fdd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5580f4c81b60 con 0x5580f49cb2f0 2024-04-25T13:51:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef0fdd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5580f4d9c960 con 0x5580f49d34d0 2024-04-25T13:51:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef0fdd8740 10 client.0 _ll_register_callbacks cb 0x5580f4ca62b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4da5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef02d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4da5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57196/0 (socket says 172.21.15.27:57196) 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.479+0000 7fef02d9d640 1 -- 172.21.15.27:0/1644440837 learned_addr learned my addr 172.21.15.27:0/1644440837 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.480+0000 7fef02d9d640 1 -- 172.21.15.27:0/1644440837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 msgr2=0x5580f4d9ec20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.480+0000 7fef02d9d640 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4d9ec20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.480+0000 7fef02d9d640 1 -- 172.21.15.27:0/1644440837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 msgr2=0x5580f4d9e6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.480+0000 7fef02d9d640 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d9e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.480+0000 7fef02d9d640 1 -- 172.21.15.27:0/1644440837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580f4a0c210 con 0x5580f49f41b0 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.480+0000 7fef02d9d640 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4da5b80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7feefc032e60 tx=0x7feefc0c7a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f14d8ff472774d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.480+0000 7feee9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.481+0000 7feee9ffb640 1 -- 172.21.15.27:0/1644440837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc0c7710 con 0x5580f49f41b0 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.481+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1644440837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef400ebf0 con 0x5580f49f41b0 2024-04-25T13:51:17.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.481+0000 7feee9ffb640 1 -- 172.21.15.27:0/1644440837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feefc0e2290 con 0x5580f49f41b0 2024-04-25T13:51:17.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.481+0000 7feee9ffb640 1 -- 172.21.15.27:0/1644440837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7feefc0ea050 con 0x5580f49f41b0 2024-04-25T13:51:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.483+0000 7fef0fdd8740 10 client.17472 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.483+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1644440837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5580f4ad8620 con 0x5580f49f41b0 2024-04-25T13:51:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.485+0000 7feee9ffb640 1 -- 172.21.15.27:0/1644440837 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef400ebf0 con 0x5580f49f41b0 2024-04-25T13:51:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.485+0000 7feee9ffb640 1 -- 172.21.15.27:0/1644440837 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7feefc0d05d0 con 0x5580f49f41b0 2024-04-25T13:51:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.487+0000 7fef0fdd8740 10 client.17472 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.487+0000 7fef0fdd8740 10 client.17472 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.487+0000 7fef0fdd8740 20 client.17472 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.487+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580f49b2d80 0x5580f4dac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.487+0000 7fef0fdd8740 4 client.17472 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.487+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1644440837 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580f4d08070 con 0x5580f49b2d80 2024-04-25T13:51:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.490+0000 7fef0259c640 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580f49b2d80 0x5580f4dac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.497+0000 7fef0259c640 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580f49b2d80 0x5580f4dac430 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c9201cd79210cdc server_cookie=b1e31754a097bf95 in_seq=0 out_seq=0 2024-04-25T13:51:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.498+0000 7feee9ffb640 10 client.17472 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.503+0000 7feee9ffb640 1 -- 172.21.15.27:0/1644440837 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5580f4d08070 con 0x5580f49b2d80 2024-04-25T13:51:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.503+0000 7feee9ffb640 10 client.17472 handle_command_reply: tid=0 2024-04-25T13:51:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.503+0000 7fef0fdd8740 1 client.17472 shutdown 2024-04-25T13:51:17.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.508+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1644440837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 msgr2=0x5580f4da5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.507 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.508+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4da5b80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7feefc032e60 tx=0x7feefc0c7a90 comp rx=0 tx=0).stop 2024-04-25T13:51:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.511+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1644440837 shutdown_connections 2024-04-25T13:51:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.511+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580f49b2d80 0x5580f4dac430 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.511+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4da5b80 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.511+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4d9ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.511+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1644440837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4d9e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.511+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1644440837 >> 172.21.15.27:0/1644440837 conn(0x5580f49e8300 msgr2=0x5580f4d93210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.513+0000 7feee9ffb640 0 client.17472 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.518 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.519+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1644440837 shutdown_connections 2024-04-25T13:51:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.523+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1644440837 wait complete. 2024-04-25T13:51:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.524+0000 7fef0fdd8740 20 client.17472 trim_cache size 0 max 16384 2024-04-25T13:51:17.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.558+0000 7fef0fdd8740 1 Processor -- start 2024-04-25T13:51:17.558 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.559+0000 7fef0fdd8740 1 -- start start 2024-04-25T13:51:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.559+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49cb2f0 0x5580f4d98b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49d34d0 0x5580f4d99070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 0x5580f4d9d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef0259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49cb2f0 0x5580f4d98b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef0fdd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feee400c3d0 con 0x5580f49cb2f0 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef0259c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49cb2f0 0x5580f4d98b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57212/0 (socket says 172.21.15.27:57212) 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef0259c640 1 -- 172.21.15.27:0/1612139152 learned_addr learned my addr 172.21.15.27:0/1612139152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1612139152 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feee400ec50 con 0x5580f49d34d0 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1612139152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feee400ca40 con 0x5580f49f41b0 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef02d9d640 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 0x5580f4d9d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef01d9b640 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49d34d0 0x5580f4d99070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef02d9d640 1 -- 172.21.15.27:0/1612139152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49d34d0 msgr2=0x5580f4d99070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef02d9d640 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49d34d0 0x5580f4d99070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef02d9d640 1 -- 172.21.15.27:0/1612139152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49cb2f0 msgr2=0x5580f4d98b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef02d9d640 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49cb2f0 0x5580f4d98b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.560+0000 7fef02d9d640 1 -- 172.21.15.27:0/1612139152 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580f4d9bef0 con 0x5580f49f41b0 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.561+0000 7fef0259c640 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49cb2f0 0x5580f4d98b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.561+0000 7fef02d9d640 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 0x5580f4d9d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7feefc0c7b90 tx=0x7feefc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e42828150022113d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.561+0000 7feeebfff640 1 -- 172.21.15.27:0/1612139152 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc0c7710 con 0x5580f49f41b0 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.561+0000 7feeebfff640 1 -- 172.21.15.27:0/1612139152 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feefc0c8300 con 0x5580f49f41b0 2024-04-25T13:51:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.561+0000 7feeebfff640 1 -- 172.21.15.27:0/1612139152 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc0d0740 con 0x5580f49f41b0 2024-04-25T13:51:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.563+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1612139152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 msgr2=0x5580f4d9d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.563+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 0x5580f4d9d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7feefc0c7b90 tx=0x7feefc005f90 comp rx=0 tx=0).stop 2024-04-25T13:51:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.563+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1612139152 shutdown_connections 2024-04-25T13:51:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.563+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49f41b0 0x5580f4d9d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.563+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49d34d0 0x5580f4d99070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.563+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/1612139152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49cb2f0 0x5580f4d98b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.563+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1612139152 >> 172.21.15.27:0/1612139152 conn(0x5580f49e8300 msgr2=0x5580f4d9cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.564+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1612139152 shutdown_connections 2024-04-25T13:51:17.564 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.564+0000 7fef0fdd8740 1 -- 172.21.15.27:0/1612139152 wait complete. 2024-04-25T13:51:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.565+0000 7fef0fdd8740 1 -- start start 2024-04-25T13:51:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.565+0000 7fef0fdd8740 1 Processor -- start 2024-04-25T13:51:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.566+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4da1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.566+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4da2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef0fdd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4e127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef0259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4da1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef0259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4da1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45670/0 (socket says 172.21.15.27:45670) 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4da2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4da2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37390/0 (socket says 172.21.15.27:37390) 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef0259c640 1 -- 172.21.15.27:0/3992122721 learned_addr learned my addr 172.21.15.27:0/3992122721 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feee400d110 con 0x5580f49f41b0 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feee400e580 con 0x5580f49cb2f0 2024-04-25T13:51:17.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.567+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feee400d7e0 con 0x5580f49d34d0 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef0fdd8740 10 client.0 _ll_register_callbacks cb 0x7feefc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef01d9b640 1 -- 172.21.15.27:0/3992122721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 msgr2=0x5580f4da1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef01d9b640 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4da1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef01d9b640 1 -- 172.21.15.27:0/3992122721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 msgr2=0x5580f4e127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef02d9d640 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4e127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef01d9b640 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4e127d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef01d9b640 1 -- 172.21.15.27:0/3992122721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580f4da5370 con 0x5580f49d34d0 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef02d9d640 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4e127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7fef01d9b640 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4da2400 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feef4000f80 tx=0x7feef400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49e33c8223a755f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.568+0000 7feee9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.569+0000 7feee9ffb640 1 -- 172.21.15.27:0/3992122721 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4021040 con 0x5580f49d34d0 2024-04-25T13:51:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.569+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feefc0c8300 con 0x5580f49d34d0 2024-04-25T13:51:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.569+0000 7feee9ffb640 1 -- 172.21.15.27:0/3992122721 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feef4003d80 con 0x5580f49d34d0 2024-04-25T13:51:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.569+0000 7feee9ffb640 1 -- 172.21.15.27:0/3992122721 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7feef4009810 con 0x5580f49d34d0 2024-04-25T13:51:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.569+0000 7fef0fdd8740 10 client.27223 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.569+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5580f4e13bd0 con 0x5580f49d34d0 2024-04-25T13:51:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.570+0000 7feee9ffb640 1 -- 172.21.15.27:0/3992122721 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feefc0c8300 con 0x5580f49d34d0 2024-04-25T13:51:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.571+0000 7feee9ffb640 1 -- 172.21.15.27:0/3992122721 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7feef400f040 con 0x5580f49d34d0 2024-04-25T13:51:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.571+0000 7fef0fdd8740 10 client.27223 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.571+0000 7fef0fdd8740 10 client.27223 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.571+0000 7fef0fdd8740 20 client.27223 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.571+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580f49881c0 0x5580f4e165f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.571+0000 7fef0fdd8740 4 client.27223 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:51:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.571+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5580f4d01ee0 con 0x5580f49881c0 2024-04-25T13:51:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.571+0000 7fef0259c640 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580f49881c0 0x5580f4e165f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.572+0000 7fef0259c640 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580f49881c0 0x5580f4e165f0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=867dabd82f4589ef server_cookie=43db5467d998a89e in_seq=0 out_seq=0 2024-04-25T13:51:17.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.573+0000 7feee9ffb640 10 client.27223 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.590+0000 7f399fa69640 1 -- 172.21.15.27:0/3408896691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810b520 msgr2=0x7f399810b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.590+0000 7f399fa69640 1 --2- 172.21.15.27:0/3408896691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810b520 0x7f399810b930 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3988008570 tx=0x7f398802c410 comp rx=0 tx=0).stop 2024-04-25T13:51:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.591+0000 7f399fa69640 1 -- 172.21.15.27:0/3408896691 shutdown_connections 2024-04-25T13:51:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.591+0000 7f399fa69640 1 --2- 172.21.15.27:0/3408896691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39981102b0 0x7f39981126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.591+0000 7f399fa69640 1 --2- 172.21.15.27:0/3408896691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810b520 0x7f399810b930 secure :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3988008570 tx=0x7f398802c410 comp rx=0 tx=0).stop 2024-04-25T13:51:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.591+0000 7f399fa69640 1 --2- 172.21.15.27:0/3408896691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f399810a380 0x7f399810afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.591+0000 7f399fa69640 1 -- 172.21.15.27:0/3408896691 >> 172.21.15.27:0/3408896691 conn(0x7f39980707b0 msgr2=0x7f3998072c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.592+0000 7f399fa69640 1 -- 172.21.15.27:0/3408896691 shutdown_connections 2024-04-25T13:51:17.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.592+0000 7f399fa69640 1 -- 172.21.15.27:0/3408896691 wait complete. 2024-04-25T13:51:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.593+0000 7f399fa69640 1 Processor -- start 2024-04-25T13:51:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.594+0000 7f399fa69640 1 -- start start 2024-04-25T13:51:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.594+0000 7f399fa69640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810a380 0x7f39981a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.594+0000 7f399fa69640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39981102b0 0x7f39981a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.594+0000 7f399d805640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810a380 0x7f39981a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.594+0000 7f399d805640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810a380 0x7f39981a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57234/0 (socket says 172.21.15.27:57234) 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.595+0000 7f399d805640 1 -- 172.21.15.27:0/2204888282 learned_addr learned my addr 172.21.15.27:0/2204888282 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.595+0000 7f399fa69640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39981a5cd0 0x7f39981a80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.595+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3998115b20 con 0x7f399810a380 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.595+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39981159a0 con 0x7f39981102b0 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.595+0000 7f399d004640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39981102b0 0x7f39981a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.595+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f3998115ca0 con 0x7f39981a5cd0 2024-04-25T13:51:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.595+0000 7f399e006640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39981a5cd0 0x7f39981a80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.596+0000 7f399e006640 1 -- 172.21.15.27:0/2204888282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39981102b0 msgr2=0x7f39981a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.596+0000 7f399e006640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39981102b0 0x7f39981a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.596+0000 7f399e006640 1 -- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810a380 msgr2=0x7f39981a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.596+0000 7f399e006640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810a380 0x7f39981a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.596+0000 7f399e006640 1 -- 172.21.15.27:0/2204888282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39981a8820 con 0x7f39981a5cd0 2024-04-25T13:51:17.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.597+0000 7f399e006640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39981a5cd0 0x7f39981a80f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f399400b410 tx=0x7f399400b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.597+0000 7f398effd640 1 -- 172.21.15.27:0/2204888282 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994015020 con 0x7f39981a5cd0 2024-04-25T13:51:17.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.598+0000 7f398effd640 1 -- 172.21.15.27:0/2204888282 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3994011ce0 con 0x7f39981a5cd0 2024-04-25T13:51:17.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.598+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39981a8b10 con 0x7f39981a5cd0 2024-04-25T13:51:17.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.598+0000 7f398effd640 1 -- 172.21.15.27:0/2204888282 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994010da0 con 0x7f39981a5cd0 2024-04-25T13:51:17.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.598+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39981a9000 con 0x7f39981a5cd0 2024-04-25T13:51:17.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.601+0000 7f398effd640 1 -- 172.21.15.27:0/2204888282 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3994018020 con 0x7f39981a5cd0 2024-04-25T13:51:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.602+0000 7f39a69f1740 1 Processor -- start 2024-04-25T13:51:17.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f39a69f1740 1 -- start start 2024-04-25T13:51:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.603+0000 7feee9ffb640 1 -- 172.21.15.27:0/3992122721 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5580f4d01ee0 con 0x5580f49881c0 2024-04-25T13:51:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.603+0000 7feee9ffb640 10 client.27223 handle_command_reply: tid=0 2024-04-25T13:51:17.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bc6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.603+0000 7fef0fdd8740 1 client.27223 shutdown 2024-04-25T13:51:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bc90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bcb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f39a69f1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a600b20d80 con 0x55a6007f14d0 2024-04-25T13:51:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f39a69f1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55a600ada8b0 con 0x55a6007f96b0 2024-04-25T13:51:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f39a69f1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55a600b1cd40 con 0x55a60081a390 2024-04-25T13:51:17.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f398effd640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39840afbe0 0x7f39840b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.603+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 msgr2=0x5580f4da2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.603+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4da2400 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feef4000f80 tx=0x7feef400edf0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f399e006640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bcb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f399d004640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bc90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f399e006640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bcb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45690/0 (socket says 172.21.15.27:45690) 2024-04-25T13:51:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f399e006640 1 -- 172.21.15.27:0/950551075 learned_addr learned my addr 172.21.15.27:0/950551075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.603+0000 7f399d004640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bc90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57248/0 (socket says 172.21.15.27:57248) 2024-04-25T13:51:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.604+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 shutdown_connections 2024-04-25T13:51:17.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.604+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5580f49881c0 0x5580f4e165f0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.604+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5580f49f41b0 0x5580f4e127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.604+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5580f49d34d0 0x5580f4da2400 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.604+0000 7fef0fdd8740 1 --2- 172.21.15.27:0/3992122721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5580f49cb2f0 0x5580f4da1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.604+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 >> 172.21.15.27:0/3992122721 conn(0x5580f49e8300 msgr2=0x5580f4d9e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.606 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.604+0000 7feee9ffb640 0 client.27223 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.604+0000 7f399d004640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39840afbe0 0x7f39840b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.604+0000 7f399d805640 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bc6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.604+0000 7f399e006640 1 -- 172.21.15.27:0/950551075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 msgr2=0x55a600bc6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.604+0000 7f399e006640 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bc6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.604+0000 7f399e006640 1 -- 172.21.15.27:0/950551075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 msgr2=0x55a600bc90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.604+0000 7f399e006640 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bc90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.607 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.604+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 shutdown_connections 2024-04-25T13:51:17.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.604+0000 7f399e006640 1 -- 172.21.15.27:0/950551075 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a600aaba40 con 0x55a6007f96b0 2024-04-25T13:51:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.605+0000 7fef0fdd8740 1 -- 172.21.15.27:0/3992122721 wait complete. 2024-04-25T13:51:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.605+0000 7f399d805640 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bc6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.605+0000 7f399e006640 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bcb880 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f39940dbd70 tx=0x7f399401f3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b2241d67f987b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.605+0000 7f3983fff640 1 -- 172.21.15.27:0/950551075 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399400c150 con 0x55a6007f96b0 2024-04-25T13:51:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.605+0000 7f3983fff640 1 -- 172.21.15.27:0/950551075 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39940a6ee0 con 0x55a6007f96b0 2024-04-25T13:51:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.605+0000 7f3983fff640 1 -- 172.21.15.27:0/950551075 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994015600 con 0x55a6007f96b0 2024-04-25T13:51:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.605+0000 7f398effd640 1 -- 172.21.15.27:0/2204888282 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f39940de330 con 0x7f39981a5cd0 2024-04-25T13:51:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.606+0000 7fef0fdd8740 20 client.27223 trim_cache size 0 max 16384 2024-04-25T13:51:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:51:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:51:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:51:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:51:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.607+0000 7f399d004640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39840afbe0 0x7f39840b20f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3988035040 tx=0x7f3988005780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.607+0000 7f39a69f1740 1 -- 172.21.15.27:0/950551075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 msgr2=0x55a600bcb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.607+0000 7f39a69f1740 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bcb880 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f39940dbd70 tx=0x7f399401f3f0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.607+0000 7fef08f26640 1 -- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feeec0afcf0 msgr2=0x7feeec0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.607+0000 7fef08f26640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feeec0afcf0 0x7feeec0b2200 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7feef4009fd0 tx=0x7feef4009ec0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.607+0000 7fef08f26640 1 -- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef04110090 msgr2=0x7fef04081d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.607+0000 7fef08f26640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef04110090 0x7fef04081d10 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feefc0303f0 tx=0x7feefc032df0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.608+0000 7fef08f26640 1 -- 172.21.15.27:0/2633610846 shutdown_connections 2024-04-25T13:51:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.608+0000 7fef08f26640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feeec0afcf0 0x7feeec0b2200 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.608+0000 7fef08f26640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef04110090 0x7fef04081d10 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.608+0000 7fef08f26640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef0410b390 0x7fef04080db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.608+0000 7fef08f26640 1 --2- 172.21.15.27:0/2633610846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef0410a150 0x7fef04080870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.608+0000 7f39a69f1740 1 -- 172.21.15.27:0/950551075 shutdown_connections 2024-04-25T13:51:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.608+0000 7fef08f26640 1 -- 172.21.15.27:0/2633610846 >> 172.21.15.27:0/2633610846 conn(0x7fef04078de0 msgr2=0x7fef0410c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.608+0000 7f39a69f1740 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bcb880 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.608+0000 7f39a69f1740 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bc90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.608+0000 7f39a69f1740 1 --2- 172.21.15.27:0/950551075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bc6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.608+0000 7f39a69f1740 1 -- 172.21.15.27:0/950551075 >> 172.21.15.27:0/950551075 conn(0x55a60080e4e0 msgr2=0x55a600bb73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.608+0000 7fef08f26640 1 -- 172.21.15.27:0/2633610846 shutdown_connections 2024-04-25T13:51:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:17.608+0000 7fef08f26640 1 -- 172.21.15.27:0/2633610846 wait complete. 2024-04-25T13:51:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.608+0000 7f39a69f1740 1 -- 172.21.15.27:0/950551075 shutdown_connections 2024-04-25T13:51:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.608+0000 7f39a69f1740 1 -- 172.21.15.27:0/950551075 wait complete. 2024-04-25T13:51:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f39a69f1740 1 -- start start 2024-04-25T13:51:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f39a69f1740 1 Processor -- start 2024-04-25T13:51:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bd56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f39a69f1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a600bcc340 con 0x55a6007f14d0 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f39a69f1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55a600aa7d10 con 0x55a6007f96b0 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f399d004640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f399d805640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f399d004640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45704/0 (socket says 172.21.15.27:45704) 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f399d004640 1 -- 172.21.15.27:0/3719778777 learned_addr learned my addr 172.21.15.27:0/3719778777 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.609+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55a600bcc4c0 con 0x55a60081a390 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f39a69f1740 10 client.0 _ll_register_callbacks cb 0x55a600ad4210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f399e006640 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bd56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f399d004640 1 -- 172.21.15.27:0/3719778777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 msgr2=0x55a600bd56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f399d004640 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bd56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f399d004640 1 -- 172.21.15.27:0/3719778777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 msgr2=0x55a600bce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f399d004640 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bce210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f399d004640 1 -- 172.21.15.27:0/3719778777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6008fc7d0 con 0x55a6007f96b0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f399d004640 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bce750 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3988005ab0 tx=0x7f398800ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bd7d50bfe6b24de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f39817fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f39817fa640 1 -- 172.21.15.27:0/3719778777 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39880035f0 con 0x55a6007f96b0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f39817fa640 1 -- 172.21.15.27:0/3719778777 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3988003790 con 0x55a6007f96b0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39940a6ee0 con 0x55a6007f96b0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f39817fa640 1 -- 172.21.15.27:0/3719778777 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f3988003990 con 0x55a6007f96b0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f39a69f1740 10 client.27188 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.610+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55a6008323f0 con 0x55a6007f96b0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f39817fa640 1 -- 172.21.15.27:0/3719778777 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39940a6ee0 con 0x55a6007f96b0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f39817fa640 1 -- 172.21.15.27:0/3719778777 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f3988041440 con 0x55a6007f96b0 2024-04-25T13:51:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f39a69f1740 10 client.27188 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f39a69f1740 10 client.27188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f39a69f1740 20 client.27188 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f39a69f1740 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a6007d8f60 0x55a600bdbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f39a69f1740 4 client.27188 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a600b2e990 con 0x55a6007d8f60 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.611+0000 7f399d805640 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a6007d8f60 0x55a600bdbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.612+0000 7f39817fa640 10 client.27188 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.612+0000 7f399d805640 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a6007d8f60 0x55a600bdbbf0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=294146bcd411959a server_cookie=3c51cdefcc08188c in_seq=0 out_seq=0 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.613+0000 7f39817fa640 1 -- 172.21.15.27:0/3719778777 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a600b2e990 con 0x55a6007d8f60 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.613+0000 7f39817fa640 10 client.27188 handle_command_reply: tid=0 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.613+0000 7f39a69f1740 1 client.27188 shutdown 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.613+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 msgr2=0x55a600bce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.613+0000 7f39a69f1740 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bce750 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3988005ab0 tx=0x7f398800ac00 comp rx=0 tx=0).stop 2024-04-25T13:51:17.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 shutdown_connections 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a6007d8f60 0x55a600bdbbf0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bd56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bce750 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 1 --2- 172.21.15.27:0/3719778777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f14d0 0x55a600bce210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 >> 172.21.15.27:0/3719778777 conn(0x55a60080e4e0 msgr2=0x55a600bb59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39817fa640 0 client.27188 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 shutdown_connections 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 1 -- 172.21.15.27:0/3719778777 wait complete. 2024-04-25T13:51:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.614+0000 7f39a69f1740 20 client.27188 trim_cache size 0 max 16384 2024-04-25T13:51:17.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.661+0000 7f39a69f1740 1 Processor -- start 2024-04-25T13:51:17.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.661+0000 7f39a69f1740 1 -- start start 2024-04-25T13:51:17.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.661+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f14d0 0x55a600bcced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.661+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f96b0 0x55a600bc4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.661+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.661+0000 7f39a69f1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f39981afcb0 con 0x55a6007f96b0 2024-04-25T13:51:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.661+0000 7f39a69f1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39981b2530 con 0x55a6007f14d0 2024-04-25T13:51:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.661+0000 7f39a69f1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f39981b0320 con 0x55a60081a390 2024-04-25T13:51:17.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.666+0000 7f399e006640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.666+0000 7f399e006640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:37434/0 (socket says 172.21.15.27:37434) 2024-04-25T13:51:17.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.666+0000 7f399e006640 1 -- 172.21.15.27:0/1622852792 learned_addr learned my addr 172.21.15.27:0/1622852792 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.666+0000 7f399d805640 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f14d0 0x55a600bcced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.667+0000 7f399d004640 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f96b0 0x55a600bc4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.670+0000 7f399d805640 1 -- 172.21.15.27:0/1622852792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 msgr2=0x55a600bce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.670+0000 7f399d805640 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bce210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.670+0000 7f399d805640 1 -- 172.21.15.27:0/1622852792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f96b0 msgr2=0x55a600bc4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.670+0000 7f399d805640 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f96b0 0x55a600bc4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.670+0000 7f399d805640 1 -- 172.21.15.27:0/1622852792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6007f20f0 con 0x55a6007f14d0 2024-04-25T13:51:17.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.671+0000 7f399e006640 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bce210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.671+0000 7f399d805640 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f14d0 0x55a600bcced0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55a600bb18a0 tx=0x7f3990009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb62014bf11cd3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.671+0000 7f39837fe640 1 -- 172.21.15.27:0/1622852792 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990007e90 con 0x55a6007f14d0 2024-04-25T13:51:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.671+0000 7f39837fe640 1 -- 172.21.15.27:0/1622852792 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f399001d680 con 0x55a6007f14d0 2024-04-25T13:51:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.671+0000 7f39837fe640 1 -- 172.21.15.27:0/1622852792 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990026a10 con 0x55a6007f14d0 2024-04-25T13:51:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.673+0000 7f39a69f1740 1 -- 172.21.15.27:0/1622852792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f14d0 msgr2=0x55a600bcced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.673+0000 7f39a69f1740 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f14d0 0x55a600bcced0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55a600bb18a0 tx=0x7f3990009730 comp rx=0 tx=0).stop 2024-04-25T13:51:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.677+0000 7f39a69f1740 1 -- 172.21.15.27:0/1622852792 shutdown_connections 2024-04-25T13:51:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.677+0000 7f39a69f1740 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a60081a390 0x55a600bce210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.677+0000 7f39a69f1740 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a6007f96b0 0x55a600bc4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.677+0000 7f39a69f1740 1 --2- 172.21.15.27:0/1622852792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f14d0 0x55a600bcced0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.677+0000 7f39a69f1740 1 -- 172.21.15.27:0/1622852792 >> 172.21.15.27:0/1622852792 conn(0x55a60080e4e0 msgr2=0x55a600bc9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.679+0000 7f39a69f1740 1 -- 172.21.15.27:0/1622852792 shutdown_connections 2024-04-25T13:51:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.682+0000 7f39a69f1740 1 -- 172.21.15.27:0/1622852792 wait complete. 2024-04-25T13:51:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.682+0000 7f39a69f1740 1 -- start start 2024-04-25T13:51:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.682+0000 7f39a69f1740 1 Processor -- start 2024-04-25T13:51:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.685+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a6007f14d0 0x55a600bd12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.685+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bd3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.685+0000 7f39a69f1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a60081a390 0x55a600bd3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.685+0000 7f39a69f1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f39981b09f0 con 0x55a60081a390 2024-04-25T13:51:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.685+0000 7f39a69f1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f39981b1e60 con 0x55a6007f96b0 2024-04-25T13:51:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.685+0000 7f39a69f1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f39981b10c0 con 0x55a6007f14d0 2024-04-25T13:51:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.685+0000 7f39a69f1740 10 client.0 _ll_register_callbacks cb 0x7f3988008230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.687+0000 7f399e006640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a60081a390 0x55a600bd3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.687+0000 7f399e006640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a60081a390 0x55a600bd3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57260/0 (socket says 172.21.15.27:57260) 2024-04-25T13:51:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.687+0000 7f399d004640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bd3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.687+0000 7f399e006640 1 -- 172.21.15.27:0/2187564111 learned_addr learned my addr 172.21.15.27:0/2187564111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.687+0000 7f399d805640 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a6007f14d0 0x55a600bd12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.688+0000 7f399e006640 1 -- 172.21.15.27:0/2187564111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a6007f14d0 msgr2=0x55a600bd12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.688+0000 7f399e006640 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a6007f14d0 0x55a600bd12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.688+0000 7f399e006640 1 -- 172.21.15.27:0/2187564111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 msgr2=0x55a600bd3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:17.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.688+0000 7f399e006640 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bd3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.688+0000 7f399e006640 1 -- 172.21.15.27:0/2187564111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a600ae14b0 con 0x55a60081a390 2024-04-25T13:51:17.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.689+0000 7f399e006640 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a60081a390 0x55a600bd3d50 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f39940af110 tx=0x7f39940c8c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e94312b2c38867 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:17.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.689+0000 7f39817fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.691+0000 7f39817fa640 1 -- 172.21.15.27:0/2187564111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39940a65b0 con 0x55a60081a390 2024-04-25T13:51:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.691+0000 7f39a69f1740 1 -- 172.21.15.27:0/2187564111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f399001d680 con 0x55a60081a390 2024-04-25T13:51:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.691+0000 7f39817fa640 1 -- 172.21.15.27:0/2187564111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f39940a2dd0 con 0x55a60081a390 2024-04-25T13:51:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.694+0000 7f39817fa640 1 -- 172.21.15.27:0/2187564111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f39940a62f0 con 0x55a60081a390 2024-04-25T13:51:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.695+0000 7f39a69f1740 10 client.17493 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.695+0000 7f39a69f1740 1 -- 172.21.15.27:0/2187564111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55a600bbd2b0 con 0x55a60081a390 2024-04-25T13:51:17.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.696+0000 7f39817fa640 1 -- 172.21.15.27:0/2187564111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f399001d680 con 0x55a60081a390 2024-04-25T13:51:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.697+0000 7f39817fa640 1 -- 172.21.15.27:0/2187564111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f39940af1b0 con 0x55a60081a390 2024-04-25T13:51:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.697+0000 7f39a69f1740 10 client.17493 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.697+0000 7f39a69f1740 10 client.17493 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.697+0000 7f39a69f1740 20 client.17493 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.697+0000 7f39a69f1740 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a6007ae3a0 0x55a600bbfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.697+0000 7f39a69f1740 4 client.17493 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:51:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.697+0000 7f39a69f1740 1 -- 172.21.15.27:0/2187564111 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a600b28800 con 0x55a6007ae3a0 2024-04-25T13:51:17.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.701+0000 7f399d805640 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a6007ae3a0 0x55a600bbfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.704+0000 7f399d805640 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a6007ae3a0 0x55a600bbfce0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32f90ac6ac816876 server_cookie=5c906623a60357f4 in_seq=0 out_seq=0 2024-04-25T13:51:17.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.704+0000 7f39817fa640 10 client.17493 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.704+0000 7f39817fa640 1 -- 172.21.15.27:0/2187564111 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a600b28800 con 0x55a6007ae3a0 2024-04-25T13:51:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.704+0000 7f39817fa640 10 client.17493 handle_command_reply: tid=0 2024-04-25T13:51:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.704+0000 7f39a69f1740 1 client.17493 shutdown 2024-04-25T13:51:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 -- 172.21.15.27:0/2187564111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a60081a390 msgr2=0x55a600bd3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a60081a390 0x55a600bd3d50 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f39940af110 tx=0x7f39940c8c90 comp rx=0 tx=0).stop 2024-04-25T13:51:17.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 -- 172.21.15.27:0/2187564111 shutdown_connections 2024-04-25T13:51:17.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a6007ae3a0 0x55a600bbfce0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a60081a390 0x55a600bd3d50 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a6007f96b0 0x55a600bd3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 --2- 172.21.15.27:0/2187564111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a6007f14d0 0x55a600bd12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 -- 172.21.15.27:0/2187564111 >> 172.21.15.27:0/2187564111 conn(0x55a60080e4e0 msgr2=0x55a600be0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39817fa640 0 client.17493 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 -- 172.21.15.27:0/2187564111 shutdown_connections 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 1 -- 172.21.15.27:0/2187564111 wait complete. 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.705+0000 7f39a69f1740 20 client.17493 trim_cache size 0 max 16384 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "3b495791-be56-47ad-b864-7acea1e3e7c3": { 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "3b495791-be56-47ad-b864-7acea1e3e7c3", 2024-04-25T13:51:17.706 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:51:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:51:17.707 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:51:17.707 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:51:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39840afbe0 msgr2=0x7f39840b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39840afbe0 0x7f39840b20f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3988035040 tx=0x7f3988005780 comp rx=0 tx=0).stop 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39981a5cd0 msgr2=0x7f39981a80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39981a5cd0 0x7f39981a80f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f399400b410 tx=0x7f399400b8d0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 shutdown_connections 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f39840afbe0 0x7f39840b20f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39981a5cd0 0x7f39981a80f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39981102b0 0x7f39981a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 --2- 172.21.15.27:0/2204888282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f399810a380 0x7f39981a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 >> 172.21.15.27:0/2204888282 conn(0x7f39980707b0 msgr2=0x7f399810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 shutdown_connections 2024-04-25T13:51:17.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:17.707+0000 7f399fa69640 1 -- 172.21.15.27:0/2204888282 wait complete. 2024-04-25T13:51:17.723 INFO:tasks.cephfs.filesystem:scrub status for tag:3b495791-be56-47ad-b864-7acea1e3e7c3 - {'path': '/', 'tag': '3b495791-be56-47ad-b864-7acea1e3e7c3', 'options': 'recursive,force'} 2024-04-25T13:51:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 610.0 s, 2117.2 tps, lat 47.353 ms stddev 55.067 2024-04-25T13:51:18.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:17 smithi122 ceph-mon[32843]: pgmap v661: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 675 op/s 2024-04-25T13:51:18.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:17 smithi045 ceph-mon[32731]: pgmap v661: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 675 op/s 2024-04-25T13:51:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:17 smithi027 ceph-mon[29140]: pgmap v661: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 675 op/s 2024-04-25T13:51:19.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:18 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:51:19.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:18 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:19.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:18 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:51:19.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:18 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:19.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:18 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:51:19.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:18 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:19.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:18 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:51:19.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:18 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:19.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:18 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:51:19.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:18 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:19.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:18 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:51:19.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:18 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:20.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:19 smithi027 ceph-mon[29140]: pgmap v662: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-04-25T13:51:20.051 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:19 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:51:20.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:19 smithi122 ceph-mon[32843]: pgmap v662: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-04-25T13:51:20.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:19 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:51:20.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:19 smithi045 ceph-mon[32731]: pgmap v662: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-04-25T13:51:20.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:19 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:51:22.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:21 smithi122 ceph-mon[32843]: pgmap v663: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-25T13:51:22.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:21 smithi045 ceph-mon[32731]: pgmap v663: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-25T13:51:22.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:21 smithi027 ceph-mon[29140]: pgmap v663: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.8 MiB/s wr, 655 op/s 2024-04-25T13:51:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 615.0 s, 1951.7 tps, lat 51.354 ms stddev 59.697 2024-04-25T13:51:23.018 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:22 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:51:23.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:22 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:51:23.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:22 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:51:24.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:23 smithi122 ceph-mon[32843]: pgmap v664: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.0 MiB/s wr, 752 op/s 2024-04-25T13:51:24.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:23 smithi045 ceph-mon[32731]: pgmap v664: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.0 MiB/s wr, 752 op/s 2024-04-25T13:51:24.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:23 smithi027 ceph-mon[29140]: pgmap v664: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.0 MiB/s wr, 752 op/s 2024-04-25T13:51:25.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:25 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:25 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:25.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:25 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:26 smithi122 ceph-mon[32843]: pgmap v665: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.4 MiB/s wr, 717 op/s 2024-04-25T13:51:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:51:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:51:26.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:26 smithi045 ceph-mon[32731]: pgmap v665: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.4 MiB/s wr, 717 op/s 2024-04-25T13:51:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:51:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:51:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:26.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:26 smithi027 ceph-mon[29140]: pgmap v665: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.4 MiB/s wr, 717 op/s 2024-04-25T13:51:26.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:51:26.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:51:26.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:51:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 620.0 s, 1692.7 tps, lat 58.556 ms stddev 64.303 2024-04-25T13:51:28.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:28 smithi122 ceph-mon[32843]: pgmap v666: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.7 MiB/s wr, 816 op/s 2024-04-25T13:51:28.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:28 smithi045 ceph-mon[32731]: pgmap v666: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.7 MiB/s wr, 816 op/s 2024-04-25T13:51:28.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:28 smithi027 ceph-mon[29140]: pgmap v666: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.7 MiB/s wr, 816 op/s 2024-04-25T13:51:30.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:30 smithi045 ceph-mon[32731]: pgmap v667: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.6 MiB/s wr, 702 op/s 2024-04-25T13:51:30.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:30 smithi027 ceph-mon[29140]: pgmap v667: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.6 MiB/s wr, 702 op/s 2024-04-25T13:51:30.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:30 smithi122 ceph-mon[32843]: pgmap v667: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.6 MiB/s wr, 702 op/s 2024-04-25T13:51:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2408790801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd88103c60 msgr2=0x7fdd88104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2408790801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd88103c60 0x7fdd88104070 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009900 tx=0x7fdd7c02a410 comp rx=0 tx=0).stop 2024-04-25T13:51:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2408790801 shutdown_connections 2024-04-25T13:51:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2408790801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd88108960 0x7fdd8810ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2408790801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd88103c60 0x7fdd88104070 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2408790801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd88102a20 0x7fdd88103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2408790801 >> 172.21.15.27:0/2408790801 conn(0x7fdd880fdc10 msgr2=0x7fdd881000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2408790801 shutdown_connections 2024-04-25T13:51:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.763+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2408790801 wait complete. 2024-04-25T13:51:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.765+0000 7fdd8e8e7640 1 Processor -- start 2024-04-25T13:51:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.765+0000 7fdd8e8e7640 1 -- start start 2024-04-25T13:51:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.765+0000 7fdd8e8e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd88102a20 0x7fdd8819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.765+0000 7fdd8e8e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd88103c60 0x7fdd8819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.765+0000 7fdd8e8e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd88108960 0x7fdd881a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.765+0000 7fdd8ce84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd88108960 0x7fdd881a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.765+0000 7fdd8ce84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd88108960 0x7fdd881a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42604/0 (socket says 172.21.15.27:42604) 2024-04-25T13:51:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.766+0000 7fdd8ce84640 1 -- 172.21.15.27:0/2052902754 learned_addr learned my addr 172.21.15.27:0/2052902754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.766+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8810d440 con 0x7fdd88102a20 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.766+0000 7fdd87fff640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd88102a20 0x7fdd8819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.766+0000 7fdd877fe640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd88103c60 0x7fdd8819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.766+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8810d2c0 con 0x7fdd88103c60 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.766+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8810d5c0 con 0x7fdd88108960 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd877fe640 1 -- 172.21.15.27:0/2052902754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd88108960 msgr2=0x7fdd881a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd877fe640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd88108960 0x7fdd881a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd877fe640 1 -- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd88102a20 msgr2=0x7fdd8819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd877fe640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd88102a20 0x7fdd8819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd877fe640 1 -- 172.21.15.27:0/2052902754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd881a3fa0 con 0x7fdd88103c60 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd877fe640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd88103c60 0x7fdd8819c9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c006010 tx=0x7fdd7c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd857fa640 1 -- 172.21.15.27:0/2052902754 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c03f070 con 0x7fdd88103c60 2024-04-25T13:51:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd857fa640 1 -- 172.21.15.27:0/2052902754 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd7c031910 con 0x7fdd88103c60 2024-04-25T13:51:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd881a4230 con 0x7fdd88103c60 2024-04-25T13:51:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd857fa640 1 -- 172.21.15.27:0/2052902754 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c035070 con 0x7fdd88103c60 2024-04-25T13:51:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.767+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd881a4720 con 0x7fdd88103c60 2024-04-25T13:51:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd95796740 1 Processor -- start 2024-04-25T13:51:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd95796740 1 -- start start 2024-04-25T13:51:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd857fa640 1 -- 172.21.15.27:0/2052902754 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdd7c0037f0 con 0x7fdd88103c60 2024-04-25T13:51:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd95796740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d2196970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd95796740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d2196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd95796740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d219ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd95796740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5555d20fc460 con 0x5555d1dd54d0 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd95796740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5555d20b6590 con 0x5555d1dcd2f0 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd95796740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5555d20f8420 con 0x5555d1df61b0 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd8ce84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d2196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd8ce84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d2196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42614/0 (socket says 172.21.15.27:42614) 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.770+0000 7fdd8ce84640 1 -- 172.21.15.27:0/2850218696 learned_addr learned my addr 172.21.15.27:0/2850218696 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd87fff640 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d2196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd877fe640 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d219ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd8ce84640 1 -- 172.21.15.27:0/2850218696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 msgr2=0x5555d2196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd8ce84640 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d2196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd8ce84640 1 -- 172.21.15.27:0/2850218696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 msgr2=0x5555d219ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd8ce84640 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d219ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd8ce84640 1 -- 172.21.15.27:0/2850218696 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555d20c30d0 con 0x5555d1df61b0 2024-04-25T13:51:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd877fe640 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d219ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd8ce84640 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d2196970 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009fd0 tx=0x7fdd800118c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b04e83e7a4345974 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd6e7fc640 1 -- 172.21.15.27:0/2850218696 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80012770 con 0x5555d1df61b0 2024-04-25T13:51:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd6e7fc640 1 -- 172.21.15.27:0/2850218696 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd80012dc0 con 0x5555d1df61b0 2024-04-25T13:51:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.771+0000 7fdd6e7fc640 1 -- 172.21.15.27:0/2850218696 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8000fe20 con 0x5555d1df61b0 2024-04-25T13:51:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd857fa640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdd700afae0 0x7fdd700b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd857fa640 1 -- 172.21.15.27:0/2052902754 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdd7c0ff270 con 0x7fdd88103c60 2024-04-25T13:51:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd87fff640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdd700afae0 0x7fdd700b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 -- 172.21.15.27:0/2850218696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 msgr2=0x5555d2196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d2196970 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009fd0 tx=0x7fdd800118c0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 -- 172.21.15.27:0/2850218696 shutdown_connections 2024-04-25T13:51:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d219ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d2196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 --2- 172.21.15.27:0/2850218696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d2196970 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 -- 172.21.15.27:0/2850218696 >> 172.21.15.27:0/2850218696 conn(0x5555d1dea300 msgr2=0x5555d2192300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 -- 172.21.15.27:0/2850218696 shutdown_connections 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd87fff640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdd700afae0 0x7fdd700b1ff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x5555d2197d50 tx=0x7fdd78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.772+0000 7fdd95796740 1 -- 172.21.15.27:0/2850218696 wait complete. 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.773+0000 7fdd95796740 1 -- start start 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.773+0000 7fdd95796740 1 Processor -- start 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.774+0000 7fdd95796740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d21a0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.774+0000 7fdd95796740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d21a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.774+0000 7fdd87fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d21a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.774+0000 7fdd87fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d21a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:42494/0 (socket says 172.21.15.27:42494) 2024-04-25T13:51:30.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.775+0000 7fdd877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d21a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.775+0000 7fdd877fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d21a0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38868/0 (socket says 172.21.15.27:38868) 2024-04-25T13:51:30.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.775+0000 7fdd87fff640 1 -- 172.21.15.27:0/3974072037 learned_addr learned my addr 172.21.15.27:0/3974072037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:30.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.775+0000 7fdd95796740 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d21a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.778 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.775+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5555d219e840 con 0x5555d1dd54d0 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.775+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5555d2083b60 con 0x5555d1dcd2f0 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.775+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5555d219e9c0 con 0x5555d1df61b0 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd95796740 10 client.0 _ll_register_callbacks cb 0x5555d20a82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd8ce84640 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d21a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd8ce84640 1 -- 172.21.15.27:0/3974072037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 msgr2=0x5555d21a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd8ce84640 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d21a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd8ce84640 1 -- 172.21.15.27:0/3974072037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 msgr2=0x5555d21a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd8ce84640 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d21a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd8ce84640 1 -- 172.21.15.27:0/3974072037 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555d1e0e210 con 0x5555d1df61b0 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd87fff640 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d21a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd877fe640 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d21a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd8ce84640 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d21a7b90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdd80005c40 tx=0x7fdd8000f6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59cd38dc796ca7f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.776+0000 7fdd677fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd677fe640 1 -- 172.21.15.27:0/3974072037 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80028070 con 0x5555d1df61b0 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd677fe640 1 -- 172.21.15.27:0/3974072037 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdd80012570 con 0x5555d1df61b0 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd80012dc0 con 0x5555d1df61b0 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd677fe640 1 -- 172.21.15.27:0/3974072037 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdd80034050 con 0x5555d1df61b0 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 10 client.27250 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5555d1eda620 con 0x5555d1df61b0 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd677fe640 1 -- 172.21.15.27:0/3974072037 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd80012dc0 con 0x5555d1df61b0 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd677fe640 1 -- 172.21.15.27:0/3974072037 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fdd80005500 con 0x5555d1df61b0 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 10 client.27250 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 10 client.27250 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 20 client.27250 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5555d1db4d80 0x5555d21ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 4 client.27250 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.777+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555d210a070 con 0x5555d1db4d80 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.778+0000 7fdd87fff640 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5555d1db4d80 0x5555d21ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.778+0000 7fdd677fe640 10 client.27250 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.778+0000 7fdd87fff640 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5555d1db4d80 0x5555d21ae0c0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd480ee836786a1e server_cookie=c6233e3dcf0cd201 in_seq=0 out_seq=0 2024-04-25T13:51:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd677fe640 1 -- 172.21.15.27:0/3974072037 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5555d210a070 con 0x5555d1db4d80 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd677fe640 10 client.27250 handle_command_reply: tid=0 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 client.27250 shutdown 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 msgr2=0x5555d21a7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d21a7b90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdd80005c40 tx=0x7fdd8000f6f0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 shutdown_connections 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5555d1db4d80 0x5555d21ae0c0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1df61b0 0x5555d21a7b90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1dd54d0 0x5555d21a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 --2- 172.21.15.27:0/3974072037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d21a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd677fe640 0 client.27250 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.779+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 >> 172.21.15.27:0/3974072037 conn(0x5555d1dea300 msgr2=0x5555d2195210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.780+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 shutdown_connections 2024-04-25T13:51:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.780+0000 7fdd95796740 1 -- 172.21.15.27:0/3974072037 wait complete. 2024-04-25T13:51:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.780+0000 7fdd95796740 20 client.27250 trim_cache size 0 max 16384 2024-04-25T13:51:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.797+0000 7fdd95796740 1 Processor -- start 2024-04-25T13:51:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.797+0000 7fdd95796740 1 -- start start 2024-04-25T13:51:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd95796740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d219b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd95796740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dd54d0 0x5555d21a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd95796740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d2195810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd95796740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd881ab3c0 con 0x5555d1df61b0 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd95796740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd881adc40 con 0x5555d1dcd2f0 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd95796740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdd881aba30 con 0x5555d1dd54d0 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd8ce84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d2195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd877fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dd54d0 0x5555d21a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd8ce84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d2195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38882/0 (socket says 172.21.15.27:38882) 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd877fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dd54d0 0x5555d21a7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42628/0 (socket says 172.21.15.27:42628) 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd8ce84640 1 -- 172.21.15.27:0/4029523591 learned_addr learned my addr 172.21.15.27:0/4029523591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd87fff640 1 --2- 172.21.15.27:0/4029523591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d219b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd877fe640 1 -- 172.21.15.27:0/4029523591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 msgr2=0x5555d219b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd877fe640 1 --2- 172.21.15.27:0/4029523591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d219b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd877fe640 1 -- 172.21.15.27:0/4029523591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 msgr2=0x5555d2195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd877fe640 1 --2- 172.21.15.27:0/4029523591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d2195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.798+0000 7fdd877fe640 1 -- 172.21.15.27:0/4029523591 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555d219eb00 con 0x5555d1dd54d0 2024-04-25T13:51:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.799+0000 7fdd877fe640 1 --2- 172.21.15.27:0/4029523591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dd54d0 0x5555d21a7ab0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c000c00 tx=0x7fdd7c02a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d56c005375663e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.799+0000 7fdd6dffb640 1 -- 172.21.15.27:0/4029523591 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c048040 con 0x5555d1dd54d0 2024-04-25T13:51:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.799+0000 7fdd6dffb640 1 -- 172.21.15.27:0/4029523591 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd7c0cee50 con 0x5555d1dd54d0 2024-04-25T13:51:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.799+0000 7fdd6dffb640 1 -- 172.21.15.27:0/4029523591 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c0d0580 con 0x5555d1dd54d0 2024-04-25T13:51:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.800+0000 7fdd95796740 1 -- 172.21.15.27:0/4029523591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dd54d0 msgr2=0x5555d21a7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.800+0000 7fdd95796740 1 --2- 172.21.15.27:0/4029523591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dd54d0 0x5555d21a7ab0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c000c00 tx=0x7fdd7c02a410 comp rx=0 tx=0).stop 2024-04-25T13:51:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.800+0000 7fdd95796740 1 -- 172.21.15.27:0/4029523591 shutdown_connections 2024-04-25T13:51:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.800+0000 7fdd95796740 1 --2- 172.21.15.27:0/4029523591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d2195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.800+0000 7fdd95796740 1 --2- 172.21.15.27:0/4029523591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dd54d0 0x5555d21a7ab0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.800+0000 7fdd95796740 1 --2- 172.21.15.27:0/4029523591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dcd2f0 0x5555d219b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.800+0000 7fdd95796740 1 -- 172.21.15.27:0/4029523591 >> 172.21.15.27:0/4029523591 conn(0x5555d1dea300 msgr2=0x5555d2195100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.801+0000 7fdd95796740 1 -- 172.21.15.27:0/4029523591 shutdown_connections 2024-04-25T13:51:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.801+0000 7fdd95796740 1 -- 172.21.15.27:0/4029523591 wait complete. 2024-04-25T13:51:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.801+0000 7fdd95796740 1 -- start start 2024-04-25T13:51:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.801+0000 7fdd95796740 1 Processor -- start 2024-04-25T13:51:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd95796740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dcd2f0 0x5555d21a6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd95796740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dd54d0 0x5555d21a7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd95796740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d221ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd95796740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdd881ac100 con 0x5555d1df61b0 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd95796740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdd881ad570 con 0x5555d1dd54d0 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd95796740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdd881ac7d0 con 0x5555d1dcd2f0 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd95796740 10 client.0 _ll_register_callbacks cb 0x7fdd7800d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd8ce84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d221ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd877fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dd54d0 0x5555d21a7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd87fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dcd2f0 0x5555d21a6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd8ce84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d221ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38896/0 (socket says 172.21.15.27:38896) 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd8ce84640 1 -- 172.21.15.27:0/2154412130 learned_addr learned my addr 172.21.15.27:0/2154412130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd87fff640 1 -- 172.21.15.27:0/2154412130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dd54d0 msgr2=0x5555d21a7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd87fff640 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dd54d0 0x5555d21a7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd87fff640 1 -- 172.21.15.27:0/2154412130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 msgr2=0x5555d221ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd87fff640 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d221ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.802+0000 7fdd87fff640 1 -- 172.21.15.27:0/2154412130 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555d221d280 con 0x5555d1dcd2f0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd87fff640 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dcd2f0 0x5555d21a6d00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x5555d21a14b0 tx=0x7fdd7800f050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fbd678618bc2fc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd677fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd677fe640 1 -- 172.21.15.27:0/2154412130 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78020050 con 0x5555d1dcd2f0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd677fe640 1 -- 172.21.15.27:0/2154412130 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdd78004230 con 0x5555d1dcd2f0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 1 -- 172.21.15.27:0/2154412130 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd7c0cee50 con 0x5555d1dcd2f0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd677fe640 1 -- 172.21.15.27:0/2154412130 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdd78004430 con 0x5555d1dcd2f0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 10 client.27256 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 1 -- 172.21.15.27:0/2154412130 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5555d221e5e0 con 0x5555d1dcd2f0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd677fe640 1 -- 172.21.15.27:0/2154412130 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd7c0cee50 con 0x5555d1dcd2f0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd677fe640 1 -- 172.21.15.27:0/2154412130 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fdd7802c2d0 con 0x5555d1dcd2f0 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 10 client.27256 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 10 client.27256 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 20 client.27256 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5555d1d8a1c0 0x5555d2221020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 4 client.27256 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:51:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.803+0000 7fdd95796740 1 -- 172.21.15.27:0/2154412130 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5555d2103ee0 con 0x5555d1d8a1c0 2024-04-25T13:51:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.804+0000 7fdd877fe640 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5555d1d8a1c0 0x5555d2221020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.804+0000 7fdd677fe640 10 client.27256 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.804+0000 7fdd877fe640 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5555d1d8a1c0 0x5555d2221020 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9379b00c6645abee server_cookie=68f4b5008f5d478c in_seq=0 out_seq=0 2024-04-25T13:51:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd677fe640 1 -- 172.21.15.27:0/2154412130 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5555d2103ee0 con 0x5555d1d8a1c0 2024-04-25T13:51:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd677fe640 10 client.27256 handle_command_reply: tid=0 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 client.27256 shutdown 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 -- 172.21.15.27:0/2154412130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dcd2f0 msgr2=0x5555d21a6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dcd2f0 0x5555d21a6d00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x5555d21a14b0 tx=0x7fdd7800f050 comp rx=0 tx=0).stop 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 -- 172.21.15.27:0/2154412130 shutdown_connections 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5555d1d8a1c0 0x5555d2221020 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5555d1df61b0 0x5555d221ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5555d1dd54d0 0x5555d21a7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd677fe640 0 client.27256 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 --2- 172.21.15.27:0/2154412130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5555d1dcd2f0 0x5555d21a6d00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.819+0000 7fdd95796740 1 -- 172.21.15.27:0/2154412130 >> 172.21.15.27:0/2154412130 conn(0x5555d1dea300 msgr2=0x5555d219ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.820+0000 7fdd95796740 1 -- 172.21.15.27:0/2154412130 shutdown_connections 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.820+0000 7fdd95796740 1 -- 172.21.15.27:0/2154412130 wait complete. 2024-04-25T13:51:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.820+0000 7fdd95796740 20 client.27256 trim_cache size 0 max 16384 2024-04-25T13:51:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:51:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:51:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:51:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:51:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.820+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdd700afae0 msgr2=0x7fdd700b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.820+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdd700afae0 0x7fdd700b1ff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x5555d2197d50 tx=0x7fdd78005f90 comp rx=0 tx=0).stop 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.820+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd88103c60 msgr2=0x7fdd8819c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.820+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd88103c60 0x7fdd8819c9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c006010 tx=0x7fdd7c004270 comp rx=0 tx=0).stop 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.821+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 shutdown_connections 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.821+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdd700afae0 0x7fdd700b1ff0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.821+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd88108960 0x7fdd881a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.821+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd88103c60 0x7fdd8819c9a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.821+0000 7fdd8e8e7640 1 --2- 172.21.15.27:0/2052902754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd88102a20 0x7fdd8819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.821+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 >> 172.21.15.27:0/2052902754 conn(0x7fdd880fdc10 msgr2=0x7fdd88104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.821+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 shutdown_connections 2024-04-25T13:51:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:30.821+0000 7fdd8e8e7640 1 -- 172.21.15.27:0/2052902754 wait complete. 2024-04-25T13:51:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:32 smithi045 ceph-mon[32731]: pgmap v668: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 722 op/s 2024-04-25T13:51:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:32 smithi027 ceph-mon[29140]: pgmap v668: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 722 op/s 2024-04-25T13:51:32.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:32 smithi122 ceph-mon[32843]: pgmap v668: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 722 op/s 2024-04-25T13:51:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 625.0 s, 1541.6 tps, lat 65.105 ms stddev 72.484 2024-04-25T13:51:34.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:34 smithi045 ceph-mon[32731]: pgmap v669: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 8.2 MiB/s wr, 758 op/s 2024-04-25T13:51:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:34 smithi027 ceph-mon[29140]: pgmap v669: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 8.2 MiB/s wr, 758 op/s 2024-04-25T13:51:34.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:34 smithi122 ceph-mon[32843]: pgmap v669: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 8.2 MiB/s wr, 758 op/s 2024-04-25T13:51:36.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:36 smithi045 ceph-mon[32731]: pgmap v670: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 6.3 MiB/s wr, 590 op/s 2024-04-25T13:51:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:36 smithi027 ceph-mon[29140]: pgmap v670: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 6.3 MiB/s wr, 590 op/s 2024-04-25T13:51:36.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:36 smithi122 ceph-mon[32843]: pgmap v670: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 6.3 MiB/s wr, 590 op/s 2024-04-25T13:51:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 630.0 s, 1560.0 tps, lat 64.224 ms stddev 69.296 2024-04-25T13:51:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:38 smithi027 ceph-mon[29140]: pgmap v671: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-04-25T13:51:38.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:38 smithi122 ceph-mon[32843]: pgmap v671: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-04-25T13:51:38.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:38 smithi045 ceph-mon[32731]: pgmap v671: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-04-25T13:51:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:40 smithi027 ceph-mon[29140]: pgmap v672: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 5.4 MiB/s wr, 540 op/s 2024-04-25T13:51:40.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:40 smithi122 ceph-mon[32843]: pgmap v672: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 5.4 MiB/s wr, 540 op/s 2024-04-25T13:51:40.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:40 smithi045 ceph-mon[32731]: pgmap v672: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 5.4 MiB/s wr, 540 op/s 2024-04-25T13:51:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:42 smithi027 ceph-mon[29140]: pgmap v673: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.2 MiB/s wr, 606 op/s 2024-04-25T13:51:42.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:42 smithi122 ceph-mon[32843]: pgmap v673: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.2 MiB/s wr, 606 op/s 2024-04-25T13:51:42.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:42 smithi045 ceph-mon[32731]: pgmap v673: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.2 MiB/s wr, 606 op/s 2024-04-25T13:51:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 635.0 s, 1577.6 tps, lat 62.853 ms stddev 69.110 2024-04-25T13:51:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.972+0000 7fb5f77fe640 1 -- 172.21.15.27:0/1908793580 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec0318a0 con 0x7fb5f00958e0 2024-04-25T13:51:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.972+0000 7fb5ff201640 1 -- 172.21.15.27:0/1908793580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f00958e0 msgr2=0x7fb5f00965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.972+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1908793580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f00958e0 0x7fb5f00965e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009900 tx=0x7fb5ec02a410 comp rx=0 tx=0).stop 2024-04-25T13:51:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.973+0000 7fb5ff201640 1 -- 172.21.15.27:0/1908793580 shutdown_connections 2024-04-25T13:51:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.973+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1908793580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f009b820 0x7fb5f009dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.973+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1908793580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0096b20 0x7fb5f0096f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.973+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1908793580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f00958e0 0x7fb5f00965e0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.973+0000 7fb5ff201640 1 -- 172.21.15.27:0/1908793580 >> 172.21.15.27:0/1908793580 conn(0x7fb5f0090ab0 msgr2=0x7fb5f0092f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.974+0000 7fb5ff201640 1 -- 172.21.15.27:0/1908793580 shutdown_connections 2024-04-25T13:51:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.974+0000 7fb5ff201640 1 -- 172.21.15.27:0/1908793580 wait complete. 2024-04-25T13:51:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.975+0000 7fb5ff201640 1 Processor -- start 2024-04-25T13:51:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.975+0000 7fb5ff201640 1 -- start start 2024-04-25T13:51:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.976+0000 7fb5ff201640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f00958e0 0x7fb5f012f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.976+0000 7fb5ff201640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0096b20 0x7fb5f012f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.976+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f00958e0 0x7fb5f012f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.976+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f00958e0 0x7fb5f012f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51058/0 (socket says 172.21.15.27:51058) 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.976+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/1568680278 learned_addr learned my addr 172.21.15.27:0/1568680278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.976+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0096b20 0x7fb5f012f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.976+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f009b820 0x7fb5f01367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f00a4710 con 0x7fb5f009b820 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f00a4590 con 0x7fb5f0096b20 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f00a4890 con 0x7fb5f00958e0 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f009b820 0x7fb5f01367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5f7fff640 1 -- 172.21.15.27:0/1568680278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f00958e0 msgr2=0x7fb5f012f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f00958e0 0x7fb5f012f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5f7fff640 1 -- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f009b820 msgr2=0x7fb5f01367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f009b820 0x7fb5f01367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.977+0000 7fb5f7fff640 1 -- 172.21.15.27:0/1568680278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f0136e30 con 0x7fb5f0096b20 2024-04-25T13:51:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.978+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0096b20 0x7fb5f012f830 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4005fd0 tx=0x7fb5e400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.978+0000 7fb5f5ffb640 1 -- 172.21.15.27:0/1568680278 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4009c30 con 0x7fb5f0096b20 2024-04-25T13:51:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.978+0000 7fb5f5ffb640 1 -- 172.21.15.27:0/1568680278 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e4009dd0 con 0x7fb5f0096b20 2024-04-25T13:51:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.978+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f0137120 con 0x7fb5f0096b20 2024-04-25T13:51:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.979+0000 7fb5f5ffb640 1 -- 172.21.15.27:0/1568680278 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4010460 con 0x7fb5f0096b20 2024-04-25T13:51:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.979+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f0137600 con 0x7fb5f0096b20 2024-04-25T13:51:43.981 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.982+0000 7fb5f5ffb640 1 -- 172.21.15.27:0/1568680278 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb5e4004030 con 0x7fb5f0096b20 2024-04-25T13:51:43.982 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.983+0000 7fb60618a740 1 Processor -- start 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.983+0000 7fb60618a740 1 -- start start 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.983+0000 7fb60618a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ce1b0 0x5586eb677320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.983+0000 7fb60618a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6778a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.983+0000 7fb60618a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ad4d0 0x5586eb67e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.983+0000 7fb60618a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5586eb5d4460 con 0x5586eb2ce1b0 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.983+0000 7fb60618a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5586eb58e590 con 0x5586eb2ad4d0 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.983+0000 7fb60618a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5586eb5d0420 con 0x5586eb2a52f0 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5f7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ad4d0 0x5586eb67e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6778a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6778a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51064/0 (socket says 172.21.15.27:51064) 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/676100475 learned_addr learned my addr 172.21.15.27:0/676100475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/676100475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ce1b0 0x5586eb677320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5f5ffb640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5e80afb70 0x7fb5e80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5f7fff640 1 -- 172.21.15.27:0/676100475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 msgr2=0x5586eb6778a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/676100475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6778a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5f7fff640 1 -- 172.21.15.27:0/676100475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ce1b0 msgr2=0x5586eb677320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/676100475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ce1b0 0x5586eb677320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5f7fff640 1 -- 172.21.15.27:0/676100475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586eb59b0d0 con 0x5586eb2ad4d0 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5e80afb70 0x7fb5e80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.984+0000 7fb5f5ffb640 1 -- 172.21.15.27:0/1568680278 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5e40de6f0 con 0x7fb5f0096b20 2024-04-25T13:51:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.985+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/676100475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ad4d0 0x5586eb67e7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40dc270 tx=0x7fb5e4022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d99f92b41035f3bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.985+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5e80afb70 0x7fb5e80b2080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8068690 tx=0x7fb5f8068cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.985+0000 7fb5e2ffd640 1 -- 172.21.15.27:0/676100475 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4003e90 con 0x5586eb2ad4d0 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.985+0000 7fb5e2ffd640 1 -- 172.21.15.27:0/676100475 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e40af3a0 con 0x5586eb2ad4d0 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.985+0000 7fb5e2ffd640 1 -- 172.21.15.27:0/676100475 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4018d90 con 0x5586eb2ad4d0 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 -- 172.21.15.27:0/676100475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ad4d0 msgr2=0x5586eb67e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 --2- 172.21.15.27:0/676100475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ad4d0 0x5586eb67e7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40dc270 tx=0x7fb5e4022000 comp rx=0 tx=0).stop 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 -- 172.21.15.27:0/676100475 shutdown_connections 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 --2- 172.21.15.27:0/676100475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ad4d0 0x5586eb67e7a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 --2- 172.21.15.27:0/676100475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6778a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 --2- 172.21.15.27:0/676100475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ce1b0 0x5586eb677320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 -- 172.21.15.27:0/676100475 >> 172.21.15.27:0/676100475 conn(0x5586eb2c2300 msgr2=0x5586eb66a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 -- 172.21.15.27:0/676100475 shutdown_connections 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 -- 172.21.15.27:0/676100475 wait complete. 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 -- start start 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.986+0000 7fb60618a740 1 Processor -- start 2024-04-25T13:51:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb60618a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2a52f0 0x5586eb670930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb60618a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2ad4d0 0x5586eb670e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb60618a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb675dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2a52f0 0x5586eb670930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2a52f0 0x5586eb670930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34028/0 (socket says 172.21.15.27:34028) 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/4159528655 learned_addr learned my addr 172.21.15.27:0/4159528655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5586eb66eaa0 con 0x5586eb2a52f0 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5586eb55bb60 con 0x5586eb2ce1b0 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5586eb66ec20 con 0x5586eb2ad4d0 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2ad4d0 0x5586eb670e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb60618a740 10 client.0 _ll_register_callbacks cb 0x5586eb5802b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb675dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/4159528655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2ad4d0 msgr2=0x5586eb670e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2ad4d0 0x5586eb670e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 msgr2=0x5586eb675dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb675dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/4159528655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586eb2e6210 con 0x5586eb2a52f0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.987+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb675dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2ad4d0 0x5586eb670e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2a52f0 0x5586eb670930 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009230 tx=0x7fb5ec031f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea2833504207a95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb5d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb5d7fff640 1 -- 172.21.15.27:0/4159528655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec035040 con 0x5586eb2a52f0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb5d7fff640 1 -- 172.21.15.27:0/4159528655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5ec004240 con 0x5586eb2a52f0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5586eb5e2070 con 0x5586eb2a52f0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb5d7fff640 1 -- 172.21.15.27:0/4159528655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb5ec04d070 con 0x5586eb2a52f0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb60618a740 10 client.17523 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5586eb3b2620 con 0x5586eb2a52f0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.988+0000 7fb5d7fff640 1 -- 172.21.15.27:0/4159528655 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5586eb5e2070 con 0x5586eb2a52f0 2024-04-25T13:51:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb5d7fff640 1 -- 172.21.15.27:0/4159528655 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb5ec044020 con 0x5586eb2a52f0 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb60618a740 10 client.17523 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb60618a740 10 client.17523 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb60618a740 20 client.17523 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb60618a740 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5586eb28cd80 0x5586eb697460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb60618a740 4 client.17523 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586eb5dbee0 con 0x5586eb28cd80 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5586eb28cd80 0x5586eb697460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb5d7fff640 10 client.17523 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.989+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5586eb28cd80 0x5586eb697460 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=398ba13315dc7d5 server_cookie=757e96dfbdcdfd84 in_seq=0 out_seq=0 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb5d7fff640 1 -- 172.21.15.27:0/4159528655 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5586eb5dbee0 con 0x5586eb28cd80 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb5d7fff640 10 client.17523 handle_command_reply: tid=0 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 client.17523 shutdown 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2a52f0 msgr2=0x5586eb670930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2a52f0 0x5586eb670930 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009230 tx=0x7fb5ec031f10 comp rx=0 tx=0).stop 2024-04-25T13:51:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 shutdown_connections 2024-04-25T13:51:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5586eb28cd80 0x5586eb697460 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb5d7fff640 0 client.17523 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb675dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2ad4d0 0x5586eb670e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 --2- 172.21.15.27:0/4159528655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2a52f0 0x5586eb670930 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.990+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 >> 172.21.15.27:0/4159528655 conn(0x5586eb2c2300 msgr2=0x5586eb66d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.992+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 shutdown_connections 2024-04-25T13:51:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.993+0000 7fb60618a740 1 -- 172.21.15.27:0/4159528655 wait complete. 2024-04-25T13:51:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:43.993+0000 7fb60618a740 20 client.17523 trim_cache size 0 max 16384 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.011+0000 7fb60618a740 1 Processor -- start 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb60618a740 1 -- start start 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb60618a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb67bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb60618a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb66d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb60618a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb69cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb60618a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f0139f00 con 0x5586eb2ad4d0 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb67bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb67bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51084/0 (socket says 172.21.15.27:51084) 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/2293295803 learned_addr learned my addr 172.21.15.27:0/2293295803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb66d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb60618a740 1 -- 172.21.15.27:0/2293295803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f013c740 con 0x5586eb2ce1b0 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb60618a740 1 -- 172.21.15.27:0/2293295803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f013a550 con 0x5586eb2a52f0 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb69cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/2293295803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 msgr2=0x5586eb69cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb69cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/2293295803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 msgr2=0x5586eb66d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb66d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/2293295803 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586eb66def0 con 0x5586eb2a52f0 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb69cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.012+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb66d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.013+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb67bb30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0096c0 tx=0x7fb5ec02a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e416b9d38facadbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.013+0000 7fb5e27fc640 1 -- 172.21.15.27:0/2293295803 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec035070 con 0x5586eb2a52f0 2024-04-25T13:51:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.013+0000 7fb5e27fc640 1 -- 172.21.15.27:0/2293295803 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5ec057070 con 0x5586eb2a52f0 2024-04-25T13:51:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.013+0000 7fb5e27fc640 1 -- 172.21.15.27:0/2293295803 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec052d30 con 0x5586eb2a52f0 2024-04-25T13:51:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 -- 172.21.15.27:0/2293295803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 msgr2=0x5586eb67bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb67bb30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0096c0 tx=0x7fb5ec02a410 comp rx=0 tx=0).stop 2024-04-25T13:51:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 -- 172.21.15.27:0/2293295803 shutdown_connections 2024-04-25T13:51:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb69cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb66d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 --2- 172.21.15.27:0/2293295803 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb67bb30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 -- 172.21.15.27:0/2293295803 >> 172.21.15.27:0/2293295803 conn(0x5586eb2c2300 msgr2=0x5586eb66d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 -- 172.21.15.27:0/2293295803 shutdown_connections 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 -- 172.21.15.27:0/2293295803 wait complete. 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.014+0000 7fb60618a740 1 -- start start 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.015+0000 7fb60618a740 1 Processor -- start 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb60618a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6a7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb60618a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb6a79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb60618a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb6a7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb60618a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f013ac00 con 0x5586eb2ad4d0 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb6a79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5fcf9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6a7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5f7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb6a79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34044/0 (socket says 172.21.15.27:34044) 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f013c070 con 0x5586eb2ce1b0 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f013b2d0 con 0x5586eb2a52f0 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5f7fff640 1 -- 172.21.15.27:0/3298229810 learned_addr learned my addr 172.21.15.27:0/3298229810 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb60618a740 10 client.0 _ll_register_callbacks cb 0x7fb5ec005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5fd79e640 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb6a7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/3298229810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 msgr2=0x5586eb6a7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb6a7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/3298229810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 msgr2=0x5586eb6a79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb6a79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.016+0000 7fb5fcf9d640 1 -- 172.21.15.27:0/3298229810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586eb6fd470 con 0x5586eb2a52f0 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb6a79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb5fcf9d640 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6a7480 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec046520 tx=0x7fb5ec000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=150b2b1acc499e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb5d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3298229810 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec035040 con 0x5586eb2a52f0 2024-04-25T13:51:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3298229810 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5ec052680 con 0x5586eb2a52f0 2024-04-25T13:51:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5ec057070 con 0x5586eb2a52f0 2024-04-25T13:51:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3298229810 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb5ec0319c0 con 0x5586eb2a52f0 2024-04-25T13:51:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb60618a740 10 client.27274 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5586eb6fe5c0 con 0x5586eb2a52f0 2024-04-25T13:51:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.017+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3298229810 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5ec057070 con 0x5586eb2a52f0 2024-04-25T13:51:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3298229810 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb5ec044070 con 0x5586eb2a52f0 2024-04-25T13:51:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb60618a740 10 client.27274 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb60618a740 10 client.27274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb60618a740 20 client.27274 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb60618a740 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5586eb2621c0 0x5586eb700ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb60618a740 4 client.27274 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:51:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5586eb5575f0 con 0x5586eb2621c0 2024-04-25T13:51:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5586eb2621c0 0x5586eb700ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.018+0000 7fb5d7fff640 10 client.27274 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.019+0000 7fb5f7fff640 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5586eb2621c0 0x5586eb700ef0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb1639dd540cd5ce server_cookie=8a5f1ec9ac425c39 in_seq=0 out_seq=0 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb5d7fff640 1 -- 172.21.15.27:0/3298229810 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5586eb5575f0 con 0x5586eb2621c0 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb5d7fff640 10 client.27274 handle_command_reply: tid=0 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 client.27274 shutdown 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 msgr2=0x5586eb6a7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6a7480 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec046520 tx=0x7fb5ec000bc0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 shutdown_connections 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5586eb2621c0 0x5586eb700ef0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5586eb2ce1b0 0x5586eb6a7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5586eb2ad4d0 0x5586eb6a79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 --2- 172.21.15.27:0/3298229810 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5586eb2a52f0 0x5586eb6a7480 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 >> 172.21.15.27:0/3298229810 conn(0x5586eb2c2300 msgr2=0x5586eb66eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.032+0000 7fb5d7fff640 0 client.27274 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.033+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 shutdown_connections 2024-04-25T13:51:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.033+0000 7fb60618a740 1 -- 172.21.15.27:0/3298229810 wait complete. 2024-04-25T13:51:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.033+0000 7fb60618a740 20 client.27274 trim_cache size 0 max 16384 2024-04-25T13:51:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:51:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:51:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:51:44.034 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:51:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.033+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5e80afb70 msgr2=0x7fb5e80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.033+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5e80afb70 0x7fb5e80b2080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8068690 tx=0x7fb5f8068cb0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.033+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0096b20 msgr2=0x7fb5f012f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.033+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0096b20 0x7fb5f012f830 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4005fd0 tx=0x7fb5e400dde0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.034+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 shutdown_connections 2024-04-25T13:51:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.034+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5e80afb70 0x7fb5e80b2080 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.034+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f009b820 0x7fb5f01367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.034+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0096b20 0x7fb5f012f830 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.034+0000 7fb5ff201640 1 --2- 172.21.15.27:0/1568680278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f00958e0 0x7fb5f012f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.034+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 >> 172.21.15.27:0/1568680278 conn(0x7fb5f0090ab0 msgr2=0x7fb5f0097800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.034+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 shutdown_connections 2024-04-25T13:51:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:44.034+0000 7fb5ff201640 1 -- 172.21.15.27:0/1568680278 wait complete. 2024-04-25T13:51:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:44 smithi027 ceph-mon[29140]: pgmap v674: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.9 MiB/s wr, 661 op/s 2024-04-25T13:51:44.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:44 smithi122 ceph-mon[32843]: pgmap v674: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.9 MiB/s wr, 661 op/s 2024-04-25T13:51:44.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:44 smithi045 ceph-mon[32731]: pgmap v674: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.9 MiB/s wr, 661 op/s 2024-04-25T13:51:46.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:46 smithi122 ceph-mon[32843]: pgmap v675: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 MiB/s wr, 557 op/s 2024-04-25T13:51:46.380 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:46 smithi045 ceph-mon[32731]: pgmap v675: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 MiB/s wr, 557 op/s 2024-04-25T13:51:46.402 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:46 smithi027 ceph-mon[29140]: pgmap v675: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 MiB/s wr, 557 op/s 2024-04-25T13:51:47.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:51:47.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:51:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 640.0 s, 1950.0 tps, lat 51.754 ms stddev 64.279 2024-04-25T13:51:47.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.922+0000 7efea959a640 1 -- 172.21.15.27:0/2530611613 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea402f310 con 0x7efeac110030 2024-04-25T13:51:47.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 -- 172.21.15.27:0/2530611613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac110030 msgr2=0x7efeac110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 --2- 172.21.15.27:0/2530611613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac110030 0x7efeac110440 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efea4009ca0 tx=0x7efea402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 -- 172.21.15.27:0/2530611613 shutdown_connections 2024-04-25T13:51:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 --2- 172.21.15.27:0/2530611613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac110030 0x7efeac110440 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 --2- 172.21.15.27:0/2530611613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeac10b310 0x7efeac10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 --2- 172.21.15.27:0/2530611613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efeac10a420 0x7efeac10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 -- 172.21.15.27:0/2530611613 >> 172.21.15.27:0/2530611613 conn(0x7efeac0704e0 msgr2=0x7efeac072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 -- 172.21.15.27:0/2530611613 shutdown_connections 2024-04-25T13:51:47.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.923+0000 7efeb0c05640 1 -- 172.21.15.27:0/2530611613 wait complete. 2024-04-25T13:51:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeb0c05640 1 Processor -- start 2024-04-25T13:51:47.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeb0c05640 1 -- start start 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeb0c05640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac10a420 0x7efeac1a9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeb0c05640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeac10b310 0x7efeac1a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeb0c05640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efeac110030 0x7efeac1b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeb0c05640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efeac11e4e0 con 0x7efeac10b310 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeb0c05640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efeac11e360 con 0x7efeac110030 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeb0c05640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efeac11e660 con 0x7efeac10a420 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efea9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeac10b310 0x7efeac1a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efea9d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeac10b310 0x7efeac1a9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34062/0 (socket says 172.21.15.27:34062) 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efea9d9b640 1 -- 172.21.15.27:0/4036623289 learned_addr learned my addr 172.21.15.27:0/4036623289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeaa59c640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac10a420 0x7efeac1a9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeaad9d640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efeac110030 0x7efeac1b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efea9d9b640 1 -- 172.21.15.27:0/4036623289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac10a420 msgr2=0x7efeac1a9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:47.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efea9d9b640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac10a420 0x7efeac1a9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efea9d9b640 1 -- 172.21.15.27:0/4036623289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efeac110030 msgr2=0x7efeac1b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efea9d9b640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efeac110030 0x7efeac1b0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efea9d9b640 1 -- 172.21.15.27:0/4036623289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeac1b0b80 con 0x7efeac10b310 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeaa59c640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac10a420 0x7efeac1a9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.924+0000 7efeaad9d640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efeac110030 0x7efeac1b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.925+0000 7efea9d9b640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeac10b310 0x7efeac1a9580 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7efea000e9f0 tx=0x7efea000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.925+0000 7efe9b7fe640 1 -- 172.21.15.27:0/4036623289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea000cd80 con 0x7efeac10b310 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.925+0000 7efeb0c05640 1 -- 172.21.15.27:0/4036623289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeac1b0e70 con 0x7efeac10b310 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.925+0000 7efe9b7fe640 1 -- 172.21.15.27:0/4036623289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efea00108e0 con 0x7efeac10b310 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.925+0000 7efe9b7fe640 1 -- 172.21.15.27:0/4036623289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0018980 con 0x7efeac10b310 2024-04-25T13:51:47.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.925+0000 7efeb0c05640 1 -- 172.21.15.27:0/4036623289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeac1b1330 con 0x7efeac10b310 2024-04-25T13:51:47.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.928+0000 7efeb7b93740 1 Processor -- start 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeb7b93740 1 -- start start 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d99362390 0x561d99703a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d99703fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efe9b7fe640 1 -- 172.21.15.27:0/4036623289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7efea0018b20 con 0x7efeac10b310 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99704c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeb7b93740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561d99668d80 con 0x561d993394d0 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeb7b93740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561d996228b0 con 0x561d993416b0 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeb7b93740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561d99664d40 con 0x561d99362390 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeaad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d99362390 0x561d99703a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeaa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d99703fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeaad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d99362390 0x561d99703a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51118/0 (socket says 172.21.15.27:51118) 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.929+0000 7efeaad9d640 1 -- 172.21.15.27:0/2357711053 learned_addr learned my addr 172.21.15.27:0/2357711053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.930+0000 7efea9d9b640 1 --2- 172.21.15.27:0/2357711053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99704c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.930+0000 7efeaad9d640 1 -- 172.21.15.27:0/2357711053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 msgr2=0x561d99704c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.930+0000 7efeaad9d640 1 --2- 172.21.15.27:0/2357711053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99704c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.930+0000 7efeaad9d640 1 -- 172.21.15.27:0/2357711053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 msgr2=0x561d99703fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.930+0000 7efeaad9d640 1 --2- 172.21.15.27:0/2357711053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d99703fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.930+0000 7efeaad9d640 1 -- 172.21.15.27:0/2357711053 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d995f3a40 con 0x561d99362390 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.930+0000 7efe9b7fe640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efe940afb70 0x7efe940b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.930+0000 7efeaad9d640 1 --2- 172.21.15.27:0/2357711053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d99362390 0x561d99703a60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efea402f390 tx=0x7efea402f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8638e6d3ead0513 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.931+0000 7efe9b7fe640 1 -- 172.21.15.27:0/4036623289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efea00defa0 con 0x7efeac10b310 2024-04-25T13:51:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.932+0000 7efeaa59c640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efe940afb70 0x7efe940b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.932+0000 7efe98ff9640 1 -- 172.21.15.27:0/2357711053 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4031650 con 0x561d99362390 2024-04-25T13:51:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.933+0000 7efe98ff9640 1 -- 172.21.15.27:0/2357711053 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efea4036040 con 0x561d99362390 2024-04-25T13:51:47.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.933+0000 7efe98ff9640 1 -- 172.21.15.27:0/2357711053 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4031650 con 0x561d99362390 2024-04-25T13:51:47.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 -- 172.21.15.27:0/2357711053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d99362390 msgr2=0x561d99703a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:47.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeaa59c640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efe940afb70 0x7efe940b2080 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x561d99703780 tx=0x7efe9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:47.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 --2- 172.21.15.27:0/2357711053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d99362390 0x561d99703a60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efea402f390 tx=0x7efea402f3c0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 -- 172.21.15.27:0/2357711053 shutdown_connections 2024-04-25T13:51:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 --2- 172.21.15.27:0/2357711053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99704c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 --2- 172.21.15.27:0/2357711053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d99703fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 --2- 172.21.15.27:0/2357711053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d99362390 0x561d99703a60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 -- 172.21.15.27:0/2357711053 >> 172.21.15.27:0/2357711053 conn(0x561d993564e0 msgr2=0x561d996ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 -- 172.21.15.27:0/2357711053 shutdown_connections 2024-04-25T13:51:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.934+0000 7efeb7b93740 1 -- 172.21.15.27:0/2357711053 wait complete. 2024-04-25T13:51:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.935+0000 7efeb7b93740 1 -- start start 2024-04-25T13:51:47.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.935+0000 7efeb7b93740 1 Processor -- start 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.935+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d9970d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.935+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 0x561d9970ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.935+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d99362390 0x561d99714dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d9970d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeb7b93740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561d99705720 con 0x561d993394d0 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeb7b93740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561d995efd10 con 0x561d99362390 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeb7b93740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561d997058a0 con 0x561d993416b0 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d9970d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34084/0 (socket says 172.21.15.27:34084) 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 -- 172.21.15.27:0/3244367248 learned_addr learned my addr 172.21.15.27:0/3244367248 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeb7b93740 10 client.0 _ll_register_callbacks cb 0x561d9961c210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaad9d640 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d99362390 0x561d99714dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efea9d9b640 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 0x561d9970ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 -- 172.21.15.27:0/3244367248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 msgr2=0x561d9970ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 0x561d9970ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 -- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d99362390 msgr2=0x561d99714dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d99362390 0x561d99714dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 -- 172.21.15.27:0/3244367248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d994447d0 con 0x561d993394d0 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.936+0000 7efeaa59c640 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d9970d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efe9c004620 tx=0x7efe9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd207ff2cce70a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.937+0000 7efe91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.937+0000 7efeaad9d640 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d99362390 0x561d99714dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.937+0000 7efe91ffb640 1 -- 172.21.15.27:0/3244367248 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00f830 con 0x561d993394d0 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.937+0000 7efe91ffb640 1 -- 172.21.15.27:0/3244367248 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efe9c00f9d0 con 0x561d993394d0 2024-04-25T13:51:47.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.937+0000 7efeb7b93740 1 -- 172.21.15.27:0/3244367248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efea4036040 con 0x561d993394d0 2024-04-25T13:51:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efe91ffb640 1 -- 172.21.15.27:0/3244367248 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7efe9c00fbd0 con 0x561d993394d0 2024-04-25T13:51:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efeb7b93740 10 client.17541 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efeb7b93740 1 -- 172.21.15.27:0/3244367248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x561d9937a3f0 con 0x561d993394d0 2024-04-25T13:51:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efe91ffb640 1 -- 172.21.15.27:0/3244367248 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efea4036040 con 0x561d993394d0 2024-04-25T13:51:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efe91ffb640 1 -- 172.21.15.27:0/3244367248 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7efe9c027070 con 0x561d993394d0 2024-04-25T13:51:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efeb7b93740 10 client.17541 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efeb7b93740 10 client.17541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efeb7b93740 20 client.17541 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efeb7b93740 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561d99320f60 0x561d9971b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efeb7b93740 4 client.17541 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.938+0000 7efeb7b93740 1 -- 172.21.15.27:0/3244367248 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d99676990 con 0x561d99320f60 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.939+0000 7efea9d9b640 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561d99320f60 0x561d9971b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.939+0000 7efe91ffb640 10 client.17541 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.939+0000 7efea9d9b640 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561d99320f60 0x561d9971b2b0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=953f993fafd07123 server_cookie=c06bbdb1c139ffac in_seq=0 out_seq=0 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efe91ffb640 1 -- 172.21.15.27:0/3244367248 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561d99676990 con 0x561d99320f60 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efe91ffb640 10 client.17541 handle_command_reply: tid=0 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 client.17541 shutdown 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 -- 172.21.15.27:0/3244367248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 msgr2=0x561d9970d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d9970d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efe9c004620 tx=0x7efe9c004730 comp rx=0 tx=0).stop 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 -- 172.21.15.27:0/3244367248 shutdown_connections 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561d99320f60 0x561d9971b2b0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d99362390 0x561d99714dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 0x561d9970ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 --2- 172.21.15.27:0/3244367248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d993394d0 0x561d9970d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 -- 172.21.15.27:0/3244367248 >> 172.21.15.27:0/3244367248 conn(0x561d993564e0 msgr2=0x561d99702180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efe91ffb640 0 client.17541 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 -- 172.21.15.27:0/3244367248 shutdown_connections 2024-04-25T13:51:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 1 -- 172.21.15.27:0/3244367248 wait complete. 2024-04-25T13:51:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.940+0000 7efeb7b93740 20 client.17541 trim_cache size 0 max 16384 2024-04-25T13:51:47.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.979+0000 7efeb7b93740 1 Processor -- start 2024-04-25T13:51:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.982+0000 7efeb7b93740 1 -- start start 2024-04-25T13:51:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.983+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993394d0 0x561d9970d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.983+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99714d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.983+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 0x561d996fec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.984+0000 7efea9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99714d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.984+0000 7efea9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99714d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52906/0 (socket says 172.21.15.27:52906) 2024-04-25T13:51:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.984+0000 7efeb7b93740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b80e0 con 0x561d99362390 2024-04-25T13:51:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.984+0000 7efea9d9b640 1 -- 172.21.15.27:0/1024859308 learned_addr learned my addr 172.21.15.27:0/1024859308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.984+0000 7efeb7b93740 1 -- 172.21.15.27:0/1024859308 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1ba960 con 0x561d993416b0 2024-04-25T13:51:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.984+0000 7efeb7b93740 1 -- 172.21.15.27:0/1024859308 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b8750 con 0x561d993394d0 2024-04-25T13:51:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.984+0000 7efeaa59c640 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993394d0 0x561d9970d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.984+0000 7efeaad9d640 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 0x561d996fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.985+0000 7efeaa59c640 1 -- 172.21.15.27:0/1024859308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 msgr2=0x561d99714d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.985+0000 7efeaa59c640 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99714d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.985+0000 7efeaa59c640 1 -- 172.21.15.27:0/1024859308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 msgr2=0x561d996fec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.985+0000 7efeaa59c640 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 0x561d996fec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.985+0000 7efeaa59c640 1 -- 172.21.15.27:0/1024859308 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d9933a0f0 con 0x561d993394d0 2024-04-25T13:51:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.986+0000 7efeaa59c640 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993394d0 0x561d9970d930 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe9c029600 tx=0x7efe9c005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3245fd9cd9035d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.986+0000 7efe93fff640 1 -- 172.21.15.27:0/1024859308 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00f830 con 0x561d993394d0 2024-04-25T13:51:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.986+0000 7efe93fff640 1 -- 172.21.15.27:0/1024859308 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe9c038020 con 0x561d993394d0 2024-04-25T13:51:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.986+0000 7efe93fff640 1 -- 172.21.15.27:0/1024859308 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c005520 con 0x561d993394d0 2024-04-25T13:51:47.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.989+0000 7efeb7b93740 1 -- 172.21.15.27:0/1024859308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993394d0 msgr2=0x561d9970d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:47.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.989+0000 7efeb7b93740 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993394d0 0x561d9970d930 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe9c029600 tx=0x7efe9c005bd0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.994+0000 7efeb7b93740 1 -- 172.21.15.27:0/1024859308 shutdown_connections 2024-04-25T13:51:47.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.994+0000 7efeb7b93740 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 0x561d996fec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.994+0000 7efeb7b93740 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993416b0 0x561d99714d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.994+0000 7efeb7b93740 1 --2- 172.21.15.27:0/1024859308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993394d0 0x561d9970d930 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:47.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.994+0000 7efeb7b93740 1 -- 172.21.15.27:0/1024859308 >> 172.21.15.27:0/1024859308 conn(0x561d993564e0 msgr2=0x561d99708780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:47.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:47.998+0000 7efeb7b93740 1 -- 172.21.15.27:0/1024859308 shutdown_connections 2024-04-25T13:51:48.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.002+0000 7efeb7b93740 1 -- 172.21.15.27:0/1024859308 wait complete. 2024-04-25T13:51:48.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.002+0000 7efeb7b93740 1 -- start start 2024-04-25T13:51:48.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.002+0000 7efeb7b93740 1 Processor -- start 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.002+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993394d0 0x561d99714d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 0x561d997118d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeb7b93740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 0x561d99711f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeb7b93740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b8e20 con 0x561d99362390 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeb7b93740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1ba290 con 0x561d993394d0 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeb7b93740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b94f0 con 0x561d993416b0 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeb7b93740 10 client.0 _ll_register_callbacks cb 0x7efe9c009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeaa59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993394d0 0x561d99714d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeaa59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993394d0 0x561d99714d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52920/0 (socket says 172.21.15.27:52920) 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeaa59c640 1 -- 172.21.15.27:0/416124319 learned_addr learned my addr 172.21.15.27:0/416124319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:48.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.003+0000 7efeaad9d640 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 0x561d99711f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.004+0000 7efeaa59c640 1 -- 172.21.15.27:0/416124319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 msgr2=0x561d997118d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:51:48.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.004+0000 7efeaa59c640 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 0x561d997118d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.004+0000 7efeaa59c640 1 -- 172.21.15.27:0/416124319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 msgr2=0x561d99711f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:48.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.004+0000 7efeaa59c640 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 0x561d99711f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.004+0000 7efeaa59c640 1 -- 172.21.15.27:0/416124319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d996294b0 con 0x561d993394d0 2024-04-25T13:51:48.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.005+0000 7efeaa59c640 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993394d0 0x561d99714d00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe9c029490 tx=0x7efe9c0298b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e55cb0193aeb6fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.006+0000 7efe91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:51:48.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.006+0000 7efe91ffb640 1 -- 172.21.15.27:0/416124319 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c041060 con 0x561d993394d0 2024-04-25T13:51:48.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.006+0000 7efe91ffb640 1 -- 172.21.15.27:0/416124319 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7efe9c029d30 con 0x561d993394d0 2024-04-25T13:51:48.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.006+0000 7efeb7b93740 1 -- 172.21.15.27:0/416124319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe9c038020 con 0x561d993394d0 2024-04-25T13:51:48.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.007+0000 7efe91ffb640 1 -- 172.21.15.27:0/416124319 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7efe9c0052a0 con 0x561d993394d0 2024-04-25T13:51:48.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.007+0000 7efeb7b93740 10 client.27245 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:48.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.007+0000 7efeb7b93740 1 -- 172.21.15.27:0/416124319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x561d99782700 con 0x561d993394d0 2024-04-25T13:51:48.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.008+0000 7efe91ffb640 1 -- 172.21.15.27:0/416124319 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe9c038020 con 0x561d993394d0 2024-04-25T13:51:48.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.008+0000 7efe91ffb640 1 -- 172.21.15.27:0/416124319 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7efe9c027040 con 0x561d993394d0 2024-04-25T13:51:48.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.008+0000 7efeb7b93740 10 client.27245 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.008+0000 7efeb7b93740 10 client.27245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.008+0000 7efeb7b93740 20 client.27245 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.008+0000 7efeb7b93740 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561d992f63a0 0x561d99785020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.008+0000 7efeb7b93740 4 client.27245 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:51:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.008+0000 7efeb7b93740 1 -- 172.21.15.27:0/416124319 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561d99670800 con 0x561d992f63a0 2024-04-25T13:51:48.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.009+0000 7efea9d9b640 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561d992f63a0 0x561d99785020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.009+0000 7efe91ffb640 10 client.27245 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.009+0000 7efea9d9b640 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561d992f63a0 0x561d99785020 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c761f54ac2ef303 server_cookie=9da4d9d7ed22edaf in_seq=0 out_seq=0 2024-04-25T13:51:48.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efe91ffb640 1 -- 172.21.15.27:0/416124319 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561d99670800 con 0x561d992f63a0 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efe91ffb640 10 client.27245 handle_command_reply: tid=0 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 client.27245 shutdown 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 -- 172.21.15.27:0/416124319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993394d0 msgr2=0x561d99714d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993394d0 0x561d99714d00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe9c029490 tx=0x7efe9c0298b0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 -- 172.21.15.27:0/416124319 shutdown_connections 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561d992f63a0 0x561d99785020 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561d99362390 0x561d99711f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561d993416b0 0x561d997118d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 --2- 172.21.15.27:0/416124319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561d993394d0 0x561d99714d00 secure :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe9c029490 tx=0x7efe9c0298b0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 -- 172.21.15.27:0/416124319 >> 172.21.15.27:0/416124319 conn(0x561d993564e0 msgr2=0x561d9970d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efe91ffb640 0 client.27245 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 -- 172.21.15.27:0/416124319 shutdown_connections 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 1 -- 172.21.15.27:0/416124319 wait complete. 2024-04-25T13:51:48.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.010+0000 7efeb7b93740 20 client.27245 trim_cache size 0 max 16384 2024-04-25T13:51:48.012 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:51:48.012 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:51:48.012 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:51:48.012 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 -- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efe940afb70 msgr2=0x7efe940b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efe940afb70 0x7efe940b2080 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x561d99703780 tx=0x7efe9c005f90 comp rx=0 tx=0).stop 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 -- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeac10b310 msgr2=0x7efeac1a9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeac10b310 0x7efeac1a9580 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7efea000e9f0 tx=0x7efea000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 -- 172.21.15.27:0/4036623289 shutdown_connections 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7efe940afb70 0x7efe940b2080 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efeac110030 0x7efeac1b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efeac10b310 0x7efeac1a9580 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 --2- 172.21.15.27:0/4036623289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efeac10a420 0x7efeac1a9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 -- 172.21.15.27:0/4036623289 >> 172.21.15.27:0/4036623289 conn(0x7efeac0704e0 msgr2=0x7efeac10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 -- 172.21.15.27:0/4036623289 shutdown_connections 2024-04-25T13:51:48.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.011+0000 7efeb0c05640 1 -- 172.21.15.27:0/4036623289 wait complete. 2024-04-25T13:51:48.028 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:51:48.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:51:48.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.204+0000 7ff2698fa640 1 -- 172.21.15.27:0/3965155116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264110030 msgr2=0x7ff264110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.204+0000 7ff2698fa640 1 --2- 172.21.15.27:0/3965155116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264110030 0x7ff264110440 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff254009ca0 tx=0x7ff25402a8d0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.211+0000 7ff2698fa640 1 -- 172.21.15.27:0/3965155116 shutdown_connections 2024-04-25T13:51:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.211+0000 7ff2698fa640 1 --2- 172.21.15.27:0/3965155116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264110030 0x7ff264110440 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.211+0000 7ff2698fa640 1 --2- 172.21.15.27:0/3965155116 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff26410b310 0x7ff26410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.211+0000 7ff2698fa640 1 --2- 172.21.15.27:0/3965155116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff26410a420 0x7ff26410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.211+0000 7ff2698fa640 1 -- 172.21.15.27:0/3965155116 >> 172.21.15.27:0/3965155116 conn(0x7ff2640704e0 msgr2=0x7ff264072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.217+0000 7ff2698fa640 1 -- 172.21.15.27:0/3965155116 shutdown_connections 2024-04-25T13:51:48.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.219+0000 7ff2698fa640 1 -- 172.21.15.27:0/3965155116 wait complete. 2024-04-25T13:51:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.220+0000 7ff2698fa640 1 Processor -- start 2024-04-25T13:51:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.221+0000 7ff2698fa640 1 -- start start 2024-04-25T13:51:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.221+0000 7ff2698fa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff26410a420 0x7ff2641a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.221+0000 7ff2698fa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff26410b310 0x7ff2641a9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.221+0000 7ff2698fa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264110030 0x7ff2641b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.221+0000 7ff2698fa640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff26411e510 con 0x7ff264110030 2024-04-25T13:51:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.221+0000 7ff2698fa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff26411e390 con 0x7ff26410a420 2024-04-25T13:51:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.221+0000 7ff2698fa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff26411e690 con 0x7ff26410b310 2024-04-25T13:51:48.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.222+0000 7ff2627fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff26410b310 0x7ff2641a9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.222+0000 7ff2627fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff26410b310 0x7ff2641a9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51146/0 (socket says 172.21.15.27:51146) 2024-04-25T13:51:48.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.222+0000 7ff2627fc640 1 -- 172.21.15.27:0/151485276 learned_addr learned my addr 172.21.15.27:0/151485276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.224+0000 7ff2637fe640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264110030 0x7ff2641b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.225+0000 7ff262ffd640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff26410a420 0x7ff2641a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.225+0000 7ff2627fc640 1 -- 172.21.15.27:0/151485276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff26410a420 msgr2=0x7ff2641a90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.225+0000 7ff2627fc640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff26410a420 0x7ff2641a90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.225+0000 7ff2627fc640 1 -- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264110030 msgr2=0x7ff2641b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.225+0000 7ff2627fc640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264110030 0x7ff2641b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.225+0000 7ff2627fc640 1 -- 172.21.15.27:0/151485276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2641b0be0 con 0x7ff26410b310 2024-04-25T13:51:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.228+0000 7ff2627fc640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff26410b310 0x7ff2641a9630 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff24c02d4a0 tx=0x7ff24c02d960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.230+0000 7ff2688f8640 1 -- 172.21.15.27:0/151485276 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c036070 con 0x7ff26410b310 2024-04-25T13:51:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.230+0000 7ff2688f8640 1 -- 172.21.15.27:0/151485276 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff24c032ac0 con 0x7ff26410b310 2024-04-25T13:51:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.230+0000 7ff2688f8640 1 -- 172.21.15.27:0/151485276 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c039c50 con 0x7ff26410b310 2024-04-25T13:51:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.230+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff264074e30 con 0x7ff26410b310 2024-04-25T13:51:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.230+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2640753b0 con 0x7ff26410b310 2024-04-25T13:51:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.233+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2440052c0 con 0x7ff26410b310 2024-04-25T13:51:48.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.238+0000 7ff2688f8640 1 -- 172.21.15.27:0/151485276 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff24c039070 con 0x7ff26410b310 2024-04-25T13:51:48.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.239+0000 7ff2688f8640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2500afd00 0x7ff2500b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.239+0000 7ff2688f8640 1 -- 172.21.15.27:0/151485276 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff24c100290 con 0x7ff26410b310 2024-04-25T13:51:48.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.242+0000 7ff262ffd640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2500afd00 0x7ff2500b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.245+0000 7ff2688f8640 1 -- 172.21.15.27:0/151485276 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff24c0c8e40 con 0x7ff26410b310 2024-04-25T13:51:48.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.246+0000 7ff262ffd640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2500afd00 0x7ff2500b2210 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0058c0 tx=0x7ff25c005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:48 smithi027 ceph-mon[29140]: pgmap v676: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.5 MiB/s wr, 656 op/s 2024-04-25T13:51:48.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:48 smithi122 ceph-mon[32843]: pgmap v676: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.5 MiB/s wr, 656 op/s 2024-04-25T13:51:48.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:48 smithi045 ceph-mon[32731]: pgmap v676: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.5 MiB/s wr, 656 op/s 2024-04-25T13:51:48.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.622+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2440055b0 con 0x7ff26410b310 2024-04-25T13:51:48.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.623+0000 7ff2688f8640 1 -- 172.21.15.27:0/151485276 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7ff24c0c8740 con 0x7ff26410b310 2024-04-25T13:51:48.623 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:51:48.623 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:51:48.623 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:51:48.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2500afd00 msgr2=0x7ff2500b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2500afd00 0x7ff2500b2210 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0058c0 tx=0x7ff25c005c60 comp rx=0 tx=0).stop 2024-04-25T13:51:48.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff26410b310 msgr2=0x7ff2641a9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff26410b310 0x7ff2641a9630 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff24c02d4a0 tx=0x7ff24c02d960 comp rx=0 tx=0).stop 2024-04-25T13:51:48.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 shutdown_connections 2024-04-25T13:51:48.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff2500afd00 0x7ff2500b2210 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff264110030 0x7ff2641b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff26410b310 0x7ff2641a9630 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 --2- 172.21.15.27:0/151485276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff26410a420 0x7ff2641a90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 >> 172.21.15.27:0/151485276 conn(0x7ff2640704e0 msgr2=0x7ff26410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 shutdown_connections 2024-04-25T13:51:48.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.625+0000 7ff2698fa640 1 -- 172.21.15.27:0/151485276 wait complete. 2024-04-25T13:51:48.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:51:48.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.828+0000 7f94c3884640 1 -- 172.21.15.27:0/1208996779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc110030 msgr2=0x7f94bc110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.828+0000 7f94c3884640 1 --2- 172.21.15.27:0/1208996779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc110030 0x7f94bc110440 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009900 tx=0x7f94b002a420 comp rx=0 tx=0).stop 2024-04-25T13:51:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.835+0000 7f94c3884640 1 -- 172.21.15.27:0/1208996779 shutdown_connections 2024-04-25T13:51:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.835+0000 7f94c3884640 1 --2- 172.21.15.27:0/1208996779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc110030 0x7f94bc110440 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.835+0000 7f94c3884640 1 --2- 172.21.15.27:0/1208996779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94bc10b310 0x7f94bc10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.835+0000 7f94c3884640 1 --2- 172.21.15.27:0/1208996779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc10a420 0x7f94bc10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.835+0000 7f94c3884640 1 -- 172.21.15.27:0/1208996779 >> 172.21.15.27:0/1208996779 conn(0x7f94bc0704e0 msgr2=0x7f94bc072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.841+0000 7f94c3884640 1 -- 172.21.15.27:0/1208996779 shutdown_connections 2024-04-25T13:51:48.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.841+0000 7f94c3884640 1 -- 172.21.15.27:0/1208996779 wait complete. 2024-04-25T13:51:48.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.842+0000 7f94c3884640 1 Processor -- start 2024-04-25T13:51:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.845+0000 7f94c3884640 1 -- start start 2024-04-25T13:51:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.845+0000 7f94c3884640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc10a420 0x7f94bc116f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.845+0000 7f94c3884640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94bc10b310 0x7f94bc1174b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.845+0000 7f94c3884640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc110030 0x7f94bc118410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.845+0000 7f94c3884640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc11e560 con 0x7f94bc10a420 2024-04-25T13:51:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.845+0000 7f94c3884640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc11e3e0 con 0x7f94bc110030 2024-04-25T13:51:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.845+0000 7f94c3884640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc11e6e0 con 0x7f94bc10b310 2024-04-25T13:51:48.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.847+0000 7f94c1620640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc10a420 0x7f94bc116f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.847+0000 7f94c1620640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc10a420 0x7f94bc116f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34138/0 (socket says 172.21.15.27:34138) 2024-04-25T13:51:48.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.847+0000 7f94c1620640 1 -- 172.21.15.27:0/2615295226 learned_addr learned my addr 172.21.15.27:0/2615295226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:48.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.848+0000 7f94c1e21640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc110030 0x7f94bc118410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.849+0000 7f94c1620640 1 -- 172.21.15.27:0/2615295226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94bc10b310 msgr2=0x7f94bc1174b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.849+0000 7f94c1620640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94bc10b310 0x7f94bc1174b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.849+0000 7f94c1620640 1 -- 172.21.15.27:0/2615295226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc110030 msgr2=0x7f94bc118410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.849+0000 7f94c1620640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc110030 0x7f94bc118410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.849+0000 7f94c1620640 1 -- 172.21.15.27:0/2615295226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc1189d0 con 0x7f94bc10a420 2024-04-25T13:51:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.850+0000 7f94c1620640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc10a420 0x7f94bc116f70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f94b400ccb0 tx=0x7f94b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.850+0000 7f94c1e21640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc110030 0x7f94bc118410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:51:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.850+0000 7f94ba7fc640 1 -- 172.21.15.27:0/2615295226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b400ce60 con 0x7f94bc10a420 2024-04-25T13:51:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.850+0000 7f94ba7fc640 1 -- 172.21.15.27:0/2615295226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94b400fb00 con 0x7f94bc10a420 2024-04-25T13:51:48.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.850+0000 7f94ba7fc640 1 -- 172.21.15.27:0/2615295226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b400eb30 con 0x7f94bc10a420 2024-04-25T13:51:48.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.851+0000 7f94c3884640 1 -- 172.21.15.27:0/2615295226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc118cc0 con 0x7f94bc10a420 2024-04-25T13:51:48.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.851+0000 7f94c3884640 1 -- 172.21.15.27:0/2615295226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc1191c0 con 0x7f94bc10a420 2024-04-25T13:51:48.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.855+0000 7f94ca807740 1 Processor -- start 2024-04-25T13:51:48.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.855+0000 7f94ca807740 1 -- start start 2024-04-25T13:51:48.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94ca807740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c23390 0x5648b4fc5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94ca807740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94c1e21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c23390 0x5648b4fc5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94c1e21640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c23390 0x5648b4fc5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34148/0 (socket says 172.21.15.27:34148) 2024-04-25T13:51:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94ca807740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c026b0 0x5648b4fc6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94c1e21640 1 -- 172.21.15.27:0/982935901 learned_addr learned my addr 172.21.15.27:0/982935901 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94ca807740 1 -- 172.21.15.27:0/982935901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5648b4f29d80 con 0x5648b4c23390 2024-04-25T13:51:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94ca807740 1 -- 172.21.15.27:0/982935901 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5648b4ee3a20 con 0x5648b4c026b0 2024-04-25T13:51:48.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.856+0000 7f94ca807740 1 -- 172.21.15.27:0/982935901 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5648b4f25d40 con 0x5648b4bfa4d0 2024-04-25T13:51:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.857+0000 7f94ba7fc640 1 -- 172.21.15.27:0/2615295226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f94b400ecd0 con 0x7f94bc10a420 2024-04-25T13:51:48.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.857+0000 7f94c1620640 1 --2- 172.21.15.27:0/982935901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.858+0000 7f94c1620640 1 -- 172.21.15.27:0/982935901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c026b0 msgr2=0x5648b4fc6a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.858+0000 7f94c1620640 1 --2- 172.21.15.27:0/982935901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c026b0 0x5648b4fc6a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.858+0000 7f94c1620640 1 -- 172.21.15.27:0/982935901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c23390 msgr2=0x5648b4fc5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.858+0000 7f94c1620640 1 --2- 172.21.15.27:0/982935901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c23390 0x5648b4fc5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.858+0000 7f94c1620640 1 -- 172.21.15.27:0/982935901 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648b4eeec30 con 0x5648b4bfa4d0 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.859+0000 7f94c1620640 1 --2- 172.21.15.27:0/982935901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc5d90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f94b40dcf10 tx=0x7f94b40e7860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8b2aad63114a4af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.859+0000 7f94ba7fc640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94a40afcf0 0x7f94a40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.859+0000 7f94a37fe640 1 -- 172.21.15.27:0/982935901 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b40a3f70 con 0x5648b4bfa4d0 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.859+0000 7f94a37fe640 1 -- 172.21.15.27:0/982935901 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94b40a8020 con 0x5648b4bfa4d0 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.859+0000 7f94a37fe640 1 -- 172.21.15.27:0/982935901 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b40a7090 con 0x5648b4bfa4d0 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.859+0000 7f94c1e21640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94a40afcf0 0x7f94a40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.859+0000 7f94ba7fc640 1 -- 172.21.15.27:0/2615295226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f94b40df470 con 0x7f94bc10a420 2024-04-25T13:51:48.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.860+0000 7f94ca807740 1 -- 172.21.15.27:0/982935901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 msgr2=0x5648b4fc5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.860+0000 7f94ca807740 1 --2- 172.21.15.27:0/982935901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc5d90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f94b40dcf10 tx=0x7f94b40e7860 comp rx=0 tx=0).stop 2024-04-25T13:51:48.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.860+0000 7f94c1e21640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94a40afcf0 0x7f94a40b2200 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f94b0031500 tx=0x7f94b0009560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.861+0000 7f94ca807740 1 -- 172.21.15.27:0/982935901 shutdown_connections 2024-04-25T13:51:48.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.861+0000 7f94ca807740 1 --2- 172.21.15.27:0/982935901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c026b0 0x5648b4fc6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.861+0000 7f94ca807740 1 --2- 172.21.15.27:0/982935901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc5d90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.861+0000 7f94ca807740 1 --2- 172.21.15.27:0/982935901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c23390 0x5648b4fc5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.861+0000 7f94ca807740 1 -- 172.21.15.27:0/982935901 >> 172.21.15.27:0/982935901 conn(0x5648b4c174e0 msgr2=0x5648b4fc1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.864+0000 7f94ca807740 1 -- 172.21.15.27:0/982935901 shutdown_connections 2024-04-25T13:51:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.865+0000 7f94ca807740 1 -- 172.21.15.27:0/982935901 wait complete. 2024-04-25T13:51:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.866+0000 7f94ca807740 1 -- start start 2024-04-25T13:51:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.866+0000 7f94ca807740 1 Processor -- start 2024-04-25T13:51:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.867+0000 7f94ca807740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fcf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.867+0000 7f94ca807740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fcfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.867+0000 7f94c1620640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fcf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.867+0000 7f94c1620640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fcf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34158/0 (socket says 172.21.15.27:34158) 2024-04-25T13:51:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.867+0000 7f94c1620640 1 -- 172.21.15.27:0/1419400029 learned_addr learned my addr 172.21.15.27:0/1419400029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.867+0000 7f94ca807740 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fd6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.867+0000 7f94c0e1f640 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fcfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5648b4fc7530 con 0x5648b4bfa4d0 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5648b4eb0dc0 con 0x5648b4c23390 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5648b4fc76b0 con 0x5648b4c026b0 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94c1e21640 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fd6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94c1620640 1 -- 172.21.15.27:0/1419400029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 msgr2=0x5648b4fcfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94c1620640 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fcfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94c1620640 1 -- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 msgr2=0x5648b4fd6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94c1620640 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fd6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.868+0000 7f94c1620640 1 -- 172.21.15.27:0/1419400029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648b4c3b3f0 con 0x5648b4bfa4d0 2024-04-25T13:51:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.869+0000 7f94ca807740 10 client.0 _ll_register_callbacks cb 0x5648b4ecc600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.869+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b40a8020 con 0x5648b4bfa4d0 2024-04-25T13:51:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.870+0000 7f94c1620640 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fcf570 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f94b400afd0 tx=0x7f94b40df6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358e27bdb96e7128 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.870+0000 7f94a0ff9640 10 client.17565 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.870+0000 7f94a0ff9640 1 -- 172.21.15.27:0/1419400029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b40a7270 con 0x5648b4bfa4d0 2024-04-25T13:51:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.870+0000 7f94a0ff9640 1 -- 172.21.15.27:0/1419400029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f94b400e3b0 con 0x5648b4bfa4d0 2024-04-25T13:51:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.870+0000 7f94a0ff9640 1 -- 172.21.15.27:0/1419400029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f94b40a7460 con 0x5648b4bfa4d0 2024-04-25T13:51:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.870+0000 7f94ca807740 10 client.17565 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.870+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({fsmap=19}) -- 0x5648b4d057d0 con 0x5648b4bfa4d0 2024-04-25T13:51:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.871+0000 7f94a0ff9640 1 -- 172.21.15.27:0/1419400029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f94b4020070 con 0x5648b4bfa4d0 2024-04-25T13:51:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.871+0000 7f94ca807740 10 client.17565 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.871+0000 7f94ca807740 10 client.17565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.871+0000 7f94ca807740 20 client.17565 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.871+0000 7f94ca807740 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5648b4be1f60 0x5648b4fdd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.871+0000 7f94ca807740 4 client.17565 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.871+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648b4f37990 con 0x5648b4be1f60 2024-04-25T13:51:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.872+0000 7f94c0e1f640 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5648b4be1f60 0x5648b4fdd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.872+0000 7f94a0ff9640 10 client.17565 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.872+0000 7f94c0e1f640 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5648b4be1f60 0x5648b4fdd0c0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d047e6aefff1469 server_cookie=e6227efe2cf86d9e in_seq=0 out_seq=0 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94a0ff9640 1 -- 172.21.15.27:0/1419400029 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5648b4f37990 con 0x5648b4be1f60 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94a0ff9640 10 client.17565 handle_command_reply: tid=0 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 client.17565 shutdown 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 msgr2=0x5648b4fcf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fcf570 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f94b400afd0 tx=0x7f94b40df6c0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 shutdown_connections 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5648b4be1f60 0x5648b4fdd0c0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fd6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fcfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 --2- 172.21.15.27:0/1419400029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fcf570 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 >> 172.21.15.27:0/1419400029 conn(0x5648b4c174e0 msgr2=0x5648b4fc3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94a0ff9640 0 client.17565 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 shutdown_connections 2024-04-25T13:51:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.889+0000 7f94ca807740 1 -- 172.21.15.27:0/1419400029 wait complete. 2024-04-25T13:51:48.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.890+0000 7f94ca807740 20 client.17565 trim_cache size 0 max 16384 2024-04-25T13:51:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.906+0000 7f94ca807740 1 Processor -- start 2024-04-25T13:51:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.906+0000 7f94ca807740 1 -- start start 2024-04-25T13:51:48.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94ca807740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94ca807740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c026b0 0x5648b4fd69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94ca807740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fcdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94ca807740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f94a800c330 con 0x5648b4c026b0 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94ca807740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94a800ebb0 con 0x5648b4c23390 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94ca807740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f94a800c9a0 con 0x5648b4bfa4d0 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c0e1f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c026b0 0x5648b4fd69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1620640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1620640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51192/0 (socket says 172.21.15.27:51192) 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1620640 1 -- 172.21.15.27:0/1412421466 learned_addr learned my addr 172.21.15.27:0/1412421466 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1e21640 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fcdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1620640 1 -- 172.21.15.27:0/1412421466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 msgr2=0x5648b4fcdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1620640 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fcdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1620640 1 -- 172.21.15.27:0/1412421466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c026b0 msgr2=0x5648b4fd69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1620640 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c026b0 0x5648b4fd69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c1620640 1 -- 172.21.15.27:0/1412421466 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648b4fcd610 con 0x5648b4bfa4d0 2024-04-25T13:51:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.907+0000 7f94c0e1f640 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c026b0 0x5648b4fd69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.908+0000 7f94c1620640 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc10f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f94b40a7430 tx=0x7f94b40b0000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31baa1d0e0c4839 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.908+0000 7f94a2ffd640 1 -- 172.21.15.27:0/1412421466 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b40a76f0 con 0x5648b4bfa4d0 2024-04-25T13:51:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.908+0000 7f94a2ffd640 1 -- 172.21.15.27:0/1412421466 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94b40a8020 con 0x5648b4bfa4d0 2024-04-25T13:51:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.908+0000 7f94a2ffd640 1 -- 172.21.15.27:0/1412421466 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b40a3290 con 0x5648b4bfa4d0 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.909+0000 7f94ca807740 1 -- 172.21.15.27:0/1412421466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 msgr2=0x5648b4fc10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.909+0000 7f94ca807740 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc10f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f94b40a7430 tx=0x7f94b40b0000 comp rx=0 tx=0).stop 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.909+0000 7f94ca807740 1 -- 172.21.15.27:0/1412421466 shutdown_connections 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.909+0000 7f94ca807740 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fcdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.909+0000 7f94ca807740 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4c026b0 0x5648b4fd69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.909+0000 7f94ca807740 1 --2- 172.21.15.27:0/1412421466 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4bfa4d0 0x5648b4fc10f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.910+0000 7f94ca807740 1 -- 172.21.15.27:0/1412421466 >> 172.21.15.27:0/1412421466 conn(0x5648b4c174e0 msgr2=0x5648b4fc09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.910+0000 7f94ca807740 1 -- 172.21.15.27:0/1412421466 shutdown_connections 2024-04-25T13:51:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.910+0000 7f94ca807740 1 -- 172.21.15.27:0/1412421466 wait complete. 2024-04-25T13:51:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.910+0000 7f94ca807740 1 -- start start 2024-04-25T13:51:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.911+0000 7f94ca807740 1 Processor -- start 2024-04-25T13:51:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.911+0000 7f94ca807740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fd6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94ca807740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fd69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94ca807740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fd3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94ca807740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f94a800d070 con 0x5648b4bfa4d0 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94ca807740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94a800e4e0 con 0x5648b4c23390 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94ca807740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f94a800d740 con 0x5648b4c026b0 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94ca807740 10 client.0 _ll_register_callbacks cb 0x7f94b40e7250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c1620640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fd6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c1620640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fd6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34172/0 (socket says 172.21.15.27:34172) 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c1620640 1 -- 172.21.15.27:0/2917311950 learned_addr learned my addr 172.21.15.27:0/2917311950 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c0e1f640 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fd69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c1e21640 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fd3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c0e1f640 1 -- 172.21.15.27:0/2917311950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 msgr2=0x5648b4fd3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c0e1f640 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fd3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c0e1f640 1 -- 172.21.15.27:0/2917311950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 msgr2=0x5648b4fd6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c0e1f640 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fd6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c0e1f640 1 -- 172.21.15.27:0/2917311950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648b4fd3890 con 0x5648b4c026b0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c1620640 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fd6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94c0e1f640 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fd69b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x5648b4fdd7f0 tx=0x7f94ac009e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2896df386fd71cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.912+0000 7f94a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94a0ff9640 1 -- 172.21.15.27:0/2917311950 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac019960 con 0x5648b4c026b0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 1 -- 172.21.15.27:0/2917311950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b40a8020 con 0x5648b4c026b0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94a0ff9640 1 -- 172.21.15.27:0/2917311950 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f94ac016420 con 0x5648b4c026b0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94a0ff9640 1 -- 172.21.15.27:0/2917311950 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f94ac016620 con 0x5648b4c026b0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 10 client.27304 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 1 -- 172.21.15.27:0/2917311950 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5648b5044270 con 0x5648b4c026b0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94a0ff9640 1 -- 172.21.15.27:0/2917311950 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94b40a8020 con 0x5648b4c026b0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94a0ff9640 1 -- 172.21.15.27:0/2917311950 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f94ac020070 con 0x5648b4c026b0 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 10 client.27304 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 10 client.27304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:48.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 20 client.27304 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:48.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5648b4bb73a0 0x5648b5046c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:48.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 4 client.27304 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:51:48.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.913+0000 7f94ca807740 1 -- 172.21.15.27:0/2917311950 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648b4f31800 con 0x5648b4bb73a0 2024-04-25T13:51:48.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.914+0000 7f94c1620640 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5648b4bb73a0 0x5648b5046c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.914+0000 7f94a0ff9640 10 client.27304 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.914+0000 7f94c1620640 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5648b4bb73a0 0x5648b5046c60 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12485492cc2fffe6 server_cookie=be9736dff8cd4d70 in_seq=0 out_seq=0 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.914+0000 7f94a0ff9640 1 -- 172.21.15.27:0/2917311950 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648b4f31800 con 0x5648b4bb73a0 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.914+0000 7f94a0ff9640 10 client.27304 handle_command_reply: tid=0 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.914+0000 7f94ca807740 1 client.27304 shutdown 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 -- 172.21.15.27:0/2917311950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 msgr2=0x5648b4fd69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fd69b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x5648b4fdd7f0 tx=0x7f94ac009e80 comp rx=0 tx=0).stop 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 -- 172.21.15.27:0/2917311950 shutdown_connections 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5648b4bb73a0 0x5648b5046c60 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648b4c23390 0x5648b4fd3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5648b4c026b0 0x5648b4fd69b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94a0ff9640 0 client.27304 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 --2- 172.21.15.27:0/2917311950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5648b4bfa4d0 0x5648b4fd6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 -- 172.21.15.27:0/2917311950 >> 172.21.15.27:0/2917311950 conn(0x5648b4c174e0 msgr2=0x5648b4fcdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 -- 172.21.15.27:0/2917311950 shutdown_connections 2024-04-25T13:51:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.915+0000 7f94ca807740 1 -- 172.21.15.27:0/2917311950 wait complete. 2024-04-25T13:51:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94ca807740 20 client.27304 trim_cache size 0 max 16384 2024-04-25T13:51:48.919 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 -- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94a40afcf0 msgr2=0x7f94a40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94a40afcf0 0x7f94a40b2200 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f94b0031500 tx=0x7f94b0009560 comp rx=0 tx=0).stop 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 -- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc10a420 msgr2=0x7f94bc116f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc10a420 0x7f94bc116f70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f94b400ccb0 tx=0x7f94b400a620 comp rx=0 tx=0).stop 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 -- 172.21.15.27:0/2615295226 shutdown_connections 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94a40afcf0 0x7f94a40b2200 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94bc110030 0x7f94bc118410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94bc10b310 0x7f94bc1174b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 --2- 172.21.15.27:0/2615295226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc10a420 0x7f94bc116f70 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 -- 172.21.15.27:0/2615295226 >> 172.21.15.27:0/2615295226 conn(0x7f94bc0704e0 msgr2=0x7f94bc10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 -- 172.21.15.27:0/2615295226 shutdown_connections 2024-04-25T13:51:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:48.916+0000 7f94c3884640 1 -- 172.21.15.27:0/2615295226 wait complete. 2024-04-25T13:51:49.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:49 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/151485276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:51:49.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:49 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/151485276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:51:49.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:49 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/151485276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:51:49.930 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:51:49.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:51:49.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:51:50.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.117+0000 7fe4553b2640 1 -- 172.21.15.27:0/276847741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe450110030 msgr2=0x7fe450110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.117+0000 7fe4553b2640 1 --2- 172.21.15.27:0/276847741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe450110030 0x7fe450110440 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe44001a910 tx=0x7fe44003b450 comp rx=0 tx=0).stop 2024-04-25T13:51:50.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.119+0000 7fe4553b2640 1 -- 172.21.15.27:0/276847741 shutdown_connections 2024-04-25T13:51:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.119+0000 7fe4553b2640 1 --2- 172.21.15.27:0/276847741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe450110030 0x7fe450110440 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.119+0000 7fe4553b2640 1 --2- 172.21.15.27:0/276847741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45010b310 0x7fe45010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.119+0000 7fe4553b2640 1 --2- 172.21.15.27:0/276847741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe45010a420 0x7fe45010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.119+0000 7fe4553b2640 1 -- 172.21.15.27:0/276847741 >> 172.21.15.27:0/276847741 conn(0x7fe4500704e0 msgr2=0x7fe450072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.121+0000 7fe4553b2640 1 -- 172.21.15.27:0/276847741 shutdown_connections 2024-04-25T13:51:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.122+0000 7fe4553b2640 1 -- 172.21.15.27:0/276847741 wait complete. 2024-04-25T13:51:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.123+0000 7fe4553b2640 1 Processor -- start 2024-04-25T13:51:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.123+0000 7fe4553b2640 1 -- start start 2024-04-25T13:51:50.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe4553b2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe45010a420 0x7fe4501a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe4553b2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45010b310 0x7fe4501a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe4553b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe450110030 0x7fe4501a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe44effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe45010a420 0x7fe4501a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe44effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe45010a420 0x7fe4501a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51224/0 (socket says 172.21.15.27:51224) 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe44e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45010b310 0x7fe4501a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe450113cd0 con 0x7fe45010b310 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe44effd640 1 -- 172.21.15.27:0/2061082458 learned_addr learned my addr 172.21.15.27:0/2061082458 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe450113b50 con 0x7fe450110030 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe450113e50 con 0x7fe45010a420 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.124+0000 7fe44f7fe640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe450110030 0x7fe4501a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe44effd640 1 -- 172.21.15.27:0/2061082458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe450110030 msgr2=0x7fe4501a7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe44effd640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe450110030 0x7fe4501a7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe44effd640 1 -- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45010b310 msgr2=0x7fe4501a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe44effd640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45010b310 0x7fe4501a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe44effd640 1 -- 172.21.15.27:0/2061082458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4501a83c0 con 0x7fe45010a420 2024-04-25T13:51:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe44effd640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe45010a420 0x7fe4501a0880 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe44800b9e0 tx=0x7fe44800bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe437fff640 1 -- 172.21.15.27:0/2061082458 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448019070 con 0x7fe45010a420 2024-04-25T13:51:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe437fff640 1 -- 172.21.15.27:0/2061082458 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe448014440 con 0x7fe45010a420 2024-04-25T13:51:50.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe437fff640 1 -- 172.21.15.27:0/2061082458 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4480125a0 con 0x7fe45010a420 2024-04-25T13:51:50.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4501a86b0 con 0x7fe45010a420 2024-04-25T13:51:50.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.125+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4501a8bb0 con 0x7fe45010a420 2024-04-25T13:51:50.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.129+0000 7fe45c269740 1 Processor -- start 2024-04-25T13:51:50.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.129+0000 7fe45c269740 1 -- start start 2024-04-25T13:51:50.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.129+0000 7fe437fff640 1 -- 172.21.15.27:0/2061082458 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe44801c070 con 0x7fe45010a420 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe45c269740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8be34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe45c269740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8be3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe45c269740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8821730 0x555eb8bea920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe45c269740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555eb8b48b00 con 0x555eb8819550 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe45c269740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555eb8ae4920 con 0x555eb8842410 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe45c269740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555eb8b44ac0 con 0x555eb8821730 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe44f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8be34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe44f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8be34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52982/0 (socket says 172.21.15.27:52982) 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe44f7fe640 1 -- 172.21.15.27:0/3880014308 learned_addr learned my addr 172.21.15.27:0/3880014308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.130+0000 7fe44effd640 1 --2- 172.21.15.27:0/3880014308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8be3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe437fff640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe4380afbb0 0x7fe4380b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe44f7fe640 1 -- 172.21.15.27:0/3880014308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8821730 msgr2=0x555eb8bea920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe44f7fe640 1 --2- 172.21.15.27:0/3880014308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8821730 0x555eb8bea920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe44f7fe640 1 -- 172.21.15.27:0/3880014308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 msgr2=0x555eb8be3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe44f7fe640 1 --2- 172.21.15.27:0/3880014308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8be3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe44f7fe640 1 -- 172.21.15.27:0/3880014308 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eb89227e0 con 0x555eb8842410 2024-04-25T13:51:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe437fff640 1 -- 172.21.15.27:0/2061082458 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe4480de0f0 con 0x7fe45010a420 2024-04-25T13:51:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe44f7fe640 1 --2- 172.21.15.27:0/3880014308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8be34a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe440044000 tx=0x7fe440004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdc31c7cd1fd96af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe44f7fe640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe4380afbb0 0x7fe4380b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.131+0000 7fe4357fa640 1 -- 172.21.15.27:0/3880014308 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440046030 con 0x555eb8842410 2024-04-25T13:51:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.132+0000 7fe4357fa640 1 -- 172.21.15.27:0/3880014308 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe440004b30 con 0x555eb8842410 2024-04-25T13:51:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.132+0000 7fe4357fa640 1 -- 172.21.15.27:0/3880014308 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440040df0 con 0x555eb8842410 2024-04-25T13:51:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.133+0000 7fe44f7fe640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe4380afbb0 0x7fe4380b20c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe440055000 tx=0x7fe44004d440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.133+0000 7fe45c269740 1 -- 172.21.15.27:0/3880014308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 msgr2=0x555eb8be34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.133+0000 7fe45c269740 1 --2- 172.21.15.27:0/3880014308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8be34a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe440044000 tx=0x7fe440004950 comp rx=0 tx=0).stop 2024-04-25T13:51:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.133+0000 7fe45c269740 1 -- 172.21.15.27:0/3880014308 shutdown_connections 2024-04-25T13:51:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.133+0000 7fe45c269740 1 --2- 172.21.15.27:0/3880014308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8821730 0x555eb8bea920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.133+0000 7fe45c269740 1 --2- 172.21.15.27:0/3880014308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8be3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.134+0000 7fe45c269740 1 --2- 172.21.15.27:0/3880014308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8be34a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.134+0000 7fe45c269740 1 -- 172.21.15.27:0/3880014308 >> 172.21.15.27:0/3880014308 conn(0x555eb8836560 msgr2=0x555eb8bdee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.134+0000 7fe45c269740 1 -- 172.21.15.27:0/3880014308 shutdown_connections 2024-04-25T13:51:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.134+0000 7fe45c269740 1 -- 172.21.15.27:0/3880014308 wait complete. 2024-04-25T13:51:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.135+0000 7fe45c269740 1 -- start start 2024-04-25T13:51:50.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.135+0000 7fe45c269740 1 Processor -- start 2024-04-25T13:51:50.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe45c269740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bed270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe45c269740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bed7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe45c269740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8842410 0x555eb8bf4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bed270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51232/0 (socket says 172.21.15.27:51232) 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44effd640 1 -- 172.21.15.27:0/284348005 learned_addr learned my addr 172.21.15.27:0/284348005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555eb8beb310 con 0x555eb8842410 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555eb8ae25d0 con 0x555eb8821730 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555eb8beb490 con 0x555eb8819550 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44e7fc640 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bed7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe45c269740 10 client.0 _ll_register_callbacks cb 0x555eb8ad0080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44f7fe640 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8842410 0x555eb8bf4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44effd640 1 -- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 msgr2=0x555eb8bed7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44effd640 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bed7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44effd640 1 -- 172.21.15.27:0/284348005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8842410 msgr2=0x555eb8bf4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44effd640 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8842410 0x555eb8bf4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44effd640 1 -- 172.21.15.27:0/284348005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eb8982f70 con 0x555eb8819550 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.136+0000 7fe44e7fc640 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bed7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.137+0000 7fe44effd640 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bed270 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe44800bfd0 tx=0x7fe4480a2570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab6bbbc8ae3e9e7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.137+0000 7fe432ff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.137+0000 7fe432ff5640 1 -- 172.21.15.27:0/284348005 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4480a5e20 con 0x555eb8819550 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.137+0000 7fe432ff5640 1 -- 172.21.15.27:0/284348005 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe4480aefd0 con 0x555eb8819550 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.137+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe440004b30 con 0x555eb8819550 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.137+0000 7fe432ff5640 1 -- 172.21.15.27:0/284348005 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe4480a61c0 con 0x555eb8819550 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.137+0000 7fe45c269740 10 client.27313 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:50.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.137+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555eb881a170 con 0x555eb8819550 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.138+0000 7fe432ff5640 1 -- 172.21.15.27:0/284348005 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe440004b30 con 0x555eb8819550 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.138+0000 7fe432ff5640 1 -- 172.21.15.27:0/284348005 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe44808db50 con 0x555eb8819550 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.138+0000 7fe45c269740 10 client.27313 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.138+0000 7fe45c269740 10 client.27313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.138+0000 7fe45c269740 20 client.27313 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.138+0000 7fe45c269740 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555eb88011f0 0x555eb8bfac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.138+0000 7fe45c269740 4 client.27313 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.138+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555eb8b56710 con 0x555eb88011f0 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.139+0000 7fe44e7fc640 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555eb88011f0 0x555eb8bfac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.139+0000 7fe44e7fc640 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555eb88011f0 0x555eb8bfac70 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=493467537a06ea53 server_cookie=f40d9df783c91c84 in_seq=0 out_seq=0 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.139+0000 7fe432ff5640 10 client.27313 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe432ff5640 1 -- 172.21.15.27:0/284348005 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555eb8b56710 con 0x555eb88011f0 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe432ff5640 10 client.27313 handle_command_reply: tid=0 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 client.27313 shutdown 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 msgr2=0x555eb8bed270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bed270 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe44800bfd0 tx=0x7fe4480a2570 comp rx=0 tx=0).stop 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 shutdown_connections 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555eb88011f0 0x555eb8bfac70 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8842410 0x555eb8bf4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bed7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe432ff5640 0 client.27313 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 --2- 172.21.15.27:0/284348005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bed270 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 >> 172.21.15.27:0/284348005 conn(0x555eb8836560 msgr2=0x555eb8bddef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 shutdown_connections 2024-04-25T13:51:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 1 -- 172.21.15.27:0/284348005 wait complete. 2024-04-25T13:51:50.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.140+0000 7fe45c269740 20 client.27313 trim_cache size 0 max 16384 2024-04-25T13:51:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.168+0000 7fe45c269740 1 Processor -- start 2024-04-25T13:51:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.168+0000 7fe45c269740 1 -- start start 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.168+0000 7fe45c269740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8bebf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.168+0000 7fe45c269740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.168+0000 7fe45c269740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8842410 0x555eb8bff930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.168+0000 7fe45c269740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501af750 con 0x555eb8819550 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.168+0000 7fe45c269740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b1fd0 con 0x555eb8821730 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe45c269740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501afdc0 con 0x555eb8842410 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8bebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44e7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8bebf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34230/0 (socket says 172.21.15.27:34230) 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 -- 172.21.15.27:0/2529595287 learned_addr learned my addr 172.21.15.27:0/2529595287 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44f7fe640 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8842410 0x555eb8bff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 -- 172.21.15.27:0/2529595287 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8842410 msgr2=0x555eb8bff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8842410 0x555eb8bff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 -- 172.21.15.27:0/2529595287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 msgr2=0x555eb8bff3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bff3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 -- 172.21.15.27:0/2529595287 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eb8be8180 con 0x555eb8819550 2024-04-25T13:51:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44e7fc640 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe44effd640 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8bebf40 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe4480a63b0 tx=0x7fe44800c590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46a9bde74975431 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe434ff9640 1 -- 172.21.15.27:0/2529595287 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4480a66b0 con 0x555eb8819550 2024-04-25T13:51:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe434ff9640 1 -- 172.21.15.27:0/2529595287 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4480c0020 con 0x555eb8819550 2024-04-25T13:51:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.169+0000 7fe434ff9640 1 -- 172.21.15.27:0/2529595287 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44800cb90 con 0x555eb8819550 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.170+0000 7fe45c269740 1 -- 172.21.15.27:0/2529595287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 msgr2=0x555eb8bebf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.170+0000 7fe45c269740 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8bebf40 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe4480a63b0 tx=0x7fe44800c590 comp rx=0 tx=0).stop 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.170+0000 7fe45c269740 1 -- 172.21.15.27:0/2529595287 shutdown_connections 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.170+0000 7fe45c269740 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8842410 0x555eb8bff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.170+0000 7fe45c269740 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8821730 0x555eb8bff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.170+0000 7fe45c269740 1 --2- 172.21.15.27:0/2529595287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8819550 0x555eb8bebf40 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.170+0000 7fe45c269740 1 -- 172.21.15.27:0/2529595287 >> 172.21.15.27:0/2529595287 conn(0x555eb8836560 msgr2=0x555eb8beb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.171+0000 7fe45c269740 1 -- 172.21.15.27:0/2529595287 shutdown_connections 2024-04-25T13:51:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.171+0000 7fe45c269740 1 -- 172.21.15.27:0/2529595287 wait complete. 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 1 -- start start 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 1 Processor -- start 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bf3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8821730 0x555eb8c5ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8c614e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b0490 con 0x555eb8821730 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b1900 con 0x555eb8842410 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b0b60 con 0x555eb8819550 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe45c269740 10 client.0 _ll_register_callbacks cb 0x7fe4480a2d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8821730 0x555eb8c5ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bf3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8c614e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44effd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bf3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51260/0 (socket says 172.21.15.27:51260) 2024-04-25T13:51:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44effd640 1 -- 172.21.15.27:0/3414191017 learned_addr learned my addr 172.21.15.27:0/3414191017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44effd640 1 -- 172.21.15.27:0/3414191017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 msgr2=0x555eb8c614e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44effd640 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8c614e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44effd640 1 -- 172.21.15.27:0/3414191017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8821730 msgr2=0x555eb8c5ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44effd640 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8821730 0x555eb8c5ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.172+0000 7fe44effd640 1 -- 172.21.15.27:0/3414191017 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eb8c61bf0 con 0x555eb8819550 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe44e7fc640 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8821730 0x555eb8c5ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe44effd640 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bf3c10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe4480a3920 tx=0x7fe44800c600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33e136e53e32075b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe432ff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe432ff5640 1 -- 172.21.15.27:0/3414191017 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4480a66b0 con 0x555eb8819550 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe432ff5640 1 -- 172.21.15.27:0/3414191017 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe4480a5910 con 0x555eb8819550 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 1 -- 172.21.15.27:0/3414191017 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4480c0020 con 0x555eb8819550 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe432ff5640 1 -- 172.21.15.27:0/3414191017 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe4480a5b50 con 0x555eb8819550 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 10 client.27319 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 1 -- 172.21.15.27:0/3414191017 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555eb8c62eb0 con 0x555eb8819550 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe432ff5640 1 -- 172.21.15.27:0/3414191017 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4480a2020 con 0x555eb8819550 2024-04-25T13:51:50.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe432ff5640 1 -- 172.21.15.27:0/3414191017 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe4480abb40 con 0x555eb8819550 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 10 client.27319 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 10 client.27319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 20 client.27319 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555eb87d6630 0x555eb8c65630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 4 client.27319 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.173+0000 7fe45c269740 1 -- 172.21.15.27:0/3414191017 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fe4480c0020 con 0x555eb87d6630 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.174+0000 7fe44e7fc640 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555eb87d6630 0x555eb8c65630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.174+0000 7fe432ff5640 10 client.27319 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.174+0000 7fe44e7fc640 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555eb87d6630 0x555eb8c65630 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=489287cf75dbc185 server_cookie=4df452f98f16ff2e in_seq=0 out_seq=0 2024-04-25T13:51:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe432ff5640 1 -- 172.21.15.27:0/3414191017 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fe4480c0020 con 0x555eb87d6630 2024-04-25T13:51:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe432ff5640 10 client.27319 handle_command_reply: tid=0 2024-04-25T13:51:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 client.27319 shutdown 2024-04-25T13:51:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 -- 172.21.15.27:0/3414191017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 msgr2=0x555eb8bf3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bf3c10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe4480a3920 tx=0x7fe44800c600 comp rx=0 tx=0).stop 2024-04-25T13:51:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 -- 172.21.15.27:0/3414191017 shutdown_connections 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555eb87d6630 0x555eb8c65630 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe432ff5640 0 client.27319 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555eb8842410 0x555eb8c614e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555eb8821730 0x555eb8c5ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 --2- 172.21.15.27:0/3414191017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555eb8819550 0x555eb8bf3c10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 -- 172.21.15.27:0/3414191017 >> 172.21.15.27:0/3414191017 conn(0x555eb8836560 msgr2=0x555eb8be1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 -- 172.21.15.27:0/3414191017 shutdown_connections 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 1 -- 172.21.15.27:0/3414191017 wait complete. 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.178+0000 7fe45c269740 20 client.27319 trim_cache size 0 max 16384 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:51:50.179 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "ee206699-cab3-41cc-845a-49b276011bd3", 2024-04-25T13:51:50.180 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:51:50.180 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:51:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe4380afbb0 msgr2=0x7fe4380b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe4380afbb0 0x7fe4380b20c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe440055000 tx=0x7fe44004d440 comp rx=0 tx=0).stop 2024-04-25T13:51:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe45010a420 msgr2=0x7fe4501a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe45010a420 0x7fe4501a0880 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe44800b9e0 tx=0x7fe44800bea0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 shutdown_connections 2024-04-25T13:51:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe4380afbb0 0x7fe4380b20c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe450110030 0x7fe4501a7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45010b310 0x7fe4501a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 --2- 172.21.15.27:0/2061082458 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe45010a420 0x7fe4501a0880 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 >> 172.21.15.27:0/2061082458 conn(0x7fe4500704e0 msgr2=0x7fe45010e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 shutdown_connections 2024-04-25T13:51:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.179+0000 7fe4553b2640 1 -- 172.21.15.27:0/2061082458 wait complete. 2024-04-25T13:51:50.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:51:50.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:51:50.222 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:50 smithi027 ceph-mon[29140]: pgmap v677: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.7 MiB/s wr, 560 op/s 2024-04-25T13:51:50.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.396+0000 7fa84c4f9640 1 -- 172.21.15.27:0/1816817154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa84410bb70 msgr2=0x7fa844110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.396+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/1816817154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa84410bb70 0x7fa844110460 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa840013080 tx=0x7fa840032710 comp rx=0 tx=0).stop 2024-04-25T13:51:50.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.396+0000 7fa84c4f9640 1 -- 172.21.15.27:0/1816817154 shutdown_connections 2024-04-25T13:51:50.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.396+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/1816817154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8441109a0 0x7fa844112dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.396+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/1816817154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa84410bb70 0x7fa844110460 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.396+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/1816817154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa84410b220 0x7fa84410b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.396+0000 7fa84c4f9640 1 -- 172.21.15.27:0/1816817154 >> 172.21.15.27:0/1816817154 conn(0x7fa844070940 msgr2=0x7fa844072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.396+0000 7fa84c4f9640 1 -- 172.21.15.27:0/1816817154 shutdown_connections 2024-04-25T13:51:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.397+0000 7fa84c4f9640 1 -- 172.21.15.27:0/1816817154 wait complete. 2024-04-25T13:51:50.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.397+0000 7fa84c4f9640 1 Processor -- start 2024-04-25T13:51:50.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84c4f9640 1 -- start start 2024-04-25T13:51:50.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84c4f9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa84410b220 0x7fa8441a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84c4f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84410bb70 0x7fa8441a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84c4f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8441109a0 0x7fa8441a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84c4f9640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa844115c40 con 0x7fa8441109a0 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84c4f9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa844115ac0 con 0x7fa84410b220 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84c4f9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa844115dc0 con 0x7fa84410bb70 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa849a94640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84410bb70 0x7fa8441a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa849a94640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84410bb70 0x7fa8441a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51276/0 (socket says 172.21.15.27:51276) 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa849a94640 1 -- 172.21.15.27:0/3138719329 learned_addr learned my addr 172.21.15.27:0/3138719329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84a295640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa84410b220 0x7fa8441a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa84aa96640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8441109a0 0x7fa8441a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa849a94640 1 -- 172.21.15.27:0/3138719329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa84410b220 msgr2=0x7fa8441a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.398+0000 7fa849a94640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa84410b220 0x7fa8441a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa849a94640 1 -- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8441109a0 msgr2=0x7fa8441a7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa849a94640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8441109a0 0x7fa8441a7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa849a94640 1 -- 172.21.15.27:0/3138719329 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8441b93f0 con 0x7fa84410bb70 2024-04-25T13:51:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa84aa96640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8441109a0 0x7fa8441a7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:51:50.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa849a94640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84410bb70 0x7fa8441a0de0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa840004530 tx=0x7fa840004560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa83b7fe640 1 -- 172.21.15.27:0/3138719329 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84003e070 con 0x7fa84410bb70 2024-04-25T13:51:50.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa84c4f9640 1 -- 172.21.15.27:0/3138719329 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8441b9680 con 0x7fa84410bb70 2024-04-25T13:51:50.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa83b7fe640 1 -- 172.21.15.27:0/3138719329 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa840039400 con 0x7fa84410bb70 2024-04-25T13:51:50.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa83b7fe640 1 -- 172.21.15.27:0/3138719329 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8400110d0 con 0x7fa84410bb70 2024-04-25T13:51:50.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.399+0000 7fa84c4f9640 1 -- 172.21.15.27:0/3138719329 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8441b9c20 con 0x7fa84410bb70 2024-04-25T13:51:50.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.403+0000 7fa8533ac740 1 Processor -- start 2024-04-25T13:51:50.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.404+0000 7fa8533ac740 1 -- start start 2024-04-25T13:51:50.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.405+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacd1390 0x5623db072a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.405+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623daca84d0 0x5623db072fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.405+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacb06b0 0x5623db073c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.405+0000 7fa84a295640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623daca84d0 0x5623db072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.405+0000 7fa84a295640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623daca84d0 0x5623db072fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51292/0 (socket says 172.21.15.27:51292) 2024-04-25T13:51:50.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.405+0000 7fa84a295640 1 -- 172.21.15.27:0/713492454 learned_addr learned my addr 172.21.15.27:0/713492454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.405+0000 7fa8533ac740 1 -- 172.21.15.27:0/713492454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5623dafd7d80 con 0x5623dacb06b0 2024-04-25T13:51:50.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.406+0000 7fa8533ac740 1 -- 172.21.15.27:0/713492454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5623daf918b0 con 0x5623dacd1390 2024-04-25T13:51:50.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.406+0000 7fa8533ac740 1 -- 172.21.15.27:0/713492454 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5623dafd3d40 con 0x5623daca84d0 2024-04-25T13:51:50.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.406+0000 7fa83b7fe640 1 -- 172.21.15.27:0/3138719329 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa840041070 con 0x7fa84410bb70 2024-04-25T13:51:50.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.406+0000 7fa84aa96640 1 --2- 172.21.15.27:0/713492454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacd1390 0x5623db072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.407+0000 7fa84aa96640 1 -- 172.21.15.27:0/713492454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623daca84d0 msgr2=0x5623db072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.407+0000 7fa84aa96640 1 --2- 172.21.15.27:0/713492454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623daca84d0 0x5623db072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.407+0000 7fa84aa96640 1 -- 172.21.15.27:0/713492454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacb06b0 msgr2=0x5623db073c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.407+0000 7fa84aa96640 1 --2- 172.21.15.27:0/713492454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacb06b0 0x5623db073c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.407+0000 7fa84aa96640 1 -- 172.21.15.27:0/713492454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623daf62a40 con 0x5623dacd1390 2024-04-25T13:51:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.407+0000 7fa83b7fe640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8300afc30 0x7fa8300b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.408+0000 7fa83b7fe640 1 -- 172.21.15.27:0/3138719329 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa8400fe940 con 0x7fa84410bb70 2024-04-25T13:51:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.408+0000 7fa84a295640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8300afc30 0x7fa8300b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.408+0000 7fa84aa96640 1 --2- 172.21.15.27:0/713492454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacd1390 0x5623db072a60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00bf00 tx=0x7fa83c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91303362a23ca271 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.409+0000 7fa838bfe640 1 -- 172.21.15.27:0/713492454 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0157d0 con 0x5623dacd1390 2024-04-25T13:51:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.409+0000 7fa838bfe640 1 -- 172.21.15.27:0/713492454 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c01b070 con 0x5623dacd1390 2024-04-25T13:51:50.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.409+0000 7fa838bfe640 1 -- 172.21.15.27:0/713492454 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c016d60 con 0x5623dacd1390 2024-04-25T13:51:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.410+0000 7fa84a295640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8300afc30 0x7fa8300b2140 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x5623db072780 tx=0x7fa834011040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.410+0000 7fa8533ac740 1 -- 172.21.15.27:0/713492454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacd1390 msgr2=0x5623db072a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.410+0000 7fa8533ac740 1 --2- 172.21.15.27:0/713492454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacd1390 0x5623db072a60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00bf00 tx=0x7fa83c009260 comp rx=0 tx=0).stop 2024-04-25T13:51:50.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.410+0000 7fa8533ac740 1 -- 172.21.15.27:0/713492454 shutdown_connections 2024-04-25T13:51:50.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.410+0000 7fa8533ac740 1 --2- 172.21.15.27:0/713492454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacb06b0 0x5623db073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.410+0000 7fa8533ac740 1 --2- 172.21.15.27:0/713492454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623daca84d0 0x5623db072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.410+0000 7fa8533ac740 1 --2- 172.21.15.27:0/713492454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacd1390 0x5623db072a60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.410+0000 7fa8533ac740 1 -- 172.21.15.27:0/713492454 >> 172.21.15.27:0/713492454 conn(0x5623dacc54e0 msgr2=0x5623db06e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.411+0000 7fa8533ac740 1 -- 172.21.15.27:0/713492454 shutdown_connections 2024-04-25T13:51:50.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.411+0000 7fa8533ac740 1 -- 172.21.15.27:0/713492454 wait complete. 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.411+0000 7fa8533ac740 1 -- start start 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa8533ac740 1 Processor -- start 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623daca84d0 0x5623db07cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacb06b0 0x5623db07cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacd1390 0x5623db083f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa8533ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5623db0747d0 con 0x5623dacd1390 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa8533ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5623daf5ed10 con 0x5623daca84d0 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa8533ac740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5623db074950 con 0x5623dacb06b0 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa84a295640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623daca84d0 0x5623db07cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa84a295640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623daca84d0 0x5623db07cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:53026/0 (socket says 172.21.15.27:53026) 2024-04-25T13:51:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa84a295640 1 -- 172.21.15.27:0/2677826971 learned_addr learned my addr 172.21.15.27:0/2677826971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa849a94640 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacb06b0 0x5623db07cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.412+0000 7fa8533ac740 10 client.0 _ll_register_callbacks cb 0x5623daf8b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa84aa96640 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacd1390 0x5623db083f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa84a295640 1 -- 172.21.15.27:0/2677826971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacb06b0 msgr2=0x5623db07cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa84a295640 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacb06b0 0x5623db07cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa84a295640 1 -- 172.21.15.27:0/2677826971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacd1390 msgr2=0x5623db083f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa84a295640 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacd1390 0x5623db083f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa84a295640 1 -- 172.21.15.27:0/2677826971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623dadb37d0 con 0x5623daca84d0 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa84aa96640 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacd1390 0x5623db083f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa84a295640 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623daca84d0 0x5623db07cab0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa83400a780 tx=0x7fa834004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0a47b2f1a986126 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa82dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa82dffb640 1 -- 172.21.15.27:0/2677826971 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83402a020 con 0x5623daca84d0 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa8533ac740 1 -- 172.21.15.27:0/2677826971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa83c01b070 con 0x5623daca84d0 2024-04-25T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.413+0000 7fa82dffb640 1 -- 172.21.15.27:0/2677826971 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa83401b620 con 0x5623daca84d0 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa82dffb640 1 -- 172.21.15.27:0/2677826971 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa834022040 con 0x5623daca84d0 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa8533ac740 10 client.27284 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa8533ac740 1 -- 172.21.15.27:0/2677826971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5623dace93f0 con 0x5623daca84d0 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa82dffb640 1 -- 172.21.15.27:0/2677826971 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c01b070 con 0x5623daca84d0 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa82dffb640 1 -- 172.21.15.27:0/2677826971 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa834025030 con 0x5623daca84d0 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa8533ac740 10 client.27284 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa8533ac740 10 client.27284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa8533ac740 20 client.27284 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5623dac8ff60 0x5623db08a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa8533ac740 4 client.27284 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:50.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa8533ac740 1 -- 172.21.15.27:0/2677826971 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623dafe5990 con 0x5623dac8ff60 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.414+0000 7fa849a94640 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5623dac8ff60 0x5623db08a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.415+0000 7fa82dffb640 10 client.27284 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.415+0000 7fa849a94640 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5623dac8ff60 0x5623db08a8c0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=582f8c5a529242b7 server_cookie=988aebfaaeec0f92 in_seq=0 out_seq=0 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.416+0000 7fa82dffb640 1 -- 172.21.15.27:0/2677826971 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5623dafe5990 con 0x5623dac8ff60 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.416+0000 7fa82dffb640 10 client.27284 handle_command_reply: tid=0 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.416+0000 7fa8533ac740 1 client.27284 shutdown 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.416+0000 7fa8533ac740 1 -- 172.21.15.27:0/2677826971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623daca84d0 msgr2=0x5623db07cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623daca84d0 0x5623db07cab0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa83400a780 tx=0x7fa834004730 comp rx=0 tx=0).stop 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 -- 172.21.15.27:0/2677826971 shutdown_connections 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5623dac8ff60 0x5623db08a8c0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623dacd1390 0x5623db083f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa82dffb640 0 client.27284 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:50.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacb06b0 0x5623db07cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2677826971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623daca84d0 0x5623db07cab0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 -- 172.21.15.27:0/2677826971 >> 172.21.15.27:0/2677826971 conn(0x5623dacc54e0 msgr2=0x5623db06d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 -- 172.21.15.27:0/2677826971 shutdown_connections 2024-04-25T13:51:50.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.417+0000 7fa8533ac740 1 -- 172.21.15.27:0/2677826971 wait complete. 2024-04-25T13:51:50.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.420+0000 7fa8533ac740 20 client.27284 trim_cache size 0 max 16384 2024-04-25T13:51:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.447+0000 7fa8533ac740 1 Processor -- start 2024-04-25T13:51:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.447+0000 7fa8533ac740 1 -- start start 2024-04-25T13:51:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.447+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db074dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.447+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db083ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.447+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db084420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa8533ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa8441c0810 con 0x5623daca84d0 2024-04-25T13:51:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa8533ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa8441c3090 con 0x5623dacb06b0 2024-04-25T13:51:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa8533ac740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa8441c0e80 con 0x5623dacd1390 2024-04-25T13:51:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84a295640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db074dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db084420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db084420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51306/0 (socket says 172.21.15.27:51306) 2024-04-25T13:51:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 -- 172.21.15.27:0/3584254815 learned_addr learned my addr 172.21.15.27:0/3584254815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa849a94640 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db083ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 -- 172.21.15.27:0/3584254815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 msgr2=0x5623db083ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db083ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 -- 172.21.15.27:0/3584254815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 msgr2=0x5623db074dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db074dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 -- 172.21.15.27:0/3584254815 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623daca90f0 con 0x5623dacd1390 2024-04-25T13:51:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84a295640 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db074dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:50.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.448+0000 7fa84aa96640 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db084420 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa83c000f80 tx=0x7fa83c00bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f59b55527b93af6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.449+0000 7fa82ffff640 1 -- 172.21.15.27:0/3584254815 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c016df0 con 0x5623dacd1390 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.449+0000 7fa82ffff640 1 -- 172.21.15.27:0/3584254815 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c01b040 con 0x5623dacd1390 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.449+0000 7fa82ffff640 1 -- 172.21.15.27:0/3584254815 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0050d0 con 0x5623dacd1390 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 -- 172.21.15.27:0/3584254815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 msgr2=0x5623db084420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db084420 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa83c000f80 tx=0x7fa83c00bc50 comp rx=0 tx=0).stop 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 -- 172.21.15.27:0/3584254815 shutdown_connections 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db084420 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db083ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 --2- 172.21.15.27:0/3584254815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db074dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 -- 172.21.15.27:0/3584254815 >> 172.21.15.27:0/3584254815 conn(0x5623dacc54e0 msgr2=0x5623db07b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 -- 172.21.15.27:0/3584254815 shutdown_connections 2024-04-25T13:51:50.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.450+0000 7fa8533ac740 1 -- 172.21.15.27:0/3584254815 wait complete. 2024-04-25T13:51:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 1 -- start start 2024-04-25T13:51:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 1 Processor -- start 2024-04-25T13:51:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db081000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db081540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db0f0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa8441c1550 con 0x5623daca84d0 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa8441c29c0 con 0x5623dacb06b0 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa8441c1c20 con 0x5623dacd1390 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa8533ac740 10 client.0 _ll_register_callbacks cb 0x7fa83400a480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa84a295640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db081000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa849a94640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db081540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa84a295640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db081000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34304/0 (socket says 172.21.15.27:34304) 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa84a295640 1 -- 172.21.15.27:0/2159328468 learned_addr learned my addr 172.21.15.27:0/2159328468 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa849a94640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db081540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:53048/0 (socket says 172.21.15.27:53048) 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.451+0000 7fa84aa96640 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db0f0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.452+0000 7fa84a295640 1 -- 172.21.15.27:0/2159328468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 msgr2=0x5623db0f0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.452+0000 7fa84a295640 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db0f0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.452+0000 7fa84a295640 1 -- 172.21.15.27:0/2159328468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 msgr2=0x5623db081540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.452+0000 7fa84a295640 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db081540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.452+0000 7fa84a295640 1 -- 172.21.15.27:0/2159328468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623daf984b0 con 0x5623daca84d0 2024-04-25T13:51:50.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.452+0000 7fa849a94640 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db081540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:51:50.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.452+0000 7fa84a295640 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db081000 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa83400a780 tx=0x7fa83400f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac19b034649a8a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:50.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.452+0000 7fa82dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:50.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.453+0000 7fa82dffb640 1 -- 172.21.15.27:0/2159328468 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83402a040 con 0x5623daca84d0 2024-04-25T13:51:50.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.453+0000 7fa82dffb640 1 -- 172.21.15.27:0/2159328468 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa834031750 con 0x5623daca84d0 2024-04-25T13:51:50.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.453+0000 7fa8533ac740 1 -- 172.21.15.27:0/2159328468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa83c01b040 con 0x5623daca84d0 2024-04-25T13:51:50.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.453+0000 7fa82dffb640 1 -- 172.21.15.27:0/2159328468 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa834005020 con 0x5623daca84d0 2024-04-25T13:51:50.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.453+0000 7fa8533ac740 10 client.17598 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:50.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.453+0000 7fa8533ac740 1 -- 172.21.15.27:0/2159328468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5623db0f1e80 con 0x5623daca84d0 2024-04-25T13:51:50.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa82dffb640 1 -- 172.21.15.27:0/2159328468 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c01b040 con 0x5623daca84d0 2024-04-25T13:51:50.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa82dffb640 1 -- 172.21.15.27:0/2159328468 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa834025040 con 0x5623daca84d0 2024-04-25T13:51:50.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa8533ac740 10 client.17598 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:50.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa8533ac740 10 client.17598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:50.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa8533ac740 20 client.17598 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:50.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5623dac653a0 0x5623db0f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:50.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa8533ac740 4 client.17598 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:51:50.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa8533ac740 1 -- 172.21.15.27:0/2159328468 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5623dafdf800 con 0x5623dac653a0 2024-04-25T13:51:50.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.454+0000 7fa849a94640 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5623dac653a0 0x5623db0f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:50.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.455+0000 7fa82dffb640 10 client.17598 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:50.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.456+0000 7fa849a94640 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5623dac653a0 0x5623db0f48f0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5c1431f4f3a01c4 server_cookie=7a819e513d7d3df7 in_seq=0 out_seq=0 2024-04-25T13:51:50.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.456+0000 7fa82dffb640 1 -- 172.21.15.27:0/2159328468 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5623dafdf800 con 0x5623dac653a0 2024-04-25T13:51:50.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.456+0000 7fa82dffb640 10 client.17598 handle_command_reply: tid=0 2024-04-25T13:51:50.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.456+0000 7fa8533ac740 1 client.17598 shutdown 2024-04-25T13:51:50.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.456+0000 7fa8533ac740 1 -- 172.21.15.27:0/2159328468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 msgr2=0x5623db081000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.456+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db081000 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa83400a780 tx=0x7fa83400f3b0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 1 -- 172.21.15.27:0/2159328468 shutdown_connections 2024-04-25T13:51:50.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5623dac653a0 0x5623db0f48f0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5623dacd1390 0x5623db0f0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5623dacb06b0 0x5623db081540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 1 --2- 172.21.15.27:0/2159328468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5623daca84d0 0x5623db081000 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa82dffb640 0 client.17598 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 1 -- 172.21.15.27:0/2159328468 >> 172.21.15.27:0/2159328468 conn(0x5623dacc54e0 msgr2=0x5623db071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 1 -- 172.21.15.27:0/2159328468 shutdown_connections 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 1 -- 172.21.15.27:0/2159328468 wait complete. 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.457+0000 7fa8533ac740 20 client.17598 trim_cache size 0 max 16384 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (124 inodes in the stack)", 2024-04-25T13:51:50.458 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stdout: "ee206699-cab3-41cc-845a-49b276011bd3": { 2024-04-25T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "ee206699-cab3-41cc-845a-49b276011bd3", 2024-04-25T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 -- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8300afc30 msgr2=0x7fa8300b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8300afc30 0x7fa8300b2140 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x5623db072780 tx=0x7fa834011040 comp rx=0 tx=0).stop 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 -- 172.21.15.27:0/3138719329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84410bb70 msgr2=0x7fa8441a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84410bb70 0x7fa8441a0de0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa840004530 tx=0x7fa840004560 comp rx=0 tx=0).stop 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 -- 172.21.15.27:0/3138719329 shutdown_connections 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8300afc30 0x7fa8300b2140 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8441109a0 0x7fa8441a7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa84410bb70 0x7fa8441a0de0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 --2- 172.21.15.27:0/3138719329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa84410b220 0x7fa8441a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 -- 172.21.15.27:0/3138719329 >> 172.21.15.27:0/3138719329 conn(0x7fa844070940 msgr2=0x7fa844071780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 -- 172.21.15.27:0/3138719329 shutdown_connections 2024-04-25T13:51:50.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:51:50.458+0000 7fa84c4f9640 1 -- 172.21.15.27:0/3138719329 wait complete. 2024-04-25T13:51:50.478 INFO:tasks.cephfs.filesystem:scrub status for tag:ee206699-cab3-41cc-845a-49b276011bd3 - {'path': '/', 'tag': 'ee206699-cab3-41cc-845a-49b276011bd3', 'options': 'recursive,force'} 2024-04-25T13:51:50.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:50 smithi122 ceph-mon[32843]: pgmap v677: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.7 MiB/s wr, 560 op/s 2024-04-25T13:51:50.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:50 smithi045 ceph-mon[32731]: pgmap v677: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.7 MiB/s wr, 560 op/s 2024-04-25T13:51:51.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:51 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:51:51.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:51 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:51.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:51 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:51:51.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:51 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:51.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:51 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:51:51.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:51 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:51.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:51 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:51:51.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:51 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:51.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:51 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:51:51.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:51 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:51.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:51 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:51:51.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:51 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:51:52.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:52 smithi027 ceph-mon[29140]: pgmap v678: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-04-25T13:51:52.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:52 smithi122 ceph-mon[32843]: pgmap v678: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-04-25T13:51:52.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:52 smithi045 ceph-mon[32731]: pgmap v678: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-04-25T13:51:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 645.0 s, 1630.8 tps, lat 60.926 ms stddev 72.577 2024-04-25T13:51:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:54 smithi027 ceph-mon[29140]: pgmap v679: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 717 op/s 2024-04-25T13:51:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:54 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:51:54.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:54 smithi122 ceph-mon[32843]: pgmap v679: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 717 op/s 2024-04-25T13:51:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:54 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:51:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:54 smithi045 ceph-mon[32731]: pgmap v679: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 717 op/s 2024-04-25T13:51:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:54 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:51:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:56 smithi027 ceph-mon[29140]: pgmap v680: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.8 MiB/s wr, 653 op/s 2024-04-25T13:51:56.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:56 smithi122 ceph-mon[32843]: pgmap v680: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.8 MiB/s wr, 653 op/s 2024-04-25T13:51:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:56 smithi045 ceph-mon[32731]: pgmap v680: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.8 MiB/s wr, 653 op/s 2024-04-25T13:51:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.183+0000 7faed330a640 1 -- 172.21.15.27:0/1775829133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faec409b810 msgr2=0x7faec409dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.183+0000 7faed330a640 1 --2- 172.21.15.27:0/1775829133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faec409b810 0x7faec409dc30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7faec800d910 tx=0x7faec802e5c0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.187+0000 7faed330a640 1 -- 172.21.15.27:0/1775829133 shutdown_connections 2024-04-25T13:51:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.187+0000 7faed330a640 1 --2- 172.21.15.27:0/1775829133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faec409b810 0x7faec409dc30 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.187+0000 7faed330a640 1 --2- 172.21.15.27:0/1775829133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faec4096b10 0x7faec4096f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.187+0000 7faed330a640 1 --2- 172.21.15.27:0/1775829133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec40958d0 0x7faec40965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.187+0000 7faed330a640 1 -- 172.21.15.27:0/1775829133 >> 172.21.15.27:0/1775829133 conn(0x7faec4090ac0 msgr2=0x7faec4092f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.191+0000 7faed330a640 1 -- 172.21.15.27:0/1775829133 shutdown_connections 2024-04-25T13:51:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.195+0000 7faed330a640 1 -- 172.21.15.27:0/1775829133 wait complete. 2024-04-25T13:51:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.196+0000 7faed330a640 1 Processor -- start 2024-04-25T13:51:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.197+0000 7faed330a640 1 -- start start 2024-04-25T13:51:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.198+0000 7faed330a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec40958d0 0x7faec413c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.198+0000 7faed330a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faec4096b10 0x7faec413c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.198+0000 7faed330a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faec409b810 0x7faec413d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.198+0000 7faed330a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faec40b13e0 con 0x7faec409b810 2024-04-25T13:51:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.198+0000 7faed330a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faec40b1260 con 0x7faec40958d0 2024-04-25T13:51:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.198+0000 7faed330a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7faec40b1560 con 0x7faec4096b10 2024-04-25T13:51:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.199+0000 7faed2308640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec40958d0 0x7faec413c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.199+0000 7faed2308640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec40958d0 0x7faec413c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:44092/0 (socket says 172.21.15.27:44092) 2024-04-25T13:51:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.199+0000 7faed2308640 1 -- 172.21.15.27:0/1896989970 learned_addr learned my addr 172.21.15.27:0/1896989970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.201+0000 7faed2b09640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faec409b810 0x7faec413d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.201+0000 7faed2308640 1 -- 172.21.15.27:0/1896989970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faec4096b10 msgr2=0x7faec413c560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:51:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.201+0000 7faed2308640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faec4096b10 0x7faec413c560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.201+0000 7faed2308640 1 -- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faec409b810 msgr2=0x7faec413d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.201+0000 7faed2308640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faec409b810 0x7faec413d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.201+0000 7faed2308640 1 -- 172.21.15.27:0/1896989970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec413db30 con 0x7faec40958d0 2024-04-25T13:51:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.203+0000 7faed2308640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec40958d0 0x7faec413c020 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7faecc0514f0 tx=0x7faecc071820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.205+0000 7faec37fe640 1 -- 172.21.15.27:0/1896989970 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc076590 con 0x7faec40958d0 2024-04-25T13:51:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.205+0000 7faed330a640 1 -- 172.21.15.27:0/1896989970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec413de20 con 0x7faec40958d0 2024-04-25T13:51:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.205+0000 7faed330a640 1 -- 172.21.15.27:0/1896989970 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec413e370 con 0x7faec40958d0 2024-04-25T13:51:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.205+0000 7faec37fe640 1 -- 172.21.15.27:0/1896989970 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faecc076be0 con 0x7faec40958d0 2024-04-25T13:51:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.205+0000 7faec37fe640 1 -- 172.21.15.27:0/1896989970 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc077c20 con 0x7faec40958d0 2024-04-25T13:51:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.210+0000 7faedb417740 1 Processor -- start 2024-04-25T13:51:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.213+0000 7faec37fe640 1 -- 172.21.15.27:0/1896989970 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7faecc07e030 con 0x7faec40958d0 2024-04-25T13:51:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.213+0000 7faedb417740 1 -- start start 2024-04-25T13:51:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.213+0000 7faedb417740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 0x5653c0e84b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.213+0000 7faedb417740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e85090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.214+0000 7faedb417740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ac34d0 0x5653c0e8bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.214+0000 7faedb417740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5653c0dea460 con 0x5653c0ae41b0 2024-04-25T13:51:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.214+0000 7faed2308640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e85090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.214+0000 7faed2308640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e85090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:44096/0 (socket says 172.21.15.27:44096) 2024-04-25T13:51:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.214+0000 7faed2308640 1 -- 172.21.15.27:0/941767542 learned_addr learned my addr 172.21.15.27:0/941767542 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.214+0000 7faedb417740 1 -- 172.21.15.27:0/941767542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5653c0da4590 con 0x5653c0abb2f0 2024-04-25T13:51:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.214+0000 7faedb417740 1 -- 172.21.15.27:0/941767542 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5653c0de6420 con 0x5653c0ac34d0 2024-04-25T13:51:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.215+0000 7faed1b07640 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ac34d0 0x5653c0e8bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.215+0000 7faed2b09640 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 0x5653c0e84b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.216+0000 7faed2b09640 1 -- 172.21.15.27:0/941767542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ac34d0 msgr2=0x5653c0e8bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.216+0000 7faed2b09640 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ac34d0 0x5653c0e8bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.216+0000 7faed2b09640 1 -- 172.21.15.27:0/941767542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 msgr2=0x5653c0e85090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.216+0000 7faed2b09640 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e85090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.216+0000 7faed2b09640 1 -- 172.21.15.27:0/941767542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c0db10d0 con 0x5653c0ae41b0 2024-04-25T13:51:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.220+0000 7faec37fe640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faeb80afd00 0x7faeb80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.220+0000 7faec37fe640 1 -- 172.21.15.27:0/1896989970 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7faecc142640 con 0x7faec40958d0 2024-04-25T13:51:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.221+0000 7faed2b09640 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 0x5653c0e84b10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7faec8004000 tx=0x7faec80040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312b14d94220091b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.222+0000 7faed2308640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faeb80afd00 0x7faeb80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.222+0000 7faec0bfe640 1 -- 172.21.15.27:0/941767542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec800d110 con 0x5653c0ae41b0 2024-04-25T13:51:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.222+0000 7faec0bfe640 1 -- 172.21.15.27:0/941767542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec8044070 con 0x5653c0ae41b0 2024-04-25T13:51:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.223+0000 7faec0bfe640 1 -- 172.21.15.27:0/941767542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8004920 con 0x5653c0ae41b0 2024-04-25T13:51:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.223+0000 7faedb417740 1 -- 172.21.15.27:0/941767542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 msgr2=0x5653c0e84b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.223+0000 7faedb417740 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 0x5653c0e84b10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7faec8004000 tx=0x7faec80040f0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faedb417740 1 -- 172.21.15.27:0/941767542 shutdown_connections 2024-04-25T13:51:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faedb417740 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ac34d0 0x5653c0e8bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faedb417740 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e85090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faedb417740 1 --2- 172.21.15.27:0/941767542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 0x5653c0e84b10 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faedb417740 1 -- 172.21.15.27:0/941767542 >> 172.21.15.27:0/941767542 conn(0x5653c0ad8300 msgr2=0x5653c0e804a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faed2308640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faeb80afd00 0x7faeb80b2210 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7faecc068a60 tx=0x7faecc148060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faedb417740 1 -- 172.21.15.27:0/941767542 shutdown_connections 2024-04-25T13:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faedb417740 1 -- 172.21.15.27:0/941767542 wait complete. 2024-04-25T13:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.224+0000 7faedb417740 1 -- start start 2024-04-25T13:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.225+0000 7faedb417740 1 Processor -- start 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.225+0000 7faedb417740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0abb2f0 0x5653c0e8e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faedb417740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e8ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faedb417740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 0x5653c0e95c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faedb417740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5653c0e8c9d0 con 0x5653c0ae41b0 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faedb417740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5653c0d71b60 con 0x5653c0ac34d0 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faed2b09640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0abb2f0 0x5653c0e8e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faed2b09640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0abb2f0 0x5653c0e8e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40854/0 (socket says 172.21.15.27:40854) 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faed2b09640 1 -- 172.21.15.27:0/3065017465 learned_addr learned my addr 172.21.15.27:0/3065017465 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faedb417740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5653c0e8cb50 con 0x5653c0abb2f0 2024-04-25T13:51:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.226+0000 7faed1b07640 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e8ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faedb417740 10 client.0 _ll_register_callbacks cb 0x5653c0d962b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faed2b09640 1 -- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 msgr2=0x5653c0e8ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faed2b09640 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e8ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faed2b09640 1 -- 172.21.15.27:0/3065017465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 msgr2=0x5653c0e95c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faed2b09640 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 0x5653c0e95c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faed2b09640 1 -- 172.21.15.27:0/3065017465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c0afc210 con 0x5653c0abb2f0 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faed1b07640 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e8ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faed2b09640 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0abb2f0 0x5653c0e8e830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7faec802f070 tx=0x7faec80121f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=874d6a4eb9c93c85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faedb417740 1 -- 172.21.15.27:0/3065017465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec8044070 con 0x5653c0abb2f0 2024-04-25T13:51:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.227+0000 7faeb5ffb640 10 client.27343 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.228+0000 7faeb5ffb640 1 -- 172.21.15.27:0/3065017465 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8058050 con 0x5653c0abb2f0 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.228+0000 7faeb5ffb640 1 -- 172.21.15.27:0/3065017465 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7faec80042e0 con 0x5653c0abb2f0 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.228+0000 7faeb5ffb640 1 -- 172.21.15.27:0/3065017465 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7faec804d030 con 0x5653c0abb2f0 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.228+0000 7faedb417740 10 client.27343 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.228+0000 7faedb417740 1 -- 172.21.15.27:0/3065017465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x5653c0bc8620 con 0x5653c0abb2f0 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.229+0000 7faeb5ffb640 1 -- 172.21.15.27:0/3065017465 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7faec8003800 con 0x5653c0abb2f0 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.229+0000 7faedb417740 10 client.27343 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.229+0000 7faedb417740 10 client.27343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.229+0000 7faedb417740 20 client.27343 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.229+0000 7faedb417740 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653c0aa2d80 0x5653c0e9bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.229+0000 7faedb417740 4 client.27343 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.229+0000 7faedb417740 1 -- 172.21.15.27:0/3065017465 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7faec8044070 con 0x5653c0aa2d80 2024-04-25T13:51:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.229+0000 7faed1b07640 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653c0aa2d80 0x5653c0e9bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.230+0000 7faeb5ffb640 10 client.27343 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.230+0000 7faed1b07640 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653c0aa2d80 0x5653c0e9bf50 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9724cb258587cd server_cookie=b8dca6ac59153e2d in_seq=0 out_seq=0 2024-04-25T13:51:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.235+0000 7faeb5ffb640 1 -- 172.21.15.27:0/3065017465 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7faec8044070 con 0x5653c0aa2d80 2024-04-25T13:51:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.235+0000 7faeb5ffb640 10 client.27343 handle_command_reply: tid=0 2024-04-25T13:51:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.235+0000 7faedb417740 1 client.27343 shutdown 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 -- 172.21.15.27:0/3065017465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0abb2f0 msgr2=0x5653c0e8e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0abb2f0 0x5653c0e8e830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7faec802f070 tx=0x7faec80121f0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 -- 172.21.15.27:0/3065017465 shutdown_connections 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653c0aa2d80 0x5653c0e9bf50 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ae41b0 0x5653c0e95c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e8ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 --2- 172.21.15.27:0/3065017465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0abb2f0 0x5653c0e8e830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 -- 172.21.15.27:0/3065017465 >> 172.21.15.27:0/3065017465 conn(0x5653c0ad8300 msgr2=0x5653c0e833b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faeb5ffb640 0 client.27343 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 -- 172.21.15.27:0/3065017465 shutdown_connections 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 1 -- 172.21.15.27:0/3065017465 wait complete. 2024-04-25T13:51:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.236+0000 7faedb417740 20 client.27343 trim_cache size 0 max 16384 2024-04-25T13:51:57.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.258+0000 7faedb417740 1 Processor -- start 2024-04-25T13:51:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.258+0000 7faedb417740 1 -- start start 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.258+0000 7faedb417740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0abb2f0 0x5653c0e8eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.258+0000 7faedb417740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e839b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.258+0000 7faedb417740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0e83ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.258+0000 7faedb417740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faeb000c450 con 0x5653c0abb2f0 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.258+0000 7faedb417740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faeb000ecd0 con 0x5653c0ac34d0 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.258+0000 7faedb417740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7faeb000cac0 con 0x5653c0ae41b0 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0e83ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0e83ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:40868/0 (socket says 172.21.15.27:40868) 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 -- 172.21.15.27:0/3524229081 learned_addr learned my addr 172.21.15.27:0/3524229081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0abb2f0 0x5653c0e8eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed1b07640 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e839b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 -- 172.21.15.27:0/3524229081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 msgr2=0x5653c0e839b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e839b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 -- 172.21.15.27:0/3524229081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0abb2f0 msgr2=0x5653c0e8eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0abb2f0 0x5653c0e8eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 -- 172.21.15.27:0/3524229081 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c0e8c350 con 0x5653c0ae41b0 2024-04-25T13:51:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.259+0000 7faed2b09640 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0e83ef0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7faec800d910 tx=0x7faec8049000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=960755fa1f202db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.260+0000 7faeb7fff640 1 -- 172.21.15.27:0/3524229081 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8058070 con 0x5653c0ae41b0 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.260+0000 7faeb7fff640 1 -- 172.21.15.27:0/3524229081 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec8044070 con 0x5653c0ae41b0 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.260+0000 7faeb7fff640 1 -- 172.21.15.27:0/3524229081 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8011070 con 0x5653c0ae41b0 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 -- 172.21.15.27:0/3524229081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 msgr2=0x5653c0e83ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0e83ef0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7faec800d910 tx=0x7faec8049000 comp rx=0 tx=0).stop 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 -- 172.21.15.27:0/3524229081 shutdown_connections 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0e83ef0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0ac34d0 0x5653c0e839b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 --2- 172.21.15.27:0/3524229081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0abb2f0 0x5653c0e8eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 -- 172.21.15.27:0/3524229081 >> 172.21.15.27:0/3524229081 conn(0x5653c0ad8300 msgr2=0x5653c0e832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 -- 172.21.15.27:0/3524229081 shutdown_connections 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.261+0000 7faedb417740 1 -- 172.21.15.27:0/3524229081 wait complete. 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 1 -- start start 2024-04-25T13:51:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 1 Processor -- start 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e94810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ac34d0 0x5653c0e94d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0f025a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7faeb000d190 con 0x5653c0ac34d0 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faeb000e600 con 0x5653c0abb2f0 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7faeb000d860 con 0x5653c0ae41b0 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faedb417740 10 client.0 _ll_register_callbacks cb 0x7faecc145ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faed1b07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ac34d0 0x5653c0e94d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faed1b07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ac34d0 0x5653c0e94d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54382/0 (socket says 172.21.15.27:54382) 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faed1b07640 1 -- 172.21.15.27:0/4088875490 learned_addr learned my addr 172.21.15.27:0/4088875490 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faed2b09640 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0f025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.262+0000 7faed2b09640 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e94810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faed1b07640 1 -- 172.21.15.27:0/4088875490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 msgr2=0x5653c0f025a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faed1b07640 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0f025a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faed1b07640 1 -- 172.21.15.27:0/4088875490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 msgr2=0x5653c0e94810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faed1b07640 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e94810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faed1b07640 1 -- 172.21.15.27:0/4088875490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c0e89640 con 0x5653c0ac34d0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faed1b07640 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ac34d0 0x5653c0e94d50 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x5653c0e81230 tx=0x7faebc009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6f40c95c540d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faeb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faeb5ffb640 1 -- 172.21.15.27:0/4088875490 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc027060 con 0x5653c0ac34d0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faeb5ffb640 1 -- 172.21.15.27:0/4088875490 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7faebc0063d0 con 0x5653c0ac34d0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faedb417740 1 -- 172.21.15.27:0/4088875490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec8044070 con 0x5653c0ac34d0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faeb5ffb640 1 -- 172.21.15.27:0/4088875490 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7faebc00f040 con 0x5653c0ac34d0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faedb417740 10 client.17613 fetch_fsmap learned FSMap version 19 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.263+0000 7faedb417740 1 -- 172.21.15.27:0/4088875490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5653c0e8c8c0 con 0x5653c0ac34d0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faeb5ffb640 1 -- 172.21.15.27:0/4088875490 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec8044070 con 0x5653c0ac34d0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faeb5ffb640 1 -- 172.21.15.27:0/4088875490 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7faebc015420 con 0x5653c0ac34d0 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faedb417740 10 client.17613 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faedb417740 10 client.17613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:51:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faedb417740 20 client.17613 populate_metadata read hostname 'smithi027' 2024-04-25T13:51:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faedb417740 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653c0a781c0 0x5653c0f06520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:51:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faedb417740 4 client.17613 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:51:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faedb417740 1 -- 172.21.15.27:0/4088875490 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5653c0df8070 con 0x5653c0a781c0 2024-04-25T13:51:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.264+0000 7faed2b09640 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653c0a781c0 0x5653c0f06520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:51:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.265+0000 7faed2b09640 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653c0a781c0 0x5653c0f06520 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40d183e873112225 server_cookie=477a572c43fa06a2 in_seq=0 out_seq=0 2024-04-25T13:51:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.265+0000 7faeb5ffb640 10 client.17613 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faeb5ffb640 1 -- 172.21.15.27:0/4088875490 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5653c0df8070 con 0x5653c0a781c0 2024-04-25T13:51:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faeb5ffb640 10 client.17613 handle_command_reply: tid=0 2024-04-25T13:51:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 client.17613 shutdown 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 -- 172.21.15.27:0/4088875490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ac34d0 msgr2=0x5653c0e94d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ac34d0 0x5653c0e94d50 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x5653c0e81230 tx=0x7faebc009f60 comp rx=0 tx=0).stop 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 -- 172.21.15.27:0/4088875490 shutdown_connections 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5653c0a781c0 0x5653c0f06520 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5653c0ae41b0 0x5653c0f025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5653c0ac34d0 0x5653c0e94d50 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 --2- 172.21.15.27:0/4088875490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5653c0abb2f0 0x5653c0e94810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 -- 172.21.15.27:0/4088875490 >> 172.21.15.27:0/4088875490 conn(0x5653c0ad8300 msgr2=0x5653c0e83cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faeb5ffb640 0 client.17613 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 -- 172.21.15.27:0/4088875490 shutdown_connections 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 1 -- 172.21.15.27:0/4088875490 wait complete. 2024-04-25T13:51:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.283+0000 7faedb417740 20 client.17613 trim_cache size 0 max 16384 2024-04-25T13:51:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:51:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:51:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:51:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 -- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faeb80afd00 msgr2=0x7faeb80b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faeb80afd00 0x7faeb80b2210 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7faecc068a60 tx=0x7faecc148060 comp rx=0 tx=0).stop 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 -- 172.21.15.27:0/1896989970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec40958d0 msgr2=0x7faec413c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec40958d0 0x7faec413c020 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7faecc0514f0 tx=0x7faecc071820 comp rx=0 tx=0).stop 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 -- 172.21.15.27:0/1896989970 shutdown_connections 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7faeb80afd00 0x7faeb80b2210 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faec409b810 0x7faec413d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faec4096b10 0x7faec413c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 --2- 172.21.15.27:0/1896989970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faec40958d0 0x7faec413c020 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 -- 172.21.15.27:0/1896989970 >> 172.21.15.27:0/1896989970 conn(0x7faec4090ac0 msgr2=0x7faec40977f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 -- 172.21.15.27:0/1896989970 shutdown_connections 2024-04-25T13:51:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:51:57.284+0000 7faed330a640 1 -- 172.21.15.27:0/1896989970 wait complete. 2024-04-25T13:51:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 650.0 s, 2008.8 tps, lat 49.735 ms stddev 57.990 2024-04-25T13:51:58.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:51:58 smithi027 ceph-mon[29140]: pgmap v681: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 6.9 MiB/s wr, 751 op/s 2024-04-25T13:51:58.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:51:58 smithi122 ceph-mon[32843]: pgmap v681: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 6.9 MiB/s wr, 751 op/s 2024-04-25T13:51:58.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:51:58 smithi045 ceph-mon[32731]: pgmap v681: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 6.9 MiB/s wr, 751 op/s 2024-04-25T13:52:00.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:00 smithi027 ceph-mon[29140]: pgmap v682: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.5 MiB/s wr, 673 op/s 2024-04-25T13:52:00.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:00 smithi122 ceph-mon[32843]: pgmap v682: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.5 MiB/s wr, 673 op/s 2024-04-25T13:52:00.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:00 smithi045 ceph-mon[32731]: pgmap v682: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.5 MiB/s wr, 673 op/s 2024-04-25T13:52:02.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:02 smithi027 ceph-mon[29140]: pgmap v683: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 7.0 MiB/s wr, 742 op/s 2024-04-25T13:52:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:02 smithi122 ceph-mon[32843]: pgmap v683: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 7.0 MiB/s wr, 742 op/s 2024-04-25T13:52:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:02 smithi045 ceph-mon[32731]: pgmap v683: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 7.0 MiB/s wr, 742 op/s 2024-04-25T13:52:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 655.0 s, 2097.6 tps, lat 47.455 ms stddev 59.144 2024-04-25T13:52:04.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:04 smithi122 ceph-mon[32843]: pgmap v684: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.3 MiB/s wr, 783 op/s 2024-04-25T13:52:04.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:04 smithi045 ceph-mon[32731]: pgmap v684: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.3 MiB/s wr, 783 op/s 2024-04-25T13:52:04.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:04 smithi027 ceph-mon[29140]: pgmap v684: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.3 MiB/s wr, 783 op/s 2024-04-25T13:52:06.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:06 smithi122 ceph-mon[32843]: pgmap v685: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-04-25T13:52:06.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:06 smithi045 ceph-mon[32731]: pgmap v685: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-04-25T13:52:06.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:06 smithi027 ceph-mon[29140]: pgmap v685: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 663 op/s 2024-04-25T13:52:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 660.0 s, 1345.8 tps, lat 75.054 ms stddev 86.072 2024-04-25T13:52:08.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:08 smithi122 ceph-mon[32843]: pgmap v686: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 747 op/s 2024-04-25T13:52:08.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:08 smithi045 ceph-mon[32731]: pgmap v686: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 747 op/s 2024-04-25T13:52:08.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:08 smithi027 ceph-mon[29140]: pgmap v686: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 747 op/s 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 -- 172.21.15.27:0/3286387189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 msgr2=0x7f941810e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 --2- 172.21.15.27:0/3286387189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 0x7f941810e090 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9410009ca0 tx=0x7f941002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 -- 172.21.15.27:0/3286387189 shutdown_connections 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 --2- 172.21.15.27:0/3286387189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 0x7f941810e090 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 --2- 172.21.15.27:0/3286387189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9418106f70 0x7f9418107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 --2- 172.21.15.27:0/3286387189 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9418105d30 0x7f9418106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 -- 172.21.15.27:0/3286387189 >> 172.21.15.27:0/3286387189 conn(0x7f9418100f00 msgr2=0x7f94181033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 -- 172.21.15.27:0/3286387189 shutdown_connections 2024-04-25T13:52:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.444+0000 7f941cdd7640 1 -- 172.21.15.27:0/3286387189 wait complete. 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.445+0000 7f941cdd7640 1 Processor -- start 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.445+0000 7f941cdd7640 1 -- start start 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941cdd7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9418105d30 0x7f941819c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941cdd7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9418106f70 0x7f941819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941cdd7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 0x7f94181a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941cdd7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f94181118b0 con 0x7f941810bc70 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941cdd7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9418111730 con 0x7f9418105d30 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941cdd7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9418111a30 con 0x7f9418106f70 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f9415d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9418106f70 0x7f941819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9418105d30 0x7f941819c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9418105d30 0x7f941819c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38320/0 (socket says 172.21.15.27:38320) 2024-04-25T13:52:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f9415d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9418106f70 0x7f941819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:59956/0 (socket says 172.21.15.27:59956) 2024-04-25T13:52:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941659c640 1 -- 172.21.15.27:0/2771894409 learned_addr learned my addr 172.21.15.27:0/2771894409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f9416d9d640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 0x7f94181a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941659c640 1 -- 172.21.15.27:0/2771894409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9418106f70 msgr2=0x7f941819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941659c640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9418106f70 0x7f941819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941659c640 1 -- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 msgr2=0x7f94181a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941659c640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 0x7f94181a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f941659c640 1 -- 172.21.15.27:0/2771894409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94181a3f00 con 0x7f9418105d30 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.446+0000 7f9416d9d640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 0x7f94181a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.447+0000 7f941659c640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9418105d30 0x7f941819c3c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f940800e9f0 tx=0x7f940800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.447+0000 7f94077fe640 1 -- 172.21.15.27:0/2771894409 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940800ccc0 con 0x7f9418105d30 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.447+0000 7f941cdd7640 1 -- 172.21.15.27:0/2771894409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94181a41f0 con 0x7f9418105d30 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.447+0000 7f941cdd7640 1 -- 172.21.15.27:0/2771894409 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94181a4740 con 0x7f9418105d30 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.447+0000 7f94077fe640 1 -- 172.21.15.27:0/2771894409 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94080035f0 con 0x7f9418105d30 2024-04-25T13:52:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.447+0000 7f94077fe640 1 -- 172.21.15.27:0/2771894409 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940800ce60 con 0x7f9418105d30 2024-04-25T13:52:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.450+0000 7f9423d66740 1 Processor -- start 2024-04-25T13:52:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.450+0000 7f94077fe640 1 -- 172.21.15.27:0/2771894409 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9408010690 con 0x7f9418105d30 2024-04-25T13:52:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.451+0000 7f9423d66740 1 -- start start 2024-04-25T13:52:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.451+0000 7f94077fe640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94000afb60 0x7f94000b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.451+0000 7f9423d66740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9265970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.451+0000 7f9423d66740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea9265ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.451+0000 7f9423d66740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9423d66740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562ea91cb460 con 0x562ea8ec51b0 2024-04-25T13:52:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9416d9d640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94000afb60 0x7f94000b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9423d66740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562ea9185590 con 0x562ea8e9c2f0 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9423d66740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562ea91c7420 con 0x562ea8ea44d0 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9265970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9265970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57892/0 (socket says 172.21.15.27:57892) 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 -- 172.21.15.27:0/2793229703 learned_addr learned my addr 172.21.15.27:0/2793229703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9415d9b640 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea9265ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f94077fe640 1 -- 172.21.15.27:0/2771894409 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f94080dee60 con 0x7f9418105d30 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9416d9d640 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 -- 172.21.15.27:0/2793229703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 msgr2=0x562ea926cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 -- 172.21.15.27:0/2793229703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 msgr2=0x562ea9265ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea9265ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 -- 172.21.15.27:0/2793229703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ea91920d0 con 0x562ea8ec51b0 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9416d9d640 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9415d9b640 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea9265ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9416d9d640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94000afb60 0x7f94000b2070 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f94100094e0 tx=0x7f941002f450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f941659c640 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9265970 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f94080dc8d0 tx=0x7f94080a6ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283cfbe35686a530 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.452+0000 7f9404ff9640 1 -- 172.21.15.27:0/2793229703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94080a6e80 con 0x562ea8ec51b0 2024-04-25T13:52:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9404ff9640 1 -- 172.21.15.27:0/2793229703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94080dce50 con 0x562ea8ec51b0 2024-04-25T13:52:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9404ff9640 1 -- 172.21.15.27:0/2793229703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94080a36f0 con 0x562ea8ec51b0 2024-04-25T13:52:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9423d66740 1 -- 172.21.15.27:0/2793229703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 msgr2=0x562ea9265970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9423d66740 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9265970 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f94080dc8d0 tx=0x7f94080a6ae0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9423d66740 1 -- 172.21.15.27:0/2793229703 shutdown_connections 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9423d66740 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9423d66740 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea9265ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9423d66740 1 --2- 172.21.15.27:0/2793229703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9265970 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.453+0000 7f9423d66740 1 -- 172.21.15.27:0/2793229703 >> 172.21.15.27:0/2793229703 conn(0x562ea8eb9300 msgr2=0x562ea9261300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 -- 172.21.15.27:0/2793229703 shutdown_connections 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 -- 172.21.15.27:0/2793229703 wait complete. 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 -- start start 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 Processor -- start 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea926f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9276b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562ea926d840 con 0x562ea8ec51b0 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562ea9152b60 con 0x562ea8e9c2f0 2024-04-25T13:52:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562ea926d9c0 con 0x562ea8ea44d0 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.454+0000 7f9423d66740 10 client.0 _ll_register_callbacks cb 0x562ea91772b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea926f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f9416d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9276b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea926f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38332/0 (socket says 172.21.15.27:38332) 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 -- 172.21.15.27:0/2456718606 learned_addr learned my addr 172.21.15.27:0/2456718606 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f9415d9b640 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 -- 172.21.15.27:0/2456718606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 msgr2=0x562ea926fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 -- 172.21.15.27:0/2456718606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 msgr2=0x562ea9276b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9276b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 -- 172.21.15.27:0/2456718606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ea8edd210 con 0x562ea8e9c2f0 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f9415d9b640 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f9416d9d640 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9276b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f941659c640 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea926f6b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f940808c650 tx=0x7f94080a6ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be4c70d69fd9738a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f93fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f93fdffb640 1 -- 172.21.15.27:0/2456718606 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94080a7370 con 0x562ea8e9c2f0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f93fdffb640 1 -- 172.21.15.27:0/2456718606 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f94080a3960 con 0x562ea8e9c2f0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f9423d66740 1 -- 172.21.15.27:0/2456718606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94080dce50 con 0x562ea8e9c2f0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f93fdffb640 1 -- 172.21.15.27:0/2456718606 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f94080ab720 con 0x562ea8e9c2f0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.455+0000 7f9423d66740 10 client.27314 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f9423d66740 1 -- 172.21.15.27:0/2456718606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562ea8fa9620 con 0x562ea8e9c2f0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f93fdffb640 1 -- 172.21.15.27:0/2456718606 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94080dce50 con 0x562ea8e9c2f0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f93fdffb640 1 -- 172.21.15.27:0/2456718606 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f94080a2ce0 con 0x562ea8e9c2f0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f9423d66740 10 client.27314 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f9423d66740 10 client.27314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f9423d66740 20 client.27314 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f9423d66740 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562ea8e83d80 0x562ea927cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f9423d66740 4 client.27314 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f9423d66740 1 -- 172.21.15.27:0/2456718606 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ea91d9070 con 0x562ea8e83d80 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.456+0000 7f9415d9b640 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562ea8e83d80 0x562ea927cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.458+0000 7f9415d9b640 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562ea8e83d80 0x562ea927cf10 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2c22fc04a94b7d2 server_cookie=112b20ce88417a74 in_seq=0 out_seq=0 2024-04-25T13:52:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.458+0000 7f93fdffb640 10 client.27314 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.458+0000 7f93fdffb640 1 -- 172.21.15.27:0/2456718606 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562ea91d9070 con 0x562ea8e83d80 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.458+0000 7f93fdffb640 10 client.27314 handle_command_reply: tid=0 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.458+0000 7f9423d66740 1 client.27314 shutdown 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.458+0000 7f9423d66740 1 -- 172.21.15.27:0/2456718606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 msgr2=0x562ea926f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.458+0000 7f9423d66740 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea926f6b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f940808c650 tx=0x7f94080a6ef0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 1 -- 172.21.15.27:0/2456718606 shutdown_connections 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562ea8e83d80 0x562ea927cf10 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8ec51b0 0x562ea9276b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea926fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f93fdffb640 0 client.27314 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 1 --2- 172.21.15.27:0/2456718606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e9c2f0 0x562ea926f6b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 1 -- 172.21.15.27:0/2456718606 >> 172.21.15.27:0/2456718606 conn(0x562ea8eb9300 msgr2=0x562ea9260250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 1 -- 172.21.15.27:0/2456718606 shutdown_connections 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 1 -- 172.21.15.27:0/2456718606 wait complete. 2024-04-25T13:52:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.459+0000 7f9423d66740 20 client.27314 trim_cache size 0 max 16384 2024-04-25T13:52:10.480 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.480+0000 7f9423d66740 1 Processor -- start 2024-04-25T13:52:10.480 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.480+0000 7f9423d66740 1 -- start start 2024-04-25T13:52:10.480 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.480+0000 7f9423d66740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8e83d80 0x562ea9260ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.480+0000 7f9423d66740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea9276ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.480+0000 7f9423d66740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8ea44d0 0x562ea92610e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9423d66740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93f800c340 con 0x562ea8e9c2f0 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9423d66740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93f800ebc0 con 0x562ea8ea44d0 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9416d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8ea44d0 0x562ea92610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9416d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8ea44d0 0x562ea92610e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:38346/0 (socket says 172.21.15.27:38346) 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9416d9d640 1 -- 172.21.15.27:0/3879948046 learned_addr learned my addr 172.21.15.27:0/3879948046 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9423d66740 1 -- 172.21.15.27:0/3879948046 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f93f800c9b0 con 0x562ea8e83d80 2024-04-25T13:52:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f941659c640 1 --2- 172.21.15.27:0/3879948046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8e83d80 0x562ea9260ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9416d9d640 1 -- 172.21.15.27:0/3879948046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8e83d80 msgr2=0x562ea9260ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9416d9d640 1 --2- 172.21.15.27:0/3879948046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8e83d80 0x562ea9260ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9416d9d640 1 -- 172.21.15.27:0/3879948046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 msgr2=0x562ea9276ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9416d9d640 1 --2- 172.21.15.27:0/3879948046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea9276ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.481+0000 7f9416d9d640 1 -- 172.21.15.27:0/3879948046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ea9261e90 con 0x562ea8ea44d0 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.482+0000 7f9416d9d640 1 --2- 172.21.15.27:0/3879948046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8ea44d0 0x562ea92610e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9410031980 tx=0x7f941003f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a9a29ded3fefd77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.482+0000 7f93fffff640 1 -- 172.21.15.27:0/3879948046 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9410054070 con 0x562ea8ea44d0 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.482+0000 7f93fffff640 1 -- 172.21.15.27:0/3879948046 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9410044070 con 0x562ea8ea44d0 2024-04-25T13:52:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.482+0000 7f93fffff640 1 -- 172.21.15.27:0/3879948046 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941002faa0 con 0x562ea8ea44d0 2024-04-25T13:52:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.483+0000 7f9423d66740 1 -- 172.21.15.27:0/3879948046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8ea44d0 msgr2=0x562ea92610e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.483+0000 7f9423d66740 1 --2- 172.21.15.27:0/3879948046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8ea44d0 0x562ea92610e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9410031980 tx=0x7f941003f040 comp rx=0 tx=0).stop 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.483+0000 7f9423d66740 1 -- 172.21.15.27:0/3879948046 shutdown_connections 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.483+0000 7f9423d66740 1 --2- 172.21.15.27:0/3879948046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8ea44d0 0x562ea92610e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 --2- 172.21.15.27:0/3879948046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea9276ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 --2- 172.21.15.27:0/3879948046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8e83d80 0x562ea9260ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 -- 172.21.15.27:0/3879948046 >> 172.21.15.27:0/3879948046 conn(0x562ea8eb9300 msgr2=0x562ea9277510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 -- 172.21.15.27:0/3879948046 shutdown_connections 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 -- 172.21.15.27:0/3879948046 wait complete. 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 -- start start 2024-04-25T13:52:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 Processor -- start 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e83d80 0x562ea9273ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea92740e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea92e3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.484+0000 7f9423d66740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93f800d080 con 0x562ea8e9c2f0 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9415d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea92740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9423d66740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93f800e4f0 con 0x562ea8e83d80 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9423d66740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f93f800d750 con 0x562ea8ea44d0 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f941659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e83d80 0x562ea9273ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9415d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea92740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57908/0 (socket says 172.21.15.27:57908) 2024-04-25T13:52:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9415d9b640 1 -- 172.21.15.27:0/281328164 learned_addr learned my addr 172.21.15.27:0/281328164 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9423d66740 10 client.0 _ll_register_callbacks cb 0x7f940c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9416d9d640 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea92e3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9415d9b640 1 -- 172.21.15.27:0/281328164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 msgr2=0x562ea92e3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9415d9b640 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea92e3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9415d9b640 1 -- 172.21.15.27:0/281328164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e83d80 msgr2=0x562ea9273ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9415d9b640 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e83d80 0x562ea9273ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.485+0000 7f9415d9b640 1 -- 172.21.15.27:0/281328164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ea92e37a0 con 0x562ea8e9c2f0 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.486+0000 7f9415d9b640 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea92740e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x562ea9262090 tx=0x7f940c011960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2a45109b922508 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.486+0000 7f93fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.486+0000 7f93fdffb640 1 -- 172.21.15.27:0/281328164 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c0205a0 con 0x562ea8e9c2f0 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.486+0000 7f93fdffb640 1 -- 172.21.15.27:0/281328164 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f940c020740 con 0x562ea8e9c2f0 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.486+0000 7f9423d66740 1 -- 172.21.15.27:0/281328164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9410044070 con 0x562ea8e9c2f0 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f93fdffb640 1 -- 172.21.15.27:0/281328164 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f940c020940 con 0x562ea8e9c2f0 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f9423d66740 10 client.17628 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f9423d66740 1 -- 172.21.15.27:0/281328164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562ea92ed320 con 0x562ea8e9c2f0 2024-04-25T13:52:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f93fdffb640 1 -- 172.21.15.27:0/281328164 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f940c030070 con 0x562ea8e9c2f0 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f93fdffb640 1 -- 172.21.15.27:0/281328164 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f940c01e420 con 0x562ea8e9c2f0 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f9423d66740 10 client.17628 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f9423d66740 10 client.17628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f9423d66740 20 client.17628 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f9423d66740 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562ea8ec51b0 0x562ea92efd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f9423d66740 4 client.17628 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.487+0000 7f9423d66740 1 -- 172.21.15.27:0/281328164 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562ea91d2ee0 con 0x562ea8ec51b0 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.488+0000 7f941659c640 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562ea8ec51b0 0x562ea92efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.488+0000 7f941659c640 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562ea8ec51b0 0x562ea92efd40 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dfa586cc59f828 server_cookie=688bdee74b805741 in_seq=0 out_seq=0 2024-04-25T13:52:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.489+0000 7f93fdffb640 10 client.17628 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.512+0000 7f93fdffb640 1 -- 172.21.15.27:0/281328164 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562ea91d2ee0 con 0x562ea8ec51b0 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.512+0000 7f93fdffb640 10 client.17628 handle_command_reply: tid=0 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.512+0000 7f9423d66740 1 client.17628 shutdown 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 -- 172.21.15.27:0/281328164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 msgr2=0x562ea92740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea92740e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x562ea9262090 tx=0x7f940c011960 comp rx=0 tx=0).stop 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 -- 172.21.15.27:0/281328164 shutdown_connections 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562ea8ec51b0 0x562ea92efd40 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562ea8ea44d0 0x562ea92e3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.513 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562ea8e9c2f0 0x562ea92740e0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f93fdffb640 0 client.17628 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 --2- 172.21.15.27:0/281328164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562ea8e83d80 0x562ea9273ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 -- 172.21.15.27:0/281328164 >> 172.21.15.27:0/281328164 conn(0x562ea8eb9300 msgr2=0x562ea9277510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 -- 172.21.15.27:0/281328164 shutdown_connections 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 1 -- 172.21.15.27:0/281328164 wait complete. 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f9423d66740 20 client.17628 trim_cache size 0 max 16384 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:52:10.514 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:52:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:52:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 -- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94000afb60 msgr2=0x7f94000b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94000afb60 0x7f94000b2070 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f94100094e0 tx=0x7f941002f450 comp rx=0 tx=0).stop 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 -- 172.21.15.27:0/2771894409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9418105d30 msgr2=0x7f941819c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9418105d30 0x7f941819c3c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f940800e9f0 tx=0x7f940800eeb0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 -- 172.21.15.27:0/2771894409 shutdown_connections 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f94000afb60 0x7f94000b2070 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941810bc70 0x7f94181a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9418106f70 0x7f941819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 --2- 172.21.15.27:0/2771894409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9418105d30 0x7f941819c3c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.513+0000 7f941cdd7640 1 -- 172.21.15.27:0/2771894409 >> 172.21.15.27:0/2771894409 conn(0x7f9418100f00 msgr2=0x7f9418107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.514+0000 7f941cdd7640 1 -- 172.21.15.27:0/2771894409 shutdown_connections 2024-04-25T13:52:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:10.514+0000 7f941cdd7640 1 -- 172.21.15.27:0/2771894409 wait complete. 2024-04-25T13:52:10.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:10 smithi122 ceph-mon[32843]: pgmap v687: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 621 op/s 2024-04-25T13:52:10.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:10 smithi045 ceph-mon[32731]: pgmap v687: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 621 op/s 2024-04-25T13:52:10.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:10 smithi027 ceph-mon[29140]: pgmap v687: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 621 op/s 2024-04-25T13:52:12.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:12 smithi122 ceph-mon[32843]: pgmap v688: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.3 MiB/s wr, 645 op/s 2024-04-25T13:52:12.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:12 smithi045 ceph-mon[32731]: pgmap v688: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.3 MiB/s wr, 645 op/s 2024-04-25T13:52:12.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:12 smithi027 ceph-mon[29140]: pgmap v688: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.3 MiB/s wr, 645 op/s 2024-04-25T13:52:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 665.0 s, 2015.2 tps, lat 49.216 ms stddev 58.198 2024-04-25T13:52:14.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:14 smithi122 ceph-mon[32843]: pgmap v689: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.0 MiB/s wr, 636 op/s 2024-04-25T13:52:14.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:14 smithi045 ceph-mon[32731]: pgmap v689: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.0 MiB/s wr, 636 op/s 2024-04-25T13:52:14.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:14 smithi027 ceph-mon[29140]: pgmap v689: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.0 MiB/s wr, 636 op/s 2024-04-25T13:52:16.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:16 smithi122 ceph-mon[32843]: pgmap v690: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.4 MiB/s wr, 625 op/s 2024-04-25T13:52:16.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:16 smithi045 ceph-mon[32731]: pgmap v690: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.4 MiB/s wr, 625 op/s 2024-04-25T13:52:16.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:16 smithi027 ceph-mon[29140]: pgmap v690: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.4 MiB/s wr, 625 op/s 2024-04-25T13:52:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 670.0 s, 2083.6 tps, lat 48.431 ms stddev 56.437 2024-04-25T13:52:18.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:18 smithi122 ceph-mon[32843]: pgmap v691: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.3 MiB/s wr, 666 op/s 2024-04-25T13:52:18.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:18 smithi045 ceph-mon[32731]: pgmap v691: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.3 MiB/s wr, 666 op/s 2024-04-25T13:52:18.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:18 smithi027 ceph-mon[29140]: pgmap v691: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.3 MiB/s wr, 666 op/s 2024-04-25T13:52:20.451 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:20 smithi027 ceph-mon[29140]: pgmap v692: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-04-25T13:52:20.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:52:20.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:52:20.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:20 smithi122 ceph-mon[32843]: pgmap v692: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-04-25T13:52:20.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:20 smithi045 ceph-mon[32731]: pgmap v692: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-04-25T13:52:20.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.653+0000 7f63c5015640 1 -- 172.21.15.27:0/4020706887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c010a420 msgr2=0x7f63c010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.653+0000 7f63c5015640 1 --2- 172.21.15.27:0/4020706887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c010a420 0x7f63c010add0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009860 tx=0x7f63b002a360 comp rx=0 tx=0).stop 2024-04-25T13:52:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.654+0000 7f63c5015640 1 -- 172.21.15.27:0/4020706887 shutdown_connections 2024-04-25T13:52:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.654+0000 7f63c5015640 1 --2- 172.21.15.27:0/4020706887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63c0110030 0x7f63c0110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.654+0000 7f63c5015640 1 --2- 172.21.15.27:0/4020706887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c010b310 0x7f63c010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.654+0000 7f63c5015640 1 --2- 172.21.15.27:0/4020706887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c010a420 0x7f63c010add0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.654+0000 7f63c5015640 1 -- 172.21.15.27:0/4020706887 >> 172.21.15.27:0/4020706887 conn(0x7f63c00704e0 msgr2=0x7f63c0072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.656+0000 7f63c5015640 1 -- 172.21.15.27:0/4020706887 shutdown_connections 2024-04-25T13:52:20.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.657+0000 7f63c5015640 1 -- 172.21.15.27:0/4020706887 wait complete. 2024-04-25T13:52:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.658+0000 7f63c5015640 1 Processor -- start 2024-04-25T13:52:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.659+0000 7f63c5015640 1 -- start start 2024-04-25T13:52:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.659+0000 7f63c5015640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63c010a420 0x7f63c011b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.659+0000 7f63c5015640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c010b310 0x7f63c011a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.659+0000 7f63c5015640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c0110030 0x7f63c011a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.659+0000 7f63c5015640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63c0112380 con 0x7f63c0110030 2024-04-25T13:52:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.659+0000 7f63c5015640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63c0112200 con 0x7f63c010a420 2024-04-25T13:52:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.659+0000 7f63c5015640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f63c0112500 con 0x7f63c010b310 2024-04-25T13:52:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.660+0000 7f63bf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c0110030 0x7f63c011a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.660+0000 7f63bf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c0110030 0x7f63c011a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43060/0 (socket says 172.21.15.27:43060) 2024-04-25T13:52:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.660+0000 7f63bf59e640 1 -- 172.21.15.27:0/510813337 learned_addr learned my addr 172.21.15.27:0/510813337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.660+0000 7f63bf59e640 1 -- 172.21.15.27:0/510813337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c010b310 msgr2=0x7f63c011a090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:52:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.660+0000 7f63bf59e640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c010b310 0x7f63c011a090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.660+0000 7f63bf59e640 1 -- 172.21.15.27:0/510813337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63c010a420 msgr2=0x7f63c011b9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.660+0000 7f63bf59e640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63c010a420 0x7f63c011b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.660+0000 7f63bf59e640 1 -- 172.21.15.27:0/510813337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c011ab10 con 0x7f63c0110030 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.661+0000 7f63bf59e640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c0110030 0x7f63c011a5d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f63b800cb00 tx=0x7f63b800a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.661+0000 7f63a7fff640 1 -- 172.21.15.27:0/510813337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8004b20 con 0x7f63c0110030 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.661+0000 7f63c5015640 1 -- 172.21.15.27:0/510813337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c011ae00 con 0x7f63c0110030 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.661+0000 7f63c5015640 1 -- 172.21.15.27:0/510813337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c01b9cc0 con 0x7f63c0110030 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.661+0000 7f63a7fff640 1 -- 172.21.15.27:0/510813337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b8004cc0 con 0x7f63c0110030 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.661+0000 7f63a7fff640 1 -- 172.21.15.27:0/510813337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b800f740 con 0x7f63c0110030 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.668+0000 7f63a7fff640 1 -- 172.21.15.27:0/510813337 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f63b800f8e0 con 0x7f63c0110030 2024-04-25T13:52:20.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.668+0000 7f63cbfa5740 1 Processor -- start 2024-04-25T13:52:20.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.669+0000 7f63a7fff640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63a80afd00 0x7f63a80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.669+0000 7f63a7fff640 1 -- 172.21.15.27:0/510813337 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f63b80def50 con 0x7f63c0110030 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.670+0000 7f63cbfa5740 1 -- start start 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.670+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efc4390 0x564a1f365a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.670+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1ef9b4d0 0x564a1f365fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.670+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f366c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.671+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a1f2cad80 con 0x564a1ef9b4d0 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.671+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564a1f2848b0 con 0x564a1efc4390 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.671+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564a1f2c6d40 con 0x564a1efa36b0 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.671+0000 7f63bf59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f366c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.671+0000 7f63bf59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f366c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54030/0 (socket says 172.21.15.27:54030) 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.671+0000 7f63bf59e640 1 -- 172.21.15.27:0/2137724515 learned_addr learned my addr 172.21.15.27:0/2137724515 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:20.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.671+0000 7f63be59c640 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1ef9b4d0 0x564a1f365fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63bf59e640 1 -- 172.21.15.27:0/2137724515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efc4390 msgr2=0x564a1f365a60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:52:20.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63bf59e640 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efc4390 0x564a1f365a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63bf59e640 1 -- 172.21.15.27:0/2137724515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1ef9b4d0 msgr2=0x564a1f365fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:20.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63bf59e640 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1ef9b4d0 0x564a1f365fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63bf59e640 1 -- 172.21.15.27:0/2137724515 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a1f255a40 con 0x564a1efa36b0 2024-04-25T13:52:20.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63be59c640 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1ef9b4d0 0x564a1f365fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:52:20.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63be59c640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63a80afd00 0x7f63a80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63bf59e640 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f366c70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f63b80a75d0 tx=0x7f63b8004f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7ac53930697cf35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.672+0000 7f63a57fa640 1 -- 172.21.15.27:0/2137724515 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b80c1070 con 0x564a1efa36b0 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.673+0000 7f63a57fa640 1 -- 172.21.15.27:0/2137724515 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b80a78d0 con 0x564a1efa36b0 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.673+0000 7f63a57fa640 1 -- 172.21.15.27:0/2137724515 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b801d030 con 0x564a1efa36b0 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.674+0000 7f63be59c640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63a80afd00 0x7f63a80b2210 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x564a1f365780 tx=0x7f63b40046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.674+0000 7f63cbfa5740 1 -- 172.21.15.27:0/2137724515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 msgr2=0x564a1f366c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.674+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f366c70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f63b80a75d0 tx=0x7f63b8004f50 comp rx=0 tx=0).stop 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.675+0000 7f63cbfa5740 1 -- 172.21.15.27:0/2137724515 shutdown_connections 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.675+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f366c70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.676+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1ef9b4d0 0x564a1f365fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.676+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/2137724515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efc4390 0x564a1f365a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.676+0000 7f63cbfa5740 1 -- 172.21.15.27:0/2137724515 >> 172.21.15.27:0/2137724515 conn(0x564a1efb84e0 msgr2=0x564a1f361270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:20.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.676+0000 7f63cbfa5740 1 -- 172.21.15.27:0/2137724515 shutdown_connections 2024-04-25T13:52:20.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.677+0000 7f63cbfa5740 1 -- 172.21.15.27:0/2137724515 wait complete. 2024-04-25T13:52:20.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.677+0000 7f63cbfa5740 1 -- start start 2024-04-25T13:52:20.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.678+0000 7f63cbfa5740 1 Processor -- start 2024-04-25T13:52:20.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.678+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.678+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efa36b0 0x564a1f36fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.678+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efc4390 0x564a1f376b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.679+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564a1f367780 con 0x564a1efa36b0 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.679+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564a1f251d10 con 0x564a1ef9b4d0 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.679+0000 7f63bed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.679+0000 7f63bed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:35228/0 (socket says 172.21.15.27:35228) 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.679+0000 7f63bed9d640 1 -- 172.21.15.27:0/1259963679 learned_addr learned my addr 172.21.15.27:0/1259963679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.679+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564a1f367900 con 0x564a1efc4390 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.679+0000 7f63cbfa5740 10 client.0 _ll_register_callbacks cb 0x564a1f27e210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.679+0000 7f63bf59e640 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efc4390 0x564a1f376b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63be59c640 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efa36b0 0x564a1f36fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63bed9d640 1 -- 172.21.15.27:0/1259963679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efc4390 msgr2=0x564a1f376b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63bed9d640 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efc4390 0x564a1f376b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63bed9d640 1 -- 172.21.15.27:0/1259963679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efa36b0 msgr2=0x564a1f36fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63bed9d640 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efa36b0 0x564a1f36fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63bed9d640 1 -- 172.21.15.27:0/1259963679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1f0a67d0 con 0x564a1ef9b4d0 2024-04-25T13:52:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63be59c640 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efa36b0 0x564a1f36fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63bed9d640 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36f6d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009780 tx=0x7f63b0005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=591b2aad28b280a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.680+0000 7f63a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:20.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.681+0000 7f63a2ff5640 1 -- 172.21.15.27:0/1259963679 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b0037040 con 0x564a1ef9b4d0 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.681+0000 7f63a2ff5640 1 -- 172.21.15.27:0/1259963679 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f63b0004040 con 0x564a1ef9b4d0 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.681+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63b80a78d0 con 0x564a1ef9b4d0 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.681+0000 7f63a2ff5640 1 -- 172.21.15.27:0/1259963679 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f63b00309b0 con 0x564a1ef9b4d0 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.681+0000 7f63cbfa5740 10 client.27323 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.681+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564a1efdc3f0 con 0x564a1ef9b4d0 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.682+0000 7f63a2ff5640 1 -- 172.21.15.27:0/1259963679 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b0005500 con 0x564a1ef9b4d0 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.682+0000 7f63a2ff5640 1 -- 172.21.15.27:0/1259963679 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f63b0043030 con 0x564a1ef9b4d0 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.683+0000 7f63cbfa5740 10 client.27323 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.683+0000 7f63cbfa5740 10 client.27323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.683+0000 7f63cbfa5740 20 client.27323 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.683+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a1ef82f60 0x564a1f37d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.683+0000 7f63cbfa5740 4 client.27323 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.683+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f63b80a78d0 con 0x564a1ef82f60 2024-04-25T13:52:20.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.683+0000 7f63be59c640 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a1ef82f60 0x564a1f37d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.685+0000 7f63be59c640 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a1ef82f60 0x564a1f37d080 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eff09aa17ae0d3 server_cookie=503e42408de7810d in_seq=0 out_seq=0 2024-04-25T13:52:20.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.685+0000 7f63a2ff5640 10 client.27323 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.686+0000 7f63a2ff5640 1 -- 172.21.15.27:0/1259963679 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f63b80a78d0 con 0x564a1ef82f60 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.686+0000 7f63a2ff5640 10 client.27323 handle_command_reply: tid=0 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.686+0000 7f63cbfa5740 1 client.27323 shutdown 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.686+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 msgr2=0x564a1f36f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.686+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36f6d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009780 tx=0x7f63b0005e40 comp rx=0 tx=0).stop 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 shutdown_connections 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a1ef82f60 0x564a1f37d080 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efc4390 0x564a1f376b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efa36b0 0x564a1f36fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/1259963679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36f6d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 >> 172.21.15.27:0/1259963679 conn(0x564a1efb84e0 msgr2=0x564a1f3603f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:20.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63a2ff5640 0 client.27323 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:20.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 shutdown_connections 2024-04-25T13:52:20.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 1 -- 172.21.15.27:0/1259963679 wait complete. 2024-04-25T13:52:20.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.687+0000 7f63cbfa5740 20 client.27323 trim_cache size 0 max 16384 2024-04-25T13:52:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.715+0000 7f63cbfa5740 1 Processor -- start 2024-04-25T13:52:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.715+0000 7f63cbfa5740 1 -- start start 2024-04-25T13:52:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f36df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f364780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac00c390 con 0x564a1efc4390 2024-04-25T13:52:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac00ec10 con 0x564a1ef9b4d0 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac00ca00 con 0x564a1efa36b0 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f364780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f364780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43082/0 (socket says 172.21.15.27:43082) 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 -- 172.21.15.27:0/244005210 learned_addr learned my addr 172.21.15.27:0/244005210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63be59c640 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f36df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bed9d640 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 -- 172.21.15.27:0/244005210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 msgr2=0x564a1f36df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f36df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 -- 172.21.15.27:0/244005210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 msgr2=0x564a1f36aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 -- 172.21.15.27:0/244005210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a1ef9c0f0 con 0x564a1efc4390 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.716+0000 7f63bf59e640 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f364780 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f63b80a3950 tx=0x7f63b80a7fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b9ecf854036e70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.717+0000 7f63a4ff9640 1 -- 172.21.15.27:0/244005210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b801d040 con 0x564a1efc4390 2024-04-25T13:52:20.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.717+0000 7f63a4ff9640 1 -- 172.21.15.27:0/244005210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b80b0260 con 0x564a1efc4390 2024-04-25T13:52:20.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.717+0000 7f63a4ff9640 1 -- 172.21.15.27:0/244005210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b80c1040 con 0x564a1efc4390 2024-04-25T13:52:20.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 -- 172.21.15.27:0/244005210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 msgr2=0x564a1f364780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f364780 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f63b80a3950 tx=0x7f63b80a7fd0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 -- 172.21.15.27:0/244005210 shutdown_connections 2024-04-25T13:52:20.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f364780 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1efa36b0 0x564a1f36df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/244005210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1ef9b4d0 0x564a1f36aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 -- 172.21.15.27:0/244005210 >> 172.21.15.27:0/244005210 conn(0x564a1efb84e0 msgr2=0x564a1f364070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:20.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 -- 172.21.15.27:0/244005210 shutdown_connections 2024-04-25T13:52:20.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 -- 172.21.15.27:0/244005210 wait complete. 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 -- start start 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.719+0000 7f63cbfa5740 1 Processor -- start 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1ef9b4d0 0x564a1f3741b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efa36b0 0x564a1f3746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63cbfa5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f3e3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac00d0d0 con 0x564a1efc4390 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac00e540 con 0x564a1efa36b0 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63cbfa5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac00d7a0 con 0x564a1ef9b4d0 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63cbfa5740 10 client.0 _ll_register_callbacks cb 0x7f63b00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63bed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1ef9b4d0 0x564a1f3741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63bed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1ef9b4d0 0x564a1f3741b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54056/0 (socket says 172.21.15.27:54056) 2024-04-25T13:52:20.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63bed9d640 1 -- 172.21.15.27:0/3261746007 learned_addr learned my addr 172.21.15.27:0/3261746007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63be59c640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efa36b0 0x564a1f3746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63be59c640 1 -- 172.21.15.27:0/3261746007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1ef9b4d0 msgr2=0x564a1f3741b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63be59c640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1ef9b4d0 0x564a1f3741b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63be59c640 1 -- 172.21.15.27:0/3261746007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 msgr2=0x564a1f3e3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63bf59e640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f3e3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63be59c640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f3e3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63be59c640 1 -- 172.21.15.27:0/3261746007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1f28b4b0 con 0x564a1efa36b0 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.720+0000 7f63bed9d640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1ef9b4d0 0x564a1f3741b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63bf59e640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f3e3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63be59c640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efa36b0 0x564a1f3746f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x564a1f37d7f0 tx=0x7f63b4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec542deb63e8736 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:20.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63a2ff5640 1 -- 172.21.15.27:0/3261746007 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b4016070 con 0x564a1efa36b0 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63cbfa5740 1 -- 172.21.15.27:0/3261746007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63b80b0260 con 0x564a1efa36b0 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63a2ff5640 1 -- 172.21.15.27:0/3261746007 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f63b4005020 con 0x564a1efa36b0 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63a2ff5640 1 -- 172.21.15.27:0/3261746007 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f63b4005220 con 0x564a1efa36b0 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63cbfa5740 10 client.27329 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63cbfa5740 1 -- 172.21.15.27:0/3261746007 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564a1f3e4980 con 0x564a1efa36b0 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.721+0000 7f63a2ff5640 1 -- 172.21.15.27:0/3261746007 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b80b0260 con 0x564a1efa36b0 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63a2ff5640 1 -- 172.21.15.27:0/3261746007 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f63b4025ce0 con 0x564a1efa36b0 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63cbfa5740 10 client.27329 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63cbfa5740 10 client.27329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63cbfa5740 20 client.27329 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a1ef583a0 0x564a1f3e7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63cbfa5740 4 client.27329 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:52:20.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63cbfa5740 1 -- 172.21.15.27:0/3261746007 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564a1f2d8990 con 0x564a1ef583a0 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63bed9d640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a1ef583a0 0x564a1f3e7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63a2ff5640 10 client.27329 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.722+0000 7f63bed9d640 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a1ef583a0 0x564a1f3e7320 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aa878f76a9b01a2 server_cookie=8475ed54228ae9b2 in_seq=0 out_seq=0 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63a2ff5640 1 -- 172.21.15.27:0/3261746007 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564a1f2d8990 con 0x564a1ef583a0 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63a2ff5640 10 client.27329 handle_command_reply: tid=0 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 client.27329 shutdown 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 -- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efa36b0 msgr2=0x564a1f3746f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efa36b0 0x564a1f3746f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x564a1f37d7f0 tx=0x7f63b4009260 comp rx=0 tx=0).stop 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 -- 172.21.15.27:0/3261746007 shutdown_connections 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564a1ef583a0 0x564a1f3e7320 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564a1efc4390 0x564a1f3e3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564a1efa36b0 0x564a1f3746f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63a2ff5640 0 client.27329 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 --2- 172.21.15.27:0/3261746007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564a1ef9b4d0 0x564a1f3741b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 -- 172.21.15.27:0/3261746007 >> 172.21.15.27:0/3261746007 conn(0x564a1efb84e0 msgr2=0x564a1f364aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:20.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 -- 172.21.15.27:0/3261746007 shutdown_connections 2024-04-25T13:52:20.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 1 -- 172.21.15.27:0/3261746007 wait complete. 2024-04-25T13:52:20.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.723+0000 7f63cbfa5740 20 client.27329 trim_cache size 0 max 16384 2024-04-25T13:52:20.730 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:52:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:52:20.730 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:52:20.730 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:52:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 -- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63a80afd00 msgr2=0x7f63a80b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63a80afd00 0x7f63a80b2210 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x564a1f365780 tx=0x7f63b40046c0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 -- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c0110030 msgr2=0x7f63c011a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c0110030 0x7f63c011a5d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f63b800cb00 tx=0x7f63b800a250 comp rx=0 tx=0).stop 2024-04-25T13:52:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 -- 172.21.15.27:0/510813337 shutdown_connections 2024-04-25T13:52:20.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63a80afd00 0x7f63a80b2210 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63c0110030 0x7f63c011a5d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f63c010b310 0x7f63c011a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 --2- 172.21.15.27:0/510813337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63c010a420 0x7f63c011b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.724+0000 7f63c5015640 1 -- 172.21.15.27:0/510813337 >> 172.21.15.27:0/510813337 conn(0x7f63c00704e0 msgr2=0x7f63c010e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.725+0000 7f63c5015640 1 -- 172.21.15.27:0/510813337 shutdown_connections 2024-04-25T13:52:20.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.725+0000 7f63c5015640 1 -- 172.21.15.27:0/510813337 wait complete. 2024-04-25T13:52:20.737 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:52:20.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:52:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 -- 172.21.15.27:0/4115000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00410a790 msgr2=0x7fd00410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 --2- 172.21.15.27:0/4115000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00410a790 0x7fd00410b320 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009900 tx=0x7fcff802a410 comp rx=0 tx=0).stop 2024-04-25T13:52:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 -- 172.21.15.27:0/4115000186 shutdown_connections 2024-04-25T13:52:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 --2- 172.21.15.27:0/4115000186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd00410c070 0x7fd00410c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 --2- 172.21.15.27:0/4115000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00410a790 0x7fd00410b320 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 --2- 172.21.15.27:0/4115000186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd00410a2f0 0x7fd00410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 -- 172.21.15.27:0/4115000186 >> 172.21.15.27:0/4115000186 conn(0x7fd004070920 msgr2=0x7fd004072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 -- 172.21.15.27:0/4115000186 shutdown_connections 2024-04-25T13:52:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.889+0000 7fd00af40640 1 -- 172.21.15.27:0/4115000186 wait complete. 2024-04-25T13:52:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.890+0000 7fd00af40640 1 Processor -- start 2024-04-25T13:52:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.890+0000 7fd00af40640 1 -- start start 2024-04-25T13:52:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.890+0000 7fd00af40640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd00410a2f0 0x7fd0041a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.890+0000 7fd00af40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00410c070 0x7fd0041a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd00af40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0041a9d00 0x7fd0041ac120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd00af40640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd00411a020 con 0x7fd00410c070 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd00af40640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd004119ea0 con 0x7fd00410a2f0 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd00af40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd00411a1a0 con 0x7fd0041a9d00 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd008cdc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd00410a2f0 0x7fd0041a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd003fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00410c070 0x7fd0041a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd008cdc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd00410a2f0 0x7fd0041a4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:35264/0 (socket says 172.21.15.27:35264) 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd008cdc640 1 -- 172.21.15.27:0/2808667648 learned_addr learned my addr 172.21.15.27:0/2808667648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd0094dd640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0041a9d00 0x7fd0041ac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd008cdc640 1 -- 172.21.15.27:0/2808667648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0041a9d00 msgr2=0x7fd0041ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd008cdc640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0041a9d00 0x7fd0041ac120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd008cdc640 1 -- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00410c070 msgr2=0x7fd0041a4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd008cdc640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00410c070 0x7fd0041a4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd008cdc640 1 -- 172.21.15.27:0/2808667648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0041ac790 con 0x7fd00410a2f0 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.891+0000 7fd0094dd640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0041a9d00 0x7fd0041ac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.892+0000 7fd008cdc640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd00410a2f0 0x7fd0041a4840 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9f0 tx=0x7fcff400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.892+0000 7fd001ffb640 1 -- 172.21.15.27:0/2808667648 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff400ccc0 con 0x7fd00410a2f0 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.892+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0041aca80 con 0x7fd00410a2f0 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.892+0000 7fd001ffb640 1 -- 172.21.15.27:0/2808667648 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcff40108e0 con 0x7fd00410a2f0 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.892+0000 7fd001ffb640 1 -- 172.21.15.27:0/2808667648 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4018860 con 0x7fd00410a2f0 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.892+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0041ad020 con 0x7fd00410a2f0 2024-04-25T13:52:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.893+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd004116290 con 0x7fd00410a2f0 2024-04-25T13:52:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.895+0000 7fd001ffb640 1 -- 172.21.15.27:0/2808667648 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcff4010430 con 0x7fd00410a2f0 2024-04-25T13:52:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.895+0000 7fd001ffb640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfec0afaf0 0x7fcfec0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.896+0000 7fd001ffb640 1 -- 172.21.15.27:0/2808667648 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fcff40df420 con 0x7fd00410a2f0 2024-04-25T13:52:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.896+0000 7fd003fff640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfec0afaf0 0x7fcfec0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:20.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.898+0000 7fd003fff640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfec0afaf0 0x7fcfec0b2000 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd0041a58e0 tx=0x7fcff8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:20.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:20.899+0000 7fd001ffb640 1 -- 172.21.15.27:0/2808667648 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcff40a7fd0 con 0x7fd00410a2f0 2024-04-25T13:52:21.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.231+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd00402db50 con 0x7fd00410a2f0 2024-04-25T13:52:21.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.232+0000 7fd001ffb640 1 -- 172.21.15.27:0/2808667648 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fcff40a78d0 con 0x7fd00410a2f0 2024-04-25T13:52:21.231 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:52:21.232 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:52:21.232 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:52:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfec0afaf0 msgr2=0x7fcfec0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfec0afaf0 0x7fcfec0b2000 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd0041a58e0 tx=0x7fcff8031400 comp rx=0 tx=0).stop 2024-04-25T13:52:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd00410a2f0 msgr2=0x7fd0041a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd00410a2f0 0x7fd0041a4840 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9f0 tx=0x7fcff400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 shutdown_connections 2024-04-25T13:52:21.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcfec0afaf0 0x7fcfec0b2000 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0041a9d00 0x7fd0041ac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00410c070 0x7fd0041a4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 --2- 172.21.15.27:0/2808667648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd00410a2f0 0x7fd0041a4840 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 >> 172.21.15.27:0/2808667648 conn(0x7fd004070920 msgr2=0x7fd0040729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 shutdown_connections 2024-04-25T13:52:21.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.234+0000 7fd00af40640 1 -- 172.21.15.27:0/2808667648 wait complete. 2024-04-25T13:52:21.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:52:21.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.419+0000 7f539cfce640 1 -- 172.21.15.27:0/3518234753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53981045e0 msgr2=0x7f5398106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.419+0000 7f539cfce640 1 --2- 172.21.15.27:0/3518234753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53981045e0 0x7f5398106a00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f538c009900 tx=0x7f538c02a410 comp rx=0 tx=0).stop 2024-04-25T13:52:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.420+0000 7f539cfce640 1 -- 172.21.15.27:0/3518234753 shutdown_connections 2024-04-25T13:52:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.420+0000 7f539cfce640 1 --2- 172.21.15.27:0/3518234753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53981098a0 0x7f539810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.420+0000 7f539cfce640 1 --2- 172.21.15.27:0/3518234753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5398106f40 0x7f5398109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.420+0000 7f539cfce640 1 --2- 172.21.15.27:0/3518234753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53981045e0 0x7f5398106a00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.420+0000 7f539cfce640 1 -- 172.21.15.27:0/3518234753 >> 172.21.15.27:0/3518234753 conn(0x7f53980fdbc0 msgr2=0x7f5398100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.420+0000 7f539cfce640 1 -- 172.21.15.27:0/3518234753 shutdown_connections 2024-04-25T13:52:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.420+0000 7f539cfce640 1 -- 172.21.15.27:0/3518234753 wait complete. 2024-04-25T13:52:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.421+0000 7f539cfce640 1 Processor -- start 2024-04-25T13:52:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.421+0000 7f539cfce640 1 -- start start 2024-04-25T13:52:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.421+0000 7f539cfce640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53981045e0 0x7f5398103a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.421+0000 7f539cfce640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5398106f40 0x7f5398102130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.421+0000 7f539cfce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53981098a0 0x7f5398102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.421+0000 7f539cfce640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f539810f5f0 con 0x7f5398106f40 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.421+0000 7f539cfce640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f539810f470 con 0x7f53981098a0 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.421+0000 7f539cfce640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f539810f770 con 0x7f53981045e0 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5395d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5398106f40 0x7f5398102130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5396d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53981098a0 0x7f5398102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5395d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5398106f40 0x7f5398102130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43142/0 (socket says 172.21.15.27:43142) 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5395d9b640 1 -- 172.21.15.27:0/4148480943 learned_addr learned my addr 172.21.15.27:0/4148480943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f539659c640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53981045e0 0x7f5398103a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5396d9d640 1 -- 172.21.15.27:0/4148480943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53981045e0 msgr2=0x7f5398103a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5396d9d640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53981045e0 0x7f5398103a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5396d9d640 1 -- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5398106f40 msgr2=0x7f5398102130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5396d9d640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5398106f40 0x7f5398102130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5396d9d640 1 -- 172.21.15.27:0/4148480943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5398102bb0 con 0x7f53981098a0 2024-04-25T13:52:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.422+0000 7f5396d9d640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53981098a0 0x7f5398102670 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5390005fd0 tx=0x7f539000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.423+0000 7f53877fe640 1 -- 172.21.15.27:0/4148480943 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539000cdf0 con 0x7f53981098a0 2024-04-25T13:52:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.423+0000 7f53877fe640 1 -- 172.21.15.27:0/4148480943 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5390005020 con 0x7f53981098a0 2024-04-25T13:52:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.423+0000 7f539cfce640 1 -- 172.21.15.27:0/4148480943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5398102ea0 con 0x7f53981098a0 2024-04-25T13:52:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.423+0000 7f53877fe640 1 -- 172.21.15.27:0/4148480943 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390010690 con 0x7f53981098a0 2024-04-25T13:52:21.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.423+0000 7f539cfce640 1 -- 172.21.15.27:0/4148480943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53981a8c80 con 0x7f53981098a0 2024-04-25T13:52:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.425+0000 7f53a3e7d740 1 Processor -- start 2024-04-25T13:52:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.425+0000 7f53877fe640 1 -- 172.21.15.27:0/4148480943 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5390010830 con 0x7f53981098a0 2024-04-25T13:52:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f53a3e7d740 1 -- start start 2024-04-25T13:52:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd401810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd401d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 0x560bfd402a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560bfd365d80 con 0x560bfd0364d0 2024-04-25T13:52:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560bfd31fa20 con 0x560bfd03e6b0 2024-04-25T13:52:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x560bfd361d40 con 0x560bfd05f390 2024-04-25T13:52:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f5395d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd401810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f5395d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd401810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54082/0 (socket says 172.21.15.27:54082) 2024-04-25T13:52:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f5395d9b640 1 -- 172.21.15.27:0/1575821464 learned_addr learned my addr 172.21.15.27:0/1575821464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f5395d9b640 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 0x560bfd402a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.426+0000 7f539659c640 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd401d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5395d9b640 1 -- 172.21.15.27:0/1575821464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 msgr2=0x560bfd402a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5395d9b640 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 0x560bfd402a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5395d9b640 1 -- 172.21.15.27:0/1575821464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 msgr2=0x560bfd401d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5395d9b640 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd401d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5395d9b640 1 -- 172.21.15.27:0/1575821464 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bfd32ac30 con 0x560bfd05f390 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f53877fe640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53800afb70 0x7f53800b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5395d9b640 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd401810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5388013910 tx=0x7f53880098d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2352b83e0a31a374 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f539659c640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53800afb70 0x7f53800b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5384ff9640 1 -- 172.21.15.27:0/1575821464 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53880117e0 con 0x560bfd05f390 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f53877fe640 1 -- 172.21.15.27:0/4148480943 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f53900df2c0 con 0x7f53981098a0 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5384ff9640 1 -- 172.21.15.27:0/1575821464 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5388011e30 con 0x560bfd05f390 2024-04-25T13:52:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.427+0000 7f5384ff9640 1 -- 172.21.15.27:0/1575821464 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538801fd80 con 0x560bfd05f390 2024-04-25T13:52:21.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 -- 172.21.15.27:0/1575821464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 msgr2=0x560bfd401810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f539659c640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53800afb70 0x7f53800b2080 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f538c009230 tx=0x7f538c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd401810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5388013910 tx=0x7f53880098d0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 -- 172.21.15.27:0/1575821464 shutdown_connections 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 0x560bfd402a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd401d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/1575821464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd401810 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 -- 172.21.15.27:0/1575821464 >> 172.21.15.27:0/1575821464 conn(0x560bfd0534e0 msgr2=0x560bfd3fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 -- 172.21.15.27:0/1575821464 shutdown_connections 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.428+0000 7f53a3e7d740 1 -- 172.21.15.27:0/1575821464 wait complete. 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 1 -- start start 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 1 Processor -- start 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd0364d0 0x560bfd40b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd05f390 0x560bfd412b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560bfd403530 con 0x560bfd05f390 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560bfd2ecdc0 con 0x560bfd0364d0 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x560bfd4036b0 con 0x560bfd03e6b0 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f53a3e7d740 10 client.0 _ll_register_callbacks cb 0x560bfd308600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f539659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd0364d0 0x560bfd40b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f5396d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd05f390 0x560bfd412b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f539659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd0364d0 0x560bfd40b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:35292/0 (socket says 172.21.15.27:35292) 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f539659c640 1 -- 172.21.15.27:0/3160055981 learned_addr learned my addr 172.21.15.27:0/3160055981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.429+0000 7f5395d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f539659c640 1 -- 172.21.15.27:0/3160055981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 msgr2=0x560bfd40bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f539659c640 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f539659c640 1 -- 172.21.15.27:0/3160055981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd05f390 msgr2=0x560bfd412b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f539659c640 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd05f390 0x560bfd412b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f539659c640 1 -- 172.21.15.27:0/3160055981 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bfd0773f0 con 0x560bfd0364d0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f5396d9d640 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd05f390 0x560bfd412b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f539659c640 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd0364d0 0x560bfd40b690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f538c005f70 tx=0x7f538c02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d93aef61738b85ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f537dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f537dffb640 1 -- 172.21.15.27:0/3160055981 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c035040 con 0x560bfd0364d0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f537dffb640 1 -- 172.21.15.27:0/3160055981 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f538c004240 con 0x560bfd0364d0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.430+0000 7f53a3e7d740 1 -- 172.21.15.27:0/3160055981 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5388011e30 con 0x560bfd0364d0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.431+0000 7f537dffb640 1 -- 172.21.15.27:0/3160055981 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f538c004440 con 0x560bfd0364d0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.431+0000 7f53a3e7d740 10 client.27347 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.431+0000 7f53a3e7d740 1 -- 172.21.15.27:0/3160055981 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560bfd1417d0 con 0x560bfd0364d0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.432+0000 7f537dffb640 1 -- 172.21.15.27:0/3160055981 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5388011e30 con 0x560bfd0364d0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.432+0000 7f537dffb640 1 -- 172.21.15.27:0/3160055981 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f538c044070 con 0x560bfd0364d0 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.432+0000 7f53a3e7d740 10 client.27347 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.432+0000 7f53a3e7d740 10 client.27347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.432+0000 7f53a3e7d740 20 client.27347 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.432+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560bfd01df60 0x560bfd419420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.432+0000 7f53a3e7d740 4 client.27347 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.432+0000 7f53a3e7d740 1 -- 172.21.15.27:0/3160055981 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bfd373990 con 0x560bfd01df60 2024-04-25T13:52:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.433+0000 7f5395d9b640 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560bfd01df60 0x560bfd419420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.433+0000 7f5395d9b640 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560bfd01df60 0x560bfd419420 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e2ce0b8cffba8a server_cookie=a10ffea13deeaff0 in_seq=0 out_seq=0 2024-04-25T13:52:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.433+0000 7f537dffb640 10 client.27347 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f537dffb640 1 -- 172.21.15.27:0/3160055981 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560bfd373990 con 0x560bfd01df60 2024-04-25T13:52:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f537dffb640 10 client.27347 handle_command_reply: tid=0 2024-04-25T13:52:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 client.27347 shutdown 2024-04-25T13:52:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 -- 172.21.15.27:0/3160055981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd0364d0 msgr2=0x560bfd40b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd0364d0 0x560bfd40b690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f538c005f70 tx=0x7f538c02a410 comp rx=0 tx=0).stop 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 -- 172.21.15.27:0/3160055981 shutdown_connections 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560bfd01df60 0x560bfd419420 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd05f390 0x560bfd412b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f537dffb640 0 client.27347 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/3160055981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd0364d0 0x560bfd40b690 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 -- 172.21.15.27:0/3160055981 >> 172.21.15.27:0/3160055981 conn(0x560bfd0534e0 msgr2=0x560bfd3fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 -- 172.21.15.27:0/3160055981 shutdown_connections 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 1 -- 172.21.15.27:0/3160055981 wait complete. 2024-04-25T13:52:21.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.434+0000 7f53a3e7d740 20 client.27347 trim_cache size 0 max 16384 2024-04-25T13:52:21.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.453+0000 7f53a3e7d740 1 Processor -- start 2024-04-25T13:52:21.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.456+0000 7f53a3e7d740 1 -- start start 2024-04-25T13:52:21.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.456+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd406a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.456+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 0x560bfd3fd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.456+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd412ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.456+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53981af6f0 con 0x560bfd0364d0 2024-04-25T13:52:21.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.456+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53981b1fb0 con 0x560bfd03e6b0 2024-04-25T13:52:21.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.456+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f53981afda0 con 0x560bfd05f390 2024-04-25T13:52:21.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.458+0000 7f5396d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd412ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.458+0000 7f5396d9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd412ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54094/0 (socket says 172.21.15.27:54094) 2024-04-25T13:52:21.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.458+0000 7f5396d9d640 1 -- 172.21.15.27:0/786383292 learned_addr learned my addr 172.21.15.27:0/786383292 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:21.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.459+0000 7f5395d9b640 1 --2- 172.21.15.27:0/786383292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 0x560bfd3fd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.465+0000 7f5396d9d640 1 -- 172.21.15.27:0/786383292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 msgr2=0x560bfd3fd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.465+0000 7f5396d9d640 1 --2- 172.21.15.27:0/786383292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 0x560bfd3fd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.465+0000 7f5396d9d640 1 -- 172.21.15.27:0/786383292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 msgr2=0x560bfd406a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.465+0000 7f5396d9d640 1 --2- 172.21.15.27:0/786383292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd406a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.465+0000 7f5396d9d640 1 -- 172.21.15.27:0/786383292 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bfd4097b0 con 0x560bfd05f390 2024-04-25T13:52:21.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.470+0000 7f5396d9d640 1 --2- 172.21.15.27:0/786383292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd412ac0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f53900a3d10 tx=0x7f53900dcc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8f0ddbaf71a3812 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:21.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.472+0000 7f537ffff640 1 -- 172.21.15.27:0/786383292 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53900b05d0 con 0x560bfd05f390 2024-04-25T13:52:21.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.472+0000 7f537ffff640 1 -- 172.21.15.27:0/786383292 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53900dcea0 con 0x560bfd05f390 2024-04-25T13:52:21.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.473+0000 7f53a3e7d740 1 -- 172.21.15.27:0/786383292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 msgr2=0x560bfd412ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.473+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/786383292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd412ac0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f53900a3d10 tx=0x7f53900dcc60 comp rx=0 tx=0).stop 2024-04-25T13:52:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.483+0000 7f53a3e7d740 1 -- 172.21.15.27:0/786383292 shutdown_connections 2024-04-25T13:52:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.483+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/786383292 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd05f390 0x560bfd412ac0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.483+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/786383292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd03e6b0 0x560bfd3fd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.483+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/786383292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd406a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.483+0000 7f53a3e7d740 1 -- 172.21.15.27:0/786383292 >> 172.21.15.27:0/786383292 conn(0x560bfd0534e0 msgr2=0x560bfd3fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:21.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.490+0000 7f53a3e7d740 1 -- 172.21.15.27:0/786383292 shutdown_connections 2024-04-25T13:52:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.494+0000 7f53a3e7d740 1 -- 172.21.15.27:0/786383292 wait complete. 2024-04-25T13:52:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 1 -- start start 2024-04-25T13:52:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 1 Processor -- start 2024-04-25T13:52:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd40f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd05f390 0x560bfd490790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53981b0470 con 0x560bfd0364d0 2024-04-25T13:52:21.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53981b18e0 con 0x560bfd05f390 2024-04-25T13:52:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f53981b0b40 con 0x560bfd03e6b0 2024-04-25T13:52:21.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.495+0000 7f53a3e7d740 10 client.0 _ll_register_callbacks cb 0x7f538c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5396d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd05f390 0x560bfd490790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5395d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5395d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54098/0 (socket says 172.21.15.27:54098) 2024-04-25T13:52:21.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5395d9b640 1 -- 172.21.15.27:0/2023840746 learned_addr learned my addr 172.21.15.27:0/2023840746 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:21.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5396d9d640 1 -- 172.21.15.27:0/2023840746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 msgr2=0x560bfd40fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:21.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5396d9d640 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5396d9d640 1 -- 172.21.15.27:0/2023840746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 msgr2=0x560bfd40f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:21.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5396d9d640 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd40f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.498+0000 7f5396d9d640 1 -- 172.21.15.27:0/2023840746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bfd412da0 con 0x560bfd05f390 2024-04-25T13:52:21.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.501+0000 7f5396d9d640 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd05f390 0x560bfd490790 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f53900b0540 tx=0x7f539000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26cdc959058569d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:21.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.501+0000 7f537dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:21.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.505+0000 7f537dffb640 1 -- 172.21.15.27:0/2023840746 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53900b05d0 con 0x560bfd05f390 2024-04-25T13:52:21.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.505+0000 7f53a3e7d740 1 -- 172.21.15.27:0/2023840746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53900dcea0 con 0x560bfd05f390 2024-04-25T13:52:21.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.505+0000 7f537dffb640 1 -- 172.21.15.27:0/2023840746 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f53900a69d0 con 0x560bfd05f390 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.509+0000 7f537dffb640 1 -- 172.21.15.27:0/2023840746 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f53900a6c10 con 0x560bfd05f390 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.511+0000 7f53a3e7d740 10 client.27353 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.511+0000 7f53a3e7d740 1 -- 172.21.15.27:0/2023840746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560bfd491cc0 con 0x560bfd05f390 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.511+0000 7f537dffb640 1 -- 172.21.15.27:0/2023840746 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53900dcea0 con 0x560bfd05f390 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.511+0000 7f537dffb640 1 -- 172.21.15.27:0/2023840746 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f53900a8c20 con 0x560bfd05f390 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.512+0000 7f53a3e7d740 10 client.27353 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.512+0000 7f53a3e7d740 10 client.27353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.512+0000 7f53a3e7d740 20 client.27353 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:21.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.512+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560bfcff33a0 0x560bfd4947d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:21.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.512+0000 7f53a3e7d740 4 client.27353 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:52:21.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.512+0000 7f53a3e7d740 1 -- 172.21.15.27:0/2023840746 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560bfd36d800 con 0x560bfcff33a0 2024-04-25T13:52:21.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.514+0000 7f539659c640 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560bfcff33a0 0x560bfd4947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:21.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.524+0000 7f539659c640 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560bfcff33a0 0x560bfd4947d0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=316a06b98a17941b server_cookie=274bd33528ffd7ed in_seq=0 out_seq=0 2024-04-25T13:52:21.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.525+0000 7f537dffb640 10 client.27353 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:21.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.528+0000 7f537dffb640 1 -- 172.21.15.27:0/2023840746 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560bfd36d800 con 0x560bfcff33a0 2024-04-25T13:52:21.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.528+0000 7f537dffb640 10 client.27353 handle_command_reply: tid=0 2024-04-25T13:52:21.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.528+0000 7f53a3e7d740 1 client.27353 shutdown 2024-04-25T13:52:21.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.532+0000 7f53a3e7d740 1 -- 172.21.15.27:0/2023840746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd05f390 msgr2=0x560bfd490790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.533+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd05f390 0x560bfd490790 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f53900b0540 tx=0x7f539000cf90 comp rx=0 tx=0).stop 2024-04-25T13:52:21.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.533+0000 7f53a3e7d740 1 -- 172.21.15.27:0/2023840746 shutdown_connections 2024-04-25T13:52:21.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.533+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560bfcff33a0 0x560bfd4947d0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.533+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560bfd05f390 0x560bfd490790 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.533+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560bfd03e6b0 0x560bfd40fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.533+0000 7f53a3e7d740 1 --2- 172.21.15.27:0/2023840746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560bfd0364d0 0x560bfd40f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.533+0000 7f53a3e7d740 1 -- 172.21.15.27:0/2023840746 >> 172.21.15.27:0/2023840746 conn(0x560bfd0534e0 msgr2=0x560bfd40b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:21.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.533+0000 7f537dffb640 0 client.27353 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:21.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.538+0000 7f53a3e7d740 1 -- 172.21.15.27:0/2023840746 shutdown_connections 2024-04-25T13:52:21.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.541+0000 7f53a3e7d740 1 -- 172.21.15.27:0/2023840746 wait complete. 2024-04-25T13:52:21.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.542+0000 7f53a3e7d740 20 client.27353 trim_cache size 0 max 16384 2024-04-25T13:52:21.548 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:52:21.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.549+0000 7f539cfce640 1 -- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53800afb70 msgr2=0x7f53800b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.550+0000 7f539cfce640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53800afb70 0x7f53800b2080 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f538c009230 tx=0x7f538c031400 comp rx=0 tx=0).stop 2024-04-25T13:52:21.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.550+0000 7f539cfce640 1 -- 172.21.15.27:0/4148480943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53981098a0 msgr2=0x7f5398102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:21.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.550+0000 7f539cfce640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53981098a0 0x7f5398102670 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5390005fd0 tx=0x7f539000ef10 comp rx=0 tx=0).stop 2024-04-25T13:52:21.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.550+0000 7f539cfce640 1 -- 172.21.15.27:0/4148480943 shutdown_connections 2024-04-25T13:52:21.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.550+0000 7f539cfce640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f53800afb70 0x7f53800b2080 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.550+0000 7f539cfce640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53981098a0 0x7f5398102670 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.551+0000 7f539cfce640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5398106f40 0x7f5398102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.551+0000 7f539cfce640 1 --2- 172.21.15.27:0/4148480943 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53981045e0 0x7f5398103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:21.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.551+0000 7f539cfce640 1 -- 172.21.15.27:0/4148480943 >> 172.21.15.27:0/4148480943 conn(0x7f53980fdbc0 msgr2=0x7f5398105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:21.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.553+0000 7f539cfce640 1 -- 172.21.15.27:0/4148480943 shutdown_connections 2024-04-25T13:52:21.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:21.553+0000 7f539cfce640 1 -- 172.21.15.27:0/4148480943 wait complete. 2024-04-25T13:52:22.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:22 smithi122 ceph-mon[32843]: pgmap v693: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 621 op/s 2024-04-25T13:52:22.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:22 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2808667648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:52:22.573 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:52:22.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:52:22.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:52:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:22 smithi045 ceph-mon[32731]: pgmap v693: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 621 op/s 2024-04-25T13:52:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:22 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2808667648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:52:22.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:22 smithi027 ceph-mon[29140]: pgmap v693: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 621 op/s 2024-04-25T13:52:22.615 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:22 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2808667648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:52:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.733+0000 7fe62c984640 1 -- 172.21.15.27:0/1172182941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 msgr2=0x7fe62810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.733+0000 7fe62c984640 1 --2- 172.21.15.27:0/1172182941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 0x7fe62810ad40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe620009ca0 tx=0x7fe62002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.733+0000 7fe62c984640 1 -- 172.21.15.27:0/1172182941 shutdown_connections 2024-04-25T13:52:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.733+0000 7fe62c984640 1 --2- 172.21.15.27:0/1172182941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 0x7fe62810ad40 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.733+0000 7fe62c984640 1 --2- 172.21.15.27:0/1172182941 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe628103c20 0x7fe628104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.733+0000 7fe62c984640 1 --2- 172.21.15.27:0/1172182941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6281029e0 0x7fe6281036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.733+0000 7fe62c984640 1 -- 172.21.15.27:0/1172182941 >> 172.21.15.27:0/1172182941 conn(0x7fe6280fdbd0 msgr2=0x7fe628100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.733+0000 7fe62c984640 1 -- 172.21.15.27:0/1172182941 shutdown_connections 2024-04-25T13:52:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.734+0000 7fe62c984640 1 -- 172.21.15.27:0/1172182941 wait complete. 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.734+0000 7fe62c984640 1 Processor -- start 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.734+0000 7fe62c984640 1 -- start start 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62c984640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6281029e0 0x7fe6280747a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62c984640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe628103c20 0x7fe6281a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62c984640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 0x7fe6281a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62c984640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe62806ce20 con 0x7fe628108920 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62c984640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe62806cca0 con 0x7fe628103c20 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62c984640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe62806cfa0 con 0x7fe6281029e0 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe626d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 0x7fe6281a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6281029e0 0x7fe6280747a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe626d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 0x7fe6281a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43206/0 (socket says 172.21.15.27:43206) 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe626d9d640 1 -- 172.21.15.27:0/1109367223 learned_addr learned my addr 172.21.15.27:0/1109367223 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe625d9b640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe628103c20 0x7fe6281a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62659c640 1 -- 172.21.15.27:0/1109367223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe628103c20 msgr2=0x7fe6281a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62659c640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe628103c20 0x7fe6281a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62659c640 1 -- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 msgr2=0x7fe6281a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62659c640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 0x7fe6281a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62659c640 1 -- 172.21.15.27:0/1109367223 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6281aa7c0 con 0x7fe6281029e0 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.735+0000 7fe62659c640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6281029e0 0x7fe6280747a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe6281022b0 tx=0x7fe61800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.736+0000 7fe6177fe640 1 -- 172.21.15.27:0/1109367223 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61800cc40 con 0x7fe6281029e0 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.736+0000 7fe6177fe640 1 -- 172.21.15.27:0/1109367223 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe61800cde0 con 0x7fe6281029e0 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.736+0000 7fe6177fe640 1 -- 172.21.15.27:0/1109367223 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe618010430 con 0x7fe6281029e0 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.736+0000 7fe62c984640 1 -- 172.21.15.27:0/1109367223 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6281aaab0 con 0x7fe6281029e0 2024-04-25T13:52:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.736+0000 7fe62c984640 1 -- 172.21.15.27:0/1109367223 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6281ab050 con 0x7fe6281029e0 2024-04-25T13:52:22.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.738+0000 7fe633835740 1 Processor -- start 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.738+0000 7fe633835740 1 -- start start 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe6177fe640 1 -- 172.21.15.27:0/1109367223 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe6180105d0 con 0x7fe6281029e0 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe633835740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1aa74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe633835740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1aa7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe633835740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16e5730 0x55f5a1aae920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe633835740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f5a1a0cb00 con 0x55f5a16dd550 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe633835740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f5a19a8920 con 0x55f5a1706410 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe633835740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f5a1a08ac0 con 0x55f5a16e5730 2024-04-25T13:52:22.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe626d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1aa74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe626d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1aa74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:35332/0 (socket says 172.21.15.27:35332) 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe626d9d640 1 -- 172.21.15.27:0/1527651445 learned_addr learned my addr 172.21.15.27:0/1527651445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe626d9d640 1 -- 172.21.15.27:0/1527651445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16e5730 msgr2=0x55f5a1aae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe626d9d640 1 --2- 172.21.15.27:0/1527651445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16e5730 0x55f5a1aae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.739+0000 7fe626d9d640 1 -- 172.21.15.27:0/1527651445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 msgr2=0x55f5a1aa7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.740+0000 7fe6177fe640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe6100afb70 0x7fe6100b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.741+0000 7fe6177fe640 1 -- 172.21.15.27:0/1109367223 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe6180def60 con 0x7fe6281029e0 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.741+0000 7fe625d9b640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe6100afb70 0x7fe6100b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.741+0000 7fe62659c640 1 --2- 172.21.15.27:0/1527651445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1aa7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.741+0000 7fe626d9d640 1 --2- 172.21.15.27:0/1527651445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1aa7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.741+0000 7fe626d9d640 1 -- 172.21.15.27:0/1527651445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5a17e67e0 con 0x55f5a1706410 2024-04-25T13:52:22.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.742+0000 7fe626d9d640 1 --2- 172.21.15.27:0/1527651445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1aa74a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe620005bd0 tx=0x7fe620005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c67c0912d8931e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:22.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.742+0000 7fe625d9b640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe6100afb70 0x7fe6100b2080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe628074140 tx=0x7fe61c009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:22.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.742+0000 7fe614ff9640 1 -- 172.21.15.27:0/1527651445 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6200307c0 con 0x55f5a1706410 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.742+0000 7fe614ff9640 1 -- 172.21.15.27:0/1527651445 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe620036030 con 0x55f5a1706410 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.742+0000 7fe614ff9640 1 -- 172.21.15.27:0/1527651445 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620030ce0 con 0x55f5a1706410 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 -- 172.21.15.27:0/1527651445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 msgr2=0x55f5a1aa74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 --2- 172.21.15.27:0/1527651445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1aa74a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe620005bd0 tx=0x7fe620005c00 comp rx=0 tx=0).stop 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 -- 172.21.15.27:0/1527651445 shutdown_connections 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 --2- 172.21.15.27:0/1527651445 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16e5730 0x55f5a1aae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 --2- 172.21.15.27:0/1527651445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1aa7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 --2- 172.21.15.27:0/1527651445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1aa74a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 -- 172.21.15.27:0/1527651445 >> 172.21.15.27:0/1527651445 conn(0x55f5a16fa560 msgr2=0x55f5a1aa2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 -- 172.21.15.27:0/1527651445 shutdown_connections 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.743+0000 7fe633835740 1 -- 172.21.15.27:0/1527651445 wait complete. 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 1 -- start start 2024-04-25T13:52:22.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 1 Processor -- start 2024-04-25T13:52:22.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1ab99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 0x55f5a1ab9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f5a1aaf190 con 0x55f5a16dd550 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f5a19a65d0 con 0x55f5a16e5730 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f5a1aaf310 con 0x55f5a1706410 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe633835740 10 client.0 _ll_register_callbacks cb 0x55f5a1994080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe62659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1ab99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe62659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1ab99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43214/0 (socket says 172.21.15.27:43214) 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe62659c640 1 -- 172.21.15.27:0/1695916026 learned_addr learned my addr 172.21.15.27:0/1695916026 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe625d9b640 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 0x55f5a1ab9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.744+0000 7fe62659c640 1 -- 172.21.15.27:0/1695916026 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a1706410 msgr2=0x55f5a1ac0e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe62659c640 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe62659c640 1 -- 172.21.15.27:0/1695916026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 msgr2=0x55f5a1ab9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe62659c640 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 0x55f5a1ab9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe62659c640 1 -- 172.21.15.27:0/1695916026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5a1846f70 con 0x55f5a16dd550 2024-04-25T13:52:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe62659c640 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1ab99f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe6180df280 tx=0x7fe6180a6be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf08cb11589fcbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe60dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:52:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe60dffb640 1 -- 172.21.15.27:0/1695916026 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6180a7470 con 0x55f5a16dd550 2024-04-25T13:52:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe633835740 1 -- 172.21.15.27:0/1695916026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe620036030 con 0x55f5a16dd550 2024-04-25T13:52:22.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.745+0000 7fe60dffb640 1 -- 172.21.15.27:0/1695916026 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe6180a3a60 con 0x55f5a16dd550 2024-04-25T13:52:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.746+0000 7fe60dffb640 1 -- 172.21.15.27:0/1695916026 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe6180a7470 con 0x55f5a16dd550 2024-04-25T13:52:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.760+0000 7fe633835740 10 client.17667 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.760+0000 7fe633835740 1 -- 172.21.15.27:0/1695916026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f5a16de170 con 0x55f5a16dd550 2024-04-25T13:52:22.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.767+0000 7fe60dffb640 1 -- 172.21.15.27:0/1695916026 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe620036030 con 0x55f5a16dd550 2024-04-25T13:52:22.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.767+0000 7fe60dffb640 1 -- 172.21.15.27:0/1695916026 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe6180c8170 con 0x55f5a16dd550 2024-04-25T13:52:22.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.769+0000 7fe633835740 10 client.17667 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:22.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.769+0000 7fe633835740 10 client.17667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:22.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.769+0000 7fe633835740 20 client.17667 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:22.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.769+0000 7fe633835740 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5a16c51f0 0x55f5a1ac7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.770+0000 7fe633835740 4 client.17667 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:22.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.770+0000 7fe633835740 1 -- 172.21.15.27:0/1695916026 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5a1a1a710 con 0x55f5a16c51f0 2024-04-25T13:52:22.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.774+0000 7fe625d9b640 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5a16c51f0 0x55f5a1ac7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.783+0000 7fe625d9b640 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5a16c51f0 0x55f5a1ac7310 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6624135dfc586430 server_cookie=9a7992a25d2ff478 in_seq=0 out_seq=0 2024-04-25T13:52:22.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.784+0000 7fe60dffb640 10 client.17667 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.786+0000 7fe60dffb640 1 -- 172.21.15.27:0/1695916026 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f5a1a1a710 con 0x55f5a16c51f0 2024-04-25T13:52:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.786+0000 7fe60dffb640 10 client.17667 handle_command_reply: tid=0 2024-04-25T13:52:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.787+0000 7fe633835740 1 client.17667 shutdown 2024-04-25T13:52:22.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.788+0000 7fe633835740 1 -- 172.21.15.27:0/1695916026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 msgr2=0x55f5a1ab99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.788+0000 7fe633835740 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1ab99f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe6180df280 tx=0x7fe6180a6be0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 1 -- 172.21.15.27:0/1695916026 shutdown_connections 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5a16c51f0 0x55f5a1ac7310 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 0x55f5a1ab9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 1 --2- 172.21.15.27:0/1695916026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16dd550 0x55f5a1ab99f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 1 -- 172.21.15.27:0/1695916026 >> 172.21.15.27:0/1695916026 conn(0x55f5a16fa560 msgr2=0x55f5a1aa1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe60dffb640 0 client.17667 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 1 -- 172.21.15.27:0/1695916026 shutdown_connections 2024-04-25T13:52:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 1 -- 172.21.15.27:0/1695916026 wait complete. 2024-04-25T13:52:22.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.790+0000 7fe633835740 20 client.17667 trim_cache size 0 max 16384 2024-04-25T13:52:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.818+0000 7fe633835740 1 Processor -- start 2024-04-25T13:52:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.818+0000 7fe633835740 1 -- start start 2024-04-25T13:52:22.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe633835740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1aa6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe633835740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 0x55f5a1aa6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe633835740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe633835740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe6281af9a0 con 0x55f5a1706410 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe633835740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe6281b2280 con 0x55f5a16e5730 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe633835740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe6281b0070 con 0x55f5a16dd550 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe62659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1aa6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe626d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe62659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1aa6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54150/0 (socket says 172.21.15.27:54150) 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe626d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43230/0 (socket says 172.21.15.27:43230) 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe62659c640 1 -- 172.21.15.27:0/4192345694 learned_addr learned my addr 172.21.15.27:0/4192345694 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe625d9b640 1 --2- 172.21.15.27:0/4192345694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 0x55f5a1aa6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe626d9d640 1 -- 172.21.15.27:0/4192345694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 msgr2=0x55f5a1aa6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe626d9d640 1 --2- 172.21.15.27:0/4192345694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1aa6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe626d9d640 1 -- 172.21.15.27:0/4192345694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 msgr2=0x55f5a1aa6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe626d9d640 1 --2- 172.21.15.27:0/4192345694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 0x55f5a1aa6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.819+0000 7fe626d9d640 1 -- 172.21.15.27:0/4192345694 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5a1aac270 con 0x55f5a1706410 2024-04-25T13:52:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.820+0000 7fe626d9d640 1 --2- 172.21.15.27:0/4192345694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097b0 tx=0x7fe62002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33386a6c2cc4bf0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.820+0000 7fe60ffff640 1 -- 172.21.15.27:0/4192345694 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620031450 con 0x55f5a1706410 2024-04-25T13:52:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.820+0000 7fe60ffff640 1 -- 172.21.15.27:0/4192345694 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe620036070 con 0x55f5a1706410 2024-04-25T13:52:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.820+0000 7fe60ffff640 1 -- 172.21.15.27:0/4192345694 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620004600 con 0x55f5a1706410 2024-04-25T13:52:22.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 -- 172.21.15.27:0/4192345694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a1706410 msgr2=0x55f5a1ac0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 --2- 172.21.15.27:0/4192345694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097b0 tx=0x7fe62002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 -- 172.21.15.27:0/4192345694 shutdown_connections 2024-04-25T13:52:22.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 --2- 172.21.15.27:0/4192345694 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a1706410 0x55f5a1ac0e20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 --2- 172.21.15.27:0/4192345694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a16e5730 0x55f5a1aa6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 --2- 172.21.15.27:0/4192345694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1aa6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 -- 172.21.15.27:0/4192345694 >> 172.21.15.27:0/4192345694 conn(0x55f5a16fa560 msgr2=0x55f5a1aa5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 -- 172.21.15.27:0/4192345694 shutdown_connections 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 -- 172.21.15.27:0/4192345694 wait complete. 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 -- start start 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.821+0000 7fe633835740 1 Processor -- start 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe633835740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1ac0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe633835740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16e5730 0x55f5a1ac0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe633835740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1b2dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe633835740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe6281b0740 con 0x55f5a16e5730 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe633835740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe6281b1bb0 con 0x55f5a1706410 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe633835740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe6281b0e10 con 0x55f5a16dd550 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe633835740 10 client.0 _ll_register_callbacks cb 0x7fe6180a7240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1ac0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1ac0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54156/0 (socket says 172.21.15.27:54156) 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 -- 172.21.15.27:0/3438151853 learned_addr learned my addr 172.21.15.27:0/3438151853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe625d9b640 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16e5730 0x55f5a1ac0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe626d9d640 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1b2dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 -- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 msgr2=0x55f5a1b2dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1b2dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 -- 172.21.15.27:0/3438151853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16e5730 msgr2=0x55f5a1ac0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16e5730 0x55f5a1ac0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 -- 172.21.15.27:0/3438151853 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5a1b2e280 con 0x55f5a16dd550 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe626d9d640 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1b2dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe62659c640 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1ac0570 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe6180a3a10 tx=0x7fe6180a3400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0133d695dffd0dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.822+0000 7fe60dffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe60dffb640 1 -- 172.21.15.27:0/3438151853 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6180a7470 con 0x55f5a16dd550 2024-04-25T13:52:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe60dffb640 1 -- 172.21.15.27:0/3438151853 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe6180c8180 con 0x55f5a16dd550 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 1 -- 172.21.15.27:0/3438151853 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe620036070 con 0x55f5a16dd550 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe60dffb640 1 -- 172.21.15.27:0/3438151853 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe618089070 con 0x55f5a16dd550 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 10 client.27415 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 1 -- 172.21.15.27:0/3438151853 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f5a1b2f540 con 0x55f5a16dd550 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe60dffb640 1 -- 172.21.15.27:0/3438151853 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe620036070 con 0x55f5a16dd550 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe60dffb640 1 -- 172.21.15.27:0/3438151853 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe61808e480 con 0x55f5a16dd550 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 10 client.27415 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 10 client.27415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 20 client.27415 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5a169a630 0x55f5a1b31f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 4 client.27415 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe633835740 1 -- 172.21.15.27:0/3438151853 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f5a1a14580 con 0x55f5a169a630 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.823+0000 7fe625d9b640 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5a169a630 0x55f5a1b31f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.824+0000 7fe625d9b640 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5a169a630 0x55f5a1b31f30 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2231fc3576aaade server_cookie=28f7af1b16341c20 in_seq=0 out_seq=0 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.824+0000 7fe60dffb640 10 client.27415 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe60dffb640 1 -- 172.21.15.27:0/3438151853 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f5a1a14580 con 0x55f5a169a630 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe60dffb640 10 client.27415 handle_command_reply: tid=0 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 client.27415 shutdown 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 -- 172.21.15.27:0/3438151853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 msgr2=0x55f5a1ac0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1ac0570 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe6180a3a10 tx=0x7fe6180a3400 comp rx=0 tx=0).stop 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 -- 172.21.15.27:0/3438151853 shutdown_connections 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5a169a630 0x55f5a1b31f30 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5a1706410 0x55f5a1b2dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe60dffb640 0 client.27415 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5a16e5730 0x55f5a1ac0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 --2- 172.21.15.27:0/3438151853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5a16dd550 0x55f5a1ac0570 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 -- 172.21.15.27:0/3438151853 >> 172.21.15.27:0/3438151853 conn(0x55f5a16fa560 msgr2=0x55f5a1aa6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 -- 172.21.15.27:0/3438151853 shutdown_connections 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.825+0000 7fe633835740 1 -- 172.21.15.27:0/3438151853 wait complete. 2024-04-25T13:52:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe633835740 20 client.27415 trim_cache size 0 max 16384 2024-04-25T13:52:22.830 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:52:22.830 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:52:22.830 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "7a3414fb-0d83-4603-80a2-dca4a69a0ed3", 2024-04-25T13:52:22.830 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:52:22.830 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 -- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe6100afb70 msgr2=0x7fe6100b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe6100afb70 0x7fe6100b2080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe628074140 tx=0x7fe61c009550 comp rx=0 tx=0).stop 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 -- 172.21.15.27:0/1109367223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6281029e0 msgr2=0x7fe6280747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6281029e0 0x7fe6280747a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe6281022b0 tx=0x7fe61800ee50 comp rx=0 tx=0).stop 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 -- 172.21.15.27:0/1109367223 shutdown_connections 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe6100afb70 0x7fe6100b2080 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628108920 0x7fe6281a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe628103c20 0x7fe6281a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 --2- 172.21.15.27:0/1109367223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6281029e0 0x7fe6280747a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:22.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.826+0000 7fe62c984640 1 -- 172.21.15.27:0/1109367223 >> 172.21.15.27:0/1109367223 conn(0x7fe6280fdbd0 msgr2=0x7fe628109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:22.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.827+0000 7fe62c984640 1 -- 172.21.15.27:0/1109367223 shutdown_connections 2024-04-25T13:52:22.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:22.827+0000 7fe62c984640 1 -- 172.21.15.27:0/1109367223 wait complete. 2024-04-25T13:52:22.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:52:22.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:52:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 675.0 s, 1912.2 tps, lat 52.205 ms stddev 57.853 2024-04-25T13:52:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.062+0000 7feab304a640 1 -- 172.21.15.27:0/3726580198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1033b0 msgr2=0x7feaac1037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.062+0000 7feab304a640 1 --2- 172.21.15.27:0/3726580198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1033b0 0x7feaac1037c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feaa4005fd0 tx=0x7feaa402a690 comp rx=0 tx=0).stop 2024-04-25T13:52:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.062+0000 7feab304a640 1 -- 172.21.15.27:0/3726580198 shutdown_connections 2024-04-25T13:52:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.062+0000 7feab304a640 1 --2- 172.21.15.27:0/3726580198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaac1080b0 0x7feaac10a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.062+0000 7feab304a640 1 --2- 172.21.15.27:0/3726580198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1033b0 0x7feaac1037c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.062+0000 7feab304a640 1 --2- 172.21.15.27:0/3726580198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaac102170 0x7feaac102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.062+0000 7feab304a640 1 -- 172.21.15.27:0/3726580198 >> 172.21.15.27:0/3726580198 conn(0x7feaac0fd340 msgr2=0x7feaac0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.063+0000 7feab304a640 1 -- 172.21.15.27:0/3726580198 shutdown_connections 2024-04-25T13:52:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.063+0000 7feab304a640 1 -- 172.21.15.27:0/3726580198 wait complete. 2024-04-25T13:52:23.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab304a640 1 Processor -- start 2024-04-25T13:52:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab304a640 1 -- start start 2024-04-25T13:52:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab304a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaac102170 0x7feaac1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab304a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaac1033b0 0x7feaac1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab304a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1080b0 0x7feaac1a7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab304a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feaac10d450 con 0x7feaac1033b0 2024-04-25T13:52:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab304a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feaac10d2d0 con 0x7feaac102170 2024-04-25T13:52:23.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab304a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feaac10d5d0 con 0x7feaac1080b0 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab15e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1080b0 0x7feaac1a7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab15e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1080b0 0x7feaac1a7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54182/0 (socket says 172.21.15.27:54182) 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab15e7640 1 -- 172.21.15.27:0/4045348476 learned_addr learned my addr 172.21.15.27:0/4045348476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.064+0000 7feab0de6640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaac102170 0x7feaac1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.065+0000 7feab15e7640 1 -- 172.21.15.27:0/4045348476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaac102170 msgr2=0x7feaac1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.065+0000 7feab15e7640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaac102170 0x7feaac1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.065+0000 7feab15e7640 1 -- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaac1033b0 msgr2=0x7feaac1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.065+0000 7feab15e7640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaac1033b0 0x7feaac1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.065+0000 7feab15e7640 1 -- 172.21.15.27:0/4045348476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac196fc0 con 0x7feaac1080b0 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.065+0000 7feab15e7640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1080b0 0x7feaac1a7d20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00c970 tx=0x7fea9c00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.066+0000 7feaa9ffb640 1 -- 172.21.15.27:0/4045348476 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c007bb0 con 0x7feaac1080b0 2024-04-25T13:52:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.066+0000 7feaa9ffb640 1 -- 172.21.15.27:0/4045348476 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea9c007d50 con 0x7feaac1080b0 2024-04-25T13:52:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.066+0000 7feaa9ffb640 1 -- 172.21.15.27:0/4045348476 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c018800 con 0x7feaac1080b0 2024-04-25T13:52:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.066+0000 7feab304a640 1 -- 172.21.15.27:0/4045348476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac1972b0 con 0x7feaac1080b0 2024-04-25T13:52:23.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.067+0000 7feab304a640 1 -- 172.21.15.27:0/4045348476 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac1977b0 con 0x7feaac1080b0 2024-04-25T13:52:23.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.070+0000 7feab9fde740 1 Processor -- start 2024-04-25T13:52:23.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.070+0000 7feaa9ffb640 1 -- 172.21.15.27:0/4045348476 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fea9c00f450 con 0x7feaac1080b0 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feab9fde740 1 -- start start 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feab9fde740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836db9390 0x55683715aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feab9fde740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x55683715afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feab9fde740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feab9fde740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5568370bfd80 con 0x556836d904d0 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feab9fde740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5568370798b0 con 0x556836db9390 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feab9fde740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5568370bbd40 con 0x556836d986b0 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feaabfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feaabfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54194/0 (socket says 172.21.15.27:54194) 2024-04-25T13:52:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feaabfff640 1 -- 172.21.15.27:0/1798488779 learned_addr learned my addr 172.21.15.27:0/1798488779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.071+0000 7feaabfff640 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836db9390 0x55683715aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feab0de6640 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x55683715afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feaa9ffb640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fea900c0c90 0x7fea900c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feaabfff640 1 -- 172.21.15.27:0/1798488779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836db9390 msgr2=0x55683715aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feaabfff640 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836db9390 0x55683715aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feaabfff640 1 -- 172.21.15.27:0/1798488779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 msgr2=0x55683715afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feaabfff640 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x55683715afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feaa9ffb640 1 -- 172.21.15.27:0/4045348476 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fea9c0deb70 con 0x7feaac1080b0 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feaabfff640 1 -- 172.21.15.27:0/1798488779 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683704aa40 con 0x556836d986b0 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feab15e7640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fea900c0c90 0x7fea900c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.072+0000 7feaabfff640 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715bc70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7feaa400aa80 tx=0x7feaa4031a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e33372ef00d6e67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.073+0000 7feaa2ffd640 1 -- 172.21.15.27:0/1798488779 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4031790 con 0x556836d986b0 2024-04-25T13:52:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.073+0000 7feaa2ffd640 1 -- 172.21.15.27:0/1798488779 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaa400a8d0 con 0x556836d986b0 2024-04-25T13:52:23.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.073+0000 7feaa2ffd640 1 -- 172.21.15.27:0/1798488779 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa402f510 con 0x556836d986b0 2024-04-25T13:52:23.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.074+0000 7feab9fde740 1 -- 172.21.15.27:0/1798488779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 msgr2=0x55683715bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.074+0000 7feab9fde740 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715bc70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7feaa400aa80 tx=0x7feaa4031a70 comp rx=0 tx=0).stop 2024-04-25T13:52:23.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.074+0000 7feab15e7640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fea900c0c90 0x7fea900c31a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00afd0 tx=0x7fea9c00fe00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.075+0000 7feab9fde740 1 -- 172.21.15.27:0/1798488779 shutdown_connections 2024-04-25T13:52:23.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.075+0000 7feab9fde740 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715bc70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.075+0000 7feab9fde740 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x55683715afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.075+0000 7feab9fde740 1 --2- 172.21.15.27:0/1798488779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836db9390 0x55683715aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.075+0000 7feab9fde740 1 -- 172.21.15.27:0/1798488779 >> 172.21.15.27:0/1798488779 conn(0x556836dad4e0 msgr2=0x556837156270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.075+0000 7feab9fde740 1 -- 172.21.15.27:0/1798488779 shutdown_connections 2024-04-25T13:52:23.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.075+0000 7feab9fde740 1 -- 172.21.15.27:0/1798488779 wait complete. 2024-04-25T13:52:23.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.076+0000 7feab9fde740 1 -- start start 2024-04-25T13:52:23.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.076+0000 7feab9fde740 1 Processor -- start 2024-04-25T13:52:23.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feab9fde740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x556837164900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feab9fde740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d986b0 0x556837164e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feab9fde740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836db9390 0x55683716bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feab9fde740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55683715c770 con 0x556836d904d0 2024-04-25T13:52:23.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feaabfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x556837164900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feab0de6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d986b0 0x556837164e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feaabfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x556837164900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43272/0 (socket says 172.21.15.27:43272) 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feaabfff640 1 -- 172.21.15.27:0/115171584 learned_addr learned my addr 172.21.15.27:0/115171584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556837046d10 con 0x556836d986b0 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55683715c8f0 con 0x556836db9390 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.077+0000 7feab9fde740 10 client.0 _ll_register_callbacks cb 0x556837073210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feaabfff640 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836db9390 0x55683716bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab0de6640 1 -- 172.21.15.27:0/115171584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836db9390 msgr2=0x55683716bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab0de6640 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836db9390 0x55683716bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab0de6640 1 -- 172.21.15.27:0/115171584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 msgr2=0x556837164900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab0de6640 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x556837164900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab0de6640 1 -- 172.21.15.27:0/115171584 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556836e9b7d0 con 0x556836d986b0 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feaabfff640 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836db9390 0x55683716bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab0de6640 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d986b0 0x556837164e40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55683715a780 tx=0x7fea9800bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6318754da91a3ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7fea8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7fea8ffff640 1 -- 172.21.15.27:0/115171584 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98004230 con 0x556836d986b0 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaa400a8d0 con 0x556836d986b0 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7fea8ffff640 1 -- 172.21.15.27:0/115171584 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fea980043d0 con 0x556836d986b0 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7fea8ffff640 1 -- 172.21.15.27:0/115171584 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fea98017720 con 0x556836d986b0 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab9fde740 10 client.27383 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.078+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556836dd13f0 con 0x556836d986b0 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7fea8ffff640 1 -- 172.21.15.27:0/115171584 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaa400a8d0 con 0x556836d986b0 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7fea8ffff640 1 -- 172.21.15.27:0/115171584 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fea98003c40 con 0x556836d986b0 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7feab9fde740 10 client.27383 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7feab9fde740 10 client.27383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7feab9fde740 20 client.27383 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7feab9fde740 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556836d77f60 0x556837172370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7feab9fde740 4 client.27383 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568370cd990 con 0x556836d77f60 2024-04-25T13:52:23.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.079+0000 7feaabfff640 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556836d77f60 0x556837172370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.080+0000 7fea8ffff640 10 client.27383 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:23.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.080+0000 7feaabfff640 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556836d77f60 0x556837172370 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df22a40839f73954 server_cookie=6f92cb7c5e91422e in_seq=0 out_seq=0 2024-04-25T13:52:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.080+0000 7fea8ffff640 1 -- 172.21.15.27:0/115171584 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5568370cd990 con 0x556836d77f60 2024-04-25T13:52:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.080+0000 7fea8ffff640 10 client.27383 handle_command_reply: tid=0 2024-04-25T13:52:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.080+0000 7feab9fde740 1 client.27383 shutdown 2024-04-25T13:52:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.080+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d986b0 msgr2=0x556837164e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d986b0 0x556837164e40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55683715a780 tx=0x7fea9800bf90 comp rx=0 tx=0).stop 2024-04-25T13:52:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 shutdown_connections 2024-04-25T13:52:23.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556836d77f60 0x556837172370 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836db9390 0x55683716bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7fea8ffff640 0 client.27383 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d986b0 0x556837164e40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 --2- 172.21.15.27:0/115171584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836d904d0 0x556837164900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 >> 172.21.15.27:0/115171584 conn(0x556836dad4e0 msgr2=0x556837159180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 shutdown_connections 2024-04-25T13:52:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 1 -- 172.21.15.27:0/115171584 wait complete. 2024-04-25T13:52:23.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.081+0000 7feab9fde740 20 client.27383 trim_cache size 0 max 16384 2024-04-25T13:52:23.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.105+0000 7feab9fde740 1 Processor -- start 2024-04-25T13:52:23.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.105+0000 7feab9fde740 1 -- start start 2024-04-25T13:52:23.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.105+0000 7feab9fde740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x55683715fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.105+0000 7feab9fde740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.105+0000 7feab9fde740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 0x556837159780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.105+0000 7feab9fde740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feaac19e3a0 con 0x556836db9390 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feab9fde740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feaac06da90 con 0x556836d904d0 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feab9fde740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feaac19ea10 con 0x556836d986b0 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feaabfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x55683715fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feab0de6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feab0de6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54206/0 (socket says 172.21.15.27:54206) 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feab0de6640 1 -- 172.21.15.27:0/2175751297 learned_addr learned my addr 172.21.15.27:0/2175751297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feaabfff640 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 0x556837159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feaabfff640 1 -- 172.21.15.27:0/2175751297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 msgr2=0x55683715ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.106+0000 7feaabfff640 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.107+0000 7feaabfff640 1 -- 172.21.15.27:0/2175751297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 msgr2=0x556837159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.107+0000 7feaabfff640 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 0x556837159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.107+0000 7feaabfff640 1 -- 172.21.15.27:0/2175751297 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556836d910f0 con 0x556836d904d0 2024-04-25T13:52:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.107+0000 7feab0de6640 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.108+0000 7feaabfff640 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x55683715fcf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x556837172a70 tx=0x7feaa4000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=523512c56bf3d883 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.108+0000 7feaa27fc640 1 -- 172.21.15.27:0/2175751297 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4031740 con 0x556836d904d0 2024-04-25T13:52:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.108+0000 7feaa27fc640 1 -- 172.21.15.27:0/2175751297 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaa4041050 con 0x556836d904d0 2024-04-25T13:52:23.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.108+0000 7feaa27fc640 1 -- 172.21.15.27:0/2175751297 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa402f770 con 0x556836d904d0 2024-04-25T13:52:23.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 -- 172.21.15.27:0/2175751297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 msgr2=0x55683715fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x55683715fcf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x556837172a70 tx=0x7feaa4000f40 comp rx=0 tx=0).stop 2024-04-25T13:52:23.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 -- 172.21.15.27:0/2175751297 shutdown_connections 2024-04-25T13:52:23.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 0x556837159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x55683715ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 --2- 172.21.15.27:0/2175751297 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x55683715fcf0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 -- 172.21.15.27:0/2175751297 >> 172.21.15.27:0/2175751297 conn(0x556836dad4e0 msgr2=0x556837159070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 -- 172.21.15.27:0/2175751297 shutdown_connections 2024-04-25T13:52:23.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.109+0000 7feab9fde740 1 -- 172.21.15.27:0/2175751297 wait complete. 2024-04-25T13:52:23.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.110+0000 7feab9fde740 1 -- start start 2024-04-25T13:52:23.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.110+0000 7feab9fde740 1 Processor -- start 2024-04-25T13:52:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.110+0000 7feab9fde740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x556837169720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.110+0000 7feab9fde740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x556837169c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feab9fde740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 0x5568371d86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feab9fde740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feaac19f0e0 con 0x556836db9390 2024-04-25T13:52:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feab9fde740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feaac06d3c0 con 0x556836d904d0 2024-04-25T13:52:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feab9fde740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feaac06c640 con 0x556836d986b0 2024-04-25T13:52:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x556837169720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feab0de6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x556837169c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x556837169720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:35392/0 (socket says 172.21.15.27:35392) 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 -- 172.21.15.27:0/1018525491 learned_addr learned my addr 172.21.15.27:0/1018525491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feab0de6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x556837169c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54214/0 (socket says 172.21.15.27:54214) 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feab9fde740 10 client.0 _ll_register_callbacks cb 0x7fea9800b930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 0x5568371d86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 -- 172.21.15.27:0/1018525491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 msgr2=0x556837169c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x556837169c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 -- 172.21.15.27:0/1018525491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 msgr2=0x5568371d86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 0x5568371d86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 -- 172.21.15.27:0/1018525491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568370804b0 con 0x556836d904d0 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7feaabfff640 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x556837169720 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7feaa400a6f0 tx=0x7feaa402ab90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c02191f13765d83b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.111+0000 7fea8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7fea8ffff640 1 -- 172.21.15.27:0/1018525491 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4054150 con 0x556836d904d0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7fea8ffff640 1 -- 172.21.15.27:0/1018525491 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feaa40542f0 con 0x556836d904d0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 1 -- 172.21.15.27:0/1018525491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaa4041050 con 0x556836d904d0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7fea8ffff640 1 -- 172.21.15.27:0/1018525491 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7feaa4067020 con 0x556836d904d0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 10 client.27389 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 1 -- 172.21.15.27:0/1018525491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5568371d9ce0 con 0x556836d904d0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7fea8ffff640 1 -- 172.21.15.27:0/1018525491 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaa40544f0 con 0x556836d904d0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7fea8ffff640 1 -- 172.21.15.27:0/1018525491 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7feaa403d430 con 0x556836d904d0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 10 client.27389 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 10 client.27389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 20 client.27389 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556836d4d3a0 0x5568371dc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 4 client.27389 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.112+0000 7feab9fde740 1 -- 172.21.15.27:0/1018525491 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7feaa4041050 con 0x556836d4d3a0 2024-04-25T13:52:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.113+0000 7feab0de6640 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556836d4d3a0 0x5568371dc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.113+0000 7fea8ffff640 10 client.27389 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.113+0000 7feab0de6640 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556836d4d3a0 0x5568371dc1d0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8239f5ac4d7a58d7 server_cookie=a03f05cb12313f3e in_seq=0 out_seq=0 2024-04-25T13:52:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7fea8ffff640 1 -- 172.21.15.27:0/1018525491 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7feaa4041050 con 0x556836d4d3a0 2024-04-25T13:52:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7fea8ffff640 10 client.27389 handle_command_reply: tid=0 2024-04-25T13:52:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 client.27389 shutdown 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 -- 172.21.15.27:0/1018525491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 msgr2=0x556837169720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x556837169720 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7feaa400a6f0 tx=0x7feaa402ab90 comp rx=0 tx=0).stop 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 -- 172.21.15.27:0/1018525491 shutdown_connections 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556836d4d3a0 0x5568371dc1d0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556836db9390 0x5568371d86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7fea8ffff640 0 client.27389 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556836d986b0 0x556837169c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 --2- 172.21.15.27:0/1018525491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556836d904d0 0x556837169720 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 -- 172.21.15.27:0/1018525491 >> 172.21.15.27:0/1018525491 conn(0x556836dad4e0 msgr2=0x5568371549c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.114+0000 7feab9fde740 1 -- 172.21.15.27:0/1018525491 shutdown_connections 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.115+0000 7feab9fde740 1 -- 172.21.15.27:0/1018525491 wait complete. 2024-04-25T13:52:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.115+0000 7feab9fde740 20 client.27389 trim_cache size 0 max 16384 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: "7a3414fb-0d83-4603-80a2-dca4a69a0ed3": { 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "7a3414fb-0d83-4603-80a2-dca4a69a0ed3", 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:52:23.118 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:52:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.115+0000 7feab304a640 1 -- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fea900c0c90 msgr2=0x7fea900c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fea900c0c90 0x7fea900c31a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00afd0 tx=0x7fea9c00fe00 comp rx=0 tx=0).stop 2024-04-25T13:52:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 -- 172.21.15.27:0/4045348476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1080b0 msgr2=0x7feaac1a7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1080b0 0x7feaac1a7d20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00c970 tx=0x7fea9c00ce30 comp rx=0 tx=0).stop 2024-04-25T13:52:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 -- 172.21.15.27:0/4045348476 shutdown_connections 2024-04-25T13:52:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fea900c0c90 0x7fea900c31a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feaac1080b0 0x7feaac1a7d20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaac1033b0 0x7feaac1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 --2- 172.21.15.27:0/4045348476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaac102170 0x7feaac1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 -- 172.21.15.27:0/4045348476 >> 172.21.15.27:0/4045348476 conn(0x7feaac0fd340 msgr2=0x7feaac104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 -- 172.21.15.27:0/4045348476 shutdown_connections 2024-04-25T13:52:23.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:23.116+0000 7feab304a640 1 -- 172.21.15.27:0/4045348476 wait complete. 2024-04-25T13:52:23.130 INFO:tasks.cephfs.filesystem:scrub status for tag:7a3414fb-0d83-4603-80a2-dca4a69a0ed3 - {'path': '/', 'tag': '7a3414fb-0d83-4603-80a2-dca4a69a0ed3', 'options': 'recursive,force'} 2024-04-25T13:52:23.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:23 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:52:23.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:23 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:23.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:23 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:52:23.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:23 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:52:23.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:23 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:23.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:23 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:52:23.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:23 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:52:23.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:23 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:23.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:23 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.653+0000 7feeb18af640 1 -- 172.21.15.27:0/3539976240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 msgr2=0x7feeac111b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.653+0000 7feeb18af640 1 --2- 172.21.15.27:0/3539976240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 0x7feeac111b80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7feea0009900 tx=0x7feea002a3f0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.654+0000 7feeb18af640 1 -- 172.21.15.27:0/3539976240 shutdown_connections 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.654+0000 7feeb18af640 1 --2- 172.21.15.27:0/3539976240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1120c0 0x7feeac114560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.654+0000 7feeb18af640 1 --2- 172.21.15.27:0/3539976240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 0x7feeac111b80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.654+0000 7feeb18af640 1 --2- 172.21.15.27:0/3539976240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac109fc0 0x7feeac10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.654+0000 7feeb18af640 1 -- 172.21.15.27:0/3539976240 >> 172.21.15.27:0/3539976240 conn(0x7feeac070540 msgr2=0x7feeac0729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.654+0000 7feeb18af640 1 -- 172.21.15.27:0/3539976240 shutdown_connections 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.654+0000 7feeb18af640 1 -- 172.21.15.27:0/3539976240 wait complete. 2024-04-25T13:52:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.654+0000 7feeb18af640 1 Processor -- start 2024-04-25T13:52:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeb18af640 1 -- start start 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeb18af640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac109fc0 0x7feeac1a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeb18af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 0x7feeac1a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeb18af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1120c0 0x7feeac1ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeb18af640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feeac11a120 con 0x7feeac1120c0 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeb18af640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feeac119fa0 con 0x7feeac109fc0 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeb18af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feeac11a2a0 con 0x7feeac10f760 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeaaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac109fc0 0x7feeac1a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeaaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac109fc0 0x7feeac1a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:35406/0 (socket says 172.21.15.27:35406) 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeaa7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 0x7feeac1a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeaaffd640 1 -- 172.21.15.27:0/2884327592 learned_addr learned my addr 172.21.15.27:0/2884327592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.655+0000 7feeab7fe640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1120c0 0x7feeac1ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeaaffd640 1 -- 172.21.15.27:0/2884327592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 msgr2=0x7feeac1a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeaaffd640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 0x7feeac1a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeaaffd640 1 -- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1120c0 msgr2=0x7feeac1ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeaaffd640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1120c0 0x7feeac1ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeaaffd640 1 -- 172.21.15.27:0/2884327592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac1ac830 con 0x7feeac109fc0 2024-04-25T13:52:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeaaffd640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac109fc0 0x7feeac1a4d00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7feea400d770 tx=0x7feea400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.657 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 0x7feeac1a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeb08ad640 1 -- 172.21.15.27:0/2884327592 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4017070 con 0x7feeac109fc0 2024-04-25T13:52:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeb08ad640 1 -- 172.21.15.27:0/2884327592 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feea4009c30 con 0x7feeac109fc0 2024-04-25T13:52:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeb08ad640 1 -- 172.21.15.27:0/2884327592 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea400f620 con 0x7feeac109fc0 2024-04-25T13:52:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.656+0000 7feeb18af640 1 -- 172.21.15.27:0/2884327592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeac114e60 con 0x7feeac109fc0 2024-04-25T13:52:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.657+0000 7feeb18af640 1 -- 172.21.15.27:0/2884327592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeac115360 con 0x7feeac109fc0 2024-04-25T13:52:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb08ad640 1 -- 172.21.15.27:0/2884327592 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7feea401a070 con 0x7feeac109fc0 2024-04-25T13:52:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb8765740 1 Processor -- start 2024-04-25T13:52:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb8765740 1 -- start start 2024-04-25T13:52:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb8765740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de34d1b0 0x55f0de6ed970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb8765740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6edef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb8765740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de32c4d0 0x55f0de6f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb8765740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f0de653460 con 0x55f0de32c4d0 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb8765740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f0de60d590 con 0x55f0de3242f0 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.660+0000 7feeb8765740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f0de64f420 con 0x55f0de34d1b0 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de34d1b0 0x55f0de6ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de34d1b0 0x55f0de6ed970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54228/0 (socket says 172.21.15.27:54228) 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeaaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 -- 172.21.15.27:0/2306513232 learned_addr learned my addr 172.21.15.27:0/2306513232 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de32c4d0 0x55f0de6f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 -- 172.21.15.27:0/2306513232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 msgr2=0x55f0de6edef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6edef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 -- 172.21.15.27:0/2306513232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de32c4d0 msgr2=0x55f0de6f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de32c4d0 0x55f0de6f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 -- 172.21.15.27:0/2306513232 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0de61a0d0 con 0x55f0de34d1b0 2024-04-25T13:52:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeb08ad640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee940afc60 0x7fee940b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeaaffd640 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6edef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeab7fe640 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de34d1b0 0x55f0de6ed970 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fee9c009fc0 tx=0x7fee9c010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46198918d48a7915 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.661+0000 7feeb08ad640 1 -- 172.21.15.27:0/2884327592 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feea40de5d0 con 0x7feeac109fc0 2024-04-25T13:52:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.662+0000 7feeaaffd640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee940afc60 0x7fee940b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.662+0000 7fee91ffb640 1 -- 172.21.15.27:0/2306513232 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c003b70 con 0x55f0de34d1b0 2024-04-25T13:52:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.662+0000 7fee91ffb640 1 -- 172.21.15.27:0/2306513232 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee9c003d10 con 0x55f0de34d1b0 2024-04-25T13:52:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.663+0000 7feeb8765740 1 -- 172.21.15.27:0/2306513232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de34d1b0 msgr2=0x55f0de6ed970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.663+0000 7feeb8765740 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de34d1b0 0x55f0de6ed970 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fee9c009fc0 tx=0x7fee9c010860 comp rx=0 tx=0).stop 2024-04-25T13:52:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.663+0000 7feeb8765740 1 -- 172.21.15.27:0/2306513232 shutdown_connections 2024-04-25T13:52:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.663+0000 7feeb8765740 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de32c4d0 0x55f0de6f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.663+0000 7feeaaffd640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee940afc60 0x7fee940b2170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x55f0de6eed50 tx=0x7feea40e4110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.663+0000 7feeb8765740 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6edef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.663+0000 7feeb8765740 1 --2- 172.21.15.27:0/2306513232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de34d1b0 0x55f0de6ed970 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.663+0000 7feeb8765740 1 -- 172.21.15.27:0/2306513232 >> 172.21.15.27:0/2306513232 conn(0x55f0de341300 msgr2=0x55f0de6e9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.664+0000 7feeb8765740 1 -- 172.21.15.27:0/2306513232 shutdown_connections 2024-04-25T13:52:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.665+0000 7feeb8765740 1 -- 172.21.15.27:0/2306513232 wait complete. 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.665+0000 7feeb8765740 1 -- start start 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeb8765740 1 Processor -- start 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeb8765740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeb8765740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeb8765740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6fed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeb8765740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f0de6f58e0 con 0x55f0de34d1b0 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeb8765740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f0de5dab60 con 0x55f0de32c4d0 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeab7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeb8765740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f0de6f5a60 con 0x55f0de3242f0 2024-04-25T13:52:23.668 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeab7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:54232/0 (socket says 172.21.15.27:54232) 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeab7fe640 1 -- 172.21.15.27:0/502495954 learned_addr learned my addr 172.21.15.27:0/502495954 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeb8765740 10 client.0 _ll_register_callbacks cb 0x55f0de5ff2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.666+0000 7feeab7fe640 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6fed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeab7fe640 1 -- 172.21.15.27:0/502495954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 msgr2=0x55f0de6f7e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeab7fe640 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6f7e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeab7fe640 1 -- 172.21.15.27:0/502495954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 msgr2=0x55f0de6fed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeab7fe640 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6fed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeab7fe640 1 -- 172.21.15.27:0/502495954 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0de365210 con 0x55f0de3242f0 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeab7fe640 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fee9c005d50 tx=0x7fee9c013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3a095e3b922176a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7fee8affd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7fee8affd640 1 -- 172.21.15.27:0/502495954 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c003970 con 0x55f0de3242f0 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7fee8affd640 1 -- 172.21.15.27:0/502495954 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fee9c003b10 con 0x55f0de3242f0 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeb8765740 1 -- 172.21.15.27:0/502495954 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f0de661070 con 0x55f0de3242f0 2024-04-25T13:52:23.669 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7fee8affd640 1 -- 172.21.15.27:0/502495954 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fee9c003d10 con 0x55f0de3242f0 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeb8765740 10 client.27442 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7feeb8765740 1 -- 172.21.15.27:0/502495954 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f0de431620 con 0x55f0de3242f0 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.667+0000 7fee8affd640 1 -- 172.21.15.27:0/502495954 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f0de661070 con 0x55f0de3242f0 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7fee8affd640 1 -- 172.21.15.27:0/502495954 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fee9c029020 con 0x55f0de3242f0 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7feeb8765740 10 client.27442 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7feeb8765740 10 client.27442 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7feeb8765740 20 client.27442 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7feeb8765740 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f0de30bd80 0x55f0de705430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7feeb8765740 4 client.27442 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7feeb8765740 1 -- 172.21.15.27:0/502495954 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0de65aee0 con 0x55f0de30bd80 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f0de30bd80 0x55f0de705430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7fee8affd640 10 client.27442 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.668+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f0de30bd80 0x55f0de705430 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f573c25b2ff36c9 server_cookie=2d6137505d68e09e in_seq=0 out_seq=0 2024-04-25T13:52:23.671 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7fee8affd640 1 -- 172.21.15.27:0/502495954 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f0de65aee0 con 0x55f0de30bd80 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7fee8affd640 10 client.27442 handle_command_reply: tid=0 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 client.27442 shutdown 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 -- 172.21.15.27:0/502495954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 msgr2=0x55f0de6f7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fee9c005d50 tx=0x7fee9c013040 comp rx=0 tx=0).stop 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 -- 172.21.15.27:0/502495954 shutdown_connections 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f0de30bd80 0x55f0de705430 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6fed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7fee8affd640 0 client.27442 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 --2- 172.21.15.27:0/502495954 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7930 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.669+0000 7feeb8765740 1 -- 172.21.15.27:0/502495954 >> 172.21.15.27:0/502495954 conn(0x55f0de341300 msgr2=0x55f0de6e82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.670+0000 7feeb8765740 1 -- 172.21.15.27:0/502495954 shutdown_connections 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.670+0000 7feeb8765740 1 -- 172.21.15.27:0/502495954 wait complete. 2024-04-25T13:52:23.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.670+0000 7feeb8765740 20 client.27442 trim_cache size 0 max 16384 2024-04-25T13:52:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeb8765740 1 Processor -- start 2024-04-25T13:52:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeb8765740 1 -- start start 2024-04-25T13:52:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeb8765740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeb8765740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6fecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeb8765740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeb8765740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1af820 con 0x55f0de34d1b0 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeb8765740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b20a0 con 0x55f0de32c4d0 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeb8765740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1afe90 con 0x55f0de3242f0 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeaa7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6fecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeab7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeaa7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6fecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43102/0 (socket says 172.21.15.27:43102) 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.688+0000 7feeaa7fc640 1 -- 172.21.15.27:0/1483063440 learned_addr learned my addr 172.21.15.27:0/1483063440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.689+0000 7feeab7fe640 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.689+0000 7feeaa7fc640 1 -- 172.21.15.27:0/1483063440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 msgr2=0x55f0de6f7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.689+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.689+0000 7feeaa7fc640 1 -- 172.21.15.27:0/1483063440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 msgr2=0x55f0de6f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.689+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6f5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.689+0000 7feeaa7fc640 1 -- 172.21.15.27:0/1483063440 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0de6e3550 con 0x55f0de32c4d0 2024-04-25T13:52:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.689+0000 7feeab7fe640 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.689+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6fecf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feea002a9b0 tx=0x7feea00314a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc62399937e207e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.690+0000 7fee917fa640 1 -- 172.21.15.27:0/1483063440 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0035040 con 0x55f0de32c4d0 2024-04-25T13:52:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.690+0000 7fee917fa640 1 -- 172.21.15.27:0/1483063440 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feea0052030 con 0x55f0de32c4d0 2024-04-25T13:52:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.690+0000 7fee917fa640 1 -- 172.21.15.27:0/1483063440 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0003b20 con 0x55f0de32c4d0 2024-04-25T13:52:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.691+0000 7feeb8765740 1 -- 172.21.15.27:0/1483063440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 msgr2=0x55f0de6fecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.691+0000 7feeb8765740 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6fecf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feea002a9b0 tx=0x7feea00314a0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.692+0000 7feeb8765740 1 -- 172.21.15.27:0/1483063440 shutdown_connections 2024-04-25T13:52:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.692+0000 7feeb8765740 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.692+0000 7feeb8765740 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de32c4d0 0x55f0de6fecf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.692+0000 7feeb8765740 1 --2- 172.21.15.27:0/1483063440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de3242f0 0x55f0de6f7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.692+0000 7feeb8765740 1 -- 172.21.15.27:0/1483063440 >> 172.21.15.27:0/1483063440 conn(0x55f0de341300 msgr2=0x55f0de6f7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.692+0000 7feeb8765740 1 -- 172.21.15.27:0/1483063440 shutdown_connections 2024-04-25T13:52:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.692+0000 7feeb8765740 1 -- 172.21.15.27:0/1483063440 wait complete. 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.692+0000 7feeb8765740 1 -- start start 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.693+0000 7feeb8765740 1 Processor -- start 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.693+0000 7feeb8765740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6feba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.693+0000 7feeb8765740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de32c4d0 0x55f0de6fb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.693+0000 7feeb8765740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.693+0000 7feeb8765740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b0560 con 0x55f0de34d1b0 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.693+0000 7feeb8765740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b19d0 con 0x55f0de3242f0 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.693+0000 7feeb8765740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feeac1b0c30 con 0x55f0de32c4d0 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeb8765740 10 client.0 _ll_register_callbacks cb 0x7feea0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de32c4d0 0x55f0de6fb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeab7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6feba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de32c4d0 0x55f0de6fb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43602/0 (socket says 172.21.15.27:43602) 2024-04-25T13:52:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeab7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6feba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:43110/0 (socket says 172.21.15.27:43110) 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 -- 172.21.15.27:0/3996082900 learned_addr learned my addr 172.21.15.27:0/3996082900 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeab7fe640 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 -- 172.21.15.27:0/3996082900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 msgr2=0x55f0de6feba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6feba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 -- 172.21.15.27:0/3996082900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 msgr2=0x55f0de6fbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6fbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 -- 172.21.15.27:0/3996082900 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0de76b790 con 0x55f0de32c4d0 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7feeaa7fc640 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de32c4d0 0x55f0de6fb7c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7feea0042010 tx=0x7feea002a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6cb6ac90e751581 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7fee8affd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7fee8affd640 1 -- 172.21.15.27:0/3996082900 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0035040 con 0x55f0de32c4d0 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.694+0000 7fee8affd640 1 -- 172.21.15.27:0/3996082900 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feea0003aa0 con 0x55f0de32c4d0 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 1 -- 172.21.15.27:0/3996082900 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feea0052030 con 0x55f0de32c4d0 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7fee8affd640 1 -- 172.21.15.27:0/3996082900 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7feea0003ca0 con 0x55f0de32c4d0 2024-04-25T13:52:23.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 10 client.27448 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 1 -- 172.21.15.27:0/3996082900 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f0de76c8d0 con 0x55f0de32c4d0 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7fee8affd640 1 -- 172.21.15.27:0/3996082900 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feea0052030 con 0x55f0de32c4d0 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7fee8affd640 1 -- 172.21.15.27:0/3996082900 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7feea004e160 con 0x55f0de32c4d0 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 10 client.27448 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 10 client.27448 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 20 client.27448 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f0de2e11c0 0x55f0de76f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 4 client.27448 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeb8765740 1 -- 172.21.15.27:0/3996082900 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f0de5d65f0 con 0x55f0de2e11c0 2024-04-25T13:52:23.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.695+0000 7feeab7fe640 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f0de2e11c0 0x55f0de76f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.696+0000 7fee8affd640 10 client.27448 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:23.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.696+0000 7feeab7fe640 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f0de2e11c0 0x55f0de76f290 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=654dca3e16fbe014 server_cookie=861f2dd339788d1c in_seq=0 out_seq=0 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.708+0000 7fee8affd640 1 -- 172.21.15.27:0/3996082900 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f0de5d65f0 con 0x55f0de2e11c0 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.708+0000 7fee8affd640 10 client.27448 handle_command_reply: tid=0 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 client.27448 shutdown 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 -- 172.21.15.27:0/3996082900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de32c4d0 msgr2=0x55f0de6fb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de32c4d0 0x55f0de6fb7c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7feea0042010 tx=0x7feea002a3f0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 -- 172.21.15.27:0/3996082900 shutdown_connections 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f0de2e11c0 0x55f0de76f290 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f0de34d1b0 0x55f0de6fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f0de32c4d0 0x55f0de6fb7c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 --2- 172.21.15.27:0/3996082900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f0de3242f0 0x55f0de6feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7fee8affd640 0 client.27448 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 -- 172.21.15.27:0/3996082900 >> 172.21.15.27:0/3996082900 conn(0x55f0de341300 msgr2=0x55f0de6f5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 -- 172.21.15.27:0/3996082900 shutdown_connections 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 1 -- 172.21.15.27:0/3996082900 wait complete. 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.709+0000 7feeb8765740 20 client.27448 trim_cache size 0 max 16384 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:52:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:52:23.711 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 -- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee940afc60 msgr2=0x7fee940b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee940afc60 0x7fee940b2170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x55f0de6eed50 tx=0x7feea40e4110 comp rx=0 tx=0).stop 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 -- 172.21.15.27:0/2884327592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac109fc0 msgr2=0x7feeac1a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac109fc0 0x7feeac1a4d00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7feea400d770 tx=0x7feea400dc30 comp rx=0 tx=0).stop 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 -- 172.21.15.27:0/2884327592 shutdown_connections 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee940afc60 0x7fee940b2170 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feeac1120c0 0x7feeac1ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feeac10f760 0x7feeac1a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 --2- 172.21.15.27:0/2884327592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feeac109fc0 0x7feeac1a4d00 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 -- 172.21.15.27:0/2884327592 >> 172.21.15.27:0/2884327592 conn(0x7feeac070540 msgr2=0x7feeac1106c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 -- 172.21.15.27:0/2884327592 shutdown_connections 2024-04-25T13:52:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:23.710+0000 7feeb18af640 1 -- 172.21.15.27:0/2884327592 wait complete. 2024-04-25T13:52:24.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:24 smithi122 ceph-mon[32843]: pgmap v694: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 MiB/s wr, 667 op/s 2024-04-25T13:52:24.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:24 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:24.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:24 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:52:24.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:24 smithi045 ceph-mon[32731]: pgmap v694: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 MiB/s wr, 667 op/s 2024-04-25T13:52:24.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:24 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:24.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:24 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:52:24.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:24 smithi027 ceph-mon[29140]: pgmap v694: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6.6 MiB/s wr, 667 op/s 2024-04-25T13:52:24.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:24 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:24.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:24 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:52:26.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:26 smithi122 ceph-mon[32843]: pgmap v695: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.3 MiB/s wr, 642 op/s 2024-04-25T13:52:26.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:26 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:52:26.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:26 smithi045 ceph-mon[32731]: pgmap v695: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.3 MiB/s wr, 642 op/s 2024-04-25T13:52:26.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:26 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:52:26.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:26 smithi027 ceph-mon[29140]: pgmap v695: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.3 MiB/s wr, 642 op/s 2024-04-25T13:52:26.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:26 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:52:27.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:27 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:27 smithi027 ceph-mon[29140]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.3 MiB/s wr, 729 op/s 2024-04-25T13:52:27.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:27 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:27 smithi122 ceph-mon[32843]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.3 MiB/s wr, 729 op/s 2024-04-25T13:52:27.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:27 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:27.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:27 smithi045 ceph-mon[32731]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.3 MiB/s wr, 729 op/s 2024-04-25T13:52:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 680.0 s, 1163.8 tps, lat 84.989 ms stddev 91.850 2024-04-25T13:52:28.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:52:28.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:52:28.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:28.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:52:28.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:52:28.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:28.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:28 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:52:28.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:28 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:52:28.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:28 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:52:29.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:29 smithi027 ceph-mon[29140]: pgmap v697: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.0 MiB/s wr, 674 op/s 2024-04-25T13:52:29.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:29 smithi122 ceph-mon[32843]: pgmap v697: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.0 MiB/s wr, 674 op/s 2024-04-25T13:52:29.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:29 smithi045 ceph-mon[32731]: pgmap v697: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.0 MiB/s wr, 674 op/s 2024-04-25T13:52:32.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:31 smithi122 ceph-mon[32843]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 6.5 MiB/s wr, 737 op/s 2024-04-25T13:52:32.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:31 smithi045 ceph-mon[32731]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 6.5 MiB/s wr, 737 op/s 2024-04-25T13:52:32.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:31 smithi027 ceph-mon[29140]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 6.5 MiB/s wr, 737 op/s 2024-04-25T13:52:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 685.0 s, 1960.0 tps, lat 51.396 ms stddev 57.250 2024-04-25T13:52:34.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:33 smithi122 ceph-mon[32843]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.7 MiB/s wr, 797 op/s 2024-04-25T13:52:34.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:33 smithi045 ceph-mon[32731]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.7 MiB/s wr, 797 op/s 2024-04-25T13:52:34.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:33 smithi027 ceph-mon[29140]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.7 MiB/s wr, 797 op/s 2024-04-25T13:52:36.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:35 smithi122 ceph-mon[32843]: pgmap v700: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-04-25T13:52:36.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:35 smithi045 ceph-mon[32731]: pgmap v700: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-04-25T13:52:36.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:35 smithi027 ceph-mon[29140]: pgmap v700: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-04-25T13:52:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.848+0000 7f51ed4ee640 1 -- 172.21.15.27:0/3303756197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e81050e0 msgr2=0x7f51e81054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.848+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/3303756197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e81050e0 0x7f51e81054f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009900 tx=0x7f51d802a410 comp rx=0 tx=0).stop 2024-04-25T13:52:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.848+0000 7f51ed4ee640 1 -- 172.21.15.27:0/3303756197 shutdown_connections 2024-04-25T13:52:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.848+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/3303756197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e8102290 0x7f51e810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.848+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/3303756197 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51e8101790 0x7f51e8101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.848+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/3303756197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e81050e0 0x7f51e81054f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.849+0000 7f51ed4ee640 1 -- 172.21.15.27:0/3303756197 >> 172.21.15.27:0/3303756197 conn(0x7f51e80fd390 msgr2=0x7f51e80ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.849+0000 7f51ed4ee640 1 -- 172.21.15.27:0/3303756197 shutdown_connections 2024-04-25T13:52:36.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.849+0000 7f51ed4ee640 1 -- 172.21.15.27:0/3303756197 wait complete. 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.850+0000 7f51ed4ee640 1 Processor -- start 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.850+0000 7f51ed4ee640 1 -- start start 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.850+0000 7f51ed4ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e8101790 0x7f51e81a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.850+0000 7f51ed4ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e8102290 0x7f51e81a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51ed4ee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51e81050e0 0x7f51e81a7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51ed4ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51e8103fe0 con 0x7f51e8102290 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51ed4ee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f51e8103e60 con 0x7f51e8101790 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51ed4ee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f51e8104160 con 0x7f51e81050e0 2024-04-25T13:52:36.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e8102290 0x7f51e81a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e8101790 0x7f51e81a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e8102290 0x7f51e81a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52068/0 (socket says 172.21.15.27:52068) 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e8101790 0x7f51e81a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52366/0 (socket says 172.21.15.27:52366) 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 -- 172.21.15.27:0/2099677000 learned_addr learned my addr 172.21.15.27:0/2099677000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e77fe640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51e81050e0 0x7f51e81a7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 -- 172.21.15.27:0/2099677000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51e81050e0 msgr2=0x7f51e81a7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51e81050e0 0x7f51e81a7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 -- 172.21.15.27:0/2099677000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e8101790 msgr2=0x7f51e81a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e8101790 0x7f51e81a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 -- 172.21.15.27:0/2099677000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e81a8440 con 0x7f51e8102290 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e6ffd640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e8101790 0x7f51e81a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:36.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.851+0000 7f51e67fc640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e8102290 0x7f51e81a0e40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00b4a0 tx=0x7f51dc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:36.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.852+0000 7f51cffff640 1 -- 172.21.15.27:0/2099677000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc015020 con 0x7f51e8102290 2024-04-25T13:52:36.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.852+0000 7f51cffff640 1 -- 172.21.15.27:0/2099677000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51dc011ce0 con 0x7f51e8102290 2024-04-25T13:52:36.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.852+0000 7f51ed4ee640 1 -- 172.21.15.27:0/2099677000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e81a8730 con 0x7f51e8102290 2024-04-25T13:52:36.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.852+0000 7f51cffff640 1 -- 172.21.15.27:0/2099677000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc010dc0 con 0x7f51e8102290 2024-04-25T13:52:36.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.852+0000 7f51ed4ee640 1 -- 172.21.15.27:0/2099677000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e81a8d00 con 0x7f51e8102290 2024-04-25T13:52:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51f43a0740 1 Processor -- start 2024-04-25T13:52:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51f43a0740 1 -- start start 2024-04-25T13:52:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627951b0 0x55f862b35970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b35ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.857 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627744d0 0x55f862b3cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51f43a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f862a9b460 con 0x55f8627744d0 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51f43a0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f862a55590 con 0x55f86276c2f0 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51f43a0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f862a97420 con 0x55f8627951b0 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51cffff640 1 -- 172.21.15.27:0/2099677000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f51dc018020 con 0x7f51e8102290 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.855+0000 7f51e77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627951b0 0x55f862b35970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b35ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627951b0 0x55f862b35970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50344/0 (socket says 172.21.15.27:50344) 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e77fe640 1 -- 172.21.15.27:0/3838180811 learned_addr learned my addr 172.21.15.27:0/3838180811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e67fc640 1 --2- 172.21.15.27:0/3838180811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627744d0 0x55f862b3cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e77fe640 1 -- 172.21.15.27:0/3838180811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 msgr2=0x55f862b35ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e77fe640 1 --2- 172.21.15.27:0/3838180811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b35ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.858 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e77fe640 1 -- 172.21.15.27:0/3838180811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627744d0 msgr2=0x55f862b3cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e77fe640 1 --2- 172.21.15.27:0/3838180811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627744d0 0x55f862b3cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e77fe640 1 -- 172.21.15.27:0/3838180811 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f862a620d0 con 0x55f8627951b0 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51e77fe640 1 --2- 172.21.15.27:0/3838180811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627951b0 0x55f862b35970 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f51e0011440 tx=0x7f51e0011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a05466a188ed00d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51cd7fa640 1 -- 172.21.15.27:0/3838180811 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0003c30 con 0x55f8627951b0 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51cd7fa640 1 -- 172.21.15.27:0/3838180811 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e0003dd0 con 0x55f8627951b0 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51cd7fa640 1 -- 172.21.15.27:0/3838180811 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e000fa20 con 0x55f8627951b0 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.856+0000 7f51cffff640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51d00afae0 0x7f51d00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51e6ffd640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51d00afae0 0x7f51d00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51cffff640 1 -- 172.21.15.27:0/2099677000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f51dc0de400 con 0x7f51e8102290 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 -- 172.21.15.27:0/3838180811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627951b0 msgr2=0x55f862b35970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 --2- 172.21.15.27:0/3838180811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627951b0 0x55f862b35970 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f51e0011440 tx=0x7f51e0011900 comp rx=0 tx=0).stop 2024-04-25T13:52:36.859 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51e6ffd640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51d00afae0 0x7f51d00b1ff0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f51d8005ab0 tx=0x7f51d8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 -- 172.21.15.27:0/3838180811 shutdown_connections 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 --2- 172.21.15.27:0/3838180811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627744d0 0x55f862b3cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 --2- 172.21.15.27:0/3838180811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b35ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 --2- 172.21.15.27:0/3838180811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627951b0 0x55f862b35970 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 -- 172.21.15.27:0/3838180811 >> 172.21.15.27:0/3838180811 conn(0x55f862789300 msgr2=0x55f862b31300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 -- 172.21.15.27:0/3838180811 shutdown_connections 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.857+0000 7f51f43a0740 1 -- 172.21.15.27:0/3838180811 wait complete. 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 1 -- start start 2024-04-25T13:52:36.860 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 1 Processor -- start 2024-04-25T13:52:36.864 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b3f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862b3fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627951b0 0x55f862b46ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f862b3d840 con 0x55f8627951b0 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f862a22b60 con 0x55f86276c2f0 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f862b3d9c0 con 0x55f8627744d0 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51f43a0740 10 client.0 _ll_register_callbacks cb 0x55f862a472b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51e77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627951b0 0x55f862b46ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b3f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862b3fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b3f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52382/0 (socket says 172.21.15.27:52382) 2024-04-25T13:52:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51e6ffd640 1 -- 172.21.15.27:0/1632032655 learned_addr learned my addr 172.21.15.27:0/1632032655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.858+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862b3fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50352/0 (socket says 172.21.15.27:50352) 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.859+0000 7f51e77fe640 1 -- 172.21.15.27:0/1632032655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 msgr2=0x55f862b3fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.859+0000 7f51e77fe640 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862b3fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.859+0000 7f51e77fe640 1 -- 172.21.15.27:0/1632032655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 msgr2=0x55f862b3f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.859+0000 7f51e77fe640 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b3f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.859+0000 7f51e77fe640 1 -- 172.21.15.27:0/1632032655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8627ad210 con 0x55f8627951b0 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.860+0000 7f51e77fe640 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627951b0 0x55f862b46ba0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f51e001b040 tx=0x7f51e0004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6552cb8ea84f468 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.860+0000 7f51caff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.860+0000 7f51caff5640 1 -- 172.21.15.27:0/1632032655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0029020 con 0x55f8627951b0 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.860+0000 7f51caff5640 1 -- 172.21.15.27:0/1632032655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f51e000fb70 con 0x55f8627951b0 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.861+0000 7f51f43a0740 1 -- 172.21.15.27:0/1632032655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e0003dd0 con 0x55f8627951b0 2024-04-25T13:52:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.861+0000 7f51caff5640 1 -- 172.21.15.27:0/1632032655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f51e000fd70 con 0x55f8627951b0 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.861+0000 7f51f43a0740 10 client.17715 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.861+0000 7f51f43a0740 1 -- 172.21.15.27:0/1632032655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f862879620 con 0x55f8627951b0 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.861+0000 7f51caff5640 1 -- 172.21.15.27:0/1632032655 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e00044b0 con 0x55f8627951b0 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51caff5640 1 -- 172.21.15.27:0/1632032655 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f51e00053e0 con 0x55f8627951b0 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51f43a0740 10 client.17715 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51f43a0740 10 client.17715 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51f43a0740 20 client.17715 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51f43a0740 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f862753d80 0x55f862b4d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51f43a0740 4 client.17715 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51f43a0740 1 -- 172.21.15.27:0/1632032655 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f51e0003dd0 con 0x55f862753d80 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51e6ffd640 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f862753d80 0x55f862b4d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51caff5640 10 client.17715 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.862+0000 7f51e6ffd640 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f862753d80 0x55f862b4d720 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac1cf1f7903255c7 server_cookie=bce750df79428e10 in_seq=0 out_seq=0 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.863+0000 7f51caff5640 1 -- 172.21.15.27:0/1632032655 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f51e0003dd0 con 0x55f862753d80 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.863+0000 7f51caff5640 10 client.17715 handle_command_reply: tid=0 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 client.17715 shutdown 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 -- 172.21.15.27:0/1632032655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627951b0 msgr2=0x55f862b46ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627951b0 0x55f862b46ba0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f51e001b040 tx=0x7f51e0004cf0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 -- 172.21.15.27:0/1632032655 shutdown_connections 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f862753d80 0x55f862b4d720 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51caff5640 0 client.17715 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8627951b0 0x55f862b46ba0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862b3fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 --2- 172.21.15.27:0/1632032655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862b3f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 -- 172.21.15.27:0/1632032655 >> 172.21.15.27:0/1632032655 conn(0x55f862789300 msgr2=0x55f862b34210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 -- 172.21.15.27:0/1632032655 shutdown_connections 2024-04-25T13:52:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 1 -- 172.21.15.27:0/1632032655 wait complete. 2024-04-25T13:52:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.864+0000 7f51f43a0740 20 client.17715 trim_cache size 0 max 16384 2024-04-25T13:52:36.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51f43a0740 1 Processor -- start 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51f43a0740 1 -- start start 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f862753d80 0x55f862b34810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86276c2f0 0x55f862b46ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8627744d0 0x55f862b3bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51f43a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51e81af9e0 con 0x55f86276c2f0 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51f43a0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f51e81b2260 con 0x55f8627744d0 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51f43a0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f51e81b0050 con 0x55f862753d80 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86276c2f0 0x55f862b46ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f862753d80 0x55f862b34810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86276c2f0 0x55f862b46ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52082/0 (socket says 172.21.15.27:52082) 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f862753d80 0x55f862b34810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50356/0 (socket says 172.21.15.27:50356) 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e67fc640 1 -- 172.21.15.27:0/625393025 learned_addr learned my addr 172.21.15.27:0/625393025 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e77fe640 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8627744d0 0x55f862b3bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e67fc640 1 -- 172.21.15.27:0/625393025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f862753d80 msgr2=0x55f862b34810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e67fc640 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f862753d80 0x55f862b34810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e67fc640 1 -- 172.21.15.27:0/625393025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8627744d0 msgr2=0x55f862b3bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e67fc640 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8627744d0 0x55f862b3bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e67fc640 1 -- 172.21.15.27:0/625393025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f862b47000 con 0x55f86276c2f0 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.885+0000 7f51e6ffd640 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f862753d80 0x55f862b34810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.886+0000 7f51e67fc640 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86276c2f0 0x55f862b46ac0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0a2990 tx=0x7f51dc0a21d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd9ff587ab17aac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.886+0000 7f51ccff9640 1 -- 172.21.15.27:0/625393025 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0a6420 con 0x55f86276c2f0 2024-04-25T13:52:36.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.886+0000 7f51ccff9640 1 -- 172.21.15.27:0/625393025 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51dc0a7010 con 0x55f86276c2f0 2024-04-25T13:52:36.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.886+0000 7f51ccff9640 1 -- 172.21.15.27:0/625393025 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0de0f0 con 0x55f86276c2f0 2024-04-25T13:52:36.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.887+0000 7f51f43a0740 1 -- 172.21.15.27:0/625393025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86276c2f0 msgr2=0x55f862b46ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.887+0000 7f51f43a0740 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86276c2f0 0x55f862b46ac0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0a2990 tx=0x7f51dc0a21d0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.888+0000 7f51f43a0740 1 -- 172.21.15.27:0/625393025 shutdown_connections 2024-04-25T13:52:36.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.888+0000 7f51f43a0740 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8627744d0 0x55f862b3bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.888+0000 7f51f43a0740 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f86276c2f0 0x55f862b46ac0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.888+0000 7f51f43a0740 1 --2- 172.21.15.27:0/625393025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f862753d80 0x55f862b34810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.888+0000 7f51f43a0740 1 -- 172.21.15.27:0/625393025 >> 172.21.15.27:0/625393025 conn(0x55f862789300 msgr2=0x55f862b34100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.888+0000 7f51f43a0740 1 -- 172.21.15.27:0/625393025 shutdown_connections 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.888+0000 7f51f43a0740 1 -- 172.21.15.27:0/625393025 wait complete. 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 1 -- start start 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 1 Processor -- start 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f862753d80 0x55f862bac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862bacee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862bb3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f51e81b0720 con 0x55f862753d80 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f51e81b1b90 con 0x55f86276c2f0 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f51e81b0df0 con 0x55f8627744d0 2024-04-25T13:52:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51e6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f862753d80 0x55f862bac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862bacee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862bacee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52404/0 (socket says 172.21.15.27:52404) 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51e67fc640 1 -- 172.21.15.27:0/2905982550 learned_addr learned my addr 172.21.15.27:0/2905982550 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51f43a0740 10 client.0 _ll_register_callbacks cb 0x7f51d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.889+0000 7f51e77fe640 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862bb3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51e67fc640 1 -- 172.21.15.27:0/2905982550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 msgr2=0x55f862bb3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51e67fc640 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862bb3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51e67fc640 1 -- 172.21.15.27:0/2905982550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f862753d80 msgr2=0x55f862bac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51e67fc640 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f862753d80 0x55f862bac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51e67fc640 1 -- 172.21.15.27:0/2905982550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f862bb4400 con 0x55f86276c2f0 2024-04-25T13:52:36.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51e67fc640 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862bacee0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0a68a0 tx=0x7f51dc0047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e4c63c13922f64b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51caff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51caff5640 1 -- 172.21.15.27:0/2905982550 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0c0060 con 0x55f86276c2f0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51caff5640 1 -- 172.21.15.27:0/2905982550 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f51dc0c90d0 con 0x55f86276c2f0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51f43a0740 1 -- 172.21.15.27:0/2905982550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51dc0a7010 con 0x55f86276c2f0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51caff5640 1 -- 172.21.15.27:0/2905982550 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f51dc0aacc0 con 0x55f86276c2f0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51f43a0740 10 client.27422 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51f43a0740 1 -- 172.21.15.27:0/2905982550 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f862bb5690 con 0x55f86276c2f0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51caff5640 1 -- 172.21.15.27:0/2905982550 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51dc0a7010 con 0x55f86276c2f0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.890+0000 7f51caff5640 1 -- 172.21.15.27:0/2905982550 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f51dc0af2e0 con 0x55f86276c2f0 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51f43a0740 10 client.27422 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51f43a0740 10 client.27422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51f43a0740 20 client.27422 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51f43a0740 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8627951b0 0x55f862bb7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51f43a0740 4 client.27422 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:52:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51f43a0740 1 -- 172.21.15.27:0/2905982550 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f862aa9070 con 0x55f8627951b0 2024-04-25T13:52:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51e6ffd640 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8627951b0 0x55f862bb7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51caff5640 10 client.27422 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.891+0000 7f51e6ffd640 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8627951b0 0x55f862bb7f30 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffa572608f5e8b68 server_cookie=fbc979486150fcdd in_seq=0 out_seq=0 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.906+0000 7f51caff5640 1 -- 172.21.15.27:0/2905982550 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f862aa9070 con 0x55f8627951b0 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51caff5640 10 client.27422 handle_command_reply: tid=0 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 client.27422 shutdown 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 -- 172.21.15.27:0/2905982550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 msgr2=0x55f862bacee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862bacee0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0a68a0 tx=0x7f51dc0047d0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 -- 172.21.15.27:0/2905982550 shutdown_connections 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8627951b0 0x55f862bb7f30 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8627744d0 0x55f862bb3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f86276c2f0 0x55f862bacee0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51caff5640 0 client.27422 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 --2- 172.21.15.27:0/2905982550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f862753d80 0x55f862bac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 -- 172.21.15.27:0/2905982550 >> 172.21.15.27:0/2905982550 conn(0x55f862789300 msgr2=0x55f862b34890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 -- 172.21.15.27:0/2905982550 shutdown_connections 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 1 -- 172.21.15.27:0/2905982550 wait complete. 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.907+0000 7f51f43a0740 20 client.27422 trim_cache size 0 max 16384 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:52:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:52:36.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:52:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 -- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51d00afae0 msgr2=0x7f51d00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51d00afae0 0x7f51d00b1ff0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f51d8005ab0 tx=0x7f51d8031400 comp rx=0 tx=0).stop 2024-04-25T13:52:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 -- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e8102290 msgr2=0x7f51e81a0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e8102290 0x7f51e81a0e40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00b4a0 tx=0x7f51dc00b8a0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 -- 172.21.15.27:0/2099677000 shutdown_connections 2024-04-25T13:52:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f51d00afae0 0x7f51d00b1ff0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f51e81050e0 0x7f51e81a7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51e8102290 0x7f51e81a0e40 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 --2- 172.21.15.27:0/2099677000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51e8101790 0x7f51e81a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 -- 172.21.15.27:0/2099677000 >> 172.21.15.27:0/2099677000 conn(0x7f51e80fd390 msgr2=0x7f51e8105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 -- 172.21.15.27:0/2099677000 shutdown_connections 2024-04-25T13:52:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:36.908+0000 7f51ed4ee640 1 -- 172.21.15.27:0/2099677000 wait complete. 2024-04-25T13:52:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 690.0 s, 1562.0 tps, lat 64.001 ms stddev 70.745 2024-04-25T13:52:38.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:38 smithi122 ceph-mon[32843]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 7.1 MiB/s wr, 754 op/s 2024-04-25T13:52:38.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:38 smithi045 ceph-mon[32731]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 7.1 MiB/s wr, 754 op/s 2024-04-25T13:52:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:38 smithi027 ceph-mon[29140]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 7.1 MiB/s wr, 754 op/s 2024-04-25T13:52:40.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:40 smithi122 ceph-mon[32843]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.1 MiB/s wr, 595 op/s 2024-04-25T13:52:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:40 smithi045 ceph-mon[32731]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.1 MiB/s wr, 595 op/s 2024-04-25T13:52:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:40 smithi027 ceph-mon[29140]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.1 MiB/s wr, 595 op/s 2024-04-25T13:52:42.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:42 smithi045 ceph-mon[32731]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 MiB/s wr, 644 op/s 2024-04-25T13:52:42.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:42 smithi027 ceph-mon[29140]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 MiB/s wr, 644 op/s 2024-04-25T13:52:42.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:42 smithi122 ceph-mon[32843]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 MiB/s wr, 644 op/s 2024-04-25T13:52:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 695.0 s, 1476.4 tps, lat 67.983 ms stddev 70.591 2024-04-25T13:52:44.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:44 smithi045 ceph-mon[32731]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.7 MiB/s wr, 658 op/s 2024-04-25T13:52:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:44 smithi027 ceph-mon[29140]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.7 MiB/s wr, 658 op/s 2024-04-25T13:52:44.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:44 smithi122 ceph-mon[32843]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.7 MiB/s wr, 658 op/s 2024-04-25T13:52:46.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:46 smithi045 ceph-mon[32731]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.9 MiB/s wr, 541 op/s 2024-04-25T13:52:46.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:46 smithi122 ceph-mon[32843]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.9 MiB/s wr, 541 op/s 2024-04-25T13:52:46.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:46 smithi027 ceph-mon[29140]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.9 MiB/s wr, 541 op/s 2024-04-25T13:52:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 700.0 s, 1556.0 tps, lat 64.214 ms stddev 73.216 2024-04-25T13:52:48.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:48 smithi027 ceph-mon[29140]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 662 op/s 2024-04-25T13:52:48.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:48 smithi122 ceph-mon[32843]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 662 op/s 2024-04-25T13:52:48.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:48 smithi045 ceph-mon[32731]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.1 MiB/s wr, 662 op/s 2024-04-25T13:52:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 -- 172.21.15.27:0/557681684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 msgr2=0x7fe310103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 --2- 172.21.15.27:0/557681684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 0x7fe310103820 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe300009860 tx=0x7fe30002a340 comp rx=0 tx=0).stop 2024-04-25T13:52:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 -- 172.21.15.27:0/557681684 shutdown_connections 2024-04-25T13:52:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 --2- 172.21.15.27:0/557681684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe310108110 0x7fe31010a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 --2- 172.21.15.27:0/557681684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 0x7fe310103820 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 --2- 172.21.15.27:0/557681684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3101021d0 0x7fe310102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 -- 172.21.15.27:0/557681684 >> 172.21.15.27:0/557681684 conn(0x7fe3100fd3c0 msgr2=0x7fe3100ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 -- 172.21.15.27:0/557681684 shutdown_connections 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.048+0000 7fe318651640 1 -- 172.21.15.27:0/557681684 wait complete. 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe318651640 1 Processor -- start 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe318651640 1 -- start start 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe318651640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3101021d0 0x7fe31019a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe318651640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 0x7fe31019a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe318651640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe310108110 0x7fe3101a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe318651640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe31010d460 con 0x7fe310108110 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe318651640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe31010d2e0 con 0x7fe3101021d0 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe318651640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe31010d5e0 con 0x7fe310103410 2024-04-25T13:52:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3101021d0 0x7fe31019a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe315bec640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 0x7fe31019a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe315bec640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 0x7fe31019a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50826/0 (socket says 172.21.15.27:50826) 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3101021d0 0x7fe31019a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:46052/0 (socket says 172.21.15.27:46052) 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe315bec640 1 -- 172.21.15.27:0/2168696638 learned_addr learned my addr 172.21.15.27:0/2168696638 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.049+0000 7fe316bee640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe310108110 0x7fe3101a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe315bec640 1 -- 172.21.15.27:0/2168696638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3101021d0 msgr2=0x7fe31019a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe315bec640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3101021d0 0x7fe31019a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe315bec640 1 -- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe310108110 msgr2=0x7fe3101a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe315bec640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe310108110 0x7fe3101a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe315bec640 1 -- 172.21.15.27:0/2168696638 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3101a1da0 con 0x7fe310103410 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe3163ed640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3101021d0 0x7fe31019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe315bec640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 0x7fe31019a810 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe300009980 tx=0x7fe300032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe3077fe640 1 -- 172.21.15.27:0/2168696638 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300040020 con 0x7fe310103410 2024-04-25T13:52:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe318651640 1 -- 172.21.15.27:0/2168696638 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3101a2030 con 0x7fe310103410 2024-04-25T13:52:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe3077fe640 1 -- 172.21.15.27:0/2168696638 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe300030dd0 con 0x7fe310103410 2024-04-25T13:52:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe3077fe640 1 -- 172.21.15.27:0/2168696638 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300037040 con 0x7fe310103410 2024-04-25T13:52:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.050+0000 7fe318651640 1 -- 172.21.15.27:0/2168696638 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3101a2570 con 0x7fe310103410 2024-04-25T13:52:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe31f5e1740 1 Processor -- start 2024-04-25T13:52:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe3077fe640 1 -- 172.21.15.27:0/2168696638 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe300043020 con 0x7fe310103410 2024-04-25T13:52:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe31f5e1740 1 -- start start 2024-04-25T13:52:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a80d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a812d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226bc4d0 0x55b422a7da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b4229e3460 con 0x55b4226bc4d0 2024-04-25T13:52:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b42299d590 con 0x55b4226dd1b0 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b4229df420 con 0x55b4226b42f0 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe316bee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a80d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe315bec640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226bc4d0 0x55b422a7da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a812d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a812d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50840/0 (socket says 172.21.15.27:50840) 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.054+0000 7fe3163ed640 1 -- 172.21.15.27:0/4034683830 learned_addr learned my addr 172.21.15.27:0/4034683830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe3163ed640 1 -- 172.21.15.27:0/4034683830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 msgr2=0x55b422a80d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe3163ed640 1 --2- 172.21.15.27:0/4034683830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a80d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe3077fe640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe2fc0afb60 0x7fe2fc0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe3163ed640 1 -- 172.21.15.27:0/4034683830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226bc4d0 msgr2=0x55b422a7da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe3163ed640 1 --2- 172.21.15.27:0/4034683830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226bc4d0 0x55b422a7da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe3163ed640 1 -- 172.21.15.27:0/4034683830 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4229aa0d0 con 0x55b4226b42f0 2024-04-25T13:52:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe316bee640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe2fc0afb60 0x7fe2fc0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe3077fe640 1 -- 172.21.15.27:0/2168696638 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe3000fe970 con 0x7fe310103410 2024-04-25T13:52:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe3163ed640 1 --2- 172.21.15.27:0/4034683830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a812d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00e550 tx=0x7fe30c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd1ca08e7072b793 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe304bfe640 1 -- 172.21.15.27:0/4034683830 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c01c070 con 0x55b4226b42f0 2024-04-25T13:52:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe304bfe640 1 -- 172.21.15.27:0/4034683830 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe30c004230 con 0x55b4226b42f0 2024-04-25T13:52:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.055+0000 7fe304bfe640 1 -- 172.21.15.27:0/4034683830 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0038e0 con 0x55b4226b42f0 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- 172.21.15.27:0/4034683830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 msgr2=0x55b422a812d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe316bee640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe2fc0afb60 0x7fe2fc0b2070 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55b422a7eac0 tx=0x7fe308010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/4034683830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a812d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00e550 tx=0x7fe30c005f90 comp rx=0 tx=0).stop 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- 172.21.15.27:0/4034683830 shutdown_connections 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/4034683830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226bc4d0 0x55b422a7da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/4034683830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a812d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/4034683830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a80d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- 172.21.15.27:0/4034683830 >> 172.21.15.27:0/4034683830 conn(0x55b4226d1300 msgr2=0x55b422a79300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- 172.21.15.27:0/4034683830 shutdown_connections 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- 172.21.15.27:0/4034683830 wait complete. 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- start start 2024-04-25T13:52:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 Processor -- start 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226b42f0 0x55b422a87510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226bc4d0 0x55b422a87a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a8c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55b42296ab60 con 0x55b4226b42f0 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55b422a819d0 con 0x55b4226dd1b0 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55b422a8cee0 con 0x55b4226bc4d0 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.057+0000 7fe31f5e1740 10 client.0 _ll_register_callbacks cb 0x55b42298f2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226b42f0 0x55b422a87510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226b42f0 0x55b422a87510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54404/0 (socket says 172.21.15.27:54404) 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 -- 172.21.15.27:0/2791340872 learned_addr learned my addr 172.21.15.27:0/2791340872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe316bee640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a8c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe315bec640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226bc4d0 0x55b422a87a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 -- 172.21.15.27:0/2791340872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226bc4d0 msgr2=0x55b422a87a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226bc4d0 0x55b422a87a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 -- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 msgr2=0x55b422a8c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a8c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 -- 172.21.15.27:0/2791340872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4226f5210 con 0x55b4226b42f0 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe316bee640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a8c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe315bec640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226bc4d0 0x55b422a87a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe3163ed640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226b42f0 0x55b422a87510 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00a750 tx=0x7fe30c00d6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403c42f567b0024e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe2f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/2791340872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c01c040 con 0x55b4226b42f0 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/2791340872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe30c00da90 con 0x55b4226b42f0 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe31f5e1740 1 -- 172.21.15.27:0/2791340872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe30c004230 con 0x55b4226b42f0 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.058+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/2791340872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe30c00dc90 con 0x55b4226b42f0 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe31f5e1740 10 client.17733 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe31f5e1740 1 -- 172.21.15.27:0/2791340872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b4227c1620 con 0x55b4226b42f0 2024-04-25T13:52:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/2791340872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe30c004230 con 0x55b4226b42f0 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/2791340872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe30c022040 con 0x55b4226b42f0 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe31f5e1740 10 client.17733 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe31f5e1740 10 client.17733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe31f5e1740 20 client.17733 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b42269bd80 0x55b422aa60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe31f5e1740 4 client.17733 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe31f5e1740 1 -- 172.21.15.27:0/2791340872 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4229f1070 con 0x55b42269bd80 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.059+0000 7fe315bec640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b42269bd80 0x55b422aa60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.060+0000 7fe315bec640 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b42269bd80 0x55b422aa60a0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c957a98f0f1e8f5a server_cookie=404aa424c7936898 in_seq=0 out_seq=0 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.060+0000 7fe2f9ffb640 10 client.17733 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/2791340872 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b4229f1070 con 0x55b42269bd80 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe2f9ffb640 10 client.17733 handle_command_reply: tid=0 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 client.17733 shutdown 2024-04-25T13:52:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 -- 172.21.15.27:0/2791340872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226b42f0 msgr2=0x55b422a87510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226b42f0 0x55b422a87510 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00a750 tx=0x7fe30c00d6a0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 -- 172.21.15.27:0/2791340872 shutdown_connections 2024-04-25T13:52:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b42269bd80 0x55b422aa60a0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226dd1b0 0x55b422a8c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226bc4d0 0x55b422a87a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/2791340872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226b42f0 0x55b422a87510 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe2f9ffb640 0 client.17733 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.062+0000 7fe31f5e1740 1 -- 172.21.15.27:0/2791340872 >> 172.21.15.27:0/2791340872 conn(0x55b4226d1300 msgr2=0x55b422a78250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.063+0000 7fe31f5e1740 1 -- 172.21.15.27:0/2791340872 shutdown_connections 2024-04-25T13:52:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.063+0000 7fe31f5e1740 1 -- 172.21.15.27:0/2791340872 wait complete. 2024-04-25T13:52:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.063+0000 7fe31f5e1740 20 client.17733 trim_cache size 0 max 16384 2024-04-25T13:52:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.082+0000 7fe31f5e1740 1 Processor -- start 2024-04-25T13:52:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.082+0000 7fe31f5e1740 1 -- start start 2024-04-25T13:52:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a78ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422a791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422a870e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101a6f70 con 0x55b4226dd1b0 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101a97f0 con 0x55b4226bc4d0 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101a75e0 con 0x55b4226b42f0 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe316bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422a870e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a78ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe315bec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422a791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a78ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50850/0 (socket says 172.21.15.27:50850) 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe316bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422a870e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54408/0 (socket says 172.21.15.27:54408) 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe3163ed640 1 -- 172.21.15.27:0/1357352389 learned_addr learned my addr 172.21.15.27:0/1357352389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe316bee640 1 -- 172.21.15.27:0/1357352389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 msgr2=0x55b422a78ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe316bee640 1 --2- 172.21.15.27:0/1357352389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a78ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe316bee640 1 -- 172.21.15.27:0/1357352389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 msgr2=0x55b422a791e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe316bee640 1 --2- 172.21.15.27:0/1357352389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422a791e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe316bee640 1 -- 172.21.15.27:0/1357352389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b42299d7e0 con 0x55b4226dd1b0 2024-04-25T13:52:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.083+0000 7fe316bee640 1 --2- 172.21.15.27:0/1357352389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422a870e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe308009f50 tx=0x7fe308017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bacb0b8fe59ed46b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.085+0000 7fe2fbfff640 1 -- 172.21.15.27:0/1357352389 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30800da70 con 0x55b4226dd1b0 2024-04-25T13:52:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.085+0000 7fe2fbfff640 1 -- 172.21.15.27:0/1357352389 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe30802c070 con 0x55b4226dd1b0 2024-04-25T13:52:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.085+0000 7fe2fbfff640 1 -- 172.21.15.27:0/1357352389 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308027680 con 0x55b4226dd1b0 2024-04-25T13:52:50.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1357352389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 msgr2=0x55b422a870e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1357352389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422a870e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fe308009f50 tx=0x7fe308017000 comp rx=0 tx=0).stop 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1357352389 shutdown_connections 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1357352389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422a870e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1357352389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422a791e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1357352389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422a78ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1357352389 >> 172.21.15.27:0/1357352389 conn(0x55b4226d1300 msgr2=0x55b422a85b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1357352389 shutdown_connections 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.086+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1357352389 wait complete. 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 1 -- start start 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 1 Processor -- start 2024-04-25T13:52:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422ab4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422ab4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422ab5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101a7cb0 con 0x55b4226dd1b0 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101a9120 con 0x55b4226bc4d0 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101a8380 con 0x55b4226b42f0 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe31f5e1740 10 client.0 _ll_register_callbacks cb 0x7fe30c01a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422ab4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe316bee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422ab5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe3163ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422ab4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50866/0 (socket says 172.21.15.27:50866) 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe3163ed640 1 -- 172.21.15.27:0/1479425425 learned_addr learned my addr 172.21.15.27:0/1479425425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.087+0000 7fe315bec640 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422ab4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe3163ed640 1 -- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 msgr2=0x55b422ab4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe3163ed640 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422ab4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe3163ed640 1 -- 172.21.15.27:0/1479425425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 msgr2=0x55b422ab5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe3163ed640 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422ab5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe3163ed640 1 -- 172.21.15.27:0/1479425425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b422ab6350 con 0x55b4226b42f0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe316bee640 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422ab5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe315bec640 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422ab4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe3163ed640 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422ab4920 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0059f0 tx=0x7fe30c004190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d62866bb944310dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe2f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/1479425425 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c01c040 con 0x55b4226b42f0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1479425425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe30802c070 con 0x55b4226b42f0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/1479425425 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe30c005040 con 0x55b4226b42f0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/1479425425 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fe30c00de60 con 0x55b4226b42f0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe31f5e1740 10 client.27481 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.088+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1479425425 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55b422ab75e0 con 0x55b4226b42f0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/1479425425 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe30802c070 con 0x55b4226b42f0 2024-04-25T13:52:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/1479425425 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fe30c022070 con 0x55b4226b42f0 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe31f5e1740 10 client.27481 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe31f5e1740 10 client.27481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe31f5e1740 20 client.27481 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4226711c0 0x55b422b10300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe31f5e1740 4 client.27481 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1479425425 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b4229eaee0 con 0x55b4226711c0 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.089+0000 7fe315bec640 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4226711c0 0x55b422b10300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.090+0000 7fe2f9ffb640 10 client.27481 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.090+0000 7fe315bec640 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4226711c0 0x55b422b10300 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=254fa6fa8b7b6c6 server_cookie=5da8bf44eca8ff9d in_seq=0 out_seq=0 2024-04-25T13:52:50.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.105+0000 7fe2f9ffb640 1 -- 172.21.15.27:0/1479425425 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b4229eaee0 con 0x55b4226711c0 2024-04-25T13:52:50.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe2f9ffb640 10 client.27481 handle_command_reply: tid=0 2024-04-25T13:52:50.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 client.27481 shutdown 2024-04-25T13:52:50.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1479425425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 msgr2=0x55b422ab4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422ab4920 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0059f0 tx=0x7fe30c004190 comp rx=0 tx=0).stop 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1479425425 shutdown_connections 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55b4226711c0 0x55b422b10300 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b4226dd1b0 0x55b422ab5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe2f9ffb640 0 client.27481 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4226bc4d0 0x55b422ab4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 --2- 172.21.15.27:0/1479425425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b4226b42f0 0x55b422ab4920 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1479425425 >> 172.21.15.27:0/1479425425 conn(0x55b4226d1300 msgr2=0x55b422a78ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1479425425 shutdown_connections 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 1 -- 172.21.15.27:0/1479425425 wait complete. 2024-04-25T13:52:50.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.106+0000 7fe31f5e1740 20 client.27481 trim_cache size 0 max 16384 2024-04-25T13:52:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:52:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:52:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:52:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:52:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 -- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe2fc0afb60 msgr2=0x7fe2fc0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe2fc0afb60 0x7fe2fc0b2070 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55b422a7eac0 tx=0x7fe308010040 comp rx=0 tx=0).stop 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 -- 172.21.15.27:0/2168696638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 msgr2=0x7fe31019a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 0x7fe31019a810 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe300009980 tx=0x7fe300032dc0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 -- 172.21.15.27:0/2168696638 shutdown_connections 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe2fc0afb60 0x7fe2fc0b2070 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe310108110 0x7fe3101a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe310103410 0x7fe31019a810 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 --2- 172.21.15.27:0/2168696638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3101021d0 0x7fe31019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.107+0000 7fe318651640 1 -- 172.21.15.27:0/2168696638 >> 172.21.15.27:0/2168696638 conn(0x7fe3100fd3c0 msgr2=0x7fe3101040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.108+0000 7fe318651640 1 -- 172.21.15.27:0/2168696638 shutdown_connections 2024-04-25T13:52:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:52:50.108+0000 7fe318651640 1 -- 172.21.15.27:0/2168696638 wait complete. 2024-04-25T13:52:50.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:50 smithi027 ceph-mon[29140]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.9 MiB/s wr, 536 op/s 2024-04-25T13:52:50.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:50 smithi122 ceph-mon[32843]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.9 MiB/s wr, 536 op/s 2024-04-25T13:52:50.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:50 smithi045 ceph-mon[32731]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.9 MiB/s wr, 536 op/s 2024-04-25T13:52:52.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:52 smithi027 ceph-mon[29140]: pgmap v708: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-25T13:52:52.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:52 smithi122 ceph-mon[32843]: pgmap v708: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-25T13:52:52.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:52 smithi045 ceph-mon[32731]: pgmap v708: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-25T13:52:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 705.0 s, 2323.2 tps, lat 43.248 ms stddev 51.172 2024-04-25T13:52:53.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:52:53.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:52:53.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.316+0000 7f2b87dc5640 1 -- 172.21.15.27:0/4267245486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 msgr2=0x7f2b80104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.316+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/4267245486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 0x7f2b80104800 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009900 tx=0x7f2b7002a410 comp rx=0 tx=0).stop 2024-04-25T13:52:53.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.316+0000 7f2b87dc5640 1 -- 172.21.15.27:0/4267245486 shutdown_connections 2024-04-25T13:52:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.316+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/4267245486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b80109a40 0x7f2b8010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.316+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/4267245486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b80104d40 0x7f2b80105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.316+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/4267245486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 0x7f2b80104800 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.316+0000 7f2b87dc5640 1 -- 172.21.15.27:0/4267245486 >> 172.21.15.27:0/4267245486 conn(0x7f2b800fed10 msgr2=0x7f2b801011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:53.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.317+0000 7f2b87dc5640 1 -- 172.21.15.27:0/4267245486 shutdown_connections 2024-04-25T13:52:53.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.318+0000 7f2b87dc5640 1 -- 172.21.15.27:0/4267245486 wait complete. 2024-04-25T13:52:53.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b87dc5640 1 Processor -- start 2024-04-25T13:52:53.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b87dc5640 1 -- start start 2024-04-25T13:52:53.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b87dc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 0x7f2b80198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b87dc5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b80104d40 0x7f2b801985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b87dc5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b80109a40 0x7f2b8019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b87dc5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010f600 con 0x7f2b80103b00 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b87dc5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010f480 con 0x7f2b80104d40 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b87dc5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010f780 con 0x7f2b80109a40 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b85b61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 0x7f2b80198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b85360640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b80104d40 0x7f2b801985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b85b61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 0x7f2b80198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54430/0 (socket says 172.21.15.27:54430) 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b85360640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b80104d40 0x7f2b801985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:46098/0 (socket says 172.21.15.27:46098) 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b85b61640 1 -- 172.21.15.27:0/525373276 learned_addr learned my addr 172.21.15.27:0/525373276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.319+0000 7f2b86362640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b80109a40 0x7f2b8019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b86362640 1 -- 172.21.15.27:0/525373276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b80104d40 msgr2=0x7f2b801985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b86362640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b80104d40 0x7f2b801985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b86362640 1 -- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 msgr2=0x7f2b80198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b86362640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 0x7f2b80198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b86362640 1 -- 172.21.15.27:0/525373276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8019fbd0 con 0x7f2b80109a40 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b85360640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b80104d40 0x7f2b801985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b86362640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b80109a40 0x7f2b8019f560 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009fd0 tx=0x7f2b7800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b7effd640 1 -- 172.21.15.27:0/525373276 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7800ce20 con 0x7f2b80109a40 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b87dc5640 1 -- 172.21.15.27:0/525373276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8019fec0 con 0x7f2b80109a40 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b7effd640 1 -- 172.21.15.27:0/525373276 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b78005020 con 0x7f2b80109a40 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b7effd640 1 -- 172.21.15.27:0/525373276 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78010690 con 0x7f2b80109a40 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.320+0000 7f2b87dc5640 1 -- 172.21.15.27:0/525373276 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b801a0410 con 0x7f2b80109a40 2024-04-25T13:52:53.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.323+0000 7f2b7effd640 1 -- 172.21.15.27:0/525373276 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2b78010830 con 0x7f2b80109a40 2024-04-25T13:52:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.323+0000 7f2b8ec79740 1 Processor -- start 2024-04-25T13:52:53.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b8ec79740 1 -- start start 2024-04-25T13:52:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a385da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a385dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a385ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5649a37c2d80 con 0x5649a349b6b0 2024-04-25T13:52:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5649a377c8b0 con 0x5649a34934d0 2024-04-25T13:52:53.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5649a37bed40 con 0x5649a34bc390 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b7effd640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b680afaf0 0x7f2b680b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a385da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85b61640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a385dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a385da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50896/0 (socket says 172.21.15.27:50896) 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 -- 172.21.15.27:0/3845844440 learned_addr learned my addr 172.21.15.27:0/3845844440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 --2- 172.21.15.27:0/3845844440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a385ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b86362640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b680afaf0 0x7f2b680b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 -- 172.21.15.27:0/3845844440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 msgr2=0x5649a385dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 --2- 172.21.15.27:0/3845844440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a385dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 -- 172.21.15.27:0/3845844440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 msgr2=0x5649a385ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 --2- 172.21.15.27:0/3845844440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a385ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.324+0000 7f2b85360640 1 -- 172.21.15.27:0/3845844440 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649a374da40 con 0x5649a34bc390 2024-04-25T13:52:53.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.325+0000 7f2b7effd640 1 -- 172.21.15.27:0/525373276 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2b780df2c0 con 0x7f2b80109a40 2024-04-25T13:52:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.325+0000 7f2b85360640 1 --2- 172.21.15.27:0/3845844440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a385da60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2b74011850 tx=0x7f2b74009b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=143774fab8e497f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.325+0000 7f2b86362640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b680afaf0 0x7f2b680b2000 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009000 tx=0x7f2b780ca040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.325+0000 7f2b67fff640 1 -- 172.21.15.27:0/3845844440 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74004030 con 0x5649a34bc390 2024-04-25T13:52:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.325+0000 7f2b67fff640 1 -- 172.21.15.27:0/3845844440 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7400f8c0 con 0x5649a34bc390 2024-04-25T13:52:53.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.325+0000 7f2b67fff640 1 -- 172.21.15.27:0/3845844440 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74003bf0 con 0x5649a34bc390 2024-04-25T13:52:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.327+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3845844440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 msgr2=0x5649a385da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.327+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3845844440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a385da60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2b74011850 tx=0x7f2b74009b70 comp rx=0 tx=0).stop 2024-04-25T13:52:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.328+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3845844440 shutdown_connections 2024-04-25T13:52:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.328+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3845844440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a385ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.328+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3845844440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a385dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.328+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3845844440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a385da60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.328+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3845844440 >> 172.21.15.27:0/3845844440 conn(0x5649a34b04e0 msgr2=0x5649a3859270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:53.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.328+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3845844440 shutdown_connections 2024-04-25T13:52:53.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.328+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3845844440 wait complete. 2024-04-25T13:52:53.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.329+0000 7f2b8ec79740 1 -- start start 2024-04-25T13:52:53.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.329+0000 7f2b8ec79740 1 Processor -- start 2024-04-25T13:52:53.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.329+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34934d0 0x5649a3867780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3867cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34bc390 0x5649a386ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5649a385f780 con 0x5649a34934d0 2024-04-25T13:52:53.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5649a3749d10 con 0x5649a34bc390 2024-04-25T13:52:53.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85360640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34934d0 0x5649a3867780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5649a385f900 con 0x5649a349b6b0 2024-04-25T13:52:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85360640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34934d0 0x5649a3867780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54442/0 (socket says 172.21.15.27:54442) 2024-04-25T13:52:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85360640 1 -- 172.21.15.27:0/4003118475 learned_addr learned my addr 172.21.15.27:0/4003118475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85b61640 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3867cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b8ec79740 10 client.0 _ll_register_callbacks cb 0x5649a3776210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85360640 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34bc390 0x5649a386ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85b61640 1 -- 172.21.15.27:0/4003118475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34bc390 msgr2=0x5649a386ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:53.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85b61640 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34bc390 0x5649a386ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85b61640 1 -- 172.21.15.27:0/4003118475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34934d0 msgr2=0x5649a3867780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85b61640 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34934d0 0x5649a3867780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.330+0000 7f2b85b61640 1 -- 172.21.15.27:0/4003118475 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649a359e7d0 con 0x5649a349b6b0 2024-04-25T13:52:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.331+0000 7f2b85b61640 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3867cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2b70000c00 tx=0x7f2b70032e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9693d1762aec95c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.331+0000 7f2b657fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:52:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.331+0000 7f2b657fa640 1 -- 172.21.15.27:0/4003118475 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70035040 con 0x5649a349b6b0 2024-04-25T13:52:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.331+0000 7f2b657fa640 1 -- 172.21.15.27:0/4003118475 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2b70031860 con 0x5649a349b6b0 2024-04-25T13:52:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.331+0000 7f2b8ec79740 1 -- 172.21.15.27:0/4003118475 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b7400f8c0 con 0x5649a349b6b0 2024-04-25T13:52:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.331+0000 7f2b657fa640 1 -- 172.21.15.27:0/4003118475 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2b70004040 con 0x5649a349b6b0 2024-04-25T13:52:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.331+0000 7f2b8ec79740 10 client.27493 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.331+0000 7f2b8ec79740 1 -- 172.21.15.27:0/4003118475 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5649a34d43f0 con 0x5649a349b6b0 2024-04-25T13:52:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.332+0000 7f2b657fa640 1 -- 172.21.15.27:0/4003118475 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7400f8c0 con 0x5649a349b6b0 2024-04-25T13:52:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.332+0000 7f2b657fa640 1 -- 172.21.15.27:0/4003118475 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2b700047f0 con 0x5649a349b6b0 2024-04-25T13:52:53.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.332+0000 7f2b8ec79740 10 client.27493 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.332+0000 7f2b8ec79740 10 client.27493 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.332+0000 7f2b8ec79740 20 client.27493 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.332+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5649a347af60 0x5649a38750f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.332+0000 7f2b8ec79740 4 client.27493 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.332+0000 7f2b8ec79740 1 -- 172.21.15.27:0/4003118475 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649a37d0990 con 0x5649a347af60 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.336+0000 7f2b85360640 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5649a347af60 0x5649a38750f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.339+0000 7f2b85360640 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5649a347af60 0x5649a38750f0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce73f5a4043b51aa server_cookie=2713cc5e6d4e1a33 in_seq=0 out_seq=0 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.340+0000 7f2b657fa640 10 client.27493 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.343+0000 7f2b657fa640 1 -- 172.21.15.27:0/4003118475 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5649a37d0990 con 0x5649a347af60 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.343+0000 7f2b657fa640 10 client.27493 handle_command_reply: tid=0 2024-04-25T13:52:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.344+0000 7f2b8ec79740 1 client.27493 shutdown 2024-04-25T13:52:53.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.351+0000 7f2b8ec79740 1 -- 172.21.15.27:0/4003118475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 msgr2=0x5649a3867cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.351+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3867cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2b70000c00 tx=0x7f2b70032e70 comp rx=0 tx=0).stop 2024-04-25T13:52:53.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.354+0000 7f2b8ec79740 1 -- 172.21.15.27:0/4003118475 shutdown_connections 2024-04-25T13:52:53.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.354+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5649a347af60 0x5649a38750f0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.354+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34bc390 0x5649a386ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.354+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3867cc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.354+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/4003118475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34934d0 0x5649a3867780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.354+0000 7f2b8ec79740 1 -- 172.21.15.27:0/4003118475 >> 172.21.15.27:0/4003118475 conn(0x5649a34b04e0 msgr2=0x5649a385c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:53.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.354+0000 7f2b657fa640 0 client.27493 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:53.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.356+0000 7f2b8ec79740 1 -- 172.21.15.27:0/4003118475 shutdown_connections 2024-04-25T13:52:53.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.356+0000 7f2b8ec79740 1 -- 172.21.15.27:0/4003118475 wait complete. 2024-04-25T13:52:53.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.359+0000 7f2b8ec79740 20 client.27493 trim_cache size 0 max 16384 2024-04-25T13:52:53.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.399+0000 7f2b8ec79740 1 Processor -- start 2024-04-25T13:52:53.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.399+0000 7f2b8ec79740 1 -- start start 2024-04-25T13:52:53.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.399+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a3858c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.399+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3859150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.399+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34bc390 0x5649a386eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.399+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2b801b7f40 con 0x5649a34bc390 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.399+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2b801ba7c0 con 0x5649a34934d0 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.399+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2b801b85b0 con 0x5649a349b6b0 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85360640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a3858c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3859150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3859150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50920/0 (socket says 172.21.15.27:50920) 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 -- 172.21.15.27:0/91101608 learned_addr learned my addr 172.21.15.27:0/91101608 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 -- 172.21.15.27:0/91101608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 msgr2=0x5649a3858c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85360640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a3858c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:46132/0 (socket says 172.21.15.27:46132) 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 --2- 172.21.15.27:0/91101608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a3858c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 -- 172.21.15.27:0/91101608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34bc390 msgr2=0x5649a386eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 --2- 172.21.15.27:0/91101608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34bc390 0x5649a386eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 -- 172.21.15.27:0/91101608 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649a3867530 con 0x5649a349b6b0 2024-04-25T13:52:53.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85360640 1 --2- 172.21.15.27:0/91101608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a3858c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:52:53.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.400+0000 7f2b85b61640 1 --2- 172.21.15.27:0/91101608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3859150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009850 tx=0x7f2b7002a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5b1f16873fbff74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:53.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.401+0000 7f2b677fe640 1 -- 172.21.15.27:0/91101608 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70035030 con 0x5649a349b6b0 2024-04-25T13:52:53.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.401+0000 7f2b677fe640 1 -- 172.21.15.27:0/91101608 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b70003cb0 con 0x5649a349b6b0 2024-04-25T13:52:53.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.401+0000 7f2b677fe640 1 -- 172.21.15.27:0/91101608 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70031950 con 0x5649a349b6b0 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 -- 172.21.15.27:0/91101608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 msgr2=0x5649a3859150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/91101608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3859150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009850 tx=0x7f2b7002a410 comp rx=0 tx=0).stop 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 -- 172.21.15.27:0/91101608 shutdown_connections 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/91101608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a34bc390 0x5649a386eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/91101608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a349b6b0 0x5649a3859150 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/91101608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a3858c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 -- 172.21.15.27:0/91101608 >> 172.21.15.27:0/91101608 conn(0x5649a34b04e0 msgr2=0x5649a38624d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 -- 172.21.15.27:0/91101608 shutdown_connections 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 -- 172.21.15.27:0/91101608 wait complete. 2024-04-25T13:52:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 -- start start 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.402+0000 7f2b8ec79740 1 Processor -- start 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a386ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a386f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b8ec79740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a386b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2b801b8c80 con 0x5649a349b6b0 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2b801ba0f0 con 0x5649a34934d0 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b85360640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a386ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b85b61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a386f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b85360640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a386ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:46134/0 (socket says 172.21.15.27:46134) 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b85360640 1 -- 172.21.15.27:0/3629878737 learned_addr learned my addr 172.21.15.27:0/3629878737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b8ec79740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2b801b9350 con 0x5649a34bc390 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b85b61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a386f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54454/0 (socket says 172.21.15.27:54454) 2024-04-25T13:52:53.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b85360640 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a386b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.403+0000 7f2b8ec79740 10 client.0 _ll_register_callbacks cb 0x7f2b70009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.404+0000 7f2b85b61640 1 -- 172.21.15.27:0/3629878737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 msgr2=0x5649a386b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.404+0000 7f2b85b61640 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a386b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.404+0000 7f2b85b61640 1 -- 172.21.15.27:0/3629878737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 msgr2=0x5649a386ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.404+0000 7f2b85b61640 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a386ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.404+0000 7f2b85b61640 1 -- 172.21.15.27:0/3629878737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649a34940f0 con 0x5649a349b6b0 2024-04-25T13:52:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.404+0000 7f2b85360640 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a386ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.404+0000 7f2b85b61640 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a386f290 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2b70004d30 tx=0x7f2b70031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7b83b0ba18040f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.405+0000 7f2b657fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:52:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.405+0000 7f2b657fa640 1 -- 172.21.15.27:0/3629878737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70035070 con 0x5649a349b6b0 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.405+0000 7f2b657fa640 1 -- 172.21.15.27:0/3629878737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2b70003a10 con 0x5649a349b6b0 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.406+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3629878737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b70003cb0 con 0x5649a349b6b0 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.406+0000 7f2b657fa640 1 -- 172.21.15.27:0/3629878737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2b70003e60 con 0x5649a349b6b0 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.407+0000 7f2b8ec79740 10 client.17754 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.407+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3629878737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5649a37834b0 con 0x5649a349b6b0 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b657fa640 1 -- 172.21.15.27:0/3629878737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b70031b80 con 0x5649a349b6b0 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b657fa640 1 -- 172.21.15.27:0/3629878737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2b70060020 con 0x5649a349b6b0 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b8ec79740 10 client.17754 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b8ec79740 10 client.17754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b8ec79740 20 client.17754 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5649a34503a0 0x5649a38de580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b8ec79740 4 client.17754 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3629878737 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f2b70003cb0 con 0x5649a34503a0 2024-04-25T13:52:53.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b85360640 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5649a34503a0 0x5649a38de580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.408+0000 7f2b657fa640 10 client.17754 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b85360640 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5649a34503a0 0x5649a38de580 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73690a6d1cb48922 server_cookie=b2e3d348cafc32ca in_seq=0 out_seq=0 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b657fa640 1 -- 172.21.15.27:0/3629878737 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f2b70003cb0 con 0x5649a34503a0 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b657fa640 10 client.17754 handle_command_reply: tid=0 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 client.17754 shutdown 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3629878737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 msgr2=0x5649a386f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a386f290 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2b70004d30 tx=0x7f2b70031400 comp rx=0 tx=0).stop 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3629878737 shutdown_connections 2024-04-25T13:52:53.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5649a34503a0 0x5649a38de580 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5649a34bc390 0x5649a386b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b657fa640 0 client.17754 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5649a349b6b0 0x5649a386f290 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 --2- 172.21.15.27:0/3629878737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649a34934d0 0x5649a386ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3629878737 >> 172.21.15.27:0/3629878737 conn(0x5649a34b04e0 msgr2=0x5649a3858c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3629878737 shutdown_connections 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.409+0000 7f2b8ec79740 1 -- 172.21.15.27:0/3629878737 wait complete. 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b8ec79740 20 client.17754 trim_cache size 0 max 16384 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:52:53.411 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 -- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b680afaf0 msgr2=0x7f2b680b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b680afaf0 0x7f2b680b2000 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009000 tx=0x7f2b780ca040 comp rx=0 tx=0).stop 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 -- 172.21.15.27:0/525373276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b80109a40 msgr2=0x7f2b8019f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b80109a40 0x7f2b8019f560 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009fd0 tx=0x7f2b7800ef10 comp rx=0 tx=0).stop 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 -- 172.21.15.27:0/525373276 shutdown_connections 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2b680afaf0 0x7f2b680b2000 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b80109a40 0x7f2b8019f560 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b80104d40 0x7f2b801985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 --2- 172.21.15.27:0/525373276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b80103b00 0x7f2b80198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.410+0000 7f2b87dc5640 1 -- 172.21.15.27:0/525373276 >> 172.21.15.27:0/525373276 conn(0x7f2b800fed10 msgr2=0x7f2b80101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.411+0000 7f2b87dc5640 1 -- 172.21.15.27:0/525373276 shutdown_connections 2024-04-25T13:52:53.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.411+0000 7f2b87dc5640 1 -- 172.21.15.27:0/525373276 wait complete. 2024-04-25T13:52:53.425 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:52:53.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:52:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.594+0000 7f7144ecc640 1 -- 172.21.15.27:0/3486062630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010fe90 msgr2=0x7f71401102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.594+0000 7f7144ecc640 1 --2- 172.21.15.27:0/3486062630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010fe90 0x7f71401102a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7138009ca0 tx=0x7f713802a8f0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.599+0000 7f7144ecc640 1 -- 172.21.15.27:0/3486062630 shutdown_connections 2024-04-25T13:52:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.599+0000 7f7144ecc640 1 --2- 172.21.15.27:0/3486062630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010fe90 0x7f71401102a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.599+0000 7f7144ecc640 1 --2- 172.21.15.27:0/3486062630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f714010b190 0x7f714010b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.599+0000 7f7144ecc640 1 --2- 172.21.15.27:0/3486062630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f714010a340 0x7f714010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.599+0000 7f7144ecc640 1 -- 172.21.15.27:0/3486062630 >> 172.21.15.27:0/3486062630 conn(0x7f7140070320 msgr2=0x7f71400727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.604+0000 7f7144ecc640 1 -- 172.21.15.27:0/3486062630 shutdown_connections 2024-04-25T13:52:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.607+0000 7f7144ecc640 1 -- 172.21.15.27:0/3486062630 wait complete. 2024-04-25T13:52:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.608+0000 7f7144ecc640 1 Processor -- start 2024-04-25T13:52:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.610+0000 7f7144ecc640 1 -- start start 2024-04-25T13:52:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.611+0000 7f7144ecc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f714010a340 0x7f7140119e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.611+0000 7f7144ecc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010b190 0x7f714011a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.611+0000 7f7144ecc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f714010fe90 0x7f714011b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.611+0000 7f7144ecc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7140112030 con 0x7f714010fe90 2024-04-25T13:52:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.611+0000 7f7144ecc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7140111eb0 con 0x7f714010b190 2024-04-25T13:52:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.611+0000 7f7144ecc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f71401121b0 con 0x7f714010a340 2024-04-25T13:52:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.616+0000 7f713effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010b190 0x7f714011a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.616+0000 7f713effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010b190 0x7f714011a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:46144/0 (socket says 172.21.15.27:46144) 2024-04-25T13:52:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.616+0000 7f713effd640 1 -- 172.21.15.27:0/499503231 learned_addr learned my addr 172.21.15.27:0/499503231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.617+0000 7f713f7fe640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f714010a340 0x7f7140119e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.617+0000 7f713effd640 1 -- 172.21.15.27:0/499503231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f714010a340 msgr2=0x7f7140119e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.617+0000 7f713effd640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f714010a340 0x7f7140119e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.617+0000 7f713effd640 1 -- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f714010fe90 msgr2=0x7f714011b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.617+0000 7f713effd640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f714010fe90 0x7f714011b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.617+0000 7f713effd640 1 -- 172.21.15.27:0/499503231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714011b920 con 0x7f714010b190 2024-04-25T13:52:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.618+0000 7f713effd640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010b190 0x7f714011a350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f713000e9f0 tx=0x7f713000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.619+0000 7f713cff9640 1 -- 172.21.15.27:0/499503231 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713000cd80 con 0x7f714010b190 2024-04-25T13:52:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.619+0000 7f713cff9640 1 -- 172.21.15.27:0/499503231 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71300108e0 con 0x7f714010b190 2024-04-25T13:52:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.619+0000 7f713cff9640 1 -- 172.21.15.27:0/499503231 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7130018860 con 0x7f714010b190 2024-04-25T13:52:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.620+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f714011bc10 con 0x7f714010b190 2024-04-25T13:52:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.620+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f714011c0e0 con 0x7f714010b190 2024-04-25T13:52:53.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.625+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7140113970 con 0x7f714010b190 2024-04-25T13:52:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.629+0000 7f713cff9640 1 -- 172.21.15.27:0/499503231 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7130010430 con 0x7f714010b190 2024-04-25T13:52:53.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.630+0000 7f713cff9640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71280afcf0 0x7f71280b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:53.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.631+0000 7f713cff9640 1 -- 172.21.15.27:0/499503231 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f71300df200 con 0x7f714010b190 2024-04-25T13:52:53.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.636+0000 7f713f7fe640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71280afcf0 0x7f71280b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:53.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.637+0000 7f713f7fe640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71280afcf0 0x7f71280b2200 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7134006fd0 tx=0x7f7134008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.639+0000 7f713cff9640 1 -- 172.21.15.27:0/499503231 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f71300a7db0 con 0x7f714010b190 2024-04-25T13:52:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:53.999+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71401b9c20 con 0x7f714010b190 2024-04-25T13:52:54.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.002+0000 7f713cff9640 1 -- 172.21.15.27:0/499503231 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f71300a76b0 con 0x7f714010b190 2024-04-25T13:52:54.002 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:52:54.002 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:52:54.002 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:52:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.005+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71280afcf0 msgr2=0x7f71280b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.005+0000 7f7144ecc640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71280afcf0 0x7f71280b2200 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7134006fd0 tx=0x7f7134008040 comp rx=0 tx=0).stop 2024-04-25T13:52:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.005+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010b190 msgr2=0x7f714011a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.005+0000 7f7144ecc640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010b190 0x7f714011a350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f713000e9f0 tx=0x7f713000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.006+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 shutdown_connections 2024-04-25T13:52:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.006+0000 7f7144ecc640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f71280afcf0 0x7f71280b2200 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.006+0000 7f7144ecc640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f714010fe90 0x7f714011b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.006+0000 7f7144ecc640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f714010b190 0x7f714011a350 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.006+0000 7f7144ecc640 1 --2- 172.21.15.27:0/499503231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f714010a340 0x7f7140119e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.006+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 >> 172.21.15.27:0/499503231 conn(0x7f7140070320 msgr2=0x7f714010de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.010+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 shutdown_connections 2024-04-25T13:52:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.010+0000 7f7144ecc640 1 -- 172.21.15.27:0/499503231 wait complete. 2024-04-25T13:52:54.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:52:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 -- 172.21.15.27:0/67513916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 msgr2=0x7f65200abb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 --2- 172.21.15.27:0/67513916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 0x7f65200abb50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f651801a910 tx=0x7f651803b430 comp rx=0 tx=0).stop 2024-04-25T13:52:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 -- 172.21.15.27:0/67513916 shutdown_connections 2024-04-25T13:52:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 --2- 172.21.15.27:0/67513916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 0x7f65200abb50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 --2- 172.21.15.27:0/67513916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65200a6a20 0x7f65200a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 --2- 172.21.15.27:0/67513916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65200a5920 0x7f65200a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 -- 172.21.15.27:0/67513916 >> 172.21.15.27:0/67513916 conn(0x7f65200a0990 msgr2=0x7f65200a2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 -- 172.21.15.27:0/67513916 shutdown_connections 2024-04-25T13:52:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.209+0000 7f652d391640 1 -- 172.21.15.27:0/67513916 wait complete. 2024-04-25T13:52:54.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.210+0000 7f652d391640 1 Processor -- start 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.210+0000 7f652d391640 1 -- start start 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f652d391640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65200a5920 0x7f652013beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f652d391640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65200a6a20 0x7f652013c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f652d391640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 0x7f6520143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f652d391640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65200af370 con 0x7f65200a6a20 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f652d391640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65200af1f0 con 0x7f65200ab740 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f652d391640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f65200af4f0 con 0x7f65200a5920 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f65277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65200a6a20 0x7f652013c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f65277fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65200a6a20 0x7f652013c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34208/0 (socket says 172.21.15.27:34208) 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f65277fe640 1 -- 172.21.15.27:0/3068666569 learned_addr learned my addr 172.21.15.27:0/3068666569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.211+0000 7f652cb90640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 0x7f6520143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.212+0000 7f65277fe640 1 -- 172.21.15.27:0/3068666569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65200a5920 msgr2=0x7f652013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.212+0000 7f65277fe640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65200a5920 0x7f652013beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.212+0000 7f65277fe640 1 -- 172.21.15.27:0/3068666569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 msgr2=0x7f6520143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.212+0000 7f65277fe640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 0x7f6520143380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.212+0000 7f65277fe640 1 -- 172.21.15.27:0/3068666569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6520143940 con 0x7f65200a6a20 2024-04-25T13:52:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.212+0000 7f652cb90640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 0x7f6520143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.212+0000 7f65277fe640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65200a6a20 0x7f652013c3f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6528050a10 tx=0x7f652806faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.212+0000 7f65257fa640 1 -- 172.21.15.27:0/3068666569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652807e070 con 0x7f65200a6a20 2024-04-25T13:52:54.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.213+0000 7f65257fa640 1 -- 172.21.15.27:0/3068666569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f652806ccb0 con 0x7f65200a6a20 2024-04-25T13:52:54.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.213+0000 7f65257fa640 1 -- 172.21.15.27:0/3068666569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528079e10 con 0x7f65200a6a20 2024-04-25T13:52:54.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.213+0000 7f652d391640 1 -- 172.21.15.27:0/3068666569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6520143c30 con 0x7f65200a6a20 2024-04-25T13:52:54.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.213+0000 7f652d391640 1 -- 172.21.15.27:0/3068666569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6520144130 con 0x7f65200a6a20 2024-04-25T13:52:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.216+0000 7f653557d740 1 Processor -- start 2024-04-25T13:52:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.217+0000 7f653557d740 1 -- start start 2024-04-25T13:52:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.218+0000 7f653557d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31928f390 0x55e319631a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.218+0000 7f653557d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e319631f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.218+0000 7f653557d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e319632c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.218+0000 7f653557d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e319595d80 con 0x55e3192664d0 2024-04-25T13:52:54.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.218+0000 7f653557d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e31954fa20 con 0x55e31926e6b0 2024-04-25T13:52:54.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.218+0000 7f653557d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e319591d40 con 0x55e31928f390 2024-04-25T13:52:54.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.218+0000 7f652cb90640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31928f390 0x55e319631a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f652cb90640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31928f390 0x55e319631a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50860/0 (socket says 172.21.15.27:50860) 2024-04-25T13:52:54.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f652cb90640 1 -- 172.21.15.27:0/880473540 learned_addr learned my addr 172.21.15.27:0/880473540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:54.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f6527fff640 1 --2- 172.21.15.27:0/880473540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e319631f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f652cb90640 1 -- 172.21.15.27:0/880473540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 msgr2=0x55e319632c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:52:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f652cb90640 1 --2- 172.21.15.27:0/880473540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e319632c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f652cb90640 1 -- 172.21.15.27:0/880473540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 msgr2=0x55e319631f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f652cb90640 1 --2- 172.21.15.27:0/880473540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e319631f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f652cb90640 1 -- 172.21.15.27:0/880473540 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e31955ac30 con 0x55e31928f390 2024-04-25T13:52:54.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.219+0000 7f652cb90640 1 --2- 172.21.15.27:0/880473540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31928f390 0x55e319631a00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6518044000 tx=0x7f6518004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=540a61649ad7f15a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:54.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.220+0000 7f650e7fc640 1 -- 172.21.15.27:0/880473540 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518046070 con 0x55e31928f390 2024-04-25T13:52:54.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.220+0000 7f650e7fc640 1 -- 172.21.15.27:0/880473540 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6518004a20 con 0x55e31928f390 2024-04-25T13:52:54.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.220+0000 7f65257fa640 1 -- 172.21.15.27:0/3068666569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6528081070 con 0x7f65200a6a20 2024-04-25T13:52:54.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.221+0000 7f650e7fc640 1 -- 172.21.15.27:0/880473540 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65180054e0 con 0x55e31928f390 2024-04-25T13:52:54.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.221+0000 7f653557d740 1 -- 172.21.15.27:0/880473540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31928f390 msgr2=0x55e319631a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.221+0000 7f653557d740 1 --2- 172.21.15.27:0/880473540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31928f390 0x55e319631a00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6518044000 tx=0x7f6518004770 comp rx=0 tx=0).stop 2024-04-25T13:52:54.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.221+0000 7f65257fa640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65100afc30 0x7f65100b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f65257fa640 1 -- 172.21.15.27:0/3068666569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6528142cc0 con 0x7f65200a6a20 2024-04-25T13:52:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f653557d740 1 -- 172.21.15.27:0/880473540 shutdown_connections 2024-04-25T13:52:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f653557d740 1 --2- 172.21.15.27:0/880473540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e319632c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f653557d740 1 --2- 172.21.15.27:0/880473540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e319631f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f653557d740 1 --2- 172.21.15.27:0/880473540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31928f390 0x55e319631a00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f653557d740 1 -- 172.21.15.27:0/880473540 >> 172.21.15.27:0/880473540 conn(0x55e3192834e0 msgr2=0x55e31962d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f652cb90640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65100afc30 0x7f65100b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f653557d740 1 -- 172.21.15.27:0/880473540 shutdown_connections 2024-04-25T13:52:54.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.222+0000 7f653557d740 1 -- 172.21.15.27:0/880473540 wait complete. 2024-04-25T13:52:54.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.223+0000 7f653557d740 1 -- start start 2024-04-25T13:52:54.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.223+0000 7f652cb90640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65100afc30 0x7f65100b2140 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6518004600 tx=0x7f651803b430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.223+0000 7f653557d740 1 Processor -- start 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f653557d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e31963b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f653557d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31926e6b0 0x55e31963bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f653557d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31928f390 0x55e319642c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f653557d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e319633760 con 0x55e3192664d0 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f653557d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e31951cdc0 con 0x55e31928f390 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f653557d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e3196338e0 con 0x55e31926e6b0 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f653557d740 10 client.0 _ll_register_callbacks cb 0x55e319538600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f6527fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e31963b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f6527fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e31963b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34226/0 (socket says 172.21.15.27:34226) 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f6527fff640 1 -- 172.21.15.27:0/2930651260 learned_addr learned my addr 172.21.15.27:0/2930651260 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f652cb90640 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31928f390 0x55e319642c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f65277fe640 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31926e6b0 0x55e31963bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f6527fff640 1 -- 172.21.15.27:0/2930651260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31926e6b0 msgr2=0x55e31963bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f6527fff640 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31926e6b0 0x55e31963bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f6527fff640 1 -- 172.21.15.27:0/2930651260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31928f390 msgr2=0x55e319642c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f6527fff640 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31928f390 0x55e319642c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.224+0000 7f6527fff640 1 -- 172.21.15.27:0/2930651260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3192a73f0 con 0x55e3192664d0 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.225+0000 7f6527fff640 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e31963b760 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f651c00f670 tx=0x7f651c00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7eca4ae22af5dbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.225+0000 7f65077fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.225+0000 7f65077fe640 1 -- 172.21.15.27:0/2930651260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c00da80 con 0x55e3192664d0 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.225+0000 7f653557d740 1 -- 172.21.15.27:0/2930651260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6518004a20 con 0x55e3192664d0 2024-04-25T13:52:54.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.225+0000 7f65077fe640 1 -- 172.21.15.27:0/2930651260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f651c00dc20 con 0x55e3192664d0 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.225+0000 7f65077fe640 1 -- 172.21.15.27:0/2930651260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f651c00de20 con 0x55e3192664d0 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.225+0000 7f653557d740 10 client.17766 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.225+0000 7f653557d740 1 -- 172.21.15.27:0/2930651260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e3193717d0 con 0x55e3192664d0 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f65077fe640 1 -- 172.21.15.27:0/2930651260 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6518004a20 con 0x55e3192664d0 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f65077fe640 1 -- 172.21.15.27:0/2930651260 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f651c018670 con 0x55e3192664d0 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f653557d740 10 client.17766 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f653557d740 10 client.17766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f653557d740 20 client.17766 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:54.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f653557d740 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e31924df60 0x55e319649970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f653557d740 4 client.17766 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:54.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f653557d740 1 -- 172.21.15.27:0/2930651260 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3195a3990 con 0x55e31924df60 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.226+0000 7f65277fe640 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e31924df60 0x55e319649970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.227+0000 7f65277fe640 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e31924df60 0x55e319649970 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dcd69405a3e2705 server_cookie=684c6a0640596af6 in_seq=0 out_seq=0 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.228+0000 7f65077fe640 10 client.17766 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.228+0000 7f65077fe640 1 -- 172.21.15.27:0/2930651260 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e3195a3990 con 0x55e31924df60 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.228+0000 7f65077fe640 10 client.17766 handle_command_reply: tid=0 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.228+0000 7f653557d740 1 client.17766 shutdown 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 -- 172.21.15.27:0/2930651260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 msgr2=0x55e31963b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e31963b760 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f651c00f670 tx=0x7f651c00af90 comp rx=0 tx=0).stop 2024-04-25T13:52:54.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 -- 172.21.15.27:0/2930651260 shutdown_connections 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e31924df60 0x55e319649970 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31928f390 0x55e319642c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e31926e6b0 0x55e31963bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 --2- 172.21.15.27:0/2930651260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e3192664d0 0x55e31963b760 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 -- 172.21.15.27:0/2930651260 >> 172.21.15.27:0/2930651260 conn(0x55e3192834e0 msgr2=0x55e31962c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f65077fe640 0 client.17766 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 -- 172.21.15.27:0/2930651260 shutdown_connections 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 1 -- 172.21.15.27:0/2930651260 wait complete. 2024-04-25T13:52:54.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.229+0000 7f653557d740 20 client.17766 trim_cache size 0 max 16384 2024-04-25T13:52:54.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.248+0000 7f653557d740 1 Processor -- start 2024-04-25T13:52:54.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.248+0000 7f653557d740 1 -- start start 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f653557d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f653557d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e319642b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f653557d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e31963b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f653557d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f652014acf0 con 0x55e31928f390 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f653557d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f652014d570 con 0x55e31926e6b0 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f653557d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f652014b360 con 0x55e3192664d0 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f6527fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f6527fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50872/0 (socket says 172.21.15.27:50872) 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f6527fff640 1 -- 172.21.15.27:0/2217247528 learned_addr learned my addr 172.21.15.27:0/2217247528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f65277fe640 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e319642b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f652cb90640 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e31963b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f6527fff640 1 -- 172.21.15.27:0/2217247528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 msgr2=0x55e319642b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f6527fff640 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e319642b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f6527fff640 1 -- 172.21.15.27:0/2217247528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 msgr2=0x55e31963b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f6527fff640 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e31963b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.249+0000 7f6527fff640 1 -- 172.21.15.27:0/2217247528 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e319643060 con 0x55e3192664d0 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.250+0000 7f652cb90640 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e31963b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.250+0000 7f6527fff640 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963b330 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f651c000c00 tx=0x7f651c00aeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e1101310488244f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.250+0000 7f650dffb640 1 -- 172.21.15.27:0/2217247528 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c00d880 con 0x55e3192664d0 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.250+0000 7f650dffb640 1 -- 172.21.15.27:0/2217247528 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f651c00da20 con 0x55e3192664d0 2024-04-25T13:52:54.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.250+0000 7f650dffb640 1 -- 172.21.15.27:0/2217247528 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c0047a0 con 0x55e3192664d0 2024-04-25T13:52:54.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 -- 172.21.15.27:0/2217247528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 msgr2=0x55e31963b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963b330 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f651c000c00 tx=0x7f651c00aeb0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 -- 172.21.15.27:0/2217247528 shutdown_connections 2024-04-25T13:52:54.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e31963b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e319642b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 --2- 172.21.15.27:0/2217247528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963b330 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 -- 172.21.15.27:0/2217247528 >> 172.21.15.27:0/2217247528 conn(0x55e3192834e0 msgr2=0x55e319631980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:54.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 -- 172.21.15.27:0/2217247528 shutdown_connections 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 -- 172.21.15.27:0/2217247528 wait complete. 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.251+0000 7f653557d740 1 -- start start 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f653557d740 1 Processor -- start 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f653557d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f653557d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e31963fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f653557d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e3196afd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f6527fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f65277fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e31963fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f653557d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f652014ba30 con 0x55e31928f390 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f653557d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f652014cea0 con 0x55e31926e6b0 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f6527fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50884/0 (socket says 172.21.15.27:50884) 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f65277fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e31963fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52238/0 (socket says 172.21.15.27:52238) 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f65277fe640 1 -- 172.21.15.27:0/530318995 learned_addr learned my addr 172.21.15.27:0/530318995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:54.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f652014c100 con 0x55e3192664d0 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f652cb90640 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e3196afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.252+0000 7f653557d740 10 client.0 _ll_register_callbacks cb 0x7f651c00a550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.253+0000 7f65277fe640 1 -- 172.21.15.27:0/530318995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 msgr2=0x55e31963f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.253+0000 7f65277fe640 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.253+0000 7f65277fe640 1 -- 172.21.15.27:0/530318995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 msgr2=0x55e3196afd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.253+0000 7f65277fe640 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e3196afd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.253+0000 7f65277fe640 1 -- 172.21.15.27:0/530318995 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e319642c20 con 0x55e31926e6b0 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.253+0000 7f65277fe640 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e31963fcb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6528140790 tx=0x7f6528142040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa4578be44e0dcc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:54.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.254+0000 7f65077fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:54.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.254+0000 7f65077fe640 1 -- 172.21.15.27:0/530318995 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652810f520 con 0x55e31926e6b0 2024-04-25T13:52:54.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.254+0000 7f65077fe640 1 -- 172.21.15.27:0/530318995 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6528079530 con 0x55e31926e6b0 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.254+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f651c00da20 con 0x55e31926e6b0 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.255+0000 7f65077fe640 1 -- 172.21.15.27:0/530318995 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6528113fd0 con 0x55e31926e6b0 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.255+0000 7f653557d740 10 client.27476 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.255+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e3196b1120 con 0x55e31926e6b0 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f65077fe640 1 -- 172.21.15.27:0/530318995 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6528124070 con 0x55e31926e6b0 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f65077fe640 1 -- 172.21.15.27:0/530318995 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f652810a460 con 0x55e31926e6b0 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f653557d740 10 client.27476 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f653557d740 10 client.27476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f653557d740 20 client.27476 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f653557d740 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e3192233a0 0x55e3196b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:54.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f653557d740 4 client.27476 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f651c00da20 con 0x55e3192233a0 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.256+0000 7f6527fff640 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e3192233a0 0x55e3196b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f65077fe640 10 client.27476 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f6527fff640 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e3192233a0 0x55e3196b3a70 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78b3e0686fd61bc3 server_cookie=1bcb86ecaee0a5ac in_seq=0 out_seq=0 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f65077fe640 1 -- 172.21.15.27:0/530318995 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f651c00da20 con 0x55e3192233a0 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f65077fe640 10 client.27476 handle_command_reply: tid=0 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 client.27476 shutdown 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 msgr2=0x55e31963fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e31963fcb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6528140790 tx=0x7f6528142040 comp rx=0 tx=0).stop 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 shutdown_connections 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e3192233a0 0x55e3196b3a70 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e31928f390 0x55e3196afd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e31926e6b0 0x55e31963fcb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 --2- 172.21.15.27:0/530318995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e3192664d0 0x55e31963f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f65077fe640 0 client.27476 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 >> 172.21.15.27:0/530318995 conn(0x55e3192834e0 msgr2=0x55e31962b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 shutdown_connections 2024-04-25T13:52:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.257+0000 7f653557d740 1 -- 172.21.15.27:0/530318995 wait complete. 2024-04-25T13:52:54.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f653557d740 20 client.27476 trim_cache size 0 max 16384 2024-04-25T13:52:54.260 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 -- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65100afc30 msgr2=0x7f65100b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65100afc30 0x7f65100b2140 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6518004600 tx=0x7f651803b430 comp rx=0 tx=0).stop 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 -- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65200a6a20 msgr2=0x7f652013c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65200a6a20 0x7f652013c3f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6528050a10 tx=0x7f652806faa0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 -- 172.21.15.27:0/3068666569 shutdown_connections 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65100afc30 0x7f65100b2140 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65200ab740 0x7f6520143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65200a6a20 0x7f652013c3f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 --2- 172.21.15.27:0/3068666569 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65200a5920 0x7f652013beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.258+0000 7f652d391640 1 -- 172.21.15.27:0/3068666569 >> 172.21.15.27:0/3068666569 conn(0x7f65200a0990 msgr2=0x7f65200a0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.259+0000 7f652d391640 1 -- 172.21.15.27:0/3068666569 shutdown_connections 2024-04-25T13:52:54.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:54.259+0000 7f652d391640 1 -- 172.21.15.27:0/3068666569 wait complete. 2024-04-25T13:52:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:54 smithi027 ceph-mon[29140]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 MiB/s wr, 639 op/s 2024-04-25T13:52:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/499503231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:52:54.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:54 smithi122 ceph-mon[32843]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 MiB/s wr, 639 op/s 2024-04-25T13:52:54.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:54 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/499503231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:52:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:54 smithi045 ceph-mon[32731]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 MiB/s wr, 639 op/s 2024-04-25T13:52:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:54 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/499503231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:52:55.274 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:52:55.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:52:55.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.429+0000 7f447dcd1640 1 -- 172.21.15.27:0/1423056617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4478103c30 msgr2=0x7f4478104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.429+0000 7f447dcd1640 1 --2- 172.21.15.27:0/1423056617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4478103c30 0x7f4478104040 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f446c009860 tx=0x7f446c02a340 comp rx=0 tx=0).stop 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.429+0000 7f447dcd1640 1 -- 172.21.15.27:0/1423056617 shutdown_connections 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.429+0000 7f447dcd1640 1 --2- 172.21.15.27:0/1423056617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4478108930 0x7f447810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.429+0000 7f447dcd1640 1 --2- 172.21.15.27:0/1423056617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4478103c30 0x7f4478104040 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.429+0000 7f447dcd1640 1 --2- 172.21.15.27:0/1423056617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44781029f0 0x7f44781036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.430+0000 7f447dcd1640 1 -- 172.21.15.27:0/1423056617 >> 172.21.15.27:0/1423056617 conn(0x7f44780fdc00 msgr2=0x7f4478100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.430+0000 7f447dcd1640 1 -- 172.21.15.27:0/1423056617 shutdown_connections 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.430+0000 7f447dcd1640 1 -- 172.21.15.27:0/1423056617 wait complete. 2024-04-25T13:52:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.430+0000 7f447dcd1640 1 Processor -- start 2024-04-25T13:52:55.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.430+0000 7f447dcd1640 1 -- start start 2024-04-25T13:52:55.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f447dcd1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44781029f0 0x7f447806bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f447dcd1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4478103c30 0x7f447819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f447dcd1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4478108930 0x7f44781a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f447dcd1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f447810e570 con 0x7f4478108930 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f447dcd1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f447810e3f0 con 0x7f44781029f0 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f447dcd1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f447810e6f0 con 0x7f4478103c30 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44781029f0 0x7f447806bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f4477fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4478108930 0x7f44781a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44781029f0 0x7f447806bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52246/0 (socket says 172.21.15.27:52246) 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 -- 172.21.15.27:0/2279459503 learned_addr learned my addr 172.21.15.27:0/2279459503 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f4476ffd640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4478103c30 0x7f447819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 -- 172.21.15.27:0/2279459503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4478103c30 msgr2=0x7f447819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4478103c30 0x7f447819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 -- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4478108930 msgr2=0x7f44781a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4478108930 0x7f44781a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 -- 172.21.15.27:0/2279459503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44781a1d70 con 0x7f44781029f0 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.431+0000 7f44777fe640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44781029f0 0x7f447806bce0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f446800e9c0 tx=0x7f446800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.432+0000 7f4474ff9640 1 -- 172.21.15.27:0/2279459503 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446800ccc0 con 0x7f44781029f0 2024-04-25T13:52:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.432+0000 7f4474ff9640 1 -- 172.21.15.27:0/2279459503 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44680035f0 con 0x7f44781029f0 2024-04-25T13:52:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.432+0000 7f4474ff9640 1 -- 172.21.15.27:0/2279459503 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446800ce60 con 0x7f44781029f0 2024-04-25T13:52:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.434+0000 7f447dcd1640 1 -- 172.21.15.27:0/2279459503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44781a2060 con 0x7f44781029f0 2024-04-25T13:52:55.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.434+0000 7f447dcd1640 1 -- 172.21.15.27:0/2279459503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44781a2530 con 0x7f44781029f0 2024-04-25T13:52:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4484b7f740 1 Processor -- start 2024-04-25T13:52:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4484b7f740 1 -- start start 2024-04-25T13:52:55.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4474ff9640 1 -- 172.21.15.27:0/2279459503 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4468004000 con 0x7f44781029f0 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a86332a410 0x55a8636cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 0x55a8636cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4484b7f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a863630b00 con 0x55a863301550 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4484b7f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55a8635cc920 con 0x55a86332a410 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.437+0000 7f4484b7f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55a86362cac0 con 0x55a863309730 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4477fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a86332a410 0x55a8636cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4477fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a86332a410 0x55a8636cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52258/0 (socket says 172.21.15.27:52258) 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4476ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4477fff640 1 -- 172.21.15.27:0/3592418824 learned_addr learned my addr 172.21.15.27:0/3592418824 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f44777fe640 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 0x55a8636cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4477fff640 1 -- 172.21.15.27:0/3592418824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 msgr2=0x55a8636d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4477fff640 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4477fff640 1 -- 172.21.15.27:0/3592418824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 msgr2=0x55a8636cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4477fff640 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 0x55a8636cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4477fff640 1 -- 172.21.15.27:0/3592418824 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86340a7e0 con 0x55a86332a410 2024-04-25T13:52:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.438+0000 7f4474ff9640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44600afcc0 0x7f44600b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.439+0000 7f4474ff9640 1 -- 172.21.15.27:0/2279459503 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f44680dedf0 con 0x7f44781029f0 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.439+0000 7f4477fff640 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a86332a410 0x55a8636cb4a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f447000a430 tx=0x7f4470009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19732498c7033afc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.439+0000 7f4476ffd640 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.439+0000 7f44777fe640 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 0x55a8636cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.439+0000 7f445dffb640 1 -- 172.21.15.27:0/3592418824 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470013790 con 0x55a86332a410 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.439+0000 7f445dffb640 1 -- 172.21.15.27:0/3592418824 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4470019070 con 0x55a86332a410 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.439+0000 7f445dffb640 1 -- 172.21.15.27:0/3592418824 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470014de0 con 0x55a86332a410 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.439+0000 7f44777fe640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44600afcc0 0x7f44600b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.440+0000 7f4484b7f740 1 -- 172.21.15.27:0/3592418824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a86332a410 msgr2=0x55a8636cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.440+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a86332a410 0x55a8636cb4a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f447000a430 tx=0x7f4470009650 comp rx=0 tx=0).stop 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.440+0000 7f44777fe640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44600afcc0 0x7f44600b21d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f44680a7230 tx=0x7f44680e27b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.440+0000 7f4484b7f740 1 -- 172.21.15.27:0/3592418824 shutdown_connections 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.441+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.441+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 0x55a8636cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.441+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3592418824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a86332a410 0x55a8636cb4a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.441+0000 7f4484b7f740 1 -- 172.21.15.27:0/3592418824 >> 172.21.15.27:0/3592418824 conn(0x55a86331e560 msgr2=0x55a8636c6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.441+0000 7f4484b7f740 1 -- 172.21.15.27:0/3592418824 shutdown_connections 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.441+0000 7f4484b7f740 1 -- 172.21.15.27:0/3592418824 wait complete. 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.441+0000 7f4484b7f740 1 -- start start 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.441+0000 7f4484b7f740 1 Processor -- start 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863301550 0x55a8636d9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636e0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4484b7f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a8636d7780 con 0x55a86332a410 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4484b7f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55a8635ca5d0 con 0x55a863301550 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4484b7f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55a8636d7900 con 0x55a863309730 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4484b7f740 10 client.0 _ll_register_callbacks cb 0x55a8635b8080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4477fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636e0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4476ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4476ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50914/0 (socket says 172.21.15.27:50914) 2024-04-25T13:52:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4476ffd640 1 -- 172.21.15.27:0/4178146510 learned_addr learned my addr 172.21.15.27:0/4178146510 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4477fff640 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863301550 0x55a8636d9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4477fff640 1 -- 172.21.15.27:0/4178146510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 msgr2=0x55a8636d9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4477fff640 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4477fff640 1 -- 172.21.15.27:0/4178146510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863301550 msgr2=0x55a8636d9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4477fff640 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863301550 0x55a8636d9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.442+0000 7f4477fff640 1 -- 172.21.15.27:0/4178146510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86346af70 con 0x55a86332a410 2024-04-25T13:52:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.443+0000 7f4477fff640 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636e0bd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f44700148c0 tx=0x7f4470004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8c71456a90a1a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.443+0000 7f4456ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:52:55.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.443+0000 7f4456ffd640 1 -- 172.21.15.27:0/4178146510 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470024070 con 0x55a86332a410 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.443+0000 7f4456ffd640 1 -- 172.21.15.27:0/4178146510 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f44700139a0 con 0x55a86332a410 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.444+0000 7f4484b7f740 1 -- 172.21.15.27:0/4178146510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4470019070 con 0x55a86332a410 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.444+0000 7f4456ffd640 1 -- 172.21.15.27:0/4178146510 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4470013ba0 con 0x55a86332a410 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.444+0000 7f4484b7f740 10 client.17784 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.444+0000 7f4484b7f740 1 -- 172.21.15.27:0/4178146510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55a863302170 con 0x55a86332a410 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.445+0000 7f4456ffd640 1 -- 172.21.15.27:0/4178146510 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4470013d40 con 0x55a86332a410 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.445+0000 7f4456ffd640 1 -- 172.21.15.27:0/4178146510 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f447002d410 con 0x55a86332a410 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.446+0000 7f4484b7f740 10 client.17784 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.446+0000 7f4484b7f740 10 client.17784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.446+0000 7f4484b7f740 20 client.17784 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.446+0000 7f4484b7f740 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a8632e91f0 0x55a8636d6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.446+0000 7f4484b7f740 4 client.17784 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.446+0000 7f4484b7f740 1 -- 172.21.15.27:0/4178146510 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4470019070 con 0x55a8632e91f0 2024-04-25T13:52:55.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.446+0000 7f4476ffd640 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a8632e91f0 0x55a8636d6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.447+0000 7f4476ffd640 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a8632e91f0 0x55a8636d6200 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4da3351cf5946f0 server_cookie=e6f3f9001af4b909 in_seq=0 out_seq=0 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.447+0000 7f4456ffd640 10 client.17784 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.447+0000 7f4456ffd640 1 -- 172.21.15.27:0/4178146510 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f4470019070 con 0x55a8632e91f0 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.447+0000 7f4456ffd640 10 client.17784 handle_command_reply: tid=0 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.447+0000 7f4484b7f740 1 client.17784 shutdown 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 -- 172.21.15.27:0/4178146510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 msgr2=0x55a8636e0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636e0bd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f44700148c0 tx=0x7f4470004b50 comp rx=0 tx=0).stop 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 -- 172.21.15.27:0/4178146510 shutdown_connections 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a8632e91f0 0x55a8636d6200 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636e0bd0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863309730 0x55a8636d9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4456ffd640 0 client.17784 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 --2- 172.21.15.27:0/4178146510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863301550 0x55a8636d9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 -- 172.21.15.27:0/4178146510 >> 172.21.15.27:0/4178146510 conn(0x55a86331e560 msgr2=0x55a8636c5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 -- 172.21.15.27:0/4178146510 shutdown_connections 2024-04-25T13:52:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 1 -- 172.21.15.27:0/4178146510 wait complete. 2024-04-25T13:52:55.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.448+0000 7f4484b7f740 20 client.17784 trim_cache size 0 max 16384 2024-04-25T13:52:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.480+0000 7f4484b7f740 1 Processor -- start 2024-04-25T13:52:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.480+0000 7f4484b7f740 1 -- start start 2024-04-25T13:52:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.481+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863301550 0x55a8636d7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.481+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636e0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.482+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636d8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.482+0000 7f4477fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863301550 0x55a8636d7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.482+0000 7f4477fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863301550 0x55a8636d7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50926/0 (socket says 172.21.15.27:50926) 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.482+0000 7f4477fff640 1 -- 172.21.15.27:0/3815903344 learned_addr learned my addr 172.21.15.27:0/3815903344 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.482+0000 7f4484b7f740 1 -- 172.21.15.27:0/3815903344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f44781a70f0 con 0x55a86332a410 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.482+0000 7f4484b7f740 1 -- 172.21.15.27:0/3815903344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f44781a9970 con 0x55a863309730 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.482+0000 7f4484b7f740 1 -- 172.21.15.27:0/3815903344 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f44781a7760 con 0x55a863301550 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.482+0000 7f4477fff640 1 --2- 172.21.15.27:0/3815903344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636d8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.483+0000 7f4477fff640 1 -- 172.21.15.27:0/3815903344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 msgr2=0x55a8636e0b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.483+0000 7f4477fff640 1 --2- 172.21.15.27:0/3815903344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636e0b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.483+0000 7f4477fff640 1 -- 172.21.15.27:0/3815903344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 msgr2=0x55a8636d8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.483+0000 7f4477fff640 1 --2- 172.21.15.27:0/3815903344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636d8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.483+0000 7f4477fff640 1 -- 172.21.15.27:0/3815903344 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8636d0190 con 0x55a863301550 2024-04-25T13:52:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.483+0000 7f4477fff640 1 --2- 172.21.15.27:0/3815903344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863301550 0x55a8636d7cd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4470014520 tx=0x7f4470000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4b0fcbc2c7ff877 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.489+0000 7f445d7fa640 1 -- 172.21.15.27:0/3815903344 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470024030 con 0x55a863301550 2024-04-25T13:52:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.489+0000 7f445d7fa640 1 -- 172.21.15.27:0/3815903344 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4470025040 con 0x55a863301550 2024-04-25T13:52:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.489+0000 7f445d7fa640 1 -- 172.21.15.27:0/3815903344 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447002d3f0 con 0x55a863301550 2024-04-25T13:52:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.494+0000 7f4484b7f740 1 -- 172.21.15.27:0/3815903344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863301550 msgr2=0x55a8636d7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.494+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3815903344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863301550 0x55a8636d7cd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4470014520 tx=0x7f4470000f40 comp rx=0 tx=0).stop 2024-04-25T13:52:55.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.500+0000 7f4484b7f740 1 -- 172.21.15.27:0/3815903344 shutdown_connections 2024-04-25T13:52:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.500+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3815903344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a86332a410 0x55a8636d8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.500+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3815903344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636e0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.500+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3815903344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a863301550 0x55a8636d7cd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.500+0000 7f4484b7f740 1 -- 172.21.15.27:0/3815903344 >> 172.21.15.27:0/3815903344 conn(0x55a86331e560 msgr2=0x55a8636d6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.505+0000 7f4484b7f740 1 -- 172.21.15.27:0/3815903344 shutdown_connections 2024-04-25T13:52:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.506+0000 7f4484b7f740 1 -- 172.21.15.27:0/3815903344 wait complete. 2024-04-25T13:52:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.507+0000 7f4484b7f740 1 -- start start 2024-04-25T13:52:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.507+0000 7f4484b7f740 1 Processor -- start 2024-04-25T13:52:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.508+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 0x55a8636e0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.508+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636dd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.508+0000 7f4484b7f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a86332a410 0x55a8636ddbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.508+0000 7f4484b7f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f44781a7e30 con 0x55a863301550 2024-04-25T13:52:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.508+0000 7f4484b7f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f44781a92a0 con 0x55a863309730 2024-04-25T13:52:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.508+0000 7f4484b7f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f44781a8500 con 0x55a86332a410 2024-04-25T13:52:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.508+0000 7f4484b7f740 10 client.0 _ll_register_callbacks cb 0x7f44700145c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:55.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.509+0000 7f4476ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636dd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.509+0000 7f4476ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636dd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52282/0 (socket says 172.21.15.27:52282) 2024-04-25T13:52:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.509+0000 7f4476ffd640 1 -- 172.21.15.27:0/3615652702 learned_addr learned my addr 172.21.15.27:0/3615652702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.509+0000 7f4476ffd640 1 -- 172.21.15.27:0/3615652702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a86332a410 msgr2=0x55a8636ddbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:52:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.509+0000 7f4476ffd640 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a86332a410 0x55a8636ddbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.509+0000 7f4476ffd640 1 -- 172.21.15.27:0/3615652702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 msgr2=0x55a8636e0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.509+0000 7f4476ffd640 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 0x55a8636e0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.509+0000 7f4476ffd640 1 -- 172.21.15.27:0/3615652702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8636e7be0 con 0x55a863309730 2024-04-25T13:52:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.510+0000 7f4476ffd640 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636dd6a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f446c02a900 tx=0x7f446c030640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84562e92dac0c122 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.510+0000 7f4456ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.510+0000 7f4456ffd640 1 -- 172.21.15.27:0/3615652702 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c04c020 con 0x55a863309730 2024-04-25T13:52:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.511+0000 7f4484b7f740 1 -- 172.21.15.27:0/3615652702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4470025040 con 0x55a863309730 2024-04-25T13:52:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.511+0000 7f4456ffd640 1 -- 172.21.15.27:0/3615652702 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f446c0307b0 con 0x55a863309730 2024-04-25T13:52:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.511+0000 7f4456ffd640 1 -- 172.21.15.27:0/3615652702 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f446c037040 con 0x55a863309730 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.511+0000 7f4484b7f740 10 client.27491 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.511+0000 7f4484b7f740 1 -- 172.21.15.27:0/3615652702 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55a8636e8fa0 con 0x55a863309730 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.512+0000 7f4456ffd640 1 -- 172.21.15.27:0/3615652702 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4470025040 con 0x55a863309730 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.512+0000 7f4456ffd640 1 -- 172.21.15.27:0/3615652702 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f446c048790 con 0x55a863309730 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.512+0000 7f4484b7f740 10 client.27491 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.512+0000 7f4484b7f740 10 client.27491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.512+0000 7f4484b7f740 20 client.27491 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.512+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a8632be630 0x55a8636eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.512+0000 7f4484b7f740 4 client.27491 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:52:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.512+0000 7f4484b7f740 1 -- 172.21.15.27:0/3615652702 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a86363e710 con 0x55a8632be630 2024-04-25T13:52:55.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.525+0000 7f4477fff640 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a8632be630 0x55a8636eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.534+0000 7f4477fff640 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a8632be630 0x55a8636eb9d0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d884fbdea35f8ed server_cookie=718eab18d4ccfcf4 in_seq=0 out_seq=0 2024-04-25T13:52:55.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.535+0000 7f4456ffd640 10 client.27491 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.539+0000 7f4456ffd640 1 -- 172.21.15.27:0/3615652702 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a86363e710 con 0x55a8632be630 2024-04-25T13:52:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.539+0000 7f4456ffd640 10 client.27491 handle_command_reply: tid=0 2024-04-25T13:52:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.539+0000 7f4484b7f740 1 client.27491 shutdown 2024-04-25T13:52:55.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.543+0000 7f4484b7f740 1 -- 172.21.15.27:0/3615652702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 msgr2=0x55a8636dd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.543+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636dd6a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f446c02a900 tx=0x7f446c030640 comp rx=0 tx=0).stop 2024-04-25T13:52:55.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 1 -- 172.21.15.27:0/3615652702 shutdown_connections 2024-04-25T13:52:55.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a8632be630 0x55a8636eb9d0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a86332a410 0x55a8636ddbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a863309730 0x55a8636dd6a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 1 --2- 172.21.15.27:0/3615652702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a863301550 0x55a8636e0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 1 -- 172.21.15.27:0/3615652702 >> 172.21.15.27:0/3615652702 conn(0x55a86331e560 msgr2=0x55a8636d7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4456ffd640 0 client.27491 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:55.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 1 -- 172.21.15.27:0/3615652702 shutdown_connections 2024-04-25T13:52:55.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 1 -- 172.21.15.27:0/3615652702 wait complete. 2024-04-25T13:52:55.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.546+0000 7f4484b7f740 20 client.27491 trim_cache size 0 max 16384 2024-04-25T13:52:55.547 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:52:55.547 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:52:55.547 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "3a4ef94b-9f30-4035-975d-fc7e478aa892", 2024-04-25T13:52:55.547 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:52:55.547 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:52:55.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.549+0000 7f447dcd1640 1 -- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44600afcc0 msgr2=0x7f44600b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.549+0000 7f447dcd1640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44600afcc0 0x7f44600b21d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f44680a7230 tx=0x7f44680e27b0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.549+0000 7f447dcd1640 1 -- 172.21.15.27:0/2279459503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44781029f0 msgr2=0x7f447806bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.549+0000 7f447dcd1640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44781029f0 0x7f447806bce0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f446800e9c0 tx=0x7f446800ee80 comp rx=0 tx=0).stop 2024-04-25T13:52:55.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.551+0000 7f447dcd1640 1 -- 172.21.15.27:0/2279459503 shutdown_connections 2024-04-25T13:52:55.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.551+0000 7f447dcd1640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f44600afcc0 0x7f44600b21d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.551+0000 7f447dcd1640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4478108930 0x7f44781a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.551+0000 7f447dcd1640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4478103c30 0x7f447819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.551+0000 7f447dcd1640 1 --2- 172.21.15.27:0/2279459503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44781029f0 0x7f447806bce0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.551+0000 7f447dcd1640 1 -- 172.21.15.27:0/2279459503 >> 172.21.15.27:0/2279459503 conn(0x7f44780fdc00 msgr2=0x7f4478104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.556+0000 7f447dcd1640 1 -- 172.21.15.27:0/2279459503 shutdown_connections 2024-04-25T13:52:55.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.556+0000 7f447dcd1640 1 -- 172.21.15.27:0/2279459503 wait complete. 2024-04-25T13:52:55.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:52:55.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 -- 172.21.15.27:0/1541121919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4103b00 msgr2=0x7f57c4104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1541121919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4103b00 0x7f57c4104800 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009900 tx=0x7f57b802a3f0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 -- 172.21.15.27:0/1541121919 shutdown_connections 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1541121919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57c4109a40 0x7f57c410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1541121919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4104d40 0x7f57c4105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1541121919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4103b00 0x7f57c4104800 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 -- 172.21.15.27:0/1541121919 >> 172.21.15.27:0/1541121919 conn(0x7f57c40fed10 msgr2=0x7f57c41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 -- 172.21.15.27:0/1541121919 shutdown_connections 2024-04-25T13:52:55.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.751+0000 7f57cafb2640 1 -- 172.21.15.27:0/1541121919 wait complete. 2024-04-25T13:52:55.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57cafb2640 1 Processor -- start 2024-04-25T13:52:55.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57cafb2640 1 -- start start 2024-04-25T13:52:55.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57cafb2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4103b00 0x7f57c419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57cafb2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4104d40 0x7f57c419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57cafb2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57c4109a40 0x7f57c41a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57cafb2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57c410f5e0 con 0x7f57c4103b00 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57cafb2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f57c410f460 con 0x7f57c4104d40 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57cafb2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f57c410f760 con 0x7f57c4109a40 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4104d40 0x7f57c419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4104d40 0x7f57c419ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52304/0 (socket says 172.21.15.27:52304) 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.752+0000 7f57c3fff640 1 -- 172.21.15.27:0/1222666367 learned_addr learned my addr 172.21.15.27:0/1222666367 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c954f640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57c4109a40 0x7f57c41a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c8d4e640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4103b00 0x7f57c419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c3fff640 1 -- 172.21.15.27:0/1222666367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57c4109a40 msgr2=0x7f57c41a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57c4109a40 0x7f57c41a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c3fff640 1 -- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4103b00 msgr2=0x7f57c419c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4103b00 0x7f57c419c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c3fff640 1 -- 172.21.15.27:0/1222666367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c41a4010 con 0x7f57c4104d40 2024-04-25T13:52:55.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c954f640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57c4109a40 0x7f57c41a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:55.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.753+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4104d40 0x7f57c419ca10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f57b000e9f0 tx=0x7f57b000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.754+0000 7f57c1ffb640 1 -- 172.21.15.27:0/1222666367 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b000ccc0 con 0x7f57c4104d40 2024-04-25T13:52:55.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.754+0000 7f57c1ffb640 1 -- 172.21.15.27:0/1222666367 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b00108e0 con 0x7f57c4104d40 2024-04-25T13:52:55.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.754+0000 7f57c1ffb640 1 -- 172.21.15.27:0/1222666367 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b0018880 con 0x7f57c4104d40 2024-04-25T13:52:55.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.754+0000 7f57cafb2640 1 -- 172.21.15.27:0/1222666367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c4194d20 con 0x7f57c4104d40 2024-04-25T13:52:55.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.754+0000 7f57cafb2640 1 -- 172.21.15.27:0/1222666367 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c4195270 con 0x7f57c4104d40 2024-04-25T13:52:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.758+0000 7f57c1ffb640 1 -- 172.21.15.27:0/1222666367 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f57b0010430 con 0x7f57c4104d40 2024-04-25T13:52:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.758+0000 7f57d1e5e740 1 Processor -- start 2024-04-25T13:52:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.758+0000 7f57d1e5e740 1 -- start start 2024-04-25T13:52:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5caca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cacfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cadc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5652c5c11d80 con 0x5652c590b390 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5652c5bcb8b0 con 0x5652c58e24d0 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5652c5c0dd40 con 0x5652c58ea6b0 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c954f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5caca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c954f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5caca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34354/0 (socket says 172.21.15.27:34354) 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c954f640 1 -- 172.21.15.27:0/562053531 learned_addr learned my addr 172.21.15.27:0/562053531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c3fff640 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cadc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c8d4e640 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cacfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c1ffb640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f57b40afc40 0x7f57b40b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c954f640 1 -- 172.21.15.27:0/562053531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 msgr2=0x5652c5cadc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c954f640 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cadc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c954f640 1 -- 172.21.15.27:0/562053531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 msgr2=0x5652c5cacfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c954f640 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cacfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c954f640 1 -- 172.21.15.27:0/562053531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652c5b9ca40 con 0x5652c590b390 2024-04-25T13:52:55.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.759+0000 7f57c1ffb640 1 -- 172.21.15.27:0/1222666367 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f57b00def70 con 0x7f57c4104d40 2024-04-25T13:52:55.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.760+0000 7f57c954f640 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5caca60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f57bc014f90 tx=0x7f57bc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39ae986f04aff91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.760+0000 7f57c954f640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f57b40afc40 0x7f57b40b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.760+0000 7f57aeffd640 1 -- 172.21.15.27:0/562053531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc022040 con 0x5652c590b390 2024-04-25T13:52:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.760+0000 7f57aeffd640 1 -- 172.21.15.27:0/562053531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57bc023020 con 0x5652c590b390 2024-04-25T13:52:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.761+0000 7f57aeffd640 1 -- 172.21.15.27:0/562053531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc02b750 con 0x5652c590b390 2024-04-25T13:52:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.761+0000 7f57c954f640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f57b40afc40 0x7f57b40b2150 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00bfd0 tx=0x7f57bc021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.761+0000 7f57d1e5e740 1 -- 172.21.15.27:0/562053531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 msgr2=0x5652c5caca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.761+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5caca60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f57bc014f90 tx=0x7f57bc009500 comp rx=0 tx=0).stop 2024-04-25T13:52:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 -- 172.21.15.27:0/562053531 shutdown_connections 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cadc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cacfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/562053531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5caca60 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 -- 172.21.15.27:0/562053531 >> 172.21.15.27:0/562053531 conn(0x5652c58ff4e0 msgr2=0x5652c5ca8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 -- 172.21.15.27:0/562053531 shutdown_connections 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 -- 172.21.15.27:0/562053531 wait complete. 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 -- start start 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.762+0000 7f57d1e5e740 1 Processor -- start 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cb6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cb6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5cbddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5652c5cae820 con 0x5652c590b390 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5652c5b98d10 con 0x5652c58e24d0 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5652c5cae9a0 con 0x5652c58ea6b0 2024-04-25T13:52:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cb6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cb6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50958/0 (socket says 172.21.15.27:50958) 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c8d4e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cb6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 -- 172.21.15.27:0/1725796428 learned_addr learned my addr 172.21.15.27:0/1725796428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c954f640 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5cbddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57d1e5e740 10 client.0 _ll_register_callbacks cb 0x5652c5bc5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 -- 172.21.15.27:0/1725796428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 msgr2=0x5652c5cb6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cb6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 -- 172.21.15.27:0/1725796428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 msgr2=0x5652c5cbddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5cbddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 -- 172.21.15.27:0/1725796428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652c59ed7d0 con 0x5652c58ea6b0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cb6e70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f57b00a3500 tx=0x7f57b00e2640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd762b233c3e724b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.763+0000 7f57a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.764+0000 7f57a3fff640 1 -- 172.21.15.27:0/1725796428 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b0010720 con 0x5652c58ea6b0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.764+0000 7f57a3fff640 1 -- 172.21.15.27:0/1725796428 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f57b0010a90 con 0x5652c58ea6b0 2024-04-25T13:52:55.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.764+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1725796428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57bc023020 con 0x5652c58ea6b0 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.764+0000 7f57a3fff640 1 -- 172.21.15.27:0/1725796428 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f57b00a6f90 con 0x5652c58ea6b0 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.764+0000 7f57d1e5e740 10 client.27538 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.764+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1725796428 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5652c59233f0 con 0x5652c58ea6b0 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.764+0000 7f57a3fff640 1 -- 172.21.15.27:0/1725796428 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57bc023020 con 0x5652c58ea6b0 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.764+0000 7f57a3fff640 1 -- 172.21.15.27:0/1725796428 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f57b00e10d0 con 0x5652c58ea6b0 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57d1e5e740 10 client.27538 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57d1e5e740 10 client.27538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57d1e5e740 20 client.27538 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5652c58c9f60 0x5652c5cc4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57d1e5e740 4 client.27538 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1725796428 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652c5c1f990 con 0x5652c58c9f60 2024-04-25T13:52:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57c8d4e640 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5652c58c9f60 0x5652c5cc4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57a3fff640 10 client.27538 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.765+0000 7f57c8d4e640 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5652c58c9f60 0x5652c5cc4330 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af5984017e771dd server_cookie=135bf7f528786154 in_seq=0 out_seq=0 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.766+0000 7f57a3fff640 1 -- 172.21.15.27:0/1725796428 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5652c5c1f990 con 0x5652c58c9f60 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.766+0000 7f57a3fff640 10 client.27538 handle_command_reply: tid=0 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.766+0000 7f57d1e5e740 1 client.27538 shutdown 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1725796428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 msgr2=0x5652c5cb6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cb6e70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f57b00a3500 tx=0x7f57b00e2640 comp rx=0 tx=0).stop 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1725796428 shutdown_connections 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5652c58c9f60 0x5652c5cc4330 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c590b390 0x5652c5cbddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cb6e70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57a3fff640 0 client.27538 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1725796428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cb6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1725796428 >> 172.21.15.27:0/1725796428 conn(0x5652c58ff4e0 msgr2=0x5652c5ca73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1725796428 shutdown_connections 2024-04-25T13:52:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1725796428 wait complete. 2024-04-25T13:52:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.767+0000 7f57d1e5e740 20 client.27538 trim_cache size 0 max 16384 2024-04-25T13:52:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.795+0000 7f57d1e5e740 1 Processor -- start 2024-04-25T13:52:55.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.795+0000 7f57d1e5e740 1 -- start start 2024-04-25T13:52:55.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58c9f60 0x5652c5cb1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58e24d0 0x5652c5cab780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cabcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41a7120 con 0x5652c58e24d0 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41a99a0 con 0x5652c58c9f60 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41a7790 con 0x5652c58ea6b0 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58e24d0 0x5652c5cab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c8d4e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58c9f60 0x5652c5cb1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58e24d0 0x5652c5cab780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34376/0 (socket says 172.21.15.27:34376) 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 -- 172.21.15.27:0/1621696967 learned_addr learned my addr 172.21.15.27:0/1621696967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c954f640 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cabcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 -- 172.21.15.27:0/1621696967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 msgr2=0x5652c5cabcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cabcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 -- 172.21.15.27:0/1621696967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58c9f60 msgr2=0x5652c5cb1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58c9f60 0x5652c5cb1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 -- 172.21.15.27:0/1621696967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652c58e30f0 con 0x5652c58e24d0 2024-04-25T13:52:55.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c954f640 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cabcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:52:55.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c8d4e640 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58c9f60 0x5652c5cb1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:55.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.796+0000 7f57c3fff640 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58e24d0 0x5652c5cab780 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f57b00a7410 tx=0x7f57b000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637a58a95cbdca22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.797+0000 7f57ae7fc640 1 -- 172.21.15.27:0/1621696967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a6dd0 con 0x5652c58e24d0 2024-04-25T13:52:55.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.797+0000 7f57ae7fc640 1 -- 172.21.15.27:0/1621696967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b00a7b80 con 0x5652c58e24d0 2024-04-25T13:52:55.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.797+0000 7f57ae7fc640 1 -- 172.21.15.27:0/1621696967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a3120 con 0x5652c58e24d0 2024-04-25T13:52:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1621696967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58e24d0 msgr2=0x5652c5cab780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58e24d0 0x5652c5cab780 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f57b00a7410 tx=0x7f57b000e940 comp rx=0 tx=0).stop 2024-04-25T13:52:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1621696967 shutdown_connections 2024-04-25T13:52:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58ea6b0 0x5652c5cabcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58e24d0 0x5652c5cab780 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/1621696967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58c9f60 0x5652c5cb1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1621696967 >> 172.21.15.27:0/1621696967 conn(0x5652c58ff4e0 msgr2=0x5652c5cab070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1621696967 shutdown_connections 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 -- 172.21.15.27:0/1621696967 wait complete. 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.798+0000 7f57d1e5e740 1 -- start start 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.799+0000 7f57d1e5e740 1 Processor -- start 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.799+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58c9f60 0x5652c5cbb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.799+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cbbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.799+0000 7f57d1e5e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58ea6b0 0x5652c5cbcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41a7e60 con 0x5652c58ea6b0 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41a92d0 con 0x5652c58e24d0 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57d1e5e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41a8530 con 0x5652c58c9f60 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c954f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58ea6b0 0x5652c5cbcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57d1e5e740 10 client.0 _ll_register_callbacks cb 0x7f57b00e2740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cbbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cbbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52340/0 (socket says 172.21.15.27:52340) 2024-04-25T13:52:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 -- 172.21.15.27:0/4264746203 learned_addr learned my addr 172.21.15.27:0/4264746203 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c8d4e640 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58c9f60 0x5652c5cbb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 -- 172.21.15.27:0/4264746203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58c9f60 msgr2=0x5652c5cbb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58c9f60 0x5652c5cbb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 -- 172.21.15.27:0/4264746203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58ea6b0 msgr2=0x5652c5cbcbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58ea6b0 0x5652c5cbcbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 -- 172.21.15.27:0/4264746203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652c5bd24b0 con 0x5652c58e24d0 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c954f640 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58ea6b0 0x5652c5cbcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57c3fff640 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cbbc80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f57b00a3a20 tx=0x7f57b00a6910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7273b5dfcb85f065 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.800+0000 7f57a3fff640 1 -- 172.21.15.27:0/4264746203 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a6dd0 con 0x5652c58e24d0 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57a3fff640 1 -- 172.21.15.27:0/4264746203 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f57b00a6360 con 0x5652c58e24d0 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 1 -- 172.21.15.27:0/4264746203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b00a7b80 con 0x5652c58e24d0 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57a3fff640 1 -- 172.21.15.27:0/4264746203 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f57b00c8020 con 0x5652c58e24d0 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 10 client.27509 fetch_fsmap learned FSMap version 19 2024-04-25T13:52:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 1 -- 172.21.15.27:0/4264746203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5652c5cbe230 con 0x5652c58e24d0 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57a3fff640 1 -- 172.21.15.27:0/4264746203 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b00b02e0 con 0x5652c58e24d0 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57a3fff640 1 -- 172.21.15.27:0/4264746203 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f57b00e2240 con 0x5652c58e24d0 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 10 client.27509 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 10 client.27509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 20 client.27509 populate_metadata read hostname 'smithi027' 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5652c590b390 0x5652c5d2e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 4 client.27509 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57d1e5e740 1 -- 172.21.15.27:0/4264746203 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f57b00a7b80 con 0x5652c590b390 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.801+0000 7f57c8d4e640 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5652c590b390 0x5652c5d2e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.802+0000 7f57a3fff640 10 client.27509 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.802+0000 7f57c8d4e640 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5652c590b390 0x5652c5d2e6b0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce1c68641da4bcd0 server_cookie=d2ad62ac1f9ff24f in_seq=0 out_seq=0 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.802+0000 7f57a3fff640 1 -- 172.21.15.27:0/4264746203 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f57b00a7b80 con 0x5652c590b390 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.802+0000 7f57a3fff640 10 client.27509 handle_command_reply: tid=0 2024-04-25T13:52:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.803+0000 7f57d1e5e740 1 client.27509 shutdown 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.804+0000 7f57d1e5e740 1 -- 172.21.15.27:0/4264746203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 msgr2=0x5652c5cbbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.804+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cbbc80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f57b00a3a20 tx=0x7f57b00a6910 comp rx=0 tx=0).stop 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 1 -- 172.21.15.27:0/4264746203 shutdown_connections 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5652c590b390 0x5652c5d2e6b0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5652c58ea6b0 0x5652c5cbcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652c58e24d0 0x5652c5cbbc80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 1 --2- 172.21.15.27:0/4264746203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5652c58c9f60 0x5652c5cbb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 1 -- 172.21.15.27:0/4264746203 >> 172.21.15.27:0/4264746203 conn(0x5652c58ff4e0 msgr2=0x5652c5cabaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57a3fff640 0 client.27509 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 1 -- 172.21.15.27:0/4264746203 shutdown_connections 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 1 -- 172.21.15.27:0/4264746203 wait complete. 2024-04-25T13:52:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.805+0000 7f57d1e5e740 20 client.27509 trim_cache size 0 max 16384 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout: "3a4ef94b-9f30-4035-975d-fc7e478aa892": { 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "3a4ef94b-9f30-4035-975d-fc7e478aa892", 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:52:55.808 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:52:55.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 -- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f57b40afc40 msgr2=0x7f57b40b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f57b40afc40 0x7f57b40b2150 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00bfd0 tx=0x7f57bc021000 comp rx=0 tx=0).stop 2024-04-25T13:52:55.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 -- 172.21.15.27:0/1222666367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4104d40 msgr2=0x7f57c419ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:52:55.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4104d40 0x7f57c419ca10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f57b000e9f0 tx=0x7f57b000edf0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 -- 172.21.15.27:0/1222666367 shutdown_connections 2024-04-25T13:52:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f57b40afc40 0x7f57b40b2150 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57c4109a40 0x7f57c41a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57c4104d40 0x7f57c419ca10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 --2- 172.21.15.27:0/1222666367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4103b00 0x7f57c419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:52:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 -- 172.21.15.27:0/1222666367 >> 172.21.15.27:0/1222666367 conn(0x7f57c40fed10 msgr2=0x7f57c4101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:52:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 -- 172.21.15.27:0/1222666367 shutdown_connections 2024-04-25T13:52:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:52:55.806+0000 7f57cafb2640 1 -- 172.21.15.27:0/1222666367 wait complete. 2024-04-25T13:52:55.820 INFO:tasks.cephfs.filesystem:scrub status for tag:3a4ef94b-9f30-4035-975d-fc7e478aa892 - {'path': '/', 'tag': '3a4ef94b-9f30-4035-975d-fc7e478aa892', 'options': 'recursive,force'} 2024-04-25T13:52:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:56 smithi027 ceph-mon[29140]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.7 MiB/s wr, 588 op/s 2024-04-25T13:52:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:56 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:52:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:56 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:56 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:52:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:56 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:56.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:56 smithi122 ceph-mon[32843]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.7 MiB/s wr, 588 op/s 2024-04-25T13:52:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:56 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:52:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:56 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:56 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:52:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:56 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:56 smithi045 ceph-mon[32731]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.7 MiB/s wr, 588 op/s 2024-04-25T13:52:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:56 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:52:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:56 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:56 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:52:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:56 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:52:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 710.0 s, 1869.6 tps, lat 52.773 ms stddev 56.094 2024-04-25T13:52:58.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:58 smithi122 ceph-mon[32843]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.1 MiB/s wr, 666 op/s 2024-04-25T13:52:58.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:58 smithi045 ceph-mon[32731]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.1 MiB/s wr, 666 op/s 2024-04-25T13:52:58.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:58 smithi027 ceph-mon[29140]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.1 MiB/s wr, 666 op/s 2024-04-25T13:52:59.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:52:59 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:52:59.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:52:59 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:52:59.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:52:59 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:53:00.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:00 smithi027 ceph-mon[29140]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 6.1 MiB/s wr, 659 op/s 2024-04-25T13:53:00.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:00 smithi122 ceph-mon[32843]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 6.1 MiB/s wr, 659 op/s 2024-04-25T13:53:00.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:00 smithi045 ceph-mon[32731]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 6.1 MiB/s wr, 659 op/s 2024-04-25T13:53:02.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:02 smithi027 ceph-mon[29140]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.8 MiB/s wr, 750 op/s 2024-04-25T13:53:02.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:02 smithi122 ceph-mon[32843]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.8 MiB/s wr, 750 op/s 2024-04-25T13:53:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:02 smithi045 ceph-mon[32731]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.8 MiB/s wr, 750 op/s 2024-04-25T13:53:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 715.0 s, 2258.0 tps, lat 44.682 ms stddev 51.317 2024-04-25T13:53:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.254+0000 7f36610ee640 1 -- 172.21.15.27:0/3996457042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 msgr2=0x7f365c1033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.254+0000 7f36610ee640 1 --2- 172.21.15.27:0/3996457042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 0x7f365c1033c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f364c009900 tx=0x7f364c02a410 comp rx=0 tx=0).stop 2024-04-25T13:53:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.254+0000 7f36610ee640 1 -- 172.21.15.27:0/3996457042 shutdown_connections 2024-04-25T13:53:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.254+0000 7f36610ee640 1 --2- 172.21.15.27:0/3996457042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f365c1044b0 0x7f365c10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.255+0000 7f36610ee640 1 --2- 172.21.15.27:0/3996457042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f365c103990 0x7f365c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.255+0000 7f36610ee640 1 --2- 172.21.15.27:0/3996457042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 0x7f365c1033c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.255+0000 7f36610ee640 1 -- 172.21.15.27:0/3996457042 >> 172.21.15.27:0/3996457042 conn(0x7f365c0fd390 msgr2=0x7f365c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.256+0000 7f36610ee640 1 -- 172.21.15.27:0/3996457042 shutdown_connections 2024-04-25T13:53:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.257+0000 7f36610ee640 1 -- 172.21.15.27:0/3996457042 wait complete. 2024-04-25T13:53:03.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.257+0000 7f36610ee640 1 Processor -- start 2024-04-25T13:53:03.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f36610ee640 1 -- start start 2024-04-25T13:53:03.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f36610ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 0x7f365c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f36610ee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f365c103990 0x7f365c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f36610ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f365c1044b0 0x7f365c1a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f36610ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f365c111880 con 0x7f365c1044b0 2024-04-25T13:53:03.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f36610ee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f365c111700 con 0x7f365c102fb0 2024-04-25T13:53:03.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f36610ee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f365c111a00 con 0x7f365c103990 2024-04-25T13:53:03.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f365b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f365c1044b0 0x7f365c1a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f365b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f365c1044b0 0x7f365c1a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34404/0 (socket says 172.21.15.27:34404) 2024-04-25T13:53:03.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f365b59e640 1 -- 172.21.15.27:0/2968280965 learned_addr learned my addr 172.21.15.27:0/2968280965 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:03.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f365a59c640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f365c103990 0x7f365c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.259 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.258+0000 7f365ad9d640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 0x7f365c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f365a59c640 1 -- 172.21.15.27:0/2968280965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 msgr2=0x7f365c19c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f365a59c640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 0x7f365c19c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f365a59c640 1 -- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f365c1044b0 msgr2=0x7f365c1a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f365a59c640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f365c1044b0 0x7f365c1a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f365a59c640 1 -- 172.21.15.27:0/2968280965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365c1a3fa0 con 0x7f365c103990 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f365b59e640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f365c1044b0 0x7f365c1a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f365ad9d640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 0x7f365c19c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f365a59c640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f365c103990 0x7f365c19c9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f365000e9f0 tx=0x7f365000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f3643fff640 1 -- 172.21.15.27:0/2968280965 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365000ccc0 con 0x7f365c103990 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f36610ee640 1 -- 172.21.15.27:0/2968280965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365c10bb90 con 0x7f365c103990 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f3643fff640 1 -- 172.21.15.27:0/2968280965 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3650005020 con 0x7f365c103990 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.259+0000 7f3643fff640 1 -- 172.21.15.27:0/2968280965 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650004a00 con 0x7f365c103990 2024-04-25T13:53:03.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.260+0000 7f36610ee640 1 -- 172.21.15.27:0/2968280965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f365c10c0e0 con 0x7f365c103990 2024-04-25T13:53:03.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.262+0000 7f3643fff640 1 -- 172.21.15.27:0/2968280965 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3650004ba0 con 0x7f365c103990 2024-04-25T13:53:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.263+0000 7f3667f9f740 1 Processor -- start 2024-04-25T13:53:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.264+0000 7f3643fff640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36440afbe0 0x7f36440b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.264+0000 7f3643fff640 1 -- 172.21.15.27:0/2968280965 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f36500deb70 con 0x7f365c103990 2024-04-25T13:53:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.264+0000 7f365b59e640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36440afbe0 0x7f36440b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.264+0000 7f3667f9f740 1 -- start start 2024-04-25T13:53:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db364970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8daf9b2f0 0x55f8db364ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8dafa34d0 0x55f8db36bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f3667f9f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f8db2ca460 con 0x55f8dafc41b0 2024-04-25T13:53:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f3667f9f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f8db284590 con 0x55f8dafa34d0 2024-04-25T13:53:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f3667f9f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f8db2c6420 con 0x55f8daf9b2f0 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f365a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8daf9b2f0 0x55f8db364ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f365a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8daf9b2f0 0x55f8db364ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51022/0 (socket says 172.21.15.27:51022) 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f365a59c640 1 -- 172.21.15.27:0/421464976 learned_addr learned my addr 172.21.15.27:0/421464976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f365ad9d640 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db364970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f365b59e640 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8dafa34d0 0x55f8db36bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f365a59c640 1 -- 172.21.15.27:0/421464976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8dafa34d0 msgr2=0x55f8db36bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f365a59c640 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8dafa34d0 0x55f8db36bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.265+0000 7f365a59c640 1 -- 172.21.15.27:0/421464976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 msgr2=0x55f8db364970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.266+0000 7f365a59c640 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db364970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.266+0000 7f365a59c640 1 -- 172.21.15.27:0/421464976 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8db2910d0 con 0x55f8daf9b2f0 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.266+0000 7f365b59e640 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8dafa34d0 0x55f8db36bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:53:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.266+0000 7f365b59e640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36440afbe0 0x7f36440b20f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3654005c40 tx=0x7f365400f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.266+0000 7f365a59c640 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8daf9b2f0 0x55f8db364ef0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f36500a6fb0 tx=0x7f3650005460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28dd3764834b1dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.266+0000 7f36417fa640 1 -- 172.21.15.27:0/421464976 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36500c1070 con 0x55f8daf9b2f0 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.266+0000 7f36417fa640 1 -- 172.21.15.27:0/421464976 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36500a7020 con 0x55f8daf9b2f0 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.266+0000 7f36417fa640 1 -- 172.21.15.27:0/421464976 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365001c040 con 0x55f8daf9b2f0 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 -- 172.21.15.27:0/421464976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8daf9b2f0 msgr2=0x55f8db364ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8daf9b2f0 0x55f8db364ef0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f36500a6fb0 tx=0x7f3650005460 comp rx=0 tx=0).stop 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 -- 172.21.15.27:0/421464976 shutdown_connections 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8dafa34d0 0x55f8db36bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8daf9b2f0 0x55f8db364ef0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 --2- 172.21.15.27:0/421464976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db364970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.269 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 -- 172.21.15.27:0/421464976 >> 172.21.15.27:0/421464976 conn(0x55f8dafb8300 msgr2=0x55f8db360300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:03.269 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 -- 172.21.15.27:0/421464976 shutdown_connections 2024-04-25T13:53:03.269 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 -- 172.21.15.27:0/421464976 wait complete. 2024-04-25T13:53:03.269 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 -- start start 2024-04-25T13:53:03.269 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.268+0000 7f3667f9f740 1 Processor -- start 2024-04-25T13:53:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db36e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafa34d0 0x55f8db36ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafc41b0 0x55f8db375b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f3667f9f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f8db36c840 con 0x55f8dafa34d0 2024-04-25T13:53:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f3667f9f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f8db251b60 con 0x55f8daf9b2f0 2024-04-25T13:53:03.272 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f3667f9f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f8db36c9c0 con 0x55f8dafc41b0 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f3667f9f740 10 client.0 _ll_register_callbacks cb 0x55f8db2762b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafa34d0 0x55f8db36ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafa34d0 0x55f8db36ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34426/0 (socket says 172.21.15.27:34426) 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365a59c640 1 -- 172.21.15.27:0/919871895 learned_addr learned my addr 172.21.15.27:0/919871895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365b59e640 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafc41b0 0x55f8db375b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365ad9d640 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db36e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365a59c640 1 -- 172.21.15.27:0/919871895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafc41b0 msgr2=0x55f8db375b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365a59c640 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafc41b0 0x55f8db375b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365a59c640 1 -- 172.21.15.27:0/919871895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 msgr2=0x55f8db36e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365a59c640 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db36e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365a59c640 1 -- 172.21.15.27:0/919871895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8dafdc210 con 0x55f8dafa34d0 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.269+0000 7f365b59e640 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafc41b0 0x55f8db375b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f365a59c640 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafa34d0 0x55f8db36ebe0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f36500a4290 tx=0x7f36500ab4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d534a9fd9443474 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f363eff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f363eff5640 1 -- 172.21.15.27:0/919871895 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36500c1050 con 0x55f8dafa34d0 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f363eff5640 1 -- 172.21.15.27:0/919871895 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f36500af9f0 con 0x55f8dafa34d0 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f3667f9f740 1 -- 172.21.15.27:0/919871895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36500a7020 con 0x55f8dafa34d0 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f363eff5640 1 -- 172.21.15.27:0/919871895 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f365001c040 con 0x55f8dafa34d0 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f3667f9f740 10 client.17817 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f3667f9f740 1 -- 172.21.15.27:0/919871895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f8db0a8620 con 0x55f8dafa34d0 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f363eff5640 1 -- 172.21.15.27:0/919871895 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36500a7020 con 0x55f8dafa34d0 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.270+0000 7f363eff5640 1 -- 172.21.15.27:0/919871895 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f36500a2a20 con 0x55f8dafa34d0 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f3667f9f740 10 client.17817 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f3667f9f740 10 client.17817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f3667f9f740 20 client.17817 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f3667f9f740 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8daf82d80 0x55f8db37bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f3667f9f740 4 client.17817 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f3667f9f740 1 -- 172.21.15.27:0/919871895 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8db2d8070 con 0x55f8daf82d80 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f365ad9d640 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8daf82d80 0x55f8db37bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f363eff5640 10 client.17817 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.271+0000 7f365ad9d640 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8daf82d80 0x55f8db37bf00 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffdb0e5c0daf54d2 server_cookie=1efeee6f69b45c2d in_seq=0 out_seq=0 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f363eff5640 1 -- 172.21.15.27:0/919871895 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f8db2d8070 con 0x55f8daf82d80 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f363eff5640 10 client.17817 handle_command_reply: tid=0 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 client.17817 shutdown 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 -- 172.21.15.27:0/919871895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafa34d0 msgr2=0x55f8db36ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafa34d0 0x55f8db36ebe0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f36500a4290 tx=0x7f36500ab4e0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 -- 172.21.15.27:0/919871895 shutdown_connections 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8daf82d80 0x55f8db37bf00 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafc41b0 0x55f8db375b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafa34d0 0x55f8db36ebe0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 --2- 172.21.15.27:0/919871895 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db36e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f363eff5640 0 client.17817 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 -- 172.21.15.27:0/919871895 >> 172.21.15.27:0/919871895 conn(0x55f8dafb8300 msgr2=0x55f8db363210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.272+0000 7f3667f9f740 1 -- 172.21.15.27:0/919871895 shutdown_connections 2024-04-25T13:53:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.273+0000 7f3667f9f740 1 -- 172.21.15.27:0/919871895 wait complete. 2024-04-25T13:53:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.273+0000 7f3667f9f740 20 client.17817 trim_cache size 0 max 16384 2024-04-25T13:53:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.298+0000 7f3667f9f740 1 Processor -- start 2024-04-25T13:53:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.298+0000 7f3667f9f740 1 -- start start 2024-04-25T13:53:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.298+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db35fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.298+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db360190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.298+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db375a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.298+0000 7f3667f9f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f365c1a7000 con 0x55f8dafc41b0 2024-04-25T13:53:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.298+0000 7f3667f9f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f365c1a9860 con 0x55f8daf9b2f0 2024-04-25T13:53:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.298+0000 7f3667f9f740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f365c1a7650 con 0x55f8dafa34d0 2024-04-25T13:53:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.299+0000 7f365ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db35fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.299+0000 7f365ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db35fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52374/0 (socket says 172.21.15.27:52374) 2024-04-25T13:53:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.299+0000 7f365ad9d640 1 -- 172.21.15.27:0/2192916380 learned_addr learned my addr 172.21.15.27:0/2192916380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.299+0000 7f365a59c640 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db360190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.299+0000 7f365b59e640 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db375a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.300+0000 7f365ad9d640 1 -- 172.21.15.27:0/2192916380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 msgr2=0x55f8db360190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.300+0000 7f365ad9d640 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db360190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.300+0000 7f365ad9d640 1 -- 172.21.15.27:0/2192916380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 msgr2=0x55f8db375a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.300+0000 7f365ad9d640 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db375a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.300+0000 7f365ad9d640 1 -- 172.21.15.27:0/2192916380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8db375fa0 con 0x55f8daf9b2f0 2024-04-25T13:53:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.300+0000 7f365a59c640 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db360190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.300+0000 7f365ad9d640 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db35fc50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f364c000c00 tx=0x7f364c005f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73b44f47b6b281be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.301+0000 7f3640ff9640 1 -- 172.21.15.27:0/2192916380 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c052070 con 0x55f8daf9b2f0 2024-04-25T13:53:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.301+0000 7f3640ff9640 1 -- 172.21.15.27:0/2192916380 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f364c004400 con 0x55f8daf9b2f0 2024-04-25T13:53:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.301+0000 7f3640ff9640 1 -- 172.21.15.27:0/2192916380 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c035040 con 0x55f8daf9b2f0 2024-04-25T13:53:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 -- 172.21.15.27:0/2192916380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 msgr2=0x55f8db35fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db35fc50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f364c000c00 tx=0x7f364c005f20 comp rx=0 tx=0).stop 2024-04-25T13:53:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 -- 172.21.15.27:0/2192916380 shutdown_connections 2024-04-25T13:53:03.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db375a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db360190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 --2- 172.21.15.27:0/2192916380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db35fc50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 -- 172.21.15.27:0/2192916380 >> 172.21.15.27:0/2192916380 conn(0x55f8dafb8300 msgr2=0x55f8db36cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 -- 172.21.15.27:0/2192916380 shutdown_connections 2024-04-25T13:53:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.302+0000 7f3667f9f740 1 -- 172.21.15.27:0/2192916380 wait complete. 2024-04-25T13:53:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.303+0000 7f3667f9f740 1 -- start start 2024-04-25T13:53:03.303 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.303+0000 7f3667f9f740 1 Processor -- start 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.303+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db372950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.303+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db372e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.303+0000 7f3667f9f740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db3e2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f3667f9f740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f365c1a7d20 con 0x55f8dafc41b0 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db372950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db372e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db372e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51038/0 (socket says 172.21.15.27:51038) 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db372950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:52384/0 (socket says 172.21.15.27:52384) 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 -- 172.21.15.27:0/3414176780 learned_addr learned my addr 172.21.15.27:0/3414176780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f365c1a9190 con 0x55f8daf9b2f0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365b59e640 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db3e2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f365c1a83f0 con 0x55f8dafa34d0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f3667f9f740 10 client.0 _ll_register_callbacks cb 0x7f36500bf070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 -- 172.21.15.27:0/3414176780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 msgr2=0x55f8db372950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db372950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 -- 172.21.15.27:0/3414176780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 msgr2=0x55f8db3e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db3e2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 -- 172.21.15.27:0/3414176780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8db3e26c0 con 0x55f8dafa34d0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365ad9d640 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db372950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f365a59c640 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db372e90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f36500a30a0 tx=0x7f3650009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfd2e8cee01c0aca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f363eff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.304+0000 7f363eff5640 1 -- 172.21.15.27:0/3414176780 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36500c1040 con 0x55f8dafa34d0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.305+0000 7f363eff5640 1 -- 172.21.15.27:0/3414176780 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f36500a3610 con 0x55f8dafa34d0 2024-04-25T13:53:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.305+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f364c004400 con 0x55f8dafa34d0 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.305+0000 7f363eff5640 1 -- 172.21.15.27:0/3414176780 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f36500afc00 con 0x55f8dafa34d0 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.305+0000 7f3667f9f740 10 client.27562 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.305+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f8db3e3770 con 0x55f8dafa34d0 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.305+0000 7f363eff5640 1 -- 172.21.15.27:0/3414176780 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36500c6070 con 0x55f8dafa34d0 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.305+0000 7f363eff5640 1 -- 172.21.15.27:0/3414176780 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f36500dcb00 con 0x55f8dafa34d0 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.305+0000 7f3667f9f740 10 client.27562 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.306+0000 7f3667f9f740 10 client.27562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.306+0000 7f3667f9f740 20 client.27562 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.306+0000 7f3667f9f740 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8daf581c0 0x55f8db3e5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.306+0000 7f3667f9f740 4 client.27562 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.306+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f364c004400 con 0x55f8daf581c0 2024-04-25T13:53:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.306+0000 7f365ad9d640 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8daf581c0 0x55f8db3e5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.307+0000 7f363eff5640 10 client.27562 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.307+0000 7f365ad9d640 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8daf581c0 0x55f8db3e5fa0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b7806bfbc284c75 server_cookie=17c70a1739b34b56 in_seq=0 out_seq=0 2024-04-25T13:53:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.427+0000 7f363eff5640 1 -- 172.21.15.27:0/3414176780 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f364c004400 con 0x55f8daf581c0 2024-04-25T13:53:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.427+0000 7f363eff5640 10 client.27562 handle_command_reply: tid=0 2024-04-25T13:53:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.427+0000 7f3667f9f740 1 client.27562 shutdown 2024-04-25T13:53:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.427+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 msgr2=0x55f8db372e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.427+0000 7f3667f9f740 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db372e90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f36500a30a0 tx=0x7f3650009f60 comp rx=0 tx=0).stop 2024-04-25T13:53:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 shutdown_connections 2024-04-25T13:53:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f8daf581c0 0x55f8db3e5fa0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.428 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f8dafc41b0 0x55f8db3e2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f8dafa34d0 0x55f8db372e90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 1 --2- 172.21.15.27:0/3414176780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8daf9b2f0 0x55f8db372950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 >> 172.21.15.27:0/3414176780 conn(0x55f8dafb8300 msgr2=0x55f8db368b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f363eff5640 0 client.27562 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 shutdown_connections 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 1 -- 172.21.15.27:0/3414176780 wait complete. 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.428+0000 7f3667f9f740 20 client.27562 trim_cache size 0 max 16384 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:53:03.429 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:53:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:53:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.429+0000 7f36610ee640 1 -- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36440afbe0 msgr2=0x7f36440b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.429+0000 7f36610ee640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36440afbe0 0x7f36440b20f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3654005c40 tx=0x7f365400f040 comp rx=0 tx=0).stop 2024-04-25T13:53:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.429+0000 7f36610ee640 1 -- 172.21.15.27:0/2968280965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f365c103990 msgr2=0x7f365c19c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.429+0000 7f36610ee640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f365c103990 0x7f365c19c9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f365000e9f0 tx=0x7f365000eeb0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.430+0000 7f36610ee640 1 -- 172.21.15.27:0/2968280965 shutdown_connections 2024-04-25T13:53:03.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.430+0000 7f36610ee640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f36440afbe0 0x7f36440b20f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.430+0000 7f36610ee640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f365c1044b0 0x7f365c1a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.430+0000 7f36610ee640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f365c103990 0x7f365c19c9a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.430+0000 7f36610ee640 1 --2- 172.21.15.27:0/2968280965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f365c102fb0 0x7f365c19c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.430+0000 7f36610ee640 1 -- 172.21.15.27:0/2968280965 >> 172.21.15.27:0/2968280965 conn(0x7f365c0fd390 msgr2=0x7f365c104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.430+0000 7f36610ee640 1 -- 172.21.15.27:0/2968280965 shutdown_connections 2024-04-25T13:53:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:03.430+0000 7f36610ee640 1 -- 172.21.15.27:0/2968280965 wait complete. 2024-04-25T13:53:04.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:04 smithi027 ceph-mon[29140]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.0 MiB/s wr, 801 op/s 2024-04-25T13:53:04.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:04 smithi122 ceph-mon[32843]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.0 MiB/s wr, 801 op/s 2024-04-25T13:53:04.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:04 smithi045 ceph-mon[32731]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.0 MiB/s wr, 801 op/s 2024-04-25T13:53:06.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:06 smithi027 ceph-mon[29140]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.9 MiB/s wr, 697 op/s 2024-04-25T13:53:06.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:06 smithi122 ceph-mon[32843]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.9 MiB/s wr, 697 op/s 2024-04-25T13:53:06.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:06 smithi045 ceph-mon[32731]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.9 MiB/s wr, 697 op/s 2024-04-25T13:53:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 720.0 s, 1737.8 tps, lat 57.304 ms stddev 61.795 2024-04-25T13:53:08.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:08 smithi027 ceph-mon[29140]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.9 MiB/s wr, 731 op/s 2024-04-25T13:53:08.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:08 smithi122 ceph-mon[32843]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.9 MiB/s wr, 731 op/s 2024-04-25T13:53:08.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:08 smithi045 ceph-mon[32731]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 6.9 MiB/s wr, 731 op/s 2024-04-25T13:53:10.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:10 smithi027 ceph-mon[29140]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.5 MiB/s wr, 729 op/s 2024-04-25T13:53:10.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:10 smithi122 ceph-mon[32843]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.5 MiB/s wr, 729 op/s 2024-04-25T13:53:10.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:10 smithi045 ceph-mon[32731]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.5 MiB/s wr, 729 op/s 2024-04-25T13:53:12.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:12 smithi027 ceph-mon[29140]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.0 MiB/s wr, 681 op/s 2024-04-25T13:53:12.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:12 smithi122 ceph-mon[32843]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.0 MiB/s wr, 681 op/s 2024-04-25T13:53:12.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:12 smithi045 ceph-mon[32731]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.0 MiB/s wr, 681 op/s 2024-04-25T13:53:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 725.0 s, 1681.6 tps, lat 59.705 ms stddev 70.016 2024-04-25T13:53:14.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:14 smithi027 ceph-mon[29140]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.6 MiB/s wr, 649 op/s 2024-04-25T13:53:14.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:14 smithi122 ceph-mon[32843]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.6 MiB/s wr, 649 op/s 2024-04-25T13:53:14.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:14 smithi045 ceph-mon[32731]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.6 MiB/s wr, 649 op/s 2024-04-25T13:53:16.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:16 smithi027 ceph-mon[29140]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 614 op/s 2024-04-25T13:53:16.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:16 smithi122 ceph-mon[32843]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 614 op/s 2024-04-25T13:53:16.566 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.563+0000 7f4f4214d640 1 -- 172.21.15.27:0/98843192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f3c1050e0 msgr2=0x7f4f3c1054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.566 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.563+0000 7f4f4214d640 1 --2- 172.21.15.27:0/98843192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f3c1050e0 0x7f4f3c1054f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009860 tx=0x7f4f2c02a340 comp rx=0 tx=0).stop 2024-04-25T13:53:16.566 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.563+0000 7f4f4214d640 1 -- 172.21.15.27:0/98843192 shutdown_connections 2024-04-25T13:53:16.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.563+0000 7f4f4214d640 1 --2- 172.21.15.27:0/98843192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f3c102290 0x7f4f3c10bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.563+0000 7f4f4214d640 1 --2- 172.21.15.27:0/98843192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f3c101790 0x7f4f3c101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.563+0000 7f4f4214d640 1 --2- 172.21.15.27:0/98843192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f3c1050e0 0x7f4f3c1054f0 secure :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009860 tx=0x7f4f2c02a340 comp rx=0 tx=0).stop 2024-04-25T13:53:16.567 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.563+0000 7f4f4214d640 1 -- 172.21.15.27:0/98843192 >> 172.21.15.27:0/98843192 conn(0x7f4f3c0fd390 msgr2=0x7f4f3c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.569+0000 7f4f4214d640 1 -- 172.21.15.27:0/98843192 shutdown_connections 2024-04-25T13:53:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.572+0000 7f4f4214d640 1 -- 172.21.15.27:0/98843192 wait complete. 2024-04-25T13:53:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.572+0000 7f4f4214d640 1 Processor -- start 2024-04-25T13:53:16.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:16 smithi045 ceph-mon[32731]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 614 op/s 2024-04-25T13:53:16.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.577+0000 7f4f4214d640 1 -- start start 2024-04-25T13:53:16.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.577+0000 7f4f4214d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f3c101790 0x7f4f3c113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.577+0000 7f4f4214d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f3c102290 0x7f4f3c113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.577+0000 7f4f4214d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f3c110380 0x7f4f3c110860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.577+0000 7f4f4214d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c10f630 con 0x7f4f3c110380 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.577+0000 7f4f4214d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c10f4b0 con 0x7f4f3c102290 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.577+0000 7f4f4214d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c10f7b0 con 0x7f4f3c101790 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.580+0000 7f4f3affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f3c102290 0x7f4f3c113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.581+0000 7f4f3b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f3c101790 0x7f4f3c113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.583+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f3c110380 0x7f4f3c110860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.583+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f3c110380 0x7f4f3c110860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40054/0 (socket says 172.21.15.27:40054) 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.583+0000 7f4f3bfff640 1 -- 172.21.15.27:0/2490906922 learned_addr learned my addr 172.21.15.27:0/2490906922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.585+0000 7f4f3affd640 1 -- 172.21.15.27:0/2490906922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f3c101790 msgr2=0x7f4f3c113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.585+0000 7f4f3affd640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f3c101790 0x7f4f3c113830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.585+0000 7f4f3affd640 1 -- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f3c110380 msgr2=0x7f4f3c110860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.585+0000 7f4f3affd640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f3c110380 0x7f4f3c110860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.585+0000 7f4f3affd640 1 -- 172.21.15.27:0/2490906922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c073640 con 0x7f4f3c102290 2024-04-25T13:53:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.585+0000 7f4f3affd640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f3c102290 0x7f4f3c113df0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000e790 tx=0x7f4f3000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.592+0000 7f4f38ff9640 1 -- 172.21.15.27:0/2490906922 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3000eea0 con 0x7f4f3c102290 2024-04-25T13:53:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.592+0000 7f4f38ff9640 1 -- 172.21.15.27:0/2490906922 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f3000ccc0 con 0x7f4f3c102290 2024-04-25T13:53:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.592+0000 7f4f38ff9640 1 -- 172.21.15.27:0/2490906922 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30018710 con 0x7f4f3c102290 2024-04-25T13:53:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.594+0000 7f4f4214d640 1 -- 172.21.15.27:0/2490906922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3c073930 con 0x7f4f3c102290 2024-04-25T13:53:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.594+0000 7f4f4214d640 1 -- 172.21.15.27:0/2490906922 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3c073e00 con 0x7f4f3c102290 2024-04-25T13:53:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.597+0000 7f4f49000740 1 Processor -- start 2024-04-25T13:53:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.598+0000 7f4f49000740 1 -- start start 2024-04-25T13:53:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.599+0000 7f4f49000740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 0x563b71bfa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.599+0000 7f4f49000740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718312f0 0x563b71bfaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.599+0000 7f4f49000740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c01df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.599+0000 7f4f49000740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563b71b60460 con 0x563b718312f0 2024-04-25T13:53:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.599+0000 7f4f49000740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563b71b1a590 con 0x563b718394d0 2024-04-25T13:53:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.599+0000 7f4f49000740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563b71b5c420 con 0x563b7185a1b0 2024-04-25T13:53:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.601+0000 7f4f3affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c01df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.602+0000 7f4f3affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c01df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37784/0 (socket says 172.21.15.27:37784) 2024-04-25T13:53:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.602+0000 7f4f3affd640 1 -- 172.21.15.27:0/3878803435 learned_addr learned my addr 172.21.15.27:0/3878803435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.603+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3878803435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718312f0 0x563b71bfaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.605+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3878803435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 msgr2=0x563b71bfa970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:53:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.605+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3878803435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 0x563b71bfa970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.605+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3878803435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 msgr2=0x563b71c01df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.605+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3878803435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c01df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.605+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3878803435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b71b270d0 con 0x563b718312f0 2024-04-25T13:53:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.608+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3878803435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718312f0 0x563b71bfaef0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c069c90 tx=0x7f4f2c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e5580670598d26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.609+0000 7f4f38ff9640 1 -- 172.21.15.27:0/2490906922 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4f30010430 con 0x7f4f3c102290 2024-04-25T13:53:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.610+0000 7f4f38ff9640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f240afcb0 0x7f4f240b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.610 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.610+0000 7f4f38ff9640 1 -- 172.21.15.27:0/2490906922 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4f300dee60 con 0x7f4f3c102290 2024-04-25T13:53:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.612+0000 7f4f21ffb640 1 -- 172.21.15.27:0/3878803435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c040020 con 0x563b718312f0 2024-04-25T13:53:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.612+0000 7f4f21ffb640 1 -- 172.21.15.27:0/3878803435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f2c038040 con 0x563b718312f0 2024-04-25T13:53:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.613+0000 7f4f49000740 1 -- 172.21.15.27:0/3878803435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718312f0 msgr2=0x563b71bfaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.613+0000 7f4f49000740 1 --2- 172.21.15.27:0/3878803435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718312f0 0x563b71bfaef0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c069c90 tx=0x7f4f2c005dd0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.613+0000 7f4f3bfff640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f240afcb0 0x7f4f240b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.618+0000 7f4f3bfff640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f240afcb0 0x7f4f240b21c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x563b71bfbb40 tx=0x7f4f34009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.618+0000 7f4f49000740 1 -- 172.21.15.27:0/3878803435 shutdown_connections 2024-04-25T13:53:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.618+0000 7f4f49000740 1 --2- 172.21.15.27:0/3878803435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c01df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.618+0000 7f4f49000740 1 --2- 172.21.15.27:0/3878803435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718312f0 0x563b71bfaef0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.618+0000 7f4f49000740 1 --2- 172.21.15.27:0/3878803435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 0x563b71bfa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.618+0000 7f4f49000740 1 -- 172.21.15.27:0/3878803435 >> 172.21.15.27:0/3878803435 conn(0x563b7184e300 msgr2=0x563b71bf6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.620+0000 7f4f49000740 1 -- 172.21.15.27:0/3878803435 shutdown_connections 2024-04-25T13:53:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.620+0000 7f4f49000740 1 -- 172.21.15.27:0/3878803435 wait complete. 2024-04-25T13:53:16.623 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.621+0000 7f4f49000740 1 -- start start 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.621+0000 7f4f49000740 1 Processor -- start 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.622+0000 7f4f49000740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718312f0 0x563b71c04650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.622+0000 7f4f49000740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c04b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.622+0000 7f4f49000740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c0baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.622+0000 7f4f3b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718312f0 0x563b71c04650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.622+0000 7f4f3b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718312f0 0x563b71c04650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:50374/0 (socket says 172.21.15.27:50374) 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.622+0000 7f4f3affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c04b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.622+0000 7f4f3affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c04b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:37792/0 (socket says 172.21.15.27:37792) 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3584886531 learned_addr learned my addr 172.21.15.27:0/3584886531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563b71c02830 con 0x563b7185a1b0 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563b71ae7b60 con 0x563b718394d0 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563b71c029b0 con 0x563b718312f0 2024-04-25T13:53:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f3bfff640 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c0baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f49000740 10 client.0 _ll_register_callbacks cb 0x563b71b0c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3584886531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 msgr2=0x563b71c04b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c04b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3584886531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 msgr2=0x563b71c0baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c0baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3584886531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b71872210 con 0x563b718312f0 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.623+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718312f0 0x563b71c04650 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c000c00 tx=0x7f4f2c003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4638a21b4753fb6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.624+0000 7f4f1affd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.624+0000 7f4f1affd640 1 -- 172.21.15.27:0/3584886531 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c040040 con 0x563b718312f0 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.624+0000 7f4f1affd640 1 -- 172.21.15.27:0/3584886531 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4f2c003da0 con 0x563b718312f0 2024-04-25T13:53:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.624+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f2c038040 con 0x563b718312f0 2024-04-25T13:53:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.624+0000 7f4f1affd640 1 -- 172.21.15.27:0/3584886531 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4f2c030550 con 0x563b718312f0 2024-04-25T13:53:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.625+0000 7f4f49000740 10 client.27568 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.625+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563b7193e620 con 0x563b718312f0 2024-04-25T13:53:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.627+0000 7f4f1affd640 1 -- 172.21.15.27:0/3584886531 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f2c038040 con 0x563b718312f0 2024-04-25T13:53:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.628+0000 7f4f1affd640 1 -- 172.21.15.27:0/3584886531 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f4f2c032860 con 0x563b718312f0 2024-04-25T13:53:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.630+0000 7f4f49000740 10 client.27568 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.630+0000 7f4f49000740 10 client.27568 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.630+0000 7f4f49000740 20 client.27568 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.630+0000 7f4f49000740 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563b71818d80 0x563b71c33f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.630+0000 7f4f49000740 4 client.27568 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.630+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b71b6e070 con 0x563b71818d80 2024-04-25T13:53:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.633+0000 7f4f3affd640 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563b71818d80 0x563b71c33f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.639+0000 7f4f3affd640 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563b71818d80 0x563b71c33f10 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c14cfc744573ac4c server_cookie=eb9f148f0384e229 in_seq=0 out_seq=0 2024-04-25T13:53:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.640+0000 7f4f1affd640 10 client.27568 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.640+0000 7f4f1affd640 1 -- 172.21.15.27:0/3584886531 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563b71b6e070 con 0x563b71818d80 2024-04-25T13:53:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.640+0000 7f4f1affd640 10 client.27568 handle_command_reply: tid=0 2024-04-25T13:53:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.640+0000 7f4f49000740 1 client.27568 shutdown 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718312f0 msgr2=0x563b71c04650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f49000740 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718312f0 0x563b71c04650 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c000c00 tx=0x7f4f2c003820 comp rx=0 tx=0).stop 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 shutdown_connections 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f49000740 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563b71818d80 0x563b71c33f10 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f49000740 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c0baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f49000740 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b718394d0 0x563b71c04b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f49000740 1 --2- 172.21.15.27:0/3584886531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718312f0 0x563b71c04650 secure :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c000c00 tx=0x7f4f2c003820 comp rx=0 tx=0).stop 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 >> 172.21.15.27:0/3584886531 conn(0x563b7184e300 msgr2=0x563b71bf9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.641+0000 7f4f1affd640 0 client.27568 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.642+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 shutdown_connections 2024-04-25T13:53:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.644+0000 7f4f49000740 1 -- 172.21.15.27:0/3584886531 wait complete. 2024-04-25T13:53:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.645+0000 7f4f49000740 20 client.27568 trim_cache size 0 max 16384 2024-04-25T13:53:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.671+0000 7f4f49000740 1 Processor -- start 2024-04-25T13:53:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.671+0000 7f4f49000740 1 -- start start 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.671+0000 7f4f49000740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718394d0 0x563b71bf9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.671+0000 7f4f49000740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 0x563b71bf9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.671+0000 7f4f49000740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71bf5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.671+0000 7f4f49000740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2800c410 con 0x563b718394d0 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.671+0000 7f4f49000740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2800ebe0 con 0x563b717ee1c0 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.671+0000 7f4f49000740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2800c9d0 con 0x563b7185a1b0 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718394d0 0x563b71bf9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718394d0 0x563b71bf9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40084/0 (socket says 172.21.15.27:40084) 2024-04-25T13:53:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/2499151042 learned_addr learned my addr 172.21.15.27:0/2499151042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3bfff640 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71bf5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3affd640 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 0x563b71bf9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/2499151042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 msgr2=0x563b71bf9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 0x563b71bf9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/2499151042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 msgr2=0x563b71bf5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71bf5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/2499151042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b71818fb0 con 0x563b718394d0 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718394d0 0x563b71bf9100 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c032da0 tx=0x7f4f2c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e597ecb11c12a1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.672+0000 7f4f217fa640 1 -- 172.21.15.27:0/2499151042 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c040070 con 0x563b718394d0 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.673+0000 7f4f217fa640 1 -- 172.21.15.27:0/2499151042 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f2c04a070 con 0x563b718394d0 2024-04-25T13:53:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.673+0000 7f4f217fa640 1 -- 172.21.15.27:0/2499151042 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c003da0 con 0x563b718394d0 2024-04-25T13:53:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.673+0000 7f4f49000740 1 -- 172.21.15.27:0/2499151042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718394d0 msgr2=0x563b71bf9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.673+0000 7f4f49000740 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718394d0 0x563b71bf9100 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c032da0 tx=0x7f4f2c001e50 comp rx=0 tx=0).stop 2024-04-25T13:53:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.673+0000 7f4f49000740 1 -- 172.21.15.27:0/2499151042 shutdown_connections 2024-04-25T13:53:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.673+0000 7f4f49000740 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71bf5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.673+0000 7f4f49000740 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b7185a1b0 0x563b71bf9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.673+0000 7f4f49000740 1 --2- 172.21.15.27:0/2499151042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b718394d0 0x563b71bf9100 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.674+0000 7f4f49000740 1 -- 172.21.15.27:0/2499151042 >> 172.21.15.27:0/2499151042 conn(0x563b7184e300 msgr2=0x563b71bfea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.674+0000 7f4f49000740 1 -- 172.21.15.27:0/2499151042 shutdown_connections 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.674+0000 7f4f49000740 1 -- 172.21.15.27:0/2499151042 wait complete. 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.674+0000 7f4f49000740 1 -- start start 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.674+0000 7f4f49000740 1 Processor -- start 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f49000740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71c07f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f49000740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718394d0 0x563b71c08470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f49000740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c9a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f49000740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2800d0a0 con 0x563b7185a1b0 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f49000740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2800e510 con 0x563b717ee1c0 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f49000740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2800d770 con 0x563b718394d0 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f49000740 10 client.0 _ll_register_callbacks cb 0x7f4f30005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c9a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c9a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40100/0 (socket says 172.21.15.27:40100) 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3bfff640 1 -- 172.21.15.27:0/3706532106 learned_addr learned my addr 172.21.15.27:0/3706532106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71c07f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3affd640 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718394d0 0x563b71c08470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3706532106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718394d0 msgr2=0x563b71c08470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718394d0 0x563b71c08470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3706532106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 msgr2=0x563b71c9a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c9a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.675+0000 7f4f3b7fe640 1 -- 172.21.15.27:0/3706532106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b71c0b430 con 0x563b717ee1c0 2024-04-25T13:53:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.676+0000 7f4f3affd640 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718394d0 0x563b71c08470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.676+0000 7f4f3b7fe640 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71c07f30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x563b71845ac0 tx=0x7f4f2c003fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb51fb26993b6234 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.676+0000 7f4f1affd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.676+0000 7f4f1affd640 1 -- 172.21.15.27:0/3706532106 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c05f020 con 0x563b717ee1c0 2024-04-25T13:53:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.676+0000 7f4f1affd640 1 -- 172.21.15.27:0/3706532106 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f4f2c045750 con 0x563b717ee1c0 2024-04-25T13:53:16.679 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.677+0000 7f4f49000740 1 -- 172.21.15.27:0/3706532106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f2c04a070 con 0x563b717ee1c0 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.677+0000 7f4f1affd640 1 -- 172.21.15.27:0/3706532106 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4f2c040040 con 0x563b717ee1c0 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.677+0000 7f4f49000740 10 client.27542 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.677+0000 7f4f49000740 1 -- 172.21.15.27:0/3706532106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563b71c9b650 con 0x563b717ee1c0 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.677+0000 7f4f1affd640 1 -- 172.21.15.27:0/3706532106 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f2c04a070 con 0x563b717ee1c0 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f1affd640 1 -- 172.21.15.27:0/3706532106 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f4f2c059070 con 0x563b717ee1c0 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f49000740 10 client.27542 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f49000740 10 client.27542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f49000740 20 client.27542 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f49000740 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563b71c9be10 0x563b71c9e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f49000740 4 client.27542 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f49000740 1 -- 172.21.15.27:0/3706532106 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563b71b67ee0 con 0x563b71c9be10 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f3affd640 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563b71c9be10 0x563b71c9e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f3affd640 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563b71c9be10 0x563b71c9e230 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=644d79c0a3b46246 server_cookie=2b6daf322e5bea4c in_seq=0 out_seq=0 2024-04-25T13:53:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.678+0000 7f4f1affd640 10 client.27542 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.702+0000 7f4f1affd640 1 -- 172.21.15.27:0/3706532106 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563b71b67ee0 con 0x563b71c9be10 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.702+0000 7f4f1affd640 10 client.27542 handle_command_reply: tid=0 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.702+0000 7f4f49000740 1 client.27542 shutdown 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 -- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 msgr2=0x563b71c07f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71c07f30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x563b71845ac0 tx=0x7f4f2c003fd0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 -- 172.21.15.27:0/3706532106 shutdown_connections 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563b71c9be10 0x563b71c9e230 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563b7185a1b0 0x563b71c9a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563b718394d0 0x563b71c08470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 --2- 172.21.15.27:0/3706532106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563b717ee1c0 0x563b71c07f30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 -- 172.21.15.27:0/3706532106 >> 172.21.15.27:0/3706532106 conn(0x563b7184e300 msgr2=0x563b71bf9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f1affd640 0 client.27542 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 -- 172.21.15.27:0/3706532106 shutdown_connections 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 1 -- 172.21.15.27:0/3706532106 wait complete. 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.703+0000 7f4f49000740 20 client.27542 trim_cache size 0 max 16384 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:53:16.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 -- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f240afcb0 msgr2=0x7f4f240b21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f240afcb0 0x7f4f240b21c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x563b71bfbb40 tx=0x7f4f34009500 comp rx=0 tx=0).stop 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 -- 172.21.15.27:0/2490906922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f3c102290 msgr2=0x7f4f3c113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f3c102290 0x7f4f3c113df0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000e790 tx=0x7f4f3000ec50 comp rx=0 tx=0).stop 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 -- 172.21.15.27:0/2490906922 shutdown_connections 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f4f240afcb0 0x7f4f240b21c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f3c110380 0x7f4f3c110860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f3c102290 0x7f4f3c113df0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 --2- 172.21.15.27:0/2490906922 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f3c101790 0x7f4f3c113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 -- 172.21.15.27:0/2490906922 >> 172.21.15.27:0/2490906922 conn(0x7f4f3c0fd390 msgr2=0x7f4f3c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 -- 172.21.15.27:0/2490906922 shutdown_connections 2024-04-25T13:53:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:16.704+0000 7f4f4214d640 1 -- 172.21.15.27:0/2490906922 wait complete. 2024-04-25T13:53:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 730.0 s, 1773.2 tps, lat 56.496 ms stddev 67.113 2024-04-25T13:53:18.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:18 smithi027 ceph-mon[29140]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.9 MiB/s wr, 633 op/s 2024-04-25T13:53:18.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:18 smithi122 ceph-mon[32843]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.9 MiB/s wr, 633 op/s 2024-04-25T13:53:18.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:18 smithi045 ceph-mon[32731]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.9 MiB/s wr, 633 op/s 2024-04-25T13:53:20.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:20 smithi027 ceph-mon[29140]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.8 MiB/s wr, 608 op/s 2024-04-25T13:53:20.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:20 smithi122 ceph-mon[32843]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.8 MiB/s wr, 608 op/s 2024-04-25T13:53:20.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:20 smithi045 ceph-mon[32731]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.8 MiB/s wr, 608 op/s 2024-04-25T13:53:22.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:22 smithi027 ceph-mon[29140]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.9 MiB/s wr, 625 op/s 2024-04-25T13:53:22.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:22 smithi122 ceph-mon[32843]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.9 MiB/s wr, 625 op/s 2024-04-25T13:53:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:22 smithi045 ceph-mon[32731]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.9 MiB/s wr, 625 op/s 2024-04-25T13:53:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 735.0 s, 2049.2 tps, lat 48.977 ms stddev 56.193 2024-04-25T13:53:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:24 smithi027 ceph-mon[29140]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.6 MiB/s wr, 618 op/s 2024-04-25T13:53:24.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:24 smithi122 ceph-mon[32843]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.6 MiB/s wr, 618 op/s 2024-04-25T13:53:24.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:24 smithi045 ceph-mon[32731]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5.6 MiB/s wr, 618 op/s 2024-04-25T13:53:25.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:53:25.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:53:25.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 -- 172.21.15.27:0/576146728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4109770 msgr2=0x7f05d410bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:25.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 --2- 172.21.15.27:0/576146728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4109770 0x7f05d410bc80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009900 tx=0x7f05bc02a410 comp rx=0 tx=0).stop 2024-04-25T13:53:25.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 -- 172.21.15.27:0/576146728 shutdown_connections 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 --2- 172.21.15.27:0/576146728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4109770 0x7f05d410bc80 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 --2- 172.21.15.27:0/576146728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05d4106c60 0x7f05d41090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 --2- 172.21.15.27:0/576146728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d4104270 0x7f05d4106690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 -- 172.21.15.27:0/576146728 >> 172.21.15.27:0/576146728 conn(0x7f05d40fdbd0 msgr2=0x7f05d4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 -- 172.21.15.27:0/576146728 shutdown_connections 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.952+0000 7f05dab40640 1 -- 172.21.15.27:0/576146728 wait complete. 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.953+0000 7f05dab40640 1 Processor -- start 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.953+0000 7f05dab40640 1 -- start start 2024-04-25T13:53:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05dab40640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4104270 0x7f05d419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05dab40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05d4106c60 0x7f05d419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05dab40640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d4109770 0x7f05d41a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05dab40640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f05d410f600 con 0x7f05d4104270 2024-04-25T13:53:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05dab40640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f05d410f480 con 0x7f05d4109770 2024-04-25T13:53:25.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05dab40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f05d410f780 con 0x7f05d4106c60 2024-04-25T13:53:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d88dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4104270 0x7f05d419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d90dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d4109770 0x7f05d41a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d88dc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4104270 0x7f05d419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45488/0 (socket says 172.21.15.27:45488) 2024-04-25T13:53:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d88dc640 1 -- 172.21.15.27:0/995529743 learned_addr learned my addr 172.21.15.27:0/995529743 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d3fff640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05d4106c60 0x7f05d419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d90dd640 1 -- 172.21.15.27:0/995529743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05d4106c60 msgr2=0x7f05d419aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d90dd640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05d4106c60 0x7f05d419aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d90dd640 1 -- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4104270 msgr2=0x7f05d419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d90dd640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4104270 0x7f05d419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d90dd640 1 -- 172.21.15.27:0/995529743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d406c6f0 con 0x7f05d4109770 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.954+0000 7f05d88dc640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4104270 0x7f05d419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.955+0000 7f05d90dd640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d4109770 0x7f05d41a1990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009230 tx=0x7f05bc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.955+0000 7f05d1ffb640 1 -- 172.21.15.27:0/995529743 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc047030 con 0x7f05d4109770 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.955+0000 7f05d1ffb640 1 -- 172.21.15.27:0/995529743 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05bc031910 con 0x7f05d4109770 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.955+0000 7f05dab40640 1 -- 172.21.15.27:0/995529743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d4102060 con 0x7f05d4109770 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.955+0000 7f05d1ffb640 1 -- 172.21.15.27:0/995529743 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc035070 con 0x7f05d4109770 2024-04-25T13:53:25.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.955+0000 7f05dab40640 1 -- 172.21.15.27:0/995529743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d4102550 con 0x7f05d4109770 2024-04-25T13:53:25.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05e19f1740 1 Processor -- start 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05d1ffb640 1 -- 172.21.15.27:0/995529743 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f05bc04a030 con 0x7f05d4109770 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05e19f1740 1 -- start start 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a0da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a0dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e3a0ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05e19f1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5582e3961d80 con 0x5582e363a6b0 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05e19f1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5582e391b8b0 con 0x5582e36324d0 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.958+0000 7f05e19f1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5582e395dd40 con 0x5582e365b390 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a0da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a0da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51734/0 (socket says 172.21.15.27:51734) 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 -- 172.21.15.27:0/1436165075 learned_addr learned my addr 172.21.15.27:0/1436165075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e3a0ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d88dc640 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a0dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 -- 172.21.15.27:0/1436165075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 msgr2=0x5582e3a0dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:25.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a0dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 -- 172.21.15.27:0/1436165075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 msgr2=0x5582e3a0ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:25.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e3a0ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d3fff640 1 -- 172.21.15.27:0/1436165075 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582e38eca40 con 0x5582e365b390 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.959+0000 7f05d1ffb640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f05c00afb70 0x7f05c00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.960+0000 7f05d1ffb640 1 -- 172.21.15.27:0/995529743 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f05bc1075f0 con 0x7f05d4109770 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.960+0000 7f05d3fff640 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a0da70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f05cc012b00 tx=0x7f05cc009a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8275465e38eac288 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.960+0000 7f05d88dc640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f05c00afb70 0x7f05c00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.960+0000 7f05caffd640 1 -- 172.21.15.27:0/1436165075 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc003970 con 0x5582e365b390 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.960+0000 7f05caffd640 1 -- 172.21.15.27:0/1436165075 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05cc010890 con 0x5582e365b390 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.960+0000 7f05caffd640 1 -- 172.21.15.27:0/1436165075 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0044e0 con 0x5582e365b390 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 -- 172.21.15.27:0/1436165075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 msgr2=0x5582e3a0da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05d88dc640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f05c00afb70 0x7f05c00b2080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x5582e3a0d790 tx=0x7f05c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a0da70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f05cc012b00 tx=0x7f05cc009a80 comp rx=0 tx=0).stop 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 -- 172.21.15.27:0/1436165075 shutdown_connections 2024-04-25T13:53:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e3a0ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a0dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 --2- 172.21.15.27:0/1436165075 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a0da70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 -- 172.21.15.27:0/1436165075 >> 172.21.15.27:0/1436165075 conn(0x5582e364f4e0 msgr2=0x5582e39f8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 -- 172.21.15.27:0/1436165075 shutdown_connections 2024-04-25T13:53:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.961+0000 7f05e19f1740 1 -- 172.21.15.27:0/1436165075 wait complete. 2024-04-25T13:53:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 1 -- start start 2024-04-25T13:53:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 1 Processor -- start 2024-04-25T13:53:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a17ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e363a6b0 0x5582e3a18010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e365b390 0x5582e3a1ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5582e3a0f7e0 con 0x5582e365b390 2024-04-25T13:53:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5582e38e8d10 con 0x5582e36324d0 2024-04-25T13:53:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5582e3a0f960 con 0x5582e363a6b0 2024-04-25T13:53:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05e19f1740 10 client.0 _ll_register_callbacks cb 0x5582e3915210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05d88dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a17ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e363a6b0 0x5582e3a18010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.962+0000 7f05d90dd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e365b390 0x5582e3a1ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d88dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a17ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60462/0 (socket says 172.21.15.27:60462) 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d88dc640 1 -- 172.21.15.27:0/4195781673 learned_addr learned my addr 172.21.15.27:0/4195781673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d88dc640 1 -- 172.21.15.27:0/4195781673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e363a6b0 msgr2=0x5582e3a18010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d88dc640 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e363a6b0 0x5582e3a18010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d88dc640 1 -- 172.21.15.27:0/4195781673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e365b390 msgr2=0x5582e3a1ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d88dc640 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e365b390 0x5582e3a1ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d88dc640 1 -- 172.21.15.27:0/4195781673 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582e373d7d0 con 0x5582e36324d0 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.963+0000 7f05d88dc640 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a17ad0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f05c400f850 tx=0x7f05c4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4861d3bcfbcedb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.964+0000 7f05b3fff640 1 -- 172.21.15.27:0/4195781673 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4011940 con 0x5582e36324d0 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.964+0000 7f05b3fff640 1 -- 172.21.15.27:0/4195781673 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f05c4011ae0 con 0x5582e36324d0 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.964+0000 7f05e19f1740 1 -- 172.21.15.27:0/4195781673 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05cc010890 con 0x5582e36324d0 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.964+0000 7f05b3fff640 1 -- 172.21.15.27:0/4195781673 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f05c4011ce0 con 0x5582e36324d0 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.964+0000 7f05e19f1740 10 client.27554 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.964+0000 7f05e19f1740 1 -- 172.21.15.27:0/4195781673 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5582e36733f0 con 0x5582e36324d0 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05b3fff640 1 -- 172.21.15.27:0/4195781673 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05cc010890 con 0x5582e36324d0 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05b3fff640 1 -- 172.21.15.27:0/4195781673 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f05c4027070 con 0x5582e36324d0 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05e19f1740 10 client.27554 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05e19f1740 10 client.27554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05e19f1740 20 client.27554 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05e19f1740 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5582e3619f60 0x5582e3a25530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05e19f1740 4 client.27554 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05e19f1740 1 -- 172.21.15.27:0/4195781673 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582e396f990 con 0x5582e3619f60 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.965+0000 7f05d3fff640 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5582e3619f60 0x5582e3a25530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.966+0000 7f05b3fff640 10 client.27554 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.966+0000 7f05d3fff640 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5582e3619f60 0x5582e3a25530 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e9cfd832f1cfa29 server_cookie=bff55b110a8919a7 in_seq=0 out_seq=0 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.966+0000 7f05b3fff640 1 -- 172.21.15.27:0/4195781673 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5582e396f990 con 0x5582e3619f60 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.966+0000 7f05b3fff640 10 client.27554 handle_command_reply: tid=0 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.966+0000 7f05e19f1740 1 client.27554 shutdown 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 -- 172.21.15.27:0/4195781673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 msgr2=0x5582e3a17ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a17ad0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f05c400f850 tx=0x7f05c4004730 comp rx=0 tx=0).stop 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 -- 172.21.15.27:0/4195781673 shutdown_connections 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5582e3619f60 0x5582e3a25530 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e365b390 0x5582e3a1ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e363a6b0 0x5582e3a18010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 --2- 172.21.15.27:0/4195781673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a17ad0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05b3fff640 0 client.27554 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 -- 172.21.15.27:0/4195781673 >> 172.21.15.27:0/4195781673 conn(0x5582e364f4e0 msgr2=0x5582e39f73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 -- 172.21.15.27:0/4195781673 shutdown_connections 2024-04-25T13:53:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 1 -- 172.21.15.27:0/4195781673 wait complete. 2024-04-25T13:53:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:25.967+0000 7f05e19f1740 20 client.27554 trim_cache size 0 max 16384 2024-04-25T13:53:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.009+0000 7f05e19f1740 1 Processor -- start 2024-04-25T13:53:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.009+0000 7f05e19f1740 1 -- start start 2024-04-25T13:53:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.010+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a16760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.010+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e39fb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.010+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a2ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.010+0000 7f05e19f1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4071600 con 0x5582e363a6b0 2024-04-25T13:53:26.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.010+0000 7f05e19f1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4073e80 con 0x5582e36324d0 2024-04-25T13:53:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.010+0000 7f05e19f1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4071c70 con 0x5582e365b390 2024-04-25T13:53:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.014+0000 7f05d90dd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a2ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.014+0000 7f05d90dd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a2ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51748/0 (socket says 172.21.15.27:51748) 2024-04-25T13:53:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.014+0000 7f05d90dd640 1 -- 172.21.15.27:0/1284073452 learned_addr learned my addr 172.21.15.27:0/1284073452 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.015+0000 7f05d88dc640 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a16760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.016+0000 7f05d3fff640 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e39fb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.016+0000 7f05d88dc640 1 -- 172.21.15.27:0/1284073452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 msgr2=0x5582e3a2ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.016+0000 7f05d88dc640 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a2ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.016+0000 7f05d88dc640 1 -- 172.21.15.27:0/1284073452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 msgr2=0x5582e39fb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.017+0000 7f05d88dc640 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e39fb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.017+0000 7f05d88dc640 1 -- 172.21.15.27:0/1284073452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582e36330f0 con 0x5582e36324d0 2024-04-25T13:53:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.017+0000 7f05d88dc640 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a16760 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f05c401f5a0 tx=0x7f05c4011f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14b65b3d4d672f2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.020+0000 7f05ca7fc640 1 -- 172.21.15.27:0/1284073452 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4011a10 con 0x5582e36324d0 2024-04-25T13:53:26.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.020+0000 7f05ca7fc640 1 -- 172.21.15.27:0/1284073452 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c4005020 con 0x5582e36324d0 2024-04-25T13:53:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.021+0000 7f05e19f1740 1 -- 172.21.15.27:0/1284073452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 msgr2=0x5582e3a16760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.021+0000 7f05e19f1740 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a16760 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f05c401f5a0 tx=0x7f05c4011f00 comp rx=0 tx=0).stop 2024-04-25T13:53:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.025+0000 7f05e19f1740 1 -- 172.21.15.27:0/1284073452 shutdown_connections 2024-04-25T13:53:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.025+0000 7f05e19f1740 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a2ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.025+0000 7f05e19f1740 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e39fb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.025+0000 7f05e19f1740 1 --2- 172.21.15.27:0/1284073452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a16760 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.025+0000 7f05e19f1740 1 -- 172.21.15.27:0/1284073452 >> 172.21.15.27:0/1284073452 conn(0x5582e364f4e0 msgr2=0x5582e39fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.029+0000 7f05e19f1740 1 -- 172.21.15.27:0/1284073452 shutdown_connections 2024-04-25T13:53:26.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.033+0000 7f05e19f1740 1 -- 172.21.15.27:0/1284073452 wait complete. 2024-04-25T13:53:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.034+0000 7f05e19f1740 1 -- start start 2024-04-25T13:53:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.034+0000 7f05e19f1740 1 Processor -- start 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a1e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e3a1e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05e19f1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a1ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05d88dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a1e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05d88dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a1e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60472/0 (socket says 172.21.15.27:60472) 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05d88dc640 1 -- 172.21.15.27:0/3430990273 learned_addr learned my addr 172.21.15.27:0/3430990273 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4072340 con 0x5582e363a6b0 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f05d40737b0 con 0x5582e36324d0 2024-04-25T13:53:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.035+0000 7f05d90dd640 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a1ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.036+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4072a10 con 0x5582e365b390 2024-04-25T13:53:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.036+0000 7f05e19f1740 10 client.0 _ll_register_callbacks cb 0x7f05c4009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.036+0000 7f05d88dc640 1 -- 172.21.15.27:0/3430990273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 msgr2=0x5582e3a1ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.036+0000 7f05d88dc640 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a1ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.036+0000 7f05d88dc640 1 -- 172.21.15.27:0/3430990273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 msgr2=0x5582e3a1e590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:53:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.036+0000 7f05d88dc640 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e3a1e590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.036+0000 7f05d88dc640 1 -- 172.21.15.27:0/3430990273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582e39224b0 con 0x5582e36324d0 2024-04-25T13:53:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.036+0000 7f05d88dc640 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a1e050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f05c40058c0 tx=0x7f05c4004510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a94aa96e271a0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.037+0000 7f05b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.039+0000 7f05b3fff640 1 -- 172.21.15.27:0/3430990273 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c401f9a0 con 0x5582e36324d0 2024-04-25T13:53:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.039+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5582e3969800 con 0x5582e36324d0 2024-04-25T13:53:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.039+0000 7f05b3fff640 1 -- 172.21.15.27:0/3430990273 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f05c401fb40 con 0x5582e36324d0 2024-04-25T13:53:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.043+0000 7f05b3fff640 1 -- 172.21.15.27:0/3430990273 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f05c401fd40 con 0x5582e36324d0 2024-04-25T13:53:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.045+0000 7f05e19f1740 10 client.27560 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.045+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5582e3a8ca90 con 0x5582e36324d0 2024-04-25T13:53:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.048+0000 7f05b3fff640 1 -- 172.21.15.27:0/3430990273 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5582e3969800 con 0x5582e36324d0 2024-04-25T13:53:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.048+0000 7f05b3fff640 1 -- 172.21.15.27:0/3430990273 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f05c4027050 con 0x5582e36324d0 2024-04-25T13:53:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.050+0000 7f05e19f1740 10 client.27560 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.050+0000 7f05e19f1740 10 client.27560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.050+0000 7f05e19f1740 20 client.27560 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.050+0000 7f05e19f1740 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5582e35ef3a0 0x5582e3a8f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.050+0000 7f05e19f1740 4 client.27560 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:53:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.050+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5582e38e44a0 con 0x5582e35ef3a0 2024-04-25T13:53:26.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.051+0000 7f05d3fff640 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5582e35ef3a0 0x5582e3a8f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.054+0000 7f05d3fff640 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5582e35ef3a0 0x5582e3a8f360 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=653828b0ff12552 server_cookie=8d969f85812dc9de in_seq=0 out_seq=0 2024-04-25T13:53:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.054+0000 7f05b3fff640 10 client.27560 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.054+0000 7f05b3fff640 1 -- 172.21.15.27:0/3430990273 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5582e38e44a0 con 0x5582e35ef3a0 2024-04-25T13:53:26.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.054+0000 7f05b3fff640 10 client.27560 handle_command_reply: tid=0 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.055+0000 7f05e19f1740 1 client.27560 shutdown 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.055+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 msgr2=0x5582e3a1e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.055+0000 7f05e19f1740 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a1e050 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f05c40058c0 tx=0x7f05c4004510 comp rx=0 tx=0).stop 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 shutdown_connections 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05e19f1740 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5582e35ef3a0 0x5582e3a8f360 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05e19f1740 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5582e365b390 0x5582e3a1ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05e19f1740 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5582e363a6b0 0x5582e3a1e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05e19f1740 1 --2- 172.21.15.27:0/3430990273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582e36324d0 0x5582e3a1e050 secure :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f05c40058c0 tx=0x7f05c4004510 comp rx=0 tx=0).stop 2024-04-25T13:53:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 >> 172.21.15.27:0/3430990273 conn(0x5582e364f4e0 msgr2=0x5582e39fbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05b3fff640 0 client.27560 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:26.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 shutdown_connections 2024-04-25T13:53:26.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.056+0000 7f05e19f1740 1 -- 172.21.15.27:0/3430990273 wait complete. 2024-04-25T13:53:26.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.059+0000 7f05e19f1740 20 client.27560 trim_cache size 0 max 16384 2024-04-25T13:53:26.060 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:53:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:53:26.060 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:53:26.060 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:53:26.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 -- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f05c00afb70 msgr2=0x7f05c00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f05c00afb70 0x7f05c00b2080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x5582e3a0d790 tx=0x7f05c4005f90 comp rx=0 tx=0).stop 2024-04-25T13:53:26.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 -- 172.21.15.27:0/995529743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d4109770 msgr2=0x7f05d41a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d4109770 0x7f05d41a1990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009230 tx=0x7f05bc0042f0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 -- 172.21.15.27:0/995529743 shutdown_connections 2024-04-25T13:53:26.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f05c00afb70 0x7f05c00b2080 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d4109770 0x7f05d41a1990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05d4106c60 0x7f05d419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 --2- 172.21.15.27:0/995529743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05d4104270 0x7f05d419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.061+0000 7f05dab40640 1 -- 172.21.15.27:0/995529743 >> 172.21.15.27:0/995529743 conn(0x7f05d40fdbd0 msgr2=0x7f05d4105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.062+0000 7f05dab40640 1 -- 172.21.15.27:0/995529743 shutdown_connections 2024-04-25T13:53:26.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.062+0000 7f05dab40640 1 -- 172.21.15.27:0/995529743 wait complete. 2024-04-25T13:53:26.079 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:53:26.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:53:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.235+0000 7fee9a935640 1 -- 172.21.15.27:0/1782711582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee9410a420 msgr2=0x7fee9410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.235+0000 7fee9a935640 1 --2- 172.21.15.27:0/1782711582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee9410a420 0x7fee9410add0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fee8c009ca0 tx=0x7fee8c02a8f0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.241+0000 7fee9a935640 1 -- 172.21.15.27:0/1782711582 shutdown_connections 2024-04-25T13:53:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.241+0000 7fee9a935640 1 --2- 172.21.15.27:0/1782711582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee94110030 0x7fee94110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.241+0000 7fee9a935640 1 --2- 172.21.15.27:0/1782711582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee9410b310 0x7fee9410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.241+0000 7fee9a935640 1 --2- 172.21.15.27:0/1782711582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee9410a420 0x7fee9410add0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.241+0000 7fee9a935640 1 -- 172.21.15.27:0/1782711582 >> 172.21.15.27:0/1782711582 conn(0x7fee940704e0 msgr2=0x7fee94072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.242+0000 7fee9a935640 1 -- 172.21.15.27:0/1782711582 shutdown_connections 2024-04-25T13:53:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.242+0000 7fee9a935640 1 -- 172.21.15.27:0/1782711582 wait complete. 2024-04-25T13:53:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee9a935640 1 Processor -- start 2024-04-25T13:53:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee9a935640 1 -- start start 2024-04-25T13:53:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee9a935640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee9410a420 0x7fee941a9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee9a935640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee9410b310 0x7fee941a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee9a935640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee94110030 0x7fee941b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee9a935640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee9411e520 con 0x7fee94110030 2024-04-25T13:53:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee9a935640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fee9411e3a0 con 0x7fee9410b310 2024-04-25T13:53:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee9a935640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fee9411e6a0 con 0x7fee9410a420 2024-04-25T13:53:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee93fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee9410a420 0x7fee941a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee93fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee9410a420 0x7fee941a9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51776/0 (socket says 172.21.15.27:51776) 2024-04-25T13:53:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.243+0000 7fee93fff640 1 -- 172.21.15.27:0/18661939 learned_addr learned my addr 172.21.15.27:0/18661939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.244+0000 7fee937fe640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee9410b310 0x7fee941a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.244+0000 7fee98ed2640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee94110030 0x7fee941b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.244+0000 7fee98ed2640 1 -- 172.21.15.27:0/18661939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee9410a420 msgr2=0x7fee941a9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.244+0000 7fee98ed2640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee9410a420 0x7fee941a9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.244+0000 7fee98ed2640 1 -- 172.21.15.27:0/18661939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee9410b310 msgr2=0x7fee941a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.244+0000 7fee98ed2640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee9410b310 0x7fee941a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.244+0000 7fee98ed2640 1 -- 172.21.15.27:0/18661939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee941b0c20 con 0x7fee94110030 2024-04-25T13:53:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.245+0000 7fee98ed2640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee94110030 0x7fee941b05b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fee8800b410 tx=0x7fee8800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.245+0000 7fee917fa640 1 -- 172.21.15.27:0/18661939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88015020 con 0x7fee94110030 2024-04-25T13:53:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.245+0000 7fee917fa640 1 -- 172.21.15.27:0/18661939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee88011ce0 con 0x7fee94110030 2024-04-25T13:53:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.246+0000 7fee917fa640 1 -- 172.21.15.27:0/18661939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee88010da0 con 0x7fee94110030 2024-04-25T13:53:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.246+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee941b0f10 con 0x7fee94110030 2024-04-25T13:53:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.246+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee941b13f0 con 0x7fee94110030 2024-04-25T13:53:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.247+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee9410ee40 con 0x7fee94110030 2024-04-25T13:53:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.249+0000 7fee917fa640 1 -- 172.21.15.27:0/18661939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fee88018020 con 0x7fee94110030 2024-04-25T13:53:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.251+0000 7fee917fa640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee7c0afd00 0x7fee7c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.251+0000 7fee917fa640 1 -- 172.21.15.27:0/18661939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fee880de7b0 con 0x7fee94110030 2024-04-25T13:53:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.252+0000 7fee93fff640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee7c0afd00 0x7fee7c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.253+0000 7fee93fff640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee7c0afd00 0x7fee7c0b2210 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee8c031a00 tx=0x7fee8c009460 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.256+0000 7fee917fa640 1 -- 172.21.15.27:0/18661939 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee8801f020 con 0x7fee94110030 2024-04-25T13:53:26.403 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:26 smithi027 ceph-mon[29140]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6.3 MiB/s wr, 629 op/s 2024-04-25T13:53:26.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:26 smithi122 ceph-mon[32843]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6.3 MiB/s wr, 629 op/s 2024-04-25T13:53:26.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:26 smithi045 ceph-mon[32731]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6.3 MiB/s wr, 629 op/s 2024-04-25T13:53:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.578+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee9402db50 con 0x7fee94110030 2024-04-25T13:53:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.579+0000 7fee917fa640 1 -- 172.21.15.27:0/18661939 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fee880a73e0 con 0x7fee94110030 2024-04-25T13:53:26.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:53:26.579 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:53:26.580 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:53:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee7c0afd00 msgr2=0x7fee7c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee7c0afd00 0x7fee7c0b2210 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee8c031a00 tx=0x7fee8c009460 comp rx=0 tx=0).stop 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee94110030 msgr2=0x7fee941b05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee94110030 0x7fee941b05b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fee8800b410 tx=0x7fee8800b8d0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 shutdown_connections 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fee7c0afd00 0x7fee7c0b2210 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee94110030 0x7fee941b05b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee9410b310 0x7fee941a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 --2- 172.21.15.27:0/18661939 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee9410a420 0x7fee941a9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 >> 172.21.15.27:0/18661939 conn(0x7fee940704e0 msgr2=0x7fee9410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 shutdown_connections 2024-04-25T13:53:26.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.582+0000 7fee9a935640 1 -- 172.21.15.27:0/18661939 wait complete. 2024-04-25T13:53:26.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:53:26.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 -- 172.21.15.27:0/1965051968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800104d20 msgr2=0x7fc800105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 --2- 172.21.15.27:0/1965051968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800104d20 0x7fc800105130 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8009900 tx=0x7fc7e802a3f0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 -- 172.21.15.27:0/1965051968 shutdown_connections 2024-04-25T13:53:26.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 --2- 172.21.15.27:0/1965051968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc800109a20 0x7fc80010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 --2- 172.21.15.27:0/1965051968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800104d20 0x7fc800105130 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 --2- 172.21.15.27:0/1965051968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc800103ae0 0x7fc8001047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 -- 172.21.15.27:0/1965051968 >> 172.21.15.27:0/1965051968 conn(0x7fc8000fecd0 msgr2=0x7fc800101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 -- 172.21.15.27:0/1965051968 shutdown_connections 2024-04-25T13:53:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.754+0000 7fc805a61640 1 -- 172.21.15.27:0/1965051968 wait complete. 2024-04-25T13:53:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc805a61640 1 Processor -- start 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc805a61640 1 -- start start 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc805a61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc800103ae0 0x7fc800076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc805a61640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc800104d20 0x7fc800074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc805a61640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800109a20 0x7fc800075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc805a61640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010f630 con 0x7fc800103ae0 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc805a61640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010f4b0 con 0x7fc800104d20 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc805a61640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc80010f7b0 con 0x7fc800109a20 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.755+0000 7fc7feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc800103ae0 0x7fc800076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7feffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc800103ae0 0x7fc800076850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45546/0 (socket says 172.21.15.27:45546) 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7feffd640 1 -- 172.21.15.27:0/2222718549 learned_addr learned my addr 172.21.15.27:0/2222718549 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800109a20 0x7fc800075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc800104d20 0x7fc800074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7feffd640 1 -- 172.21.15.27:0/2222718549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800109a20 msgr2=0x7fc800075430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7feffd640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800109a20 0x7fc800075430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7feffd640 1 -- 172.21.15.27:0/2222718549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc800104d20 msgr2=0x7fc800074ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7feffd640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc800104d20 0x7fc800074ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7feffd640 1 -- 172.21.15.27:0/2222718549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc800075b60 con 0x7fc800103ae0 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800109a20 0x7fc800075430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc7feffd640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc800103ae0 0x7fc800076850 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc7f000d740 tx=0x7fc7f000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc804a5f640 1 -- 172.21.15.27:0/2222718549 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0017070 con 0x7fc800103ae0 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc805a61640 1 -- 172.21.15.27:0/2222718549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8001a42c0 con 0x7fc800103ae0 2024-04-25T13:53:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc805a61640 1 -- 172.21.15.27:0/2222718549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8001a4780 con 0x7fc800103ae0 2024-04-25T13:53:26.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.756+0000 7fc804a5f640 1 -- 172.21.15.27:0/2222718549 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f000de50 con 0x7fc800103ae0 2024-04-25T13:53:26.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.757+0000 7fc804a5f640 1 -- 172.21.15.27:0/2222718549 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0003c30 con 0x7fc800103ae0 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.759+0000 7fc804a5f640 1 -- 172.21.15.27:0/2222718549 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc7f0010460 con 0x7fc800103ae0 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.759+0000 7fc80c9ee740 1 Processor -- start 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.759+0000 7fc80c9ee740 1 -- start start 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.759+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68a6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.759+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.759+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559df680ad80 con 0x559df64db4d0 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559df67c4a20 con 0x559df6504390 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559df6806d40 con 0x559df64e36b0 2024-04-25T13:53:26.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68a6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60518/0 (socket says 172.21.15.27:60518) 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/3732272384 learned_addr learned my addr 172.21.15.27:0/3732272384 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7feffd640 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/3732272384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 msgr2=0x559df68a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/3732272384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 msgr2=0x559df68a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/3732272384 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559df67cfc30 con 0x559df6504390 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc804a5f640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc7ec0afb70 0x7fc7ec0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68a6810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80117e0 tx=0x7fc7f8011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a50f88d3a9847e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7feffd640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc7ec0afb70 0x7fc7ec0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc804a5f640 1 -- 172.21.15.27:0/2222718549 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc7f00de430 con 0x7fc800103ae0 2024-04-25T13:53:26.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7f5ffb640 1 -- 172.21.15.27:0/3732272384 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8011310 con 0x559df6504390 2024-04-25T13:53:26.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7f5ffb640 1 -- 172.21.15.27:0/3732272384 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f8003e20 con 0x559df6504390 2024-04-25T13:53:26.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.760+0000 7fc7f5ffb640 1 -- 172.21.15.27:0/3732272384 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80044f0 con 0x559df6504390 2024-04-25T13:53:26.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.762+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3732272384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 msgr2=0x559df68a6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.762+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68a6810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80117e0 tx=0x7fc7f8011ca0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.762+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3732272384 shutdown_connections 2024-04-25T13:53:26.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.762+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.762+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.762+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/3732272384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68a6810 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.762+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3732272384 >> 172.21.15.27:0/3732272384 conn(0x559df64f84e0 msgr2=0x559df68a2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.763+0000 7fc7feffd640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc7ec0afb70 0x7fc7ec0b2080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x559df68a6530 tx=0x7fc7f00e1f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.763+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3732272384 shutdown_connections 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.763+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3732272384 wait complete. 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 1 -- start start 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 1 Processor -- start 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x559df68a8530 con 0x559df64db4d0 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x559df6791dc0 con 0x559df6504390 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x559df68a86b0 con 0x559df64e36b0 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.764+0000 7fc80c9ee740 10 client.0 _ll_register_callbacks cb 0x559df67ad600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7fe7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45556/0 (socket says 172.21.15.27:45556) 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/2597431915 learned_addr learned my addr 172.21.15.27:0/2597431915 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/2597431915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 msgr2=0x559df68b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/2597431915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 msgr2=0x559df68b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/2597431915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559df651c3f0 con 0x559df64db4d0 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b0370 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80115f0 tx=0x7fc7f8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcb49c54f852ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7deffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7deffd640 1 -- 172.21.15.27:0/2597431915 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8004270 con 0x559df64db4d0 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7deffd640 1 -- 172.21.15.27:0/2597431915 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc7f8004410 con 0x559df64db4d0 2024-04-25T13:53:26.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc80c9ee740 1 -- 172.21.15.27:0/2597431915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f8003e20 con 0x559df64db4d0 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc7deffd640 1 -- 172.21.15.27:0/2597431915 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc7f8004610 con 0x559df64db4d0 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc80c9ee740 10 client.17871 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.765+0000 7fc80c9ee740 1 -- 172.21.15.27:0/2597431915 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559df65e67d0 con 0x559df64db4d0 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc7deffd640 1 -- 172.21.15.27:0/2597431915 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f80047b0 con 0x559df64db4d0 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc7deffd640 1 -- 172.21.15.27:0/2597431915 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc7f8025070 con 0x559df64db4d0 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc80c9ee740 10 client.17871 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc80c9ee740 10 client.17871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc80c9ee740 20 client.17871 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559df64c2f60 0x559df68bde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc80c9ee740 4 client.17871 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc80c9ee740 1 -- 172.21.15.27:0/2597431915 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc7f8003e20 con 0x559df64c2f60 2024-04-25T13:53:26.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.766+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559df64c2f60 0x559df68bde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.767+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559df64c2f60 0x559df68bde40 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48b7a6ccff726f72 server_cookie=cdc86b4c8ada8d2f in_seq=0 out_seq=0 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.767+0000 7fc7deffd640 10 client.17871 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc7deffd640 1 -- 172.21.15.27:0/2597431915 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fc7f8003e20 con 0x559df64c2f60 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc7deffd640 10 client.17871 handle_command_reply: tid=0 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 client.17871 shutdown 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 -- 172.21.15.27:0/2597431915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 msgr2=0x559df68b0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b0370 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80115f0 tx=0x7fc7f8005fd0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 -- 172.21.15.27:0/2597431915 shutdown_connections 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559df64c2f60 0x559df68bde40 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/2597431915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b0370 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc80c9ee740 1 -- 172.21.15.27:0/2597431915 >> 172.21.15.27:0/2597431915 conn(0x559df64f84e0 msgr2=0x559df68a4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.768+0000 7fc7deffd640 0 client.17871 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.769+0000 7fc80c9ee740 1 -- 172.21.15.27:0/2597431915 shutdown_connections 2024-04-25T13:53:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.769+0000 7fc80c9ee740 1 -- 172.21.15.27:0/2597431915 wait complete. 2024-04-25T13:53:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.769+0000 7fc80c9ee740 20 client.17871 trim_cache size 0 max 16384 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc80c9ee740 1 Processor -- start 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc80c9ee740 1 -- start start 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68af390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e000c390 con 0x559df64db4d0 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e000ec10 con 0x559df6504390 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e000ca00 con 0x559df64e36b0 2024-04-25T13:53:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68af390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45566/0 (socket says 172.21.15.27:45566) 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/3177428029 learned_addr learned my addr 172.21.15.27:0/3177428029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.787+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/3177428029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 msgr2=0x559df68a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/3177428029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 msgr2=0x559df68b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/3177428029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559df68aba90 con 0x559df64db4d0 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68af390 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8005d70 tx=0x7fc7f80021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432c3b818d3dc36f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7f57fa640 1 -- 172.21.15.27:0/3177428029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f801aa00 con 0x559df64db4d0 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7f57fa640 1 -- 172.21.15.27:0/3177428029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f803e020 con 0x559df64db4d0 2024-04-25T13:53:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.788+0000 7fc7f57fa640 1 -- 172.21.15.27:0/3177428029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f801aba0 con 0x559df64db4d0 2024-04-25T13:53:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.789+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3177428029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 msgr2=0x559df68af390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.789+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68af390 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8005d70 tx=0x7fc7f80021f0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.790+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3177428029 shutdown_connections 2024-04-25T13:53:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.790+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.790+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.790+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/3177428029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68af390 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.790+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3177428029 >> 172.21.15.27:0/3177428029 conn(0x559df64f84e0 msgr2=0x559df68a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.791+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3177428029 shutdown_connections 2024-04-25T13:53:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.791+0000 7fc80c9ee740 1 -- 172.21.15.27:0/3177428029 wait complete. 2024-04-25T13:53:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 1 -- start start 2024-04-25T13:53:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 1 Processor -- start 2024-04-25T13:53:26.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e000d0d0 con 0x559df64db4d0 2024-04-25T13:53:26.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e000e540 con 0x559df6504390 2024-04-25T13:53:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e000d7a0 con 0x559df64e36b0 2024-04-25T13:53:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc80c9ee740 10 client.0 _ll_register_callbacks cb 0x7fc7f00e1de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc7ff7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45570/0 (socket says 172.21.15.27:45570) 2024-04-25T13:53:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc7ff7fe640 1 -- 172.21.15.27:0/724256934 learned_addr learned my addr 172.21.15.27:0/724256934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.792+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.793+0000 7fc7fe7fc640 1 -- 172.21.15.27:0/724256934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 msgr2=0x559df68b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.793+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.793+0000 7fc7fe7fc640 1 -- 172.21.15.27:0/724256934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 msgr2=0x559df68b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.793+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.793+0000 7fc7fe7fc640 1 -- 172.21.15.27:0/724256934 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559df68b4a60 con 0x559df64e36b0 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.793+0000 7fc7fe7fc640 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b3fe0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc7e802a9e0 tx=0x7fc7e802ab70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48c31701dd2bfdd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.793+0000 7fc7deffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.793+0000 7fc7deffd640 1 -- 172.21.15.27:0/724256934 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8035040 con 0x559df64e36b0 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc7deffd640 1 -- 172.21.15.27:0/724256934 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc7e8046790 con 0x559df64e36b0 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 1 -- 172.21.15.27:0/724256934 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f803e020 con 0x559df64e36b0 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc7deffd640 1 -- 172.21.15.27:0/724256934 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc7e8046990 con 0x559df64e36b0 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 10 client.27613 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 1 -- 172.21.15.27:0/724256934 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x559df6925130 con 0x559df64e36b0 2024-04-25T13:53:26.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc7deffd640 1 -- 172.21.15.27:0/724256934 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f803e020 con 0x559df64e36b0 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc7deffd640 1 -- 172.21.15.27:0/724256934 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fc7e8031910 con 0x559df64e36b0 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 10 client.27613 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 10 client.27613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 20 client.27613 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559df64983a0 0x559df6927b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 4 client.27613 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.794+0000 7fc80c9ee740 1 -- 172.21.15.27:0/724256934 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559df6818990 con 0x559df64983a0 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559df64983a0 0x559df6927b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc7deffd640 10 client.27613 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc7ff7fe640 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559df64983a0 0x559df6927b20 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf48ead19b412ec server_cookie=39bdd4fdd4de49f3 in_seq=0 out_seq=0 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc7deffd640 1 -- 172.21.15.27:0/724256934 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559df6818990 con 0x559df64983a0 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc7deffd640 10 client.27613 handle_command_reply: tid=0 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 client.27613 shutdown 2024-04-25T13:53:26.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 -- 172.21.15.27:0/724256934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 msgr2=0x559df68b3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b3fe0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc7e802a9e0 tx=0x7fc7e802ab70 comp rx=0 tx=0).stop 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 -- 172.21.15.27:0/724256934 shutdown_connections 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x559df64983a0 0x559df6927b20 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559df6504390 0x559df68b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559df64e36b0 0x559df68b3fe0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 --2- 172.21.15.27:0/724256934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559df64db4d0 0x559df68b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc7deffd640 0 client.27613 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 -- 172.21.15.27:0/724256934 >> 172.21.15.27:0/724256934 conn(0x559df64f84e0 msgr2=0x559df68af950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 -- 172.21.15.27:0/724256934 shutdown_connections 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.795+0000 7fc80c9ee740 1 -- 172.21.15.27:0/724256934 wait complete. 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc80c9ee740 20 client.27613 trim_cache size 0 max 16384 2024-04-25T13:53:26.798 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:53:26.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 -- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc7ec0afb70 msgr2=0x7fc7ec0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc7ec0afb70 0x7fc7ec0b2080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x559df68a6530 tx=0x7fc7f00e1f10 comp rx=0 tx=0).stop 2024-04-25T13:53:26.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 -- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc800103ae0 msgr2=0x7fc800076850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:26.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc800103ae0 0x7fc800076850 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc7f000d740 tx=0x7fc7f000dc00 comp rx=0 tx=0).stop 2024-04-25T13:53:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 -- 172.21.15.27:0/2222718549 shutdown_connections 2024-04-25T13:53:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc7ec0afb70 0x7fc7ec0b2080 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc800109a20 0x7fc800075430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc800104d20 0x7fc800074ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 --2- 172.21.15.27:0/2222718549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc800103ae0 0x7fc800076850 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 -- 172.21.15.27:0/2222718549 >> 172.21.15.27:0/2222718549 conn(0x7fc8000fecd0 msgr2=0x7fc800105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 -- 172.21.15.27:0/2222718549 shutdown_connections 2024-04-25T13:53:26.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:26.796+0000 7fc805a61640 1 -- 172.21.15.27:0/2222718549 wait complete. 2024-04-25T13:53:27.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:27 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/18661939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:53:27.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:27 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/18661939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:53:27.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:27 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/18661939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:53:27.811 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:53:27.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:53:27.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:53:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 740.0 s, 1721.2 tps, lat 57.212 ms stddev 61.710 2024-04-25T13:53:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:27.999+0000 7fac22282640 1 -- 172.21.15.27:0/223706931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10b1d0 msgr2=0x7fac1c10b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:27.999+0000 7fac1affd640 1 -- 172.21.15.27:0/223706931 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10046020 con 0x7fac1c10b1d0 2024-04-25T13:53:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:27.999+0000 7fac22282640 1 --2- 172.21.15.27:0/223706931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10b1d0 0x7fac1c10b5e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fac1001a910 tx=0x7fac1003b450 comp rx=0 tx=0).stop 2024-04-25T13:53:28.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.001+0000 7fac22282640 1 -- 172.21.15.27:0/223706931 shutdown_connections 2024-04-25T13:53:28.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.001+0000 7fac22282640 1 --2- 172.21.15.27:0/223706931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac1c10fef0 0x7fac1c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.001+0000 7fac22282640 1 --2- 172.21.15.27:0/223706931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10b1d0 0x7fac1c10b5e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.001+0000 7fac22282640 1 --2- 172.21.15.27:0/223706931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c10a3e0 0x7fac1c10ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.001+0000 7fac22282640 1 -- 172.21.15.27:0/223706931 >> 172.21.15.27:0/223706931 conn(0x7fac1c070400 msgr2=0x7fac1c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.004+0000 7fac22282640 1 -- 172.21.15.27:0/223706931 shutdown_connections 2024-04-25T13:53:28.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.006+0000 7fac22282640 1 -- 172.21.15.27:0/223706931 wait complete. 2024-04-25T13:53:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.007+0000 7fac22282640 1 Processor -- start 2024-04-25T13:53:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.007+0000 7fac22282640 1 -- start start 2024-04-25T13:53:28.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.007+0000 7fac22282640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10a3e0 0x7fac1c1a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.007+0000 7fac22282640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c10b1d0 0x7fac1c1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.007+0000 7fac22282640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac1c10fef0 0x7fac1c1a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.007+0000 7fac22282640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c113b20 con 0x7fac1c10b1d0 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.007+0000 7fac22282640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1139a0 con 0x7fac1c10a3e0 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.007+0000 7fac22282640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c113ca0 con 0x7fac1c10fef0 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10a3e0 0x7fac1c1a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10a3e0 0x7fac1c1a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60562/0 (socket says 172.21.15.27:60562) 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 -- 172.21.15.27:0/3978531025 learned_addr learned my addr 172.21.15.27:0/3978531025 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 -- 172.21.15.27:0/3978531025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac1c10fef0 msgr2=0x7fac1c1a7c10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac1c10fef0 0x7fac1c1a7c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 -- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c10b1d0 msgr2=0x7fac1c1a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c10b1d0 0x7fac1c1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 -- 172.21.15.27:0/3978531025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1c1a8280 con 0x7fac1c10a3e0 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.008+0000 7fac1bfff640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10a3e0 0x7fac1c1a0740 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fac1400e9c0 tx=0x7fac1400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.009+0000 7fac197fa640 1 -- 172.21.15.27:0/3978531025 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1400ccc0 con 0x7fac1c10a3e0 2024-04-25T13:53:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.009+0000 7fac197fa640 1 -- 172.21.15.27:0/3978531025 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac140054d0 con 0x7fac1c10a3e0 2024-04-25T13:53:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.009+0000 7fac197fa640 1 -- 172.21.15.27:0/3978531025 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14010430 con 0x7fac1c10a3e0 2024-04-25T13:53:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.009+0000 7fac22282640 1 -- 172.21.15.27:0/3978531025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1c1a8570 con 0x7fac1c10a3e0 2024-04-25T13:53:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.009+0000 7fac22282640 1 -- 172.21.15.27:0/3978531025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac1c1a8af0 con 0x7fac1c10a3e0 2024-04-25T13:53:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.013+0000 7fac197fa640 1 -- 172.21.15.27:0/3978531025 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fac140046c0 con 0x7fac1c10a3e0 2024-04-25T13:53:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.013+0000 7fac2920d740 1 Processor -- start 2024-04-25T13:53:28.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac2920d740 1 -- start start 2024-04-25T13:53:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac2920d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c89660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac2920d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c89be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac2920d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c90ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac2920d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560d39beeb00 con 0x560d398e8410 2024-04-25T13:53:28.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac2920d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560d39b8a920 con 0x560d398bf550 2024-04-25T13:53:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac2920d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x560d39beaac0 con 0x560d398c7730 2024-04-25T13:53:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c89be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac2081f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c89660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac1b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c90ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac1b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c90ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51844/0 (socket says 172.21.15.27:51844) 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.014+0000 7fac1b7fe640 1 -- 172.21.15.27:0/3315744197 learned_addr learned my addr 172.21.15.27:0/3315744197 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.015+0000 7fac197fa640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac040d1bb0 0x7fac040d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.015+0000 7fac1b7fe640 1 -- 172.21.15.27:0/3315744197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 msgr2=0x560d39c89be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.015+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/3315744197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c89be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.015+0000 7fac1b7fe640 1 -- 172.21.15.27:0/3315744197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 msgr2=0x560d39c89660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.015+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/3315744197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c89660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.015+0000 7fac1b7fe640 1 -- 172.21.15.27:0/3315744197 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d399c87e0 con 0x560d398c7730 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.015+0000 7fac197fa640 1 -- 172.21.15.27:0/3978531025 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fac140deff0 con 0x7fac1c10a3e0 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.016+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/3315744197 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c90ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fac10005e00 tx=0x7fac10005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b06e9c8a1a16a00e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.016+0000 7fac027fc640 1 -- 172.21.15.27:0/3315744197 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10012020 con 0x560d398c7730 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.016+0000 7fac027fc640 1 -- 172.21.15.27:0/3315744197 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac10004ee0 con 0x560d398c7730 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.016+0000 7fac027fc640 1 -- 172.21.15.27:0/3315744197 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10046030 con 0x560d398c7730 2024-04-25T13:53:28.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.016+0000 7fac2081f640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac040d1bb0 0x7fac040d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 -- 172.21.15.27:0/3315744197 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 msgr2=0x560d39c90ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 --2- 172.21.15.27:0/3315744197 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c90ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fac10005e00 tx=0x7fac10005e30 comp rx=0 tx=0).stop 2024-04-25T13:53:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2081f640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac040d1bb0 0x7fac040d40c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x560d39c8a830 tx=0x7fac0c004a80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 -- 172.21.15.27:0/3315744197 shutdown_connections 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 --2- 172.21.15.27:0/3315744197 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c90ae0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 --2- 172.21.15.27:0/3315744197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c89be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 --2- 172.21.15.27:0/3315744197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c89660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 -- 172.21.15.27:0/3315744197 >> 172.21.15.27:0/3315744197 conn(0x560d398dc560 msgr2=0x560d39c84ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 -- 172.21.15.27:0/3315744197 shutdown_connections 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.017+0000 7fac2920d740 1 -- 172.21.15.27:0/3315744197 wait complete. 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.018+0000 7fac2920d740 1 -- start start 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.018+0000 7fac2920d740 1 Processor -- start 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.018+0000 7fac2920d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c93240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac2920d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c93780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac2920d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c9a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac2920d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x560d39c914d0 con 0x560d398e8410 2024-04-25T13:53:28.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c93240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac2920d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560d39b885d0 con 0x560d398bf550 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac2920d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x560d39c91650 con 0x560d398c7730 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c93240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60594/0 (socket says 172.21.15.27:60594) 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1bfff640 1 -- 172.21.15.27:0/2578723221 learned_addr learned my addr 172.21.15.27:0/2578723221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac2920d740 10 client.0 _ll_register_callbacks cb 0x560d39b76080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1b7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c93780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac2081f640 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c9a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1bfff640 1 -- 172.21.15.27:0/2578723221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 msgr2=0x560d39c93780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1bfff640 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c93780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1bfff640 1 -- 172.21.15.27:0/2578723221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 msgr2=0x560d39c9a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1bfff640 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c9a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac1bfff640 1 -- 172.21.15.27:0/2578723221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d39a28f70 con 0x560d398bf550 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.019+0000 7fac2081f640 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c9a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fac1bfff640 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c93240 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c6a0 tx=0x7fac14017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47bcf6276e96ae1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fabfb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fabfb7fe640 1 -- 172.21.15.27:0/2578723221 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac140e1320 con 0x560d398bf550 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fac2920d740 1 -- 172.21.15.27:0/2578723221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac10004ee0 con 0x560d398bf550 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fabfb7fe640 1 -- 172.21.15.27:0/2578723221 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fac140afed0 con 0x560d398bf550 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fabfb7fe640 1 -- 172.21.15.27:0/2578723221 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fac140a7c00 con 0x560d398bf550 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fac2920d740 10 client.27596 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fac2920d740 1 -- 172.21.15.27:0/2578723221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560d398c0170 con 0x560d398bf550 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fabfb7fe640 1 -- 172.21.15.27:0/2578723221 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac10004ee0 con 0x560d398bf550 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.020+0000 7fabfb7fe640 1 -- 172.21.15.27:0/2578723221 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fac14004fd0 con 0x560d398bf550 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fac2920d740 10 client.27596 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fac2920d740 10 client.27596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fac2920d740 20 client.27596 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fac2920d740 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560d398a71f0 0x560d39ca0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fac2920d740 4 client.27596 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fac2920d740 1 -- 172.21.15.27:0/2578723221 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d39bfc710 con 0x560d398a71f0 2024-04-25T13:53:28.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560d398a71f0 0x560d39ca0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fabfb7fe640 10 client.27596 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:28.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.021+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560d398a71f0 0x560d39ca0c10 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a80c57b8e477c00b server_cookie=8aef1b418412bf7d in_seq=0 out_seq=0 2024-04-25T13:53:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.022+0000 7fabfb7fe640 1 -- 172.21.15.27:0/2578723221 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560d39bfc710 con 0x560d398a71f0 2024-04-25T13:53:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.022+0000 7fabfb7fe640 10 client.27596 handle_command_reply: tid=0 2024-04-25T13:53:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.022+0000 7fac2920d740 1 client.27596 shutdown 2024-04-25T13:53:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 -- 172.21.15.27:0/2578723221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 msgr2=0x560d39c93240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c93240 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fac1400c6a0 tx=0x7fac14017000 comp rx=0 tx=0).stop 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 -- 172.21.15.27:0/2578723221 shutdown_connections 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560d398a71f0 0x560d39ca0c10 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fabfb7fe640 0 client.27596 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398e8410 0x560d39c9a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398c7730 0x560d39c93780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 --2- 172.21.15.27:0/2578723221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398bf550 0x560d39c93240 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 -- 172.21.15.27:0/2578723221 >> 172.21.15.27:0/2578723221 conn(0x560d398dc560 msgr2=0x560d39c83f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.023+0000 7fac2920d740 1 -- 172.21.15.27:0/2578723221 shutdown_connections 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.024+0000 7fac2920d740 1 -- 172.21.15.27:0/2578723221 wait complete. 2024-04-25T13:53:28.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.024+0000 7fac2920d740 20 client.27596 trim_cache size 0 max 16384 2024-04-25T13:53:28.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.046+0000 7fac2920d740 1 Processor -- start 2024-04-25T13:53:28.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.046+0000 7fac2920d740 1 -- start start 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac2920d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398a71f0 0x560d39c84ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac2920d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398bf550 0x560d39c92e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac2920d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c93350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac2920d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1af6a0 con 0x560d398a71f0 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac2920d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1b1f20 con 0x560d398c7730 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac2920d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1afd10 con 0x560d398bf550 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398a71f0 0x560d39c84ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac2081f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c93350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398a71f0 0x560d39c84ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45630/0 (socket says 172.21.15.27:45630) 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1bfff640 1 -- 172.21.15.27:0/2026924630 learned_addr learned my addr 172.21.15.27:0/2026924630 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398bf550 0x560d39c92e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1bfff640 1 -- 172.21.15.27:0/2026924630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398bf550 msgr2=0x560d39c92e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1bfff640 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398bf550 0x560d39c92e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1bfff640 1 -- 172.21.15.27:0/2026924630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 msgr2=0x560d39c93350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1bfff640 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c93350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.047+0000 7fac1bfff640 1 -- 172.21.15.27:0/2026924630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d39c8e150 con 0x560d398a71f0 2024-04-25T13:53:28.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.048+0000 7fac2081f640 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c93350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:28.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.048+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398bf550 0x560d39c92e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:28.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.048+0000 7fac1bfff640 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398a71f0 0x560d39c84ff0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fac140a7490 tx=0x7fac14009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13edd4bdf69d0f04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.049+0000 7fac01ffb640 1 -- 172.21.15.27:0/2026924630 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac140a33c0 con 0x560d398a71f0 2024-04-25T13:53:28.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.049+0000 7fac01ffb640 1 -- 172.21.15.27:0/2026924630 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac140a7500 con 0x560d398a71f0 2024-04-25T13:53:28.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.049+0000 7fac01ffb640 1 -- 172.21.15.27:0/2026924630 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac140afed0 con 0x560d398a71f0 2024-04-25T13:53:28.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 -- 172.21.15.27:0/2026924630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398a71f0 msgr2=0x560d39c84ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398a71f0 0x560d39c84ff0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fac140a7490 tx=0x7fac14009550 comp rx=0 tx=0).stop 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 -- 172.21.15.27:0/2026924630 shutdown_connections 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c93350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398bf550 0x560d39c92e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 --2- 172.21.15.27:0/2026924630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398a71f0 0x560d39c84ff0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 -- 172.21.15.27:0/2026924630 >> 172.21.15.27:0/2026924630 conn(0x560d398dc560 msgr2=0x560d39c848e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 -- 172.21.15.27:0/2026924630 shutdown_connections 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 -- 172.21.15.27:0/2026924630 wait complete. 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.050+0000 7fac2920d740 1 -- start start 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2920d740 1 Processor -- start 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2920d740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398a71f0 0x560d39c99690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2920d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398bf550 0x560d39c99bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2920d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c97de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2920d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1b03e0 con 0x560d398bf550 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2920d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1b1850 con 0x560d398c7730 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2920d740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1b0ab0 con 0x560d398a71f0 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398bf550 0x560d39c99bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2920d740 10 client.0 _ll_register_callbacks cb 0x7fac140a72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398bf550 0x560d39c99bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45638/0 (socket says 172.21.15.27:45638) 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1b7fe640 1 -- 172.21.15.27:0/1775885786 learned_addr learned my addr 172.21.15.27:0/1775885786 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2081f640 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c97de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1bfff640 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398a71f0 0x560d39c99690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1b7fe640 1 -- 172.21.15.27:0/1775885786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398a71f0 msgr2=0x560d39c99690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398a71f0 0x560d39c99690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1b7fe640 1 -- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 msgr2=0x560d39c97de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c97de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac1b7fe640 1 -- 172.21.15.27:0/1775885786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d39c984d0 con 0x560d398bf550 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.051+0000 7fac2081f640 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c97de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fac1b7fe640 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398bf550 0x560d39c99bd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fac10005ea0 tx=0x7fac10005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31cb43d368afb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fabfb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fabfb7fe640 1 -- 172.21.15.27:0/1775885786 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10012040 con 0x560d398bf550 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fac2920d740 1 -- 172.21.15.27:0/1775885786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac140a7500 con 0x560d398bf550 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fabfb7fe640 1 -- 172.21.15.27:0/1775885786 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fac100051b0 con 0x560d398bf550 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fabfb7fe640 1 -- 172.21.15.27:0/1775885786 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fac10046040 con 0x560d398bf550 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fac2920d740 10 client.17889 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fac2920d740 1 -- 172.21.15.27:0/1775885786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x560d39d09980 con 0x560d398bf550 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.052+0000 7fabfb7fe640 1 -- 172.21.15.27:0/1775885786 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac140a7500 con 0x560d398bf550 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fabfb7fe640 1 -- 172.21.15.27:0/1775885786 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fac10040cd0 con 0x560d398bf550 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fac2920d740 10 client.17889 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fac2920d740 10 client.17889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fac2920d740 20 client.17889 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fac2920d740 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560d398e8410 0x560d39d0bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fac2920d740 4 client.17889 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:53:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fac2920d740 1 -- 172.21.15.27:0/1775885786 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560d39bf6580 con 0x560d398e8410 2024-04-25T13:53:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fac1bfff640 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560d398e8410 0x560d39d0bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fabfb7fe640 10 client.17889 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:28.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.053+0000 7fac1bfff640 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560d398e8410 0x560d39d0bee0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd63bb7a47c6b05e server_cookie=b24e2b620ace746a in_seq=0 out_seq=0 2024-04-25T13:53:28.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.054+0000 7fabfb7fe640 1 -- 172.21.15.27:0/1775885786 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560d39bf6580 con 0x560d398e8410 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.054+0000 7fabfb7fe640 10 client.17889 handle_command_reply: tid=0 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.054+0000 7fac2920d740 1 client.17889 shutdown 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 -- 172.21.15.27:0/1775885786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398bf550 msgr2=0x560d39c99bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398bf550 0x560d39c99bd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fac10005ea0 tx=0x7fac10005d50 comp rx=0 tx=0).stop 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 -- 172.21.15.27:0/1775885786 shutdown_connections 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x560d398e8410 0x560d39d0bee0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fabfb7fe640 0 client.17889 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560d398c7730 0x560d39c97de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560d398bf550 0x560d39c99bd0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 --2- 172.21.15.27:0/1775885786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560d398a71f0 0x560d39c99690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 -- 172.21.15.27:0/1775885786 >> 172.21.15.27:0/1775885786 conn(0x560d398dc560 msgr2=0x560d39c92e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 -- 172.21.15.27:0/1775885786 shutdown_connections 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.055+0000 7fac2920d740 1 -- 172.21.15.27:0/1775885786 wait complete. 2024-04-25T13:53:28.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.056+0000 7fac2920d740 20 client.17889 trim_cache size 0 max 16384 2024-04-25T13:53:28.059 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:53:28.059 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:53:28.059 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "86524397-d1d8-49e1-8c08-8878d29ae245", 2024-04-25T13:53:28.059 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:53:28.059 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:53:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 -- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac040d1bb0 msgr2=0x7fac040d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac040d1bb0 0x7fac040d40c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x560d39c8a830 tx=0x7fac0c004a80 comp rx=0 tx=0).stop 2024-04-25T13:53:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 -- 172.21.15.27:0/3978531025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10a3e0 msgr2=0x7fac1c1a0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10a3e0 0x7fac1c1a0740 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fac1400e9c0 tx=0x7fac1400ee80 comp rx=0 tx=0).stop 2024-04-25T13:53:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 -- 172.21.15.27:0/3978531025 shutdown_connections 2024-04-25T13:53:28.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fac040d1bb0 0x7fac040d40c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fac1c10fef0 0x7fac1c1a7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1c10b1d0 0x7fac1c1a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.059+0000 7fac22282640 1 --2- 172.21.15.27:0/3978531025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac1c10a3e0 0x7fac1c1a0740 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.060+0000 7fac22282640 1 -- 172.21.15.27:0/3978531025 >> 172.21.15.27:0/3978531025 conn(0x7fac1c070400 msgr2=0x7fac1c06a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.060+0000 7fac22282640 1 -- 172.21.15.27:0/3978531025 shutdown_connections 2024-04-25T13:53:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.060+0000 7fac22282640 1 -- 172.21.15.27:0/3978531025 wait complete. 2024-04-25T13:53:28.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:53:28.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 -- 172.21.15.27:0/2720105630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac1120c0 msgr2=0x7f1bac114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/2720105630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac1120c0 0x7f1bac114520 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009ca0 tx=0x7f1ba402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 -- 172.21.15.27:0/2720105630 shutdown_connections 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/2720105630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac1120c0 0x7f1bac114520 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/2720105630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bac10f760 0x7f1bac111b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/2720105630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bac109fc0 0x7f1bac10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 -- 172.21.15.27:0/2720105630 >> 172.21.15.27:0/2720105630 conn(0x7f1bac0704e0 msgr2=0x7f1bac072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 -- 172.21.15.27:0/2720105630 shutdown_connections 2024-04-25T13:53:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.249+0000 7f1bb0d84640 1 -- 172.21.15.27:0/2720105630 wait complete. 2024-04-25T13:53:28.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.251+0000 7f1bb0d84640 1 Processor -- start 2024-04-25T13:53:28.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.251+0000 7f1bb0d84640 1 -- start start 2024-04-25T13:53:28.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1bb0d84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bac109fc0 0x7f1bac1a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1bb0d84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac10f760 0x7f1bac1a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1bb0d84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bac1120c0 0x7f1bac1a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1bb0d84640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac115cc0 con 0x7f1bac10f760 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1bb0d84640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac115b40 con 0x7f1bac1120c0 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1bb0d84640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac115e40 con 0x7f1bac109fc0 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1baa59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bac109fc0 0x7f1bac1a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1baa59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bac109fc0 0x7f1bac1a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51900/0 (socket says 172.21.15.27:51900) 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1baa59c640 1 -- 172.21.15.27:0/1966732672 learned_addr learned my addr 172.21.15.27:0/1966732672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1baad9d640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bac1120c0 0x7f1bac1a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.252+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac10f760 0x7f1bac1a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.253+0000 7f1baad9d640 1 -- 172.21.15.27:0/1966732672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bac109fc0 msgr2=0x7f1bac1a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.253+0000 7f1baad9d640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bac109fc0 0x7f1bac1a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.253+0000 7f1baad9d640 1 -- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac10f760 msgr2=0x7f1bac1a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.253+0000 7f1baad9d640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac10f760 0x7f1bac1a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.253+0000 7f1baad9d640 1 -- 172.21.15.27:0/1966732672 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bac1a8370 con 0x7f1bac1120c0 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.253+0000 7f1baad9d640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bac1120c0 0x7f1bac1a7d00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4033040 tx=0x7f1ba4031680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.254+0000 7f1b9b7fe640 1 -- 172.21.15.27:0/1966732672 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4003b70 con 0x7f1bac1120c0 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.254+0000 7f1b9b7fe640 1 -- 172.21.15.27:0/1966732672 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba4003d10 con 0x7f1bac1120c0 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.254+0000 7f1b9b7fe640 1 -- 172.21.15.27:0/1966732672 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba402fac0 con 0x7f1bac1120c0 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.254+0000 7f1bb0d84640 1 -- 172.21.15.27:0/1966732672 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bac1a8600 con 0x7f1bac1120c0 2024-04-25T13:53:28.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.254+0000 7f1bb0d84640 1 -- 172.21.15.27:0/1966732672 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bac1a8ae0 con 0x7f1bac1120c0 2024-04-25T13:53:28.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.258+0000 7f1bb7c33740 1 Processor -- start 2024-04-25T13:53:28.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.258+0000 7f1b9b7fe640 1 -- 172.21.15.27:0/1966732672 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1ba4004580 con 0x7f1bac1120c0 2024-04-25T13:53:28.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.258+0000 7f1bb7c33740 1 -- start start 2024-04-25T13:53:28.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.261+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65c1a390 0x55aa65fc4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.261+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fc47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.261+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bf96b0 0x55aa65fc5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.261+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aa65f20d80 con 0x55aa65bf14d0 2024-04-25T13:53:28.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.261+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55aa65eda8b0 con 0x55aa65c1a390 2024-04-25T13:53:28.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.261+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55aa65f1cd40 con 0x55aa65bf96b0 2024-04-25T13:53:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.262+0000 7f1b9b7fe640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1b940afbf0 0x7f1b940b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.262+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65c1a390 0x55aa65fc4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.262+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bf96b0 0x55aa65fc5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.262+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65c1a390 0x55aa65fc4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60642/0 (socket says 172.21.15.27:60642) 2024-04-25T13:53:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.262+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/1787883436 learned_addr learned my addr 172.21.15.27:0/1787883436 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.262+0000 7f1b9b7fe640 1 -- 172.21.15.27:0/1966732672 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1ba4100000 con 0x7f1bac1120c0 2024-04-25T13:53:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.262+0000 7f1baad9d640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1b940afbf0 0x7f1b940b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1baa59c640 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fc47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/1787883436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bf96b0 msgr2=0x55aa65fc5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bf96b0 0x55aa65fc5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/1787883436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 msgr2=0x55aa65fc47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fc47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/1787883436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa65eaba40 con 0x55aa65c1a390 2024-04-25T13:53:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1baa59c640 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fc47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1baad9d640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1b940afbf0 0x7f1b940b2100 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40094e0 tx=0x7f1ba402a8c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.263+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65c1a390 0x55aa65fc4270 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0018810 tx=0x7f1ba0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ad13bb4b2473d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.264+0000 7f1b98ff9640 1 -- 172.21.15.27:0/1787883436 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0021070 con 0x55aa65c1a390 2024-04-25T13:53:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.264+0000 7f1b98ff9640 1 -- 172.21.15.27:0/1787883436 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba0013c60 con 0x55aa65c1a390 2024-04-25T13:53:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.264+0000 7f1b98ff9640 1 -- 172.21.15.27:0/1787883436 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0021070 con 0x55aa65c1a390 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 -- 172.21.15.27:0/1787883436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65c1a390 msgr2=0x55aa65fc4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65c1a390 0x55aa65fc4270 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0018810 tx=0x7f1ba0005f90 comp rx=0 tx=0).stop 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 -- 172.21.15.27:0/1787883436 shutdown_connections 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bf96b0 0x55aa65fc5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fc47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/1787883436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65c1a390 0x55aa65fc4270 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 -- 172.21.15.27:0/1787883436 >> 172.21.15.27:0/1787883436 conn(0x55aa65c0e4e0 msgr2=0x55aa65fbfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 -- 172.21.15.27:0/1787883436 shutdown_connections 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 -- 172.21.15.27:0/1787883436 wait complete. 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.265+0000 7f1bb7c33740 1 -- start start 2024-04-25T13:53:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1bb7c33740 1 Processor -- start 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65c1a390 0x55aa65fd5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55aa65fc5f90 con 0x55aa65bf14d0 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55aa65ea7d10 con 0x55aa65bf96b0 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55aa65fc6110 con 0x55aa65c1a390 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1bb7c33740 10 client.0 _ll_register_callbacks cb 0x55aa65ed4210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1baa59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1baa59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60652/0 (socket says 172.21.15.27:60652) 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1baa59c640 1 -- 172.21.15.27:0/3090501924 learned_addr learned my addr 172.21.15.27:0/3090501924 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.266+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65c1a390 0x55aa65fd5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1baa59c640 1 -- 172.21.15.27:0/3090501924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65c1a390 msgr2=0x55aa65fd5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1baa59c640 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65c1a390 0x55aa65fd5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1baa59c640 1 -- 172.21.15.27:0/3090501924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 msgr2=0x55aa65fce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1baa59c640 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1baa59c640 1 -- 172.21.15.27:0/3090501924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa65cfc7d0 con 0x55aa65bf96b0 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65c1a390 0x55aa65fd5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1baa59c640 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fce740 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c012700 tx=0x7f1b9c012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd4efa3e22e62789 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1b91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1b91ffb640 1 -- 172.21.15.27:0/3090501924 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c004030 con 0x55aa65bf96b0 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.267+0000 7f1b91ffb640 1 -- 172.21.15.27:0/3090501924 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1b9c0041d0 con 0x55aa65bf96b0 2024-04-25T13:53:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.268+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3090501924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ba0013c60 con 0x55aa65bf96b0 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.268+0000 7f1b91ffb640 1 -- 172.21.15.27:0/3090501924 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1b9c0043f0 con 0x55aa65bf96b0 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.268+0000 7f1bb7c33740 10 client.27614 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.268+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3090501924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55aa65c323f0 con 0x55aa65bf96b0 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.269+0000 7f1b91ffb640 1 -- 172.21.15.27:0/3090501924 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba0013c60 con 0x55aa65bf96b0 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.269+0000 7f1b91ffb640 1 -- 172.21.15.27:0/3090501924 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1b9c003c40 con 0x55aa65bf96b0 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.269+0000 7f1bb7c33740 10 client.27614 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.269+0000 7f1bb7c33740 10 client.27614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.269+0000 7f1bb7c33740 20 client.27614 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.269+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa65bd8f60 0x55aa65fdbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.269+0000 7f1bb7c33740 4 client.27614 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.269+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3090501924 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa65f2e990 con 0x55aa65bd8f60 2024-04-25T13:53:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.270+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa65bd8f60 0x55aa65fdbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.270+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa65bd8f60 0x55aa65fdbc00 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1222bf6c05ffed79 server_cookie=760fce893743d198 in_seq=0 out_seq=0 2024-04-25T13:53:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.270+0000 7f1b91ffb640 10 client.27614 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.271+0000 7f1b91ffb640 1 -- 172.21.15.27:0/3090501924 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55aa65f2e990 con 0x55aa65bd8f60 2024-04-25T13:53:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.271+0000 7f1b91ffb640 10 client.27614 handle_command_reply: tid=0 2024-04-25T13:53:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.271+0000 7f1bb7c33740 1 client.27614 shutdown 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3090501924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 msgr2=0x55aa65fce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fce740 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c012700 tx=0x7f1b9c012bc0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3090501924 shutdown_connections 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa65bd8f60 0x55aa65fdbc00 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65c1a390 0x55aa65fd5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1b91ffb640 0 client.27614 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fce740 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3090501924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3090501924 >> 172.21.15.27:0/3090501924 conn(0x55aa65c0e4e0 msgr2=0x55aa65fb59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3090501924 shutdown_connections 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.272+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3090501924 wait complete. 2024-04-25T13:53:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.273+0000 7f1bb7c33740 20 client.27614 trim_cache size 0 max 16384 2024-04-25T13:53:28.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.302+0000 7f1bb7c33740 1 Processor -- start 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1bb7c33740 1 -- start start 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fbfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf14d0 0x55aa65fb59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf96b0 0x55aa65fb5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac1af750 con 0x55aa65bf96b0 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac1b1fd0 con 0x55aa65bf14d0 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac1afdc0 con 0x55aa65bd8f60 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fbfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fbfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51922/0 (socket says 172.21.15.27:51922) 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/3734802279 learned_addr learned my addr 172.21.15.27:0/3734802279 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1baa59c640 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf14d0 0x55aa65fb59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.303+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf96b0 0x55aa65fb5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/3734802279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf14d0 msgr2=0x55aa65fb59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf14d0 0x55aa65fb59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/3734802279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf96b0 msgr2=0x55aa65fb5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf96b0 0x55aa65fb5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/3734802279 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa65bf20f0 con 0x55aa65bd8f60 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fbfbc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000dd60 tx=0x7f1ba000df60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d38c5225749c58d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1b93fff640 1 -- 172.21.15.27:0/3734802279 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0021070 con 0x55aa65bd8f60 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1b93fff640 1 -- 172.21.15.27:0/3734802279 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba00054f0 con 0x55aa65bd8f60 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.304+0000 7f1b93fff640 1 -- 172.21.15.27:0/3734802279 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0003900 con 0x55aa65bd8f60 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.305+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3734802279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 msgr2=0x55aa65fbfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.305+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fbfbc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000dd60 tx=0x7f1ba000df60 comp rx=0 tx=0).stop 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.305+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3734802279 shutdown_connections 2024-04-25T13:53:28.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.305+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf96b0 0x55aa65fb5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.305+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf14d0 0x55aa65fb59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.305+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/3734802279 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fbfbc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.305+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3734802279 >> 172.21.15.27:0/3734802279 conn(0x55aa65c0e4e0 msgr2=0x55aa65fbf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.307+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3734802279 shutdown_connections 2024-04-25T13:53:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.307+0000 7f1bb7c33740 1 -- 172.21.15.27:0/3734802279 wait complete. 2024-04-25T13:53:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.307+0000 7f1bb7c33740 1 -- start start 2024-04-25T13:53:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.307+0000 7f1bb7c33740 1 Processor -- start 2024-04-25T13:53:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fd3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fd38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1bb7c33740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fd47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac1b0490 con 0x55aa65bf14d0 2024-04-25T13:53:28.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac1b1900 con 0x55aa65bf96b0 2024-04-25T13:53:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1bb7c33740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac1b0b60 con 0x55aa65bd8f60 2024-04-25T13:53:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1bb7c33740 10 client.0 _ll_register_callbacks cb 0x7f1b9c009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fd3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1ba9d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fd3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51924/0 (socket says 172.21.15.27:51924) 2024-04-25T13:53:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.308+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/196974538 learned_addr learned my addr 172.21.15.27:0/196974538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.317+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fd47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.318+0000 7f1baa59c640 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fd38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.318+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/196974538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 msgr2=0x55aa65fd3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.318+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fd3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.318+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/196974538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 msgr2=0x55aa65fd38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.318+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fd38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.318+0000 7f1ba9d9b640 1 -- 172.21.15.27:0/196974538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa65ee14b0 con 0x55aa65bf96b0 2024-04-25T13:53:28.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.323+0000 7f1ba9d9b640 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fd47d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000de80 tx=0x7f1ba000dd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5db004e2279cede server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:28.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.323+0000 7f1b91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:28.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.327+0000 7f1b91ffb640 1 -- 172.21.15.27:0/196974538 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0021040 con 0x55aa65bf96b0 2024-04-25T13:53:28.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.327+0000 7f1b91ffb640 1 -- 172.21.15.27:0/196974538 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1ba00050e0 con 0x55aa65bf96b0 2024-04-25T13:53:28.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.329+0000 7f1bb7c33740 1 -- 172.21.15.27:0/196974538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ba00054f0 con 0x55aa65bf96b0 2024-04-25T13:53:28.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.334+0000 7f1b91ffb640 1 -- 172.21.15.27:0/196974538 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1ba00052e0 con 0x55aa65bf96b0 2024-04-25T13:53:28.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.339+0000 7f1bb7c33740 10 client.27620 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:28.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.339+0000 7f1bb7c33740 1 -- 172.21.15.27:0/196974538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55aa65fd5e50 con 0x55aa65bf96b0 2024-04-25T13:53:28.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.346+0000 7f1b91ffb640 1 -- 172.21.15.27:0/196974538 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba00054f0 con 0x55aa65bf96b0 2024-04-25T13:53:28.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.346+0000 7f1b91ffb640 1 -- 172.21.15.27:0/196974538 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1ba0036020 con 0x55aa65bf96b0 2024-04-25T13:53:28.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.347+0000 7f1bb7c33740 10 client.27620 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:28.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.347+0000 7f1bb7c33740 10 client.27620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:28.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.347+0000 7f1bb7c33740 20 client.27620 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:28.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.347+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa65c1a390 0x55aa66046160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:28.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.347+0000 7f1bb7c33740 4 client.27620 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:53:28.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.347+0000 7f1bb7c33740 1 -- 172.21.15.27:0/196974538 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aa65f28800 con 0x55aa65c1a390 2024-04-25T13:53:28.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.356+0000 7f1baa59c640 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa65c1a390 0x55aa66046160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.368+0000 7f1baa59c640 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa65c1a390 0x55aa66046160 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6be04485f1e617a2 server_cookie=f1756994e0fb8de2 in_seq=0 out_seq=0 2024-04-25T13:53:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.369+0000 7f1b91ffb640 10 client.27620 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.373+0000 7f1b91ffb640 1 -- 172.21.15.27:0/196974538 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55aa65f28800 con 0x55aa65c1a390 2024-04-25T13:53:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.373+0000 7f1b91ffb640 10 client.27620 handle_command_reply: tid=0 2024-04-25T13:53:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.374+0000 7f1bb7c33740 1 client.27620 shutdown 2024-04-25T13:53:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.378+0000 7f1bb7c33740 1 -- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 msgr2=0x55aa65fd47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.379+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fd47d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000de80 tx=0x7f1ba000dd90 comp rx=0 tx=0).stop 2024-04-25T13:53:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.380+0000 7f1bb7c33740 1 -- 172.21.15.27:0/196974538 shutdown_connections 2024-04-25T13:53:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.380+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aa65c1a390 0x55aa66046160 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.380+0000 7f1b91ffb640 0 client.27620 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.380+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa65bf96b0 0x55aa65fd47d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.380+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aa65bf14d0 0x55aa65fd38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.380+0000 7f1bb7c33740 1 --2- 172.21.15.27:0/196974538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aa65bd8f60 0x55aa65fd3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.380+0000 7f1bb7c33740 1 -- 172.21.15.27:0/196974538 >> 172.21.15.27:0/196974538 conn(0x55aa65c0e4e0 msgr2=0x55aa65fbf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.382+0000 7f1bb7c33740 1 -- 172.21.15.27:0/196974538 shutdown_connections 2024-04-25T13:53:28.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.383+0000 7f1bb7c33740 1 -- 172.21.15.27:0/196974538 wait complete. 2024-04-25T13:53:28.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.383+0000 7f1bb7c33740 20 client.27620 trim_cache size 0 max 16384 2024-04-25T13:53:28.383 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:53:28.383 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:53:28.383 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:53:28.383 INFO:teuthology.orchestra.run.smithi027.stdout: "86524397-d1d8-49e1-8c08-8878d29ae245": { 2024-04-25T13:53:28.383 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:53:28.383 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "86524397-d1d8-49e1-8c08-8878d29ae245", 2024-04-25T13:53:28.384 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:53:28.384 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:53:28.384 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:53:28.384 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:53:28.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 -- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1b940afbf0 msgr2=0x7f1b940b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1b940afbf0 0x7f1b940b2100 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40094e0 tx=0x7f1ba402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 -- 172.21.15.27:0/1966732672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bac1120c0 msgr2=0x7f1bac1a7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bac1120c0 0x7f1bac1a7d00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4033040 tx=0x7f1ba4031680 comp rx=0 tx=0).stop 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 -- 172.21.15.27:0/1966732672 shutdown_connections 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1b940afbf0 0x7f1b940b2100 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bac1120c0 0x7f1bac1a7d00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bac10f760 0x7f1bac1a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 --2- 172.21.15.27:0/1966732672 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bac109fc0 0x7f1bac1a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:28.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.385+0000 7f1bb0d84640 1 -- 172.21.15.27:0/1966732672 >> 172.21.15.27:0/1966732672 conn(0x7f1bac0704e0 msgr2=0x7f1bac1106a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:28.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.386+0000 7f1bb0d84640 1 -- 172.21.15.27:0/1966732672 shutdown_connections 2024-04-25T13:53:28.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:28.386+0000 7f1bb0d84640 1 -- 172.21.15.27:0/1966732672 wait complete. 2024-04-25T13:53:28.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:28 smithi027 ceph-mon[29140]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.8 MiB/s wr, 617 op/s 2024-04-25T13:53:28.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:28 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:53:28.403 INFO:tasks.cephfs.filesystem:scrub status for tag:86524397-d1d8-49e1-8c08-8878d29ae245 - {'path': '/', 'tag': '86524397-d1d8-49e1-8c08-8878d29ae245', 'options': 'recursive,force'} 2024-04-25T13:53:28.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:28 smithi122 ceph-mon[32843]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.8 MiB/s wr, 617 op/s 2024-04-25T13:53:28.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:28 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:53:28.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:28 smithi045 ceph-mon[32731]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.8 MiB/s wr, 617 op/s 2024-04-25T13:53:28.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:28 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:53:29.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:29 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:53:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:29 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:53:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:29 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:53:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:29 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:53:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:53:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:53:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:53:29.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:29 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:53:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:29 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:53:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:29 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:53:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:29 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:53:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:29 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:53:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:53:29.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:53:29.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:53:29.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:29 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:53:29.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:29 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:53:29.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:29 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:53:29.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:29 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:53:29.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:29 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:53:29.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:53:29.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:53:29.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:53:29.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:29 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:53:29.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.872+0000 7ffaabfff640 1 -- 172.21.15.27:0/2742830754 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4030840 con 0x7ffab4110070 2024-04-25T13:53:29.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.873+0000 7ffabb572640 1 -- 172.21.15.27:0/2742830754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffab4110070 msgr2=0x7ffab4110480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:29.873 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.873+0000 7ffabb572640 1 --2- 172.21.15.27:0/2742830754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffab4110070 0x7ffab4110480 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009860 tx=0x7ffaa402a380 comp rx=0 tx=0).stop 2024-04-25T13:53:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.874+0000 7ffabb572640 1 -- 172.21.15.27:0/2742830754 shutdown_connections 2024-04-25T13:53:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.874+0000 7ffabb572640 1 --2- 172.21.15.27:0/2742830754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffab4110070 0x7ffab4110480 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.874+0000 7ffabb572640 1 --2- 172.21.15.27:0/2742830754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffab410b350 0x7ffab410b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.874+0000 7ffabb572640 1 --2- 172.21.15.27:0/2742830754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab410a420 0x7ffab410ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.874+0000 7ffabb572640 1 -- 172.21.15.27:0/2742830754 >> 172.21.15.27:0/2742830754 conn(0x7ffab4070540 msgr2=0x7ffab40729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:29.874 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.875+0000 7ffabb572640 1 -- 172.21.15.27:0/2742830754 shutdown_connections 2024-04-25T13:53:29.875 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.875+0000 7ffabb572640 1 -- 172.21.15.27:0/2742830754 wait complete. 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.875+0000 7ffabb572640 1 Processor -- start 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffabb572640 1 -- start start 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffabb572640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffab410a420 0x7ffab41a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffabb572640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab410b350 0x7ffab41a95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffabb572640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffab4110070 0x7ffab41b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffabb572640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffab411e520 con 0x7ffab410b350 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffabb572640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffab411e3a0 con 0x7ffab410a420 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffabb572640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ffab411e6a0 con 0x7ffab4110070 2024-04-25T13:53:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffab8b0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab410b350 0x7ffab41a95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffab8b0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab410b350 0x7ffab41a95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45728/0 (socket says 172.21.15.27:45728) 2024-04-25T13:53:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.876+0000 7ffab8b0d640 1 -- 172.21.15.27:0/576676869 learned_addr learned my addr 172.21.15.27:0/576676869 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.877+0000 7ffab8b0d640 1 -- 172.21.15.27:0/576676869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffab4110070 msgr2=0x7ffab41b0550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:53:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.877+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffab4110070 0x7ffab41b0550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.877+0000 7ffab8b0d640 1 -- 172.21.15.27:0/576676869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffab410a420 msgr2=0x7ffab41a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.877+0000 7ffab930e640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffab410a420 0x7ffab41a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.877+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffab410a420 0x7ffab41a9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.877+0000 7ffab8b0d640 1 -- 172.21.15.27:0/576676869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab41b0bc0 con 0x7ffab410b350 2024-04-25T13:53:29.878 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.878+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab410b350 0x7ffab41a95c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e9c0 tx=0x7ffab000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.878+0000 7ffaaa7fc640 1 -- 172.21.15.27:0/576676869 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab000cc40 con 0x7ffab410b350 2024-04-25T13:53:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.878+0000 7ffaaa7fc640 1 -- 172.21.15.27:0/576676869 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab000cde0 con 0x7ffab410b350 2024-04-25T13:53:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.879+0000 7ffaaa7fc640 1 -- 172.21.15.27:0/576676869 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0010430 con 0x7ffab410b350 2024-04-25T13:53:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.879+0000 7ffabb572640 1 -- 172.21.15.27:0/576676869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab41b0eb0 con 0x7ffab410b350 2024-04-25T13:53:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.879+0000 7ffabb572640 1 -- 172.21.15.27:0/576676869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab41b1320 con 0x7ffab410b350 2024-04-25T13:53:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.882+0000 7ffac2427740 1 Processor -- start 2024-04-25T13:53:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.882+0000 7ffac2427740 1 -- start start 2024-04-25T13:53:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.882+0000 7ffac2427740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcfb1b0 0x562f4009b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.882+0000 7ffac2427740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcd22f0 0x562f4009bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffac2427740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 0x562f400a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffac2427740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562f40001460 con 0x562f3fcfb1b0 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffac2427740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562f3ffbb590 con 0x562f3fcda4d0 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffac2427740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562f3fffd420 con 0x562f3fcd22f0 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab9b0f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcfb1b0 0x562f4009b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab9b0f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcfb1b0 0x562f4009b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45738/0 (socket says 172.21.15.27:45738) 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab9b0f640 1 -- 172.21.15.27:0/3198271978 learned_addr learned my addr 172.21.15.27:0/3198271978 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab930e640 1 --2- 172.21.15.27:0/3198271978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcd22f0 0x562f4009bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab930e640 1 -- 172.21.15.27:0/3198271978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 msgr2=0x562f400a2df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:53:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab930e640 1 --2- 172.21.15.27:0/3198271978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 0x562f400a2df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab930e640 1 -- 172.21.15.27:0/3198271978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcfb1b0 msgr2=0x562f4009b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab930e640 1 --2- 172.21.15.27:0/3198271978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcfb1b0 0x562f4009b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffab930e640 1 -- 172.21.15.27:0/3198271978 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f3ffc80d0 con 0x562f3fcd22f0 2024-04-25T13:53:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.883+0000 7ffaaa7fc640 1 -- 172.21.15.27:0/576676869 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ffab0004ee0 con 0x7ffab410b350 2024-04-25T13:53:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.884+0000 7ffab930e640 1 --2- 172.21.15.27:0/3198271978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcd22f0 0x562f4009bef0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffaac005fd0 tx=0x7ffaac00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0390be101944394 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.884+0000 7ffa9f7fe640 1 -- 172.21.15.27:0/3198271978 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac011440 con 0x562f3fcd22f0 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.884+0000 7ffa9f7fe640 1 -- 172.21.15.27:0/3198271978 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaac011a90 con 0x562f3fcd22f0 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.884+0000 7ffa9f7fe640 1 -- 172.21.15.27:0/3198271978 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac019bf0 con 0x562f3fcd22f0 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.884+0000 7ffaaa7fc640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ffaa00afbb0 0x7ffaa00b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.884+0000 7ffaaa7fc640 1 -- 172.21.15.27:0/576676869 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ffab00deed0 con 0x7ffab410b350 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffab9b0f640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ffaa00afbb0 0x7ffaa00b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 -- 172.21.15.27:0/3198271978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcd22f0 msgr2=0x562f4009bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 --2- 172.21.15.27:0/3198271978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcd22f0 0x562f4009bef0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffaac005fd0 tx=0x7ffaac00f730 comp rx=0 tx=0).stop 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 -- 172.21.15.27:0/3198271978 shutdown_connections 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 --2- 172.21.15.27:0/3198271978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 0x562f400a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 --2- 172.21.15.27:0/3198271978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcd22f0 0x562f4009bef0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 --2- 172.21.15.27:0/3198271978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcfb1b0 0x562f4009b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 -- 172.21.15.27:0/3198271978 >> 172.21.15.27:0/3198271978 conn(0x562f3fcef300 msgr2=0x562f40097300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffab9b0f640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ffaa00afbb0 0x7ffaa00b20c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40057c0 tx=0x7ffaa402a880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 -- 172.21.15.27:0/3198271978 shutdown_connections 2024-04-25T13:53:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.885+0000 7ffac2427740 1 -- 172.21.15.27:0/3198271978 wait complete. 2024-04-25T13:53:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.886+0000 7ffac2427740 1 -- start start 2024-04-25T13:53:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.886+0000 7ffac2427740 1 Processor -- start 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.886+0000 7ffac2427740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.886+0000 7ffac2427740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 0x562f400a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffac2427740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcfb1b0 0x562f400acb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab930e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab930e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45744/0 (socket says 172.21.15.27:45744) 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab930e640 1 -- 172.21.15.27:0/3920363935 learned_addr learned my addr 172.21.15.27:0/3920363935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562f400a3840 con 0x562f3fcd22f0 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562f3ff88b60 con 0x562f3fcda4d0 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562f400a39c0 con 0x562f3fcfb1b0 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 0x562f400a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab9b0f640 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcfb1b0 0x562f400acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffac2427740 10 client.0 _ll_register_callbacks cb 0x562f3ffad2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab8b0d640 1 -- 172.21.15.27:0/3920363935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcfb1b0 msgr2=0x562f400acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcfb1b0 0x562f400acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab8b0d640 1 -- 172.21.15.27:0/3920363935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 msgr2=0x562f400a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab8b0d640 1 -- 172.21.15.27:0/3920363935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f3fd13210 con 0x562f3fcda4d0 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab930e640 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 0x562f400a5c20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ffab00dcd70 tx=0x7ffab0005730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24ef96fd4785c551 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.887+0000 7ffa9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.888+0000 7ffa9cff9640 1 -- 172.21.15.27:0/3920363935 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab00c0020 con 0x562f3fcda4d0 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.888+0000 7ffa9cff9640 1 -- 172.21.15.27:0/3920363935 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ffab00a7850 con 0x562f3fcda4d0 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.888+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaac011a90 con 0x562f3fcda4d0 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.888+0000 7ffa9cff9640 1 -- 172.21.15.27:0/3920363935 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ffab00c6050 con 0x562f3fcda4d0 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.889+0000 7ffac2427740 10 client.27626 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.889+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562f3fddf620 con 0x562f3fcda4d0 2024-04-25T13:53:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.889+0000 7ffa9cff9640 1 -- 172.21.15.27:0/3920363935 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab00ab790 con 0x562f3fcda4d0 2024-04-25T13:53:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffa9cff9640 1 -- 172.21.15.27:0/3920363935 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ffab00a1b70 con 0x562f3fcda4d0 2024-04-25T13:53:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffac2427740 10 client.27626 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffac2427740 10 client.27626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffac2427740 20 client.27626 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffac2427740 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562f3fcb9d80 0x562f400b3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffac2427740 4 client.27626 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffaac011a90 con 0x562f3fcb9d80 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffab930e640 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562f3fcb9d80 0x562f400b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffab930e640 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562f3fcb9d80 0x562f400b3190 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77255c070ee5f17b server_cookie=56cb022f78ce4fdb in_seq=0 out_seq=0 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.890+0000 7ffa9cff9640 10 client.27626 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffa9cff9640 1 -- 172.21.15.27:0/3920363935 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7ffaac011a90 con 0x562f3fcb9d80 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffa9cff9640 10 client.27626 handle_command_reply: tid=0 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 client.27626 shutdown 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 msgr2=0x562f400a5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 0x562f400a5c20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ffab00dcd70 tx=0x7ffab0005730 comp rx=0 tx=0).stop 2024-04-25T13:53:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 shutdown_connections 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562f3fcb9d80 0x562f400b3190 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcfb1b0 0x562f400acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcda4d0 0x562f400a5c20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 --2- 172.21.15.27:0/3920363935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffa9cff9640 0 client.27626 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.891+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 >> 172.21.15.27:0/3920363935 conn(0x562f3fcef300 msgr2=0x562f4009a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.892+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 shutdown_connections 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.892+0000 7ffac2427740 1 -- 172.21.15.27:0/3920363935 wait complete. 2024-04-25T13:53:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.892+0000 7ffac2427740 20 client.27626 trim_cache size 0 max 16384 2024-04-25T13:53:29.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.915+0000 7ffac2427740 1 Processor -- start 2024-04-25T13:53:29.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.915+0000 7ffac2427740 1 -- start start 2024-04-25T13:53:29.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffac2427740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcb9d80 0x562f4009eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffac2427740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcd22f0 0x562f400acaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.916 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffac2427740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcda4d0 0x562f4009f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffac2427740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffab41b8080 con 0x562f3fcda4d0 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffac2427740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffab41ba900 con 0x562f3fcd22f0 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffac2427740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ffab41b86f0 con 0x562f3fcb9d80 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab9b0f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcda4d0 0x562f4009f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcb9d80 0x562f4009eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcb9d80 0x562f4009eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:51992/0 (socket says 172.21.15.27:51992) 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 -- 172.21.15.27:0/4066010040 learned_addr learned my addr 172.21.15.27:0/4066010040 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcd22f0 0x562f400acaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 -- 172.21.15.27:0/4066010040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcd22f0 msgr2=0x562f400acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcd22f0 0x562f400acaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 -- 172.21.15.27:0/4066010040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcda4d0 msgr2=0x562f4009f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcda4d0 0x562f4009f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 -- 172.21.15.27:0/4066010040 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f400a2ec0 con 0x562f3fcb9d80 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab9b0f640 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcda4d0 0x562f4009f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.916+0000 7ffab930e640 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcb9d80 0x562f4009eb80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009b60 tx=0x7ffaac000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fba03c443b2c4d8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:29.918 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.917+0000 7ffa9effd640 1 -- 172.21.15.27:0/4066010040 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac004ba0 con 0x562f3fcb9d80 2024-04-25T13:53:29.918 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.917+0000 7ffa9effd640 1 -- 172.21.15.27:0/4066010040 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaac02f070 con 0x562f3fcb9d80 2024-04-25T13:53:29.918 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.917+0000 7ffa9effd640 1 -- 172.21.15.27:0/4066010040 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0116a0 con 0x562f3fcb9d80 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 -- 172.21.15.27:0/4066010040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcb9d80 msgr2=0x562f4009eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcb9d80 0x562f4009eb80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009b60 tx=0x7ffaac000f40 comp rx=0 tx=0).stop 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 -- 172.21.15.27:0/4066010040 shutdown_connections 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcda4d0 0x562f4009f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcd22f0 0x562f400acaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 --2- 172.21.15.27:0/4066010040 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcb9d80 0x562f4009eb80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 -- 172.21.15.27:0/4066010040 >> 172.21.15.27:0/4066010040 conn(0x562f3fcef300 msgr2=0x562f4009fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 -- 172.21.15.27:0/4066010040 shutdown_connections 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 -- 172.21.15.27:0/4066010040 wait complete. 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.918+0000 7ffac2427740 1 -- start start 2024-04-25T13:53:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.919+0000 7ffac2427740 1 Processor -- start 2024-04-25T13:53:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffac2427740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcb9d80 0x562f400a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffac2427740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffac2427740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcda4d0 0x562f40119560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffac2427740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffab41b8dc0 con 0x562f3fcd22f0 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffac2427740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffab41ba230 con 0x562f3fcb9d80 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffac2427740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ffab41b9490 con 0x562f3fcda4d0 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffac2427740 10 client.0 _ll_register_callbacks cb 0x7ffab00a32a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffab930e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcb9d80 0x562f400a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffab930e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcb9d80 0x562f400a9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60712/0 (socket says 172.21.15.27:60712) 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffab930e640 1 -- 172.21.15.27:0/2218785872 learned_addr learned my addr 172.21.15.27:0/2218785872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffab9b0f640 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcda4d0 0x562f40119560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.920+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffab9b0f640 1 -- 172.21.15.27:0/2218785872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcb9d80 msgr2=0x562f400a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffab9b0f640 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcb9d80 0x562f400a9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffab9b0f640 1 -- 172.21.15.27:0/2218785872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 msgr2=0x562f400a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffab9b0f640 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffab9b0f640 1 -- 172.21.15.27:0/2218785872 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f40119aa0 con 0x562f3fcda4d0 2024-04-25T13:53:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffab8b0d640 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffab9b0f640 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcda4d0 0x562f40119560 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40304e0 tx=0x7ffaa402a380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a12d5032c74c719 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffa9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffa9cff9640 1 -- 172.21.15.27:0/2218785872 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4037040 con 0x562f3fcda4d0 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffa9cff9640 1 -- 172.21.15.27:0/2218785872 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ffaa4004020 con 0x562f3fcda4d0 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.921+0000 7ffac2427740 1 -- 172.21.15.27:0/2218785872 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaac02f070 con 0x562f3fcda4d0 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.922+0000 7ffa9cff9640 1 -- 172.21.15.27:0/2218785872 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ffaa4058070 con 0x562f3fcda4d0 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.922+0000 7ffac2427740 10 client.27658 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.922+0000 7ffac2427740 1 -- 172.21.15.27:0/2218785872 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562f4011aca0 con 0x562f3fcda4d0 2024-04-25T13:53:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.922+0000 7ffa9cff9640 1 -- 172.21.15.27:0/2218785872 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaac02f070 con 0x562f3fcda4d0 2024-04-25T13:53:29.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.922+0000 7ffa9cff9640 1 -- 172.21.15.27:0/2218785872 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ffaa4047040 con 0x562f3fcda4d0 2024-04-25T13:53:29.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.922+0000 7ffac2427740 10 client.27658 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:29.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.923+0000 7ffac2427740 10 client.27658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:29.924 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.923+0000 7ffac2427740 20 client.27658 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.923+0000 7ffac2427740 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562f3fcfb1b0 0x562f4011d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.923+0000 7ffac2427740 4 client.27658 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:53:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.923+0000 7ffac2427740 1 -- 172.21.15.27:0/2218785872 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562f4000f070 con 0x562f3fcfb1b0 2024-04-25T13:53:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.923+0000 7ffab930e640 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562f3fcfb1b0 0x562f4011d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.923+0000 7ffa9cff9640 10 client.27658 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.923+0000 7ffab930e640 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562f3fcfb1b0 0x562f4011d690 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15aa499466e2244 server_cookie=da856b05705830ba in_seq=0 out_seq=0 2024-04-25T13:53:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffa9cff9640 1 -- 172.21.15.27:0/2218785872 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562f4000f070 con 0x562f3fcfb1b0 2024-04-25T13:53:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffa9cff9640 10 client.27658 handle_command_reply: tid=0 2024-04-25T13:53:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 client.27658 shutdown 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 -- 172.21.15.27:0/2218785872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcda4d0 msgr2=0x562f40119560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcda4d0 0x562f40119560 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40304e0 tx=0x7ffaa402a380 comp rx=0 tx=0).stop 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 -- 172.21.15.27:0/2218785872 shutdown_connections 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562f3fcfb1b0 0x562f4011d690 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562f3fcda4d0 0x562f40119560 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f3fcd22f0 0x562f400a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 --2- 172.21.15.27:0/2218785872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f3fcb9d80 0x562f400a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffac2427740 1 -- 172.21.15.27:0/2218785872 >> 172.21.15.27:0/2218785872 conn(0x562f3fcef300 msgr2=0x562f4009eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.937+0000 7ffa9cff9640 0 client.27658 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.938+0000 7ffac2427740 1 -- 172.21.15.27:0/2218785872 shutdown_connections 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.938+0000 7ffac2427740 1 -- 172.21.15.27:0/2218785872 wait complete. 2024-04-25T13:53:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.938+0000 7ffac2427740 20 client.27658 trim_cache size 0 max 16384 2024-04-25T13:53:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:53:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:53:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:53:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 -- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ffaa00afbb0 msgr2=0x7ffaa00b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ffaa00afbb0 0x7ffaa00b20c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40057c0 tx=0x7ffaa402a880 comp rx=0 tx=0).stop 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 -- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab410b350 msgr2=0x7ffab41a95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab410b350 0x7ffab41a95c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e9c0 tx=0x7ffab000ee80 comp rx=0 tx=0).stop 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 -- 172.21.15.27:0/576676869 shutdown_connections 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ffaa00afbb0 0x7ffaa00b20c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffab4110070 0x7ffab41b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab410b350 0x7ffab41a95c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 --2- 172.21.15.27:0/576676869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffab410a420 0x7ffab41a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 -- 172.21.15.27:0/576676869 >> 172.21.15.27:0/576676869 conn(0x7ffab4070540 msgr2=0x7ffab410c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 -- 172.21.15.27:0/576676869 shutdown_connections 2024-04-25T13:53:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:29.939+0000 7ffabb572640 1 -- 172.21.15.27:0/576676869 wait complete. 2024-04-25T13:53:30.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:30 smithi122 ceph-mon[32843]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.7 MiB/s wr, 598 op/s 2024-04-25T13:53:30.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:30 smithi045 ceph-mon[32731]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.7 MiB/s wr, 598 op/s 2024-04-25T13:53:30.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:30 smithi027 ceph-mon[29140]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.7 MiB/s wr, 598 op/s 2024-04-25T13:53:32.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:32 smithi122 ceph-mon[32843]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.9 MiB/s wr, 709 op/s 2024-04-25T13:53:32.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:32 smithi045 ceph-mon[32731]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.9 MiB/s wr, 709 op/s 2024-04-25T13:53:32.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:32 smithi027 ceph-mon[29140]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.9 MiB/s wr, 709 op/s 2024-04-25T13:53:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 745.0 s, 1069.5 tps, lat 93.864 ms stddev 101.827 2024-04-25T13:53:34.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:34 smithi122 ceph-mon[32843]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.6 MiB/s wr, 748 op/s 2024-04-25T13:53:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:34 smithi045 ceph-mon[32731]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.6 MiB/s wr, 748 op/s 2024-04-25T13:53:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:34 smithi027 ceph-mon[29140]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.6 MiB/s wr, 748 op/s 2024-04-25T13:53:36.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:36 smithi122 ceph-mon[32843]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.4 MiB/s wr, 694 op/s 2024-04-25T13:53:36.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:36 smithi045 ceph-mon[32731]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.4 MiB/s wr, 694 op/s 2024-04-25T13:53:36.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:36 smithi027 ceph-mon[29140]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.4 MiB/s wr, 694 op/s 2024-04-25T13:53:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 750.0 s, 1419.4 tps, lat 71.092 ms stddev 78.658 2024-04-25T13:53:38.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:38 smithi122 ceph-mon[32843]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.1 MiB/s wr, 702 op/s 2024-04-25T13:53:38.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:38 smithi045 ceph-mon[32731]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.1 MiB/s wr, 702 op/s 2024-04-25T13:53:38.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:38 smithi027 ceph-mon[29140]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.1 MiB/s wr, 702 op/s 2024-04-25T13:53:40.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:40 smithi122 ceph-mon[32843]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.9 MiB/s wr, 683 op/s 2024-04-25T13:53:40.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:40 smithi045 ceph-mon[32731]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.9 MiB/s wr, 683 op/s 2024-04-25T13:53:40.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:40 smithi027 ceph-mon[29140]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.9 MiB/s wr, 683 op/s 2024-04-25T13:53:42.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:42 smithi122 ceph-mon[32843]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 6.5 MiB/s wr, 728 op/s 2024-04-25T13:53:42.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:42 smithi045 ceph-mon[32731]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 6.5 MiB/s wr, 728 op/s 2024-04-25T13:53:42.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:42 smithi027 ceph-mon[29140]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 6.5 MiB/s wr, 728 op/s 2024-04-25T13:53:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 755.0 s, 1711.2 tps, lat 57.930 ms stddev 69.781 2024-04-25T13:53:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 -- 172.21.15.27:0/2291687178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e0102a20 msgr2=0x7f96e0103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 --2- 172.21.15.27:0/2291687178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e0102a20 0x7f96e0103720 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009900 tx=0x7f96d002a410 comp rx=0 tx=0).stop 2024-04-25T13:53:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 -- 172.21.15.27:0/2291687178 shutdown_connections 2024-04-25T13:53:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 --2- 172.21.15.27:0/2291687178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e0108960 0x7f96e010ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 --2- 172.21.15.27:0/2291687178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e0103c60 0x7f96e0104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 --2- 172.21.15.27:0/2291687178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e0102a20 0x7f96e0103720 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 -- 172.21.15.27:0/2291687178 >> 172.21.15.27:0/2291687178 conn(0x7f96e00fdc10 msgr2=0x7f96e01000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 -- 172.21.15.27:0/2291687178 shutdown_connections 2024-04-25T13:53:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 -- 172.21.15.27:0/2291687178 wait complete. 2024-04-25T13:53:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 Processor -- start 2024-04-25T13:53:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.061+0000 7f96e6470640 1 -- start start 2024-04-25T13:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96e6470640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e0102a20 0x7f96e006ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96e6470640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e0103c60 0x7f96e019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96e6470640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e0108960 0x7f96e01a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96e6470640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f96e010e520 con 0x7f96e0103c60 2024-04-25T13:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96e6470640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96e010e3a0 con 0x7f96e0102a20 2024-04-25T13:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96e6470640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f96e010e6a0 con 0x7f96e0108960 2024-04-25T13:53:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e0102a20 0x7f96e006ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e0102a20 0x7f96e006ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:60438/0 (socket says 172.21.15.27:60438) 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96dffff640 1 -- 172.21.15.27:0/3520470201 learned_addr learned my addr 172.21.15.27:0/3520470201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96df7fe640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e0103c60 0x7f96e019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.062+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e0108960 0x7f96e01a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.063+0000 7f96dffff640 1 -- 172.21.15.27:0/3520470201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e0108960 msgr2=0x7f96e01a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.063+0000 7f96dffff640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e0108960 0x7f96e01a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.063+0000 7f96dffff640 1 -- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e0103c60 msgr2=0x7f96e019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.063+0000 7f96dffff640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e0103c60 0x7f96e019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.063+0000 7f96dffff640 1 -- 172.21.15.27:0/3520470201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e0198dd0 con 0x7f96e0102a20 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.063+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e0108960 0x7f96e01a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.064+0000 7f96dffff640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e0102a20 0x7f96e006ae50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f96d0006010 tx=0x7f96d0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.064+0000 7f96dd7fa640 1 -- 172.21.15.27:0/3520470201 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d003f030 con 0x7f96e0102a20 2024-04-25T13:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.064+0000 7f96dd7fa640 1 -- 172.21.15.27:0/3520470201 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d0031910 con 0x7f96e0102a20 2024-04-25T13:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.064+0000 7f96e6470640 1 -- 172.21.15.27:0/3520470201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e0199060 con 0x7f96e0102a20 2024-04-25T13:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.064+0000 7f96e6470640 1 -- 172.21.15.27:0/3520470201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e0199550 con 0x7f96e0102a20 2024-04-25T13:53:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.065+0000 7f96dd7fa640 1 -- 172.21.15.27:0/3520470201 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d003f030 con 0x7f96e0102a20 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96ed3f5740 1 Processor -- start 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96ed3f5740 1 -- start start 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a037970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a037ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a03edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555779f9d460 con 0x555779c764d0 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555779f57590 con 0x555779c971b0 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555779f99420 con 0x555779c6e2f0 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.067+0000 7f96dd7fa640 1 -- 172.21.15.27:0/3520470201 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f96d0042030 con 0x7f96e0102a20 2024-04-25T13:53:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a03edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96e4a0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a037970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a03edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60108/0 (socket says 172.21.15.27:60108) 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 -- 172.21.15.27:0/4113342207 learned_addr learned my addr 172.21.15.27:0/4113342207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 -- 172.21.15.27:0/4113342207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 msgr2=0x55577a037ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 --2- 172.21.15.27:0/4113342207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a037ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 -- 172.21.15.27:0/4113342207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 msgr2=0x55577a037970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 --2- 172.21.15.27:0/4113342207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a037970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 -- 172.21.15.27:0/4113342207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555779f640d0 con 0x555779c764d0 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/4113342207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a037970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96df7fe640 1 --2- 172.21.15.27:0/4113342207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a03edf0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009c20 tx=0x7f96d40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa944a8f3107767f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.068+0000 7f96dd7fa640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f96c80afcd0 0x7f96c80b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.069+0000 7f96c67fc640 1 -- 172.21.15.27:0/4113342207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4004030 con 0x555779c764d0 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.069+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f96c80afcd0 0x7f96c80b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.069+0000 7f96c67fc640 1 -- 172.21.15.27:0/4113342207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d40035f0 con 0x555779c764d0 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.069+0000 7f96c67fc640 1 -- 172.21.15.27:0/4113342207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d40126b0 con 0x555779c764d0 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.069+0000 7f96dd7fa640 1 -- 172.21.15.27:0/3520470201 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f96d00feca0 con 0x7f96e0102a20 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.069+0000 7f96ed3f5740 1 -- 172.21.15.27:0/4113342207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 msgr2=0x55577a03edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.069+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f96c80afcd0 0x7f96c80b21e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x55577a038b40 tx=0x7f96d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.069+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/4113342207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a03edf0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009c20 tx=0x7f96d40117b0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 -- 172.21.15.27:0/4113342207 shutdown_connections 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/4113342207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a03edf0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/4113342207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a037ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/4113342207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a037970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 -- 172.21.15.27:0/4113342207 >> 172.21.15.27:0/4113342207 conn(0x555779c8b300 msgr2=0x55577a033300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 -- 172.21.15.27:0/4113342207 shutdown_connections 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 -- 172.21.15.27:0/4113342207 wait complete. 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 -- start start 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.070+0000 7f96ed3f5740 1 Processor -- start 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c6e2f0 0x55577a0416b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a041bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c971b0 0x55577a048b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55577a03f840 con 0x555779c764d0 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555779f24b60 con 0x555779c6e2f0 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55577a03f9c0 con 0x555779c971b0 2024-04-25T13:53:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96ed3f5740 10 client.0 _ll_register_callbacks cb 0x555779f492b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a041bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a041bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60118/0 (socket says 172.21.15.27:60118) 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c6e2f0 0x55577a0416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 -- 172.21.15.27:0/1280421601 learned_addr learned my addr 172.21.15.27:0/1280421601 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c971b0 0x55577a048b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 -- 172.21.15.27:0/1280421601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c971b0 msgr2=0x55577a048b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c971b0 0x55577a048b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 -- 172.21.15.27:0/1280421601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c6e2f0 msgr2=0x55577a0416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c6e2f0 0x55577a0416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 -- 172.21.15.27:0/1280421601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555779caf210 con 0x555779c764d0 2024-04-25T13:53:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96dffff640 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c6e2f0 0x55577a0416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96df7fe640 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a041bf0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f96d40057c0 tx=0x7f96d4004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442703a8aa2794b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1280421601 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d401f040 con 0x555779c764d0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1280421601 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f96d4004480 con 0x555779c764d0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.071+0000 7f96ed3f5740 1 -- 172.21.15.27:0/1280421601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96d40035f0 con 0x555779c764d0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.072+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1280421601 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f96d4005050 con 0x555779c764d0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.072+0000 7f96ed3f5740 10 client.17937 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.072+0000 7f96ed3f5740 1 -- 172.21.15.27:0/1280421601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555779d7b620 con 0x555779c764d0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.072+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1280421601 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d40035f0 con 0x555779c764d0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.072+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1280421601 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f96d4012450 con 0x555779c764d0 2024-04-25T13:53:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96ed3f5740 10 client.17937 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96ed3f5740 10 client.17937 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96ed3f5740 20 client.17937 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555779c55d80 0x55577a04efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96ed3f5740 4 client.17937 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96ed3f5740 1 -- 172.21.15.27:0/1280421601 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555779fab070 con 0x555779c55d80 2024-04-25T13:53:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96dffff640 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555779c55d80 0x55577a04efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.080 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96dffff640 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555779c55d80 0x55577a04efd0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8492503d315cbc29 server_cookie=f8a1ebfcba947f4e in_seq=0 out_seq=0 2024-04-25T13:53:43.080 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.073+0000 7f96bf7fe640 10 client.17937 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:43.080 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.074+0000 7f96bf7fe640 1 -- 172.21.15.27:0/1280421601 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555779fab070 con 0x555779c55d80 2024-04-25T13:53:43.080 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.074+0000 7f96bf7fe640 10 client.17937 handle_command_reply: tid=0 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.075+0000 7f96ed3f5740 1 client.17937 shutdown 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.075+0000 7f96ed3f5740 1 -- 172.21.15.27:0/1280421601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 msgr2=0x55577a041bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.075+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a041bf0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f96d40057c0 tx=0x7f96d4004260 comp rx=0 tx=0).stop 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 1 -- 172.21.15.27:0/1280421601 shutdown_connections 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555779c55d80 0x55577a04efd0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c971b0 0x55577a048b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a041bf0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/1280421601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c6e2f0 0x55577a0416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 1 -- 172.21.15.27:0/1280421601 >> 172.21.15.27:0/1280421601 conn(0x555779c8b300 msgr2=0x55577a032250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96bf7fe640 0 client.17937 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 1 -- 172.21.15.27:0/1280421601 shutdown_connections 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 1 -- 172.21.15.27:0/1280421601 wait complete. 2024-04-25T13:53:43.081 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.076+0000 7f96ed3f5740 20 client.17937 trim_cache size 0 max 16384 2024-04-25T13:53:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.101+0000 7f96ed3f5740 1 Processor -- start 2024-04-25T13:53:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.101+0000 7f96ed3f5740 1 -- start start 2024-04-25T13:53:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.101+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a032cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.101+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c764d0 0x55577a048a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.101+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c971b0 0x55577a033200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.102+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f96e01ab3f0 con 0x555779c971b0 2024-04-25T13:53:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.102+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96e01adc70 con 0x555779c764d0 2024-04-25T13:53:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.102+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f96e01aba60 con 0x555779c6e2f0 2024-04-25T13:53:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.106+0000 7f96e4a0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c971b0 0x55577a033200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.106+0000 7f96e4a0d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c971b0 0x55577a033200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60120/0 (socket says 172.21.15.27:60120) 2024-04-25T13:53:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.106+0000 7f96e4a0d640 1 -- 172.21.15.27:0/3989755407 learned_addr learned my addr 172.21.15.27:0/3989755407 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.107+0000 7f96dffff640 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a032cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.109+0000 7f96e4a0d640 1 -- 172.21.15.27:0/3989755407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 msgr2=0x55577a032cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.109+0000 7f96df7fe640 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c764d0 0x55577a048a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.109+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a032cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.109+0000 7f96e4a0d640 1 -- 172.21.15.27:0/3989755407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c764d0 msgr2=0x55577a048a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.109+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c764d0 0x55577a048a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.109+0000 7f96e4a0d640 1 -- 172.21.15.27:0/3989755407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55577a048fb0 con 0x555779c971b0 2024-04-25T13:53:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.109+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c971b0 0x55577a033200 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f96d8012b90 tx=0x7f96d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3b0b8c33faff3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.110+0000 7f96c5ffb640 1 -- 172.21.15.27:0/3989755407 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8020df0 con 0x555779c971b0 2024-04-25T13:53:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.110+0000 7f96c5ffb640 1 -- 172.21.15.27:0/3989755407 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d8005020 con 0x555779c971b0 2024-04-25T13:53:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.110+0000 7f96c5ffb640 1 -- 172.21.15.27:0/3989755407 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d801b600 con 0x555779c971b0 2024-04-25T13:53:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.111+0000 7f96ed3f5740 1 -- 172.21.15.27:0/3989755407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c971b0 msgr2=0x55577a033200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.111+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c971b0 0x55577a033200 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f96d8012b90 tx=0x7f96d8014000 comp rx=0 tx=0).stop 2024-04-25T13:53:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.113+0000 7f96ed3f5740 1 -- 172.21.15.27:0/3989755407 shutdown_connections 2024-04-25T13:53:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.113+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c971b0 0x55577a033200 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.113+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c764d0 0x55577a048a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.113+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/3989755407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a032cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.113+0000 7f96ed3f5740 1 -- 172.21.15.27:0/3989755407 >> 172.21.15.27:0/3989755407 conn(0x555779c8b300 msgr2=0x55577a03bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:43.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.116+0000 7f96ed3f5740 1 -- 172.21.15.27:0/3989755407 shutdown_connections 2024-04-25T13:53:43.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.120+0000 7f96ed3f5740 1 -- 172.21.15.27:0/3989755407 wait complete. 2024-04-25T13:53:43.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.120+0000 7f96ed3f5740 1 -- start start 2024-04-25T13:53:43.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.120+0000 7f96ed3f5740 1 Processor -- start 2024-04-25T13:53:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.121+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a045880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.121+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a045dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.121+0000 7f96ed3f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a0c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.121+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f96e01ac130 con 0x555779c764d0 2024-04-25T13:53:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.121+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96e01ad5a0 con 0x555779c971b0 2024-04-25T13:53:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.121+0000 7f96ed3f5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f96e01ac800 con 0x555779c6e2f0 2024-04-25T13:53:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.121+0000 7f96ed3f5740 10 client.0 _ll_register_callbacks cb 0x7f96d4009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.122+0000 7f96e4a0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a0c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.123+0000 7f96df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a045dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.123+0000 7f96df7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a045dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60128/0 (socket says 172.21.15.27:60128) 2024-04-25T13:53:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.123+0000 7f96df7fe640 1 -- 172.21.15.27:0/847171821 learned_addr learned my addr 172.21.15.27:0/847171821 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:43.124 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.124+0000 7f96dffff640 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a045880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.126+0000 7f96e4a0d640 1 -- 172.21.15.27:0/847171821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 msgr2=0x55577a045880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.126+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a045880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.126+0000 7f96e4a0d640 1 -- 172.21.15.27:0/847171821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 msgr2=0x55577a045dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.126+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a045dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.126+0000 7f96e4a0d640 1 -- 172.21.15.27:0/847171821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55577a0463d0 con 0x555779c971b0 2024-04-25T13:53:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.127+0000 7f96df7fe640 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a045dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:43.128 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.128+0000 7f96e4a0d640 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a0c62d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f96d8012b60 tx=0x7f96d8004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5acff623ab12f628 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:43.128 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.129+0000 7f96bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:43.128 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.129+0000 7f96bf7fe640 1 -- 172.21.15.27:0/847171821 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8037020 con 0x555779c971b0 2024-04-25T13:53:43.128 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.129+0000 7f96bf7fe640 1 -- 172.21.15.27:0/847171821 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f96d80053d0 con 0x555779c971b0 2024-04-25T13:53:43.129 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.129+0000 7f96ed3f5740 1 -- 172.21.15.27:0/847171821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96d8005020 con 0x555779c971b0 2024-04-25T13:53:43.130 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.131+0000 7f96bf7fe640 1 -- 172.21.15.27:0/847171821 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f96d802d040 con 0x555779c971b0 2024-04-25T13:53:43.130 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.131+0000 7f96ed3f5740 10 client.27644 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:43.131 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.131+0000 7f96ed3f5740 1 -- 172.21.15.27:0/847171821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55577a0c78a0 con 0x555779c971b0 2024-04-25T13:53:43.131 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.131+0000 7f96bf7fe640 1 -- 172.21.15.27:0/847171821 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d8005020 con 0x555779c971b0 2024-04-25T13:53:43.131 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.132+0000 7f96bf7fe640 1 -- 172.21.15.27:0/847171821 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f96d8027070 con 0x555779c971b0 2024-04-25T13:53:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.132+0000 7f96ed3f5740 10 client.27644 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.132+0000 7f96ed3f5740 10 client.27644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.132+0000 7f96ed3f5740 20 client.27644 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.132+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555779c2b1c0 0x55577a0ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.132+0000 7f96ed3f5740 4 client.27644 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:53:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.132+0000 7f96ed3f5740 1 -- 172.21.15.27:0/847171821 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555779fa4ee0 con 0x555779c2b1c0 2024-04-25T13:53:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.134+0000 7f96dffff640 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555779c2b1c0 0x55577a0ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.135+0000 7f96bf7fe640 10 client.27644 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.135+0000 7f96dffff640 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555779c2b1c0 0x55577a0ca210 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a038d50ce575aca6 server_cookie=6d7a745ccef2f26d in_seq=0 out_seq=0 2024-04-25T13:53:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96bf7fe640 1 -- 172.21.15.27:0/847171821 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555779fa4ee0 con 0x555779c2b1c0 2024-04-25T13:53:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96bf7fe640 10 client.27644 handle_command_reply: tid=0 2024-04-25T13:53:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 client.27644 shutdown 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 -- 172.21.15.27:0/847171821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 msgr2=0x55577a0c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a0c62d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f96d8012b60 tx=0x7f96d8004c70 comp rx=0 tx=0).stop 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 -- 172.21.15.27:0/847171821 shutdown_connections 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555779c2b1c0 0x55577a0ca210 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555779c971b0 0x55577a0c62d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555779c764d0 0x55577a045dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 --2- 172.21.15.27:0/847171821 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555779c6e2f0 0x55577a045880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96ed3f5740 1 -- 172.21.15.27:0/847171821 >> 172.21.15.27:0/847171821 conn(0x555779c8b300 msgr2=0x55577a032be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.149+0000 7f96bf7fe640 0 client.27644 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96ed3f5740 1 -- 172.21.15.27:0/847171821 shutdown_connections 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96ed3f5740 1 -- 172.21.15.27:0/847171821 wait complete. 2024-04-25T13:53:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96ed3f5740 20 client.27644 trim_cache size 0 max 16384 2024-04-25T13:53:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:53:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:53:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:53:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 -- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f96c80afcd0 msgr2=0x7f96c80b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f96c80afcd0 0x7f96c80b21e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x55577a038b40 tx=0x7f96d8009500 comp rx=0 tx=0).stop 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 -- 172.21.15.27:0/3520470201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e0102a20 msgr2=0x7f96e006ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e0102a20 0x7f96e006ae50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f96d0006010 tx=0x7f96d0004270 comp rx=0 tx=0).stop 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 -- 172.21.15.27:0/3520470201 shutdown_connections 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f96c80afcd0 0x7f96c80b21e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f96e0108960 0x7f96e01a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96e0103c60 0x7f96e019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 --2- 172.21.15.27:0/3520470201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e0102a20 0x7f96e006ae50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 -- 172.21.15.27:0/3520470201 >> 172.21.15.27:0/3520470201 conn(0x7f96e00fdc10 msgr2=0x7f96e0104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 -- 172.21.15.27:0/3520470201 shutdown_connections 2024-04-25T13:53:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:43.150+0000 7f96e6470640 1 -- 172.21.15.27:0/3520470201 wait complete. 2024-04-25T13:53:44.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:44 smithi122 ceph-mon[32843]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 6.0 MiB/s wr, 678 op/s 2024-04-25T13:53:44.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:44 smithi045 ceph-mon[32731]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 6.0 MiB/s wr, 678 op/s 2024-04-25T13:53:44.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:44 smithi027 ceph-mon[29140]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 6.0 MiB/s wr, 678 op/s 2024-04-25T13:53:46.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:46 smithi122 ceph-mon[32843]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.1 MiB/s wr, 544 op/s 2024-04-25T13:53:46.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:46 smithi045 ceph-mon[32731]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.1 MiB/s wr, 544 op/s 2024-04-25T13:53:46.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:46 smithi027 ceph-mon[29140]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.1 MiB/s wr, 544 op/s 2024-04-25T13:53:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 760.0 s, 1649.6 tps, lat 61.130 ms stddev 64.954 2024-04-25T13:53:48.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:48 smithi122 ceph-mon[32843]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-04-25T13:53:48.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:48 smithi045 ceph-mon[32731]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-04-25T13:53:48.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:48 smithi027 ceph-mon[29140]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-04-25T13:53:50.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:50 smithi122 ceph-mon[32843]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.4 MiB/s wr, 589 op/s 2024-04-25T13:53:50.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:50 smithi045 ceph-mon[32731]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.4 MiB/s wr, 589 op/s 2024-04-25T13:53:50.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:50 smithi027 ceph-mon[29140]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.4 MiB/s wr, 589 op/s 2024-04-25T13:53:52.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:52 smithi122 ceph-mon[32843]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.8 MiB/s wr, 617 op/s 2024-04-25T13:53:52.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:52 smithi045 ceph-mon[32731]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.8 MiB/s wr, 617 op/s 2024-04-25T13:53:52.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:52 smithi027 ceph-mon[29140]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.8 MiB/s wr, 617 op/s 2024-04-25T13:53:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 765.0 s, 2269.0 tps, lat 44.158 ms stddev 51.555 2024-04-25T13:53:54.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:54 smithi122 ceph-mon[32843]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.7 MiB/s wr, 620 op/s 2024-04-25T13:53:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:54 smithi045 ceph-mon[32731]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.7 MiB/s wr, 620 op/s 2024-04-25T13:53:54.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:54 smithi027 ceph-mon[29140]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.7 MiB/s wr, 620 op/s 2024-04-25T13:53:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.294+0000 7f2a32d38640 1 -- 172.21.15.27:0/254612753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c1021c0 msgr2=0x7f2a2c102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.294+0000 7f2a32d38640 1 --2- 172.21.15.27:0/254612753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c1021c0 0x7f2a2c102ec0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000dc90 tx=0x7f2a2002e370 comp rx=0 tx=0).stop 2024-04-25T13:53:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.294+0000 7f2a32d38640 1 -- 172.21.15.27:0/254612753 shutdown_connections 2024-04-25T13:53:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.294+0000 7f2a32d38640 1 --2- 172.21.15.27:0/254612753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a2c108100 0x7f2a2c10a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.294+0000 7f2a32d38640 1 --2- 172.21.15.27:0/254612753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2c103400 0x7f2a2c103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.294+0000 7f2a32d38640 1 --2- 172.21.15.27:0/254612753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c1021c0 0x7f2a2c102ec0 secure :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000dc90 tx=0x7f2a2002e370 comp rx=0 tx=0).stop 2024-04-25T13:53:56.295 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.294+0000 7f2a32d38640 1 -- 172.21.15.27:0/254612753 >> 172.21.15.27:0/254612753 conn(0x7f2a2c0fd390 msgr2=0x7f2a2c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.295+0000 7f2a32d38640 1 -- 172.21.15.27:0/254612753 shutdown_connections 2024-04-25T13:53:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.295+0000 7f2a32d38640 1 -- 172.21.15.27:0/254612753 wait complete. 2024-04-25T13:53:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a32d38640 1 Processor -- start 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a32d38640 1 -- start start 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a32d38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2c103400 0x7f2a2c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a32d38640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c108100 0x7f2a2c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a32d38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a2c19d5a0 0x7f2a2c19f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a32d38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c10d450 con 0x7f2a2c19d5a0 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a32d38640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c10d2d0 con 0x7f2a2c103400 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a32d38640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c10d5d0 con 0x7f2a2c108100 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a30ad4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2c103400 0x7f2a2c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a2bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c108100 0x7f2a2c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a30ad4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2c103400 0x7f2a2c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56626/0 (socket says 172.21.15.27:56626) 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.296+0000 7f2a30ad4640 1 -- 172.21.15.27:0/2175348701 learned_addr learned my addr 172.21.15.27:0/2175348701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.297+0000 7f2a312d5640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a2c19d5a0 0x7f2a2c19f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.297+0000 7f2a2bfff640 1 -- 172.21.15.27:0/2175348701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2c103400 msgr2=0x7f2a2c1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.297+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2c103400 0x7f2a2c1980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.297+0000 7f2a2bfff640 1 -- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a2c19d5a0 msgr2=0x7f2a2c19f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.297+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a2c19d5a0 0x7f2a2c19f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.297+0000 7f2a2bfff640 1 -- 172.21.15.27:0/2175348701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c1a00f0 con 0x7f2a2c108100 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.297+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2c103400 0x7f2a2c1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.298+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c108100 0x7f2a2c1986a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800e9f0 tx=0x7f2a1800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.298+0000 7f2a29ffb640 1 -- 172.21.15.27:0/2175348701 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1800ccc0 con 0x7f2a2c108100 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.298+0000 7f2a29ffb640 1 -- 172.21.15.27:0/2175348701 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a18005020 con 0x7f2a2c108100 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.298+0000 7f2a29ffb640 1 -- 172.21.15.27:0/2175348701 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1800ccc0 con 0x7f2a2c108100 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.299+0000 7f2a32d38640 1 -- 172.21.15.27:0/2175348701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2c1a03e0 con 0x7f2a2c108100 2024-04-25T13:53:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.299+0000 7f2a32d38640 1 -- 172.21.15.27:0/2175348701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a2c1a08e0 con 0x7f2a2c108100 2024-04-25T13:53:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a29ffb640 1 -- 172.21.15.27:0/2175348701 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2a180046c0 con 0x7f2a2c108100 2024-04-25T13:53:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a39bef740 1 Processor -- start 2024-04-25T13:53:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a39bef740 1 -- start start 2024-04-25T13:53:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17441b0 0x5628b1ae4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b171b2f0 0x5628b1ae4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1aebdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a39bef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5628b1a4a460 con 0x5628b17234d0 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a39bef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5628b1a04590 con 0x5628b171b2f0 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a39bef740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5628b1a46420 con 0x5628b17441b0 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a312d5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17441b0 0x5628b1ae4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a312d5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17441b0 0x5628b1ae4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33272/0 (socket says 172.21.15.27:33272) 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.303+0000 7f2a312d5640 1 -- 172.21.15.27:0/1068589758 learned_addr learned my addr 172.21.15.27:0/1068589758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1aebdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a29ffb640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a1c0afcd0 0x7f2a1c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b171b2f0 0x5628b1ae4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a312d5640 1 -- 172.21.15.27:0/1068589758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b171b2f0 msgr2=0x5628b1ae4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a312d5640 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b171b2f0 0x5628b1ae4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a312d5640 1 -- 172.21.15.27:0/1068589758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 msgr2=0x5628b1aebdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a312d5640 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1aebdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a29ffb640 1 -- 172.21.15.27:0/2175348701 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2a180de790 con 0x7f2a2c108100 2024-04-25T13:53:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a312d5640 1 -- 172.21.15.27:0/1068589758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b1a110d0 con 0x5628b17441b0 2024-04-25T13:53:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.304+0000 7f2a312d5640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a1c0afcd0 0x7f2a1c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.305+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b171b2f0 0x5628b1ae4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.305+0000 7f2a312d5640 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17441b0 0x5628b1ae4970 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2a240104f0 tx=0x7f2a240109b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf4a5bf2788f10b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.305+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1aebdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.305+0000 7f2a16ffd640 1 -- 172.21.15.27:0/1068589758 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24022070 con 0x5628b17441b0 2024-04-25T13:53:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.305+0000 7f2a16ffd640 1 -- 172.21.15.27:0/1068589758 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a24013e30 con 0x5628b17441b0 2024-04-25T13:53:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.305+0000 7f2a16ffd640 1 -- 172.21.15.27:0/1068589758 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24022070 con 0x5628b17441b0 2024-04-25T13:53:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 -- 172.21.15.27:0/1068589758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17441b0 msgr2=0x5628b1ae4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17441b0 0x5628b1ae4970 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2a240104f0 tx=0x7f2a240109b0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a312d5640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a1c0afcd0 0x7f2a1c0b21e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2a2401f000 tx=0x7f2a2400eb80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 -- 172.21.15.27:0/1068589758 shutdown_connections 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1aebdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b171b2f0 0x5628b1ae4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1068589758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17441b0 0x5628b1ae4970 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 -- 172.21.15.27:0/1068589758 >> 172.21.15.27:0/1068589758 conn(0x5628b1738300 msgr2=0x5628b1ae0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 -- 172.21.15.27:0/1068589758 shutdown_connections 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.306+0000 7f2a39bef740 1 -- 172.21.15.27:0/1068589758 wait complete. 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.307+0000 7f2a39bef740 1 -- start start 2024-04-25T13:53:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.307+0000 7f2a39bef740 1 Processor -- start 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1aee620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17234d0 0x5628b1aeeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17441b0 0x5628b1af5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a39bef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5628b1aec840 con 0x5628b17441b0 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a39bef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5628b19d1b60 con 0x5628b17234d0 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a39bef740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5628b1aec9c0 con 0x5628b171b2f0 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a39bef740 10 client.0 _ll_register_callbacks cb 0x5628b19f62b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a30ad4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1aee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a30ad4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1aee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33274/0 (socket says 172.21.15.27:33274) 2024-04-25T13:53:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a30ad4640 1 -- 172.21.15.27:0/1834532270 learned_addr learned my addr 172.21.15.27:0/1834532270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.308+0000 7f2a312d5640 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17441b0 0x5628b1af5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.309+0000 7f2a30ad4640 1 -- 172.21.15.27:0/1834532270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17234d0 msgr2=0x5628b1aeeb60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:53:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.309+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17234d0 0x5628b1aeeb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.309+0000 7f2a30ad4640 1 -- 172.21.15.27:0/1834532270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17441b0 msgr2=0x5628b1af5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.309+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17441b0 0x5628b1af5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.309+0000 7f2a30ad4640 1 -- 172.21.15.27:0/1834532270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b175c210 con 0x5628b171b2f0 2024-04-25T13:53:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.309+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1aee620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2a20004bd0 tx=0x7f2a2000b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a192509f138f5de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.310+0000 7f2a0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:53:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.310+0000 7f2a0bfff640 1 -- 172.21.15.27:0/1834532270 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20046070 con 0x5628b171b2f0 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.310+0000 7f2a0bfff640 1 -- 172.21.15.27:0/1834532270 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2a20004240 con 0x5628b171b2f0 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.310+0000 7f2a39bef740 1 -- 172.21.15.27:0/1834532270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a24013e30 con 0x5628b171b2f0 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.310+0000 7f2a0bfff640 1 -- 172.21.15.27:0/1834532270 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2a2003b040 con 0x5628b171b2f0 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.310+0000 7f2a39bef740 10 client.27679 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.310+0000 7f2a39bef740 1 -- 172.21.15.27:0/1834532270 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5628b1828620 con 0x5628b171b2f0 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.311+0000 7f2a0bfff640 1 -- 172.21.15.27:0/1834532270 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a24013e30 con 0x5628b171b2f0 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.311+0000 7f2a0bfff640 1 -- 172.21.15.27:0/1834532270 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2a20003710 con 0x5628b171b2f0 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.311+0000 7f2a39bef740 10 client.27679 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.311+0000 7f2a39bef740 10 client.27679 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.311+0000 7f2a39bef740 20 client.27679 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.311+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628b1702d80 0x5628b1b0d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.311+0000 7f2a39bef740 4 client.27679 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.311+0000 7f2a39bef740 1 -- 172.21.15.27:0/1834532270 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628b1a58070 con 0x5628b1702d80 2024-04-25T13:53:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.312+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628b1702d80 0x5628b1b0d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.313+0000 7f2a0bfff640 10 client.27679 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.313+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628b1702d80 0x5628b1b0d330 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6d4801476fb4256 server_cookie=a5fa3f1f08a5d367 in_seq=0 out_seq=0 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a0bfff640 1 -- 172.21.15.27:0/1834532270 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5628b1a58070 con 0x5628b1702d80 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a0bfff640 10 client.27679 handle_command_reply: tid=0 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 client.27679 shutdown 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 -- 172.21.15.27:0/1834532270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 msgr2=0x5628b1aee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1aee620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2a20004bd0 tx=0x7f2a2000b6e0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 -- 172.21.15.27:0/1834532270 shutdown_connections 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628b1702d80 0x5628b1b0d330 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17441b0 0x5628b1af5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17234d0 0x5628b1aeeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1834532270 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1aee620 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a0bfff640 0 client.27679 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 -- 172.21.15.27:0/1834532270 >> 172.21.15.27:0/1834532270 conn(0x5628b1738300 msgr2=0x5628b1adf250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 -- 172.21.15.27:0/1834532270 shutdown_connections 2024-04-25T13:53:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.314+0000 7f2a39bef740 1 -- 172.21.15.27:0/1834532270 wait complete. 2024-04-25T13:53:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.315+0000 7f2a39bef740 20 client.27679 trim_cache size 0 max 16384 2024-04-25T13:53:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.342+0000 7f2a39bef740 1 Processor -- start 2024-04-25T13:53:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.342+0000 7f2a39bef740 1 -- start start 2024-04-25T13:53:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.342+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b171b2f0 0x5628b1adfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.342+0000 7f2a30ad4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b171b2f0 0x5628b1adfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.342+0000 7f2a30ad4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b171b2f0 0x5628b1adfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38306/0 (socket says 172.21.15.27:38306) 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.342+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17234d0 0x5628b1ae0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a30ad4640 1 -- 172.21.15.27:0/1323828876 learned_addr learned my addr 172.21.15.27:0/1323828876 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1aee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a39bef740 1 -- 172.21.15.27:0/1323828876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c0730b0 con 0x5628b171b2f0 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a39bef740 1 -- 172.21.15.27:0/1323828876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c1a1700 con 0x5628b17441b0 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a39bef740 1 -- 172.21.15.27:0/1323828876 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c073720 con 0x5628b17234d0 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17234d0 0x5628b1ae0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a312d5640 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1aee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a30ad4640 1 -- 172.21.15.27:0/1323828876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17234d0 msgr2=0x5628b1ae0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17234d0 0x5628b1ae0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.344 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a30ad4640 1 -- 172.21.15.27:0/1323828876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 msgr2=0x5628b1aee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1aee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a30ad4640 1 -- 172.21.15.27:0/1323828876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b1aebe60 con 0x5628b171b2f0 2024-04-25T13:53:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.343+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b171b2f0 0x5628b1adfcc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2a20003c50 tx=0x7f2a2000d670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58027432b25fd71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.344+0000 7f2a167fc640 1 -- 172.21.15.27:0/1323828876 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20046040 con 0x5628b171b2f0 2024-04-25T13:53:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.344+0000 7f2a167fc640 1 -- 172.21.15.27:0/1323828876 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a200037f0 con 0x5628b171b2f0 2024-04-25T13:53:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.344+0000 7f2a167fc640 1 -- 172.21.15.27:0/1323828876 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20004300 con 0x5628b171b2f0 2024-04-25T13:53:56.345 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.346+0000 7f2a39bef740 1 -- 172.21.15.27:0/1323828876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b171b2f0 msgr2=0x5628b1adfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.346+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b171b2f0 0x5628b1adfcc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2a20003c50 tx=0x7f2a2000d670 comp rx=0 tx=0).stop 2024-04-25T13:53:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.346+0000 7f2a39bef740 1 -- 172.21.15.27:0/1323828876 shutdown_connections 2024-04-25T13:53:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.346+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1aee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.346+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b17234d0 0x5628b1ae0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.346+0000 7f2a39bef740 1 --2- 172.21.15.27:0/1323828876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b171b2f0 0x5628b1adfcc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.346 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.346+0000 7f2a39bef740 1 -- 172.21.15.27:0/1323828876 >> 172.21.15.27:0/1323828876 conn(0x5628b1738300 msgr2=0x5628b1ae8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.347+0000 7f2a39bef740 1 -- 172.21.15.27:0/1323828876 shutdown_connections 2024-04-25T13:53:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.347+0000 7f2a39bef740 1 -- 172.21.15.27:0/1323828876 wait complete. 2024-04-25T13:53:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.347+0000 7f2a39bef740 1 -- start start 2024-04-25T13:53:56.347 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.348+0000 7f2a39bef740 1 Processor -- start 2024-04-25T13:53:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.348+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1af2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.348+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1af30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.348+0000 7f2a39bef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1af4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.348+0000 7f2a39bef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c073df0 con 0x5628b17234d0 2024-04-25T13:53:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.348+0000 7f2a39bef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c1a1030 con 0x5628b17441b0 2024-04-25T13:53:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.348+0000 7f2a39bef740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c0744c0 con 0x5628b171b2f0 2024-04-25T13:53:56.348 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.348+0000 7f2a39bef740 10 client.0 _ll_register_callbacks cb 0x7f2a18005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:56.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.353+0000 7f2a312d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1af4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.353+0000 7f2a312d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1af4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56666/0 (socket says 172.21.15.27:56666) 2024-04-25T13:53:56.352 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.353+0000 7f2a312d5640 1 -- 172.21.15.27:0/3967340240 learned_addr learned my addr 172.21.15.27:0/3967340240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:56.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.353+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1af30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.353 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.353+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1af2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.358+0000 7f2a30ad4640 1 -- 172.21.15.27:0/3967340240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 msgr2=0x5628b1af4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.358+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1af4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.358+0000 7f2a30ad4640 1 -- 172.21.15.27:0/3967340240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 msgr2=0x5628b1af30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:56.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.358+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1af30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.358 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.358+0000 7f2a30ad4640 1 -- 172.21.15.27:0/3967340240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b1af45d0 con 0x5628b171b2f0 2024-04-25T13:53:56.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.362+0000 7f2a30ad4640 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1af2ba0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x5628b1aef9d0 tx=0x7f2a20005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a37b064935dcdb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:56.361 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.362+0000 7f2a0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:53:56.364 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.364+0000 7f2a0bfff640 1 -- 172.21.15.27:0/3967340240 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20046040 con 0x5628b171b2f0 2024-04-25T13:53:56.364 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.364+0000 7f2a0bfff640 1 -- 172.21.15.27:0/3967340240 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2a200035f0 con 0x5628b171b2f0 2024-04-25T13:53:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.366+0000 7f2a39bef740 1 -- 172.21.15.27:0/3967340240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a200037f0 con 0x5628b171b2f0 2024-04-25T13:53:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.369+0000 7f2a0bfff640 1 -- 172.21.15.27:0/3967340240 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2a20063050 con 0x5628b171b2f0 2024-04-25T13:53:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.370+0000 7f2a39bef740 10 client.27685 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:56.369 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.370+0000 7f2a39bef740 1 -- 172.21.15.27:0/3967340240 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5628b1b6f850 con 0x5628b171b2f0 2024-04-25T13:53:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.374+0000 7f2a0bfff640 1 -- 172.21.15.27:0/3967340240 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a200037f0 con 0x5628b171b2f0 2024-04-25T13:53:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.374+0000 7f2a0bfff640 1 -- 172.21.15.27:0/3967340240 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2a200050e0 con 0x5628b171b2f0 2024-04-25T13:53:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.374+0000 7f2a39bef740 10 client.27685 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.374+0000 7f2a39bef740 10 client.27685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.374+0000 7f2a39bef740 20 client.27685 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.374+0000 7f2a39bef740 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628b16d81c0 0x5628b1b721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.374+0000 7f2a39bef740 4 client.27685 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:53:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.374+0000 7f2a39bef740 1 -- 172.21.15.27:0/3967340240 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5628b1a51ee0 con 0x5628b16d81c0 2024-04-25T13:53:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.379+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628b16d81c0 0x5628b1b721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.380+0000 7f2a2bfff640 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628b16d81c0 0x5628b1b721c0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de9d9f8002320fd6 server_cookie=66567e0ed7838ce8 in_seq=0 out_seq=0 2024-04-25T13:53:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.380+0000 7f2a0bfff640 10 client.27685 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:56.422 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.422+0000 7f2a0bfff640 1 -- 172.21.15.27:0/3967340240 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5628b1a51ee0 con 0x5628b16d81c0 2024-04-25T13:53:56.422 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.422+0000 7f2a0bfff640 10 client.27685 handle_command_reply: tid=0 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.422+0000 7f2a39bef740 1 client.27685 shutdown 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.422+0000 7f2a39bef740 1 -- 172.21.15.27:0/3967340240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 msgr2=0x5628b1af2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.422+0000 7f2a39bef740 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1af2ba0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x5628b1aef9d0 tx=0x7f2a20005fd0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 1 -- 172.21.15.27:0/3967340240 shutdown_connections 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628b16d81c0 0x5628b1b721c0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628b17441b0 0x5628b1af4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628b17234d0 0x5628b1af30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 1 --2- 172.21.15.27:0/3967340240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628b171b2f0 0x5628b1af2ba0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 1 -- 172.21.15.27:0/3967340240 >> 172.21.15.27:0/3967340240 conn(0x5628b1738300 msgr2=0x5628b1ae8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a0bfff640 0 client.27685 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 1 -- 172.21.15.27:0/3967340240 shutdown_connections 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 1 -- 172.21.15.27:0/3967340240 wait complete. 2024-04-25T13:53:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.423+0000 7f2a39bef740 20 client.27685 trim_cache size 0 max 16384 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 -- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a1c0afcd0 msgr2=0x7f2a1c0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a1c0afcd0 0x7f2a1c0b21e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2a2401f000 tx=0x7f2a2400eb80 comp rx=0 tx=0).stop 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 -- 172.21.15.27:0/2175348701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c108100 msgr2=0x7f2a2c1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c108100 0x7f2a2c1986a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800e9f0 tx=0x7f2a1800eeb0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.424 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 -- 172.21.15.27:0/2175348701 shutdown_connections 2024-04-25T13:53:56.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2a1c0afcd0 0x7f2a1c0b21e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a2c19d5a0 0x7f2a2c19f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a2c108100 0x7f2a2c1986a0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 --2- 172.21.15.27:0/2175348701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2c103400 0x7f2a2c1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:56.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 -- 172.21.15.27:0/2175348701 >> 172.21.15.27:0/2175348701 conn(0x7f2a2c0fd390 msgr2=0x7f2a2c1040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:56.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 -- 172.21.15.27:0/2175348701 shutdown_connections 2024-04-25T13:53:56.425 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:53:56.424+0000 7f2a32d38640 1 -- 172.21.15.27:0/2175348701 wait complete. 2024-04-25T13:53:56.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:56 smithi122 ceph-mon[32843]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5.1 MiB/s wr, 549 op/s 2024-04-25T13:53:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:56 smithi045 ceph-mon[32731]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5.1 MiB/s wr, 549 op/s 2024-04-25T13:53:56.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:56 smithi027 ceph-mon[29140]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5.1 MiB/s wr, 549 op/s 2024-04-25T13:53:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 770.0 s, 2234.4 tps, lat 44.168 ms stddev 47.948 2024-04-25T13:53:58.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:53:58.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:53:58.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:58 smithi122 ceph-mon[32843]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.1 MiB/s wr, 637 op/s 2024-04-25T13:53:58.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:58 smithi045 ceph-mon[32731]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.1 MiB/s wr, 637 op/s 2024-04-25T13:53:58.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.579+0000 7f4782253640 1 -- 172.21.15.27:0/3735330435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f477c10b310 msgr2=0x7f477c10b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.579+0000 7f4782253640 1 --2- 172.21.15.27:0/3735330435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f477c10b310 0x7f477c10b720 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f476c01a910 tx=0x7f476c03b470 comp rx=0 tx=0).stop 2024-04-25T13:53:58.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.579+0000 7f4782253640 1 -- 172.21.15.27:0/3735330435 shutdown_connections 2024-04-25T13:53:58.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.579+0000 7f4782253640 1 --2- 172.21.15.27:0/3735330435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f477c110030 0x7f477c110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.579+0000 7f4782253640 1 --2- 172.21.15.27:0/3735330435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f477c10b310 0x7f477c10b720 secure :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f476c01a910 tx=0x7f476c03b470 comp rx=0 tx=0).stop 2024-04-25T13:53:58.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.579+0000 7f4782253640 1 --2- 172.21.15.27:0/3735330435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f477c10a420 0x7f477c10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.579+0000 7f4782253640 1 -- 172.21.15.27:0/3735330435 >> 172.21.15.27:0/3735330435 conn(0x7f477c0704e0 msgr2=0x7f477c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:58.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.585+0000 7f4782253640 1 -- 172.21.15.27:0/3735330435 shutdown_connections 2024-04-25T13:53:58.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.585+0000 7f4782253640 1 -- 172.21.15.27:0/3735330435 wait complete. 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.586+0000 7f4782253640 1 Processor -- start 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.587+0000 7f4782253640 1 -- start start 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.587+0000 7f4782253640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f477c10a420 0x7f477c1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.587+0000 7f4782253640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f477c110030 0x7f477c1a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.587+0000 7f4782253640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f477c1c7e80 0x7f477c1ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.587+0000 7f4782253640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f477c113dc0 con 0x7f477c110030 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.587+0000 7f4782253640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f477c113c40 con 0x7f477c10a420 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.587+0000 7f4782253640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f477c113f40 con 0x7f477c1c7e80 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.588+0000 7f477bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f477c1c7e80 0x7f477c1ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.588+0000 7f477bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f477c1c7e80 0x7f477c1ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33302/0 (socket says 172.21.15.27:33302) 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.588+0000 7f477bfff640 1 -- 172.21.15.27:0/1666070981 learned_addr learned my addr 172.21.15.27:0/1666070981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.589+0000 7f477bfff640 1 -- 172.21.15.27:0/1666070981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f477c10a420 msgr2=0x7f477c1a09b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:53:58.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.589+0000 7f477bfff640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f477c10a420 0x7f477c1a09b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.589+0000 7f477bfff640 1 -- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f477c110030 msgr2=0x7f477c1a0f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:53:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.589+0000 7f477bfff640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f477c110030 0x7f477c1a0f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.589+0000 7f477bfff640 1 -- 172.21.15.27:0/1666070981 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477c1ca9d0 con 0x7f477c1c7e80 2024-04-25T13:53:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.589+0000 7f477bfff640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f477c1c7e80 0x7f477c1ca2a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f477400b970 tx=0x7f477400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.590+0000 7f4778ff9640 1 -- 172.21.15.27:0/1666070981 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774019070 con 0x7f477c1c7e80 2024-04-25T13:53:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.590+0000 7f4782253640 1 -- 172.21.15.27:0/1666070981 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f477c1cacc0 con 0x7f477c1c7e80 2024-04-25T13:53:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.590+0000 7f4782253640 1 -- 172.21.15.27:0/1666070981 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f477c1cb230 con 0x7f477c1c7e80 2024-04-25T13:53:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.590+0000 7f4778ff9640 1 -- 172.21.15.27:0/1666070981 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4774014440 con 0x7f477c1c7e80 2024-04-25T13:53:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.590+0000 7f4778ff9640 1 -- 172.21.15.27:0/1666070981 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47740075b0 con 0x7f477c1c7e80 2024-04-25T13:53:58.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.595+0000 7f4778ff9640 1 -- 172.21.15.27:0/1666070981 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f477401c070 con 0x7f477c1c7e80 2024-04-25T13:53:58.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.596+0000 7f47891e0740 1 Processor -- start 2024-04-25T13:53:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.596+0000 7f47891e0740 1 -- start start 2024-04-25T13:53:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.597+0000 7f47891e0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e64a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.597+0000 7f47891e0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e64fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.597+0000 7f47891e0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 0x555e39e65c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.597+0000 7f47891e0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555e39dc9d80 con 0x555e39a9a4d0 2024-04-25T13:53:58.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.597+0000 7f47891e0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555e39d838b0 con 0x555e39ac3390 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.597+0000 7f47891e0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555e39dc5d40 con 0x555e39aa26b0 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.597+0000 7f4778ff9640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47640afc20 0x7f47640b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e64a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e64a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56698/0 (socket says 172.21.15.27:56698) 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 -- 172.21.15.27:0/2934546761 learned_addr learned my addr 172.21.15.27:0/2934546761 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 0x555e39e65c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477b7fe640 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e64fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 -- 172.21.15.27:0/2934546761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 msgr2=0x555e39e65c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 0x555e39e65c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 -- 172.21.15.27:0/2934546761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 msgr2=0x555e39e64fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e64fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.598+0000 7f477affd640 1 -- 172.21.15.27:0/2934546761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e39d54a40 con 0x555e39ac3390 2024-04-25T13:53:58.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.599+0000 7f4778ff9640 1 -- 172.21.15.27:0/1666070981 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f47740df170 con 0x7f477c1c7e80 2024-04-25T13:53:58.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.599+0000 7f477affd640 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e64a60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f476c005ab0 tx=0x7f476c004d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7093ddc8353d2f88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:58.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.599+0000 7f477bfff640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47640afc20 0x7f47640b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.599+0000 7f4761ffb640 1 -- 172.21.15.27:0/2934546761 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c046040 con 0x555e39ac3390 2024-04-25T13:53:58.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.599+0000 7f4761ffb640 1 -- 172.21.15.27:0/2934546761 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f476c004a00 con 0x555e39ac3390 2024-04-25T13:53:58.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.599+0000 7f4761ffb640 1 -- 172.21.15.27:0/2934546761 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c005000 con 0x555e39ac3390 2024-04-25T13:53:58.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.600+0000 7f477bfff640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47640afc20 0x7f47640b2130 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4774014dd0 tx=0x7f477400bf60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 -- 172.21.15.27:0/2934546761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 msgr2=0x555e39e64a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e64a60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f476c005ab0 tx=0x7f476c004d80 comp rx=0 tx=0).stop 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 -- 172.21.15.27:0/2934546761 shutdown_connections 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 0x555e39e65c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e64fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 --2- 172.21.15.27:0/2934546761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e64a60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 -- 172.21.15.27:0/2934546761 >> 172.21.15.27:0/2934546761 conn(0x555e39ab74e0 msgr2=0x555e39e60270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 -- 172.21.15.27:0/2934546761 shutdown_connections 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.601+0000 7f47891e0740 1 -- 172.21.15.27:0/2934546761 wait complete. 2024-04-25T13:53:58.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.602+0000 7f47891e0740 1 -- start start 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.602+0000 7f47891e0740 1 Processor -- start 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.602+0000 7f47891e0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e6e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.603+0000 7f47891e0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 0x555e39e6ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.603+0000 7f47891e0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e75d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.603+0000 7f47891e0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555e39e666c0 con 0x555e39a9a4d0 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.603+0000 7f47891e0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555e39d50d10 con 0x555e39ac3390 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.603+0000 7f47891e0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555e39e66840 con 0x555e39aa26b0 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.603+0000 7f47891e0740 10 client.0 _ll_register_callbacks cb 0x555e39d7d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.604+0000 7f477affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e75d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.604+0000 7f477affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e6e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.604+0000 7f477affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e6e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38336/0 (socket says 172.21.15.27:38336) 2024-04-25T13:53:58.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.604+0000 7f477affd640 1 -- 172.21.15.27:0/2761111951 learned_addr learned my addr 172.21.15.27:0/2761111951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:58.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.604+0000 7f477b7fe640 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 0x555e39e6ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.605+0000 7f477affd640 1 -- 172.21.15.27:0/2761111951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 msgr2=0x555e39e6ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.605+0000 7f477affd640 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 0x555e39e6ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.605+0000 7f477affd640 1 -- 172.21.15.27:0/2761111951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 msgr2=0x555e39e6e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:58.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.605+0000 7f477affd640 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e6e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.605+0000 7f477affd640 1 -- 172.21.15.27:0/2761111951 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e39ba57d0 con 0x555e39ac3390 2024-04-25T13:53:58.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.605+0000 7f477affd640 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e75d90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f476c03b9e0 tx=0x7f476c01a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc9170ff6ef3310 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:58.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.605+0000 7f475affd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:58.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.605+0000 7f475affd640 1 -- 172.21.15.27:0/2761111951 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c053060 con 0x555e39ac3390 2024-04-25T13:53:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.606+0000 7f475affd640 1 -- 172.21.15.27:0/2761111951 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f476c040990 con 0x555e39ac3390 2024-04-25T13:53:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.606+0000 7f47891e0740 1 -- 172.21.15.27:0/2761111951 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f476c004a00 con 0x555e39ac3390 2024-04-25T13:53:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.606+0000 7f475affd640 1 -- 172.21.15.27:0/2761111951 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f476c046070 con 0x555e39ac3390 2024-04-25T13:53:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.606+0000 7f47891e0740 10 client.27665 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.606+0000 7f47891e0740 1 -- 172.21.15.27:0/2761111951 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555e39adb3f0 con 0x555e39ac3390 2024-04-25T13:53:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.606+0000 7f475affd640 1 -- 172.21.15.27:0/2761111951 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f476c004a00 con 0x555e39ac3390 2024-04-25T13:53:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.607+0000 7f475affd640 1 -- 172.21.15.27:0/2761111951 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f476c04b040 con 0x555e39ac3390 2024-04-25T13:53:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.607+0000 7f47891e0740 10 client.27665 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.607+0000 7f47891e0740 10 client.27665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.607+0000 7f47891e0740 20 client.27665 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.607+0000 7f47891e0740 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e39a81f60 0x555e39e7bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.607+0000 7f47891e0740 4 client.27665 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.607+0000 7f47891e0740 1 -- 172.21.15.27:0/2761111951 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e39dd7990 con 0x555e39a81f60 2024-04-25T13:53:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.607+0000 7f477b7fe640 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e39a81f60 0x555e39e7bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.608+0000 7f477b7fe640 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e39a81f60 0x555e39e7bfc0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a7bdff84b6ba69f server_cookie=e77f00c8e820d4b2 in_seq=0 out_seq=0 2024-04-25T13:53:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.608+0000 7f475affd640 10 client.27665 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.609+0000 7f475affd640 1 -- 172.21.15.27:0/2761111951 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555e39dd7990 con 0x555e39a81f60 2024-04-25T13:53:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.609+0000 7f475affd640 10 client.27665 handle_command_reply: tid=0 2024-04-25T13:53:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.609+0000 7f47891e0740 1 client.27665 shutdown 2024-04-25T13:53:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.609+0000 7f47891e0740 1 -- 172.21.15.27:0/2761111951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 msgr2=0x555e39e75d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.609+0000 7f47891e0740 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e75d90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f476c03b9e0 tx=0x7f476c01a270 comp rx=0 tx=0).stop 2024-04-25T13:53:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.609+0000 7f47891e0740 1 -- 172.21.15.27:0/2761111951 shutdown_connections 2024-04-25T13:53:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.610+0000 7f47891e0740 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e39a81f60 0x555e39e7bfc0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.610+0000 7f47891e0740 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e75d90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.610+0000 7f47891e0740 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39aa26b0 0x555e39e6ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.610+0000 7f47891e0740 1 --2- 172.21.15.27:0/2761111951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39a9a4d0 0x555e39e6e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.610+0000 7f47891e0740 1 -- 172.21.15.27:0/2761111951 >> 172.21.15.27:0/2761111951 conn(0x555e39ab74e0 msgr2=0x555e39e5f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.611+0000 7f475affd640 0 client.27665 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:58.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.612+0000 7f47891e0740 1 -- 172.21.15.27:0/2761111951 shutdown_connections 2024-04-25T13:53:58.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.612+0000 7f47891e0740 1 -- 172.21.15.27:0/2761111951 wait complete. 2024-04-25T13:53:58.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.612+0000 7f47891e0740 20 client.27665 trim_cache size 0 max 16384 2024-04-25T13:53:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.640+0000 7f47891e0740 1 Processor -- start 2024-04-25T13:53:58.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.640+0000 7f47891e0740 1 -- start start 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f47891e0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e6cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f47891e0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39aa26b0 0x555e39e6d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f47891e0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39ac3390 0x555e39e807b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f47891e0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f476800c380 con 0x555e39ac3390 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f47891e0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f476800ec00 con 0x555e39aa26b0 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f47891e0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f476800c9f0 con 0x555e39a9a4d0 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39ac3390 0x555e39e807b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39ac3390 0x555e39e807b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38342/0 (socket says 172.21.15.27:38342) 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 -- 172.21.15.27:0/1805391982 learned_addr learned my addr 172.21.15.27:0/1805391982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e6cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477b7fe640 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39aa26b0 0x555e39e6d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 -- 172.21.15.27:0/1805391982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 msgr2=0x555e39e6cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e6cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 -- 172.21.15.27:0/1805391982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39aa26b0 msgr2=0x555e39e6d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39aa26b0 0x555e39e6d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477affd640 1 -- 172.21.15.27:0/1805391982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e39e6e6a0 con 0x555e39ac3390 2024-04-25T13:53:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.641+0000 7f477b7fe640 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39aa26b0 0x555e39e6d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.642+0000 7f477affd640 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39ac3390 0x555e39e807b0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f476c03bbc0 tx=0x7f476c014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5260868534eaeb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.642+0000 7f47617fa640 1 -- 172.21.15.27:0/1805391982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c004750 con 0x555e39ac3390 2024-04-25T13:53:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.643+0000 7f47617fa640 1 -- 172.21.15.27:0/1805391982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f476c05f040 con 0x555e39ac3390 2024-04-25T13:53:58.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.643+0000 7f47617fa640 1 -- 172.21.15.27:0/1805391982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c003970 con 0x555e39ac3390 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 -- 172.21.15.27:0/1805391982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39ac3390 msgr2=0x555e39e807b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39ac3390 0x555e39e807b0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f476c03bbc0 tx=0x7f476c014040 comp rx=0 tx=0).stop 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 -- 172.21.15.27:0/1805391982 shutdown_connections 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39ac3390 0x555e39e807b0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39aa26b0 0x555e39e6d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 --2- 172.21.15.27:0/1805391982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e6cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 -- 172.21.15.27:0/1805391982 >> 172.21.15.27:0/1805391982 conn(0x555e39ab74e0 msgr2=0x555e39e69680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 -- 172.21.15.27:0/1805391982 shutdown_connections 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 -- 172.21.15.27:0/1805391982 wait complete. 2024-04-25T13:53:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.644+0000 7f47891e0740 1 -- start start 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f47891e0740 1 Processor -- start 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f47891e0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e75340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f47891e0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39aa26b0 0x555e39e71f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f47891e0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f477affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e75340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f477affd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e75340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33336/0 (socket says 172.21.15.27:33336) 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f477affd640 1 -- 172.21.15.27:0/1532890363 learned_addr learned my addr 172.21.15.27:0/1532890363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f476800d0c0 con 0x555e39aa26b0 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f476800e530 con 0x555e39ac3390 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f476800d790 con 0x555e39a9a4d0 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.645+0000 7f477b7fe640 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39aa26b0 0x555e39e71f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f477affd640 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f477b7fe640 1 -- 172.21.15.27:0/1532890363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 msgr2=0x555e39e75340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f477b7fe640 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e75340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f477b7fe640 1 -- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 msgr2=0x555e39e724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f477b7fe640 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f477b7fe640 1 -- 172.21.15.27:0/1532890363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e39a9b0f0 con 0x555e39aa26b0 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f477affd640 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f47891e0740 10 client.0 _ll_register_callbacks cb 0x7f4770006e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f476c05f040 con 0x555e39aa26b0 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f477b7fe640 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39aa26b0 0x555e39e71f10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x555e39e61000 tx=0x7f4770009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebaa362c390ffeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f475affd640 10 client.17976 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f475affd640 1 -- 172.21.15.27:0/1532890363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477000f040 con 0x555e39aa26b0 2024-04-25T13:53:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f475affd640 1 -- 172.21.15.27:0/1532890363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f477001a9c0 con 0x555e39aa26b0 2024-04-25T13:53:58.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f475affd640 1 -- 172.21.15.27:0/1532890363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4770005990 con 0x555e39aa26b0 2024-04-25T13:53:58.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f47891e0740 10 client.17976 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:58.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.646+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({fsmap=19}) -- 0x555e39d8a4b0 con 0x555e39aa26b0 2024-04-25T13:53:58.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.647+0000 7f475affd640 1 -- 172.21.15.27:0/1532890363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f4770005b30 con 0x555e39aa26b0 2024-04-25T13:53:58.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.647+0000 7f47891e0740 10 client.17976 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:58.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.647+0000 7f47891e0740 10 client.17976 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:58.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.647+0000 7f47891e0740 20 client.17976 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:58.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.647+0000 7f47891e0740 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e39a573a0 0x555e39ee5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.647+0000 7f47891e0740 4 client.17976 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:53:58.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.647+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e39dd1800 con 0x555e39a573a0 2024-04-25T13:53:58.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.647+0000 7f477affd640 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e39a573a0 0x555e39ee5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.648+0000 7f477affd640 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e39a573a0 0x555e39ee5a50 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9105df97ed5355a server_cookie=c1bb5dcee12d2066 in_seq=0 out_seq=0 2024-04-25T13:53:58.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.648+0000 7f475affd640 10 client.17976 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f475affd640 1 -- 172.21.15.27:0/1532890363 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555e39dd1800 con 0x555e39a573a0 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f475affd640 10 client.17976 handle_command_reply: tid=0 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 client.17976 shutdown 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39aa26b0 msgr2=0x555e39e71f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39aa26b0 0x555e39e71f10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x555e39e61000 tx=0x7f4770009ee0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 shutdown_connections 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e39a573a0 0x555e39ee5a50 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e39ac3390 0x555e39e724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e39aa26b0 0x555e39e71f10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 --2- 172.21.15.27:0/1532890363 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e39a9a4d0 0x555e39e75340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 >> 172.21.15.27:0/1532890363 conn(0x555e39ab74e0 msgr2=0x555e39e6ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:58.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f475affd640 0 client.17976 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:58.658 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:58 smithi027 ceph-mon[29140]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 6.1 MiB/s wr, 637 op/s 2024-04-25T13:53:58.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 shutdown_connections 2024-04-25T13:53:58.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.650+0000 7f47891e0740 1 -- 172.21.15.27:0/1532890363 wait complete. 2024-04-25T13:53:58.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f47891e0740 20 client.17976 trim_cache size 0 max 16384 2024-04-25T13:53:58.659 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:53:58.659 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:53:58.659 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:53:58.659 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:53:58.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 -- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47640afc20 msgr2=0x7f47640b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47640afc20 0x7f47640b2130 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4774014dd0 tx=0x7f477400bf60 comp rx=0 tx=0).stop 2024-04-25T13:53:58.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 -- 172.21.15.27:0/1666070981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f477c1c7e80 msgr2=0x7f477c1ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f477c1c7e80 0x7f477c1ca2a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f477400b970 tx=0x7f477400be30 comp rx=0 tx=0).stop 2024-04-25T13:53:58.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 -- 172.21.15.27:0/1666070981 shutdown_connections 2024-04-25T13:53:58.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47640afc20 0x7f47640b2130 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f477c1c7e80 0x7f477c1ca2a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f477c110030 0x7f477c1a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 --2- 172.21.15.27:0/1666070981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f477c10a420 0x7f477c1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 -- 172.21.15.27:0/1666070981 >> 172.21.15.27:0/1666070981 conn(0x7f477c0704e0 msgr2=0x7f477c10e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:58.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 -- 172.21.15.27:0/1666070981 shutdown_connections 2024-04-25T13:53:58.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.651+0000 7f4782253640 1 -- 172.21.15.27:0/1666070981 wait complete. 2024-04-25T13:53:58.664 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:53:58.664 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:53:58.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 -- 172.21.15.27:0/1732787850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83801029d0 msgr2=0x7f83801036d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 --2- 172.21.15.27:0/1732787850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83801029d0 0x7f83801036d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8370009900 tx=0x7f837002a410 comp rx=0 tx=0).stop 2024-04-25T13:53:58.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 -- 172.21.15.27:0/1732787850 shutdown_connections 2024-04-25T13:53:58.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 --2- 172.21.15.27:0/1732787850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8380108910 0x7f838010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 --2- 172.21.15.27:0/1732787850 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8380103c10 0x7f8380104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 --2- 172.21.15.27:0/1732787850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83801029d0 0x7f83801036d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 -- 172.21.15.27:0/1732787850 >> 172.21.15.27:0/1732787850 conn(0x7f83800fdba0 msgr2=0x7f8380100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 -- 172.21.15.27:0/1732787850 shutdown_connections 2024-04-25T13:53:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.824+0000 7f8386492640 1 -- 172.21.15.27:0/1732787850 wait complete. 2024-04-25T13:53:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8386492640 1 Processor -- start 2024-04-25T13:53:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8386492640 1 -- start start 2024-04-25T13:53:58.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8386492640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83801029d0 0x7f838006bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8386492640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8380103c10 0x7f838019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8386492640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8380108910 0x7f83801a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8386492640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f838010e500 con 0x7f8380103c10 2024-04-25T13:53:58.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8386492640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f838010e380 con 0x7f83801029d0 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8386492640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f838010e680 con 0x7f8380108910 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f837f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8380103c10 0x7f838019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f837f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8380103c10 0x7f838019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38376/0 (socket says 172.21.15.27:38376) 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f837f7fe640 1 -- 172.21.15.27:0/394446069 learned_addr learned my addr 172.21.15.27:0/394446069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f837ffff640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83801029d0 0x7f838006bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.825+0000 7f8384a2f640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8380108910 0x7f83801a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837ffff640 1 -- 172.21.15.27:0/394446069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8380108910 msgr2=0x7f83801a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837ffff640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8380108910 0x7f83801a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837ffff640 1 -- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8380103c10 msgr2=0x7f838019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837ffff640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8380103c10 0x7f838019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837ffff640 1 -- 172.21.15.27:0/394446069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83801a1d30 con 0x7f83801029d0 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837f7fe640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8380103c10 0x7f838019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837ffff640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83801029d0 0x7f838006bd40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f837002aa00 tx=0x7f8370003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837d7fa640 1 -- 172.21.15.27:0/394446069 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837003f070 con 0x7f83801029d0 2024-04-25T13:53:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837d7fa640 1 -- 172.21.15.27:0/394446069 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8370031bd0 con 0x7f83801029d0 2024-04-25T13:53:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83801a1fc0 con 0x7f83801029d0 2024-04-25T13:53:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f837d7fa640 1 -- 172.21.15.27:0/394446069 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370035030 con 0x7f83801029d0 2024-04-25T13:53:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.826+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83801a2420 con 0x7f83801029d0 2024-04-25T13:53:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.828+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8380107720 con 0x7f83801029d0 2024-04-25T13:53:58.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.829+0000 7f837d7fa640 1 -- 172.21.15.27:0/394446069 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8370042070 con 0x7f83801029d0 2024-04-25T13:53:58.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.830+0000 7f837d7fa640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83680afb70 0x7f83680b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:58.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.830+0000 7f837d7fa640 1 -- 172.21.15.27:0/394446069 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f83700ff4d0 con 0x7f83801029d0 2024-04-25T13:53:58.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.830+0000 7f837f7fe640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83680afb70 0x7f83680b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:58.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.830+0000 7f837f7fe640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83680afb70 0x7f83680b2080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f838006b6e0 tx=0x7f8374005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:58.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:58.835+0000 7f837d7fa640 1 -- 172.21.15.27:0/394446069 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f83700c7a90 con 0x7f83801029d0 2024-04-25T13:53:59.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.187+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f838002db50 con 0x7f83801029d0 2024-04-25T13:53:59.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.189+0000 7f837d7fa640 1 -- 172.21.15.27:0/394446069 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f83700d0160 con 0x7f83801029d0 2024-04-25T13:53:59.189 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:53:59.189 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:53:59.189 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:53:59.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83680afb70 msgr2=0x7f83680b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83680afb70 0x7f83680b2080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f838006b6e0 tx=0x7f8374005f90 comp rx=0 tx=0).stop 2024-04-25T13:53:59.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83801029d0 msgr2=0x7f838006bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83801029d0 0x7f838006bd40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f837002aa00 tx=0x7f8370003c10 comp rx=0 tx=0).stop 2024-04-25T13:53:59.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 shutdown_connections 2024-04-25T13:53:59.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f83680afb70 0x7f83680b2080 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8380108910 0x7f83801a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8380103c10 0x7f838019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 --2- 172.21.15.27:0/394446069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83801029d0 0x7f838006bd40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 >> 172.21.15.27:0/394446069 conn(0x7f83800fdba0 msgr2=0x7f83801048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:59.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 shutdown_connections 2024-04-25T13:53:59.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.193+0000 7f8386492640 1 -- 172.21.15.27:0/394446069 wait complete. 2024-04-25T13:53:59.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:53:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.395+0000 7f3890c69640 1 -- 172.21.15.27:0/215075449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10b310 msgr2=0x7f388c10b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.395+0000 7f3890c69640 1 --2- 172.21.15.27:0/215075449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10b310 0x7f388c10b720 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3880009900 tx=0x7f388002a420 comp rx=0 tx=0).stop 2024-04-25T13:53:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.398+0000 7f3890c69640 1 -- 172.21.15.27:0/215075449 shutdown_connections 2024-04-25T13:53:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.398+0000 7f3890c69640 1 --2- 172.21.15.27:0/215075449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f388c110030 0x7f388c110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.398+0000 7f3890c69640 1 --2- 172.21.15.27:0/215075449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10b310 0x7f388c10b720 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.398+0000 7f3890c69640 1 --2- 172.21.15.27:0/215075449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c10a420 0x7f388c10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.398+0000 7f3890c69640 1 -- 172.21.15.27:0/215075449 >> 172.21.15.27:0/215075449 conn(0x7f388c0704e0 msgr2=0x7f388c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.401+0000 7f3890c69640 1 -- 172.21.15.27:0/215075449 shutdown_connections 2024-04-25T13:53:59.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.403+0000 7f3890c69640 1 -- 172.21.15.27:0/215075449 wait complete. 2024-04-25T13:53:59.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.404+0000 7f3890c69640 1 Processor -- start 2024-04-25T13:53:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.408+0000 7f3890c69640 1 -- start start 2024-04-25T13:53:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.408+0000 7f3890c69640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10a420 0x7f388c1a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.408+0000 7f3890c69640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c10b310 0x7f388c1a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.408+0000 7f3890c69640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f388c110030 0x7f388c1b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.408+0000 7f3890c69640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f388c11e4e0 con 0x7f388c10b310 2024-04-25T13:53:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.408+0000 7f3890c69640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f388c11e360 con 0x7f388c110030 2024-04-25T13:53:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.408+0000 7f3890c69640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f388c11e660 con 0x7f388c10a420 2024-04-25T13:53:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.409+0000 7f3889d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c10b310 0x7f388c1a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.409+0000 7f388a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10a420 0x7f388c1a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.409+0000 7f3889d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c10b310 0x7f388c1a95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38394/0 (socket says 172.21.15.27:38394) 2024-04-25T13:53:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.409+0000 7f3889d9b640 1 -- 172.21.15.27:0/2719024947 learned_addr learned my addr 172.21.15.27:0/2719024947 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:59.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.410+0000 7f388a59c640 1 -- 172.21.15.27:0/2719024947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f388c110030 msgr2=0x7f388c1b0570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:53:59.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.410+0000 7f388a59c640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f388c110030 0x7f388c1b0570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.410+0000 7f388a59c640 1 -- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c10b310 msgr2=0x7f388c1a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.410+0000 7f388a59c640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c10b310 0x7f388c1a95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.410+0000 7f388a59c640 1 -- 172.21.15.27:0/2719024947 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c1b0be0 con 0x7f388c10a420 2024-04-25T13:53:59.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.411+0000 7f388a59c640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10a420 0x7f388c1a90a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f387c00e9c0 tx=0x7f387c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:59.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.411+0000 7f387b7fe640 1 -- 172.21.15.27:0/2719024947 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c00ccc0 con 0x7f388c10a420 2024-04-25T13:53:59.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.412+0000 7f387b7fe640 1 -- 172.21.15.27:0/2719024947 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f387c0108e0 con 0x7f388c10a420 2024-04-25T13:53:59.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.412+0000 7f387b7fe640 1 -- 172.21.15.27:0/2719024947 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c0187f0 con 0x7f388c10a420 2024-04-25T13:53:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.415+0000 7f3890c69640 1 -- 172.21.15.27:0/2719024947 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388c1b0ed0 con 0x7f388c10a420 2024-04-25T13:53:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.415+0000 7f3890c69640 1 -- 172.21.15.27:0/2719024947 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388c1b12e0 con 0x7f388c10a420 2024-04-25T13:53:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.419+0000 7f3897bf0740 1 Processor -- start 2024-04-25T13:53:59.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.419+0000 7f3897bf0740 1 -- start start 2024-04-25T13:53:59.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.419+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4d014390 0x556a4d3b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.419+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3b6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.419+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4cff36b0 0x556a4d3b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.419+0000 7f3897bf0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556a4d31ad80 con 0x556a4d014390 2024-04-25T13:53:59.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.419+0000 7f3897bf0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556a4d2d4a20 con 0x556a4cfeb4d0 2024-04-25T13:53:59.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.419+0000 7f3897bf0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556a4d316d40 con 0x556a4cff36b0 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f388ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4d014390 0x556a4d3b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f3889d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4cff36b0 0x556a4d3b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f3889d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4cff36b0 0x556a4d3b7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33370/0 (socket says 172.21.15.27:33370) 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f3889d9b640 1 -- 172.21.15.27:0/2157243054 learned_addr learned my addr 172.21.15.27:0/2157243054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f387b7fe640 1 -- 172.21.15.27:0/2719024947 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f387c01f020 con 0x7f388c10a420 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f388ad9d640 1 -- 172.21.15.27:0/2157243054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4cff36b0 msgr2=0x556a4d3b7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f388ad9d640 1 --2- 172.21.15.27:0/2157243054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4cff36b0 0x556a4d3b7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f388ad9d640 1 -- 172.21.15.27:0/2157243054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 msgr2=0x556a4d3b6d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f388ad9d640 1 --2- 172.21.15.27:0/2157243054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3b6d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f388ad9d640 1 -- 172.21.15.27:0/2157243054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a4d2dfc30 con 0x556a4d014390 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f3889d9b640 1 --2- 172.21.15.27:0/2157243054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4cff36b0 0x556a4d3b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.420+0000 7f388ad9d640 1 --2- 172.21.15.27:0/2157243054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4d014390 0x556a4d3b6810 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f388400ab40 tx=0x7f388400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb7735b8d128d6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.421+0000 7f3878ff9640 1 -- 172.21.15.27:0/2157243054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3884004b20 con 0x556a4d014390 2024-04-25T13:53:59.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.421+0000 7f3878ff9640 1 -- 172.21.15.27:0/2157243054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3884004cc0 con 0x556a4d014390 2024-04-25T13:53:59.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.421+0000 7f3878ff9640 1 -- 172.21.15.27:0/2157243054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388400f780 con 0x556a4d014390 2024-04-25T13:53:59.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.421+0000 7f387b7fe640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f38740afc80 0x7f38740b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.421+0000 7f387b7fe640 1 -- 172.21.15.27:0/2719024947 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f387c0decc0 con 0x7f388c10a420 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.421+0000 7f388a59c640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f38740afc80 0x7f38740b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 -- 172.21.15.27:0/2157243054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4d014390 msgr2=0x556a4d3b6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2157243054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4d014390 0x556a4d3b6810 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f388400ab40 tx=0x7f388400c2d0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 -- 172.21.15.27:0/2157243054 shutdown_connections 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2157243054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4cff36b0 0x556a4d3b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2157243054 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3b6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2157243054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4d014390 0x556a4d3b6810 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f388a59c640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f38740afc80 0x7f38740b2190 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f387c0df020 tx=0x7f387c0a3090 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 -- 172.21.15.27:0/2157243054 >> 172.21.15.27:0/2157243054 conn(0x556a4d0084e0 msgr2=0x556a4d3b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 -- 172.21.15.27:0/2157243054 shutdown_connections 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.422+0000 7f3897bf0740 1 -- 172.21.15.27:0/2157243054 wait complete. 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 1 -- start start 2024-04-25T13:53:59.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 1 Processor -- start 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556a4d3b8530 con 0x556a4cfeb4d0 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556a4d2a1dc0 con 0x556a4cff36b0 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556a4d3b86b0 con 0x556a4d014390 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.423+0000 7f3897bf0740 10 client.0 _ll_register_callbacks cb 0x556a4d2bd600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.424+0000 7f3889d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3c0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.424+0000 7f388ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.424+0000 7f388ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38414/0 (socket says 172.21.15.27:38414) 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.424+0000 7f3889d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3c0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56782/0 (socket says 172.21.15.27:56782) 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.424+0000 7f3889d9b640 1 -- 172.21.15.27:0/1819841356 learned_addr learned my addr 172.21.15.27:0/1819841356 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:59.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.424+0000 7f388ad9d640 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.424+0000 7f3889d9b640 1 -- 172.21.15.27:0/1819841356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 msgr2=0x556a4d3c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.424+0000 7f3889d9b640 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.425+0000 7f3889d9b640 1 -- 172.21.15.27:0/1819841356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 msgr2=0x556a4d3c05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.425+0000 7f3889d9b640 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.425+0000 7f3889d9b640 1 -- 172.21.15.27:0/1819841356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a4d02c3f0 con 0x556a4cff36b0 2024-04-25T13:53:59.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.425+0000 7f388ad9d640 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:59.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.426+0000 7f3889d9b640 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3c0ae0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3880006010 tx=0x7f3880004470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17bca6ebe5f35e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:59.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.426+0000 7f3871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:59.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.426+0000 7f3871ffb640 1 -- 172.21.15.27:0/1819841356 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880035040 con 0x556a4cff36b0 2024-04-25T13:53:59.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.426+0000 7f3897bf0740 1 -- 172.21.15.27:0/1819841356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3884004cc0 con 0x556a4cff36b0 2024-04-25T13:53:59.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.426+0000 7f3871ffb640 1 -- 172.21.15.27:0/1819841356 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3880005590 con 0x556a4cff36b0 2024-04-25T13:53:59.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.427+0000 7f3871ffb640 1 -- 172.21.15.27:0/1819841356 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f388004d070 con 0x556a4cff36b0 2024-04-25T13:53:59.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.427+0000 7f3897bf0740 10 client.27680 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.427+0000 7f3897bf0740 1 -- 172.21.15.27:0/1819841356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556a4d0f67d0 con 0x556a4cff36b0 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.427+0000 7f3871ffb640 1 -- 172.21.15.27:0/1819841356 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3884004cc0 con 0x556a4cff36b0 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.428+0000 7f3871ffb640 1 -- 172.21.15.27:0/1819841356 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f3880003990 con 0x556a4cff36b0 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.428+0000 7f3897bf0740 10 client.27680 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.428+0000 7f3897bf0740 10 client.27680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.428+0000 7f3897bf0740 20 client.27680 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.428+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a4cfd2f60 0x556a4d3ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.428+0000 7f3897bf0740 4 client.27680 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:53:59.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.428+0000 7f3897bf0740 1 -- 172.21.15.27:0/1819841356 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a4d328990 con 0x556a4cfd2f60 2024-04-25T13:53:59.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.428+0000 7f388ad9d640 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a4cfd2f60 0x556a4d3ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.429+0000 7f3871ffb640 10 client.27680 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:59.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.429+0000 7f388ad9d640 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a4cfd2f60 0x556a4d3ce260 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a39b2841b075dc71 server_cookie=4f6fa8211950d49e in_seq=0 out_seq=0 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3871ffb640 1 -- 172.21.15.27:0/1819841356 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556a4d328990 con 0x556a4cfd2f60 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3871ffb640 10 client.27680 handle_command_reply: tid=0 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 client.27680 shutdown 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 -- 172.21.15.27:0/1819841356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 msgr2=0x556a4d3c0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3c0ae0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3880006010 tx=0x7f3880004470 comp rx=0 tx=0).stop 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 -- 172.21.15.27:0/1819841356 shutdown_connections 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a4cfd2f60 0x556a4d3ce260 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3871ffb640 0 client.27680 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3c0ae0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1819841356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 -- 172.21.15.27:0/1819841356 >> 172.21.15.27:0/1819841356 conn(0x556a4d0084e0 msgr2=0x556a4d3b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 -- 172.21.15.27:0/1819841356 shutdown_connections 2024-04-25T13:53:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 1 -- 172.21.15.27:0/1819841356 wait complete. 2024-04-25T13:53:59.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.430+0000 7f3897bf0740 20 client.27680 trim_cache size 0 max 16384 2024-04-25T13:53:59.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.450+0000 7f3897bf0740 1 Processor -- start 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f3897bf0740 1 -- start start 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3bff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f3897bf0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1b8000 con 0x556a4cfeb4d0 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f3897bf0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1ba880 con 0x556a4cff36b0 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f3897bf0740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1b8670 con 0x556a4d014390 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f388ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3bff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f388ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3bff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38420/0 (socket says 172.21.15.27:38420) 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f3889d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f388ad9d640 1 -- 172.21.15.27:0/1214942426 learned_addr learned my addr 172.21.15.27:0/1214942426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:59.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.451+0000 7f388ad9d640 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3889d9b640 1 -- 172.21.15.27:0/1214942426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 msgr2=0x556a4d3c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3889d9b640 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3889d9b640 1 -- 172.21.15.27:0/1214942426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 msgr2=0x556a4d3bff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3889d9b640 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3bff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3889d9b640 1 -- 172.21.15.27:0/1214942426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a4d3c7ea0 con 0x556a4cff36b0 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f388ad9d640 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3bff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3889d9b640 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3b1a50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3880003d00 tx=0x7f3880004120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b8bb4d1d3eb0ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3873fff640 1 -- 172.21.15.27:0/1214942426 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880035070 con 0x556a4cff36b0 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3873fff640 1 -- 172.21.15.27:0/1214942426 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3880056020 con 0x556a4cff36b0 2024-04-25T13:53:59.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.452+0000 7f3873fff640 1 -- 172.21.15.27:0/1214942426 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880031900 con 0x556a4cff36b0 2024-04-25T13:53:59.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 -- 172.21.15.27:0/1214942426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 msgr2=0x556a4d3b1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3b1a50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3880003d00 tx=0x7f3880004120 comp rx=0 tx=0).stop 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 -- 172.21.15.27:0/1214942426 shutdown_connections 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cff36b0 0x556a4d3b1a50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 --2- 172.21.15.27:0/1214942426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cfeb4d0 0x556a4d3bff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 -- 172.21.15.27:0/1214942426 >> 172.21.15.27:0/1214942426 conn(0x556a4d0084e0 msgr2=0x556a4d3bf830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 -- 172.21.15.27:0/1214942426 shutdown_connections 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.453+0000 7f3897bf0740 1 -- 172.21.15.27:0/1214942426 wait complete. 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.454+0000 7f3897bf0740 1 -- start start 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.454+0000 7f3897bf0740 1 Processor -- start 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cff36b0 0x556a4d3c5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3897bf0740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3897bf0740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1b8d40 con 0x556a4cff36b0 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56808/0 (socket says 172.21.15.27:56808) 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3889d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cff36b0 0x556a4d3c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 -- 172.21.15.27:0/2376882361 learned_addr learned my addr 172.21.15.27:0/2376882361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1ba1b0 con 0x556a4cfeb4d0 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1b9410 con 0x556a4d014390 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3897bf0740 10 client.0 _ll_register_callbacks cb 0x7f3880009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 -- 172.21.15.27:0/2376882361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 msgr2=0x556a4d3c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 -- 172.21.15.27:0/2376882361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cff36b0 msgr2=0x556a4d3c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:53:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cff36b0 0x556a4d3c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f388ad9d640 1 -- 172.21.15.27:0/2376882361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a4d3c6a10 con 0x556a4cfeb4d0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.455+0000 7f3889d9b640 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cff36b0 0x556a4d3c5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f388ad9d640 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c5050 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f388400a7e0 tx=0x7f388400a450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ef45af67d2ed028 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f3871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f3871ffb640 1 -- 172.21.15.27:0/2376882361 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3884023050 con 0x556a4cfeb4d0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f3871ffb640 1 -- 172.21.15.27:0/2376882361 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3884003970 con 0x556a4cfeb4d0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3880056020 con 0x556a4cfeb4d0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f3871ffb640 1 -- 172.21.15.27:0/2376882361 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f3884003b70 con 0x556a4cfeb4d0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f3897bf0740 10 client.27686 fetch_fsmap learned FSMap version 19 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556a4d3c7cc0 con 0x556a4cfeb4d0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.456+0000 7f3871ffb640 1 -- 172.21.15.27:0/2376882361 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3880056020 con 0x556a4cfeb4d0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3871ffb640 1 -- 172.21.15.27:0/2376882361 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f3884004020 con 0x556a4cfeb4d0 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3897bf0740 10 client.27686 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:53:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3897bf0740 10 client.27686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3897bf0740 20 client.27686 populate_metadata read hostname 'smithi027' 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a4cfa83a0 0x556a4d438ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3897bf0740 4 client.27686 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a4d322800 con 0x556a4cfa83a0 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3889d9b640 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a4cfa83a0 0x556a4d438ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3871ffb640 10 client.27686 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3889d9b640 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a4cfa83a0 0x556a4d438ae0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b21692cf087df9 server_cookie=a7a7c4a7d0234814 in_seq=0 out_seq=0 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3871ffb640 1 -- 172.21.15.27:0/2376882361 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a4d322800 con 0x556a4cfa83a0 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.457+0000 7f3871ffb640 10 client.27686 handle_command_reply: tid=0 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 client.27686 shutdown 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 msgr2=0x556a4d3c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c5050 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f388400a7e0 tx=0x7f388400a450 comp rx=0 tx=0).stop 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 shutdown_connections 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556a4cfa83a0 0x556a4d438ae0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556a4d014390 0x556a4d3c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556a4cff36b0 0x556a4d3c5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 --2- 172.21.15.27:0/2376882361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556a4cfeb4d0 0x556a4d3c5050 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3871ffb640 0 client.27686 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:53:59.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 >> 172.21.15.27:0/2376882361 conn(0x556a4d0084e0 msgr2=0x556a4d3b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:59.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 shutdown_connections 2024-04-25T13:53:59.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 1 -- 172.21.15.27:0/2376882361 wait complete. 2024-04-25T13:53:59.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.458+0000 7f3897bf0740 20 client.27686 trim_cache size 0 max 16384 2024-04-25T13:53:59.466 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.460+0000 7f3890c69640 1 -- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f38740afc80 msgr2=0x7f38740b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.460+0000 7f3890c69640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f38740afc80 0x7f38740b2190 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f387c0df020 tx=0x7f387c0a3090 comp rx=0 tx=0).stop 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.460+0000 7f3890c69640 1 -- 172.21.15.27:0/2719024947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10a420 msgr2=0x7f388c1a90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.460+0000 7f3890c69640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10a420 0x7f388c1a90a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f387c00e9c0 tx=0x7f387c00ee80 comp rx=0 tx=0).stop 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.461+0000 7f3890c69640 1 -- 172.21.15.27:0/2719024947 shutdown_connections 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.461+0000 7f3890c69640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f38740afc80 0x7f38740b2190 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.461+0000 7f3890c69640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f388c110030 0x7f388c1b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.461+0000 7f3890c69640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f388c10b310 0x7f388c1a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.461+0000 7f3890c69640 1 --2- 172.21.15.27:0/2719024947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f388c10a420 0x7f388c1a90a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.461+0000 7f3890c69640 1 -- 172.21.15.27:0/2719024947 >> 172.21.15.27:0/2719024947 conn(0x7f388c0704e0 msgr2=0x7f388c10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.461+0000 7f3890c69640 1 -- 172.21.15.27:0/2719024947 shutdown_connections 2024-04-25T13:53:59.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:53:59.461+0000 7f3890c69640 1 -- 172.21.15.27:0/2719024947 wait complete. 2024-04-25T13:53:59.472 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:53:59 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/394446069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:53:59.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:53:59 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/394446069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:53:59.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:53:59 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/394446069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:54:00.474 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:54:00.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:54:00.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:54:00.503 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:00 smithi027 ceph-mon[29140]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.7 MiB/s wr, 666 op/s 2024-04-25T13:54:00.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:00 smithi122 ceph-mon[32843]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.7 MiB/s wr, 666 op/s 2024-04-25T13:54:00.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:00 smithi045 ceph-mon[32731]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.7 MiB/s wr, 666 op/s 2024-04-25T13:54:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.639+0000 7f8bc4005640 1 -- 172.21.15.27:0/122413978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bbc10b310 msgr2=0x7f8bbc10b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.639+0000 7f8bc4005640 1 --2- 172.21.15.27:0/122413978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bbc10b310 0x7f8bbc10b720 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009900 tx=0x7f8bac02a440 comp rx=0 tx=0).stop 2024-04-25T13:54:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.639+0000 7f8bc4005640 1 -- 172.21.15.27:0/122413978 shutdown_connections 2024-04-25T13:54:00.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.639+0000 7f8bc4005640 1 --2- 172.21.15.27:0/122413978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bbc110030 0x7f8bbc110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.639+0000 7f8bc4005640 1 --2- 172.21.15.27:0/122413978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bbc10b310 0x7f8bbc10b720 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.639+0000 7f8bc4005640 1 --2- 172.21.15.27:0/122413978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bbc10a420 0x7f8bbc10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.640+0000 7f8bc4005640 1 -- 172.21.15.27:0/122413978 >> 172.21.15.27:0/122413978 conn(0x7f8bbc0704e0 msgr2=0x7f8bbc072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.640+0000 7f8bc4005640 1 -- 172.21.15.27:0/122413978 shutdown_connections 2024-04-25T13:54:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.640+0000 7f8bc4005640 1 -- 172.21.15.27:0/122413978 wait complete. 2024-04-25T13:54:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.641+0000 7f8bc4005640 1 Processor -- start 2024-04-25T13:54:00.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.641+0000 7f8bc4005640 1 -- start start 2024-04-25T13:54:00.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.641+0000 7f8bc4005640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bbc10a420 0x7f8bbc11b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.641+0000 7f8bc4005640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bbc10b310 0x7f8bbc119fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.641+0000 7f8bc4005640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bbc110030 0x7f8bbc11a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.641+0000 7f8bc4005640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc112380 con 0x7f8bbc10b310 2024-04-25T13:54:00.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.641+0000 7f8bc4005640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc112200 con 0x7f8bbc110030 2024-04-25T13:54:00.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.641+0000 7f8bc4005640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc112500 con 0x7f8bbc10a420 2024-04-25T13:54:00.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.642+0000 7f8bc1da1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bbc10a420 0x7f8bbc11b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.642+0000 7f8bc1da1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bbc10a420 0x7f8bbc11b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33406/0 (socket says 172.21.15.27:33406) 2024-04-25T13:54:00.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.642+0000 7f8bc1da1640 1 -- 172.21.15.27:0/3146056081 learned_addr learned my addr 172.21.15.27:0/3146056081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.643+0000 7f8bc1da1640 1 -- 172.21.15.27:0/3146056081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bbc110030 msgr2=0x7f8bbc11a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.643+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bbc110030 0x7f8bbc11a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.643+0000 7f8bc1da1640 1 -- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bbc10b310 msgr2=0x7f8bbc119fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.643+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bbc10b310 0x7f8bbc119fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.643+0000 7f8bc1da1640 1 -- 172.21.15.27:0/3146056081 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc11aa40 con 0x7f8bbc10a420 2024-04-25T13:54:00.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.644+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bbc10a420 0x7f8bbc11b920 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800e9c0 tx=0x7f8bb800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.644+0000 7f8bb2ffd640 1 -- 172.21.15.27:0/3146056081 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb800cd80 con 0x7f8bbc10a420 2024-04-25T13:54:00.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.644+0000 7f8bb2ffd640 1 -- 172.21.15.27:0/3146056081 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bb80054d0 con 0x7f8bbc10a420 2024-04-25T13:54:00.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.644+0000 7f8bb2ffd640 1 -- 172.21.15.27:0/3146056081 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8010430 con 0x7f8bbc10a420 2024-04-25T13:54:00.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.644+0000 7f8bc4005640 1 -- 172.21.15.27:0/3146056081 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc11ad30 con 0x7f8bbc10a420 2024-04-25T13:54:00.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.644+0000 7f8bc4005640 1 -- 172.21.15.27:0/3146056081 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bbc1b9bf0 con 0x7f8bbc10a420 2024-04-25T13:54:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.649+0000 7f8bcaf90740 1 Processor -- start 2024-04-25T13:54:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.651+0000 7f8bcaf90740 1 -- start start 2024-04-25T13:54:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.651+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1603e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.651+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b1609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.651+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 0x555d5b167ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.651+0000 7f8bb2ffd640 1 -- 172.21.15.27:0/3146056081 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8bb8005020 con 0x7f8bbc10a420 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.652+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555d5b0c5b00 con 0x555d5ad9e730 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.652+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555d5b061920 con 0x555d5ad96550 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.652+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555d5b0c1ac0 con 0x555d5adbf410 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.652+0000 7f8bc25a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.652+0000 7f8bc25a2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1603e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33412/0 (socket says 172.21.15.27:33412) 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.652+0000 7f8bc25a2640 1 -- 172.21.15.27:0/336794819 learned_addr learned my addr 172.21.15.27:0/336794819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.652+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b1609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.652+0000 7f8bc15a0640 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 0x555d5b167ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.653+0000 7f8bb2ffd640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ba80afd00 0x7f8ba80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.653+0000 7f8bc1da1640 1 -- 172.21.15.27:0/336794819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 msgr2=0x555d5b1603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.653+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.653+0000 7f8bc1da1640 1 -- 172.21.15.27:0/336794819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 msgr2=0x555d5b167ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.653+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 0x555d5b167ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.653+0000 7f8bc1da1640 1 -- 172.21.15.27:0/336794819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d5ae9f7e0 con 0x555d5ad96550 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.654+0000 7f8bb2ffd640 1 -- 172.21.15.27:0/3146056081 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8bb80df0b0 con 0x7f8bbc10a420 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.654+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b1609e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80dcb20 tx=0x7f8bb80e70e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=611fbf1651f72a3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.654+0000 7f8ba77fe640 1 -- 172.21.15.27:0/336794819 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80a3420 con 0x555d5ad96550 2024-04-25T13:54:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.654+0000 7f8ba77fe640 1 -- 172.21.15.27:0/336794819 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bb8005310 con 0x555d5ad96550 2024-04-25T13:54:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.655+0000 7f8ba77fe640 1 -- 172.21.15.27:0/336794819 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80a6cd0 con 0x555d5ad96550 2024-04-25T13:54:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.655+0000 7f8bc25a2640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ba80afd00 0x7f8ba80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.657+0000 7f8bcaf90740 1 -- 172.21.15.27:0/336794819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 msgr2=0x555d5b1609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.657+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b1609e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80dcb20 tx=0x7f8bb80e70e0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.658+0000 7f8bcaf90740 1 -- 172.21.15.27:0/336794819 shutdown_connections 2024-04-25T13:54:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.658+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 0x555d5b167ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.658+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b1609e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.658+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/336794819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.658+0000 7f8bcaf90740 1 -- 172.21.15.27:0/336794819 >> 172.21.15.27:0/336794819 conn(0x555d5adb3560 msgr2=0x555d5b15c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.658+0000 7f8bc25a2640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ba80afd00 0x7f8ba80b2210 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x555d5b161820 tx=0x7f8bb4009ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.658+0000 7f8bcaf90740 1 -- 172.21.15.27:0/336794819 shutdown_connections 2024-04-25T13:54:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.658+0000 7f8bcaf90740 1 -- 172.21.15.27:0/336794819 wait complete. 2024-04-25T13:54:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.659+0000 7f8bcaf90740 1 -- start start 2024-04-25T13:54:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.659+0000 7f8bcaf90740 1 Processor -- start 2024-04-25T13:54:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.660+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b16a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.660+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 0x555d5b16a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.660+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.660+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555d5b168340 con 0x555d5ad9e730 2024-04-25T13:54:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.660+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555d5b05f5d0 con 0x555d5ad96550 2024-04-25T13:54:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.660+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555d5b1684c0 con 0x555d5adbf410 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.660+0000 7f8bcaf90740 10 client.0 _ll_register_callbacks cb 0x555d5b04d080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b16a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b16a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56844/0 (socket says 172.21.15.27:56844) 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 -- 172.21.15.27:0/2508389064 learned_addr learned my addr 172.21.15.27:0/2508389064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc15a0640 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 0x555d5b16a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc25a2640 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 -- 172.21.15.27:0/2508389064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 msgr2=0x555d5b1717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 -- 172.21.15.27:0/2508389064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 msgr2=0x555d5b16a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 0x555d5b16a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 -- 172.21.15.27:0/2508389064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d5aefff70 con 0x555d5ad96550 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b16a3a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80df3e0 tx=0x7f8bb80a3a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a911158f9cfb6497 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8ba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.661+0000 7f8ba57fa640 1 -- 172.21.15.27:0/2508389064 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80c0020 con 0x555d5ad96550 2024-04-25T13:54:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8ba57fa640 1 -- 172.21.15.27:0/2508389064 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8bb80a79d0 con 0x555d5ad96550 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2508389064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555d5b0d3710 con 0x555d5ad96550 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8ba57fa640 1 -- 172.21.15.27:0/2508389064 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f8bb80a6eb0 con 0x555d5ad96550 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 10 client.27695 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2508389064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555d5ad97170 con 0x555d5ad96550 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8ba57fa640 1 -- 172.21.15.27:0/2508389064 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555d5b0d3710 con 0x555d5ad96550 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8ba57fa640 1 -- 172.21.15.27:0/2508389064 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f8bb80aff30 con 0x555d5ad96550 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 10 client.27695 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 10 client.27695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 20 client.27695 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d5ad7e1f0 0x555d5b177890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 4 client.27695 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.662+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2508389064 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d5b0cd580 con 0x555d5ad7e1f0 2024-04-25T13:54:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.663+0000 7f8bc15a0640 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d5ad7e1f0 0x555d5b177890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.665+0000 7f8ba57fa640 10 client.27695 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.665+0000 7f8bc15a0640 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d5ad7e1f0 0x555d5b177890 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da1b5fc70deff4bc server_cookie=7a9fd71e3d4a9058 in_seq=0 out_seq=0 2024-04-25T13:54:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.666+0000 7f8ba57fa640 1 -- 172.21.15.27:0/2508389064 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555d5b0cd580 con 0x555d5ad7e1f0 2024-04-25T13:54:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.666+0000 7f8ba57fa640 10 client.27695 handle_command_reply: tid=0 2024-04-25T13:54:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.666+0000 7f8bcaf90740 1 client.27695 shutdown 2024-04-25T13:54:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2508389064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 msgr2=0x555d5b16a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b16a3a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80df3e0 tx=0x7f8bb80a3a20 comp rx=0 tx=0).stop 2024-04-25T13:54:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2508389064 shutdown_connections 2024-04-25T13:54:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d5ad7e1f0 0x555d5b177890 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8ba57fa640 0 client.27695 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5adbf410 0x555d5b1717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad9e730 0x555d5b16a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2508389064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad96550 0x555d5b16a3a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.667+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2508389064 >> 172.21.15.27:0/2508389064 conn(0x555d5adb3560 msgr2=0x555d5b15b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.668+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2508389064 shutdown_connections 2024-04-25T13:54:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.668+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2508389064 wait complete. 2024-04-25T13:54:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.668+0000 7f8bcaf90740 20 client.27695 trim_cache size 0 max 16384 2024-04-25T13:54:00.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.691+0000 7f8bcaf90740 1 Processor -- start 2024-04-25T13:54:00.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.691+0000 7f8bcaf90740 1 -- start start 2024-04-25T13:54:00.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.691+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b169410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.691+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b15f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1c0860 con 0x555d5ad96550 2024-04-25T13:54:00.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1c30e0 con 0x555d5ad7e1f0 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1c0ed0 con 0x555d5ad9e730 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc1da1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b169410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38462/0 (socket says 172.21.15.27:38462) 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 -- 172.21.15.27:0/2873686598 learned_addr learned my addr 172.21.15.27:0/2873686598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc25a2640 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b15f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 -- 172.21.15.27:0/2873686598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 msgr2=0x555d5b15f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b15f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 -- 172.21.15.27:0/2873686598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 msgr2=0x555d5b169410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b169410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 -- 172.21.15.27:0/2873686598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d5b161850 con 0x555d5ad96550 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b169410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8bc15a0640 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16a5e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8bac005ab0 tx=0x7f8bac005f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb13e2bf20cb0b7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.692+0000 7f8ba6ffd640 1 -- 172.21.15.27:0/2873686598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac004cb0 con 0x555d5ad96550 2024-04-25T13:54:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.693+0000 7f8ba6ffd640 1 -- 172.21.15.27:0/2873686598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bac005300 con 0x555d5ad96550 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.693+0000 7f8ba6ffd640 1 -- 172.21.15.27:0/2873686598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac048bd0 con 0x555d5ad96550 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.693+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2873686598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 msgr2=0x555d5b16a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.693+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16a5e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8bac005ab0 tx=0x7f8bac005f20 comp rx=0 tx=0).stop 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.693+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2873686598 shutdown_connections 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.693+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b15f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.693+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16a5e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.693+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/2873686598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b169410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.694+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2873686598 >> 172.21.15.27:0/2873686598 conn(0x555d5adb3560 msgr2=0x555d5b15eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.694+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2873686598 shutdown_connections 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.694+0000 7f8bcaf90740 1 -- 172.21.15.27:0/2873686598 wait complete. 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.694+0000 7f8bcaf90740 1 -- start start 2024-04-25T13:54:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.694+0000 7f8bcaf90740 1 Processor -- start 2024-04-25T13:54:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b16f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bcaf90740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b170b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bc1da1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b16f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1c15a0 con 0x555d5ad96550 2024-04-25T13:54:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bc1da1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b16f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56862/0 (socket says 172.21.15.27:56862) 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bcaf90740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1c2a10 con 0x555d5ad7e1f0 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1c1c70 con 0x555d5ad9e730 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.695+0000 7f8bc1da1640 1 -- 172.21.15.27:0/554326268 learned_addr learned my addr 172.21.15.27:0/554326268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bcaf90740 10 client.0 _ll_register_callbacks cb 0x7f8bb80dcb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bc25a2640 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b170b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bc15a0640 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bc25a2640 1 -- 172.21.15.27:0/554326268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 msgr2=0x555d5b16f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bc25a2640 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b16f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bc25a2640 1 -- 172.21.15.27:0/554326268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 msgr2=0x555d5b16fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bc25a2640 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bc25a2640 1 -- 172.21.15.27:0/554326268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d5b171110 con 0x555d5ad9e730 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8bc25a2640 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b170b50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8bb40049a0 tx=0x7f8bb4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52ddc06bed923ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8ba4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.696+0000 7f8ba4ff9640 1 -- 172.21.15.27:0/554326268 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb402e060 con 0x555d5ad9e730 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8ba4ff9640 1 -- 172.21.15.27:0/554326268 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8bb401e5c0 con 0x555d5ad9e730 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bac005300 con 0x555d5ad9e730 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8ba4ff9640 1 -- 172.21.15.27:0/554326268 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f8bb401e7c0 con 0x555d5ad9e730 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 10 client.27739 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555d5b1dfa70 con 0x555d5ad9e730 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8ba4ff9640 1 -- 172.21.15.27:0/554326268 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bac005300 con 0x555d5ad9e730 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8ba4ff9640 1 -- 172.21.15.27:0/554326268 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f8bb4029a30 con 0x555d5ad9e730 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 10 client.27739 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 10 client.27739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 20 client.27739 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d5adbf410 0x555d5b1e2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 4 client.27739 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555d5b048ac0 con 0x555d5adbf410 2024-04-25T13:54:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.697+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d5adbf410 0x555d5b1e2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.698+0000 7f8ba4ff9640 10 client.27739 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:00.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.698+0000 7f8bc1da1640 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d5adbf410 0x555d5b1e2370 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9df7c64ef41c2d6a server_cookie=9b465e69e4a9f375 in_seq=0 out_seq=0 2024-04-25T13:54:00.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.699+0000 7f8ba4ff9640 1 -- 172.21.15.27:0/554326268 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555d5b048ac0 con 0x555d5adbf410 2024-04-25T13:54:00.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.699+0000 7f8ba4ff9640 10 client.27739 handle_command_reply: tid=0 2024-04-25T13:54:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.699+0000 7f8bcaf90740 1 client.27739 shutdown 2024-04-25T13:54:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 msgr2=0x555d5b170b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b170b50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8bb40049a0 tx=0x7f8bb4004a40 comp rx=0 tx=0).stop 2024-04-25T13:54:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 shutdown_connections 2024-04-25T13:54:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555d5adbf410 0x555d5b1e2370 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555d5ad9e730 0x555d5b170b50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555d5ad96550 0x555d5b16fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 --2- 172.21.15.27:0/554326268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d5ad7e1f0 0x555d5b16f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 >> 172.21.15.27:0/554326268 conn(0x555d5adb3560 msgr2=0x555d5b15f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8ba4ff9640 0 client.27739 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 shutdown_connections 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.700+0000 7f8bcaf90740 1 -- 172.21.15.27:0/554326268 wait complete. 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.701+0000 7f8bcaf90740 20 client.27739 trim_cache size 0 max 16384 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "e604440f-5881-4bc2-abb8-cb436fadd62d", 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:54:00.707 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:54:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.702+0000 7f8bb0ff9640 1 -- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ba80afd00 msgr2=0x7f8ba80b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ba80afd00 0x7f8ba80b2210 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x555d5b161820 tx=0x7f8bb4009ba0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 -- 172.21.15.27:0/3146056081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bbc10a420 msgr2=0x7f8bbc11b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bbc10a420 0x7f8bbc11b920 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800e9c0 tx=0x7f8bb800ee80 comp rx=0 tx=0).stop 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 -- 172.21.15.27:0/3146056081 shutdown_connections 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8ba80afd00 0x7f8ba80b2210 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bbc110030 0x7f8bbc11a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bbc10b310 0x7f8bbc119fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 --2- 172.21.15.27:0/3146056081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8bbc10a420 0x7f8bbc11b920 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 -- 172.21.15.27:0/3146056081 >> 172.21.15.27:0/3146056081 conn(0x7f8bbc0704e0 msgr2=0x7f8bbc10e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 -- 172.21.15.27:0/3146056081 shutdown_connections 2024-04-25T13:54:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.703+0000 7f8bb0ff9640 1 -- 172.21.15.27:0/3146056081 wait complete. 2024-04-25T13:54:00.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:54:00.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:54:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 -- 172.21.15.27:0/2295325430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca010f710 msgr2=0x7f9ca010fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/2295325430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca010f710 0x7f9ca010fb20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009860 tx=0x7f9c9002a340 comp rx=0 tx=0).stop 2024-04-25T13:54:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 -- 172.21.15.27:0/2295325430 shutdown_connections 2024-04-25T13:54:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/2295325430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca0114410 0x7f9ca0116830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/2295325430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca010f710 0x7f9ca010fb20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/2295325430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca010e4d0 0x7f9ca010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 -- 172.21.15.27:0/2295325430 >> 172.21.15.27:0/2295325430 conn(0x7f9ca006e3b0 msgr2=0x7f9ca010bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 -- 172.21.15.27:0/2295325430 shutdown_connections 2024-04-25T13:54:00.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.932+0000 7f9ca83f3640 1 -- 172.21.15.27:0/2295325430 wait complete. 2024-04-25T13:54:00.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.934+0000 7f9ca83f3640 1 Processor -- start 2024-04-25T13:54:00.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.934+0000 7f9ca83f3640 1 -- start start 2024-04-25T13:54:00.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.934+0000 7f9ca83f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca010e4d0 0x7f9ca01a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.934+0000 7f9ca83f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca010f710 0x7f9ca01a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.935+0000 7f9ca618f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca010e4d0 0x7f9ca01a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.935+0000 7f9ca618f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca010e4d0 0x7f9ca01a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33454/0 (socket says 172.21.15.27:33454) 2024-04-25T13:54:00.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.935+0000 7f9ca83f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca0114410 0x7f9ca01ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.935+0000 7f9ca618f640 1 -- 172.21.15.27:0/3792337256 learned_addr learned my addr 172.21.15.27:0/3792337256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.935+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0119ff0 con 0x7f9ca0114410 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.935+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0119e70 con 0x7f9ca010f710 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.935+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca011a170 con 0x7f9ca010e4d0 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.936+0000 7f9ca618f640 1 -- 172.21.15.27:0/3792337256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca010f710 msgr2=0x7f9ca01a5160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.936+0000 7f9ca618f640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca010f710 0x7f9ca01a5160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.936+0000 7f9ca618f640 1 -- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca0114410 msgr2=0x7f9ca01ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.936+0000 7f9ca618f640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca0114410 0x7f9ca01ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.936+0000 7f9ca618f640 1 -- 172.21.15.27:0/3792337256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca01ac760 con 0x7f9ca010e4d0 2024-04-25T13:54:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.937+0000 7f9ca618f640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca010e4d0 0x7f9ca01a4c20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800e9c0 tx=0x7f9c9800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.937+0000 7f9c977fe640 1 -- 172.21.15.27:0/3792337256 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9800ccc0 con 0x7f9ca010e4d0 2024-04-25T13:54:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.937+0000 7f9c977fe640 1 -- 172.21.15.27:0/3792337256 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c980035f0 con 0x7f9ca010e4d0 2024-04-25T13:54:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.937+0000 7f9c977fe640 1 -- 172.21.15.27:0/3792337256 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9800ce60 con 0x7f9ca010e4d0 2024-04-25T13:54:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.937+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca01aca50 con 0x7f9ca010e4d0 2024-04-25T13:54:00.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.937+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01acf50 con 0x7f9ca010e4d0 2024-04-25T13:54:00.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9caf37e740 1 Processor -- start 2024-04-25T13:54:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9c977fe640 1 -- 172.21.15.27:0/3792337256 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9c98010750 con 0x7f9ca010e4d0 2024-04-25T13:54:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9caf37e740 1 -- start start 2024-04-25T13:54:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3efa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 0x5573b3efa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3efb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9caf37e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5573b3e56d80 con 0x5573b3b50390 2024-04-25T13:54:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9caf37e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5573b3e108b0 con 0x5573b3b2f6b0 2024-04-25T13:54:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.941+0000 7f9caf37e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5573b3e52d40 con 0x5573b3b274d0 2024-04-25T13:54:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9ca6990640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3efa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9ca6990640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3efa270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38504/0 (socket says 172.21.15.27:38504) 2024-04-25T13:54:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9ca6990640 1 -- 172.21.15.27:0/601916259 learned_addr learned my addr 172.21.15.27:0/601916259 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9ca6990640 1 -- 172.21.15.27:0/601916259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 msgr2=0x5573b3efa7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9ca6990640 1 --2- 172.21.15.27:0/601916259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 0x5573b3efa7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9ca6990640 1 -- 172.21.15.27:0/601916259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 msgr2=0x5573b3efb480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9ca6990640 1 --2- 172.21.15.27:0/601916259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3efb480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9c977fe640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c8c0afb70 0x7f9c8c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.942+0000 7f9ca6990640 1 -- 172.21.15.27:0/601916259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573b3de1a40 con 0x5573b3b50390 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.943+0000 7f9c977fe640 1 -- 172.21.15.27:0/3792337256 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9c980def00 con 0x7f9ca010e4d0 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.943+0000 7f9ca6990640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c8c0afb70 0x7f9c8c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.943+0000 7f9ca6990640 1 --2- 172.21.15.27:0/601916259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3efa270 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009ea0 tx=0x7f9c9c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5ec47d5bf25abd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.943+0000 7f9c94bfe640 1 -- 172.21.15.27:0/601916259 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0104f0 con 0x5573b3b50390 2024-04-25T13:54:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.944+0000 7f9ca6990640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c8c0afb70 0x7f9c8c0b2080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c01e000 tx=0x7f9c9c021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.944+0000 7f9c94bfe640 1 -- 172.21.15.27:0/601916259 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c9c01aaa0 con 0x5573b3b50390 2024-04-25T13:54:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.944+0000 7f9c94bfe640 1 -- 172.21.15.27:0/601916259 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c023b90 con 0x5573b3b50390 2024-04-25T13:54:00.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.945+0000 7f9caf37e740 1 -- 172.21.15.27:0/601916259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 msgr2=0x5573b3efa270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.945+0000 7f9caf37e740 1 --2- 172.21.15.27:0/601916259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3efa270 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009ea0 tx=0x7f9c9c009f90 comp rx=0 tx=0).stop 2024-04-25T13:54:00.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.945+0000 7f9caf37e740 1 -- 172.21.15.27:0/601916259 shutdown_connections 2024-04-25T13:54:00.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.945+0000 7f9caf37e740 1 --2- 172.21.15.27:0/601916259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3efb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.945+0000 7f9caf37e740 1 --2- 172.21.15.27:0/601916259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 0x5573b3efa7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.945+0000 7f9caf37e740 1 --2- 172.21.15.27:0/601916259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3efa270 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.945+0000 7f9caf37e740 1 -- 172.21.15.27:0/601916259 >> 172.21.15.27:0/601916259 conn(0x5573b3b444e0 msgr2=0x5573b3ef5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.946+0000 7f9caf37e740 1 -- 172.21.15.27:0/601916259 shutdown_connections 2024-04-25T13:54:00.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.946+0000 7f9caf37e740 1 -- 172.21.15.27:0/601916259 wait complete. 2024-04-25T13:54:00.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.947+0000 7f9caf37e740 1 -- start start 2024-04-25T13:54:00.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.947+0000 7f9caf37e740 1 Processor -- start 2024-04-25T13:54:00.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.947+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.947+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3f04580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.947+0000 7f9ca618f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.947+0000 7f9ca618f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33472/0 (socket says 172.21.15.27:33472) 2024-04-25T13:54:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.947+0000 7f9ca618f640 1 -- 172.21.15.27:0/654796598 learned_addr learned my addr 172.21.15.27:0/654796598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9caf37e740 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3f0b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5573b3efbf90 con 0x5573b3b50390 2024-04-25T13:54:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5573b3dddd10 con 0x5573b3b2f6b0 2024-04-25T13:54:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5573b3efc110 con 0x5573b3b274d0 2024-04-25T13:54:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9ca6990640 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3f0b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9ca598e640 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3f04580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9caf37e740 10 client.0 _ll_register_callbacks cb 0x5573b3e0a210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9ca6990640 1 -- 172.21.15.27:0/654796598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 msgr2=0x5573b3f04040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9ca6990640 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9ca6990640 1 -- 172.21.15.27:0/654796598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 msgr2=0x5573b3f04580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9ca6990640 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3f04580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.948+0000 7f9ca6990640 1 -- 172.21.15.27:0/654796598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573b3c327d0 con 0x5573b3b50390 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.949+0000 7f9ca6990640 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3f0b4e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009ea0 tx=0x7f9c9c023ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34bf64152fbca9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.949+0000 7f9c89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.949+0000 7f9c89ffb640 1 -- 172.21.15.27:0/654796598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c034060 con 0x5573b3b50390 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.949+0000 7f9c89ffb640 1 -- 172.21.15.27:0/654796598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9c9c026690 con 0x5573b3b50390 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.949+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c9c01aaa0 con 0x5573b3b50390 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.949+0000 7f9c89ffb640 1 -- 172.21.15.27:0/654796598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9c9c01a900 con 0x5573b3b50390 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9caf37e740 10 client.18024 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5573b3b683f0 con 0x5573b3b50390 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9c89ffb640 1 -- 172.21.15.27:0/654796598 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c9c01aaa0 con 0x5573b3b50390 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9c89ffb640 1 -- 172.21.15.27:0/654796598 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9c9c02e030 con 0x5573b3b50390 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9caf37e740 10 client.18024 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9caf37e740 10 client.18024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9caf37e740 20 client.18024 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9caf37e740 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573b3b0ef60 0x5573b3f115e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9caf37e740 4 client.18024 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:00.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.950+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573b3e64990 con 0x5573b3b0ef60 2024-04-25T13:54:00.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.951+0000 7f9ca618f640 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573b3b0ef60 0x5573b3f115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.951+0000 7f9c89ffb640 10 client.18024 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.951+0000 7f9ca618f640 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573b3b0ef60 0x5573b3f115e0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e39967734a60c4b server_cookie=a2eb6bcd1039c490 in_seq=0 out_seq=0 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.952+0000 7f9c89ffb640 1 -- 172.21.15.27:0/654796598 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5573b3e64990 con 0x5573b3b0ef60 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.952+0000 7f9c89ffb640 10 client.18024 handle_command_reply: tid=0 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.952+0000 7f9caf37e740 1 client.18024 shutdown 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.952+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 msgr2=0x5573b3f0b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.952+0000 7f9caf37e740 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3f0b4e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009ea0 tx=0x7f9c9c023ee0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.953+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 shutdown_connections 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.953+0000 7f9caf37e740 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573b3b0ef60 0x5573b3f115e0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.953+0000 7f9caf37e740 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b50390 0x5573b3f0b4e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.953+0000 7f9caf37e740 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3f04580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.953+0000 7f9caf37e740 1 --2- 172.21.15.27:0/654796598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.953+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 >> 172.21.15.27:0/654796598 conn(0x5573b3b444e0 msgr2=0x5573b3eec3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.953+0000 7f9c89ffb640 0 client.18024 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.954+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 shutdown_connections 2024-04-25T13:54:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.955+0000 7f9caf37e740 1 -- 172.21.15.27:0/654796598 wait complete. 2024-04-25T13:54:00.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.955+0000 7f9caf37e740 20 client.18024 trim_cache size 0 max 16384 2024-04-25T13:54:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.982+0000 7f9caf37e740 1 Processor -- start 2024-04-25T13:54:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.982+0000 7f9caf37e740 1 -- start start 2024-04-25T13:54:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.982+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.982+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3f0b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.982+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b50390 0x5573b3f0b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.982+0000 7f9caf37e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01af9f0 con 0x5573b3b274d0 2024-04-25T13:54:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9caf37e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b2270 con 0x5573b3b2f6b0 2024-04-25T13:54:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca618f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca618f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38530/0 (socket says 172.21.15.27:38530) 2024-04-25T13:54:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca618f640 1 -- 172.21.15.27:0/830524930 learned_addr learned my addr 172.21.15.27:0/830524930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca6990640 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b50390 0x5573b3f0b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9caf37e740 1 -- 172.21.15.27:0/830524930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b0060 con 0x5573b3b50390 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca6990640 1 -- 172.21.15.27:0/830524930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 msgr2=0x5573b3f0b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca6990640 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3f0b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca6990640 1 -- 172.21.15.27:0/830524930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 msgr2=0x5573b3f04040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca6990640 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca6990640 1 -- 172.21.15.27:0/830524930 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573b3f02e40 con 0x5573b3b50390 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.983+0000 7f9ca618f640 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.984+0000 7f9ca6990640 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b50390 0x5573b3f0b940 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009b30 tx=0x7f9c9c009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b7715e8e0a4fb5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.984+0000 7f9c8bfff640 1 -- 172.21.15.27:0/830524930 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c03f020 con 0x5573b3b50390 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.984+0000 7f9c8bfff640 1 -- 172.21.15.27:0/830524930 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c9c01aaa0 con 0x5573b3b50390 2024-04-25T13:54:00.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.984+0000 7f9c8bfff640 1 -- 172.21.15.27:0/830524930 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c034040 con 0x5573b3b50390 2024-04-25T13:54:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 -- 172.21.15.27:0/830524930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b50390 msgr2=0x5573b3f0b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b50390 0x5573b3f0b940 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009b30 tx=0x7f9c9c009fd0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 -- 172.21.15.27:0/830524930 shutdown_connections 2024-04-25T13:54:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b50390 0x5573b3f0b940 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b2f6b0 0x5573b3f0b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 --2- 172.21.15.27:0/830524930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f04040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 -- 172.21.15.27:0/830524930 >> 172.21.15.27:0/830524930 conn(0x5573b3b444e0 msgr2=0x5573b3efb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 -- 172.21.15.27:0/830524930 shutdown_connections 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.985+0000 7f9caf37e740 1 -- 172.21.15.27:0/830524930 wait complete. 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9caf37e740 1 -- start start 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9caf37e740 1 Processor -- start 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f0bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b2f6b0 0x5573b3f088a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9caf37e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b50390 0x5573b3f08ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9caf37e740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b0730 con 0x5573b3b274d0 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9caf37e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b1ba0 con 0x5573b3b50390 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9caf37e740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01b0e00 con 0x5573b3b2f6b0 2024-04-25T13:54:00.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.986+0000 7f9ca618f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f0bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca618f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f0bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38540/0 (socket says 172.21.15.27:38540) 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca598e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b2f6b0 0x5573b3f088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca618f640 1 -- 172.21.15.27:0/73916744 learned_addr learned my addr 172.21.15.27:0/73916744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9caf37e740 10 client.0 _ll_register_callbacks cb 0x7f9c9c01d020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca6990640 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b50390 0x5573b3f08ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca618f640 1 -- 172.21.15.27:0/73916744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b2f6b0 msgr2=0x5573b3f088a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca618f640 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b2f6b0 0x5573b3f088a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca618f640 1 -- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b50390 msgr2=0x5573b3f08ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca618f640 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b50390 0x5573b3f08ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca618f640 1 -- 172.21.15.27:0/73916744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573b3b280f0 con 0x5573b3b274d0 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca6990640 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b50390 0x5573b3f08ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9ca618f640 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f0bd20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9c980a7340 tx=0x7f9c980a7b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e208e57de40548c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9c89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:54:00.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9c89ffb640 1 -- 172.21.15.27:0/73916744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c980b0210 con 0x5573b3b274d0 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9c89ffb640 1 -- 172.21.15.27:0/73916744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9c980037a0 con 0x5573b3b274d0 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.987+0000 7f9caf37e740 1 -- 172.21.15.27:0/73916744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c9c01aaa0 con 0x5573b3b274d0 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9c89ffb640 1 -- 172.21.15.27:0/73916744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9c980c8070 con 0x5573b3b274d0 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9caf37e740 10 client.18030 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9caf37e740 1 -- 172.21.15.27:0/73916744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5573b3e174b0 con 0x5573b3b274d0 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9c89ffb640 1 -- 172.21.15.27:0/73916744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c9c01aaa0 con 0x5573b3b274d0 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9c89ffb640 1 -- 172.21.15.27:0/73916744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9c980afd80 con 0x5573b3b274d0 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9caf37e740 10 client.18030 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9caf37e740 10 client.18030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9caf37e740 20 client.18030 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9caf37e740 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573b3ae43a0 0x5573b3f7b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9caf37e740 4 client.18030 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9caf37e740 1 -- 172.21.15.27:0/73916744 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5573b3e5e800 con 0x5573b3ae43a0 2024-04-25T13:54:00.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.988+0000 7f9ca598e640 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573b3ae43a0 0x5573b3f7b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9c89ffb640 10 client.18030 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9ca598e640 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573b3ae43a0 0x5573b3f7b2d0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7c70604579f8909 server_cookie=836c801682eadc5b in_seq=0 out_seq=0 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9c89ffb640 1 -- 172.21.15.27:0/73916744 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5573b3e5e800 con 0x5573b3ae43a0 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9c89ffb640 10 client.18030 handle_command_reply: tid=0 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 client.18030 shutdown 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 -- 172.21.15.27:0/73916744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 msgr2=0x5573b3f0bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f0bd20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9c980a7340 tx=0x7f9c980a7b70 comp rx=0 tx=0).stop 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 -- 172.21.15.27:0/73916744 shutdown_connections 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5573b3ae43a0 0x5573b3f7b2d0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5573b3b50390 0x5573b3f08ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5573b3b2f6b0 0x5573b3f088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 --2- 172.21.15.27:0/73916744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5573b3b274d0 0x5573b3f0bd20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9c89ffb640 0 client.18030 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 -- 172.21.15.27:0/73916744 >> 172.21.15.27:0/73916744 conn(0x5573b3b444e0 msgr2=0x5573b3f04040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 -- 172.21.15.27:0/73916744 shutdown_connections 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 1 -- 172.21.15.27:0/73916744 wait complete. 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.989+0000 7f9caf37e740 20 client.18030 trim_cache size 0 max 16384 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "e604440f-5881-4bc2-abb8-cb436fadd62d": { 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "e604440f-5881-4bc2-abb8-cb436fadd62d", 2024-04-25T13:54:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:54:00.996 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:54:00.996 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:54:00.996 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:54:00.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c8c0afb70 msgr2=0x7f9c8c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c8c0afb70 0x7f9c8c0b2080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c01e000 tx=0x7f9c9c021000 comp rx=0 tx=0).stop 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca010e4d0 msgr2=0x7f9ca01a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca010e4d0 0x7f9ca01a4c20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800e9c0 tx=0x7f9c9800ee80 comp rx=0 tx=0).stop 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 shutdown_connections 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c8c0afb70 0x7f9c8c0b2080 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca0114410 0x7f9ca01ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca010f710 0x7f9ca01a5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 --2- 172.21.15.27:0/3792337256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca010e4d0 0x7f9ca01a4c20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 >> 172.21.15.27:0/3792337256 conn(0x7f9ca006e3b0 msgr2=0x7f9ca01103f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 shutdown_connections 2024-04-25T13:54:00.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:00.990+0000 7f9ca83f3640 1 -- 172.21.15.27:0/3792337256 wait complete. 2024-04-25T13:54:01.005 INFO:tasks.cephfs.filesystem:scrub status for tag:e604440f-5881-4bc2-abb8-cb436fadd62d - {'path': '/', 'tag': 'e604440f-5881-4bc2-abb8-cb436fadd62d', 'options': 'recursive,force'} 2024-04-25T13:54:01.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:01 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:54:01.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:01 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:01.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:01 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:54:01.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:01 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:54:01.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:01 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:01.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:01 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:54:01.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:01 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:54:01.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:01 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:01.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:01 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:54:02.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:02 smithi122 ceph-mon[32843]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.2 MiB/s wr, 639 op/s 2024-04-25T13:54:02.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:02 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:02 smithi045 ceph-mon[32731]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.2 MiB/s wr, 639 op/s 2024-04-25T13:54:02.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:02 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:02.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:02 smithi027 ceph-mon[29140]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.2 MiB/s wr, 639 op/s 2024-04-25T13:54:02.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:02 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 775.0 s, 1417.0 tps, lat 69.478 ms stddev 74.342 2024-04-25T13:54:04.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:04 smithi122 ceph-mon[32843]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 7.0 MiB/s wr, 753 op/s 2024-04-25T13:54:04.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:04 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:54:04.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:04 smithi045 ceph-mon[32731]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 7.0 MiB/s wr, 753 op/s 2024-04-25T13:54:04.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:04 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:54:04.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:04 smithi027 ceph-mon[29140]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 7.0 MiB/s wr, 753 op/s 2024-04-25T13:54:04.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:04 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:54:06.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:06 smithi122 ceph-mon[32843]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.1 MiB/s wr, 670 op/s 2024-04-25T13:54:06.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:06 smithi045 ceph-mon[32731]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.1 MiB/s wr, 670 op/s 2024-04-25T13:54:06.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:06 smithi027 ceph-mon[29140]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.1 MiB/s wr, 670 op/s 2024-04-25T13:54:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 780.0 s, 1156.6 tps, lat 87.954 ms stddev 96.359 2024-04-25T13:54:08.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:08 smithi122 ceph-mon[32843]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 6.9 MiB/s wr, 788 op/s 2024-04-25T13:54:08.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:08 smithi045 ceph-mon[32731]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 6.9 MiB/s wr, 788 op/s 2024-04-25T13:54:08.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:08 smithi027 ceph-mon[29140]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 6.9 MiB/s wr, 788 op/s 2024-04-25T13:54:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.576+0000 7fdf86f00640 1 -- 172.21.15.27:0/924589598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 msgr2=0x7fdf80104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.576+0000 7fdf86f00640 1 --2- 172.21.15.27:0/924589598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 0x7fdf80104810 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009860 tx=0x7fdf7002a340 comp rx=0 tx=0).stop 2024-04-25T13:54:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.576+0000 7fdf86f00640 1 -- 172.21.15.27:0/924589598 shutdown_connections 2024-04-25T13:54:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.577+0000 7fdf86f00640 1 --2- 172.21.15.27:0/924589598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf801076b0 0x7fdf80107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.577+0000 7fdf86f00640 1 --2- 172.21.15.27:0/924589598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf80104d50 0x7fdf80107170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.577+0000 7fdf86f00640 1 --2- 172.21.15.27:0/924589598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 0x7fdf80104810 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.577+0000 7fdf86f00640 1 -- 172.21.15.27:0/924589598 >> 172.21.15.27:0/924589598 conn(0x7fdf800fdc00 msgr2=0x7fdf801000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.577+0000 7fdf86f00640 1 -- 172.21.15.27:0/924589598 shutdown_connections 2024-04-25T13:54:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.577+0000 7fdf86f00640 1 -- 172.21.15.27:0/924589598 wait complete. 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.577+0000 7fdf86f00640 1 Processor -- start 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.577+0000 7fdf86f00640 1 -- start start 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf86f00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 0x7fdf801a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf86f00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80104d50 0x7fdf801a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf86f00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf801076b0 0x7fdf801a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf86f00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80115c30 con 0x7fdf80104d50 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf86f00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80115ab0 con 0x7fdf801076b0 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf86f00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80115db0 con 0x7fdf801023f0 2024-04-25T13:54:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80104d50 0x7fdf801a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80104d50 0x7fdf801a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34674/0 (socket says 172.21.15.27:34674) 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf7ffff640 1 -- 172.21.15.27:0/2779956997 learned_addr learned my addr 172.21.15.27:0/2779956997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 0x7fdf801a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf8549d640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf801076b0 0x7fdf801a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf7ffff640 1 -- 172.21.15.27:0/2779956997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 msgr2=0x7fdf801a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 0x7fdf801a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf7ffff640 1 -- 172.21.15.27:0/2779956997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf801076b0 msgr2=0x7fdf801a7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf801076b0 0x7fdf801a7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf7ffff640 1 -- 172.21.15.27:0/2779956997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf801a83a0 con 0x7fdf80104d50 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.578+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 0x7fdf801a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.579+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80104d50 0x7fdf801a0da0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400e9f0 tx=0x7fdf7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.579+0000 7fdf7dffb640 1 -- 172.21.15.27:0/2779956997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7400cd80 con 0x7fdf80104d50 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.579+0000 7fdf86f00640 1 -- 172.21.15.27:0/2779956997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf801a8690 con 0x7fdf80104d50 2024-04-25T13:54:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.579+0000 7fdf86f00640 1 -- 172.21.15.27:0/2779956997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf801a8c30 con 0x7fdf80104d50 2024-04-25T13:54:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.579+0000 7fdf7dffb640 1 -- 172.21.15.27:0/2779956997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf740108e0 con 0x7fdf80104d50 2024-04-25T13:54:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.579+0000 7fdf7dffb640 1 -- 172.21.15.27:0/2779956997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74018820 con 0x7fdf80104d50 2024-04-25T13:54:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.582+0000 7fdf8ddb1740 1 Processor -- start 2024-04-25T13:54:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8ddb1740 1 -- start start 2024-04-25T13:54:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d3d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d3def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.583 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 0x55ce17d44df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce17ca3460 con 0x55ce179742f0 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ce17c5d590 con 0x55ce1799d1b0 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ce17c9f420 con 0x55ce1797c4d0 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf84c9c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d3def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8549d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d3d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8549d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d3d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57078/0 (socket says 172.21.15.27:57078) 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf84c9c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d3def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34690/0 (socket says 172.21.15.27:34690) 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8549d640 1 -- 172.21.15.27:0/858607021 learned_addr learned my addr 172.21.15.27:0/858607021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8549d640 1 -- 172.21.15.27:0/858607021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 msgr2=0x55ce17d44df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8549d640 1 --2- 172.21.15.27:0/858607021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 0x55ce17d44df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8549d640 1 -- 172.21.15.27:0/858607021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 msgr2=0x55ce17d3def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8549d640 1 --2- 172.21.15.27:0/858607021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d3def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf8549d640 1 -- 172.21.15.27:0/858607021 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce17c6a0d0 con 0x55ce1799d1b0 2024-04-25T13:54:09.584 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.583+0000 7fdf7dffb640 1 -- 172.21.15.27:0/2779956997 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdf740189c0 con 0x7fdf80104d50 2024-04-25T13:54:09.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.584+0000 7fdf8549d640 1 --2- 172.21.15.27:0/858607021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d3d970 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800e4d0 tx=0x7fdf780095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c597170eee7cb281 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:09.585 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.584+0000 7fdf66ffd640 1 -- 172.21.15.27:0/858607021 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78003970 con 0x55ce1799d1b0 2024-04-25T13:54:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.584+0000 7fdf66ffd640 1 -- 172.21.15.27:0/858607021 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf7800d770 con 0x55ce1799d1b0 2024-04-25T13:54:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.584+0000 7fdf7dffb640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf680afae0 0x7fdf680b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.584+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf680afae0 0x7fdf680b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.584+0000 7fdf7dffb640 1 -- 172.21.15.27:0/2779956997 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdf740deee0 con 0x7fdf80104d50 2024-04-25T13:54:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf66ffd640 1 -- 172.21.15.27:0/858607021 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780043c0 con 0x55ce1799d1b0 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/858607021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 msgr2=0x55ce17d3d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/858607021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d3d970 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800e4d0 tx=0x7fdf780095c0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf680afae0 0x7fdf680b1ff0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdf7002a930 tx=0x7fdf7002c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/858607021 shutdown_connections 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/858607021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 0x55ce17d44df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/858607021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d3def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/858607021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d3d970 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/858607021 >> 172.21.15.27:0/858607021 conn(0x55ce17991300 msgr2=0x55ce17d39300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/858607021 shutdown_connections 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/858607021 wait complete. 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 -- start start 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.585+0000 7fdf8ddb1740 1 Processor -- start 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d47820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 0x55ce17d47d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d4ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55ce17d458e0 con 0x55ce179742f0 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55ce17c2ab60 con 0x55ce1799d1b0 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55ce17d45a60 con 0x55ce1797c4d0 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8ddb1740 10 client.0 _ll_register_callbacks cb 0x55ce17c4f2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf84c9c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d47820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8549d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d4ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf84c9c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d47820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34696/0 (socket says 172.21.15.27:34696) 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf84c9c640 1 -- 172.21.15.27:0/494806050 learned_addr learned my addr 172.21.15.27:0/494806050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 0x55ce17d47d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8549d640 1 -- 172.21.15.27:0/494806050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 msgr2=0x55ce17d47d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8549d640 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 0x55ce17d47d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8549d640 1 -- 172.21.15.27:0/494806050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 msgr2=0x55ce17d47820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8549d640 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d47820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf8549d640 1 -- 172.21.15.27:0/494806050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce179b5210 con 0x55ce1799d1b0 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.586+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 0x55ce17d47d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:54:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf8549d640 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d4ecc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdf78006010 tx=0x7fdf7800de90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=984de8ae6498f5b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf5ffff640 1 -- 172.21.15.27:0/494806050 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78027020 con 0x55ce1799d1b0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/494806050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf7800d770 con 0x55ce1799d1b0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf5ffff640 1 -- 172.21.15.27:0/494806050 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdf78003c80 con 0x55ce1799d1b0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf5ffff640 1 -- 172.21.15.27:0/494806050 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdf78003e80 con 0x55ce1799d1b0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf8ddb1740 10 client.27722 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/494806050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ce17a81620 con 0x55ce1799d1b0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf5ffff640 1 -- 172.21.15.27:0/494806050 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf7800d770 con 0x55ce1799d1b0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf5ffff640 1 -- 172.21.15.27:0/494806050 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fdf7800b040 con 0x55ce1799d1b0 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf8ddb1740 10 client.27722 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf8ddb1740 10 client.27722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.587+0000 7fdf8ddb1740 20 client.27722 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.588+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ce1795bd80 0x55ce17d55280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.588+0000 7fdf8ddb1740 4 client.27722 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.588+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/494806050 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce17cb1070 con 0x55ce1795bd80 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.588+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ce1795bd80 0x55ce17d55280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.588+0000 7fdf5ffff640 10 client.27722 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.588+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ce1795bd80 0x55ce17d55280 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7ddf15758aaa5a6 server_cookie=42ee04b3a53681a3 in_seq=0 out_seq=0 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf5ffff640 1 -- 172.21.15.27:0/494806050 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ce17cb1070 con 0x55ce1795bd80 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf5ffff640 10 client.27722 handle_command_reply: tid=0 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 client.27722 shutdown 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/494806050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 msgr2=0x55ce17d4ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d4ecc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdf78006010 tx=0x7fdf7800de90 comp rx=0 tx=0).stop 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/494806050 shutdown_connections 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ce1795bd80 0x55ce17d55280 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1799d1b0 0x55ce17d4ecc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf5ffff640 0 client.27722 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1797c4d0 0x55ce17d47d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/494806050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d47820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.589+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/494806050 >> 172.21.15.27:0/494806050 conn(0x55ce17991300 msgr2=0x55ce17d38250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.590+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/494806050 shutdown_connections 2024-04-25T13:54:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.590+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/494806050 wait complete. 2024-04-25T13:54:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.590+0000 7fdf8ddb1740 20 client.27722 trim_cache size 0 max 16384 2024-04-25T13:54:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf8ddb1740 1 Processor -- start 2024-04-25T13:54:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf8ddb1740 1 -- start start 2024-04-25T13:54:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d38cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1799d1b0 0x55ce17d39200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c00c390 con 0x55ce179742f0 2024-04-25T13:54:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c00ec10 con 0x55ce1797c4d0 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c00ca00 con 0x55ce1799d1b0 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf84c9c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d38cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.611+0000 7fdf84c9c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d38cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34700/0 (socket says 172.21.15.27:34700) 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf84c9c640 1 -- 172.21.15.27:0/2696868403 learned_addr learned my addr 172.21.15.27:0/2696868403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf8549d640 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1799d1b0 0x55ce17d39200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf84c9c640 1 -- 172.21.15.27:0/2696868403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1799d1b0 msgr2=0x55ce17d39200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1799d1b0 0x55ce17d39200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf84c9c640 1 -- 172.21.15.27:0/2696868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 msgr2=0x55ce17d4ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf84c9c640 1 -- 172.21.15.27:0/2696868403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce17d44f00 con 0x55ce179742f0 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d38cc0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdf7002a860 tx=0x7fdf7003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ac7617664bf45a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf667fc640 1 -- 172.21.15.27:0/2696868403 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70054070 con 0x55ce179742f0 2024-04-25T13:54:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf667fc640 1 -- 172.21.15.27:0/2696868403 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf700327f0 con 0x55ce179742f0 2024-04-25T13:54:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.612+0000 7fdf667fc640 1 -- 172.21.15.27:0/2696868403 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70037040 con 0x55ce179742f0 2024-04-25T13:54:09.617 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/2696868403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 msgr2=0x55ce17d38cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.617 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d38cc0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdf7002a860 tx=0x7fdf7003e000 comp rx=0 tx=0).stop 2024-04-25T13:54:09.617 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/2696868403 shutdown_connections 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce1799d1b0 0x55ce17d39200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/2696868403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce179742f0 0x55ce17d38cc0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/2696868403 >> 172.21.15.27:0/2696868403 conn(0x55ce17991300 msgr2=0x55ce17d41b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/2696868403 shutdown_connections 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.613+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/2696868403 wait complete. 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8ddb1740 1 -- start start 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8ddb1740 1 Processor -- start 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce179742f0 0x55ce17d4bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8ddb1740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce1799d1b0 0x55ce17dbb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c00d0d0 con 0x55ce1799d1b0 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c00e540 con 0x55ce1797c4d0 2024-04-25T13:54:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8ddb1740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c00d7a0 con 0x55ce179742f0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8549d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce1799d1b0 0x55ce17dbb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8549d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce1799d1b0 0x55ce17dbb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34706/0 (socket says 172.21.15.27:34706) 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.614+0000 7fdf8549d640 1 -- 172.21.15.27:0/70411224 learned_addr learned my addr 172.21.15.27:0/70411224 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce179742f0 0x55ce17d4bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf8ddb1740 10 client.0 _ll_register_callbacks cb 0x7fdf78009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf84c9c640 1 -- 172.21.15.27:0/70411224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 msgr2=0x55ce17d4c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf84c9c640 1 -- 172.21.15.27:0/70411224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce1799d1b0 msgr2=0x55ce17dbb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce1799d1b0 0x55ce17dbb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf84c9c640 1 -- 172.21.15.27:0/70411224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce17dbba40 con 0x55ce179742f0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf84c9c640 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce179742f0 0x55ce17d4bb20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55ce17d41500 tx=0x7fdf700035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1f35098d2970f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.615+0000 7fdf5ffff640 1 -- 172.21.15.27:0/70411224 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70054070 con 0x55ce179742f0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.616+0000 7fdf5ffff640 1 -- 172.21.15.27:0/70411224 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdf70004380 con 0x55ce179742f0 2024-04-25T13:54:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.616+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/70411224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf700327f0 con 0x55ce179742f0 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.616+0000 7fdf5ffff640 1 -- 172.21.15.27:0/70411224 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdf70037040 con 0x55ce179742f0 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.616+0000 7fdf8ddb1740 10 client.27769 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.616+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/70411224 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55ce17dbcb30 con 0x55ce179742f0 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.616+0000 7fdf5ffff640 1 -- 172.21.15.27:0/70411224 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf700327f0 con 0x55ce179742f0 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.617+0000 7fdf5ffff640 1 -- 172.21.15.27:0/70411224 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fdf7004f110 con 0x55ce179742f0 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.617+0000 7fdf8ddb1740 10 client.27769 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.617+0000 7fdf8ddb1740 10 client.27769 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.617+0000 7fdf8ddb1740 20 client.27769 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.617+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ce179311c0 0x55ce17dbf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.617+0000 7fdf8ddb1740 4 client.27769 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.617+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/70411224 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce17caaee0 con 0x55ce179311c0 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.617+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ce179311c0 0x55ce17dbf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.618+0000 7fdf5ffff640 10 client.27769 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.618+0000 7fdf7ffff640 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ce179311c0 0x55ce17dbf510 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dc0f3819663b2d1 server_cookie=95e821b99e5e116e in_seq=0 out_seq=0 2024-04-25T13:54:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf5ffff640 1 -- 172.21.15.27:0/70411224 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce17caaee0 con 0x55ce179311c0 2024-04-25T13:54:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf5ffff640 10 client.27769 handle_command_reply: tid=0 2024-04-25T13:54:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 client.27769 shutdown 2024-04-25T13:54:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/70411224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce179742f0 msgr2=0x55ce17d4bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce179742f0 0x55ce17d4bb20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55ce17d41500 tx=0x7fdf700035f0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/70411224 shutdown_connections 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55ce179311c0 0x55ce17dbf510 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ce1799d1b0 0x55ce17dbb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ce1797c4d0 0x55ce17d4c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf5ffff640 0 client.27769 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 --2- 172.21.15.27:0/70411224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ce179742f0 0x55ce17d4bb20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.649+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/70411224 >> 172.21.15.27:0/70411224 conn(0x55ce17991300 msgr2=0x55ce17d38c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/70411224 shutdown_connections 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf8ddb1740 1 -- 172.21.15.27:0/70411224 wait complete. 2024-04-25T13:54:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf8ddb1740 20 client.27769 trim_cache size 0 max 16384 2024-04-25T13:54:09.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:54:09.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:54:09.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:54:09.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:54:09.651 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 -- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf680afae0 msgr2=0x7fdf680b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf680afae0 0x7fdf680b1ff0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fdf7002a930 tx=0x7fdf7002c040 comp rx=0 tx=0).stop 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 -- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80104d50 msgr2=0x7fdf801a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80104d50 0x7fdf801a0da0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400e9f0 tx=0x7fdf7400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 -- 172.21.15.27:0/2779956997 shutdown_connections 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdf680afae0 0x7fdf680b1ff0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf801076b0 0x7fdf801a7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80104d50 0x7fdf801a0da0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 --2- 172.21.15.27:0/2779956997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf801023f0 0x7fdf801a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.650+0000 7fdf86f00640 1 -- 172.21.15.27:0/2779956997 >> 172.21.15.27:0/2779956997 conn(0x7fdf800fdc00 msgr2=0x7fdf801032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.651+0000 7fdf86f00640 1 -- 172.21.15.27:0/2779956997 shutdown_connections 2024-04-25T13:54:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:09.651+0000 7fdf86f00640 1 -- 172.21.15.27:0/2779956997 wait complete. 2024-04-25T13:54:10.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:10 smithi122 ceph-mon[32843]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.3 MiB/s wr, 723 op/s 2024-04-25T13:54:10.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:10 smithi045 ceph-mon[32731]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.3 MiB/s wr, 723 op/s 2024-04-25T13:54:10.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:10 smithi027 ceph-mon[29140]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.3 MiB/s wr, 723 op/s 2024-04-25T13:54:12.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:12 smithi122 ceph-mon[32843]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.7 MiB/s wr, 697 op/s 2024-04-25T13:54:12.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:12 smithi045 ceph-mon[32731]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.7 MiB/s wr, 697 op/s 2024-04-25T13:54:12.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:12 smithi027 ceph-mon[29140]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.7 MiB/s wr, 697 op/s 2024-04-25T13:54:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 785.0 s, 1485.0 tps, lat 67.382 ms stddev 86.149 2024-04-25T13:54:14.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:14 smithi122 ceph-mon[32843]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-25T13:54:14.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:14 smithi045 ceph-mon[32731]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-25T13:54:14.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:14 smithi027 ceph-mon[29140]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-25T13:54:16.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:16 smithi122 ceph-mon[32843]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.3 MiB/s wr, 593 op/s 2024-04-25T13:54:16.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:16 smithi045 ceph-mon[32731]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.3 MiB/s wr, 593 op/s 2024-04-25T13:54:16.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:16 smithi027 ceph-mon[29140]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.3 MiB/s wr, 593 op/s 2024-04-25T13:54:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 790.0 s, 1684.4 tps, lat 59.504 ms stddev 66.735 2024-04-25T13:54:18.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:18 smithi122 ceph-mon[32843]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.2 MiB/s wr, 679 op/s 2024-04-25T13:54:18.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:18 smithi045 ceph-mon[32731]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.2 MiB/s wr, 679 op/s 2024-04-25T13:54:18.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:18 smithi027 ceph-mon[29140]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.2 MiB/s wr, 679 op/s 2024-04-25T13:54:20.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:20 smithi122 ceph-mon[32843]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.3 MiB/s wr, 570 op/s 2024-04-25T13:54:20.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:20 smithi045 ceph-mon[32731]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.3 MiB/s wr, 570 op/s 2024-04-25T13:54:20.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:20 smithi027 ceph-mon[29140]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.3 MiB/s wr, 570 op/s 2024-04-25T13:54:22.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:22 smithi122 ceph-mon[32843]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-04-25T13:54:22.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:22 smithi045 ceph-mon[32731]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-04-25T13:54:22.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:22 smithi027 ceph-mon[29140]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-04-25T13:54:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.790+0000 7f4774627640 1 -- 172.21.15.27:0/1361455751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 msgr2=0x7f476c104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.790+0000 7f4774627640 1 --2- 172.21.15.27:0/1361455751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 0x7f476c104070 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f475c009900 tx=0x7f475c02a410 comp rx=0 tx=0).stop 2024-04-25T13:54:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.790+0000 7f4774627640 1 -- 172.21.15.27:0/1361455751 shutdown_connections 2024-04-25T13:54:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.790+0000 7f4774627640 1 --2- 172.21.15.27:0/1361455751 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f476c108960 0x7f476c10ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.790+0000 7f4774627640 1 --2- 172.21.15.27:0/1361455751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 0x7f476c104070 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.790+0000 7f4774627640 1 --2- 172.21.15.27:0/1361455751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f476c102a20 0x7f476c103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 -- 172.21.15.27:0/1361455751 >> 172.21.15.27:0/1361455751 conn(0x7f476c0fdc10 msgr2=0x7f476c1000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 -- 172.21.15.27:0/1361455751 shutdown_connections 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 -- 172.21.15.27:0/1361455751 wait complete. 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 Processor -- start 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 -- start start 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f476c102a20 0x7f476c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 0x7f476c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f476c108960 0x7f476c1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f476c10e540 con 0x7f476c103c60 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f476c10e3c0 con 0x7f476c102a20 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.791+0000 7f4774627640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f476c10e6c0 con 0x7f476c108960 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 0x7f476c1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 0x7f476c1996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45424/0 (socket says 172.21.15.27:45424) 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 -- 172.21.15.27:0/2264641977 learned_addr learned my addr 172.21.15.27:0/2264641977 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4772bc4640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f476c108960 0x7f476c1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f47723c3640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f476c102a20 0x7f476c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 -- 172.21.15.27:0/2264641977 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f476c108960 msgr2=0x7f476c1a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f476c108960 0x7f476c1a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 -- 172.21.15.27:0/2264641977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f476c102a20 msgr2=0x7f476c199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f476c102a20 0x7f476c199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 -- 172.21.15.27:0/2264641977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c1a0cb0 con 0x7f476c103c60 2024-04-25T13:54:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f4771bc2640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 0x7f476c1996b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f475c006010 tx=0x7f475c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.792+0000 7f47637fe640 1 -- 172.21.15.27:0/2264641977 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c03f030 con 0x7f476c103c60 2024-04-25T13:54:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.793+0000 7f47637fe640 1 -- 172.21.15.27:0/2264641977 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f475c031910 con 0x7f476c103c60 2024-04-25T13:54:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.793+0000 7f47637fe640 1 -- 172.21.15.27:0/2264641977 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c035070 con 0x7f476c103c60 2024-04-25T13:54:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.793+0000 7f4774627640 1 -- 172.21.15.27:0/2264641977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476c1a0f40 con 0x7f476c103c60 2024-04-25T13:54:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.793+0000 7f4774627640 1 -- 172.21.15.27:0/2264641977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f476c1a1350 con 0x7f476c103c60 2024-04-25T13:54:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.796+0000 7f477b5bc740 1 Processor -- start 2024-04-25T13:54:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.796+0000 7f477b5bc740 1 -- start start 2024-04-25T13:54:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.796+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fdc1b0 0x555e9737c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fb32f0 0x555e9737cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 0x555e97383df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f477b5bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555e972e2460 con 0x555e96fbb4d0 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f477b5bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555e9729c590 con 0x555e96fb32f0 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f477b5bc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555e972de420 con 0x555e96fdc1b0 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fb32f0 0x555e9737cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f4772bc4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fdc1b0 0x555e9737c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fb32f0 0x555e9737cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49964/0 (socket says 172.21.15.27:49964) 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 -- 172.21.15.27:0/4291331098 learned_addr learned my addr 172.21.15.27:0/4291331098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47637fe640 1 -- 172.21.15.27:0/2264641977 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f475c042030 con 0x7f476c103c60 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 -- 172.21.15.27:0/4291331098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fdc1b0 msgr2=0x555e9737c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 --2- 172.21.15.27:0/4291331098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fdc1b0 0x555e9737c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 -- 172.21.15.27:0/4291331098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 msgr2=0x555e97383df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 --2- 172.21.15.27:0/4291331098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 0x555e97383df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 -- 172.21.15.27:0/4291331098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e972a90d0 con 0x555e96fb32f0 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f4772bc4640 1 --2- 172.21.15.27:0/4291331098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fdc1b0 0x555e9737c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.797+0000 7f47723c3640 1 --2- 172.21.15.27:0/4291331098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fb32f0 0x555e9737cef0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4768009c60 tx=0x7f4768011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b80e7e779eea7f07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.798+0000 7f4760bfe640 1 -- 172.21.15.27:0/4291331098 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768004220 con 0x555e96fb32f0 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.798+0000 7f4760bfe640 1 -- 172.21.15.27:0/4291331098 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47680043c0 con 0x555e96fb32f0 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.798+0000 7f4760bfe640 1 -- 172.21.15.27:0/4291331098 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768019830 con 0x555e96fb32f0 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.798+0000 7f47637fe640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47580afb60 0x7f47580b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.798+0000 7f4772bc4640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47580afb60 0x7f47580b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.798+0000 7f47637fe640 1 -- 172.21.15.27:0/2264641977 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f475c0ff090 con 0x7f476c103c60 2024-04-25T13:54:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.799+0000 7f477b5bc740 1 -- 172.21.15.27:0/4291331098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fb32f0 msgr2=0x555e9737cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.799+0000 7f477b5bc740 1 --2- 172.21.15.27:0/4291331098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fb32f0 0x555e9737cef0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4768009c60 tx=0x7f4768011810 comp rx=0 tx=0).stop 2024-04-25T13:54:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.799+0000 7f4772bc4640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47580afb60 0x7f47580b2070 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x555e9737db40 tx=0x7f4764009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.799+0000 7f477b5bc740 1 -- 172.21.15.27:0/4291331098 shutdown_connections 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.799+0000 7f477b5bc740 1 --2- 172.21.15.27:0/4291331098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 0x555e97383df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.799+0000 7f477b5bc740 1 --2- 172.21.15.27:0/4291331098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fb32f0 0x555e9737cef0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.799+0000 7f477b5bc740 1 --2- 172.21.15.27:0/4291331098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fdc1b0 0x555e9737c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.799+0000 7f477b5bc740 1 -- 172.21.15.27:0/4291331098 >> 172.21.15.27:0/4291331098 conn(0x555e96fd0300 msgr2=0x555e97378300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.800+0000 7f477b5bc740 1 -- 172.21.15.27:0/4291331098 shutdown_connections 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.800+0000 7f477b5bc740 1 -- 172.21.15.27:0/4291331098 wait complete. 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.800+0000 7f477b5bc740 1 -- start start 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f477b5bc740 1 Processor -- start 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e97391840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e97393d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e973962e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f477b5bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x555e97386a50 con 0x555e96fb32f0 2024-04-25T13:54:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f477b5bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x555e97269b60 con 0x555e96fdc1b0 2024-04-25T13:54:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f477b5bc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x555e97386bd0 con 0x555e96fbb4d0 2024-04-25T13:54:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f477b5bc740 10 client.0 _ll_register_callbacks cb 0x555e9728e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f4772bc4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e973962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f4771bc2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e97393d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f4772bc4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e973962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:49978/0 (socket says 172.21.15.27:49978) 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f4772bc4640 1 -- 172.21.15.27:0/1991650256 learned_addr learned my addr 172.21.15.27:0/1991650256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.801+0000 7f47723c3640 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e97391840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4771bc2640 1 -- 172.21.15.27:0/1991650256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 msgr2=0x555e973962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4771bc2640 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e973962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4771bc2640 1 -- 172.21.15.27:0/1991650256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 msgr2=0x555e97391840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4771bc2640 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e97391840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4771bc2640 1 -- 172.21.15.27:0/1991650256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e96ff4210 con 0x555e96fbb4d0 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f47723c3640 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e97391840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4771bc2640 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e97393d90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f475c0c72a0 tx=0x7f475c0c7370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f34e23d95118ec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4755ffb640 1 -- 172.21.15.27:0/1991650256 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c049070 con 0x555e96fbb4d0 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4755ffb640 1 -- 172.21.15.27:0/1991650256 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f475c0c7800 con 0x555e96fbb4d0 2024-04-25T13:54:22.806 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f477b5bc740 1 -- 172.21.15.27:0/1991650256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47680043c0 con 0x555e96fbb4d0 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4755ffb640 1 -- 172.21.15.27:0/1991650256 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f475c0c7a80 con 0x555e96fbb4d0 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f477b5bc740 10 client.27781 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f477b5bc740 1 -- 172.21.15.27:0/1991650256 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555e970c0620 con 0x555e96fbb4d0 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.802+0000 7f4755ffb640 1 -- 172.21.15.27:0/1991650256 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47680043c0 con 0x555e96fbb4d0 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f4755ffb640 1 -- 172.21.15.27:0/1991650256 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f475c0cff70 con 0x555e96fbb4d0 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f477b5bc740 10 client.27781 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f477b5bc740 10 client.27781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f477b5bc740 20 client.27781 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e96f9ad80 0x555e9739ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f477b5bc740 4 client.27781 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f477b5bc740 1 -- 172.21.15.27:0/1991650256 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e972f0070 con 0x555e96f9ad80 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f47723c3640 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e96f9ad80 0x555e9739ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f47723c3640 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e96f9ad80 0x555e9739ca90 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=869faccd1f972fe4 server_cookie=77cfcace76da0e55 in_seq=0 out_seq=0 2024-04-25T13:54:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.803+0000 7f4755ffb640 10 client.27781 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f4755ffb640 1 -- 172.21.15.27:0/1991650256 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555e972f0070 con 0x555e96f9ad80 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f4755ffb640 10 client.27781 handle_command_reply: tid=0 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 client.27781 shutdown 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 -- 172.21.15.27:0/1991650256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 msgr2=0x555e97393d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e97393d90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f475c0c72a0 tx=0x7f475c0c7370 comp rx=0 tx=0).stop 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 -- 172.21.15.27:0/1991650256 shutdown_connections 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e96f9ad80 0x555e9739ca90 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e973962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e97393d90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f4755ffb640 0 client.27781 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1991650256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e97391840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 -- 172.21.15.27:0/1991650256 >> 172.21.15.27:0/1991650256 conn(0x555e96fd0300 msgr2=0x555e97376820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.804+0000 7f477b5bc740 1 -- 172.21.15.27:0/1991650256 shutdown_connections 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.805+0000 7f477b5bc740 1 -- 172.21.15.27:0/1991650256 wait complete. 2024-04-25T13:54:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.805+0000 7f477b5bc740 20 client.27781 trim_cache size 0 max 16384 2024-04-25T13:54:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.823+0000 7f477b5bc740 1 Processor -- start 2024-04-25T13:54:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.823+0000 7f477b5bc740 1 -- start start 2024-04-25T13:54:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.823+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e973a2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.823+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e973a2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.823+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e9737b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.823+0000 7f477b5bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f476c1a6f10 con 0x555e96fb32f0 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.823+0000 7f477b5bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f476c1a9790 con 0x555e96fdc1b0 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.823+0000 7f477b5bc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f476c1a7580 con 0x555e96fbb4d0 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f47723c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e973a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f47723c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e973a2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45440/0 (socket says 172.21.15.27:45440) 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f47723c3640 1 -- 172.21.15.27:0/1302822066 learned_addr learned my addr 172.21.15.27:0/1302822066 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f4772bc4640 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e9737b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f4771bc2640 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e973a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f47723c3640 1 -- 172.21.15.27:0/1302822066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 msgr2=0x555e973a2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f47723c3640 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e973a2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f47723c3640 1 -- 172.21.15.27:0/1302822066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 msgr2=0x555e9737b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f47723c3640 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e9737b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.824+0000 7f47723c3640 1 -- 172.21.15.27:0/1302822066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e9737f2f0 con 0x555e96fb32f0 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.825+0000 7f47723c3640 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e973a2230 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4768009c60 tx=0x7f47680051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06a68ef527753d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.825+0000 7f4757fff640 1 -- 172.21.15.27:0/1302822066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47680052c0 con 0x555e96fb32f0 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.825+0000 7f4757fff640 1 -- 172.21.15.27:0/1302822066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4768005460 con 0x555e96fb32f0 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.825+0000 7f4757fff640 1 -- 172.21.15.27:0/1302822066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768003c40 con 0x555e96fb32f0 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 -- 172.21.15.27:0/1302822066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 msgr2=0x555e973a2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e973a2230 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4768009c60 tx=0x7f47680051d0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 -- 172.21.15.27:0/1302822066 shutdown_connections 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e9737b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fbb4d0 0x555e973a2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 --2- 172.21.15.27:0/1302822066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fb32f0 0x555e973a2230 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 -- 172.21.15.27:0/1302822066 >> 172.21.15.27:0/1302822066 conn(0x555e96fd0300 msgr2=0x555e97382000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 -- 172.21.15.27:0/1302822066 shutdown_connections 2024-04-25T13:54:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 -- 172.21.15.27:0/1302822066 wait complete. 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 -- start start 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.827+0000 7f477b5bc740 1 Processor -- start 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fb32f0 0x555e973956b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 0x555e97395bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f477b5bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e97402fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f477b5bc740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f476c1a7c50 con 0x555e96fbb4d0 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f477b5bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f476c1a90c0 con 0x555e96fdc1b0 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f477b5bc740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f476c1a8320 con 0x555e96fb32f0 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f477b5bc740 10 client.0 _ll_register_callbacks cb 0x7f475c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f47723c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fb32f0 0x555e973956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f4771bc2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 0x555e97395bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f47723c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fb32f0 0x555e973956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47326/0 (socket says 172.21.15.27:47326) 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f47723c3640 1 -- 172.21.15.27:0/3026012588 learned_addr learned my addr 172.21.15.27:0/3026012588 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f4772bc4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e97402fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f47723c3640 1 -- 172.21.15.27:0/3026012588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 msgr2=0x555e97402fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f47723c3640 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e97402fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f47723c3640 1 -- 172.21.15.27:0/3026012588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 msgr2=0x555e97395bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f47723c3640 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 0x555e97395bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.828+0000 7f47723c3640 1 -- 172.21.15.27:0/3026012588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e97403590 con 0x555e96fb32f0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f47723c3640 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fb32f0 0x555e973956b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4768011850 tx=0x7f4768009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=978a357f0755dab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f4755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f4755ffb640 1 -- 172.21.15.27:0/3026012588 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476803c060 con 0x555e96fb32f0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f4755ffb640 1 -- 172.21.15.27:0/3026012588 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f476802b040 con 0x555e96fb32f0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 1 -- 172.21.15.27:0/3026012588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4768005460 con 0x555e96fb32f0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f4755ffb640 1 -- 172.21.15.27:0/3026012588 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4768005c50 con 0x555e96fb32f0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 10 client.27787 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 1 -- 172.21.15.27:0/3026012588 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x555e97404800 con 0x555e96fb32f0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f4755ffb640 1 -- 172.21.15.27:0/3026012588 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4768005460 con 0x555e96fb32f0 2024-04-25T13:54:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f4755ffb640 1 -- 172.21.15.27:0/3026012588 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f47680198f0 con 0x555e96fb32f0 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 10 client.27787 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 10 client.27787 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 20 client.27787 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e96f701c0 0x555e97406ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 4 client.27787 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.829+0000 7f477b5bc740 1 -- 172.21.15.27:0/3026012588 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555e972e9ee0 con 0x555e96f701c0 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.830+0000 7f4771bc2640 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e96f701c0 0x555e97406ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.830+0000 7f4755ffb640 10 client.27787 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.830+0000 7f4771bc2640 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e96f701c0 0x555e97406ef0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c461a23dba41b834 server_cookie=21561c23b34af241 in_seq=0 out_seq=0 2024-04-25T13:54:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f4755ffb640 1 -- 172.21.15.27:0/3026012588 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555e972e9ee0 con 0x555e96f701c0 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f4755ffb640 10 client.27787 handle_command_reply: tid=0 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 client.27787 shutdown 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 -- 172.21.15.27:0/3026012588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fb32f0 msgr2=0x555e973956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fb32f0 0x555e973956b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4768011850 tx=0x7f4768009b60 comp rx=0 tx=0).stop 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 -- 172.21.15.27:0/3026012588 shutdown_connections 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x555e96f701c0 0x555e97406ef0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e96fdc1b0 0x555e97402fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f4755ffb640 0 client.27787 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x555e96fbb4d0 0x555e97395bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 --2- 172.21.15.27:0/3026012588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e96fb32f0 0x555e973956b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.848+0000 7f477b5bc740 1 -- 172.21.15.27:0/3026012588 >> 172.21.15.27:0/3026012588 conn(0x555e96fd0300 msgr2=0x555e9737b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.849+0000 7f477b5bc740 1 -- 172.21.15.27:0/3026012588 shutdown_connections 2024-04-25T13:54:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.849+0000 7f477b5bc740 1 -- 172.21.15.27:0/3026012588 wait complete. 2024-04-25T13:54:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.849+0000 7f477b5bc740 20 client.27787 trim_cache size 0 max 16384 2024-04-25T13:54:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:54:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:54:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:54:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.850+0000 7f4774627640 1 -- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47580afb60 msgr2=0x7f47580b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47580afb60 0x7f47580b2070 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x555e9737db40 tx=0x7f4764009500 comp rx=0 tx=0).stop 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 -- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 msgr2=0x7f476c1996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 0x7f476c1996b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f475c006010 tx=0x7f475c0042f0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 -- 172.21.15.27:0/2264641977 shutdown_connections 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f47580afb60 0x7f47580b2070 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f476c108960 0x7f476c1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f476c103c60 0x7f476c1996b0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 --2- 172.21.15.27:0/2264641977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f476c102a20 0x7f476c199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.851+0000 7f4774627640 1 -- 172.21.15.27:0/2264641977 >> 172.21.15.27:0/2264641977 conn(0x7f476c0fdc10 msgr2=0x7f476c104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.862+0000 7f4774627640 1 -- 172.21.15.27:0/2264641977 shutdown_connections 2024-04-25T13:54:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:22.864+0000 7f4774627640 1 -- 172.21.15.27:0/2264641977 wait complete. 2024-04-25T13:54:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 795.0 s, 1443.4 tps, lat 69.248 ms stddev 70.601 2024-04-25T13:54:24.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:24 smithi045 ceph-mon[32731]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6.1 MiB/s wr, 649 op/s 2024-04-25T13:54:24.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:24 smithi027 ceph-mon[29140]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6.1 MiB/s wr, 649 op/s 2024-04-25T13:54:24.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:24 smithi122 ceph-mon[32843]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6.1 MiB/s wr, 649 op/s 2024-04-25T13:54:26.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:26 smithi045 ceph-mon[32731]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5.0 MiB/s wr, 540 op/s 2024-04-25T13:54:26.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:26 smithi027 ceph-mon[29140]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5.0 MiB/s wr, 540 op/s 2024-04-25T13:54:26.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:26 smithi122 ceph-mon[32843]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5.0 MiB/s wr, 540 op/s 2024-04-25T13:54:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 800.0 s, 1392.0 tps, lat 72.067 ms stddev 79.887 2024-04-25T13:54:28.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:28 smithi045 ceph-mon[32731]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.3 MiB/s wr, 589 op/s 2024-04-25T13:54:28.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:28 smithi027 ceph-mon[29140]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.3 MiB/s wr, 589 op/s 2024-04-25T13:54:28.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:28 smithi122 ceph-mon[32843]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.3 MiB/s wr, 589 op/s 2024-04-25T13:54:29.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:29 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:54:29.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:29 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:54:29.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:29 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:54:30.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:30 smithi027 ceph-mon[29140]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.2 MiB/s wr, 568 op/s 2024-04-25T13:54:30.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:54:30.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:54:30.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:54:30.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:30 smithi122 ceph-mon[32843]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.2 MiB/s wr, 568 op/s 2024-04-25T13:54:30.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:54:30.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:54:30.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:54:30.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:30 smithi045 ceph-mon[32731]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.2 MiB/s wr, 568 op/s 2024-04-25T13:54:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:54:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:54:30.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:54:31.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:54:31.007 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:54:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 -- 172.21.15.27:0/2122553495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428108930 msgr2=0x7f142810ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2122553495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428108930 0x7f142810ad50 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1420009ca0 tx=0x7f142002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 -- 172.21.15.27:0/2122553495 shutdown_connections 2024-04-25T13:54:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2122553495 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428108930 0x7f142810ad50 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2122553495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1428103c30 0x7f1428104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2122553495 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14281029f0 0x7f14281036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 -- 172.21.15.27:0/2122553495 >> 172.21.15.27:0/2122553495 conn(0x7f14280fdc00 msgr2=0x7f1428100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 -- 172.21.15.27:0/2122553495 shutdown_connections 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.160+0000 7f142eb6c640 1 -- 172.21.15.27:0/2122553495 wait complete. 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142eb6c640 1 Processor -- start 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142eb6c640 1 -- start start 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142eb6c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14281029f0 0x7f142819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142eb6c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428103c30 0x7f142819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142eb6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1428108930 0x7f14281a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142eb6c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f142810e520 con 0x7f1428103c30 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142eb6c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f142810e3a0 con 0x7f1428108930 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142eb6c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f142810e6a0 con 0x7f14281029f0 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f1427fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428103c30 0x7f142819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f1427fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428103c30 0x7f142819c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51090/0 (socket says 172.21.15.27:51090) 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f1427fff640 1 -- 172.21.15.27:0/2430068907 learned_addr learned my addr 172.21.15.27:0/2430068907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142c908640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14281029f0 0x7f142819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.161+0000 7f142d109640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1428108930 0x7f14281a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1427fff640 1 -- 172.21.15.27:0/2430068907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14281029f0 msgr2=0x7f142819c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1427fff640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14281029f0 0x7f142819c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1427fff640 1 -- 172.21.15.27:0/2430068907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1428108930 msgr2=0x7f14281a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1427fff640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1428108930 0x7f14281a38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1427fff640 1 -- 172.21.15.27:0/2430068907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14281a3f30 con 0x7f1428103c30 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f142c908640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14281029f0 0x7f142819c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1427fff640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428103c30 0x7f142819c930 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f141400e9f0 tx=0x7f141400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1425ffb640 1 -- 172.21.15.27:0/2430068907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141400cd80 con 0x7f1428103c30 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1425ffb640 1 -- 172.21.15.27:0/2430068907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14140108e0 con 0x7f1428103c30 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f1425ffb640 1 -- 172.21.15.27:0/2430068907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1414018980 con 0x7f1428103c30 2024-04-25T13:54:31.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.162+0000 7f142eb6c640 1 -- 172.21.15.27:0/2430068907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14281a4220 con 0x7f1428103c30 2024-04-25T13:54:31.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.163+0000 7f142eb6c640 1 -- 172.21.15.27:0/2430068907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14281a4660 con 0x7f1428103c30 2024-04-25T13:54:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.166+0000 7f1435a17740 1 Processor -- start 2024-04-25T13:54:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.166+0000 7f1425ffb640 1 -- 172.21.15.27:0/2430068907 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1414010430 con 0x7f1428103c30 2024-04-25T13:54:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.167+0000 7f1425ffb640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14180afcc0 0x7f14180b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.167+0000 7f1425ffb640 1 -- 172.21.15.27:0/2430068907 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f14140def80 con 0x7f1428103c30 2024-04-25T13:54:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.176+0000 7f1435a17740 1 -- start start 2024-04-25T13:54:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.176+0000 7f1435a17740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6abc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.176+0000 7f1435a17740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6abc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.176+0000 7f1435a17740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6abd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.176+0000 7f1435a17740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556ed6a18d80 con 0x556ed6712390 2024-04-25T13:54:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.176+0000 7f1435a17740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556ed69d28b0 con 0x556ed66f16b0 2024-04-25T13:54:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.176+0000 7f1435a17740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556ed6a14d40 con 0x556ed66e94d0 2024-04-25T13:54:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.176+0000 7f1427fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6abc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.177+0000 7f1427fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6abc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58208/0 (socket says 172.21.15.27:58208) 2024-04-25T13:54:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.177+0000 7f1427fff640 1 -- 172.21.15.27:0/2343735079 learned_addr learned my addr 172.21.15.27:0/2343735079 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.180+0000 7f142c908640 1 --2- 172.21.15.27:0/2343735079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6abc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.180+0000 7f1427fff640 1 -- 172.21.15.27:0/2343735079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 msgr2=0x556ed6abd480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:54:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.180+0000 7f1427fff640 1 --2- 172.21.15.27:0/2343735079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6abd480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.180+0000 7f1427fff640 1 -- 172.21.15.27:0/2343735079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 msgr2=0x556ed6abc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.180+0000 7f1427fff640 1 --2- 172.21.15.27:0/2343735079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6abc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.180+0000 7f1427fff640 1 -- 172.21.15.27:0/2343735079 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed69a3a40 con 0x556ed66e94d0 2024-04-25T13:54:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.182+0000 7f1427fff640 1 --2- 172.21.15.27:0/2343735079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6abc7f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f14140a73c0 tx=0x7f14140107e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfc7c982ebe1bee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.187+0000 7f142d109640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14180afcc0 0x7f14180b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.187+0000 7f1412ffd640 1 -- 172.21.15.27:0/2343735079 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14140a7430 con 0x556ed66e94d0 2024-04-25T13:54:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.187+0000 7f1412ffd640 1 -- 172.21.15.27:0/2343735079 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14140ab7e0 con 0x556ed66e94d0 2024-04-25T13:54:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.187+0000 7f1412ffd640 1 -- 172.21.15.27:0/2343735079 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14140aeda0 con 0x556ed66e94d0 2024-04-25T13:54:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.188+0000 7f1435a17740 1 -- 172.21.15.27:0/2343735079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 msgr2=0x556ed6abc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.188+0000 7f1435a17740 1 --2- 172.21.15.27:0/2343735079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6abc7f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f14140a73c0 tx=0x7f14140107e0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.196+0000 7f1435a17740 1 -- 172.21.15.27:0/2343735079 shutdown_connections 2024-04-25T13:54:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.196+0000 7f1435a17740 1 --2- 172.21.15.27:0/2343735079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6abd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.196+0000 7f1435a17740 1 --2- 172.21.15.27:0/2343735079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6abc7f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.196+0000 7f1435a17740 1 --2- 172.21.15.27:0/2343735079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6abc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.196+0000 7f1435a17740 1 -- 172.21.15.27:0/2343735079 >> 172.21.15.27:0/2343735079 conn(0x556ed67064e0 msgr2=0x556ed6ab7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.198+0000 7f1435a17740 1 -- 172.21.15.27:0/2343735079 shutdown_connections 2024-04-25T13:54:31.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.200+0000 7f142d109640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14180afcc0 0x7f14180b21d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1420004af0 tx=0x7f1420009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.200+0000 7f1435a17740 1 -- 172.21.15.27:0/2343735079 wait complete. 2024-04-25T13:54:31.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.200+0000 7f1435a17740 1 -- start start 2024-04-25T13:54:31.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.200+0000 7f1435a17740 1 Processor -- start 2024-04-25T13:54:31.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.201+0000 7f1435a17740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6ac6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.201+0000 7f1435a17740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.201+0000 7f1435a17740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6acd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.201+0000 7f1435a17740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x556ed6abded0 con 0x556ed6712390 2024-04-25T13:54:31.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.201+0000 7f1435a17740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x556ed699fd10 con 0x556ed66f16b0 2024-04-25T13:54:31.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.201+0000 7f1435a17740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x556ed6abe050 con 0x556ed66e94d0 2024-04-25T13:54:31.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.201+0000 7f1435a17740 10 client.0 _ll_register_callbacks cb 0x556ed69cc210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:31.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.203+0000 7f142d109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6acd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.204+0000 7f142d109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6acd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51112/0 (socket says 172.21.15.27:51112) 2024-04-25T13:54:31.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.204+0000 7f142d109640 1 -- 172.21.15.27:0/2520108198 learned_addr learned my addr 172.21.15.27:0/2520108198 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.204+0000 7f142c908640 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6ac6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.205+0000 7f1427fff640 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.209+0000 7f1427fff640 1 -- 172.21.15.27:0/2520108198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 msgr2=0x556ed6ac6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.209+0000 7f1427fff640 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6ac6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.209+0000 7f1427fff640 1 -- 172.21.15.27:0/2520108198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 msgr2=0x556ed6acd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.209+0000 7f1427fff640 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6acd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.209+0000 7f1427fff640 1 -- 172.21.15.27:0/2520108198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed67f47d0 con 0x556ed66f16b0 2024-04-25T13:54:31.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.209+0000 7f1427fff640 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac66b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f14140a73c0 tx=0x7f14140a3960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfe2f22b998956bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.210+0000 7f1407fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:54:31.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.210+0000 7f1407fff640 1 -- 172.21.15.27:0/2520108198 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14140aed70 con 0x556ed66f16b0 2024-04-25T13:54:31.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.210+0000 7f1435a17740 1 -- 172.21.15.27:0/2520108198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14140ab7e0 con 0x556ed66f16b0 2024-04-25T13:54:31.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.210+0000 7f1407fff640 1 -- 172.21.15.27:0/2520108198 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f14140c8300 con 0x556ed66f16b0 2024-04-25T13:54:31.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.211+0000 7f1407fff640 1 -- 172.21.15.27:0/2520108198 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f14140b7050 con 0x556ed66f16b0 2024-04-25T13:54:31.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.211+0000 7f1435a17740 10 client.27755 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:31.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.211+0000 7f1435a17740 1 -- 172.21.15.27:0/2520108198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556ed672a3f0 con 0x556ed66f16b0 2024-04-25T13:54:31.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.212+0000 7f1407fff640 1 -- 172.21.15.27:0/2520108198 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14140ab7e0 con 0x556ed66f16b0 2024-04-25T13:54:31.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.212+0000 7f1407fff640 1 -- 172.21.15.27:0/2520108198 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f14140bc430 con 0x556ed66f16b0 2024-04-25T13:54:31.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.212+0000 7f1435a17740 10 client.27755 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:31.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.212+0000 7f1435a17740 10 client.27755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:31.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.213+0000 7f1435a17740 20 client.27755 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:31.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.213+0000 7f1435a17740 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556ed66d0f60 0x556ed6ae4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.213+0000 7f1435a17740 4 client.27755 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.213+0000 7f1435a17740 1 -- 172.21.15.27:0/2520108198 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ed6a26990 con 0x556ed66d0f60 2024-04-25T13:54:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.213+0000 7f142c908640 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556ed66d0f60 0x556ed6ae4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.218+0000 7f142c908640 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556ed66d0f60 0x556ed6ae4c70 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43a04ba1930acd26 server_cookie=4d8bf9ac0b3a85dc in_seq=0 out_seq=0 2024-04-25T13:54:31.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.218+0000 7f1407fff640 10 client.27755 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:31.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.219+0000 7f1407fff640 1 -- 172.21.15.27:0/2520108198 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556ed6a26990 con 0x556ed66d0f60 2024-04-25T13:54:31.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.219+0000 7f1407fff640 10 client.27755 handle_command_reply: tid=0 2024-04-25T13:54:31.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.220+0000 7f1435a17740 1 client.27755 shutdown 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 -- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 msgr2=0x556ed6ac66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac66b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f14140a73c0 tx=0x7f14140a3960 comp rx=0 tx=0).stop 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 -- 172.21.15.27:0/2520108198 shutdown_connections 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556ed66d0f60 0x556ed6ae4c70 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6acd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac66b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1407fff640 0 client.27755 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 --2- 172.21.15.27:0/2520108198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6ac6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 -- 172.21.15.27:0/2520108198 >> 172.21.15.27:0/2520108198 conn(0x556ed67064e0 msgr2=0x556ed6aad9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 -- 172.21.15.27:0/2520108198 shutdown_connections 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 1 -- 172.21.15.27:0/2520108198 wait complete. 2024-04-25T13:54:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.221+0000 7f1435a17740 20 client.27755 trim_cache size 0 max 16384 2024-04-25T13:54:31.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.245+0000 7f1435a17740 1 Processor -- start 2024-04-25T13:54:31.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f1435a17740 1 -- start start 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f1435a17740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6ac4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f1435a17740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f1435a17740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6ac5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f1435a17740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f142806ec20 con 0x556ed6712390 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f1435a17740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1428071460 con 0x556ed66f16b0 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f1435a17740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f142806f270 con 0x556ed66e94d0 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6ac5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6ac5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51126/0 (socket says 172.21.15.27:51126) 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 -- 172.21.15.27:0/3397436737 learned_addr learned my addr 172.21.15.27:0/3397436737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f1427fff640 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142c908640 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6ac4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 -- 172.21.15.27:0/3397436737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 msgr2=0x556ed6ac4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6ac4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 -- 172.21.15.27:0/3397436737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 msgr2=0x556ed6ac5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 -- 172.21.15.27:0/3397436737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed66ea0f0 con 0x556ed6712390 2024-04-25T13:54:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.246+0000 7f142d109640 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6ac5940 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1420009a60 tx=0x7f142003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a110ddda307ad19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.247+0000 7f14127fc640 1 -- 172.21.15.27:0/3397436737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420003d70 con 0x556ed6712390 2024-04-25T13:54:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.247+0000 7f14127fc640 1 -- 172.21.15.27:0/3397436737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1420036040 con 0x556ed6712390 2024-04-25T13:54:31.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.247+0000 7f14127fc640 1 -- 172.21.15.27:0/3397436737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142002fb20 con 0x556ed6712390 2024-04-25T13:54:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.247+0000 7f1435a17740 1 -- 172.21.15.27:0/3397436737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 msgr2=0x556ed6ac5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.248+0000 7f1435a17740 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6ac5940 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1420009a60 tx=0x7f142003e000 comp rx=0 tx=0).stop 2024-04-25T13:54:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.248+0000 7f1435a17740 1 -- 172.21.15.27:0/3397436737 shutdown_connections 2024-04-25T13:54:31.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.248+0000 7f1435a17740 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6ac5940 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.248+0000 7f1435a17740 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66f16b0 0x556ed6ac5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.248+0000 7f1435a17740 1 --2- 172.21.15.27:0/3397436737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66e94d0 0x556ed6ac4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.248+0000 7f1435a17740 1 -- 172.21.15.27:0/3397436737 >> 172.21.15.27:0/3397436737 conn(0x556ed67064e0 msgr2=0x556ed6ac4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.248+0000 7f1435a17740 1 -- 172.21.15.27:0/3397436737 shutdown_connections 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.248+0000 7f1435a17740 1 -- 172.21.15.27:0/3397436737 wait complete. 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.249+0000 7f1435a17740 1 -- start start 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.249+0000 7f1435a17740 1 Processor -- start 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.249+0000 7f1435a17740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66e94d0 0x556ed6ac99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1435a17740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66f16b0 0x556ed6ac9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1435a17740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6b4b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1435a17740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f142806f920 con 0x556ed6712390 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1435a17740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1428070d90 con 0x556ed66e94d0 2024-04-25T13:54:31.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1435a17740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f142806fff0 con 0x556ed66f16b0 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1435a17740 10 client.0 _ll_register_callbacks cb 0x7f14140a6ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66f16b0 0x556ed6ac9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66f16b0 0x556ed6ac9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58224/0 (socket says 172.21.15.27:58224) 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 -- 172.21.15.27:0/2595109899 learned_addr learned my addr 172.21.15.27:0/2595109899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f142c908640 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66e94d0 0x556ed6ac99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f142d109640 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6b4b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 -- 172.21.15.27:0/2595109899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66e94d0 msgr2=0x556ed6ac99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66e94d0 0x556ed6ac99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 -- 172.21.15.27:0/2595109899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 msgr2=0x556ed6b4b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6b4b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 -- 172.21.15.27:0/2595109899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed69d94b0 con 0x556ed66f16b0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1427fff640 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66f16b0 0x556ed6ac9f10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f14140a46a0 tx=0x7f14140107e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8c53cf5b737f9c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1407fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.250+0000 7f1407fff640 1 -- 172.21.15.27:0/2595109899 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14140ab7e0 con 0x556ed66f16b0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1407fff640 1 -- 172.21.15.27:0/2595109899 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f14140a61f0 con 0x556ed66f16b0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 1 -- 172.21.15.27:0/2595109899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1420036040 con 0x556ed66f16b0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1407fff640 1 -- 172.21.15.27:0/2595109899 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f14140b0290 con 0x556ed66f16b0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 10 client.27805 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 1 -- 172.21.15.27:0/2595109899 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x556ed6b4c340 con 0x556ed66f16b0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1407fff640 1 -- 172.21.15.27:0/2595109899 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1420036040 con 0x556ed66f16b0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1407fff640 1 -- 172.21.15.27:0/2595109899 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1414018bb0 con 0x556ed66f16b0 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 10 client.27805 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 10 client.27805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 20 client.27805 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556ed66a63a0 0x556ed6b4ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 4 client.27805 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f1435a17740 1 -- 172.21.15.27:0/2595109899 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ed6a20800 con 0x556ed66a63a0 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.251+0000 7f142c908640 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556ed66a63a0 0x556ed6b4ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1407fff640 10 client.27805 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f142c908640 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556ed66a63a0 0x556ed6b4ebd0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79924a3100424a1c server_cookie=7d61ce93cafd3ba3 in_seq=0 out_seq=0 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1407fff640 1 -- 172.21.15.27:0/2595109899 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556ed6a20800 con 0x556ed66a63a0 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1407fff640 10 client.27805 handle_command_reply: tid=0 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 client.27805 shutdown 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 -- 172.21.15.27:0/2595109899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66f16b0 msgr2=0x556ed6ac9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66f16b0 0x556ed6ac9f10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f14140a46a0 tx=0x7f14140107e0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 -- 172.21.15.27:0/2595109899 shutdown_connections 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x556ed66a63a0 0x556ed6b4ebd0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x556ed6712390 0x556ed6b4b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556ed66f16b0 0x556ed6ac9f10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1407fff640 0 client.27805 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 --2- 172.21.15.27:0/2595109899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556ed66e94d0 0x556ed6ac99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 -- 172.21.15.27:0/2595109899 >> 172.21.15.27:0/2595109899 conn(0x556ed67064e0 msgr2=0x556ed6aad9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 -- 172.21.15.27:0/2595109899 shutdown_connections 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.252+0000 7f1435a17740 1 -- 172.21.15.27:0/2595109899 wait complete. 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f1435a17740 20 client.27805 trim_cache size 0 max 16384 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:54:31.257 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:54:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 -- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14180afcc0 msgr2=0x7f14180b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14180afcc0 0x7f14180b21d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1420004af0 tx=0x7f1420009360 comp rx=0 tx=0).stop 2024-04-25T13:54:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 -- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428103c30 msgr2=0x7f142819c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428103c30 0x7f142819c930 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f141400e9f0 tx=0x7f141400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 -- 172.21.15.27:0/2430068907 shutdown_connections 2024-04-25T13:54:31.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f14180afcc0 0x7f14180b21d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1428108930 0x7f14281a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1428103c30 0x7f142819c930 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 --2- 172.21.15.27:0/2430068907 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14281029f0 0x7f142819c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 -- 172.21.15.27:0/2430068907 >> 172.21.15.27:0/2430068907 conn(0x7f14280fdc00 msgr2=0x7f142806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 -- 172.21.15.27:0/2430068907 shutdown_connections 2024-04-25T13:54:31.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.253+0000 7f142eb6c640 1 -- 172.21.15.27:0/2430068907 wait complete. 2024-04-25T13:54:31.266 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:54:31.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:54:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.417+0000 7fbe1d469640 1 -- 172.21.15.27:0/2173393271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18101a10 msgr2=0x7fbe181025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.417+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2173393271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18101a10 0x7fbe181025d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009900 tx=0x7fbe0c02a410 comp rx=0 tx=0).stop 2024-04-25T13:54:31.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.426+0000 7fbe1d469640 1 -- 172.21.15.27:0/2173393271 shutdown_connections 2024-04-25T13:54:31.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.426+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2173393271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe18107830 0x7fbe18107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.426+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2173393271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe18102b10 0x7fbe18102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.426+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2173393271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18101a10 0x7fbe181025d0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.426+0000 7fbe1d469640 1 -- 172.21.15.27:0/2173393271 >> 172.21.15.27:0/2173393271 conn(0x7fbe180fcef0 msgr2=0x7fbe180ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.433+0000 7fbe1d469640 1 -- 172.21.15.27:0/2173393271 shutdown_connections 2024-04-25T13:54:31.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.435+0000 7fbe1d469640 1 -- 172.21.15.27:0/2173393271 wait complete. 2024-04-25T13:54:31.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.436+0000 7fbe1d469640 1 Processor -- start 2024-04-25T13:54:31.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.436+0000 7fbe1d469640 1 -- start start 2024-04-25T13:54:31.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.436+0000 7fbe1d469640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe18101a10 0x7fbe1810e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.436+0000 7fbe1d469640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe18102b10 0x7fbe1810ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.436+0000 7fbe1d469640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18107830 0x7fbe1810fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.436+0000 7fbe1d469640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbe18115c90 con 0x7fbe18107830 2024-04-25T13:54:31.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.436+0000 7fbe1d469640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbe18115b10 con 0x7fbe18101a10 2024-04-25T13:54:31.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.436+0000 7fbe1d469640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbe18115e10 con 0x7fbe18102b10 2024-04-25T13:54:31.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.437+0000 7fbe177fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18107830 0x7fbe1810fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.437+0000 7fbe177fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18107830 0x7fbe1810fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51168/0 (socket says 172.21.15.27:51168) 2024-04-25T13:54:31.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.437+0000 7fbe177fe640 1 -- 172.21.15.27:0/2602745155 learned_addr learned my addr 172.21.15.27:0/2602745155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.438+0000 7fbe177fe640 1 -- 172.21.15.27:0/2602745155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe18102b10 msgr2=0x7fbe1810ebf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:54:31.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.438+0000 7fbe177fe640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe18102b10 0x7fbe1810ebf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.438+0000 7fbe177fe640 1 -- 172.21.15.27:0/2602745155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe18101a10 msgr2=0x7fbe1810e6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:54:31.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.438+0000 7fbe177fe640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe18101a10 0x7fbe1810e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.438+0000 7fbe177fe640 1 -- 172.21.15.27:0/2602745155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe18110170 con 0x7fbe18107830 2024-04-25T13:54:31.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.443+0000 7fbe177fe640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18107830 0x7fbe1810fb00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000eaa0 tx=0x7fbe1000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.448+0000 7fbdfffff640 1 -- 172.21.15.27:0/2602745155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1000cdb0 con 0x7fbe18107830 2024-04-25T13:54:31.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.448+0000 7fbdfffff640 1 -- 172.21.15.27:0/2602745155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe10005020 con 0x7fbe18107830 2024-04-25T13:54:31.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.448+0000 7fbdfffff640 1 -- 172.21.15.27:0/2602745155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe100116d0 con 0x7fbe18107830 2024-04-25T13:54:31.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.448+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe18110460 con 0x7fbe18107830 2024-04-25T13:54:31.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.448+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe18110960 con 0x7fbe18107830 2024-04-25T13:54:31.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.453+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe040052c0 con 0x7fbe18107830 2024-04-25T13:54:31.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.454+0000 7fbdfffff640 1 -- 172.21.15.27:0/2602745155 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbe10004b20 con 0x7fbe18107830 2024-04-25T13:54:31.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.455+0000 7fbdfffff640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbe000afcf0 0x7fbe000b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.456+0000 7fbdfffff640 1 -- 172.21.15.27:0/2602745155 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbe100dee00 con 0x7fbe18107830 2024-04-25T13:54:31.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.460+0000 7fbe16ffd640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbe000afcf0 0x7fbe000b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.464+0000 7fbe16ffd640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbe000afcf0 0x7fbe000b2200 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009230 tx=0x7fbe0c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.470+0000 7fbdfffff640 1 -- 172.21.15.27:0/2602745155 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe10016030 con 0x7fbe18107830 2024-04-25T13:54:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.792+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe040055b0 con 0x7fbe18107830 2024-04-25T13:54:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.798+0000 7fbdfffff640 1 -- 172.21.15.27:0/2602745155 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fbe100a79b0 con 0x7fbe18107830 2024-04-25T13:54:31.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:54:31.798 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:54:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:54:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.804+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbe000afcf0 msgr2=0x7fbe000b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.804+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbe000afcf0 0x7fbe000b2200 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009230 tx=0x7fbe0c000bc0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.804+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18107830 msgr2=0x7fbe1810fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.804+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18107830 0x7fbe1810fb00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbe1000eaa0 tx=0x7fbe1000ef60 comp rx=0 tx=0).stop 2024-04-25T13:54:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.805+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 shutdown_connections 2024-04-25T13:54:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.805+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbe000afcf0 0x7fbe000b2200 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.805+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe18107830 0x7fbe1810fb00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.805+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe18102b10 0x7fbe1810ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.805+0000 7fbe1d469640 1 --2- 172.21.15.27:0/2602745155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe18101a10 0x7fbe1810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.805+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 >> 172.21.15.27:0/2602745155 conn(0x7fbe180fcef0 msgr2=0x7fbe18103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.809+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 shutdown_connections 2024-04-25T13:54:31.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.809+0000 7fbe1d469640 1 -- 172.21.15.27:0/2602745155 wait complete. 2024-04-25T13:54:31.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:54:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.982+0000 7fb056d55640 1 -- 172.21.15.27:0/1994817048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 msgr2=0x7fb050104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.982+0000 7fb056d55640 1 --2- 172.21.15.27:0/1994817048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 0x7fb050104800 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009900 tx=0x7fb04c02a410 comp rx=0 tx=0).stop 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.982+0000 7fb056d55640 1 -- 172.21.15.27:0/1994817048 shutdown_connections 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.982+0000 7fb056d55640 1 --2- 172.21.15.27:0/1994817048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb050109a40 0x7fb05010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.982+0000 7fb056d55640 1 --2- 172.21.15.27:0/1994817048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb050104d40 0x7fb050105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.982+0000 7fb056d55640 1 --2- 172.21.15.27:0/1994817048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 0x7fb050104800 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.982+0000 7fb056d55640 1 -- 172.21.15.27:0/1994817048 >> 172.21.15.27:0/1994817048 conn(0x7fb0500fecf0 msgr2=0x7fb0501011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.983+0000 7fb056d55640 1 -- 172.21.15.27:0/1994817048 shutdown_connections 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.983+0000 7fb056d55640 1 -- 172.21.15.27:0/1994817048 wait complete. 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.983+0000 7fb056d55640 1 Processor -- start 2024-04-25T13:54:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.983+0000 7fb056d55640 1 -- start start 2024-04-25T13:54:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.983+0000 7fb056d55640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 0x7fb05019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.983+0000 7fb056d55640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb050104d40 0x7fb05019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.983+0000 7fb056d55640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb050109a40 0x7fb0501a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb056d55640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb05010f600 con 0x7fb050103b00 2024-04-25T13:54:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb056d55640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb05010f480 con 0x7fb050104d40 2024-04-25T13:54:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb056d55640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb05010f780 con 0x7fb050109a40 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 0x7fb05019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 0x7fb05019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51198/0 (socket says 172.21.15.27:51198) 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 -- 172.21.15.27:0/3641510684 learned_addr learned my addr 172.21.15.27:0/3641510684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb0552f2640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb050109a40 0x7fb0501a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb047fff640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb050104d40 0x7fb05019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 -- 172.21.15.27:0/3641510684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb050109a40 msgr2=0x7fb0501a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb050109a40 0x7fb0501a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 -- 172.21.15.27:0/3641510684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb050104d40 msgr2=0x7fb05019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb050104d40 0x7fb05019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 -- 172.21.15.27:0/3641510684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0501a1da0 con 0x7fb050103b00 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb054af1640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 0x7fb05019a260 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb04c006010 tx=0x7fb04c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb045ffb640 1 -- 172.21.15.27:0/3641510684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c03f030 con 0x7fb050103b00 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb045ffb640 1 -- 172.21.15.27:0/3641510684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb04c003aa0 con 0x7fb050103b00 2024-04-25T13:54:31.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.984+0000 7fb045ffb640 1 -- 172.21.15.27:0/3641510684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c035070 con 0x7fb050103b00 2024-04-25T13:54:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.985+0000 7fb056d55640 1 -- 172.21.15.27:0/3641510684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0501a2030 con 0x7fb050103b00 2024-04-25T13:54:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.985+0000 7fb056d55640 1 -- 172.21.15.27:0/3641510684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0501a2520 con 0x7fb050103b00 2024-04-25T13:54:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.988+0000 7fb05dc05740 1 Processor -- start 2024-04-25T13:54:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.988+0000 7fb05dc05740 1 -- start start 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0dd390 0x56154a47f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0b44d0 0x56154a47fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a480a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb05dc05740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56154a3e3d80 con 0x56154a0bc6b0 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb05dc05740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56154a39da20 con 0x56154a0b44d0 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb05dc05740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56154a3dfd40 con 0x56154a0dd390 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb0552f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0dd390 0x56154a47f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb045ffb640 1 -- 172.21.15.27:0/3641510684 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb04c042030 con 0x7fb050103b00 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb0552f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0dd390 0x56154a47f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58260/0 (socket says 172.21.15.27:58260) 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb0552f2640 1 -- 172.21.15.27:0/2725592050 learned_addr learned my addr 172.21.15.27:0/2725592050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb047fff640 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a480a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb0552f2640 1 -- 172.21.15.27:0/2725592050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0b44d0 msgr2=0x56154a47fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.989+0000 7fb054af1640 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0b44d0 0x56154a47fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb0552f2640 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0b44d0 0x56154a47fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb0552f2640 1 -- 172.21.15.27:0/2725592050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 msgr2=0x56154a480a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb0552f2640 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a480a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb0552f2640 1 -- 172.21.15.27:0/2725592050 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56154a3a8c30 con 0x56154a0dd390 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb054af1640 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0b44d0 0x56154a47fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb0552f2640 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0dd390 0x56154a47f810 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb048010370 tx=0x7fb048010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11f525b5a34f10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb036ffd640 1 -- 172.21.15.27:0/2725592050 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048016070 con 0x56154a0dd390 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb036ffd640 1 -- 172.21.15.27:0/2725592050 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb048004d20 con 0x56154a0dd390 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb036ffd640 1 -- 172.21.15.27:0/2725592050 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048005390 con 0x56154a0dd390 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb045ffb640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb0380afaf0 0x7fb0380b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb045ffb640 1 -- 172.21.15.27:0/3641510684 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb04c0ff190 con 0x7fb050103b00 2024-04-25T13:54:31.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.990+0000 7fb054af1640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb0380afaf0 0x7fb0380b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 -- 172.21.15.27:0/2725592050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0dd390 msgr2=0x56154a47f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:31.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0dd390 0x56154a47f810 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb048010370 tx=0x7fb048010830 comp rx=0 tx=0).stop 2024-04-25T13:54:31.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb054af1640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb0380afaf0 0x7fb0380b2000 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x56154a47f530 tx=0x7fb04c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 -- 172.21.15.27:0/2725592050 shutdown_connections 2024-04-25T13:54:31.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a480a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0b44d0 0x56154a47fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2725592050 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0dd390 0x56154a47f810 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 -- 172.21.15.27:0/2725592050 >> 172.21.15.27:0/2725592050 conn(0x56154a0d14e0 msgr2=0x56154a47b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 -- 172.21.15.27:0/2725592050 shutdown_connections 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.991+0000 7fb05dc05740 1 -- 172.21.15.27:0/2725592050 wait complete. 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.992+0000 7fb05dc05740 1 -- start start 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.992+0000 7fb05dc05740 1 Processor -- start 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0b44d0 0x56154a48da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0bc6b0 0x56154a48df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a494e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb05dc05740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56154a481530 con 0x56154a0b44d0 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb05dc05740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56154a36adc0 con 0x56154a0dd390 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb05dc05740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x56154a4816b0 con 0x56154a0bc6b0 2024-04-25T13:54:31.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0b44d0 0x56154a48da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0b44d0 0x56154a48da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51220/0 (socket says 172.21.15.27:51220) 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 -- 172.21.15.27:0/3963514254 learned_addr learned my addr 172.21.15.27:0/3963514254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb047fff640 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0bc6b0 0x56154a48df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb05dc05740 10 client.0 _ll_register_callbacks cb 0x56154a386600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a494e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 -- 172.21.15.27:0/3963514254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0bc6b0 msgr2=0x56154a48df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0bc6b0 0x56154a48df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 -- 172.21.15.27:0/3963514254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 msgr2=0x56154a494e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a494e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 -- 172.21.15.27:0/3963514254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56154a0f53f0 con 0x56154a0b44d0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb0552f2640 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0b44d0 0x56154a48da10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb048004ff0 tx=0x7fb048005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d355c4e7fae3a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.993+0000 7fb02ffff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb02ffff640 1 -- 172.21.15.27:0/3963514254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048016070 con 0x56154a0b44d0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb02ffff640 1 -- 172.21.15.27:0/3963514254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb0480054a0 con 0x56154a0b44d0 2024-04-25T13:54:31.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 1 -- 172.21.15.27:0/3963514254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb048004d20 con 0x56154a0b44d0 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb02ffff640 1 -- 172.21.15.27:0/3963514254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb048031020 con 0x56154a0b44d0 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 10 client.18099 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 1 -- 172.21.15.27:0/3963514254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56154a1bf7d0 con 0x56154a0b44d0 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb02ffff640 1 -- 172.21.15.27:0/3963514254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb048004d20 con 0x56154a0b44d0 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb02ffff640 1 -- 172.21.15.27:0/3963514254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb04800f040 con 0x56154a0b44d0 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 10 client.18099 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 10 client.18099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 20 client.18099 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56154a09bf60 0x56154a49b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 4 client.18099 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.994+0000 7fb05dc05740 1 -- 172.21.15.27:0/3963514254 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56154a3f1990 con 0x56154a09bf60 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.995+0000 7fb047fff640 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56154a09bf60 0x56154a49b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.996+0000 7fb047fff640 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56154a09bf60 0x56154a49b220 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1e1534298c0d28e server_cookie=bb0801a3a92a3272 in_seq=0 out_seq=0 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.996+0000 7fb02ffff640 10 client.18099 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.996+0000 7fb02ffff640 1 -- 172.21.15.27:0/3963514254 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56154a3f1990 con 0x56154a09bf60 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.996+0000 7fb02ffff640 10 client.18099 handle_command_reply: tid=0 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.996+0000 7fb05dc05740 1 client.18099 shutdown 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 -- 172.21.15.27:0/3963514254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0b44d0 msgr2=0x56154a48da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0b44d0 0x56154a48da10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb048004ff0 tx=0x7fb048005c50 comp rx=0 tx=0).stop 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 -- 172.21.15.27:0/3963514254 shutdown_connections 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56154a09bf60 0x56154a49b220 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a494e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0bc6b0 0x56154a48df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb02ffff640 0 client.18099 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 --2- 172.21.15.27:0/3963514254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0b44d0 0x56154a48da10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 -- 172.21.15.27:0/3963514254 >> 172.21.15.27:0/3963514254 conn(0x56154a0d14e0 msgr2=0x56154a47a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:32.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 -- 172.21.15.27:0/3963514254 shutdown_connections 2024-04-25T13:54:32.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 1 -- 172.21.15.27:0/3963514254 wait complete. 2024-04-25T13:54:32.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:31.997+0000 7fb05dc05740 20 client.18099 trim_cache size 0 max 16384 2024-04-25T13:54:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.023+0000 7fb05dc05740 1 Processor -- start 2024-04-25T13:54:32.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.024+0000 7fb05dc05740 1 -- start start 2024-04-25T13:54:32.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.024+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:32.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.024+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a48ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:32.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.024+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a48db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:32.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.024+0000 7fb05dc05740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb0501a6fa0 con 0x56154a0bc6b0 2024-04-25T13:54:32.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.024+0000 7fb05dc05740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0501a9820 con 0x56154a0dd390 2024-04-25T13:54:32.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.024+0000 7fb05dc05740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb0501a7610 con 0x56154a0b44d0 2024-04-25T13:54:32.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.028+0000 7fb0552f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:32.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.028+0000 7fb0552f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a48db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:32.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.030+0000 7fb047fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a48ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:32.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.030+0000 7fb047fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a48ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51224/0 (socket says 172.21.15.27:51224) 2024-04-25T13:54:32.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.030+0000 7fb047fff640 1 -- 172.21.15.27:0/4023169295 learned_addr learned my addr 172.21.15.27:0/4023169295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.037+0000 7fb0552f2640 1 -- 172.21.15.27:0/4023169295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 msgr2=0x56154a48db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.037+0000 7fb0552f2640 1 --2- 172.21.15.27:0/4023169295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a48db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.037+0000 7fb0552f2640 1 -- 172.21.15.27:0/4023169295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 msgr2=0x56154a48ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.037+0000 7fb0552f2640 1 --2- 172.21.15.27:0/4023169295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a48ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.037+0000 7fb0552f2640 1 -- 172.21.15.27:0/4023169295 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56154a4767f0 con 0x56154a0b44d0 2024-04-25T13:54:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.041+0000 7fb0552f2640 1 --2- 172.21.15.27:0/4023169295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48d5e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb048000f80 tx=0x7fb048010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21d810e8d91c82b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.043+0000 7fb0367fc640 1 -- 172.21.15.27:0/4023169295 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048016070 con 0x56154a0b44d0 2024-04-25T13:54:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.043+0000 7fb0367fc640 1 -- 172.21.15.27:0/4023169295 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb048025040 con 0x56154a0b44d0 2024-04-25T13:54:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.043+0000 7fb0367fc640 1 -- 172.21.15.27:0/4023169295 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048003bd0 con 0x56154a0b44d0 2024-04-25T13:54:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.045+0000 7fb05dc05740 1 -- 172.21.15.27:0/4023169295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 msgr2=0x56154a48d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.045+0000 7fb05dc05740 1 --2- 172.21.15.27:0/4023169295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48d5e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb048000f80 tx=0x7fb048010830 comp rx=0 tx=0).stop 2024-04-25T13:54:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.048+0000 7fb05dc05740 1 -- 172.21.15.27:0/4023169295 shutdown_connections 2024-04-25T13:54:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.048+0000 7fb05dc05740 1 --2- 172.21.15.27:0/4023169295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a48db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.048+0000 7fb05dc05740 1 --2- 172.21.15.27:0/4023169295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a48ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.048+0000 7fb05dc05740 1 --2- 172.21.15.27:0/4023169295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48d5e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.048+0000 7fb05dc05740 1 -- 172.21.15.27:0/4023169295 >> 172.21.15.27:0/4023169295 conn(0x56154a0d14e0 msgr2=0x56154a4842f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:32.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.055+0000 7fb05dc05740 1 -- 172.21.15.27:0/4023169295 shutdown_connections 2024-04-25T13:54:32.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.057+0000 7fb05dc05740 1 -- 172.21.15.27:0/4023169295 wait complete. 2024-04-25T13:54:32.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.057+0000 7fb05dc05740 1 -- start start 2024-04-25T13:54:32.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.057+0000 7fb05dc05740 1 Processor -- start 2024-04-25T13:54:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.061+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.061+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a48b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.061+0000 7fb05dc05740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a505ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.061+0000 7fb05dc05740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb0501a7ce0 con 0x56154a0bc6b0 2024-04-25T13:54:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.061+0000 7fb05dc05740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0501a9150 con 0x56154a0dd390 2024-04-25T13:54:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.061+0000 7fb05dc05740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb0501a83b0 con 0x56154a0b44d0 2024-04-25T13:54:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.061+0000 7fb05dc05740 10 client.0 _ll_register_callbacks cb 0x7fb048009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.066+0000 7fb0552f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.066+0000 7fb0552f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58274/0 (socket says 172.21.15.27:58274) 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.066+0000 7fb0552f2640 1 -- 172.21.15.27:0/2800221024 learned_addr learned my addr 172.21.15.27:0/2800221024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.066+0000 7fb0552f2640 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a505ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.067+0000 7fb0552f2640 1 -- 172.21.15.27:0/2800221024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 msgr2=0x56154a505ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.067+0000 7fb0552f2640 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a505ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.067+0000 7fb0552f2640 1 -- 172.21.15.27:0/2800221024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 msgr2=0x56154a48b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.067+0000 7fb0552f2640 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a48b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.067+0000 7fb0552f2640 1 -- 172.21.15.27:0/2800221024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56154a495140 con 0x56154a0b44d0 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.069+0000 7fb0552f2640 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48afb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb048012570 tx=0x7fb048010ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8df8f9e8f1dcd586 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.069+0000 7fb02ffff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:32.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.070+0000 7fb02ffff640 1 -- 172.21.15.27:0/2800221024 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04803b020 con 0x56154a0b44d0 2024-04-25T13:54:32.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.070+0000 7fb05dc05740 1 -- 172.21.15.27:0/2800221024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb048025040 con 0x56154a0b44d0 2024-04-25T13:54:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.070+0000 7fb02ffff640 1 -- 172.21.15.27:0/2800221024 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb04801aba0 con 0x56154a0b44d0 2024-04-25T13:54:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.071+0000 7fb02ffff640 1 -- 172.21.15.27:0/2800221024 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb048016070 con 0x56154a0b44d0 2024-04-25T13:54:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.072+0000 7fb05dc05740 10 client.27826 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.072+0000 7fb05dc05740 1 -- 172.21.15.27:0/2800221024 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x56154a518180 con 0x56154a0b44d0 2024-04-25T13:54:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.076+0000 7fb02ffff640 1 -- 172.21.15.27:0/2800221024 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb048025040 con 0x56154a0b44d0 2024-04-25T13:54:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.076+0000 7fb02ffff640 1 -- 172.21.15.27:0/2800221024 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb048044040 con 0x56154a0b44d0 2024-04-25T13:54:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.076+0000 7fb05dc05740 10 client.27826 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.076+0000 7fb05dc05740 10 client.27826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.076+0000 7fb05dc05740 20 client.27826 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.076+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56154a0713a0 0x56154a51aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.076+0000 7fb05dc05740 4 client.27826 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:54:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.076+0000 7fb05dc05740 1 -- 172.21.15.27:0/2800221024 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56154a3eb800 con 0x56154a0713a0 2024-04-25T13:54:32.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.078+0000 7fb047fff640 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56154a0713a0 0x56154a51aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.085+0000 7fb047fff640 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56154a0713a0 0x56154a51aa10 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e19c319e24a1aa3 server_cookie=a750aaa424a5f838 in_seq=0 out_seq=0 2024-04-25T13:54:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.085+0000 7fb02ffff640 10 client.27826 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.087+0000 7fb02ffff640 1 -- 172.21.15.27:0/2800221024 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56154a3eb800 con 0x56154a0713a0 2024-04-25T13:54:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.087+0000 7fb02ffff640 10 client.27826 handle_command_reply: tid=0 2024-04-25T13:54:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.087+0000 7fb05dc05740 1 client.27826 shutdown 2024-04-25T13:54:32.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.092+0000 7fb05dc05740 1 -- 172.21.15.27:0/2800221024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 msgr2=0x56154a48afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:32.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.092+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48afb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb048012570 tx=0x7fb048010ea0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.093+0000 7fb05dc05740 1 -- 172.21.15.27:0/2800221024 shutdown_connections 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.093+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x56154a0713a0 0x56154a51aa10 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.093+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56154a0dd390 0x56154a505ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.093+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56154a0bc6b0 0x56154a48b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.093+0000 7fb05dc05740 1 --2- 172.21.15.27:0/2800221024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56154a0b44d0 0x56154a48afb0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.093+0000 7fb05dc05740 1 -- 172.21.15.27:0/2800221024 >> 172.21.15.27:0/2800221024 conn(0x56154a0d14e0 msgr2=0x56154a47de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.093+0000 7fb02ffff640 0 client.27826 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.094+0000 7fb05dc05740 1 -- 172.21.15.27:0/2800221024 shutdown_connections 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.094+0000 7fb05dc05740 1 -- 172.21.15.27:0/2800221024 wait complete. 2024-04-25T13:54:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.094+0000 7fb05dc05740 20 client.27826 trim_cache size 0 max 16384 2024-04-25T13:54:32.095 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:54:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.096+0000 7fb056d55640 1 -- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb0380afaf0 msgr2=0x7fb0380b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.096+0000 7fb056d55640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb0380afaf0 0x7fb0380b2000 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x56154a47f530 tx=0x7fb04c031400 comp rx=0 tx=0).stop 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.096+0000 7fb056d55640 1 -- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 msgr2=0x7fb05019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.096+0000 7fb056d55640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 0x7fb05019a260 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb04c006010 tx=0x7fb04c004270 comp rx=0 tx=0).stop 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.097+0000 7fb056d55640 1 -- 172.21.15.27:0/3641510684 shutdown_connections 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.097+0000 7fb056d55640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb0380afaf0 0x7fb0380b2000 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.097+0000 7fb056d55640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb050109a40 0x7fb0501a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.097+0000 7fb056d55640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb050104d40 0x7fb05019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.097+0000 7fb056d55640 1 --2- 172.21.15.27:0/3641510684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb050103b00 0x7fb05019a260 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.097+0000 7fb056d55640 1 -- 172.21.15.27:0/3641510684 >> 172.21.15.27:0/3641510684 conn(0x7fb0500fecf0 msgr2=0x7fb050105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.097+0000 7fb056d55640 1 -- 172.21.15.27:0/3641510684 shutdown_connections 2024-04-25T13:54:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:32.097+0000 7fb056d55640 1 -- 172.21.15.27:0/3641510684 wait complete. 2024-04-25T13:54:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:32 smithi027 ceph-mon[29140]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 MiB/s wr, 625 op/s 2024-04-25T13:54:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:32 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2602745155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:54:32.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:32 smithi122 ceph-mon[32843]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 MiB/s wr, 625 op/s 2024-04-25T13:54:32.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:32 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2602745155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:54:32.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:32 smithi045 ceph-mon[32731]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 MiB/s wr, 625 op/s 2024-04-25T13:54:32.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:32 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2602745155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:54:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 805.0 s, 1374.4 tps, lat 71.872 ms stddev 84.795 2024-04-25T13:54:33.114 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:54:33.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:54:33.114 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:54:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.287+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2939199622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f4110030 msgr2=0x7ff0f4110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.287+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2939199622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f4110030 0x7ff0f4110440 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009ca0 tx=0x7ff0ec02a8f0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.293+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2939199622 shutdown_connections 2024-04-25T13:54:33.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.293+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2939199622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f4110030 0x7ff0f4110440 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.293+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2939199622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f410b310 0x7ff0f410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.293+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2939199622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f410a420 0x7ff0f410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.293+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2939199622 >> 172.21.15.27:0/2939199622 conn(0x7ff0f40704e0 msgr2=0x7ff0f4072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.299+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2939199622 shutdown_connections 2024-04-25T13:54:33.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.303+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2939199622 wait complete. 2024-04-25T13:54:33.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.303+0000 7ff0fbbf3640 1 Processor -- start 2024-04-25T13:54:33.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.304+0000 7ff0fbbf3640 1 -- start start 2024-04-25T13:54:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.304+0000 7ff0fbbf3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f410a420 0x7ff0f4117050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.304+0000 7ff0fbbf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f410b310 0x7ff0f4117590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.304+0000 7ff0fbbf3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f4110030 0x7ff0f41184f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.304+0000 7ff0fbbf3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f411e490 con 0x7ff0f410a420 2024-04-25T13:54:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.304+0000 7ff0fbbf3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f411e310 con 0x7ff0f410b310 2024-04-25T13:54:33.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.304+0000 7ff0fbbf3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f411e610 con 0x7ff0f4110030 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0fa190640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f4110030 0x7ff0f41184f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0f998f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f410a420 0x7ff0f4117050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0fa190640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f4110030 0x7ff0f41184f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58298/0 (socket says 172.21.15.27:58298) 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0fa190640 1 -- 172.21.15.27:0/2383736715 learned_addr learned my addr 172.21.15.27:0/2383736715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0f918e640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f410b310 0x7ff0f4117590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0f918e640 1 -- 172.21.15.27:0/2383736715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f4110030 msgr2=0x7ff0f41184f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0f918e640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f4110030 0x7ff0f41184f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0f918e640 1 -- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f410a420 msgr2=0x7ff0f4117050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0f918e640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f410a420 0x7ff0f4117050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.308+0000 7ff0f918e640 1 -- 172.21.15.27:0/2383736715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f4118b60 con 0x7ff0f410b310 2024-04-25T13:54:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.309+0000 7ff0f918e640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f410b310 0x7ff0f4117590 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400e9f0 tx=0x7ff0e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.309+0000 7ff0f998f640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f410a420 0x7ff0f4117050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.316+0000 7ff0eaffd640 1 -- 172.21.15.27:0/2383736715 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e400ccc0 con 0x7ff0f410b310 2024-04-25T13:54:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.316+0000 7ff0eaffd640 1 -- 172.21.15.27:0/2383736715 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e40108e0 con 0x7ff0f410b310 2024-04-25T13:54:33.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.316+0000 7ff0eaffd640 1 -- 172.21.15.27:0/2383736715 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40188e0 con 0x7ff0f410b310 2024-04-25T13:54:33.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.317+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2383736715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f4118e50 con 0x7ff0f410b310 2024-04-25T13:54:33.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.317+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2383736715 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f4119350 con 0x7ff0f410b310 2024-04-25T13:54:33.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.323+0000 7ff0eaffd640 1 -- 172.21.15.27:0/2383736715 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff0e401f020 con 0x7ff0f410b310 2024-04-25T13:54:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.324+0000 7ff102b7b740 1 Processor -- start 2024-04-25T13:54:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.325+0000 7ff0eaffd640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff0e00afd00 0x7ff0e00b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.325+0000 7ff0eaffd640 1 -- 172.21.15.27:0/2383736715 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff0e40deeb0 con 0x7ff0f410b310 2024-04-25T13:54:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.327+0000 7ff0fa190640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff0e00afd00 0x7ff0e00b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.327+0000 7ff102b7b740 1 -- start start 2024-04-25T13:54:33.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.327+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e98410 0x563d092394a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.327+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e6f550 0x563d09239a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.327+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09240920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.327+0000 7ff102b7b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563d0919eb00 con 0x563d08e77730 2024-04-25T13:54:33.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.327+0000 7ff102b7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563d0913a920 con 0x563d08e6f550 2024-04-25T13:54:33.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.327+0000 7ff102b7b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563d0919aac0 con 0x563d08e98410 2024-04-25T13:54:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0f918e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e6f550 0x563d09239a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0f918e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e6f550 0x563d09239a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:58982/0 (socket says 172.21.15.27:58982) 2024-04-25T13:54:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0f918e640 1 -- 172.21.15.27:0/3273434971 learned_addr learned my addr 172.21.15.27:0/3273434971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0fa190640 1 --2- 172.21.15.27:0/3273434971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09240920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0f918e640 1 -- 172.21.15.27:0/3273434971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e98410 msgr2=0x563d092394a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:54:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0f918e640 1 --2- 172.21.15.27:0/3273434971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e98410 0x563d092394a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0f918e640 1 -- 172.21.15.27:0/3273434971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 msgr2=0x563d09240920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0f918e640 1 --2- 172.21.15.27:0/3273434971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09240920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.328+0000 7ff0f918e640 1 -- 172.21.15.27:0/3273434971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d08f787e0 con 0x563d08e6f550 2024-04-25T13:54:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.329+0000 7ff0f918e640 1 --2- 172.21.15.27:0/3273434971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e6f550 0x563d09239a20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40a72f0 tx=0x7ff0e40a6e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c650d2ae2f2f4c35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.329+0000 7ff0fa190640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff0e00afd00 0x7ff0e00b2210 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec005e30 tx=0x7ff0ec004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.334+0000 7ff0dffff640 1 -- 172.21.15.27:0/3273434971 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40c1070 con 0x563d08e6f550 2024-04-25T13:54:33.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.334+0000 7ff0dffff640 1 -- 172.21.15.27:0/3273434971 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e40ab710 con 0x563d08e6f550 2024-04-25T13:54:33.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.335+0000 7ff102b7b740 1 -- 172.21.15.27:0/3273434971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e6f550 msgr2=0x563d09239a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.335+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3273434971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e6f550 0x563d09239a20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40a72f0 tx=0x7ff0e40a6e70 comp rx=0 tx=0).stop 2024-04-25T13:54:33.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.336+0000 7ff102b7b740 1 -- 172.21.15.27:0/3273434971 shutdown_connections 2024-04-25T13:54:33.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.336+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3273434971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09240920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.336+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3273434971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e6f550 0x563d09239a20 secure :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40a72f0 tx=0x7ff0e40a6e70 comp rx=0 tx=0).stop 2024-04-25T13:54:33.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.336+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3273434971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e98410 0x563d092394a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.336+0000 7ff102b7b740 1 -- 172.21.15.27:0/3273434971 >> 172.21.15.27:0/3273434971 conn(0x563d08e8c560 msgr2=0x563d09234e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.340+0000 7ff102b7b740 1 -- 172.21.15.27:0/3273434971 shutdown_connections 2024-04-25T13:54:33.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.342+0000 7ff102b7b740 1 -- 172.21.15.27:0/3273434971 wait complete. 2024-04-25T13:54:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.342+0000 7ff102b7b740 1 -- start start 2024-04-25T13:54:33.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.342+0000 7ff102b7b740 1 Processor -- start 2024-04-25T13:54:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.343+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e77730 0x563d092434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.343+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e98410 0x563d09243a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.343+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d0924a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.343+0000 7ff102b7b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563d09241360 con 0x563d08e98410 2024-04-25T13:54:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.343+0000 7ff102b7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563d091385d0 con 0x563d08e77730 2024-04-25T13:54:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.343+0000 7ff102b7b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563d092414e0 con 0x563d08e571f0 2024-04-25T13:54:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.343+0000 7ff102b7b740 10 client.0 _ll_register_callbacks cb 0x563d09126080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:33.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.344+0000 7ff0f918e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e98410 0x563d09243a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.344+0000 7ff0f918e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e98410 0x563d09243a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51272/0 (socket says 172.21.15.27:51272) 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.344+0000 7ff0f918e640 1 -- 172.21.15.27:0/3608048013 learned_addr learned my addr 172.21.15.27:0/3608048013 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.344+0000 7ff0f918e640 1 -- 172.21.15.27:0/3608048013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 msgr2=0x563d0924a990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.344+0000 7ff0f918e640 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d0924a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.344+0000 7ff0f918e640 1 -- 172.21.15.27:0/3608048013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e77730 msgr2=0x563d092434f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.344+0000 7ff0f918e640 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e77730 0x563d092434f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.344+0000 7ff0f918e640 1 -- 172.21.15.27:0/3608048013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d08fd8f70 con 0x563d08e98410 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.345+0000 7ff0f918e640 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e98410 0x563d09243a30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400e9c0 tx=0x7ff0e40a3340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322f56af3b0b9402 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.345+0000 7ff0dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:54:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.345+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/3608048013 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40c1040 con 0x563d08e98410 2024-04-25T13:54:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.345+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/3608048013 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff0e40a3950 con 0x563d08e98410 2024-04-25T13:54:33.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.346+0000 7ff102b7b740 1 -- 172.21.15.27:0/3608048013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0e40ab710 con 0x563d08e98410 2024-04-25T13:54:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.346+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/3608048013 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff0e40aff90 con 0x563d08e98410 2024-04-25T13:54:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.346+0000 7ff102b7b740 10 client.18114 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:33.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.346+0000 7ff102b7b740 1 -- 172.21.15.27:0/3608048013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563d08e70170 con 0x563d08e98410 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.347+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/3608048013 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e40ab710 con 0x563d08e98410 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.347+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/3608048013 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff0e40ab170 con 0x563d08e98410 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.347+0000 7ff102b7b740 10 client.18114 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.347+0000 7ff102b7b740 10 client.18114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.347+0000 7ff102b7b740 20 client.18114 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.347+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d08e2c630 0x563d09250ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.347+0000 7ff102b7b740 4 client.18114 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.347+0000 7ff102b7b740 1 -- 172.21.15.27:0/3608048013 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d091ac710 con 0x563d08e2c630 2024-04-25T13:54:33.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.348+0000 7ff0f998f640 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d08e2c630 0x563d09250ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.349+0000 7ff0f998f640 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d08e2c630 0x563d09250ee0 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cbdaa927d1dfda2 server_cookie=28e1faa7749ab0f in_seq=0 out_seq=0 2024-04-25T13:54:33.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.349+0000 7ff0dd7fa640 10 client.18114 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.350+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/3608048013 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563d091ac710 con 0x563d08e2c630 2024-04-25T13:54:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.350+0000 7ff0dd7fa640 10 client.18114 handle_command_reply: tid=0 2024-04-25T13:54:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.350+0000 7ff102b7b740 1 client.18114 shutdown 2024-04-25T13:54:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.350+0000 7ff102b7b740 1 -- 172.21.15.27:0/3608048013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e98410 msgr2=0x563d09243a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.350+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e98410 0x563d09243a30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400e9c0 tx=0x7ff0e40a3340 comp rx=0 tx=0).stop 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 1 -- 172.21.15.27:0/3608048013 shutdown_connections 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d08e2c630 0x563d09250ee0 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d0924a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e98410 0x563d09243a30 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 1 --2- 172.21.15.27:0/3608048013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e77730 0x563d092434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 1 -- 172.21.15.27:0/3608048013 >> 172.21.15.27:0/3608048013 conn(0x563d08e8c560 msgr2=0x563d09233ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff0dd7fa640 0 client.18114 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 1 -- 172.21.15.27:0/3608048013 shutdown_connections 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 1 -- 172.21.15.27:0/3608048013 wait complete. 2024-04-25T13:54:33.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.351+0000 7ff102b7b740 20 client.18114 trim_cache size 0 max 16384 2024-04-25T13:54:33.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.378+0000 7ff102b7b740 1 Processor -- start 2024-04-25T13:54:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff102b7b740 1 -- start start 2024-04-25T13:54:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e571f0 0x563d092556d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e77730 0x563d0924a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 0x563d0924ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff102b7b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f41c0850 con 0x563d08e571f0 2024-04-25T13:54:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff102b7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f41c30d0 con 0x563d08e98410 2024-04-25T13:54:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff102b7b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f41c0ec0 con 0x563d08e77730 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff0f918e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e77730 0x563d0924a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.379+0000 7ff0f918e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e77730 0x563d0924a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58320/0 (socket says 172.21.15.27:58320) 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0f918e640 1 -- 172.21.15.27:0/2343330878 learned_addr learned my addr 172.21.15.27:0/2343330878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0fa190640 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 0x563d0924ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0f998f640 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e571f0 0x563d092556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0f918e640 1 -- 172.21.15.27:0/2343330878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 msgr2=0x563d0924ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0f918e640 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 0x563d0924ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0f918e640 1 -- 172.21.15.27:0/2343330878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e571f0 msgr2=0x563d092556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0f918e640 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e571f0 0x563d092556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0f918e640 1 -- 172.21.15.27:0/2343330878 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d09238fb0 con 0x563d08e77730 2024-04-25T13:54:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.380+0000 7ff0f918e640 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e77730 0x563d0924a950 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400c6a0 tx=0x7ff0e40a69f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edf852725bfc32d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.381+0000 7ff0df7fe640 1 -- 172.21.15.27:0/2343330878 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40c1070 con 0x563d08e77730 2024-04-25T13:54:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.381+0000 7ff0df7fe640 1 -- 172.21.15.27:0/2343330878 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e40a67d0 con 0x563d08e77730 2024-04-25T13:54:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.382+0000 7ff0df7fe640 1 -- 172.21.15.27:0/2343330878 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40b01c0 con 0x563d08e77730 2024-04-25T13:54:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.382+0000 7ff102b7b740 1 -- 172.21.15.27:0/2343330878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e77730 msgr2=0x563d0924a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.382+0000 7ff102b7b740 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e77730 0x563d0924a950 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400c6a0 tx=0x7ff0e40a69f0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 -- 172.21.15.27:0/2343330878 shutdown_connections 2024-04-25T13:54:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 0x563d0924ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e77730 0x563d0924a950 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 --2- 172.21.15.27:0/2343330878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e571f0 0x563d092556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 -- 172.21.15.27:0/2343330878 >> 172.21.15.27:0/2343330878 conn(0x563d08e8c560 msgr2=0x563d09241b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 -- 172.21.15.27:0/2343330878 shutdown_connections 2024-04-25T13:54:33.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 -- 172.21.15.27:0/2343330878 wait complete. 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 -- start start 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.383+0000 7ff102b7b740 1 Processor -- start 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d09249530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09249a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff102b7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 0x563d09249fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff102b7b740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f41c1590 con 0x563d08e77730 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff102b7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f41c2a00 con 0x563d08e98410 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff102b7b740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f41c1c60 con 0x563d08e571f0 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff102b7b740 10 client.0 _ll_register_callbacks cb 0x7ff0e4010760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f918e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09249a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d09249530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d09249530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58324/0 (socket says 172.21.15.27:58324) 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 -- 172.21.15.27:0/344542670 learned_addr learned my addr 172.21.15.27:0/344542670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0fa190640 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 0x563d09249fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 -- 172.21.15.27:0/344542670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 msgr2=0x563d09249fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 0x563d09249fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 -- 172.21.15.27:0/344542670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 msgr2=0x563d09249a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09249a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 -- 172.21.15.27:0/344542670 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d0923e330 con 0x563d08e571f0 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f918e640 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09249a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0f998f640 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d09249530 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005940 tx=0x7ff0f00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85678dfc8d9994df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.384+0000 7ff0dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.385+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/344542670 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f001a890 con 0x563d08e571f0 2024-04-25T13:54:33.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.385+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/344542670 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff0f001aa30 con 0x563d08e571f0 2024-04-25T13:54:33.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.385+0000 7ff102b7b740 1 -- 172.21.15.27:0/344542670 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0e40a67d0 con 0x563d08e571f0 2024-04-25T13:54:33.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.385+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/344542670 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff0f0004020 con 0x563d08e571f0 2024-04-25T13:54:33.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.385+0000 7ff102b7b740 10 client.27838 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:33.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.385+0000 7ff102b7b740 1 -- 172.21.15.27:0/344542670 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563d09241900 con 0x563d08e571f0 2024-04-25T13:54:33.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.385+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/344542670 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e40a67d0 con 0x563d08e571f0 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.386+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/344542670 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff0f00035f0 con 0x563d08e571f0 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.386+0000 7ff102b7b740 10 client.27838 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.386+0000 7ff102b7b740 10 client.27838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.386+0000 7ff102b7b740 20 client.27838 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.386+0000 7ff102b7b740 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d092b8ac0 0x563d092baee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.386+0000 7ff102b7b740 4 client.27838 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.386+0000 7ff102b7b740 1 -- 172.21.15.27:0/344542670 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563d091a6580 con 0x563d092b8ac0 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.386+0000 7ff0f918e640 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d092b8ac0 0x563d092baee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.388+0000 7ff0f918e640 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d092b8ac0 0x563d092baee0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2325ae2c71058411 server_cookie=2ceafe56bc1479e0 in_seq=0 out_seq=0 2024-04-25T13:54:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.388+0000 7ff0dd7fa640 10 client.27838 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:33.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.389+0000 7ff0dd7fa640 1 -- 172.21.15.27:0/344542670 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563d091a6580 con 0x563d092b8ac0 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.389+0000 7ff0dd7fa640 10 client.27838 handle_command_reply: tid=0 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.389+0000 7ff102b7b740 1 client.27838 shutdown 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.389+0000 7ff102b7b740 1 -- 172.21.15.27:0/344542670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 msgr2=0x563d09249530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.389+0000 7ff102b7b740 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d09249530 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005940 tx=0x7ff0f00118d0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 1 -- 172.21.15.27:0/344542670 shutdown_connections 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d092b8ac0 0x563d092baee0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d08e98410 0x563d09249fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d08e77730 0x563d09249a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0dd7fa640 0 client.27838 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 1 --2- 172.21.15.27:0/344542670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d08e571f0 0x563d09249530 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 1 -- 172.21.15.27:0/344542670 >> 172.21.15.27:0/344542670 conn(0x563d08e8c560 msgr2=0x563d09237c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 1 -- 172.21.15.27:0/344542670 shutdown_connections 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 1 -- 172.21.15.27:0/344542670 wait complete. 2024-04-25T13:54:33.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff102b7b740 20 client.27838 trim_cache size 0 max 16384 2024-04-25T13:54:33.391 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:54:33.391 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:54:33.391 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "f66492b3-d9f5-488b-853b-e67b47274a80", 2024-04-25T13:54:33.391 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:54:33.391 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff0e00afd00 msgr2=0x7ff0e00b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff0e00afd00 0x7ff0e00b2210 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec005e30 tx=0x7ff0ec004a40 comp rx=0 tx=0).stop 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2383736715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f410b310 msgr2=0x7ff0f4117590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f410b310 0x7ff0f4117590 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400e9f0 tx=0x7ff0e400eeb0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2383736715 shutdown_connections 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff0e00afd00 0x7ff0e00b2210 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0f4110030 0x7ff0f41184f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0f410b310 0x7ff0f4117590 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 --2- 172.21.15.27:0/2383736715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f410a420 0x7ff0f4117050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.390+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2383736715 >> 172.21.15.27:0/2383736715 conn(0x7ff0f40704e0 msgr2=0x7ff0f410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.391+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2383736715 shutdown_connections 2024-04-25T13:54:33.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.391+0000 7ff0fbbf3640 1 -- 172.21.15.27:0/2383736715 wait complete. 2024-04-25T13:54:33.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:54:33.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:54:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.586+0000 7f496992e640 1 -- 172.21.15.27:0/2595447566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f496410a0f0 msgr2=0x7f496410adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.586+0000 7f496992e640 1 --2- 172.21.15.27:0/2595447566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f496410a0f0 0x7f496410adf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f494c009900 tx=0x7f494c02a3f0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.587+0000 7f496992e640 1 -- 172.21.15.27:0/2595447566 shutdown_connections 2024-04-25T13:54:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.587+0000 7f496992e640 1 --2- 172.21.15.27:0/2595447566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4964110030 0x7f4964112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.587+0000 7f496992e640 1 --2- 172.21.15.27:0/2595447566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f496410b330 0x7f496410b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.587+0000 7f496992e640 1 --2- 172.21.15.27:0/2595447566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f496410a0f0 0x7f496410adf0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.587+0000 7f496992e640 1 -- 172.21.15.27:0/2595447566 >> 172.21.15.27:0/2595447566 conn(0x7f4964078de0 msgr2=0x7f49641079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.587+0000 7f496992e640 1 -- 172.21.15.27:0/2595447566 shutdown_connections 2024-04-25T13:54:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.587+0000 7f496992e640 1 -- 172.21.15.27:0/2595447566 wait complete. 2024-04-25T13:54:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.587+0000 7f496992e640 1 Processor -- start 2024-04-25T13:54:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f496992e640 1 -- start start 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f496992e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f496410a0f0 0x7f49641a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f496992e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f496410b330 0x7f49641a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f496992e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4964110030 0x7f49641a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f496992e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4964115c10 con 0x7f496410a0f0 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f496992e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4964115a90 con 0x7f496410b330 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f496992e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4964115d90 con 0x7f4964110030 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f49637fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4964110030 0x7f49641a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f49637fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4964110030 0x7f49641a7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58330/0 (socket says 172.21.15.27:58330) 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.588+0000 7f49637fe640 1 -- 172.21.15.27:0/4268460971 learned_addr learned my addr 172.21.15.27:0/4268460971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.589+0000 7f495bfff640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f496410b330 0x7f49641a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.589+0000 7f49637fe640 1 -- 172.21.15.27:0/4268460971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f496410b330 msgr2=0x7f49641a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.589+0000 7f49637fe640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f496410b330 0x7f49641a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.589+0000 7f49637fe640 1 -- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f496410a0f0 msgr2=0x7f49641a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.589+0000 7f49637fe640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f496410a0f0 0x7f49641a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.589+0000 7f49637fe640 1 -- 172.21.15.27:0/4268460971 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49641a83e0 con 0x7f4964110030 2024-04-25T13:54:33.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.589+0000 7f49637fe640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4964110030 0x7f49641a7d70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f495c00eb10 tx=0x7f495c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.590+0000 7f496892c640 1 -- 172.21.15.27:0/4268460971 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c004b20 con 0x7f4964110030 2024-04-25T13:54:33.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.590+0000 7f496892c640 1 -- 172.21.15.27:0/4268460971 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f495c004cc0 con 0x7f4964110030 2024-04-25T13:54:33.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.590+0000 7f496992e640 1 -- 172.21.15.27:0/4268460971 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49641a86d0 con 0x7f4964110030 2024-04-25T13:54:33.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.590+0000 7f496892c640 1 -- 172.21.15.27:0/4268460971 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c010670 con 0x7f4964110030 2024-04-25T13:54:33.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.590+0000 7f496992e640 1 -- 172.21.15.27:0/4268460971 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49641a8c20 con 0x7f4964110030 2024-04-25T13:54:33.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.593+0000 7f496892c640 1 -- 172.21.15.27:0/4268460971 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f495c005020 con 0x7f4964110030 2024-04-25T13:54:33.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.594+0000 7f49707db740 1 Processor -- start 2024-04-25T13:54:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.594+0000 7f49707db740 1 -- start start 2024-04-25T13:54:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.594+0000 7f496892c640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afc40 0x7f49500b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f49707db740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edd0b390 0x5628ee0aca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f49707db740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f49707db740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcea6b0 0x5628ee0adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f49707db740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5628ee011d80 con 0x5628edcea6b0 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f49707db740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5628edfcb8b0 con 0x5628edd0b390 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f49707db740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5628ee00dd40 con 0x5628edce24d0 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f496892c640 1 -- 172.21.15.27:0/4268460971 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f495c0df1b0 con 0x7f4964110030 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f4962ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edd0b390 0x5628ee0aca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f4962ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edd0b390 0x5628ee0aca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:59048/0 (socket says 172.21.15.27:59048) 2024-04-25T13:54:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f4962ffd640 1 -- 172.21.15.27:0/4209876794 learned_addr learned my addr 172.21.15.27:0/4209876794 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f495bfff640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afc40 0x7f49500b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f49637fe640 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcea6b0 0x5628ee0adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f495bfff640 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.595+0000 7f4962ffd640 1 -- 172.21.15.27:0/4209876794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 msgr2=0x5628ee0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f4962ffd640 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0acfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f4962ffd640 1 -- 172.21.15.27:0/4209876794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcea6b0 msgr2=0x5628ee0adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f4962ffd640 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcea6b0 0x5628ee0adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f4962ffd640 1 -- 172.21.15.27:0/4209876794 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628edf9ca40 con 0x5628edd0b390 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f495bfff640 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f49637fe640 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcea6b0 0x5628ee0adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f4962ffd640 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edd0b390 0x5628ee0aca60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f494c02a910 tx=0x7f494c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47b4572a02bd004a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f495bfff640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afc40 0x7f49500b2150 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x5628ee0ac780 tx=0x7f495400b790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f4959ffb640 1 -- 172.21.15.27:0/4209876794 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c03e030 con 0x5628edd0b390 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f4959ffb640 1 -- 172.21.15.27:0/4209876794 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f494c032c60 con 0x5628edd0b390 2024-04-25T13:54:33.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.596+0000 7f4959ffb640 1 -- 172.21.15.27:0/4209876794 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c035040 con 0x5628edd0b390 2024-04-25T13:54:33.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.597+0000 7f49707db740 1 -- 172.21.15.27:0/4209876794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edd0b390 msgr2=0x5628ee0aca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.597+0000 7f49707db740 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edd0b390 0x5628ee0aca60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f494c02a910 tx=0x7f494c000bc0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.597+0000 7f49707db740 1 -- 172.21.15.27:0/4209876794 shutdown_connections 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.597+0000 7f49707db740 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcea6b0 0x5628ee0adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.597+0000 7f49707db740 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.597+0000 7f49707db740 1 --2- 172.21.15.27:0/4209876794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edd0b390 0x5628ee0aca60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.598+0000 7f49707db740 1 -- 172.21.15.27:0/4209876794 >> 172.21.15.27:0/4209876794 conn(0x5628edcff4e0 msgr2=0x5628ee0a8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.598+0000 7f49707db740 1 -- 172.21.15.27:0/4209876794 shutdown_connections 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.598+0000 7f49707db740 1 -- 172.21.15.27:0/4209876794 wait complete. 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.598+0000 7f49707db740 1 -- start start 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.598+0000 7f49707db740 1 Processor -- start 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f49707db740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edce24d0 0x5628ee0c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f49707db740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcea6b0 0x5628ee0c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f49707db740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edd0b390 0x5628ee0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f49707db740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5628ee0ae720 con 0x5628edd0b390 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f49707db740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5628edf98d10 con 0x5628edce24d0 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f49707db740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5628ee0ae8a0 con 0x5628edcea6b0 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f4962ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edce24d0 0x5628ee0c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f4962ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edce24d0 0x5628ee0c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:59062/0 (socket says 172.21.15.27:59062) 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f4962ffd640 1 -- 172.21.15.27:0/2286957482 learned_addr learned my addr 172.21.15.27:0/2286957482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f495bfff640 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcea6b0 0x5628ee0c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f49637fe640 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edd0b390 0x5628ee0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f49707db740 10 client.0 _ll_register_callbacks cb 0x5628edfc5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f4962ffd640 1 -- 172.21.15.27:0/2286957482 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcea6b0 msgr2=0x5628ee0c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f4962ffd640 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcea6b0 0x5628ee0c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f4962ffd640 1 -- 172.21.15.27:0/2286957482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edd0b390 msgr2=0x5628ee0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f4962ffd640 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edd0b390 0x5628ee0ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.599+0000 7f4962ffd640 1 -- 172.21.15.27:0/2286957482 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628edded7d0 con 0x5628edce24d0 2024-04-25T13:54:33.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.600+0000 7f4962ffd640 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edce24d0 0x5628ee0c78e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f494c006010 tx=0x7f494c031850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41dadc7288139abf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.600+0000 7f4942ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:54:33.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.600+0000 7f4942ffd640 1 -- 172.21.15.27:0/2286957482 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c03e040 con 0x5628edce24d0 2024-04-25T13:54:33.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.600+0000 7f4942ffd640 1 -- 172.21.15.27:0/2286957482 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f494c0052c0 con 0x5628edce24d0 2024-04-25T13:54:33.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.600+0000 7f49707db740 1 -- 172.21.15.27:0/2286957482 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f494c032c60 con 0x5628edce24d0 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.601+0000 7f4942ffd640 1 -- 172.21.15.27:0/2286957482 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f494c053070 con 0x5628edce24d0 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.601+0000 7f49707db740 10 client.27803 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.601+0000 7f49707db740 1 -- 172.21.15.27:0/2286957482 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5628edd233f0 con 0x5628edce24d0 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.601+0000 7f4942ffd640 1 -- 172.21.15.27:0/2286957482 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f494c032c60 con 0x5628edce24d0 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.602+0000 7f4942ffd640 1 -- 172.21.15.27:0/2286957482 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f494c04c480 con 0x5628edce24d0 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.602+0000 7f49707db740 10 client.27803 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.602+0000 7f49707db740 10 client.27803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.602+0000 7f49707db740 20 client.27803 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.602+0000 7f49707db740 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628edcc9f60 0x5628ee0d4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.602+0000 7f49707db740 4 client.27803 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.602+0000 7f49707db740 1 -- 172.21.15.27:0/2286957482 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628ee01f990 con 0x5628edcc9f60 2024-04-25T13:54:33.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.602+0000 7f495bfff640 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628edcc9f60 0x5628ee0d4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.603+0000 7f495bfff640 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628edcc9f60 0x5628ee0d4d90 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaac98e7a82565ad server_cookie=b8919587599c5a2e in_seq=0 out_seq=0 2024-04-25T13:54:33.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.603+0000 7f4942ffd640 10 client.27803 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f4942ffd640 1 -- 172.21.15.27:0/2286957482 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5628ee01f990 con 0x5628edcc9f60 2024-04-25T13:54:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f4942ffd640 10 client.27803 handle_command_reply: tid=0 2024-04-25T13:54:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f49707db740 1 client.27803 shutdown 2024-04-25T13:54:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f49707db740 1 -- 172.21.15.27:0/2286957482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edce24d0 msgr2=0x5628ee0c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f49707db740 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edce24d0 0x5628ee0c78e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f494c006010 tx=0x7f494c031850 comp rx=0 tx=0).stop 2024-04-25T13:54:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f49707db740 1 -- 172.21.15.27:0/2286957482 shutdown_connections 2024-04-25T13:54:33.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f49707db740 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628edcc9f60 0x5628ee0d4d90 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f4942ffd640 0 client.27803 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f49707db740 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edd0b390 0x5628ee0ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f49707db740 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcea6b0 0x5628ee0c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.604+0000 7f49707db740 1 --2- 172.21.15.27:0/2286957482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edce24d0 0x5628ee0c78e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.605+0000 7f49707db740 1 -- 172.21.15.27:0/2286957482 >> 172.21.15.27:0/2286957482 conn(0x5628edcff4e0 msgr2=0x5628ee0ab180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.605+0000 7f49707db740 1 -- 172.21.15.27:0/2286957482 shutdown_connections 2024-04-25T13:54:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.605+0000 7f49707db740 1 -- 172.21.15.27:0/2286957482 wait complete. 2024-04-25T13:54:33.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.605+0000 7f49707db740 20 client.27803 trim_cache size 0 max 16384 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.630+0000 7f49707db740 1 Processor -- start 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f49707db740 1 -- start start 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f49707db740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcc9f60 0x5628ee0c6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f49707db740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f49707db740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee0cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f49707db740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4964072e40 con 0x5628edcc9f60 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f49707db740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f49641a97e0 con 0x5628edcea6b0 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f49707db740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f49640734b0 con 0x5628edce24d0 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f4962ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcc9f60 0x5628ee0c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f495bfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f4962ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcc9f60 0x5628ee0c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51324/0 (socket says 172.21.15.27:51324) 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f4962ffd640 1 -- 172.21.15.27:0/1362154462 learned_addr learned my addr 172.21.15.27:0/1362154462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f49637fe640 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee0cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f495bfff640 1 -- 172.21.15.27:0/1362154462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 msgr2=0x5628ee0cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f495bfff640 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee0cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f495bfff640 1 -- 172.21.15.27:0/1362154462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcc9f60 msgr2=0x5628ee0c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f495bfff640 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcc9f60 0x5628ee0c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f495bfff640 1 -- 172.21.15.27:0/1362154462 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628edce30f0 con 0x5628edce24d0 2024-04-25T13:54:33.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.631+0000 7f4962ffd640 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcc9f60 0x5628ee0c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.632+0000 7f495bfff640 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0ceca0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f495400bc20 tx=0x7f4954014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=426de0416d87f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.632+0000 7f49597fa640 1 -- 172.21.15.27:0/1362154462 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49540130d0 con 0x5628edce24d0 2024-04-25T13:54:33.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.632+0000 7f49597fa640 1 -- 172.21.15.27:0/1362154462 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4954020e30 con 0x5628edce24d0 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.632+0000 7f49597fa640 1 -- 172.21.15.27:0/1362154462 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954005480 con 0x5628edce24d0 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.633+0000 7f49707db740 1 -- 172.21.15.27:0/1362154462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 msgr2=0x5628ee0ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.633+0000 7f49707db740 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0ceca0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f495400bc20 tx=0x7f4954014000 comp rx=0 tx=0).stop 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.633+0000 7f49707db740 1 -- 172.21.15.27:0/1362154462 shutdown_connections 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.633+0000 7f49707db740 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee0cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.633+0000 7f49707db740 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edce24d0 0x5628ee0ceca0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.633+0000 7f49707db740 1 --2- 172.21.15.27:0/1362154462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edcc9f60 0x5628ee0c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.633+0000 7f49707db740 1 -- 172.21.15.27:0/1362154462 >> 172.21.15.27:0/1362154462 conn(0x5628edcff4e0 msgr2=0x5628ee0c5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.633+0000 7f49707db740 1 -- 172.21.15.27:0/1362154462 shutdown_connections 2024-04-25T13:54:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.634+0000 7f49707db740 1 -- 172.21.15.27:0/1362154462 wait complete. 2024-04-25T13:54:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.634+0000 7f49707db740 1 -- start start 2024-04-25T13:54:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.635+0000 7f49707db740 1 Processor -- start 2024-04-25T13:54:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.635+0000 7f49707db740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcc9f60 0x5628ee0cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.635+0000 7f49707db740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edce24d0 0x5628ee0cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f49707db740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee13b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f49707db740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4964073b80 con 0x5628edce24d0 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f49707db740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f49641a9110 con 0x5628edcea6b0 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f49707db740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f4964074250 con 0x5628edcc9f60 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f4962ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcc9f60 0x5628ee0cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f4962ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcc9f60 0x5628ee0cc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58362/0 (socket says 172.21.15.27:58362) 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f4962ffd640 1 -- 172.21.15.27:0/3399163664 learned_addr learned my addr 172.21.15.27:0/3399163664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f49707db740 10 client.0 _ll_register_callbacks cb 0x7f494c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f495bfff640 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edce24d0 0x5628ee0cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f49637fe640 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f4962ffd640 1 -- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 msgr2=0x5628ee13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f4962ffd640 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f4962ffd640 1 -- 172.21.15.27:0/3399163664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edce24d0 msgr2=0x5628ee0cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f4962ffd640 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edce24d0 0x5628ee0cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f4962ffd640 1 -- 172.21.15.27:0/3399163664 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628edfd24b0 con 0x5628edcc9f60 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.636+0000 7f49637fe640 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee13b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f4962ffd640 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcc9f60 0x5628ee0cc3a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f494c009900 tx=0x7f494c005990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d12bfe7bfa0f99a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f4942ffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f4942ffd640 1 -- 172.21.15.27:0/3399163664 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c05c020 con 0x5628edcc9f60 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f4942ffd640 1 -- 172.21.15.27:0/3399163664 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f494c04cac0 con 0x5628edcc9f60 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f49707db740 1 -- 172.21.15.27:0/3399163664 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4954020e30 con 0x5628edcc9f60 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f4942ffd640 1 -- 172.21.15.27:0/3399163664 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f494c03e040 con 0x5628edcc9f60 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f49707db740 10 client.27853 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f49707db740 1 -- 172.21.15.27:0/3399163664 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5628ee13c580 con 0x5628edcc9f60 2024-04-25T13:54:33.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f4942ffd640 1 -- 172.21.15.27:0/3399163664 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4954020e30 con 0x5628edcc9f60 2024-04-25T13:54:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.637+0000 7f4942ffd640 1 -- 172.21.15.27:0/3399163664 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f494c0048f0 con 0x5628edcc9f60 2024-04-25T13:54:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.638+0000 7f49707db740 10 client.27853 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.638+0000 7f49707db740 10 client.27853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.638+0000 7f49707db740 20 client.27853 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.638+0000 7f49707db740 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628edd0b390 0x5628ee13efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.638+0000 7f49707db740 4 client.27853 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:54:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.638+0000 7f49707db740 1 -- 172.21.15.27:0/3399163664 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5628ee019800 con 0x5628edd0b390 2024-04-25T13:54:33.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.638+0000 7f495bfff640 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628edd0b390 0x5628ee13efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:33.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.638+0000 7f495bfff640 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628edd0b390 0x5628ee13efa0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a7101fd58b09ebc server_cookie=461006c6f41b18fd in_seq=0 out_seq=0 2024-04-25T13:54:33.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.639+0000 7f4942ffd640 10 client.27853 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:33.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.639+0000 7f4942ffd640 1 -- 172.21.15.27:0/3399163664 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5628ee019800 con 0x5628edd0b390 2024-04-25T13:54:33.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.639+0000 7f4942ffd640 10 client.27853 handle_command_reply: tid=0 2024-04-25T13:54:33.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.639+0000 7f49707db740 1 client.27853 shutdown 2024-04-25T13:54:33.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 -- 172.21.15.27:0/3399163664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcc9f60 msgr2=0x5628ee0cc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcc9f60 0x5628ee0cc3a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f494c009900 tx=0x7f494c005990 comp rx=0 tx=0).stop 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 -- 172.21.15.27:0/3399163664 shutdown_connections 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5628edd0b390 0x5628ee13efa0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5628edcea6b0 0x5628ee13b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5628edce24d0 0x5628ee0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 --2- 172.21.15.27:0/3399163664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5628edcc9f60 0x5628ee0cc3a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 -- 172.21.15.27:0/3399163664 >> 172.21.15.27:0/3399163664 conn(0x5628edcff4e0 msgr2=0x5628ee0c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f4942ffd640 0 client.27853 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 -- 172.21.15.27:0/3399163664 shutdown_connections 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 1 -- 172.21.15.27:0/3399163664 wait complete. 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f49707db740 20 client.27853 trim_cache size 0 max 16384 2024-04-25T13:54:33.642 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (179 inodes in the stack)", 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout: "f66492b3-d9f5-488b-853b-e67b47274a80": { 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "f66492b3-d9f5-488b-853b-e67b47274a80", 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:54:33.643 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:54:33.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f496992e640 1 -- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afc40 msgr2=0x7f49500b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f496992e640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afc40 0x7f49500b2150 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x5628ee0ac780 tx=0x7f495400b790 comp rx=0 tx=0).stop 2024-04-25T13:54:33.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f496992e640 1 -- 172.21.15.27:0/4268460971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4964110030 msgr2=0x7f49641a7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:33.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.640+0000 7f496992e640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4964110030 0x7f49641a7d70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f495c00eb10 tx=0x7f495c00c2d0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.641+0000 7f496992e640 1 -- 172.21.15.27:0/4268460971 shutdown_connections 2024-04-25T13:54:33.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.641+0000 7f496992e640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f49500afc40 0x7f49500b2150 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.641+0000 7f496992e640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4964110030 0x7f49641a7d70 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.641+0000 7f496992e640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f496410b330 0x7f49641a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.641+0000 7f496992e640 1 --2- 172.21.15.27:0/4268460971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f496410a0f0 0x7f49641a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:33.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.641+0000 7f496992e640 1 -- 172.21.15.27:0/4268460971 >> 172.21.15.27:0/4268460971 conn(0x7f4964078de0 msgr2=0x7f496410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:33.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.641+0000 7f496992e640 1 -- 172.21.15.27:0/4268460971 shutdown_connections 2024-04-25T13:54:33.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:54:33.641+0000 7f496992e640 1 -- 172.21.15.27:0/4268460971 wait complete. 2024-04-25T13:54:33.656 INFO:tasks.cephfs.filesystem:scrub status for tag:f66492b3-d9f5-488b-853b-e67b47274a80 - {'path': '/', 'tag': 'f66492b3-d9f5-488b-853b-e67b47274a80', 'options': 'recursive,force'} 2024-04-25T13:54:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:34 smithi027 ceph-mon[29140]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T13:54:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:34 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:54:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:34 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:34 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:54:34.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:34 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:34.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:34 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:54:34.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:34 smithi122 ceph-mon[32843]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T13:54:34.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:34 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:54:34.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:34 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:34.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:34 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:54:34.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:34 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:34.808 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:34 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:54:34.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:34 smithi045 ceph-mon[32731]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T13:54:34.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:34 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:54:34.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:34 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:34.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:34 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:54:34.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:34 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:54:34.828 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:34 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:54:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 -- 172.21.15.27:0/3607563779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 msgr2=0x7f89e4104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 --2- 172.21.15.27:0/3607563779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 0x7f89e4104090 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009900 tx=0x7f89d002a410 comp rx=0 tx=0).stop 2024-04-25T13:54:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 -- 172.21.15.27:0/3607563779 shutdown_connections 2024-04-25T13:54:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 --2- 172.21.15.27:0/3607563779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e4108980 0x7f89e410ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 --2- 172.21.15.27:0/3607563779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 0x7f89e4104090 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 --2- 172.21.15.27:0/3607563779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89e4102a40 0x7f89e4103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 -- 172.21.15.27:0/3607563779 >> 172.21.15.27:0/3607563779 conn(0x7f89e40fdc30 msgr2=0x7f89e41000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 -- 172.21.15.27:0/3607563779 shutdown_connections 2024-04-25T13:54:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.996+0000 7f89eafa8640 1 -- 172.21.15.27:0/3607563779 wait complete. 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.997+0000 7f89eafa8640 1 Processor -- start 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.997+0000 7f89eafa8640 1 -- start start 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.997+0000 7f89eafa8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89e4102a40 0x7f89e41a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.997+0000 7f89eafa8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 0x7f89e41a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89eafa8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e4108980 0x7f89e41a7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89eafa8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f89e410e560 con 0x7f89e4103c80 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89eafa8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89e410e3e0 con 0x7f89e4108980 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89eafa8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f89e410e6e0 con 0x7f89e4102a40 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e9545640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e4108980 0x7f89e41a7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e9545640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e4108980 0x7f89e41a7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40080/0 (socket says 172.21.15.27:40080) 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e9545640 1 -- 172.21.15.27:0/2751333271 learned_addr learned my addr 172.21.15.27:0/2751333271 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e8d44640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89e4102a40 0x7f89e41a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e3fff640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 0x7f89e41a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e9545640 1 -- 172.21.15.27:0/2751333271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89e4102a40 msgr2=0x7f89e41a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e9545640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89e4102a40 0x7f89e41a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e9545640 1 -- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 msgr2=0x7f89e41a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e9545640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 0x7f89e41a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e9545640 1 -- 172.21.15.27:0/2751333271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e41a8360 con 0x7f89e4108980 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.998+0000 7f89e3fff640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 0x7f89e41a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.999+0000 7f89e9545640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e4108980 0x7f89e41a7cf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00eb10 tx=0x7f89dc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.999+0000 7f89e1ffb640 1 -- 172.21.15.27:0/2751333271 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc004b20 con 0x7f89e4108980 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.999+0000 7f89e1ffb640 1 -- 172.21.15.27:0/2751333271 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89dc004cc0 con 0x7f89e4108980 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.999+0000 7f89e1ffb640 1 -- 172.21.15.27:0/2751333271 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc010670 con 0x7f89e4108980 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:35.999+0000 7f89eafa8640 1 -- 172.21.15.27:0/2751333271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e41a8650 con 0x7f89e4108980 2024-04-25T13:54:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.000+0000 7f89eafa8640 1 -- 172.21.15.27:0/2751333271 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e41a8ba0 con 0x7f89e4108980 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.002+0000 7f89f1e51740 1 Processor -- start 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.002+0000 7f89e1ffb640 1 -- 172.21.15.27:0/2751333271 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f89dc005020 con 0x7f89e4108980 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89f1e51740 1 -- start start 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558002f0 0x563d55bc9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89f1e51740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563d55b2f460 con 0x563d558002f0 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89f1e51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563d55ae9590 con 0x563d558291b0 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89f1e51740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563d55b2b420 con 0x563d558084d0 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e8d44640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558002f0 0x563d55bc9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e8d44640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558002f0 0x563d55bc9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43358/0 (socket says 172.21.15.27:43358) 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e8d44640 1 -- 172.21.15.27:0/4053001733 learned_addr learned my addr 172.21.15.27:0/4053001733 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40082/0 (socket says 172.21.15.27:40082) 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e3fff640 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e1ffb640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89d40afb60 0x7f89d40b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e8d44640 1 -- 172.21.15.27:0/4053001733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 msgr2=0x563d55bd0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e8d44640 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e8d44640 1 -- 172.21.15.27:0/4053001733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 msgr2=0x563d55bc9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e8d44640 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e8d44640 1 -- 172.21.15.27:0/4053001733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d55af60d0 con 0x563d558002f0 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e3fff640 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.003+0000 7f89e9545640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89d40afb60 0x7f89d40b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.004+0000 7f89e1ffb640 1 -- 172.21.15.27:0/2751333271 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f89dc0df1e0 con 0x7f89e4108980 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.004+0000 7f89e8d44640 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558002f0 0x563d55bc9ef0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009c60 tx=0x7f89d8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37bd4b407a3a7e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.004+0000 7f89ceffd640 1 -- 172.21.15.27:0/4053001733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8004220 con 0x563d558002f0 2024-04-25T13:54:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.004+0000 7f89e9545640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89d40afb60 0x7f89d40b2070 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f89dc0a3770 tx=0x7f89dc09c030 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.004+0000 7f89ceffd640 1 -- 172.21.15.27:0/4053001733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89d80043c0 con 0x563d558002f0 2024-04-25T13:54:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.004+0000 7f89ceffd640 1 -- 172.21.15.27:0/4053001733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8004220 con 0x563d558002f0 2024-04-25T13:54:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 -- 172.21.15.27:0/4053001733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558002f0 msgr2=0x563d55bc9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558002f0 0x563d55bc9ef0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009c60 tx=0x7f89d8011810 comp rx=0 tx=0).stop 2024-04-25T13:54:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 -- 172.21.15.27:0/4053001733 shutdown_connections 2024-04-25T13:54:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558002f0 0x563d55bc9ef0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 --2- 172.21.15.27:0/4053001733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 -- 172.21.15.27:0/4053001733 >> 172.21.15.27:0/4053001733 conn(0x563d5581d300 msgr2=0x563d55bc5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 -- 172.21.15.27:0/4053001733 shutdown_connections 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 -- 172.21.15.27:0/4053001733 wait complete. 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.005+0000 7f89f1e51740 1 -- start start 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89f1e51740 1 Processor -- start 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558002f0 0x563d55bd3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bdab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89f1e51740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x563d55bd1890 con 0x563d558291b0 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89f1e51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x563d55ab6b60 con 0x563d558002f0 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89f1e51740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x563d55bd1a10 con 0x563d558084d0 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558002f0 0x563d55bd3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558002f0 0x563d55bd3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:40088/0 (socket says 172.21.15.27:40088) 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89e3fff640 1 -- 172.21.15.27:0/2214604935 learned_addr learned my addr 172.21.15.27:0/2214604935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89e3fff640 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bdab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89e8d44640 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.006+0000 7f89f1e51740 10 client.0 _ll_register_callbacks cb 0x563d55adb2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89e3fff640 1 -- 172.21.15.27:0/2214604935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 msgr2=0x563d55bd3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89e3fff640 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89e3fff640 1 -- 172.21.15.27:0/2214604935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 msgr2=0x563d55bdab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89e3fff640 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bdab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89e3fff640 1 -- 172.21.15.27:0/2214604935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d55841210 con 0x563d558002f0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89e3fff640 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558002f0 0x563d55bd3660 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f89d0005ab0 tx=0x7f89d002a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28af1f6d5fb5903 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89c3fff640 1 -- 172.21.15.27:0/2214604935 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d0041070 con 0x563d558002f0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.007+0000 7f89c3fff640 1 -- 172.21.15.27:0/2214604935 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f89d0004480 con 0x563d558002f0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.008+0000 7f89f1e51740 1 -- 172.21.15.27:0/2214604935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d80043c0 con 0x563d558002f0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.008+0000 7f89c3fff640 1 -- 172.21.15.27:0/2214604935 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f89d0051050 con 0x563d558002f0 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.008+0000 7f89f1e51740 10 client.27821 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.008+0000 7f89f1e51740 1 -- 172.21.15.27:0/2214604935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563d5590d620 con 0x563d558002f0 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.009+0000 7f89c3fff640 1 -- 172.21.15.27:0/2214604935 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89d0003aa0 con 0x563d558002f0 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.009+0000 7f89c3fff640 1 -- 172.21.15.27:0/2214604935 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f89d004a070 con 0x563d558002f0 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.009+0000 7f89f1e51740 10 client.27821 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.009+0000 7f89f1e51740 10 client.27821 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.009+0000 7f89f1e51740 20 client.27821 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.009+0000 7f89f1e51740 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d557e7d80 0x563d55be12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.010+0000 7f89f1e51740 4 client.27821 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.010+0000 7f89f1e51740 1 -- 172.21.15.27:0/2214604935 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f89d80043c0 con 0x563d557e7d80 2024-04-25T13:54:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.010+0000 7f89e8d44640 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d557e7d80 0x563d55be12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.010+0000 7f89e8d44640 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d557e7d80 0x563d55be12a0 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37b5f6c5b7e163ae server_cookie=96c45af98f525ed0 in_seq=0 out_seq=0 2024-04-25T13:54:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.010+0000 7f89c3fff640 10 client.27821 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.011+0000 7f89c3fff640 1 -- 172.21.15.27:0/2214604935 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f89d80043c0 con 0x563d557e7d80 2024-04-25T13:54:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.011+0000 7f89c3fff640 10 client.27821 handle_command_reply: tid=0 2024-04-25T13:54:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.011+0000 7f89f1e51740 1 client.27821 shutdown 2024-04-25T13:54:36.013 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.011+0000 7f89f1e51740 1 -- 172.21.15.27:0/2214604935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558002f0 msgr2=0x563d55bd3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.011+0000 7f89f1e51740 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558002f0 0x563d55bd3660 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f89d0005ab0 tx=0x7f89d002a410 comp rx=0 tx=0).stop 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.011+0000 7f89f1e51740 1 -- 172.21.15.27:0/2214604935 shutdown_connections 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.011+0000 7f89f1e51740 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d557e7d80 0x563d55be12a0 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.011+0000 7f89f1e51740 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bdab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.012+0000 7f89f1e51740 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.012+0000 7f89f1e51740 1 --2- 172.21.15.27:0/2214604935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558002f0 0x563d55bd3660 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.012+0000 7f89c3fff640 0 client.27821 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.012+0000 7f89f1e51740 1 -- 172.21.15.27:0/2214604935 >> 172.21.15.27:0/2214604935 conn(0x563d5581d300 msgr2=0x563d55bc8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.012+0000 7f89f1e51740 1 -- 172.21.15.27:0/2214604935 shutdown_connections 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.012+0000 7f89f1e51740 1 -- 172.21.15.27:0/2214604935 wait complete. 2024-04-25T13:54:36.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.012+0000 7f89f1e51740 20 client.27821 trim_cache size 0 max 16384 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.030+0000 7f89f1e51740 1 Processor -- start 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.030+0000 7f89f1e51740 1 -- start start 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.030+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558002f0 0x563d55bc4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.030+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558084d0 0x563d55bc5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558002f0 0x563d55bc4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558002f0 0x563d55bc4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:33798/0 (socket says 172.21.15.27:33798) 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 -- 172.21.15.27:0/239027357 learned_addr learned my addr 172.21.15.27:0/239027357 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89f1e51740 1 -- 172.21.15.27:0/239027357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41af780 con 0x563d558084d0 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89f1e51740 1 -- 172.21.15.27:0/239027357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41b2000 con 0x563d558291b0 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89f1e51740 1 -- 172.21.15.27:0/239027357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41afdf0 con 0x563d558002f0 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 --2- 172.21.15.27:0/239027357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 -- 172.21.15.27:0/239027357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 msgr2=0x563d55bc8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 --2- 172.21.15.27:0/239027357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 -- 172.21.15.27:0/239027357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558084d0 msgr2=0x563d55bc5200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 --2- 172.21.15.27:0/239027357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558084d0 0x563d55bc5200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.031+0000 7f89e3fff640 1 -- 172.21.15.27:0/239027357 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d55bd0e50 con 0x563d558002f0 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.032+0000 7f89e3fff640 1 --2- 172.21.15.27:0/239027357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558002f0 0x563d55bc4cc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f89d0000c00 tx=0x7f89d00056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ef1e6c07b01f902 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.032+0000 7f89ce7fc640 1 -- 172.21.15.27:0/239027357 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d007f070 con 0x563d558002f0 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.032+0000 7f89ce7fc640 1 -- 172.21.15.27:0/239027357 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89d004c950 con 0x563d558002f0 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.033+0000 7f89f1e51740 1 -- 172.21.15.27:0/239027357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558002f0 msgr2=0x563d55bc4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.033+0000 7f89f1e51740 1 --2- 172.21.15.27:0/239027357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558002f0 0x563d55bc4cc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f89d0000c00 tx=0x7f89d00056d0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.034+0000 7f89f1e51740 1 -- 172.21.15.27:0/239027357 shutdown_connections 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.034+0000 7f89f1e51740 1 --2- 172.21.15.27:0/239027357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d558291b0 0x563d55bc8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.034+0000 7f89f1e51740 1 --2- 172.21.15.27:0/239027357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558084d0 0x563d55bc5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.034+0000 7f89f1e51740 1 --2- 172.21.15.27:0/239027357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558002f0 0x563d55bc4cc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.034+0000 7f89f1e51740 1 -- 172.21.15.27:0/239027357 >> 172.21.15.27:0/239027357 conn(0x563d5581d300 msgr2=0x563d55bcda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.038+0000 7f89f1e51740 1 -- 172.21.15.27:0/239027357 shutdown_connections 2024-04-25T13:54:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.041+0000 7f89f1e51740 1 -- 172.21.15.27:0/239027357 wait complete. 2024-04-25T13:54:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.041+0000 7f89f1e51740 1 -- start start 2024-04-25T13:54:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.041+0000 7f89f1e51740 1 Processor -- start 2024-04-25T13:54:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.042+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.042+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bd78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.042+0000 7f89f1e51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d557bd1c0 0x563d55bd8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.042+0000 7f89f1e51740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41b04c0 con 0x563d558291b0 2024-04-25T13:54:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.042+0000 7f89f1e51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41b1930 con 0x563d557bd1c0 2024-04-25T13:54:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.042+0000 7f89f1e51740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f89e41b0b90 con 0x563d558084d0 2024-04-25T13:54:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.042+0000 7f89f1e51740 10 client.0 _ll_register_callbacks cb 0x7f89d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.046+0000 7f89e8d44640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bd78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.046+0000 7f89e8d44640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bd78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43370/0 (socket says 172.21.15.27:43370) 2024-04-25T13:54:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.046+0000 7f89e8d44640 1 -- 172.21.15.27:0/690170377 learned_addr learned my addr 172.21.15.27:0/690170377 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.046+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.050+0000 7f89e3fff640 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d557bd1c0 0x563d55bd8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89e3fff640 1 -- 172.21.15.27:0/690170377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d557bd1c0 msgr2=0x563d55bd8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89e3fff640 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d557bd1c0 0x563d55bd8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89e3fff640 1 -- 172.21.15.27:0/690170377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 msgr2=0x563d55bd78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89e3fff640 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bd78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89e3fff640 1 -- 172.21.15.27:0/690170377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d55bd8d40 con 0x563d558084d0 2024-04-25T13:54:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89e3fff640 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd7380 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x563d55bddf30 tx=0x7f89d0031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=181b88a80ee3d2f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:36.051 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89c3fff640 1 -- 172.21.15.27:0/690170377 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d007f040 con 0x563d558084d0 2024-04-25T13:54:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89f1e51740 1 -- 172.21.15.27:0/690170377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563d55b3d070 con 0x563d558084d0 2024-04-25T13:54:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.051+0000 7f89c3fff640 1 -- 172.21.15.27:0/690170377 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f89d0031a20 con 0x563d558084d0 2024-04-25T13:54:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.052+0000 7f89c3fff640 1 -- 172.21.15.27:0/690170377 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f89d0035040 con 0x563d558084d0 2024-04-25T13:54:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.054+0000 7f89f1e51740 10 client.27871 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.054+0000 7f89f1e51740 1 -- 172.21.15.27:0/690170377 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x563d55bda0d0 con 0x563d558084d0 2024-04-25T13:54:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.063+0000 7f89c3fff640 1 -- 172.21.15.27:0/690170377 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563d55b3d070 con 0x563d558084d0 2024-04-25T13:54:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.064+0000 7f89c3fff640 1 -- 172.21.15.27:0/690170377 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f89d0082070 con 0x563d558084d0 2024-04-25T13:54:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.064+0000 7f89f1e51740 10 client.27871 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.064+0000 7f89f1e51740 10 client.27871 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.064+0000 7f89f1e51740 20 client.27871 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.064+0000 7f89f1e51740 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d55c49990 0x563d55c4bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.064+0000 7f89f1e51740 4 client.27871 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:54:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.064+0000 7f89f1e51740 1 -- 172.21.15.27:0/690170377 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563d55b36ee0 con 0x563d55c49990 2024-04-25T13:54:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.070+0000 7f89e8d44640 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d55c49990 0x563d55c4bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.073+0000 7f89e8d44640 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d55c49990 0x563d55c4bdb0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=532b0b31912cecb7 server_cookie=9659b9005f9d2169 in_seq=0 out_seq=0 2024-04-25T13:54:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.073+0000 7f89c3fff640 10 client.27871 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89c3fff640 1 -- 172.21.15.27:0/690170377 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563d55b36ee0 con 0x563d55c49990 2024-04-25T13:54:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89c3fff640 10 client.27871 handle_command_reply: tid=0 2024-04-25T13:54:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 client.27871 shutdown 2024-04-25T13:54:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 -- 172.21.15.27:0/690170377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 msgr2=0x563d55bd7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd7380 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x563d55bddf30 tx=0x7f89d0031400 comp rx=0 tx=0).stop 2024-04-25T13:54:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 -- 172.21.15.27:0/690170377 shutdown_connections 2024-04-25T13:54:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x563d55c49990 0x563d55c4bdb0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563d557bd1c0 0x563d55bd8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563d558291b0 0x563d55bd78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 --2- 172.21.15.27:0/690170377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563d558084d0 0x563d55bd7380 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89f1e51740 1 -- 172.21.15.27:0/690170377 >> 172.21.15.27:0/690170377 conn(0x563d5581d300 msgr2=0x563d55bc8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.103+0000 7f89c3fff640 0 client.27871 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89f1e51740 1 -- 172.21.15.27:0/690170377 shutdown_connections 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89f1e51740 1 -- 172.21.15.27:0/690170377 wait complete. 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89f1e51740 20 client.27871 trim_cache size 0 max 16384 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:54:36.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:54:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 -- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89d40afb60 msgr2=0x7f89d40b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89d40afb60 0x7f89d40b2070 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f89dc0a3770 tx=0x7f89dc09c030 comp rx=0 tx=0).stop 2024-04-25T13:54:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 -- 172.21.15.27:0/2751333271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e4108980 msgr2=0x7f89e41a7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e4108980 0x7f89e41a7cf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00eb10 tx=0x7f89dc00c2d0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 -- 172.21.15.27:0/2751333271 shutdown_connections 2024-04-25T13:54:36.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f89d40afb60 0x7f89d40b2070 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e4108980 0x7f89e41a7cf0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e4103c80 0x7f89e41a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 --2- 172.21.15.27:0/2751333271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89e4102a40 0x7f89e41a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 -- 172.21.15.27:0/2751333271 >> 172.21.15.27:0/2751333271 conn(0x7f89e40fdc30 msgr2=0x7f89e406b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 -- 172.21.15.27:0/2751333271 shutdown_connections 2024-04-25T13:54:36.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:36.104+0000 7f89eafa8640 1 -- 172.21.15.27:0/2751333271 wait complete. 2024-04-25T13:54:36.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:36 smithi027 ceph-mon[29140]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.2 MiB/s wr, 557 op/s 2024-04-25T13:54:36.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:36 smithi122 ceph-mon[32843]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.2 MiB/s wr, 557 op/s 2024-04-25T13:54:36.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:36 smithi045 ceph-mon[32731]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.2 MiB/s wr, 557 op/s 2024-04-25T13:54:37.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:37 smithi027 ceph-mon[29140]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.0 MiB/s wr, 679 op/s 2024-04-25T13:54:37.807 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:37 smithi122 ceph-mon[32843]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.0 MiB/s wr, 679 op/s 2024-04-25T13:54:37.827 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:37 smithi045 ceph-mon[32731]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.0 MiB/s wr, 679 op/s 2024-04-25T13:54:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 810.0 s, 1870.6 tps, lat 53.684 ms stddev 59.016 2024-04-25T13:54:40.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:39 smithi122 ceph-mon[32843]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.8 MiB/s wr, 718 op/s 2024-04-25T13:54:40.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:39 smithi045 ceph-mon[32731]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.8 MiB/s wr, 718 op/s 2024-04-25T13:54:40.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:39 smithi027 ceph-mon[29140]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.8 MiB/s wr, 718 op/s 2024-04-25T13:54:42.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:41 smithi122 ceph-mon[32843]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 8.2 MiB/s wr, 742 op/s 2024-04-25T13:54:42.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:41 smithi045 ceph-mon[32731]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 8.2 MiB/s wr, 742 op/s 2024-04-25T13:54:42.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:41 smithi027 ceph-mon[29140]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 8.2 MiB/s wr, 742 op/s 2024-04-25T13:54:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 815.0 s, 1393.4 tps, lat 71.674 ms stddev 83.244 2024-04-25T13:54:44.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:43 smithi122 ceph-mon[32843]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.8 MiB/s wr, 729 op/s 2024-04-25T13:54:44.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:43 smithi045 ceph-mon[32731]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.8 MiB/s wr, 729 op/s 2024-04-25T13:54:44.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:43 smithi027 ceph-mon[29140]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.8 MiB/s wr, 729 op/s 2024-04-25T13:54:46.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:45 smithi122 ceph-mon[32843]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.8 MiB/s wr, 651 op/s 2024-04-25T13:54:46.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:45 smithi045 ceph-mon[32731]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.8 MiB/s wr, 651 op/s 2024-04-25T13:54:46.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:45 smithi027 ceph-mon[29140]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.8 MiB/s wr, 651 op/s 2024-04-25T13:54:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 820.0 s, 1151.0 tps, lat 87.812 ms stddev 110.509 2024-04-25T13:54:48.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:47 smithi122 ceph-mon[32843]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 11 MiB/s wr, 747 op/s 2024-04-25T13:54:48.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:47 smithi045 ceph-mon[32731]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 11 MiB/s wr, 747 op/s 2024-04-25T13:54:48.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:47 smithi027 ceph-mon[29140]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 11 MiB/s wr, 747 op/s 2024-04-25T13:54:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.239+0000 7ff470dd5640 1 -- 172.21.15.27:0/3071296217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c102a40 msgr2=0x7ff46c103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.239+0000 7ff470dd5640 1 --2- 172.21.15.27:0/3071296217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c102a40 0x7ff46c103740 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff45c009900 tx=0x7ff45c02a440 comp rx=0 tx=0).stop 2024-04-25T13:54:49.242 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.241+0000 7ff470dd5640 1 -- 172.21.15.27:0/3071296217 shutdown_connections 2024-04-25T13:54:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.241+0000 7ff470dd5640 1 --2- 172.21.15.27:0/3071296217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff46c108980 0x7ff46c10ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.241+0000 7ff470dd5640 1 --2- 172.21.15.27:0/3071296217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff46c103c80 0x7ff46c104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.241+0000 7ff470dd5640 1 --2- 172.21.15.27:0/3071296217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c102a40 0x7ff46c103740 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.241+0000 7ff470dd5640 1 -- 172.21.15.27:0/3071296217 >> 172.21.15.27:0/3071296217 conn(0x7ff46c0fdc30 msgr2=0x7ff46c1000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.242+0000 7ff470dd5640 1 -- 172.21.15.27:0/3071296217 shutdown_connections 2024-04-25T13:54:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.242+0000 7ff470dd5640 1 -- 172.21.15.27:0/3071296217 wait complete. 2024-04-25T13:54:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.243+0000 7ff470dd5640 1 Processor -- start 2024-04-25T13:54:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.243+0000 7ff470dd5640 1 -- start start 2024-04-25T13:54:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff470dd5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff46c102a40 0x7ff46c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff470dd5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c103c80 0x7ff46c19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff470dd5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff46c108980 0x7ff46c1a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff46a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff46c102a40 0x7ff46c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff46a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff46c102a40 0x7ff46c19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:45406/0 (socket says 172.21.15.27:45406) 2024-04-25T13:54:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff46a59c640 1 -- 172.21.15.27:0/1197928904 learned_addr learned my addr 172.21.15.27:0/1197928904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff46ad9d640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff46c108980 0x7ff46c1a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c103c80 0x7ff46c19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c10e690 con 0x7ff46c108980 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c10e510 con 0x7ff46c102a40 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.244+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c10e810 con 0x7ff46c103c80 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.245+0000 7ff46ad9d640 1 -- 172.21.15.27:0/1197928904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c103c80 msgr2=0x7ff46c19a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.245+0000 7ff46ad9d640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c103c80 0x7ff46c19a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.245+0000 7ff46ad9d640 1 -- 172.21.15.27:0/1197928904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff46c102a40 msgr2=0x7ff46c19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.245+0000 7ff46ad9d640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff46c102a40 0x7ff46c19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.245+0000 7ff46ad9d640 1 -- 172.21.15.27:0/1197928904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46c1a1e90 con 0x7ff46c108980 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.245+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c103c80 0x7ff46c19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:54:49.246 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.245+0000 7ff46ad9d640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff46c108980 0x7ff46c1a1820 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff46400be20 tx=0x7ff46400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:49.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.246+0000 7ff45b7fe640 1 -- 172.21.15.27:0/1197928904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff464019070 con 0x7ff46c108980 2024-04-25T13:54:49.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.246+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46c1a2180 con 0x7ff46c108980 2024-04-25T13:54:49.247 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.246+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff46c1a2750 con 0x7ff46c108980 2024-04-25T13:54:49.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.247+0000 7ff45b7fe640 1 -- 172.21.15.27:0/1197928904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff46401d860 con 0x7ff46c108980 2024-04-25T13:54:49.248 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.248+0000 7ff45b7fe640 1 -- 172.21.15.27:0/1197928904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff464023060 con 0x7ff46c108980 2024-04-25T13:54:49.249 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.250+0000 7ff477922740 1 Processor -- start 2024-04-25T13:54:49.250 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.251+0000 7ff45b7fe640 1 -- 172.21.15.27:0/1197928904 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff46401c070 con 0x7ff46c108980 2024-04-25T13:54:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.251+0000 7ff477922740 1 -- start start 2024-04-25T13:54:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.251+0000 7ff477922740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f801b0 0x557bea32b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.251+0000 7ff477922740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea32dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.251 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.251+0000 7ff477922740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea330510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.251+0000 7ff477922740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557bea286460 con 0x557be9f5f4d0 2024-04-25T13:54:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff477922740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557bea240590 con 0x557be9f572f0 2024-04-25T13:54:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff477922740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557bea282420 con 0x557be9f801b0 2024-04-25T13:54:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff469d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea330510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff46a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea32dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff469d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea330510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33564/0 (socket says 172.21.15.27:33564) 2024-04-25T13:54:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff469d9b640 1 -- 172.21.15.27:0/3734044581 learned_addr learned my addr 172.21.15.27:0/3734044581 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:49.252 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff469d9b640 1 --2- 172.21.15.27:0/3734044581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f801b0 0x557bea32b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff45b7fe640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4540afd50 0x7ff4540b2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff46a59c640 1 -- 172.21.15.27:0/3734044581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f801b0 msgr2=0x557bea32b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff46a59c640 1 --2- 172.21.15.27:0/3734044581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f801b0 0x557bea32b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff46a59c640 1 -- 172.21.15.27:0/3734044581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 msgr2=0x557bea330510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff46a59c640 1 --2- 172.21.15.27:0/3734044581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea330510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.252+0000 7ff46a59c640 1 -- 172.21.15.27:0/3734044581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bea24d0d0 con 0x557be9f572f0 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.253+0000 7ff46ad9d640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4540afd50 0x7ff4540b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.253+0000 7ff45b7fe640 1 -- 172.21.15.27:0/1197928904 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff4640de130 con 0x7ff46c108980 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.253+0000 7ff46a59c640 1 --2- 172.21.15.27:0/3734044581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea32dd80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff45c006010 tx=0x7ff45c004390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30c8ba2cd14c940 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.253+0000 7ff46ad9d640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4540afd50 0x7ff4540b2260 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff464009290 tx=0x7ff4640d4d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.253+0000 7ff458ff9640 1 -- 172.21.15.27:0/3734044581 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45c035030 con 0x557be9f572f0 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.254+0000 7ff458ff9640 1 -- 172.21.15.27:0/3734044581 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff45c009540 con 0x557be9f572f0 2024-04-25T13:54:49.254 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.254+0000 7ff458ff9640 1 -- 172.21.15.27:0/3734044581 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45c031d40 con 0x557be9f572f0 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 -- 172.21.15.27:0/3734044581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 msgr2=0x557bea32dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 --2- 172.21.15.27:0/3734044581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea32dd80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff45c006010 tx=0x7ff45c004390 comp rx=0 tx=0).stop 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 -- 172.21.15.27:0/3734044581 shutdown_connections 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 --2- 172.21.15.27:0/3734044581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea330510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 --2- 172.21.15.27:0/3734044581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea32dd80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 --2- 172.21.15.27:0/3734044581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f801b0 0x557bea32b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 -- 172.21.15.27:0/3734044581 >> 172.21.15.27:0/3734044581 conn(0x557be9f74300 msgr2=0x557bea31c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 -- 172.21.15.27:0/3734044581 shutdown_connections 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.255+0000 7ff477922740 1 -- 172.21.15.27:0/3734044581 wait complete. 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.256+0000 7ff477922740 1 -- start start 2024-04-25T13:54:49.256 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.256+0000 7ff477922740 1 Processor -- start 2024-04-25T13:54:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.256+0000 7ff477922740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f572f0 0x557bea332e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.256+0000 7ff477922740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f5f4d0 0x557bea333350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.256+0000 7ff477922740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f801b0 0x557bea33a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff469d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f572f0 0x557bea332e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.257 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff46a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f5f4d0 0x557bea333350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff46a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f5f4d0 0x557bea333350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:45218/0 (socket says 172.21.15.27:45218) 2024-04-25T13:54:49.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff46a59c640 1 -- 172.21.15.27:0/1347412185 learned_addr learned my addr 172.21.15.27:0/1347412185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:49.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff469d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f572f0 0x557bea332e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33566/0 (socket says 172.21.15.27:33566) 2024-04-25T13:54:49.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x557bea330fd0 con 0x557be9f572f0 2024-04-25T13:54:49.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x557bea20db60 con 0x557be9f801b0 2024-04-25T13:54:49.258 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x557bea331150 con 0x557be9f5f4d0 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff46a59c640 1 -- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f801b0 msgr2=0x557bea33a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.257+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f801b0 0x557bea33a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff46a59c640 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f801b0 0x557bea33a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff46a59c640 1 -- 172.21.15.27:0/1347412185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f572f0 msgr2=0x557bea332e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff46a59c640 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f572f0 0x557bea332e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff46a59c640 1 -- 172.21.15.27:0/1347412185 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557be9f98210 con 0x557be9f5f4d0 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f801b0 0x557bea33a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff46a59c640 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f5f4d0 0x557bea333350 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff45c031da0 tx=0x7ff45c031dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b32cdaefbf0b132 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff451ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff451ffb640 1 -- 172.21.15.27:0/1347412185 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45c035070 con 0x557be9f5f4d0 2024-04-25T13:54:49.260 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.258+0000 7ff451ffb640 1 -- 172.21.15.27:0/1347412185 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff45c03c850 con 0x557be9f5f4d0 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.259+0000 7ff451ffb640 10 client.0 inactive, discarding osd_map(78..78 src has 1..78) 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.259+0000 7ff477922740 10 client.0 _ll_register_callbacks cb 0x557bea2322b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff45c009540 con 0x557be9f5f4d0 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff451ffb640 1 -- 172.21.15.27:0/1347412185 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff45c049070 con 0x557be9f5f4d0 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 10 client.27883 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x557bea064620 con 0x557be9f5f4d0 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff451ffb640 1 -- 172.21.15.27:0/1347412185 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff45c03ca50 con 0x557be9f5f4d0 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 10 client.27883 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 10 client.27883 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 20 client.27883 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557be9f3ed80 0x557bea3402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 4 client.27883 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:54:49.261 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.260+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bea294070 con 0x557be9f3ed80 2024-04-25T13:54:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.261+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557be9f3ed80 0x557bea3402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.261+0000 7ff451ffb640 10 client.27883 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.261+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557be9f3ed80 0x557bea3402b0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44158d62b5959268 server_cookie=add38e70810d7d83 in_seq=0 out_seq=0 2024-04-25T13:54:49.263 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.262+0000 7ff451ffb640 1 -- 172.21.15.27:0/1347412185 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557bea294070 con 0x557be9f3ed80 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.262+0000 7ff451ffb640 10 client.27883 handle_command_reply: tid=0 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.262+0000 7ff477922740 1 client.27883 shutdown 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f5f4d0 msgr2=0x557bea333350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f5f4d0 0x557bea333350 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff45c031da0 tx=0x7ff45c031dd0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 shutdown_connections 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557be9f3ed80 0x557bea3402b0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff451ffb640 0 client.27883 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f801b0 0x557bea33a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f5f4d0 0x557bea333350 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 --2- 172.21.15.27:0/1347412185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f572f0 0x557bea332e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 >> 172.21.15.27:0/1347412185 conn(0x557be9f74300 msgr2=0x557bea31b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 shutdown_connections 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.263+0000 7ff477922740 1 -- 172.21.15.27:0/1347412185 wait complete. 2024-04-25T13:54:49.264 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.264+0000 7ff477922740 20 client.27883 trim_cache size 0 max 16384 2024-04-25T13:54:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.282+0000 7ff477922740 1 Processor -- start 2024-04-25T13:54:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff477922740 1 -- start start 2024-04-25T13:54:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff477922740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f3ed80 0x557bea31af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff477922740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f572f0 0x557bea33a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff477922740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f5f4d0 0x557bea345a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff477922740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c00c300 con 0x557be9f3ed80 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff477922740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c00eb80 con 0x557be9f5f4d0 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff477922740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c00c970 con 0x557be9f572f0 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff469d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f3ed80 0x557bea31af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff469d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f3ed80 0x557bea31af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33568/0 (socket says 172.21.15.27:33568) 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff469d9b640 1 -- 172.21.15.27:0/1428566417 learned_addr learned my addr 172.21.15.27:0/1428566417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff46a59c640 1 --2- 172.21.15.27:0/1428566417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f572f0 0x557bea33a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff469d9b640 1 -- 172.21.15.27:0/1428566417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f572f0 msgr2=0x557bea33a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1428566417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f572f0 0x557bea33a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff469d9b640 1 -- 172.21.15.27:0/1428566417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f5f4d0 msgr2=0x557bea345a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1428566417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f5f4d0 0x557bea345a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.283+0000 7ff469d9b640 1 -- 172.21.15.27:0/1428566417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bea32e0f0 con 0x557be9f3ed80 2024-04-25T13:54:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.284+0000 7ff469d9b640 1 --2- 172.21.15.27:0/1428566417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f3ed80 0x557bea31af30 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff460016680 tx=0x7ff46000a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b79a114ee970f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.284+0000 7ff453fff640 1 -- 172.21.15.27:0/1428566417 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff460016d70 con 0x557be9f3ed80 2024-04-25T13:54:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.284+0000 7ff453fff640 1 -- 172.21.15.27:0/1428566417 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff460018d50 con 0x557be9f3ed80 2024-04-25T13:54:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.284+0000 7ff453fff640 1 -- 172.21.15.27:0/1428566417 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46001f8d0 con 0x557be9f3ed80 2024-04-25T13:54:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 -- 172.21.15.27:0/1428566417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f3ed80 msgr2=0x557bea31af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 --2- 172.21.15.27:0/1428566417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f3ed80 0x557bea31af30 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff460016680 tx=0x7ff46000a360 comp rx=0 tx=0).stop 2024-04-25T13:54:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 -- 172.21.15.27:0/1428566417 shutdown_connections 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 --2- 172.21.15.27:0/1428566417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f5f4d0 0x557bea345a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 --2- 172.21.15.27:0/1428566417 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f572f0 0x557bea33a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 --2- 172.21.15.27:0/1428566417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f3ed80 0x557bea31af30 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 -- 172.21.15.27:0/1428566417 >> 172.21.15.27:0/1428566417 conn(0x557be9f74300 msgr2=0x557bea31a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 -- 172.21.15.27:0/1428566417 shutdown_connections 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 -- 172.21.15.27:0/1428566417 wait complete. 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 -- start start 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.285+0000 7ff477922740 1 Processor -- start 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff477922740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f3ed80 0x557bea337a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff477922740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea337f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff477922740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea338eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff477922740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c00d040 con 0x557be9f5f4d0 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff477922740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c00e4b0 con 0x557be9f572f0 2024-04-25T13:54:49.289 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff477922740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c00d710 con 0x557be9f3ed80 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff477922740 10 client.0 _ll_register_callbacks cb 0x7ff46000a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff469d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f3ed80 0x557bea337a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff469d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f3ed80 0x557bea337a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:45240/0 (socket says 172.21.15.27:45240) 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff46a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea337f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff469d9b640 1 -- 172.21.15.27:0/511607866 learned_addr learned my addr 172.21.15.27:0/511607866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.286+0000 7ff469d9b640 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea338eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff469d9b640 1 -- 172.21.15.27:0/511607866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 msgr2=0x557bea337f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff469d9b640 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea337f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff469d9b640 1 -- 172.21.15.27:0/511607866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 msgr2=0x557bea338eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff469d9b640 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea338eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff469d9b640 1 -- 172.21.15.27:0/511607866 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bea339470 con 0x557be9f3ed80 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff469d9b640 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f3ed80 0x557bea337a40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff460000c00 tx=0x7ff4600069e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9b0e9d3f4ce8311 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff451ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff451ffb640 1 -- 172.21.15.27:0/511607866 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4600188a0 con 0x557be9f3ed80 2024-04-25T13:54:49.290 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff451ffb640 1 -- 172.21.15.27:0/511607866 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff460025040 con 0x557be9f3ed80 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff477922740 1 -- 172.21.15.27:0/511607866 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff460018d50 con 0x557be9f3ed80 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff451ffb640 1 -- 172.21.15.27:0/511607866 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7ff460018a40 con 0x557be9f3ed80 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff477922740 10 client.27889 fetch_fsmap learned FSMap version 19 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.287+0000 7ff477922740 1 -- 172.21.15.27:0/511607866 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x557bea3a8620 con 0x557be9f3ed80 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff451ffb640 1 -- 172.21.15.27:0/511607866 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff460018d50 con 0x557be9f3ed80 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff451ffb640 1 -- 172.21.15.27:0/511607866 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7ff46001fce0 con 0x557be9f3ed80 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff477922740 10 client.27889 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff477922740 10 client.27889 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff477922740 20 client.27889 populate_metadata read hostname 'smithi027' 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff477922740 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557be9f801b0 0x557bea3aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff477922740 4 client.27889 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff477922740 1 -- 172.21.15.27:0/511607866 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557bea28dee0 con 0x557be9f801b0 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff46a59c640 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557be9f801b0 0x557bea3aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff451ffb640 10 client.27889 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.288+0000 7ff46a59c640 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557be9f801b0 0x557bea3aaff0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dfbb376b7988847 server_cookie=4a9a0c9264f9b89 in_seq=0 out_seq=0 2024-04-25T13:54:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.305+0000 7ff451ffb640 1 -- 172.21.15.27:0/511607866 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557bea28dee0 con 0x557be9f801b0 2024-04-25T13:54:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.305+0000 7ff451ffb640 10 client.27889 handle_command_reply: tid=0 2024-04-25T13:54:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.305+0000 7ff477922740 1 client.27889 shutdown 2024-04-25T13:54:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.305+0000 7ff477922740 1 -- 172.21.15.27:0/511607866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f3ed80 msgr2=0x557bea337a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.305+0000 7ff477922740 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f3ed80 0x557bea337a40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff460000c00 tx=0x7ff4600069e0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 1 -- 172.21.15.27:0/511607866 shutdown_connections 2024-04-25T13:54:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x557be9f801b0 0x557bea3aaff0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557be9f5f4d0 0x557bea338eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557be9f572f0 0x557bea337f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 1 --2- 172.21.15.27:0/511607866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557be9f3ed80 0x557bea337a40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff451ffb640 0 client.27889 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 1 -- 172.21.15.27:0/511607866 >> 172.21.15.27:0/511607866 conn(0x557be9f74300 msgr2=0x557bea31b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 1 -- 172.21.15.27:0/511607866 shutdown_connections 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 1 -- 172.21.15.27:0/511607866 wait complete. 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff477922740 20 client.27889 trim_cache size 0 max 16384 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:54:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:54:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4540afd50 msgr2=0x7ff4540b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4540afd50 0x7ff4540b2260 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff464009290 tx=0x7ff4640d4d30 comp rx=0 tx=0).stop 2024-04-25T13:54:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff46c108980 msgr2=0x7ff46c1a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:54:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff46c108980 0x7ff46c1a1820 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff46400be20 tx=0x7ff46400bf10 comp rx=0 tx=0).stop 2024-04-25T13:54:49.308 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 shutdown_connections 2024-04-25T13:54:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff4540afd50 0x7ff4540b2260 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff46c108980 0x7ff46c1a1820 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff46c103c80 0x7ff46c19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 --2- 172.21.15.27:0/1197928904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff46c102a40 0x7ff46c19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:54:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.306+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 >> 172.21.15.27:0/1197928904 conn(0x7ff46c0fdc30 msgr2=0x7ff46c0ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:54:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.307+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 shutdown_connections 2024-04-25T13:54:49.309 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:54:49.307+0000 7ff470dd5640 1 -- 172.21.15.27:0/1197928904 wait complete. 2024-04-25T13:54:50.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:49 smithi122 ceph-mon[32843]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 745 op/s 2024-04-25T13:54:50.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:49 smithi045 ceph-mon[32731]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 745 op/s 2024-04-25T13:54:50.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:49 smithi027 ceph-mon[29140]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 745 op/s 2024-04-25T13:54:52.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:51 smithi122 ceph-mon[32843]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-25T13:54:52.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:51 smithi045 ceph-mon[32731]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-25T13:54:52.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:51 smithi027 ceph-mon[29140]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 656 op/s 2024-04-25T13:54:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 825.0 s, 1452.4 tps, lat 68.220 ms stddev 74.211 2024-04-25T13:54:54.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:53 smithi122 ceph-mon[32843]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 13 MiB/s wr, 692 op/s 2024-04-25T13:54:54.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:53 smithi045 ceph-mon[32731]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 13 MiB/s wr, 692 op/s 2024-04-25T13:54:54.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:53 smithi027 ceph-mon[29140]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 13 MiB/s wr, 692 op/s 2024-04-25T13:54:56.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:55 smithi122 ceph-mon[32843]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 13 MiB/s wr, 620 op/s 2024-04-25T13:54:56.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:55 smithi045 ceph-mon[32731]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 13 MiB/s wr, 620 op/s 2024-04-25T13:54:56.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:55 smithi027 ceph-mon[29140]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 13 MiB/s wr, 620 op/s 2024-04-25T13:54:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 830.0 s, 1538.4 tps, lat 65.674 ms stddev 72.201 2024-04-25T13:54:58.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:57 smithi122 ceph-mon[32843]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-25T13:54:58.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:57 smithi045 ceph-mon[32731]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-25T13:54:58.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:57 smithi027 ceph-mon[29140]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-25T13:55:00.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:54:59 smithi122 ceph-mon[32843]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 14 MiB/s wr, 677 op/s 2024-04-25T13:55:00.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:54:59 smithi045 ceph-mon[32731]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 14 MiB/s wr, 677 op/s 2024-04-25T13:55:00.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:54:59 smithi027 ceph-mon[29140]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 14 MiB/s wr, 677 op/s 2024-04-25T13:55:02.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:01 smithi045 ceph-mon[32731]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 601 op/s 2024-04-25T13:55:02.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:01 smithi027 ceph-mon[29140]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 601 op/s 2024-04-25T13:55:02.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:01 smithi122 ceph-mon[32843]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 601 op/s 2024-04-25T13:55:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.430+0000 7f633c915640 1 -- 172.21.15.27:0/2797521287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6338109a80 msgr2=0x7f633810bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.430+0000 7f633c915640 1 --2- 172.21.15.27:0/2797521287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6338109a80 0x7f633810bea0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6330009ca0 tx=0x7f633002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.431+0000 7f633c915640 1 -- 172.21.15.27:0/2797521287 shutdown_connections 2024-04-25T13:55:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.431+0000 7f633c915640 1 --2- 172.21.15.27:0/2797521287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6338109a80 0x7f633810bea0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.431+0000 7f633c915640 1 --2- 172.21.15.27:0/2797521287 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6338104d80 0x7f6338105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.431+0000 7f633c915640 1 --2- 172.21.15.27:0/2797521287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6338103b40 0x7f6338104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.431+0000 7f633c915640 1 -- 172.21.15.27:0/2797521287 >> 172.21.15.27:0/2797521287 conn(0x7f63380fed30 msgr2=0x7f63381011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.431+0000 7f633c915640 1 -- 172.21.15.27:0/2797521287 shutdown_connections 2024-04-25T13:55:02.431 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.431+0000 7f633c915640 1 -- 172.21.15.27:0/2797521287 wait complete. 2024-04-25T13:55:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633c915640 1 Processor -- start 2024-04-25T13:55:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633c915640 1 -- start start 2024-04-25T13:55:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633c915640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6338103b40 0x7f63381a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633c915640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6338104d80 0x7f63381a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633c915640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6338109a80 0x7f63381a7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633c915640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f633810d460 con 0x7f6338109a80 2024-04-25T13:55:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633c915640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f633810d2e0 con 0x7f6338103b40 2024-04-25T13:55:02.433 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633c915640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f633810d5e0 con 0x7f6338104d80 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6338103b40 0x7f63381a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f6336d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6338109a80 0x7f63381a7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6338103b40 0x7f63381a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34108/0 (socket says 172.21.15.27:34108) 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f633659c640 1 -- 172.21.15.27:0/3935531168 learned_addr learned my addr 172.21.15.27:0/3935531168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.432+0000 7f6335d9b640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6338104d80 0x7f63381a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f633659c640 1 -- 172.21.15.27:0/3935531168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6338104d80 msgr2=0x7f63381a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f633659c640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6338104d80 0x7f63381a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f633659c640 1 -- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6338109a80 msgr2=0x7f63381a7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f633659c640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6338109a80 0x7f63381a7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f633659c640 1 -- 172.21.15.27:0/3935531168 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63381a8320 con 0x7f6338103b40 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f633659c640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6338103b40 0x7f63381a07e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f632800e9f0 tx=0x7f632800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f63277fe640 1 -- 172.21.15.27:0/3935531168 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632800cc40 con 0x7f6338103b40 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f633c915640 1 -- 172.21.15.27:0/3935531168 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63381a8610 con 0x7f6338103b40 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f63277fe640 1 -- 172.21.15.27:0/3935531168 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f632800cde0 con 0x7f6338103b40 2024-04-25T13:55:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f633c915640 1 -- 172.21.15.27:0/3935531168 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63381a8bb0 con 0x7f6338103b40 2024-04-25T13:55:02.435 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.433+0000 7f63277fe640 1 -- 172.21.15.27:0/3935531168 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328010430 con 0x7f6338103b40 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.436+0000 7f63437c5740 1 Processor -- start 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f63437c5740 1 -- start start 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f63277fe640 1 -- 172.21.15.27:0/3935531168 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f63280105d0 con 0x7f6338103b40 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f63437c5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 0x5556bf366970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f63437c5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556bef9d2f0 0x5556bf366ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f63437c5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf36ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f63437c5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5556bf2cc460 con 0x5556befa54d0 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f63437c5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5556bf286590 con 0x5556bef9d2f0 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f63437c5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5556bf2c8420 con 0x5556befc61b0 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf36ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.437 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f633659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556bef9d2f0 0x5556bf366ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f633659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556bef9d2f0 0x5556bf366ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34110/0 (socket says 172.21.15.27:34110) 2024-04-25T13:55:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f633659c640 1 -- 172.21.15.27:0/1341471994 learned_addr learned my addr 172.21.15.27:0/1341471994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.437+0000 7f6336d9d640 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 0x5556bf366970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.439 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f633659c640 1 -- 172.21.15.27:0/1341471994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 msgr2=0x5556bf366970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f633659c640 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 0x5556bf366970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f63277fe640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63200afb30 0x7f63200b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f633659c640 1 -- 172.21.15.27:0/1341471994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 msgr2=0x5556bf36ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f633659c640 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf36ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f633659c640 1 -- 172.21.15.27:0/1341471994 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bf2930d0 con 0x5556bef9d2f0 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f6335d9b640 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf36ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f6336d9d640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63200afb30 0x7f63200b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f63277fe640 1 -- 172.21.15.27:0/3935531168 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f63280dee80 con 0x7f6338103b40 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f633659c640 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556bef9d2f0 0x5556bf366ef0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f63280a3410 tx=0x7f632800cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=998685d3b760944e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f6336d9d640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63200afb30 0x7f63200b2040 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x5556bf367b40 tx=0x7f6330009460 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.438+0000 7f6324ff9640 1 -- 172.21.15.27:0/1341471994 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280a3250 con 0x5556bef9d2f0 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.439+0000 7f6324ff9640 1 -- 172.21.15.27:0/1341471994 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63280a6ea0 con 0x5556bef9d2f0 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.439+0000 7f6324ff9640 1 -- 172.21.15.27:0/1341471994 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280a6b00 con 0x5556bef9d2f0 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.439+0000 7f63437c5740 1 -- 172.21.15.27:0/1341471994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556bef9d2f0 msgr2=0x5556bf366ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.439+0000 7f63437c5740 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556bef9d2f0 0x5556bf366ef0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f63280a3410 tx=0x7f632800cf90 comp rx=0 tx=0).stop 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 -- 172.21.15.27:0/1341471994 shutdown_connections 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf36ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556bef9d2f0 0x5556bf366ef0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 --2- 172.21.15.27:0/1341471994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 0x5556bf366970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 -- 172.21.15.27:0/1341471994 >> 172.21.15.27:0/1341471994 conn(0x5556befba300 msgr2=0x5556bf362300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 -- 172.21.15.27:0/1341471994 shutdown_connections 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 -- 172.21.15.27:0/1341471994 wait complete. 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 -- start start 2024-04-25T13:55:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.440+0000 7f63437c5740 1 Processor -- start 2024-04-25T13:55:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f63437c5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf3706b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f63437c5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befa54d0 0x5556bf370bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f63437c5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 0x5556bf377b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f63437c5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5556bf36e7e0 con 0x5556bef9d2f0 2024-04-25T13:55:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f63437c5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5556bf253b60 con 0x5556befc61b0 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f63437c5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5556bf36e960 con 0x5556befa54d0 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f63437c5740 10 client.0 _ll_register_callbacks cb 0x5556bf2782b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befa54d0 0x5556bf370bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befa54d0 0x5556bf370bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:46924/0 (socket says 172.21.15.27:46924) 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f6335d9b640 1 -- 172.21.15.27:0/4255242440 learned_addr learned my addr 172.21.15.27:0/4255242440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.441+0000 7f633659c640 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf3706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.442+0000 7f633659c640 1 -- 172.21.15.27:0/4255242440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befa54d0 msgr2=0x5556bf370bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.442+0000 7f633659c640 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befa54d0 0x5556bf370bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.442+0000 7f633659c640 1 -- 172.21.15.27:0/4255242440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 msgr2=0x5556bf377b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.442+0000 7f633659c640 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 0x5556bf377b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.442+0000 7f633659c640 1 -- 172.21.15.27:0/4255242440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556befde210 con 0x5556bef9d2f0 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.442+0000 7f633659c640 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf3706b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x5556bf367d50 tx=0x7f63280e2b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bcb957db66f9fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.442+0000 7f631dffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:55:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.443+0000 7f631dffb640 1 -- 172.21.15.27:0/4255242440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280a6ce0 con 0x5556bef9d2f0 2024-04-25T13:55:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.443+0000 7f631dffb640 1 -- 172.21.15.27:0/4255242440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f63280a6800 con 0x5556bef9d2f0 2024-04-25T13:55:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.443+0000 7f63437c5740 1 -- 172.21.15.27:0/4255242440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63280a6ea0 con 0x5556bef9d2f0 2024-04-25T13:55:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.443+0000 7f631dffb640 1 -- 172.21.15.27:0/4255242440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f63280a7160 con 0x5556bef9d2f0 2024-04-25T13:55:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f63437c5740 10 client.18177 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f63437c5740 1 -- 172.21.15.27:0/4255242440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5556bf0aa620 con 0x5556bef9d2f0 2024-04-25T13:55:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f631dffb640 1 -- 172.21.15.27:0/4255242440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63280a6ea0 con 0x5556bef9d2f0 2024-04-25T13:55:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f631dffb640 1 -- 172.21.15.27:0/4255242440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f63280e2140 con 0x5556bef9d2f0 2024-04-25T13:55:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f63437c5740 10 client.18177 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f63437c5740 10 client.18177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f63437c5740 20 client.18177 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f63437c5740 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5556bef84d80 0x5556bf37c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f63437c5740 4 client.18177 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.444+0000 7f63437c5740 1 -- 172.21.15.27:0/4255242440 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556bf2da070 con 0x5556bef84d80 2024-04-25T13:55:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.445+0000 7f6335d9b640 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5556bef84d80 0x5556bf37c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.445+0000 7f631dffb640 10 client.18177 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:02.446 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.445+0000 7f6335d9b640 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5556bef84d80 0x5556bf37c2b0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=739d53f0ed705580 server_cookie=6f611630e06d37a8 in_seq=0 out_seq=0 2024-04-25T13:55:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f631dffb640 1 -- 172.21.15.27:0/4255242440 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5556bf2da070 con 0x5556bef84d80 2024-04-25T13:55:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f631dffb640 10 client.18177 handle_command_reply: tid=0 2024-04-25T13:55:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 client.18177 shutdown 2024-04-25T13:55:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 -- 172.21.15.27:0/4255242440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 msgr2=0x5556bf3706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf3706b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x5556bf367d50 tx=0x7f63280e2b00 comp rx=0 tx=0).stop 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 -- 172.21.15.27:0/4255242440 shutdown_connections 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5556bef84d80 0x5556bf37c2b0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 0x5556bf377b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befa54d0 0x5556bf370bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 --2- 172.21.15.27:0/4255242440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf3706b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 -- 172.21.15.27:0/4255242440 >> 172.21.15.27:0/4255242440 conn(0x5556befba300 msgr2=0x5556bf365210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f631dffb640 0 client.18177 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 -- 172.21.15.27:0/4255242440 shutdown_connections 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.449+0000 7f63437c5740 1 -- 172.21.15.27:0/4255242440 wait complete. 2024-04-25T13:55:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.450+0000 7f63437c5740 20 client.18177 trim_cache size 0 max 16384 2024-04-25T13:55:02.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.469+0000 7f63437c5740 1 Processor -- start 2024-04-25T13:55:02.470 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.469+0000 7f63437c5740 1 -- start start 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.469+0000 7f63437c5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf36ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.469+0000 7f63437c5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befa54d0 0x5556bf377ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f63437c5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 0x5556bf36b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f63437c5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6338117af0 con 0x5556bef9d2f0 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f63437c5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63381a9a00 con 0x5556befa54d0 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f63437c5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6338118160 con 0x5556befc61b0 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf36ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf36ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52684/0 (socket says 172.21.15.27:52684) 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 -- 172.21.15.27:0/1299103257 learned_addr learned my addr 172.21.15.27:0/1299103257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f6335d9b640 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befa54d0 0x5556bf377ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f6336d9d640 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 0x5556bf36b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 -- 172.21.15.27:0/1299103257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 msgr2=0x5556bf36b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 0x5556bf36b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.471 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 -- 172.21.15.27:0/1299103257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befa54d0 msgr2=0x5556bf377ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:02.472 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befa54d0 0x5556bf377ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.472 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 -- 172.21.15.27:0/1299103257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bf36df50 con 0x5556bef9d2f0 2024-04-25T13:55:02.472 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f633659c640 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf36ab90 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f63280dcd20 tx=0x7f632800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3429523e39087a5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:02.472 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f631ffff640 1 -- 172.21.15.27:0/1299103257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280a6ce0 con 0x5556bef9d2f0 2024-04-25T13:55:02.472 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f631ffff640 1 -- 172.21.15.27:0/1299103257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63280a6ea0 con 0x5556bef9d2f0 2024-04-25T13:55:02.472 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.470+0000 7f631ffff640 1 -- 172.21.15.27:0/1299103257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280a6b00 con 0x5556bef9d2f0 2024-04-25T13:55:02.475 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.471+0000 7f63437c5740 1 -- 172.21.15.27:0/1299103257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 msgr2=0x5556bf36ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.471+0000 7f63437c5740 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf36ab90 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f63280dcd20 tx=0x7f632800cf90 comp rx=0 tx=0).stop 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.471+0000 7f63437c5740 1 -- 172.21.15.27:0/1299103257 shutdown_connections 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.471+0000 7f63437c5740 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556befc61b0 0x5556bf36b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.471+0000 7f63437c5740 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befa54d0 0x5556bf377ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.471+0000 7f63437c5740 1 --2- 172.21.15.27:0/1299103257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556bef9d2f0 0x5556bf36ab90 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.471+0000 7f63437c5740 1 -- 172.21.15.27:0/1299103257 >> 172.21.15.27:0/1299103257 conn(0x5556befba300 msgr2=0x5556bf370650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.472+0000 7f63437c5740 1 -- 172.21.15.27:0/1299103257 shutdown_connections 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.472+0000 7f63437c5740 1 -- 172.21.15.27:0/1299103257 wait complete. 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 1 -- start start 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 1 Processor -- start 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556bef9d2f0 0x5556bf3749d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf374f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.476 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 0x5556bf3e4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6338118830 con 0x5556befa54d0 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63381a9330 con 0x5556befc61b0 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6338118f00 con 0x5556bef9d2f0 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f63437c5740 10 client.0 _ll_register_callbacks cb 0x7f632c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf374f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf374f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52686/0 (socket says 172.21.15.27:52686) 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f6335d9b640 1 -- 172.21.15.27:0/1628745046 learned_addr learned my addr 172.21.15.27:0/1628745046 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f633659c640 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556bef9d2f0 0x5556bf3749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f6336d9d640 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 0x5556bf3e4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f633659c640 1 -- 172.21.15.27:0/1628745046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 msgr2=0x5556bf3e4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f633659c640 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 0x5556bf3e4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.477 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f633659c640 1 -- 172.21.15.27:0/1628745046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 msgr2=0x5556bf374f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.473+0000 7f633659c640 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf374f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f633659c640 1 -- 172.21.15.27:0/1628745046 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bf3e4560 con 0x5556bef9d2f0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f6335d9b640 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf374f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f633659c640 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556bef9d2f0 0x5556bf3749d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x5556bf367c80 tx=0x7f63280a3980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a6fc30ed55920 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f631dffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f631dffb640 1 -- 172.21.15.27:0/1628745046 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280a3250 con 0x5556bef9d2f0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f63437c5740 1 -- 172.21.15.27:0/1628745046 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63280a6ea0 con 0x5556bef9d2f0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f631dffb640 1 -- 172.21.15.27:0/1628745046 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f63280e2730 con 0x5556bef9d2f0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f631dffb640 1 -- 172.21.15.27:0/1628745046 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f63280a6b00 con 0x5556bef9d2f0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f63437c5740 10 client.27907 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.474+0000 7f63437c5740 1 -- 172.21.15.27:0/1628745046 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5556bf3e4560 con 0x5556bef9d2f0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.475+0000 7f631dffb640 1 -- 172.21.15.27:0/1628745046 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63280a6ea0 con 0x5556bef9d2f0 2024-04-25T13:55:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.475+0000 7f631dffb640 1 -- 172.21.15.27:0/1628745046 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f63280c8220 con 0x5556bef9d2f0 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.475+0000 7f63437c5740 10 client.27907 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.475+0000 7f63437c5740 10 client.27907 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.475+0000 7f63437c5740 20 client.27907 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.475+0000 7f63437c5740 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5556bef5a1c0 0x5556bf3e7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.475+0000 7f63437c5740 4 client.27907 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.475+0000 7f63437c5740 1 -- 172.21.15.27:0/1628745046 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5556bf2d3ee0 con 0x5556bef5a1c0 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.476+0000 7f6335d9b640 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5556bef5a1c0 0x5556bf3e7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.476+0000 7f6335d9b640 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5556bef5a1c0 0x5556bf3e7bd0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f04288f8ff56cb8f server_cookie=e72f8f7caa2feec3 in_seq=0 out_seq=0 2024-04-25T13:55:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.476+0000 7f631dffb640 10 client.27907 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.634+0000 7f631dffb640 1 -- 172.21.15.27:0/1628745046 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5556bf2d3ee0 con 0x5556bef5a1c0 2024-04-25T13:55:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.634+0000 7f631dffb640 10 client.27907 handle_command_reply: tid=0 2024-04-25T13:55:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.634+0000 7f63437c5740 1 client.27907 shutdown 2024-04-25T13:55:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f63437c5740 1 -- 172.21.15.27:0/1628745046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556bef9d2f0 msgr2=0x5556bf3749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.635 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f63437c5740 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556bef9d2f0 0x5556bf3749d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x5556bf367c80 tx=0x7f63280a3980 comp rx=0 tx=0).stop 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f63437c5740 1 -- 172.21.15.27:0/1628745046 shutdown_connections 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f63437c5740 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5556bef5a1c0 0x5556bf3e7bd0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f631dffb640 0 client.27907 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f63437c5740 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5556befc61b0 0x5556bf3e4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f63437c5740 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5556befa54d0 0x5556bf374f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f63437c5740 1 --2- 172.21.15.27:0/1628745046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556bef9d2f0 0x5556bf3749d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.635+0000 7f63437c5740 1 -- 172.21.15.27:0/1628745046 >> 172.21.15.27:0/1628745046 conn(0x5556befba300 msgr2=0x5556bf370650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.636+0000 7f63437c5740 1 -- 172.21.15.27:0/1628745046 shutdown_connections 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.636+0000 7f63437c5740 1 -- 172.21.15.27:0/1628745046 wait complete. 2024-04-25T13:55:02.636 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.636+0000 7f63437c5740 20 client.27907 trim_cache size 0 max 16384 2024-04-25T13:55:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:55:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:55:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:55:02.637 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:55:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.637+0000 7f633c915640 1 -- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63200afb30 msgr2=0x7f63200b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.637+0000 7f633c915640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63200afb30 0x7f63200b2040 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x5556bf367b40 tx=0x7f6330009460 comp rx=0 tx=0).stop 2024-04-25T13:55:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.637+0000 7f633c915640 1 -- 172.21.15.27:0/3935531168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6338103b40 msgr2=0x7f63381a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.637+0000 7f633c915640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6338103b40 0x7f63381a07e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f632800e9f0 tx=0x7f632800eeb0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.637+0000 7f633c915640 1 -- 172.21.15.27:0/3935531168 shutdown_connections 2024-04-25T13:55:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.638+0000 7f633c915640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f63200afb30 0x7f63200b2040 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.638+0000 7f633c915640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6338109a80 0x7f63381a7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.638+0000 7f633c915640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6338104d80 0x7f63381a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.638+0000 7f633c915640 1 --2- 172.21.15.27:0/3935531168 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6338103b40 0x7f63381a07e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.638+0000 7f633c915640 1 -- 172.21.15.27:0/3935531168 >> 172.21.15.27:0/3935531168 conn(0x7f63380fed30 msgr2=0x7f633810de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.638+0000 7f633c915640 1 -- 172.21.15.27:0/3935531168 shutdown_connections 2024-04-25T13:55:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:02.638+0000 7f633c915640 1 -- 172.21.15.27:0/3935531168 wait complete. 2024-04-25T13:55:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 835.0 s, 1556.0 tps, lat 63.812 ms stddev 81.366 2024-04-25T13:55:03.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:55:03.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:55:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.821+0000 7f7f1e242640 1 -- 172.21.15.27:0/2165618936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 msgr2=0x7f7f18110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.821+0000 7f7f1e242640 1 --2- 172.21.15.27:0/2165618936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 0x7f7f18110440 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7f1001acb0 tx=0x7f7f1003b900 comp rx=0 tx=0).stop 2024-04-25T13:55:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.822+0000 7f7f1e242640 1 -- 172.21.15.27:0/2165618936 shutdown_connections 2024-04-25T13:55:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.822+0000 7f7f1e242640 1 --2- 172.21.15.27:0/2165618936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 0x7f7f18110440 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.822+0000 7f7f1e242640 1 --2- 172.21.15.27:0/2165618936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f1810b310 0x7f7f1810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.822+0000 7f7f1e242640 1 --2- 172.21.15.27:0/2165618936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1810a420 0x7f7f1810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.822+0000 7f7f1e242640 1 -- 172.21.15.27:0/2165618936 >> 172.21.15.27:0/2165618936 conn(0x7f7f180704e0 msgr2=0x7f7f18072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:03.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.822+0000 7f7f1e242640 1 -- 172.21.15.27:0/2165618936 shutdown_connections 2024-04-25T13:55:03.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.822+0000 7f7f1e242640 1 -- 172.21.15.27:0/2165618936 wait complete. 2024-04-25T13:55:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.823+0000 7f7f1e242640 1 Processor -- start 2024-04-25T13:55:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.823+0000 7f7f1e242640 1 -- start start 2024-04-25T13:55:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.823+0000 7f7f1e242640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f1810a420 0x7f7f181a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.823+0000 7f7f1e242640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1810b310 0x7f7f181a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.823+0000 7f7f1e242640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 0x7f7f181a7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.823+0000 7f7f1e242640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18113d40 con 0x7f7f1810b310 2024-04-25T13:55:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.823+0000 7f7f1e242640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18113bc0 con 0x7f7f1810a420 2024-04-25T13:55:03.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.823+0000 7f7f1e242640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7f18113ec0 con 0x7f7f18110030 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f1810a420 0x7f7f181a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f1810a420 0x7f7f181a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57640/0 (socket says 172.21.15.27:57640) 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 -- 172.21.15.27:0/1189416485 learned_addr learned my addr 172.21.15.27:0/1189416485 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f17fff640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 0x7f7f181a7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1810b310 0x7f7f181a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 -- 172.21.15.27:0/1189416485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 msgr2=0x7f7f181a7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 0x7f7f181a7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 -- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1810b310 msgr2=0x7f7f181a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1810b310 0x7f7f181a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 -- 172.21.15.27:0/1189416485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f181a83d0 con 0x7f7f1810a420 2024-04-25T13:55:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f17fff640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 0x7f7f181a7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f177fe640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f1810a420 0x7f7f181a0890 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800e9c0 tx=0x7f7f0800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.824+0000 7f7f14ff9640 1 -- 172.21.15.27:0/1189416485 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800cd80 con 0x7f7f1810a420 2024-04-25T13:55:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.825+0000 7f7f1e242640 1 -- 172.21.15.27:0/1189416485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f181a86c0 con 0x7f7f1810a420 2024-04-25T13:55:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.825+0000 7f7f1e242640 1 -- 172.21.15.27:0/1189416485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f181a8c60 con 0x7f7f1810a420 2024-04-25T13:55:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.825+0000 7f7f14ff9640 1 -- 172.21.15.27:0/1189416485 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f080054d0 con 0x7f7f1810a420 2024-04-25T13:55:03.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.825+0000 7f7f14ff9640 1 -- 172.21.15.27:0/1189416485 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08010430 con 0x7f7f1810a420 2024-04-25T13:55:03.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.827+0000 7f7f14ff9640 1 -- 172.21.15.27:0/1189416485 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7f080105d0 con 0x7f7f1810a420 2024-04-25T13:55:03.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.827+0000 7f7f251cf740 1 Processor -- start 2024-04-25T13:55:03.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f251cf740 1 -- start start 2024-04-25T13:55:03.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11cec7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11cec7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11cec8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f251cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e11ce2cd80 con 0x55e11cb26390 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f251cf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e11cde68b0 con 0x55e11cb056b0 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f251cf740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e11ce28d40 con 0x55e11cafd4d0 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11cec7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11cec7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58838/0 (socket says 172.21.15.27:58838) 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f177fe640 1 -- 172.21.15.27:0/2208294979 learned_addr learned my addr 172.21.15.27:0/2208294979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f17fff640 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11cec7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.828+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11cec8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.829+0000 7f7f177fe640 1 -- 172.21.15.27:0/2208294979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 msgr2=0x55e11cec8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.829+0000 7f7f177fe640 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11cec8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.829+0000 7f7f177fe640 1 -- 172.21.15.27:0/2208294979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 msgr2=0x55e11cec7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.829+0000 7f7f177fe640 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11cec7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.829+0000 7f7f14ff9640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7f000afb70 0x7f7f000b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.829+0000 7f7f177fe640 1 -- 172.21.15.27:0/2208294979 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e11cdb7a40 con 0x55e11cafd4d0 2024-04-25T13:55:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.830+0000 7f7f177fe640 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11cec7fe0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7f080b0b70 tx=0x7f7f080a3af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53bf884eb81f3057 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.830+0000 7f7f14ff9640 1 -- 172.21.15.27:0/1189416485 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f7f080deed0 con 0x7f7f1810a420 2024-04-25T13:55:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.830+0000 7f7efdffb640 1 -- 172.21.15.27:0/2208294979 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f080a71b0 con 0x55e11cafd4d0 2024-04-25T13:55:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.830+0000 7f7efdffb640 1 -- 172.21.15.27:0/2208294979 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f080e2e50 con 0x55e11cafd4d0 2024-04-25T13:55:03.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.830+0000 7f7efdffb640 1 -- 172.21.15.27:0/2208294979 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f080a6b50 con 0x55e11cafd4d0 2024-04-25T13:55:03.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.831+0000 7f7f17fff640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7f000afb70 0x7f7f000b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 -- 172.21.15.27:0/2208294979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 msgr2=0x55e11cec7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11cec7fe0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7f080b0b70 tx=0x7f7f080a3af0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 -- 172.21.15.27:0/2208294979 shutdown_connections 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11cec8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11cec7fe0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 --2- 172.21.15.27:0/2208294979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11cec7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 -- 172.21.15.27:0/2208294979 >> 172.21.15.27:0/2208294979 conn(0x55e11cb1a4e0 msgr2=0x55e11cec3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 -- 172.21.15.27:0/2208294979 shutdown_connections 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f17fff640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7f000afb70 0x7f7f000b2080 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x55e11cec8ab0 tx=0x7f7f10040800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.832+0000 7f7f251cf740 1 -- 172.21.15.27:0/2208294979 wait complete. 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 1 -- start start 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 1 Processor -- start 2024-04-25T13:55:03.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11ced1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e11cec9780 con 0x55e11cb26390 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e11cdb3d10 con 0x55e11cb056b0 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e11cec9900 con 0x55e11cafd4d0 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58852/0 (socket says 172.21.15.27:58852) 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f177fe640 1 -- 172.21.15.27:0/1139228538 learned_addr learned my addr 172.21.15.27:0/1139228538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f251cf740 10 client.0 _ll_register_callbacks cb 0x55e11cde0210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.833+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11ced1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f17fff640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f16ffd640 1 -- 172.21.15.27:0/1139228538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 msgr2=0x55e11ced1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f16ffd640 1 -- 172.21.15.27:0/1139228538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 msgr2=0x55e11ced8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f16ffd640 1 -- 172.21.15.27:0/1139228538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e11cc087d0 con 0x55e11cb056b0 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f177fe640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f17fff640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11ced1c70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7f181a1930 tx=0x7f7f0c0119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7601c09d024543e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7ef6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/1139228538 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c013670 con 0x55e11cb056b0 2024-04-25T13:55:03.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/1139228538 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f7f0c013810 con 0x55e11cb056b0 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f251cf740 1 -- 172.21.15.27:0/1139228538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f080e2e50 con 0x55e11cb056b0 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/1139228538 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f7f0c013a50 con 0x55e11cb056b0 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f251cf740 10 client.27860 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.834+0000 7f7f251cf740 1 -- 172.21.15.27:0/1139228538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e11cb3e3f0 con 0x55e11cb056b0 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/1139228538 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f080e2e50 con 0x55e11cb056b0 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/1139228538 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f7f0c027420 con 0x55e11cb056b0 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7f251cf740 10 client.27860 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7f251cf740 10 client.27860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7f251cf740 20 client.27860 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7f251cf740 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e11cae4f60 0x55e11cedf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7f251cf740 4 client.27860 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:03.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7f251cf740 1 -- 172.21.15.27:0/1139228538 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e11ce3a990 con 0x55e11cae4f60 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.835+0000 7f7f177fe640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e11cae4f60 0x55e11cedf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.836+0000 7f7f177fe640 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e11cae4f60 0x55e11cedf0d0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7e8a54835d7b0e3 server_cookie=b05966feb98ec325 in_seq=0 out_seq=0 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.836+0000 7f7ef6ffd640 10 client.27860 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.836+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/1139228538 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e11ce3a990 con 0x55e11cae4f60 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.836+0000 7f7ef6ffd640 10 client.27860 handle_command_reply: tid=0 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.836+0000 7f7f251cf740 1 client.27860 shutdown 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 -- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 msgr2=0x55e11ced1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11ced1c70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7f181a1930 tx=0x7f7f0c0119d0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 -- 172.21.15.27:0/1139228538 shutdown_connections 2024-04-25T13:55:03.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e11cae4f60 0x55e11cedf0d0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb056b0 0x55e11ced1c70 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 --2- 172.21.15.27:0/1139228538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 -- 172.21.15.27:0/1139228538 >> 172.21.15.27:0/1139228538 conn(0x55e11cb1a4e0 msgr2=0x55e11cec23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:03.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7ef6ffd640 0 client.27860 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:03.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 -- 172.21.15.27:0/1139228538 shutdown_connections 2024-04-25T13:55:03.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 1 -- 172.21.15.27:0/1139228538 wait complete. 2024-04-25T13:55:03.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.837+0000 7f7f251cf740 20 client.27860 trim_cache size 0 max 16384 2024-04-25T13:55:03.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f251cf740 1 Processor -- start 2024-04-25T13:55:03.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f251cf740 1 -- start start 2024-04-25T13:55:03.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11cecff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb26390 0x55e11ced04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f251cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0400c390 con 0x55e11cafd4d0 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f251cf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0400ec10 con 0x55e11cb26390 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f251cf740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0400ca00 con 0x55e11cb056b0 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.863+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41206/0 (socket says 172.21.15.27:41206) 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f177fe640 1 -- 172.21.15.27:0/4139056942 learned_addr learned my addr 172.21.15.27:0/4139056942 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f17fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb26390 0x55e11ced04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11cecff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f16ffd640 1 -- 172.21.15.27:0/4139056942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb26390 msgr2=0x55e11ced04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb26390 0x55e11ced04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f16ffd640 1 -- 172.21.15.27:0/4139056942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cafd4d0 msgr2=0x55e11ced1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f16ffd640 1 -- 172.21.15.27:0/4139056942 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e11cafe0f0 con 0x55e11cb056b0 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f177fe640 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11cecff70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0279e0 tx=0x7f7f0c004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44f8f7bdd6bee29f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7efd7fa640 1 -- 172.21.15.27:0/4139056942 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c027620 con 0x55e11cb056b0 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7efd7fa640 1 -- 172.21.15.27:0/4139056942 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f0c013880 con 0x55e11cb056b0 2024-04-25T13:55:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.864+0000 7f7efd7fa640 1 -- 172.21.15.27:0/4139056942 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c014b50 con 0x55e11cb056b0 2024-04-25T13:55:03.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.865+0000 7f7f251cf740 1 -- 172.21.15.27:0/4139056942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 msgr2=0x55e11cecff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.865+0000 7f7f251cf740 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11cecff70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0279e0 tx=0x7f7f0c004770 comp rx=0 tx=0).stop 2024-04-25T13:55:03.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.865+0000 7f7f251cf740 1 -- 172.21.15.27:0/4139056942 shutdown_connections 2024-04-25T13:55:03.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.865+0000 7f7f251cf740 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cb26390 0x55e11ced04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.865+0000 7f7f251cf740 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11cecff70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.865+0000 7f7f251cf740 1 --2- 172.21.15.27:0/4139056942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cafd4d0 0x55e11ced1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.865+0000 7f7f251cf740 1 -- 172.21.15.27:0/4139056942 >> 172.21.15.27:0/4139056942 conn(0x55e11cb1a4e0 msgr2=0x55e11cedf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.866+0000 7f7f251cf740 1 -- 172.21.15.27:0/4139056942 shutdown_connections 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.866+0000 7f7f251cf740 1 -- 172.21.15.27:0/4139056942 wait complete. 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.866+0000 7f7f251cf740 1 -- start start 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.866+0000 7f7f251cf740 1 Processor -- start 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cafd4d0 0x55e11ced7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11ced4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f251cf740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f251cf740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0400d0d0 con 0x55e11cb26390 2024-04-25T13:55:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f251cf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0400e540 con 0x55e11cafd4d0 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f251cf740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0400d7a0 con 0x55e11cb056b0 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f251cf740 10 client.0 _ll_register_callbacks cb 0x7f7f0c009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11ced4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cafd4d0 0x55e11ced7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11ced4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58868/0 (socket says 172.21.15.27:58868) 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 -- 172.21.15.27:0/3480266381 learned_addr learned my addr 172.21.15.27:0/3480266381 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f17fff640 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 -- 172.21.15.27:0/3480266381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cafd4d0 msgr2=0x55e11ced7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cafd4d0 0x55e11ced7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 -- 172.21.15.27:0/3480266381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 msgr2=0x55e11ced4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 -- 172.21.15.27:0/3480266381 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e11cded4b0 con 0x55e11cb056b0 2024-04-25T13:55:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7f16ffd640 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11ced4930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009b20 tx=0x7f7f0c027650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f511c855f0abb860 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.867+0000 7f7ef6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.868+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/3480266381 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c034020 con 0x55e11cb056b0 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.868+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/3480266381 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f7f0c014850 con 0x55e11cb056b0 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.868+0000 7f7f251cf740 1 -- 172.21.15.27:0/3480266381 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f0c013880 con 0x55e11cb056b0 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.869+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/3480266381 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f7f0c014a50 con 0x55e11cb056b0 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.869+0000 7f7f251cf740 10 client.27925 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.869+0000 7f7f251cf740 1 -- 172.21.15.27:0/3480266381 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e11cf46150 con 0x55e11cb056b0 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.869+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/3480266381 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f0c013880 con 0x55e11cb056b0 2024-04-25T13:55:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.869+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/3480266381 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f7f0c006220 con 0x55e11cb056b0 2024-04-25T13:55:03.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.870+0000 7f7f251cf740 10 client.27925 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:03.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.870+0000 7f7f251cf740 10 client.27925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:03.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.870+0000 7f7f251cf740 20 client.27925 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:03.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.870+0000 7f7f251cf740 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e11caba3a0 0x55e11cf48b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:03.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.870+0000 7f7f251cf740 4 client.27925 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:55:03.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.870+0000 7f7f251cf740 1 -- 172.21.15.27:0/3480266381 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e11ce34800 con 0x55e11caba3a0 2024-04-25T13:55:03.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.870+0000 7f7f177fe640 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e11caba3a0 0x55e11cf48b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.871+0000 7f7f177fe640 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e11caba3a0 0x55e11cf48b90 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dade3a98fc63a6d server_cookie=a5079a50982c833d in_seq=0 out_seq=0 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.871+0000 7f7ef6ffd640 10 client.27925 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.871+0000 7f7ef6ffd640 1 -- 172.21.15.27:0/3480266381 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e11ce34800 con 0x55e11caba3a0 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.871+0000 7f7ef6ffd640 10 client.27925 handle_command_reply: tid=0 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.871+0000 7f7f251cf740 1 client.27925 shutdown 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.873+0000 7f7f251cf740 1 -- 172.21.15.27:0/3480266381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 msgr2=0x55e11ced4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.873+0000 7f7f251cf740 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11ced4930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009b20 tx=0x7f7f0c027650 comp rx=0 tx=0).stop 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.874+0000 7f7f251cf740 1 -- 172.21.15.27:0/3480266381 shutdown_connections 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.874+0000 7f7f251cf740 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e11caba3a0 0x55e11cf48b90 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.874+0000 7f7f251cf740 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e11cb26390 0x55e11ced4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.874+0000 7f7f251cf740 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e11cb056b0 0x55e11ced4930 secure :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009b20 tx=0x7f7f0c027650 comp rx=0 tx=0).stop 2024-04-25T13:55:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.874+0000 7f7f251cf740 1 --2- 172.21.15.27:0/3480266381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e11cafd4d0 0x55e11ced7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.874+0000 7f7f251cf740 1 -- 172.21.15.27:0/3480266381 >> 172.21.15.27:0/3480266381 conn(0x55e11cb1a4e0 msgr2=0x55e11cedf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.874+0000 7f7ef6ffd640 0 client.27925 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.876+0000 7f7f251cf740 1 -- 172.21.15.27:0/3480266381 shutdown_connections 2024-04-25T13:55:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.876+0000 7f7f251cf740 1 -- 172.21.15.27:0/3480266381 wait complete. 2024-04-25T13:55:03.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.876+0000 7f7f251cf740 20 client.27925 trim_cache size 0 max 16384 2024-04-25T13:55:03.877 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:55:03.877 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:55:03.877 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:55:03.877 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:55:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 -- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7f000afb70 msgr2=0x7f7f000b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7f000afb70 0x7f7f000b2080 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x55e11cec8ab0 tx=0x7f7f10040800 comp rx=0 tx=0).stop 2024-04-25T13:55:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 -- 172.21.15.27:0/1189416485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f1810a420 msgr2=0x7f7f181a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f1810a420 0x7f7f181a0890 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800e9c0 tx=0x7f7f0800ee80 comp rx=0 tx=0).stop 2024-04-25T13:55:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 -- 172.21.15.27:0/1189416485 shutdown_connections 2024-04-25T13:55:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f7f000afb70 0x7f7f000b2080 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7f18110030 0x7f7f181a7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f1810b310 0x7f7f181a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 --2- 172.21.15.27:0/1189416485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f1810a420 0x7f7f181a0890 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:03.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 -- 172.21.15.27:0/1189416485 >> 172.21.15.27:0/1189416485 conn(0x7f7f180704e0 msgr2=0x7f7f1810e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:03.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.877+0000 7f7f1e242640 1 -- 172.21.15.27:0/1189416485 shutdown_connections 2024-04-25T13:55:03.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:03.878+0000 7f7f1e242640 1 -- 172.21.15.27:0/1189416485 wait complete. 2024-04-25T13:55:03.892 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:55:03.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:55:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.064+0000 7f93dd538640 1 -- 172.21.15.27:0/2848090842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93d8110030 msgr2=0x7f93d8110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.064+0000 7f93dd538640 1 --2- 172.21.15.27:0/2848090842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93d8110030 0x7f93d8110440 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009900 tx=0x7f93d002a440 comp rx=0 tx=0).stop 2024-04-25T13:55:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.064+0000 7f93dd538640 1 -- 172.21.15.27:0/2848090842 shutdown_connections 2024-04-25T13:55:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.064+0000 7f93dd538640 1 --2- 172.21.15.27:0/2848090842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93d8110030 0x7f93d8110440 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.064+0000 7f93dd538640 1 --2- 172.21.15.27:0/2848090842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d810b310 0x7f93d810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.064+0000 7f93dd538640 1 --2- 172.21.15.27:0/2848090842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93d810a420 0x7f93d810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.064+0000 7f93dd538640 1 -- 172.21.15.27:0/2848090842 >> 172.21.15.27:0/2848090842 conn(0x7f93d80704e0 msgr2=0x7f93d8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:04.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.065+0000 7f93dd538640 1 -- 172.21.15.27:0/2848090842 shutdown_connections 2024-04-25T13:55:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.068+0000 7f93dd538640 1 -- 172.21.15.27:0/2848090842 wait complete. 2024-04-25T13:55:04.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.069+0000 7f93dd538640 1 Processor -- start 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.069+0000 7f93dd538640 1 -- start start 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.069+0000 7f93dd538640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93d810a420 0x7f93d81170b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.069+0000 7f93dd538640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93d810b310 0x7f93d81175f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.069+0000 7f93dd538640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d8110030 0x7f93d8118550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.069+0000 7f93dd538640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f93d811e590 con 0x7f93d810b310 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.069+0000 7f93dd538640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93d811e410 con 0x7f93d8110030 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.069+0000 7f93dd538640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f93d811e710 con 0x7f93d810a420 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.071+0000 7f93d77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d8110030 0x7f93d8118550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.071+0000 7f93d6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93d810a420 0x7f93d81170b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.072+0000 7f93d6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93d810a420 0x7f93d81170b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58872/0 (socket says 172.21.15.27:58872) 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.072+0000 7f93d6ffd640 1 -- 172.21.15.27:0/3766858661 learned_addr learned my addr 172.21.15.27:0/3766858661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.072+0000 7f93d6ffd640 1 -- 172.21.15.27:0/3766858661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d8110030 msgr2=0x7f93d8118550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.072+0000 7f93d6ffd640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d8110030 0x7f93d8118550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.072+0000 7f93d6ffd640 1 -- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93d810b310 msgr2=0x7f93d81175f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:55:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.072+0000 7f93d6ffd640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93d810b310 0x7f93d81175f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.072+0000 7f93d6ffd640 1 -- 172.21.15.27:0/3766858661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d8118bc0 con 0x7f93d810a420 2024-04-25T13:55:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.073+0000 7f93d6ffd640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93d810a420 0x7f93d81170b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f93c800b9e0 tx=0x7f93c800bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.074+0000 7f93bffff640 1 -- 172.21.15.27:0/3766858661 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8019070 con 0x7f93d810a420 2024-04-25T13:55:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.074+0000 7f93bffff640 1 -- 172.21.15.27:0/3766858661 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c800cba0 con 0x7f93d810a420 2024-04-25T13:55:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.074+0000 7f93bffff640 1 -- 172.21.15.27:0/3766858661 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80125e0 con 0x7f93d810a420 2024-04-25T13:55:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.075+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d8118eb0 con 0x7f93d810a420 2024-04-25T13:55:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.075+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d8119430 con 0x7f93d810a420 2024-04-25T13:55:04.077 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:03 smithi045 ceph-mon[32731]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 17 MiB/s wr, 744 op/s 2024-04-25T13:55:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.077+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d8119f10 con 0x7f93d810a420 2024-04-25T13:55:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.079+0000 7f93bffff640 1 -- 172.21.15.27:0/3766858661 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f93c801c070 con 0x7f93d810a420 2024-04-25T13:55:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.080+0000 7f93bffff640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93c00afc80 0x7f93c00b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.081+0000 7f93bffff640 1 -- 172.21.15.27:0/3766858661 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f93c80de7a0 con 0x7f93d810a420 2024-04-25T13:55:04.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.081+0000 7f93d67fc640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93c00afc80 0x7f93c00b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.082+0000 7f93d67fc640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93c00afc80 0x7f93c00b2190 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f93d8118150 tx=0x7f93cc006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.089+0000 7f93bffff640 1 -- 172.21.15.27:0/3766858661 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93c80a7430 con 0x7f93d810a420 2024-04-25T13:55:04.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:03 smithi027 ceph-mon[29140]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 17 MiB/s wr, 744 op/s 2024-04-25T13:55:04.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:03 smithi122 ceph-mon[32843]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 17 MiB/s wr, 744 op/s 2024-04-25T13:55:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.457+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93d811a110 con 0x7f93d810a420 2024-04-25T13:55:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.458+0000 7f93bffff640 1 -- 172.21.15.27:0/3766858661 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f93c80a6d30 con 0x7f93d810a420 2024-04-25T13:55:04.458 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:55:04.458 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:55:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:55:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93c00afc80 msgr2=0x7f93c00b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93c00afc80 0x7f93c00b2190 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f93d8118150 tx=0x7f93cc006bd0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93d810a420 msgr2=0x7f93d81170b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93d810a420 0x7f93d81170b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f93c800b9e0 tx=0x7f93c800bea0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 shutdown_connections 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f93c00afc80 0x7f93c00b2190 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d8110030 0x7f93d8118550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93d810b310 0x7f93d81175f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 --2- 172.21.15.27:0/3766858661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93d810a420 0x7f93d81170b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 >> 172.21.15.27:0/3766858661 conn(0x7f93d80704e0 msgr2=0x7f93d810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 shutdown_connections 2024-04-25T13:55:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.461+0000 7f93dd538640 1 -- 172.21.15.27:0/3766858661 wait complete. 2024-04-25T13:55:04.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:55:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.638+0000 7f33d6ad2640 1 -- 172.21.15.27:0/3641195759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d010b1d0 msgr2=0x7f33d010b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.638+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/3641195759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d010b1d0 0x7f33d010b5e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009900 tx=0x7f33c002a440 comp rx=0 tx=0).stop 2024-04-25T13:55:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.641+0000 7f33d6ad2640 1 -- 172.21.15.27:0/3641195759 shutdown_connections 2024-04-25T13:55:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.641+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/3641195759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33d010fef0 0x7f33d0110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.641+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/3641195759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d010b1d0 0x7f33d010b5e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.641+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/3641195759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d010a350 0x7f33d010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.641+0000 7f33d6ad2640 1 -- 172.21.15.27:0/3641195759 >> 172.21.15.27:0/3641195759 conn(0x7f33d0070400 msgr2=0x7f33d0072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:04.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.647+0000 7f33d6ad2640 1 -- 172.21.15.27:0/3641195759 shutdown_connections 2024-04-25T13:55:04.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.649+0000 7f33d6ad2640 1 -- 172.21.15.27:0/3641195759 wait complete. 2024-04-25T13:55:04.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.649+0000 7f33d6ad2640 1 Processor -- start 2024-04-25T13:55:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.652+0000 7f33d6ad2640 1 -- start start 2024-04-25T13:55:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.652+0000 7f33d6ad2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d010a350 0x7f33d01a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.652+0000 7f33d6ad2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33d010b1d0 0x7f33d01a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.653+0000 7f33d6ad2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d010fef0 0x7f33d01b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.653+0000 7f33d6ad2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33d011e2b0 con 0x7f33d010fef0 2024-04-25T13:55:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.653+0000 7f33d6ad2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33d011e130 con 0x7f33d010a350 2024-04-25T13:55:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.653+0000 7f33d6ad2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f33d011e430 con 0x7f33d010b1d0 2024-04-25T13:55:04.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.654+0000 7f33d5ad0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d010a350 0x7f33d01a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.654+0000 7f33d5ad0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d010a350 0x7f33d01a8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57708/0 (socket says 172.21.15.27:57708) 2024-04-25T13:55:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.654+0000 7f33d5ad0640 1 -- 172.21.15.27:0/1308979296 learned_addr learned my addr 172.21.15.27:0/1308979296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.654+0000 7f33d62d1640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d010fef0 0x7f33d01b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.655+0000 7f33d52cf640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33d010b1d0 0x7f33d01a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.660+0000 7f33d5ad0640 1 -- 172.21.15.27:0/1308979296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33d010b1d0 msgr2=0x7f33d01a93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.660+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33d010b1d0 0x7f33d01a93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.660+0000 7f33d5ad0640 1 -- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d010fef0 msgr2=0x7f33d01b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.660+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d010fef0 0x7f33d01b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.660+0000 7f33d5ad0640 1 -- 172.21.15.27:0/1308979296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d01b09a0 con 0x7f33d010a350 2024-04-25T13:55:04.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.662+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d010a350 0x7f33d01a8e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00b450 tx=0x7f33cc00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:04.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.665+0000 7f33c6ffd640 1 -- 172.21.15.27:0/1308979296 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc015020 con 0x7f33d010a350 2024-04-25T13:55:04.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.665+0000 7f33d6ad2640 1 -- 172.21.15.27:0/1308979296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d01b0c90 con 0x7f33d010a350 2024-04-25T13:55:04.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.665+0000 7f33d6ad2640 1 -- 172.21.15.27:0/1308979296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d01b1190 con 0x7f33d010a350 2024-04-25T13:55:04.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.665+0000 7f33c6ffd640 1 -- 172.21.15.27:0/1308979296 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33cc011d40 con 0x7f33d010a350 2024-04-25T13:55:04.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.666+0000 7f33c6ffd640 1 -- 172.21.15.27:0/1308979296 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc00f340 con 0x7f33d010a350 2024-04-25T13:55:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.671+0000 7f33debe3740 1 Processor -- start 2024-04-25T13:55:04.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.677+0000 7f33c6ffd640 1 -- 172.21.15.27:0/1308979296 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f33cc018020 con 0x7f33d010a350 2024-04-25T13:55:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.678+0000 7f33debe3740 1 -- start start 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33debe3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33debe3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33debe3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209ae8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33debe3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564209a4bd80 con 0x56420971c4d0 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33debe3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564209a05a20 con 0x564209745390 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33debe3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564209a47d40 con 0x5642097246b0 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33c6ffd640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33bc0afd00 0x7f33bc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33d62d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33d62d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57724/0 (socket says 172.21.15.27:57724) 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33d62d1640 1 -- 172.21.15.27:0/3824226567 learned_addr learned my addr 172.21.15.27:0/3824226567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.679+0000 7f33c6ffd640 1 -- 172.21.15.27:0/1308979296 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f33cc0de4f0 con 0x7f33d010a350 2024-04-25T13:55:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.680+0000 7f33d62d1640 1 -- 172.21.15.27:0/3824226567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 msgr2=0x564209ae8c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:55:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.680+0000 7f33d62d1640 1 --2- 172.21.15.27:0/3824226567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209ae8c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.680+0000 7f33d62d1640 1 -- 172.21.15.27:0/3824226567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 msgr2=0x564209ae7f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:55:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.680+0000 7f33d62d1640 1 --2- 172.21.15.27:0/3824226567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae7f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.680+0000 7f33d62d1640 1 -- 172.21.15.27:0/3824226567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564209a10c30 con 0x564209745390 2024-04-25T13:55:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.680+0000 7f33d62d1640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33bc0afd00 0x7f33bc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.681+0000 7f33d62d1640 1 --2- 172.21.15.27:0/3824226567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae7a00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f33c80113e0 tx=0x7f33c80118a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eb2396932e6068c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.682+0000 7f33bbfff640 1 -- 172.21.15.27:0/3824226567 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8012e40 con 0x564209745390 2024-04-25T13:55:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.682+0000 7f33bbfff640 1 -- 172.21.15.27:0/3824226567 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33c800fd80 con 0x564209745390 2024-04-25T13:55:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.683+0000 7f33debe3740 1 -- 172.21.15.27:0/3824226567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 msgr2=0x564209ae7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.683+0000 7f33debe3740 1 --2- 172.21.15.27:0/3824226567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae7a00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f33c80113e0 tx=0x7f33c80118a0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.683+0000 7f33d62d1640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33bc0afd00 0x7f33bc0b2210 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f33c8020000 tx=0x7f33c801d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.684+0000 7f33debe3740 1 -- 172.21.15.27:0/3824226567 shutdown_connections 2024-04-25T13:55:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.684+0000 7f33debe3740 1 --2- 172.21.15.27:0/3824226567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209ae8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.684+0000 7f33debe3740 1 --2- 172.21.15.27:0/3824226567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.684+0000 7f33debe3740 1 --2- 172.21.15.27:0/3824226567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae7a00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.684+0000 7f33debe3740 1 -- 172.21.15.27:0/3824226567 >> 172.21.15.27:0/3824226567 conn(0x5642097394e0 msgr2=0x564209ae3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:04.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.690+0000 7f33debe3740 1 -- 172.21.15.27:0/3824226567 shutdown_connections 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.690+0000 7f33debe3740 1 -- 172.21.15.27:0/3824226567 wait complete. 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.690+0000 7f33debe3740 1 -- start start 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.690+0000 7f33debe3740 1 Processor -- start 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.691+0000 7f33debe3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209b02830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.691+0000 7f33debe3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209b02d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.691+0000 7f33debe3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209b09cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.691+0000 7f33debe3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564209ae9770 con 0x56420971c4d0 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.691+0000 7f33debe3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5642099d2dc0 con 0x564209745390 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.691+0000 7f33debe3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564209ae98f0 con 0x5642097246b0 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.691+0000 7f33debe3740 10 client.0 _ll_register_callbacks cb 0x5642099ee600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.692+0000 7f33d5ad0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209b02830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.692+0000 7f33d5ad0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209b02830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41244/0 (socket says 172.21.15.27:41244) 2024-04-25T13:55:04.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.692+0000 7f33d5ad0640 1 -- 172.21.15.27:0/4280326357 learned_addr learned my addr 172.21.15.27:0/4280326357 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.692+0000 7f33d5ad0640 1 -- 172.21.15.27:0/4280326357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 msgr2=0x564209b02d70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.692+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209b02d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.692+0000 7f33d5ad0640 1 -- 172.21.15.27:0/4280326357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 msgr2=0x564209b09cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.692+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209b09cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.692+0000 7f33d5ad0640 1 -- 172.21.15.27:0/4280326357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56420975d3f0 con 0x56420971c4d0 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.693+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209b02830 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f33cc000c00 tx=0x7f33cc0a6200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76f33432d63c97c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.693+0000 7f33b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.693+0000 7f33b97fa640 1 -- 172.21.15.27:0/4280326357 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0c0060 con 0x56420971c4d0 2024-04-25T13:55:04.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.693+0000 7f33b97fa640 1 -- 172.21.15.27:0/4280326357 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f33cc0c90b0 con 0x56420971c4d0 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.693+0000 7f33debe3740 1 -- 172.21.15.27:0/4280326357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33c800fd80 con 0x56420971c4d0 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.693+0000 7f33b97fa640 1 -- 172.21.15.27:0/4280326357 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f33cc0aad50 con 0x56420971c4d0 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.693+0000 7f33debe3740 10 client.18210 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.693+0000 7f33debe3740 1 -- 172.21.15.27:0/4280326357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5642098277d0 con 0x56420971c4d0 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.694+0000 7f33b97fa640 1 -- 172.21.15.27:0/4280326357 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33c800fd80 con 0x56420971c4d0 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.694+0000 7f33b97fa640 1 -- 172.21.15.27:0/4280326357 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f33cc0c3070 con 0x56420971c4d0 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.694+0000 7f33debe3740 10 client.18210 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.694+0000 7f33debe3740 10 client.18210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.694+0000 7f33debe3740 20 client.18210 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.694+0000 7f33debe3740 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564209703f60 0x564209af67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.694+0000 7f33debe3740 4 client.18210 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.694+0000 7f33debe3740 1 -- 172.21.15.27:0/4280326357 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564209a59990 con 0x564209703f60 2024-04-25T13:55:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.695+0000 7f33d52cf640 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564209703f60 0x564209af67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.695+0000 7f33b97fa640 10 client.18210 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.695+0000 7f33d52cf640 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564209703f60 0x564209af67e0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95dee175dfcef362 server_cookie=fc6f1bf75371911d in_seq=0 out_seq=0 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.696+0000 7f33b97fa640 1 -- 172.21.15.27:0/4280326357 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564209a59990 con 0x564209703f60 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.696+0000 7f33b97fa640 10 client.18210 handle_command_reply: tid=0 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.696+0000 7f33debe3740 1 client.18210 shutdown 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 -- 172.21.15.27:0/4280326357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 msgr2=0x564209b02830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209b02830 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f33cc000c00 tx=0x7f33cc0a6200 comp rx=0 tx=0).stop 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 -- 172.21.15.27:0/4280326357 shutdown_connections 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564209703f60 0x564209af67e0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209b09cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209b02d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 --2- 172.21.15.27:0/4280326357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209b02830 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 -- 172.21.15.27:0/4280326357 >> 172.21.15.27:0/4280326357 conn(0x5642097394e0 msgr2=0x564209ae23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33b97fa640 0 client.18210 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 -- 172.21.15.27:0/4280326357 shutdown_connections 2024-04-25T13:55:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 1 -- 172.21.15.27:0/4280326357 wait complete. 2024-04-25T13:55:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.697+0000 7f33debe3740 20 client.18210 trim_cache size 0 max 16384 2024-04-25T13:55:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.726+0000 7f33debe3740 1 Processor -- start 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.726+0000 7f33debe3740 1 -- start start 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.726+0000 7f33debe3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.726+0000 7f33debe3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209ae1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.726+0000 7f33debe3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.726+0000 7f33debe3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33b400c380 con 0x56420971c4d0 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.726+0000 7f33debe3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33b400ec00 con 0x564209745390 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.726+0000 7f33debe3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f33b400c9f0 con 0x5642097246b0 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d5ad0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d62d1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d5ad0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41246/0 (socket says 172.21.15.27:41246) 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d5ad0640 1 -- 172.21.15.27:0/2224868737 learned_addr learned my addr 172.21.15.27:0/2224868737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d52cf640 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209ae1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d62d1640 1 -- 172.21.15.27:0/2224868737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 msgr2=0x564209ae1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d62d1640 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209ae1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d62d1640 1 -- 172.21.15.27:0/2224868737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 msgr2=0x564209ae19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d62d1640 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d62d1640 1 -- 172.21.15.27:0/2224868737 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564209704190 con 0x564209745390 2024-04-25T13:55:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d52cf640 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209ae1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.727+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.730+0000 7f33d62d1640 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae2440 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x564209ae8a50 tx=0x7f33c800a540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=423b2074e6b449b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.730+0000 7f33bb7fe640 1 -- 172.21.15.27:0/2224868737 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c800fd80 con 0x564209745390 2024-04-25T13:55:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.731+0000 7f33bb7fe640 1 -- 172.21.15.27:0/2224868737 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33c800ab30 con 0x564209745390 2024-04-25T13:55:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.731+0000 7f33bb7fe640 1 -- 172.21.15.27:0/2224868737 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8028480 con 0x564209745390 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 -- 172.21.15.27:0/2224868737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 msgr2=0x564209ae2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae2440 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x564209ae8a50 tx=0x7f33c800a540 comp rx=0 tx=0).stop 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 -- 172.21.15.27:0/2224868737 shutdown_connections 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564209745390 0x564209ae2440 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642097246b0 0x564209ae1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 --2- 172.21.15.27:0/2224868737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56420971c4d0 0x564209ae19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 -- 172.21.15.27:0/2224868737 >> 172.21.15.27:0/2224868737 conn(0x5642097394e0 msgr2=0x564209aec440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 -- 172.21.15.27:0/2224868737 shutdown_connections 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 -- 172.21.15.27:0/2224868737 wait complete. 2024-04-25T13:55:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.732+0000 7f33debe3740 1 -- start start 2024-04-25T13:55:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33debe3740 1 Processor -- start 2024-04-25T13:55:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33debe3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56420971c4d0 0x564209b09360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33debe3740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642097246b0 0x564209b855c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33debe3740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564209745390 0x564209b87c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33debe3740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33b400d0c0 con 0x5642097246b0 2024-04-25T13:55:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33debe3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f33b400e530 con 0x56420971c4d0 2024-04-25T13:55:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33debe3740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f33b400d790 con 0x564209745390 2024-04-25T13:55:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33d5ad0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56420971c4d0 0x564209b09360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33d52cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642097246b0 0x564209b855c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33d5ad0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56420971c4d0 0x564209b09360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57742/0 (socket says 172.21.15.27:57742) 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33d52cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642097246b0 0x564209b855c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41250/0 (socket says 172.21.15.27:41250) 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.733+0000 7f33d5ad0640 1 -- 172.21.15.27:0/1279311152 learned_addr learned my addr 172.21.15.27:0/1279311152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33d62d1640 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564209745390 0x564209b87c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33debe3740 10 client.0 _ll_register_callbacks cb 0x7f33cc012f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33d5ad0640 1 -- 172.21.15.27:0/1279311152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564209745390 msgr2=0x564209b87c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564209745390 0x564209b87c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33d5ad0640 1 -- 172.21.15.27:0/1279311152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642097246b0 msgr2=0x564209b855c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642097246b0 0x564209b855c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33d5ad0640 1 -- 172.21.15.27:0/1279311152 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564209b883b0 con 0x56420971c4d0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33d5ad0640 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56420971c4d0 0x564209b09360 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009fd0 tx=0x7f33cc0a2950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd3189a709e8721 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33b97fa640 1 -- 172.21.15.27:0/1279311152 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0c0030 con 0x56420971c4d0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33b97fa640 1 -- 172.21.15.27:0/1279311152 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f33cc0c90b0 con 0x56420971c4d0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33debe3740 1 -- 172.21.15.27:0/1279311152 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33c800ab30 con 0x56420971c4d0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33b97fa640 1 -- 172.21.15.27:0/1279311152 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f33cc0a2690 con 0x56420971c4d0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33debe3740 10 client.27881 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.734+0000 7f33debe3740 1 -- 172.21.15.27:0/1279311152 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564209b89640 con 0x56420971c4d0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33b97fa640 1 -- 172.21.15.27:0/1279311152 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33cc0af800 con 0x56420971c4d0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33b97fa640 1 -- 172.21.15.27:0/1279311152 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f33cc0c3070 con 0x56420971c4d0 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33debe3740 10 client.27881 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33debe3740 10 client.27881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33debe3740 20 client.27881 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33debe3740 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5642096d93a0 0x564209b8bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33debe3740 4 client.27881 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33debe3740 1 -- 172.21.15.27:0/1279311152 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f33c800ab30 con 0x5642096d93a0 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.735+0000 7f33d52cf640 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5642096d93a0 0x564209b8bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.736+0000 7f33b97fa640 10 client.27881 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.736+0000 7f33d52cf640 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5642096d93a0 0x564209b8bef0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=566ae5cfae813d5e server_cookie=c1954629581099e8 in_seq=0 out_seq=0 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.736+0000 7f33b97fa640 1 -- 172.21.15.27:0/1279311152 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f33c800ab30 con 0x5642096d93a0 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.736+0000 7f33b97fa640 10 client.27881 handle_command_reply: tid=0 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.736+0000 7f33debe3740 1 client.27881 shutdown 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.736+0000 7f33debe3740 1 -- 172.21.15.27:0/1279311152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56420971c4d0 msgr2=0x564209b09360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.736+0000 7f33debe3740 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56420971c4d0 0x564209b09360 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009fd0 tx=0x7f33cc0a2950 comp rx=0 tx=0).stop 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 1 -- 172.21.15.27:0/1279311152 shutdown_connections 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5642096d93a0 0x564209b8bef0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564209745390 0x564209b87c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5642097246b0 0x564209b855c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33b97fa640 0 client.27881 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 1 --2- 172.21.15.27:0/1279311152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56420971c4d0 0x564209b09360 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 1 -- 172.21.15.27:0/1279311152 >> 172.21.15.27:0/1279311152 conn(0x5642097394e0 msgr2=0x564209ae6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 1 -- 172.21.15.27:0/1279311152 shutdown_connections 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 1 -- 172.21.15.27:0/1279311152 wait complete. 2024-04-25T13:55:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.737+0000 7f33debe3740 20 client.27881 trim_cache size 0 max 16384 2024-04-25T13:55:04.742 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:55:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 -- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33bc0afd00 msgr2=0x7f33bc0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33bc0afd00 0x7f33bc0b2210 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f33c8020000 tx=0x7f33c801d000 comp rx=0 tx=0).stop 2024-04-25T13:55:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 -- 172.21.15.27:0/1308979296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d010a350 msgr2=0x7f33d01a8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d010a350 0x7f33d01a8e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00b450 tx=0x7f33cc00b910 comp rx=0 tx=0).stop 2024-04-25T13:55:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 -- 172.21.15.27:0/1308979296 shutdown_connections 2024-04-25T13:55:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f33bc0afd00 0x7f33bc0b2210 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33d010fef0 0x7f33d01b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f33d010b1d0 0x7f33d01a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 --2- 172.21.15.27:0/1308979296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33d010a350 0x7f33d01a8e60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 -- 172.21.15.27:0/1308979296 >> 172.21.15.27:0/1308979296 conn(0x7f33d0070400 msgr2=0x7f33d010bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 -- 172.21.15.27:0/1308979296 shutdown_connections 2024-04-25T13:55:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:04.738+0000 7f33d6ad2640 1 -- 172.21.15.27:0/1308979296 wait complete. 2024-04-25T13:55:05.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:04 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3766858661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:55:05.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:04 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3766858661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:55:05.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:04 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3766858661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:55:05.752 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:55:05.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:55:05.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:55:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.937+0000 7feac5dc5640 1 -- 172.21.15.27:0/2219972184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0103b00 msgr2=0x7feac0104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.937+0000 7feac5dc5640 1 --2- 172.21.15.27:0/2219972184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0103b00 0x7feac0104800 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7feab0009900 tx=0x7feab002a440 comp rx=0 tx=0).stop 2024-04-25T13:55:05.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.942+0000 7feac5dc5640 1 -- 172.21.15.27:0/2219972184 shutdown_connections 2024-04-25T13:55:05.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.942+0000 7feac5dc5640 1 --2- 172.21.15.27:0/2219972184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feac0109a40 0x7feac010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.942+0000 7feac5dc5640 1 --2- 172.21.15.27:0/2219972184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac0104d40 0x7feac0105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.942+0000 7feac5dc5640 1 --2- 172.21.15.27:0/2219972184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0103b00 0x7feac0104800 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.942+0000 7feac5dc5640 1 -- 172.21.15.27:0/2219972184 >> 172.21.15.27:0/2219972184 conn(0x7feac00fed10 msgr2=0x7feac01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.948+0000 7feac5dc5640 1 -- 172.21.15.27:0/2219972184 shutdown_connections 2024-04-25T13:55:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.952+0000 7feac5dc5640 1 -- 172.21.15.27:0/2219972184 wait complete. 2024-04-25T13:55:05.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.952+0000 7feac5dc5640 1 Processor -- start 2024-04-25T13:55:05.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feac5dc5640 1 -- start start 2024-04-25T13:55:05.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feac5dc5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac0103b00 0x7feac01133e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feac5dc5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0104d40 0x7feac0113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feac5dc5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feac0109a40 0x7feac0118830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feac5dc5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feac010d480 con 0x7feac0104d40 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feac5dc5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feac010d300 con 0x7feac0109a40 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feac5dc5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feac010d600 con 0x7feac0103b00 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feabf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac0103b00 0x7feac01133e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feabf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac0103b00 0x7feac01133e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58936/0 (socket says 172.21.15.27:58936) 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feabf7fe640 1 -- 172.21.15.27:0/141936499 learned_addr learned my addr 172.21.15.27:0/141936499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.953+0000 7feabeffd640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0104d40 0x7feac0113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.954+0000 7feabffff640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feac0109a40 0x7feac0118830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.954+0000 7feabeffd640 1 -- 172.21.15.27:0/141936499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac0103b00 msgr2=0x7feac01133e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.954+0000 7feabeffd640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac0103b00 0x7feac01133e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.954+0000 7feabeffd640 1 -- 172.21.15.27:0/141936499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feac0109a40 msgr2=0x7feac0118830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.954+0000 7feabeffd640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feac0109a40 0x7feac0118830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.954+0000 7feabeffd640 1 -- 172.21.15.27:0/141936499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac0118ea0 con 0x7feac0104d40 2024-04-25T13:55:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.954+0000 7feabeffd640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0104d40 0x7feac0113920 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7feab800cce0 tx=0x7feab80075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.958+0000 7feabcff9640 1 -- 172.21.15.27:0/141936499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8004b20 con 0x7feac0104d40 2024-04-25T13:55:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.958+0000 7feabcff9640 1 -- 172.21.15.27:0/141936499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab8004cc0 con 0x7feac0104d40 2024-04-25T13:55:05.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.959+0000 7feac5dc5640 1 -- 172.21.15.27:0/141936499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac0119190 con 0x7feac0104d40 2024-04-25T13:55:05.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.959+0000 7feac5dc5640 1 -- 172.21.15.27:0/141936499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac01b1530 con 0x7feac0104d40 2024-04-25T13:55:05.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.960+0000 7feabcff9640 1 -- 172.21.15.27:0/141936499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab800f6b0 con 0x7feac0104d40 2024-04-25T13:55:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.962+0000 7feaccc7a740 1 Processor -- start 2024-04-25T13:55:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.962+0000 7feaccc7a740 1 -- start start 2024-04-25T13:55:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.962+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd64410 0x55eed0105490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.962+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed0105a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd43730 0x55eed010c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feaccc7a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eed006ab00 con 0x55eecfd64410 2024-04-25T13:55:05.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feaccc7a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55eed0006920 con 0x55eecfd43730 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feaccc7a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55eed0066ac0 con 0x55eecfd3b550 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feabffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd64410 0x55eed0105490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feabffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd64410 0x55eed0105490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41266/0 (socket says 172.21.15.27:41266) 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feabffff640 1 -- 172.21.15.27:0/2121335006 learned_addr learned my addr 172.21.15.27:0/2121335006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feabf7fe640 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed0105a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.963+0000 7feabcff9640 1 -- 172.21.15.27:0/141936499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7feab800f850 con 0x7feac0104d40 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.964+0000 7feabeffd640 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd43730 0x55eed010c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.964+0000 7feabffff640 1 -- 172.21.15.27:0/2121335006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 msgr2=0x55eed0105a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.964+0000 7feabffff640 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed0105a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.964+0000 7feabffff640 1 -- 172.21.15.27:0/2121335006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd43730 msgr2=0x55eed010c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.964+0000 7feabffff640 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd43730 0x55eed010c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.964+0000 7feabffff640 1 -- 172.21.15.27:0/2121335006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eecfe447e0 con 0x55eecfd64410 2024-04-25T13:55:05.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.965+0000 7feabcff9640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feaa80afc80 0x7feaa80b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.965+0000 7feabffff640 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd64410 0x55eed0105490 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7feab402da20 tx=0x7feab402dee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217b045ca0cb3de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:05.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.965+0000 7feaa5ffb640 1 -- 172.21.15.27:0/2121335006 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4031430 con 0x55eecfd64410 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.966+0000 7feaa5ffb640 1 -- 172.21.15.27:0/2121335006 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab4031a80 con 0x55eecfd64410 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.966+0000 7feabcff9640 1 -- 172.21.15.27:0/141936499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feab80df010 con 0x7feac0104d40 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.966+0000 7feaa5ffb640 1 -- 172.21.15.27:0/2121335006 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4039a80 con 0x55eecfd64410 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.966+0000 7feabf7fe640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feaa80afc80 0x7feaa80b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 -- 172.21.15.27:0/2121335006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd64410 msgr2=0x55eed0105490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feabf7fe640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feaa80afc80 0x7feaa80b2190 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7feab0006010 tx=0x7feab0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd64410 0x55eed0105490 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7feab402da20 tx=0x7feab402dee0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 -- 172.21.15.27:0/2121335006 shutdown_connections 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd43730 0x55eed010c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed0105a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2121335006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd64410 0x55eed0105490 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 -- 172.21.15.27:0/2121335006 >> 172.21.15.27:0/2121335006 conn(0x55eecfd58560 msgr2=0x55eed0100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 -- 172.21.15.27:0/2121335006 shutdown_connections 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.967+0000 7feaccc7a740 1 -- 172.21.15.27:0/2121335006 wait complete. 2024-04-25T13:55:05.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 1 -- start start 2024-04-25T13:55:05.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 1 Processor -- start 2024-04-25T13:55:05.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd3b550 0x55eed010f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed010f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd64410 0x55eed0116840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55eed010d3b0 con 0x55eecfd43730 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55eed00045d0 con 0x55eecfd3b550 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55eed010d530 con 0x55eecfd64410 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.968+0000 7feaccc7a740 10 client.0 _ll_register_callbacks cb 0x55eecfff2080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed010f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed010f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41282/0 (socket says 172.21.15.27:41282) 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 -- 172.21.15.27:0/3012909883 learned_addr learned my addr 172.21.15.27:0/3012909883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabffff640 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd64410 0x55eed0116840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 -- 172.21.15.27:0/3012909883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd64410 msgr2=0x55eed0116840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd64410 0x55eed0116840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 -- 172.21.15.27:0/3012909883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd3b550 msgr2=0x55eed010f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd3b550 0x55eed010f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 -- 172.21.15.27:0/3012909883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eecfea4f70 con 0x55eecfd43730 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feabeffd640 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed010f8e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7feab80a7450 tx=0x7feab80a7290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2143cf9cd0d7be28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7fea9effd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7fea9effd640 1 -- 172.21.15.27:0/3012909883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab801d040 con 0x55eecfd43730 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feaccc7a740 1 -- 172.21.15.27:0/3012909883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feab4031a80 con 0x55eecfd43730 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7fea9effd640 1 -- 172.21.15.27:0/3012909883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feab80b00a0 con 0x55eecfd43730 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7fea9effd640 1 -- 172.21.15.27:0/3012909883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7feab80a6fd0 con 0x55eecfd43730 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feaccc7a740 10 client.18228 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.969+0000 7feaccc7a740 1 -- 172.21.15.27:0/3012909883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55eecfd3c170 con 0x55eecfd43730 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7fea9effd640 1 -- 172.21.15.27:0/3012909883 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab4031a80 con 0x55eecfd43730 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7fea9effd640 1 -- 172.21.15.27:0/3012909883 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7feab80e2150 con 0x55eecfd43730 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7feaccc7a740 10 client.18228 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:05.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7feaccc7a740 10 client.18228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:05.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7feaccc7a740 20 client.18228 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:05.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7feaccc7a740 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55eecfd231f0 0x55eed011cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:05.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7feaccc7a740 4 client.18228 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:05.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7feaccc7a740 1 -- 172.21.15.27:0/3012909883 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eed0078710 con 0x55eecfd231f0 2024-04-25T13:55:05.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.970+0000 7feabf7fe640 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55eecfd231f0 0x55eed011cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:05.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.971+0000 7fea9effd640 10 client.18228 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:05.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.971+0000 7feabf7fe640 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55eecfd231f0 0x55eed011cc20 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9682d36073ffee9 server_cookie=90e9034da00dfd16 in_seq=0 out_seq=0 2024-04-25T13:55:05.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.972+0000 7fea9effd640 1 -- 172.21.15.27:0/3012909883 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eed0078710 con 0x55eecfd231f0 2024-04-25T13:55:05.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.972+0000 7fea9effd640 10 client.18228 handle_command_reply: tid=0 2024-04-25T13:55:05.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.972+0000 7feaccc7a740 1 client.18228 shutdown 2024-04-25T13:55:05.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 -- 172.21.15.27:0/3012909883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 msgr2=0x55eed010f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:05.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed010f8e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7feab80a7450 tx=0x7feab80a7290 comp rx=0 tx=0).stop 2024-04-25T13:55:05.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 -- 172.21.15.27:0/3012909883 shutdown_connections 2024-04-25T13:55:05.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55eecfd231f0 0x55eed011cc20 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd64410 0x55eed0116840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7fea9effd640 0 client.18228 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:05.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed010f8e0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 --2- 172.21.15.27:0/3012909883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd3b550 0x55eed010f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:05.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 -- 172.21.15.27:0/3012909883 >> 172.21.15.27:0/3012909883 conn(0x55eecfd58560 msgr2=0x55eed00ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:05.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 -- 172.21.15.27:0/3012909883 shutdown_connections 2024-04-25T13:55:05.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 1 -- 172.21.15.27:0/3012909883 wait complete. 2024-04-25T13:55:05.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:05.973+0000 7feaccc7a740 20 client.18228 trim_cache size 0 max 16384 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feaccc7a740 1 Processor -- start 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feaccc7a740 1 -- start start 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd3b550 0x55eed010e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd43730 0x55eed0109e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 0x55eed0116760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feaccc7a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feac0120090 con 0x55eecfd3b550 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feaccc7a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feac0122910 con 0x55eecfd64410 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feaccc7a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feac0120700 con 0x55eecfd43730 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd3b550 0x55eed010e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd3b550 0x55eed010e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41284/0 (socket says 172.21.15.27:41284) 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabf7fe640 1 -- 172.21.15.27:0/2813934494 learned_addr learned my addr 172.21.15.27:0/2813934494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabeffd640 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd43730 0x55eed0109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabffff640 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 0x55eed0116760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabf7fe640 1 -- 172.21.15.27:0/2813934494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd43730 msgr2=0x55eed0109e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabf7fe640 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd43730 0x55eed0109e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabf7fe640 1 -- 172.21.15.27:0/2813934494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 msgr2=0x55eed0116760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabf7fe640 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 0x55eed0116760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.002+0000 7feabf7fe640 1 -- 172.21.15.27:0/2813934494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eed0108830 con 0x55eecfd3b550 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.003+0000 7feabf7fe640 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd3b550 0x55eed010e3f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7feab0004ba0 tx=0x7feab0041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973a066968103da9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.003+0000 7feaa57fa640 1 -- 172.21.15.27:0/2813934494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab0057070 con 0x55eecfd3b550 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.003+0000 7feaa57fa640 1 -- 172.21.15.27:0/2813934494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55eed0078710 con 0x55eecfd3b550 2024-04-25T13:55:06.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.003+0000 7feaa57fa640 1 -- 172.21.15.27:0/2813934494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab0031840 con 0x55eecfd3b550 2024-04-25T13:55:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 -- 172.21.15.27:0/2813934494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd3b550 msgr2=0x55eed010e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd3b550 0x55eed010e3f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7feab0004ba0 tx=0x7feab0041000 comp rx=0 tx=0).stop 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 -- 172.21.15.27:0/2813934494 shutdown_connections 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 0x55eed0116760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd43730 0x55eed0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2813934494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd3b550 0x55eed010e3f0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 -- 172.21.15.27:0/2813934494 >> 172.21.15.27:0/2813934494 conn(0x55eecfd58560 msgr2=0x55eed0103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 -- 172.21.15.27:0/2813934494 shutdown_connections 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.004+0000 7feaccc7a740 1 -- 172.21.15.27:0/2813934494 wait complete. 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.005+0000 7feaccc7a740 1 -- start start 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.005+0000 7feaccc7a740 1 Processor -- start 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed017ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed01814b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feaccc7a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 0x55eed0183a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feaccc7a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feac0120dd0 con 0x55eecfd43730 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feaccc7a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feac0122240 con 0x55eecfd64410 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feaccc7a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7feac01214a0 con 0x55eecfd3b550 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed01814b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed017ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 0x55eed0183a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feaccc7a740 10 client.0 _ll_register_callbacks cb 0x7feab8004e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed01814b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41292/0 (socket says 172.21.15.27:41292) 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabeffd640 1 -- 172.21.15.27:0/2998130460 learned_addr learned my addr 172.21.15.27:0/2998130460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabf7fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed017ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58970/0 (socket says 172.21.15.27:58970) 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabf7fe640 1 -- 172.21.15.27:0/2998130460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 msgr2=0x55eed0183a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabf7fe640 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 0x55eed0183a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabf7fe640 1 -- 172.21.15.27:0/2998130460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 msgr2=0x55eed01814b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabf7fe640 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed01814b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabf7fe640 1 -- 172.21.15.27:0/2998130460 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eed01841c0 con 0x55eecfd3b550 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.006+0000 7feabeffd640 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed01814b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feabf7fe640 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed017ef60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7feab002aa30 tx=0x7feab0005510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=278a1c371a5cf8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7fea9effd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7fea9effd640 1 -- 172.21.15.27:0/2998130460 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab0057040 con 0x55eecfd3b550 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7fea9effd640 1 -- 172.21.15.27:0/2998130460 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7feab0053740 con 0x55eecfd3b550 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feaccc7a740 1 -- 172.21.15.27:0/2998130460 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eed0078710 con 0x55eecfd3b550 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7fea9effd640 1 -- 172.21.15.27:0/2998130460 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7feab006b070 con 0x55eecfd3b550 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feaccc7a740 10 client.27952 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feaccc7a740 1 -- 172.21.15.27:0/2998130460 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55eed01854f0 con 0x55eecfd3b550 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7fea9effd640 1 -- 172.21.15.27:0/2998130460 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55eed0078710 con 0x55eecfd3b550 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7fea9effd640 1 -- 172.21.15.27:0/2998130460 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7feab005c570 con 0x55eecfd3b550 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feaccc7a740 10 client.27952 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feaccc7a740 10 client.27952 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feaccc7a740 20 client.27952 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55eecfcf8630 0x55eed0187eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.007+0000 7feaccc7a740 4 client.27952 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:55:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.008+0000 7feaccc7a740 1 -- 172.21.15.27:0/2998130460 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55eed0072580 con 0x55eecfcf8630 2024-04-25T13:55:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.008+0000 7feabeffd640 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55eecfcf8630 0x55eed0187eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.008+0000 7fea9effd640 10 client.27952 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:06.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.008+0000 7feabeffd640 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55eecfcf8630 0x55eed0187eb0 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9f68335666919e8 server_cookie=9023eef38dac0c48 in_seq=0 out_seq=0 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.009+0000 7fea9effd640 1 -- 172.21.15.27:0/2998130460 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55eed0072580 con 0x55eecfcf8630 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.009+0000 7fea9effd640 10 client.27952 handle_command_reply: tid=0 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.009+0000 7feaccc7a740 1 client.27952 shutdown 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 -- 172.21.15.27:0/2998130460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 msgr2=0x55eed017ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed017ef60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7feab002aa30 tx=0x7feab0005510 comp rx=0 tx=0).stop 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 -- 172.21.15.27:0/2998130460 shutdown_connections 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55eecfcf8630 0x55eed0187eb0 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eecfd64410 0x55eed0183a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55eecfd43730 0x55eed01814b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7fea9effd640 0 client.27952 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 --2- 172.21.15.27:0/2998130460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eecfd3b550 0x55eed017ef60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 -- 172.21.15.27:0/2998130460 >> 172.21.15.27:0/2998130460 conn(0x55eecfd58560 msgr2=0x55eed0104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 -- 172.21.15.27:0/2998130460 shutdown_connections 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 1 -- 172.21.15.27:0/2998130460 wait complete. 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.010+0000 7feaccc7a740 20 client.27952 trim_cache size 0 max 16384 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "1fed5ebc-3313-4f6f-88dd-4d1f68b21e7b", 2024-04-25T13:55:06.014 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:55:06.015 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:55:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 -- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feaa80afc80 msgr2=0x7feaa80b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feaa80afc80 0x7feaa80b2190 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7feab0006010 tx=0x7feab0005850 comp rx=0 tx=0).stop 2024-04-25T13:55:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 -- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0104d40 msgr2=0x7feac0113920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0104d40 0x7feac0113920 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7feab800cce0 tx=0x7feab80075f0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 -- 172.21.15.27:0/141936499 shutdown_connections 2024-04-25T13:55:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7feaa80afc80 0x7feaa80b2190 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feac0109a40 0x7feac0118830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac0104d40 0x7feac0113920 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 --2- 172.21.15.27:0/141936499 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac0103b00 0x7feac01133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 -- 172.21.15.27:0/141936499 >> 172.21.15.27:0/141936499 conn(0x7feac00fed10 msgr2=0x7feac0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 -- 172.21.15.27:0/141936499 shutdown_connections 2024-04-25T13:55:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.011+0000 7feac5dc5640 1 -- 172.21.15.27:0/141936499 wait complete. 2024-04-25T13:55:06.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:55:06.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:55:06.053 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:05 smithi027 ceph-mon[29140]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 MiB/s wr, 604 op/s 2024-04-25T13:55:06.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 -- 172.21.15.27:0/3333825560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f0103ee0 msgr2=0x7f97f01027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/3333825560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f0103ee0 0x7f97f01027c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009900 tx=0x7f97e002a410 comp rx=0 tx=0).stop 2024-04-25T13:55:06.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 -- 172.21.15.27:0/3333825560 shutdown_connections 2024-04-25T13:55:06.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/3333825560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f0108a10 0x7f97f010ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/3333825560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01043c0 0x7f97f0102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/3333825560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f0103ee0 0x7f97f01027c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 -- 172.21.15.27:0/3333825560 >> 172.21.15.27:0/3333825560 conn(0x7f97f00fd340 msgr2=0x7f97f00ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 -- 172.21.15.27:0/3333825560 shutdown_connections 2024-04-25T13:55:06.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.201+0000 7f97f5d7d640 1 -- 172.21.15.27:0/3333825560 wait complete. 2024-04-25T13:55:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.202+0000 7f97f5d7d640 1 Processor -- start 2024-04-25T13:55:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.202+0000 7f97f5d7d640 1 -- start start 2024-04-25T13:55:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.202+0000 7f97f5d7d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f0103ee0 0x7f97f010df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97f5d7d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f0108a10 0x7f97f010c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97f5d7d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f010cb50 0x7f97f010cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97ef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f0103ee0 0x7f97f010df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97ef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f0103ee0 0x7f97f010df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57824/0 (socket says 172.21.15.27:57824) 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97effff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f010cb50 0x7f97f010cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97ef7fe640 1 -- 172.21.15.27:0/2397879861 learned_addr learned my addr 172.21.15.27:0/2397879861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010f640 con 0x7f97f010cb50 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010f4c0 con 0x7f97f0103ee0 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.203+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010f7c0 con 0x7f97f0108a10 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ef7fe640 1 -- 172.21.15.27:0/2397879861 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f0108a10 msgr2=0x7f97f010c610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ef7fe640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f0108a10 0x7f97f010c610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ef7fe640 1 -- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f010cb50 msgr2=0x7f97f010cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ef7fe640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f010cb50 0x7f97f010cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ef7fe640 1 -- 172.21.15.27:0/2397879861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f019d960 con 0x7f97f0103ee0 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97effff640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f010cb50 0x7f97f010cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ef7fe640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f0103ee0 0x7f97f010df70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009850 tx=0x7f97e0003ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ecff9640 1 -- 172.21.15.27:0/2397879861 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e003f070 con 0x7f97f0103ee0 2024-04-25T13:55:06.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f01a8470 con 0x7f97f0103ee0 2024-04-25T13:55:06.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ecff9640 1 -- 172.21.15.27:0/2397879861 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97e0031d60 con 0x7f97f0103ee0 2024-04-25T13:55:06.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97ecff9640 1 -- 172.21.15.27:0/2397879861 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0035030 con 0x7f97f0103ee0 2024-04-25T13:55:06.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.204+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f01a8960 con 0x7f97f0103ee0 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97fcc2c740 1 Processor -- start 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97fcc2c740 1 -- start start 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444d1390 0x564c44872a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44872fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c44873c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564c447d7d80 con 0x564c444b06b0 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564c447918b0 con 0x564c444d1390 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564c447d3d40 con 0x564c444a84d0 2024-04-25T13:55:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97ecff9640 1 -- 172.21.15.27:0/2397879861 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f97e0042070 con 0x7f97f0103ee0 2024-04-25T13:55:06.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c44873c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444d1390 0x564c44872a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444d1390 0x564c44872a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57832/0 (socket says 172.21.15.27:57832) 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c44873c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41308/0 (socket says 172.21.15.27:41308) 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97effff640 1 -- 172.21.15.27:0/3280652953 learned_addr learned my addr 172.21.15.27:0/3280652953 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.208+0000 7f97ef7fe640 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44872fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97effff640 1 -- 172.21.15.27:0/3280652953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 msgr2=0x564c44872fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97effff640 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44872fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97effff640 1 -- 172.21.15.27:0/3280652953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 msgr2=0x564c44873c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97effff640 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c44873c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97effff640 1 -- 172.21.15.27:0/3280652953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c44762a40 con 0x564c444d1390 2024-04-25T13:55:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97ef7fe640 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44872fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:55:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97effff640 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444d1390 0x564c44872a60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f97e80117e0 tx=0x7f97e8011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=343ebe05b560b8a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97ecff9640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97d80afb20 0x7f97d80b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97d5ffb640 1 -- 172.21.15.27:0/3280652953 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8011310 con 0x564c444d1390 2024-04-25T13:55:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97d5ffb640 1 -- 172.21.15.27:0/3280652953 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97e8003e20 con 0x564c444d1390 2024-04-25T13:55:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.209+0000 7f97d5ffb640 1 -- 172.21.15.27:0/3280652953 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e80044d0 con 0x564c444d1390 2024-04-25T13:55:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.210+0000 7f97ecff9640 1 -- 172.21.15.27:0/2397879861 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f97e00ff290 con 0x7f97f0103ee0 2024-04-25T13:55:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.211+0000 7f97fcc2c740 1 -- 172.21.15.27:0/3280652953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444d1390 msgr2=0x564c44872a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.211+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444d1390 0x564c44872a60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f97e80117e0 tx=0x7f97e8011ca0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.212+0000 7f97fcc2c740 1 -- 172.21.15.27:0/3280652953 shutdown_connections 2024-04-25T13:55:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.212+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c44873c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.212+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44872fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.212+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/3280652953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444d1390 0x564c44872a60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.212+0000 7f97fcc2c740 1 -- 172.21.15.27:0/3280652953 >> 172.21.15.27:0/3280652953 conn(0x564c444c54e0 msgr2=0x564c4486e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.213+0000 7f97ef7fe640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97d80afb20 0x7f97d80b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.214+0000 7f97fcc2c740 1 -- 172.21.15.27:0/3280652953 shutdown_connections 2024-04-25T13:55:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.215+0000 7f97fcc2c740 1 -- 172.21.15.27:0/3280652953 wait complete. 2024-04-25T13:55:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.215+0000 7f97ef7fe640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97d80afb20 0x7f97d80b2030 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f97e00fcd60 tx=0x7f97e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.215+0000 7f97fcc2c740 1 -- start start 2024-04-25T13:55:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97fcc2c740 1 Processor -- start 2024-04-25T13:55:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444a84d0 0x564c4487c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c4487ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c44883dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x564c44874780 con 0x564c444b06b0 2024-04-25T13:55:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x564c4475ed10 con 0x564c444a84d0 2024-04-25T13:55:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x564c44874900 con 0x564c444d1390 2024-04-25T13:55:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97fcc2c740 10 client.0 _ll_register_callbacks cb 0x564c4478b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.216+0000 7f97effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444a84d0 0x564c4487c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97effff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c44883dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97effff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444a84d0 0x564c4487c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57846/0 (socket says 172.21.15.27:57846) 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97effff640 1 -- 172.21.15.27:0/2073764285 learned_addr learned my addr 172.21.15.27:0/2073764285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97eeffd640 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c4487ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97effff640 1 -- 172.21.15.27:0/2073764285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444a84d0 msgr2=0x564c4487c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97effff640 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444a84d0 0x564c4487c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97effff640 1 -- 172.21.15.27:0/2073764285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 msgr2=0x564c4487ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97effff640 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c4487ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.217+0000 7f97effff640 1 -- 172.21.15.27:0/2073764285 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c445b37d0 con 0x564c444d1390 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.218+0000 7f97effff640 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c44883dd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f97e800fd10 tx=0x7f97e8009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b839f36ceaaf98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.218+0000 7f97ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.218+0000 7f97ceffd640 1 -- 172.21.15.27:0/2073764285 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8022040 con 0x564c444d1390 2024-04-25T13:55:06.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.218+0000 7f97fcc2c740 1 -- 172.21.15.27:0/2073764285 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97e8003e20 con 0x564c444d1390 2024-04-25T13:55:06.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.218+0000 7f97ceffd640 1 -- 172.21.15.27:0/2073764285 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f97e8004c40 con 0x564c444d1390 2024-04-25T13:55:06.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.218+0000 7f97ceffd640 1 -- 172.21.15.27:0/2073764285 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f97e8032050 con 0x564c444d1390 2024-04-25T13:55:06.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.218+0000 7f97fcc2c740 10 client.27961 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:06.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.218+0000 7f97fcc2c740 1 -- 172.21.15.27:0/2073764285 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564c444e93f0 con 0x564c444d1390 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97ceffd640 1 -- 172.21.15.27:0/2073764285 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97e8003e20 con 0x564c444d1390 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97ceffd640 1 -- 172.21.15.27:0/2073764285 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f97e8028070 con 0x564c444d1390 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97fcc2c740 10 client.27961 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97fcc2c740 10 client.27961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97fcc2c740 20 client.27961 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564c4448ff60 0x564c4488a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97fcc2c740 4 client.27961 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97fcc2c740 1 -- 172.21.15.27:0/2073764285 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c447e5990 con 0x564c4448ff60 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.219+0000 7f97eeffd640 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564c4448ff60 0x564c4488a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.220+0000 7f97ceffd640 10 client.27961 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.220+0000 7f97eeffd640 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564c4448ff60 0x564c4488a000 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a59eb03a14157d7 server_cookie=b27e4c97b9b6b038 in_seq=0 out_seq=0 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97ceffd640 1 -- 172.21.15.27:0/2073764285 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564c447e5990 con 0x564c4448ff60 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97ceffd640 10 client.27961 handle_command_reply: tid=0 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 client.27961 shutdown 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 -- 172.21.15.27:0/2073764285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 msgr2=0x564c44883dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c44883dd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f97e800fd10 tx=0x7f97e8009500 comp rx=0 tx=0).stop 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 -- 172.21.15.27:0/2073764285 shutdown_connections 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564c4448ff60 0x564c4488a000 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c44883dd0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444b06b0 0x564c4487ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/2073764285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444a84d0 0x564c4487c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97ceffd640 0 client.27961 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 -- 172.21.15.27:0/2073764285 >> 172.21.15.27:0/2073764285 conn(0x564c444c54e0 msgr2=0x564c44871180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 -- 172.21.15.27:0/2073764285 shutdown_connections 2024-04-25T13:55:06.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.221+0000 7f97fcc2c740 1 -- 172.21.15.27:0/2073764285 wait complete. 2024-04-25T13:55:06.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.222+0000 7f97fcc2c740 20 client.27961 trim_cache size 0 max 16384 2024-04-25T13:55:06.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.248+0000 7f97fcc2c740 1 Processor -- start 2024-04-25T13:55:06.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.248+0000 7f97fcc2c740 1 -- start start 2024-04-25T13:55:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444a84d0 0x564c4487b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c4486e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c4486d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01af580 con 0x564c444a84d0 2024-04-25T13:55:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01b1e00 con 0x564c444b06b0 2024-04-25T13:55:06.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01afbf0 con 0x564c444d1390 2024-04-25T13:55:06.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444a84d0 0x564c4487b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444a84d0 0x564c4487b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41332/0 (socket says 172.21.15.27:41332) 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 -- 172.21.15.27:0/650757869 learned_addr learned my addr 172.21.15.27:0/650757869 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97eeffd640 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c4486e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c4486d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 -- 172.21.15.27:0/650757869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 msgr2=0x564c4486d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c4486d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 -- 172.21.15.27:0/650757869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 msgr2=0x564c4486e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c4486e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.249+0000 7f97effff640 1 -- 172.21.15.27:0/650757869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c444a90f0 con 0x564c444a84d0 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.250+0000 7f97effff640 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444a84d0 0x564c4487b930 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f97e8006010 tx=0x7f97e8011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2658b1e27a70a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.250+0000 7f97d57fa640 1 -- 172.21.15.27:0/650757869 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e801a640 con 0x564c444a84d0 2024-04-25T13:55:06.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.250+0000 7f97d57fa640 1 -- 172.21.15.27:0/650757869 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97e801a7e0 con 0x564c444a84d0 2024-04-25T13:55:06.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.250+0000 7f97d57fa640 1 -- 172.21.15.27:0/650757869 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8010550 con 0x564c444a84d0 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 -- 172.21.15.27:0/650757869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444a84d0 msgr2=0x564c4487b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444a84d0 0x564c4487b930 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f97e8006010 tx=0x7f97e8011ca0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 -- 172.21.15.27:0/650757869 shutdown_connections 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444d1390 0x564c4486d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c4486e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/650757869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444a84d0 0x564c4487b930 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 -- 172.21.15.27:0/650757869 >> 172.21.15.27:0/650757869 conn(0x564c444c54e0 msgr2=0x564c4486c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 -- 172.21.15.27:0/650757869 shutdown_connections 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 -- 172.21.15.27:0/650757869 wait complete. 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.251+0000 7f97fcc2c740 1 -- start start 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97fcc2c740 1 Processor -- start 2024-04-25T13:55:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44881ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c44882020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97fcc2c740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444d1390 0x564c44882f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01b02c0 con 0x564c444d1390 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01b1730 con 0x564c444b06b0 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97fcc2c740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f97f01b0990 con 0x564c444a84d0 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97fcc2c740 10 client.0 _ll_register_callbacks cb 0x7f97e80116e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c44882020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c44882020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:57878/0 (socket says 172.21.15.27:57878) 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97eeffd640 1 -- 172.21.15.27:0/1547573362 learned_addr learned my addr 172.21.15.27:0/1547573362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.252+0000 7f97effff640 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44881ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97effff640 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444d1390 0x564c44882f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97eeffd640 1 -- 172.21.15.27:0/1547573362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 msgr2=0x564c44881ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97eeffd640 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44881ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97eeffd640 1 -- 172.21.15.27:0/1547573362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444d1390 msgr2=0x564c44882f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97eeffd640 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444d1390 0x564c44882f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97eeffd640 1 -- 172.21.15.27:0/1547573362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c447984b0 con 0x564c444b06b0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97effff640 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444d1390 0x564c44882f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97eeffd640 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c44882020 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x564c4488a760 tx=0x7f97e401c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91da4ffee00cfddd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97ceffd640 1 -- 172.21.15.27:0/1547573362 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4004230 con 0x564c444b06b0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97ceffd640 1 -- 172.21.15.27:0/1547573362 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f97e40043d0 con 0x564c444b06b0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.253+0000 7f97fcc2c740 1 -- 172.21.15.27:0/1547573362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97e801a7e0 con 0x564c444b06b0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97ceffd640 1 -- 172.21.15.27:0/1547573362 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f97e402b070 con 0x564c444b06b0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97fcc2c740 10 client.27908 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97fcc2c740 1 -- 172.21.15.27:0/1547573362 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x564c44884680 con 0x564c444b06b0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97ceffd640 1 -- 172.21.15.27:0/1547573362 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97e801a7e0 con 0x564c444b06b0 2024-04-25T13:55:06.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97ceffd640 1 -- 172.21.15.27:0/1547573362 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f97e4024410 con 0x564c444b06b0 2024-04-25T13:55:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97fcc2c740 10 client.27908 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97fcc2c740 10 client.27908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97fcc2c740 20 client.27908 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564c444653a0 0x564c448f4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97fcc2c740 4 client.27908 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:55:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97fcc2c740 1 -- 172.21.15.27:0/1547573362 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564c447df800 con 0x564c444653a0 2024-04-25T13:55:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.254+0000 7f97effff640 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564c444653a0 0x564c448f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.256+0000 7f97effff640 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564c444653a0 0x564c448f4840 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d0c243bbc0d9d34 server_cookie=32ea9b4a4548ba90 in_seq=0 out_seq=0 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.256+0000 7f97ceffd640 10 client.27908 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.257+0000 7f97ceffd640 1 -- 172.21.15.27:0/1547573362 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x564c447df800 con 0x564c444653a0 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.257+0000 7f97ceffd640 10 client.27908 handle_command_reply: tid=0 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.257+0000 7f97fcc2c740 1 client.27908 shutdown 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 -- 172.21.15.27:0/1547573362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 msgr2=0x564c44882020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c44882020 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x564c4488a760 tx=0x7f97e401c6e0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 -- 172.21.15.27:0/1547573362 shutdown_connections 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x564c444653a0 0x564c448f4840 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x564c444d1390 0x564c44882f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564c444b06b0 0x564c44882020 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 --2- 172.21.15.27:0/1547573362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564c444a84d0 0x564c44881ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97ceffd640 0 client.27908 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 -- 172.21.15.27:0/1547573362 >> 172.21.15.27:0/1547573362 conn(0x564c444c54e0 msgr2=0x564c4487b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 -- 172.21.15.27:0/1547573362 shutdown_connections 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 1 -- 172.21.15.27:0/1547573362 wait complete. 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.258+0000 7f97fcc2c740 20 client.27908 trim_cache size 0 max 16384 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (88 inodes in the stack)", 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stdout: "1fed5ebc-3313-4f6f-88dd-4d1f68b21e7b": { 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "1fed5ebc-3313-4f6f-88dd-4d1f68b21e7b", 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:55:06.260 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:55:06.261 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:55:06.261 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97d80afb20 msgr2=0x7f97d80b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97d80afb20 0x7f97d80b2030 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f97e00fcd60 tx=0x7f97e0005fd0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f0103ee0 msgr2=0x7f97f010df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f0103ee0 0x7f97f010df70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009850 tx=0x7f97e0003ed0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 shutdown_connections 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f97d80afb20 0x7f97d80b2030 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97f010cb50 0x7f97f010cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97f0108a10 0x7f97f010c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 --2- 172.21.15.27:0/2397879861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f0103ee0 0x7f97f010df70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 >> 172.21.15.27:0/2397879861 conn(0x7f97f00fd340 msgr2=0x7f97f010a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.259+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 shutdown_connections 2024-04-25T13:55:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:06.260+0000 7f97f5d7d640 1 -- 172.21.15.27:0/2397879861 wait complete. 2024-04-25T13:55:06.274 INFO:tasks.cephfs.filesystem:scrub status for tag:1fed5ebc-3313-4f6f-88dd-4d1f68b21e7b - {'path': '/', 'tag': '1fed5ebc-3313-4f6f-88dd-4d1f68b21e7b', 'options': 'recursive,force'} 2024-04-25T13:55:06.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:05 smithi122 ceph-mon[32843]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 MiB/s wr, 604 op/s 2024-04-25T13:55:06.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:05 smithi045 ceph-mon[32731]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 MiB/s wr, 604 op/s 2024-04-25T13:55:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 840.0 s, 1386.4 tps, lat 72.507 ms stddev 85.728 2024-04-25T13:55:08.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:07 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:55:08.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:07 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:08.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:07 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:55:08.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:07 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:08.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:07 smithi027 ceph-mon[29140]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 678 op/s 2024-04-25T13:55:08.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:07 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:55:08.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:07 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:08.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:07 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:55:08.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:07 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:08.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:07 smithi122 ceph-mon[32843]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 678 op/s 2024-04-25T13:55:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:07 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:55:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:07 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:07 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:55:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:07 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:08.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:07 smithi045 ceph-mon[32731]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 678 op/s 2024-04-25T13:55:10.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:09 smithi027 ceph-mon[29140]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 14 MiB/s wr, 730 op/s 2024-04-25T13:55:10.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:09 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:55:10.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:09 smithi122 ceph-mon[32843]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 14 MiB/s wr, 730 op/s 2024-04-25T13:55:10.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:09 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:55:10.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:09 smithi045 ceph-mon[32731]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 14 MiB/s wr, 730 op/s 2024-04-25T13:55:10.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:09 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:55:12.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:11 smithi027 ceph-mon[29140]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 14 MiB/s wr, 699 op/s 2024-04-25T13:55:12.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:11 smithi122 ceph-mon[32843]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 14 MiB/s wr, 699 op/s 2024-04-25T13:55:12.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:11 smithi045 ceph-mon[32731]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 14 MiB/s wr, 699 op/s 2024-04-25T13:55:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 845.0 s, 1682.8 tps, lat 59.237 ms stddev 69.724 2024-04-25T13:55:14.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:13 smithi027 ceph-mon[29140]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 16 MiB/s wr, 812 op/s 2024-04-25T13:55:14.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:13 smithi122 ceph-mon[32843]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 16 MiB/s wr, 812 op/s 2024-04-25T13:55:14.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:13 smithi045 ceph-mon[32731]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 16 MiB/s wr, 812 op/s 2024-04-25T13:55:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.762+0000 7f9cb018f640 1 -- 172.21.15.27:0/1601643039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca810b9f0 msgr2=0x7f9ca810de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.762+0000 7f9cb018f640 1 --2- 172.21.15.27:0/1601643039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca810b9f0 0x7f9ca810de10 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009900 tx=0x7f9c9802a410 comp rx=0 tx=0).stop 2024-04-25T13:55:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.769+0000 7f9cb018f640 1 -- 172.21.15.27:0/1601643039 shutdown_connections 2024-04-25T13:55:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.769+0000 7f9cb018f640 1 --2- 172.21.15.27:0/1601643039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca8110f50 0x7f9ca8113460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.769+0000 7f9cb018f640 1 --2- 172.21.15.27:0/1601643039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca810e3e0 0x7f9ca81108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.769+0000 7f9cb018f640 1 --2- 172.21.15.27:0/1601643039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca810b9f0 0x7f9ca810de10 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.769+0000 7f9cb018f640 1 -- 172.21.15.27:0/1601643039 >> 172.21.15.27:0/1601643039 conn(0x7f9ca8070960 msgr2=0x7f9ca8072e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.780+0000 7f9cb018f640 1 -- 172.21.15.27:0/1601643039 shutdown_connections 2024-04-25T13:55:15.783 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.784+0000 7f9cb018f640 1 -- 172.21.15.27:0/1601643039 wait complete. 2024-04-25T13:55:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.785+0000 7f9cb018f640 1 Processor -- start 2024-04-25T13:55:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.789+0000 7f9cb018f640 1 -- start start 2024-04-25T13:55:15.788 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.789+0000 7f9cb018f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca810b9f0 0x7f9ca81a9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.789+0000 7f9cb018f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca810e3e0 0x7f9ca81a9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.793+0000 7f9cadf2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca810b9f0 0x7f9ca81a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.793+0000 7f9cadf2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca810b9f0 0x7f9ca81a9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58188/0 (socket says 172.21.15.27:58188) 2024-04-25T13:55:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.796+0000 7f9cad72a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca810e3e0 0x7f9ca81a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.796+0000 7f9cad72a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca810e3e0 0x7f9ca81a9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:33546/0 (socket says 172.21.15.27:33546) 2024-04-25T13:55:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.797+0000 7f9cb018f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca8110f50 0x7f9ca81b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.798 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.797+0000 7f9cadf2b640 1 -- 172.21.15.27:0/2605689282 learned_addr learned my addr 172.21.15.27:0/2605689282 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.797+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8116dd0 con 0x7f9ca8110f50 2024-04-25T13:55:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.797+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8116c50 con 0x7f9ca810e3e0 2024-04-25T13:55:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.797+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8116f50 con 0x7f9ca810b9f0 2024-04-25T13:55:15.799 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.798+0000 7f9cae72c640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca8110f50 0x7f9ca81b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.805+0000 7f9cad72a640 1 -- 172.21.15.27:0/2605689282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca810b9f0 msgr2=0x7f9ca81a9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.805+0000 7f9cad72a640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca810b9f0 0x7f9ca81a9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.805+0000 7f9cad72a640 1 -- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca8110f50 msgr2=0x7f9ca81b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.805+0000 7f9cad72a640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca8110f50 0x7f9ca81b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.805+0000 7f9cad72a640 1 -- 172.21.15.27:0/2605689282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca81b0c60 con 0x7f9ca810e3e0 2024-04-25T13:55:15.809 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.809+0000 7f9cad72a640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca810e3e0 0x7f9ca81a9850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4005fd0 tx=0x7f9ca400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:15.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.813+0000 7f9c9effd640 1 -- 172.21.15.27:0/2605689282 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4009c30 con 0x7f9ca810e3e0 2024-04-25T13:55:15.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.813+0000 7f9c9effd640 1 -- 172.21.15.27:0/2605689282 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca4009dd0 con 0x7f9ca810e3e0 2024-04-25T13:55:15.813 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.813+0000 7f9c9effd640 1 -- 172.21.15.27:0/2605689282 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4010460 con 0x7f9ca810e3e0 2024-04-25T13:55:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.814+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca81b0f50 con 0x7f9ca810e3e0 2024-04-25T13:55:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.814+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca81b13c0 con 0x7f9ca810e3e0 2024-04-25T13:55:15.816 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.817+0000 7f9cb7040740 1 Processor -- start 2024-04-25T13:55:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.818+0000 7f9cb7040740 1 -- start start 2024-04-25T13:55:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.818+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded9271b0 0x55adedcc7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.818+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedcc7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.818+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9064d0 0x55adedccedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.818+0000 7f9cb7040740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55adedc2d460 con 0x55aded9271b0 2024-04-25T13:55:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.818+0000 7f9cb7040740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55adedbe7590 con 0x55aded9064d0 2024-04-25T13:55:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.818+0000 7f9cb7040740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55adedc29420 con 0x55aded8fe2f0 2024-04-25T13:55:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cadf2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedcc7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cadf2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedcc7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58200/0 (socket says 172.21.15.27:58200) 2024-04-25T13:55:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cadf2b640 1 -- 172.21.15.27:0/2087633207 learned_addr learned my addr 172.21.15.27:0/2087633207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cad72a640 1 --2- 172.21.15.27:0/2087633207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9064d0 0x55adedccedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9c9effd640 1 -- 172.21.15.27:0/2605689282 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9ca4003e80 con 0x7f9ca810e3e0 2024-04-25T13:55:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cadf2b640 1 -- 172.21.15.27:0/2087633207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9064d0 msgr2=0x55adedccedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/2087633207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9064d0 0x55adedccedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cadf2b640 1 -- 172.21.15.27:0/2087633207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded9271b0 msgr2=0x55adedcc7970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:55:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/2087633207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded9271b0 0x55adedcc7970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.821+0000 7f9cadf2b640 1 -- 172.21.15.27:0/2087633207 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adedbf40d0 con 0x55aded8fe2f0 2024-04-25T13:55:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.823+0000 7f9c9effd640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c940afcb0 0x7f9c940b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.823+0000 7f9c9effd640 1 -- 172.21.15.27:0/2605689282 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9ca40de820 con 0x7f9ca810e3e0 2024-04-25T13:55:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.824+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/2087633207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedcc7ef0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9c98006010 tx=0x7f9c98031bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84fb30fd7ab035ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.827+0000 7f9c93fff640 1 -- 172.21.15.27:0/2087633207 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98035030 con 0x55aded8fe2f0 2024-04-25T13:55:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.828+0000 7f9c93fff640 1 -- 172.21.15.27:0/2087633207 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c98040070 con 0x55aded8fe2f0 2024-04-25T13:55:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.828+0000 7f9c93fff640 1 -- 172.21.15.27:0/2087633207 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98003a90 con 0x55aded8fe2f0 2024-04-25T13:55:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.831+0000 7f9cae72c640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c940afcb0 0x7f9c940b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.832+0000 7f9cb7040740 1 -- 172.21.15.27:0/2087633207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 msgr2=0x55adedcc7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.832+0000 7f9cb7040740 1 --2- 172.21.15.27:0/2087633207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedcc7ef0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9c98006010 tx=0x7f9c98031bc0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.835+0000 7f9cb7040740 1 -- 172.21.15.27:0/2087633207 shutdown_connections 2024-04-25T13:55:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.835+0000 7f9cb7040740 1 --2- 172.21.15.27:0/2087633207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9064d0 0x55adedccedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.835+0000 7f9cb7040740 1 --2- 172.21.15.27:0/2087633207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedcc7ef0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.835+0000 7f9cb7040740 1 --2- 172.21.15.27:0/2087633207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded9271b0 0x55adedcc7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.835+0000 7f9cb7040740 1 -- 172.21.15.27:0/2087633207 >> 172.21.15.27:0/2087633207 conn(0x55aded91b300 msgr2=0x55adedcc3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.843+0000 7f9cb7040740 1 -- 172.21.15.27:0/2087633207 shutdown_connections 2024-04-25T13:55:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.844+0000 7f9cae72c640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c940afcb0 0x7f9c940b21c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x55adedcc8b40 tx=0x7f9ca0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.844+0000 7f9cb7040740 1 -- 172.21.15.27:0/2087633207 wait complete. 2024-04-25T13:55:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.844+0000 7f9cb7040740 1 -- start start 2024-04-25T13:55:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.845+0000 7f9cb7040740 1 Processor -- start 2024-04-25T13:55:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedce26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded9064d0 0x55adedce2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cb7040740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55adedccf7e0 con 0x55aded9064d0 2024-04-25T13:55:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cb7040740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55adedbb4b60 con 0x55aded9271b0 2024-04-25T13:55:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cb7040740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55adedccf960 con 0x55aded8fe2f0 2024-04-25T13:55:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cb7040740 10 client.0 _ll_register_callbacks cb 0x55adedbd92b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cadf2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedce26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cadf2b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedce26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58206/0 (socket says 172.21.15.27:58206) 2024-04-25T13:55:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.846+0000 7f9cadf2b640 1 -- 172.21.15.27:0/953981552 learned_addr learned my addr 172.21.15.27:0/953981552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.847+0000 7f9cae72c640 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.847 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.848+0000 7f9cae72c640 1 -- 172.21.15.27:0/953981552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 msgr2=0x55adedce26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.848+0000 7f9cae72c640 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedce26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.848+0000 7f9cae72c640 1 -- 172.21.15.27:0/953981552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded9064d0 msgr2=0x55adedce2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.848+0000 7f9cae72c640 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded9064d0 0x55adedce2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.848+0000 7f9cae72c640 1 -- 172.21.15.27:0/953981552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aded93f210 con 0x55aded9271b0 2024-04-25T13:55:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.848+0000 7f9cae72c640 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce9b50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005fd0 tx=0x7f9ca0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c666da89fb9325e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.848+0000 7f9c917fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.851+0000 7f9c917fa640 1 -- 172.21.15.27:0/953981552 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0004d20 con 0x55aded9271b0 2024-04-25T13:55:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.851+0000 7f9c917fa640 1 -- 172.21.15.27:0/953981552 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9ca000eb50 con 0x55aded9271b0 2024-04-25T13:55:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.851+0000 7f9cb7040740 1 -- 172.21.15.27:0/953981552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c98040070 con 0x55aded9271b0 2024-04-25T13:55:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.851+0000 7f9c917fa640 1 -- 172.21.15.27:0/953981552 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9ca000ed50 con 0x55aded9271b0 2024-04-25T13:55:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.851+0000 7f9cb7040740 10 client.27917 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.851+0000 7f9cb7040740 1 -- 172.21.15.27:0/953981552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55adeda0b620 con 0x55aded9271b0 2024-04-25T13:55:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.852+0000 7f9c917fa640 1 -- 172.21.15.27:0/953981552 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c98040070 con 0x55aded9271b0 2024-04-25T13:55:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.852+0000 7f9c917fa640 1 -- 172.21.15.27:0/953981552 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9ca00264b0 con 0x55aded9271b0 2024-04-25T13:55:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.852+0000 7f9cb7040740 10 client.27917 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.853+0000 7f9cb7040740 10 client.27917 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.853+0000 7f9cb7040740 20 client.27917 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.853+0000 7f9cb7040740 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aded8e5d80 0x55adedcf0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.853+0000 7f9cb7040740 4 client.27917 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.853+0000 7f9cb7040740 1 -- 172.21.15.27:0/953981552 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adedc3b070 con 0x55aded8e5d80 2024-04-25T13:55:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.853+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aded8e5d80 0x55adedcf0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.854+0000 7f9c917fa640 10 client.27917 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.854+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aded8e5d80 0x55adedcf0570 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b1d3490e8581470 server_cookie=82a5e19fa66a2829 in_seq=0 out_seq=0 2024-04-25T13:55:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9c917fa640 1 -- 172.21.15.27:0/953981552 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55adedc3b070 con 0x55aded8e5d80 2024-04-25T13:55:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9c917fa640 10 client.27917 handle_command_reply: tid=0 2024-04-25T13:55:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 client.27917 shutdown 2024-04-25T13:55:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 -- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 msgr2=0x55adedce9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce9b50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005fd0 tx=0x7f9ca0004ab0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 -- 172.21.15.27:0/953981552 shutdown_connections 2024-04-25T13:55:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aded8e5d80 0x55adedcf0570 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce9b50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded9064d0 0x55adedce2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 --2- 172.21.15.27:0/953981552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded8fe2f0 0x55adedce26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 -- 172.21.15.27:0/953981552 >> 172.21.15.27:0/953981552 conn(0x55aded91b300 msgr2=0x55adedcc6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9c917fa640 0 client.27917 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 -- 172.21.15.27:0/953981552 shutdown_connections 2024-04-25T13:55:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.855+0000 7f9cb7040740 1 -- 172.21.15.27:0/953981552 wait complete. 2024-04-25T13:55:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.856+0000 7f9cb7040740 20 client.27917 trim_cache size 0 max 16384 2024-04-25T13:55:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.879+0000 7f9cb7040740 1 Processor -- start 2024-04-25T13:55:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.880+0000 7f9cb7040740 1 -- start start 2024-04-25T13:55:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.880+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedccfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.881+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.881+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedcd02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.881+0000 7f9cb7040740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81a72d0 con 0x55aded8fe2f0 2024-04-25T13:55:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.881+0000 7f9cb7040740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81b2110 con 0x55aded9271b0 2024-04-25T13:55:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.881+0000 7f9cb7040740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81a78f0 con 0x55aded9064d0 2024-04-25T13:55:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.881+0000 7f9cad72a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.881+0000 7f9cad72a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58210/0 (socket says 172.21.15.27:58210) 2024-04-25T13:55:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.881+0000 7f9cad72a640 1 -- 172.21.15.27:0/357835096 learned_addr learned my addr 172.21.15.27:0/357835096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.882+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/357835096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedccfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.882+0000 7f9cad72a640 1 -- 172.21.15.27:0/357835096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 msgr2=0x55adedcd02a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:55:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.882+0000 7f9cad72a640 1 --2- 172.21.15.27:0/357835096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedcd02a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.882+0000 7f9cad72a640 1 -- 172.21.15.27:0/357835096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 msgr2=0x55adedccfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.882+0000 7f9cad72a640 1 --2- 172.21.15.27:0/357835096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedccfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.882+0000 7f9cad72a640 1 -- 172.21.15.27:0/357835096 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adedcbe710 con 0x55aded9064d0 2024-04-25T13:55:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.882+0000 7f9cad72a640 1 --2- 172.21.15.27:0/357835096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce9a70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40a3fc0 tx=0x7f9ca40a67a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3651941172854fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.882+0000 7f9c937fe640 1 -- 172.21.15.27:0/357835096 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40a7450 con 0x55aded9064d0 2024-04-25T13:55:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.883+0000 7f9c937fe640 1 -- 172.21.15.27:0/357835096 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca40a6d50 con 0x55aded9064d0 2024-04-25T13:55:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.883+0000 7f9c937fe640 1 -- 172.21.15.27:0/357835096 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40043d0 con 0x55aded9064d0 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.883+0000 7f9cb7040740 1 -- 172.21.15.27:0/357835096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 msgr2=0x55adedce9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.883+0000 7f9cb7040740 1 --2- 172.21.15.27:0/357835096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce9a70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40a3fc0 tx=0x7f9ca40a67a0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.883+0000 7f9cb7040740 1 -- 172.21.15.27:0/357835096 shutdown_connections 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.884+0000 7f9cb7040740 1 --2- 172.21.15.27:0/357835096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedcd02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.884+0000 7f9cb7040740 1 --2- 172.21.15.27:0/357835096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce9a70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.884+0000 7f9cb7040740 1 --2- 172.21.15.27:0/357835096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedccfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.884+0000 7f9cb7040740 1 -- 172.21.15.27:0/357835096 >> 172.21.15.27:0/357835096 conn(0x55aded91b300 msgr2=0x55adedcc7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.884+0000 7f9cb7040740 1 -- 172.21.15.27:0/357835096 shutdown_connections 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.884+0000 7f9cb7040740 1 -- 172.21.15.27:0/357835096 wait complete. 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.884+0000 7f9cb7040740 1 -- start start 2024-04-25T13:55:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.884+0000 7f9cb7040740 1 Processor -- start 2024-04-25T13:55:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedce99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cb7040740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cb7040740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81a7fc0 con 0x55aded8fe2f0 2024-04-25T13:55:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cadf2b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedce99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cad72a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cadf2b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedce99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58894/0 (socket says 172.21.15.27:58894) 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cadf2b640 1 -- 172.21.15.27:0/185321494 learned_addr learned my addr 172.21.15.27:0/185321494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cae72c640 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81b1a40 con 0x55aded9271b0 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.885+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81a8690 con 0x55aded9064d0 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cb7040740 10 client.0 _ll_register_callbacks cb 0x7f9ca0009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cad72a640 1 -- 172.21.15.27:0/185321494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 msgr2=0x55adedce6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cad72a640 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cad72a640 1 -- 172.21.15.27:0/185321494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 msgr2=0x55adedce99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cad72a640 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedce99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cad72a640 1 -- 172.21.15.27:0/185321494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adedd56910 con 0x55aded9064d0 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedce99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cad72a640 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce6530 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40a9000 tx=0x7f9ca40a6260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c96638c6869d5bf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9c917fa640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9c917fa640 1 -- 172.21.15.27:0/185321494 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40c5070 con 0x55aded9064d0 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9c917fa640 1 -- 172.21.15.27:0/185321494 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9ca40a3340 con 0x55aded9064d0 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ca40a6d50 con 0x55aded9064d0 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9c917fa640 1 -- 172.21.15.27:0/185321494 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9ca40b5020 con 0x55aded9064d0 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.886+0000 7f9cb7040740 10 client.27985 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55adedd57b10 con 0x55aded9064d0 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9c917fa640 1 -- 172.21.15.27:0/185321494 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca40a6d50 con 0x55aded9064d0 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9c917fa640 1 -- 172.21.15.27:0/185321494 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9ca40c0020 con 0x55aded9064d0 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9cb7040740 10 client.27985 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9cb7040740 10 client.27985 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9cb7040740 20 client.27985 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9cb7040740 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aded8bb1c0 0x55adedd5a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9cb7040740 4 client.27985 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:55:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55adedc34ee0 con 0x55aded8bb1c0 2024-04-25T13:55:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.887+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aded8bb1c0 0x55adedd5a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.888+0000 7f9c917fa640 10 client.27985 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.888+0000 7f9cadf2b640 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aded8bb1c0 0x55adedd5a480 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3db20fca738df29e server_cookie=c072a9f4ce209411 in_seq=0 out_seq=0 2024-04-25T13:55:15.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.908+0000 7f9c917fa640 1 -- 172.21.15.27:0/185321494 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55adedc34ee0 con 0x55aded8bb1c0 2024-04-25T13:55:15.908 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.908+0000 7f9c917fa640 10 client.27985 handle_command_reply: tid=0 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.908+0000 7f9cb7040740 1 client.27985 shutdown 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.908+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 msgr2=0x55adedce6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.908+0000 7f9cb7040740 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce6530 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40a9000 tx=0x7f9ca40a6260 comp rx=0 tx=0).stop 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 shutdown_connections 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55aded8bb1c0 0x55adedd5a480 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aded9271b0 0x55adedce6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55aded9064d0 0x55adedce6530 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 1 --2- 172.21.15.27:0/185321494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55aded8fe2f0 0x55adedce99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 >> 172.21.15.27:0/185321494 conn(0x55aded91b300 msgr2=0x55adedcc78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9c917fa640 0 client.27985 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 shutdown_connections 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 1 -- 172.21.15.27:0/185321494 wait complete. 2024-04-25T13:55:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb7040740 20 client.27985 trim_cache size 0 max 16384 2024-04-25T13:55:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:55:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:55:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:55:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:55:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c940afcb0 msgr2=0x7f9c940b21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb018f640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c940afcb0 0x7f9c940b21c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x55adedcc8b40 tx=0x7f9ca0009500 comp rx=0 tx=0).stop 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca810e3e0 msgr2=0x7f9ca81a9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.909+0000 7f9cb018f640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca810e3e0 0x7f9ca81a9850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4005fd0 tx=0x7f9ca400dde0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.910+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 shutdown_connections 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.910+0000 7f9cb018f640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9c940afcb0 0x7f9c940b21c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.910+0000 7f9cb018f640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ca8110f50 0x7f9ca81b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.910+0000 7f9cb018f640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ca810e3e0 0x7f9ca81a9850 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.910+0000 7f9cb018f640 1 --2- 172.21.15.27:0/2605689282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ca810b9f0 0x7f9ca81a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.910+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 >> 172.21.15.27:0/2605689282 conn(0x7f9ca8070960 msgr2=0x7f9ca810be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.910+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 shutdown_connections 2024-04-25T13:55:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:15.910+0000 7f9cb018f640 1 -- 172.21.15.27:0/2605689282 wait complete. 2024-04-25T13:55:16.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:15 smithi027 ceph-mon[29140]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 636 op/s 2024-04-25T13:55:16.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:15 smithi122 ceph-mon[32843]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 636 op/s 2024-04-25T13:55:16.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:15 smithi045 ceph-mon[32731]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 636 op/s 2024-04-25T13:55:17.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 850.0 s, 1589.8 tps, lat 62.572 ms stddev 70.888 2024-04-25T13:55:18.129 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:17 smithi045 ceph-mon[32731]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 12 MiB/s wr, 696 op/s 2024-04-25T13:55:18.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:17 smithi027 ceph-mon[29140]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 12 MiB/s wr, 696 op/s 2024-04-25T13:55:18.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:17 smithi122 ceph-mon[32843]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 12 MiB/s wr, 696 op/s 2024-04-25T13:55:20.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:19 smithi027 ceph-mon[29140]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 725 op/s 2024-04-25T13:55:20.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:19 smithi122 ceph-mon[32843]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 725 op/s 2024-04-25T13:55:20.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:19 smithi045 ceph-mon[32731]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 725 op/s 2024-04-25T13:55:22.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:21 smithi027 ceph-mon[29140]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 636 op/s 2024-04-25T13:55:22.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:21 smithi122 ceph-mon[32843]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 636 op/s 2024-04-25T13:55:22.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:21 smithi045 ceph-mon[32731]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 636 op/s 2024-04-25T13:55:22.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 855.0 s, 1425.4 tps, lat 70.724 ms stddev 77.157 2024-04-25T13:55:24.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:23 smithi027 ceph-mon[29140]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 695 op/s 2024-04-25T13:55:24.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:23 smithi122 ceph-mon[32843]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 695 op/s 2024-04-25T13:55:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:23 smithi045 ceph-mon[32731]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 695 op/s 2024-04-25T13:55:26.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:25 smithi027 ceph-mon[29140]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 545 op/s 2024-04-25T13:55:26.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:25 smithi122 ceph-mon[32843]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 545 op/s 2024-04-25T13:55:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:25 smithi045 ceph-mon[32731]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 545 op/s 2024-04-25T13:55:27.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 860.0 s, 1159.4 tps, lat 86.335 ms stddev 139.164 2024-04-25T13:55:28.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:27 smithi027 ceph-mon[29140]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 882 op/s 2024-04-25T13:55:28.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:27 smithi122 ceph-mon[32843]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 882 op/s 2024-04-25T13:55:28.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:27 smithi045 ceph-mon[32731]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 882 op/s 2024-04-25T13:55:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.070+0000 7fca911ee640 1 -- 172.21.15.27:0/2525626394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c10a420 msgr2=0x7fca8c10ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.071 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.070+0000 7fca911ee640 1 --2- 172.21.15.27:0/2525626394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c10a420 0x7fca8c10ae10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fca84009ca0 tx=0x7fca8402a8b0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.072+0000 7fca911ee640 1 -- 172.21.15.27:0/2525626394 shutdown_connections 2024-04-25T13:55:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.072+0000 7fca911ee640 1 --2- 172.21.15.27:0/2525626394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca8c110070 0x7fca8c110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.072+0000 7fca911ee640 1 --2- 172.21.15.27:0/2525626394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca8c10b350 0x7fca8c10b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.072+0000 7fca911ee640 1 --2- 172.21.15.27:0/2525626394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c10a420 0x7fca8c10ae10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.072+0000 7fca911ee640 1 -- 172.21.15.27:0/2525626394 >> 172.21.15.27:0/2525626394 conn(0x7fca8c070540 msgr2=0x7fca8c0729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:29.080 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.080+0000 7fca911ee640 1 -- 172.21.15.27:0/2525626394 shutdown_connections 2024-04-25T13:55:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.081+0000 7fca911ee640 1 -- 172.21.15.27:0/2525626394 wait complete. 2024-04-25T13:55:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.082+0000 7fca911ee640 1 Processor -- start 2024-04-25T13:55:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca911ee640 1 -- start start 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca911ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c10a420 0x7fca8c1a9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca911ee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca8c10b350 0x7fca8c1a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca911ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca8c110070 0x7fca8c1b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca911ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c11e560 con 0x7fca8c110070 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca911ee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c11e3e0 con 0x7fca8c10a420 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca911ee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c11e6e0 con 0x7fca8c10b350 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca8a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca8c10b350 0x7fca8c1a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.085+0000 7fca8a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca8c10b350 0x7fca8c1a96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:56758/0 (socket says 172.21.15.27:56758) 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.086+0000 7fca8a59c640 1 -- 172.21.15.27:0/1918589121 learned_addr learned my addr 172.21.15.27:0/1918589121 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.086+0000 7fca8ad9d640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c10a420 0x7fca8c1a9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.087+0000 7fca8b59e640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca8c110070 0x7fca8c1b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.087+0000 7fca8a59c640 1 -- 172.21.15.27:0/1918589121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c10a420 msgr2=0x7fca8c1a9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.087+0000 7fca8a59c640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c10a420 0x7fca8c1a9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.087+0000 7fca8a59c640 1 -- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca8c110070 msgr2=0x7fca8c1b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.087+0000 7fca8a59c640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca8c110070 0x7fca8c1b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.087+0000 7fca8a59c640 1 -- 172.21.15.27:0/1918589121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8c111c90 con 0x7fca8c10b350 2024-04-25T13:55:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.087+0000 7fca8b59e640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca8c110070 0x7fca8c1b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:55:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.088+0000 7fca8a59c640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca8c10b350 0x7fca8c1a96a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c890 tx=0x7fca7c00cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:29.094 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.094+0000 7fca73fff640 1 -- 172.21.15.27:0/1918589121 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c004220 con 0x7fca8c10b350 2024-04-25T13:55:29.094 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.095+0000 7fca911ee640 1 -- 172.21.15.27:0/1918589121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca8c111f80 con 0x7fca8c10b350 2024-04-25T13:55:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.095+0000 7fca911ee640 1 -- 172.21.15.27:0/1918589121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca8c112550 con 0x7fca8c10b350 2024-04-25T13:55:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.095+0000 7fca73fff640 1 -- 172.21.15.27:0/1918589121 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca7c0043c0 con 0x7fca8c10b350 2024-04-25T13:55:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.095+0000 7fca73fff640 1 -- 172.21.15.27:0/1918589121 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c004a00 con 0x7fca8c10b350 2024-04-25T13:55:29.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.099+0000 7fca98175740 1 Processor -- start 2024-04-25T13:55:29.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.100+0000 7fca98175740 1 -- start start 2024-04-25T13:55:29.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.100+0000 7fca98175740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48391b0 0x55bdd4bd9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.100+0000 7fca98175740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bd9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.100+0000 7fca98175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48184d0 0x55bdd4be0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.100+0000 7fca98175740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bdd4b3f460 con 0x55bdd48391b0 2024-04-25T13:55:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.100+0000 7fca98175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55bdd4af9590 con 0x55bdd48184d0 2024-04-25T13:55:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.100+0000 7fca98175740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55bdd4b3b420 con 0x55bdd48102f0 2024-04-25T13:55:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.101+0000 7fca8b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48391b0 0x55bdd4bd9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.101+0000 7fca8ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bd9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.101+0000 7fca8b59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48391b0 0x55bdd4bd9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59682/0 (socket says 172.21.15.27:59682) 2024-04-25T13:55:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.101+0000 7fca8b59e640 1 -- 172.21.15.27:0/3054266212 learned_addr learned my addr 172.21.15.27:0/3054266212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.101+0000 7fca8b59e640 1 -- 172.21.15.27:0/3054266212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 msgr2=0x55bdd4bd9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.102+0000 7fca73fff640 1 -- 172.21.15.27:0/1918589121 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fca7c0079b0 con 0x7fca8c10b350 2024-04-25T13:55:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca73fff640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fca740afcf0 0x7fca740b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca73fff640 1 -- 172.21.15.27:0/1918589121 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fca7c0de8d0 con 0x7fca8c10b350 2024-04-25T13:55:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca8a59c640 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48184d0 0x55bdd4be0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca8b59e640 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bd9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca8b59e640 1 -- 172.21.15.27:0/3054266212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48184d0 msgr2=0x55bdd4be0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca8b59e640 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48184d0 0x55bdd4be0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca8b59e640 1 -- 172.21.15.27:0/3054266212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdd4b060d0 con 0x55bdd48391b0 2024-04-25T13:55:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca8ad9d640 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bd9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:55:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.103+0000 7fca8a59c640 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48184d0 0x55bdd4be0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:55:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.105+0000 7fca8b59e640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fca740afcf0 0x7fca740b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.105+0000 7fca8b59e640 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48391b0 0x55bdd4bd9970 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fca80007800 tx=0x7fca80007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579bcbbe0c4c7f62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.106+0000 7fca717fa640 1 -- 172.21.15.27:0/3054266212 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80015da0 con 0x55bdd48391b0 2024-04-25T13:55:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.106+0000 7fca717fa640 1 -- 172.21.15.27:0/3054266212 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca800128a0 con 0x55bdd48391b0 2024-04-25T13:55:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.106+0000 7fca717fa640 1 -- 172.21.15.27:0/3054266212 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80015da0 con 0x55bdd48391b0 2024-04-25T13:55:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.108+0000 7fca98175740 1 -- 172.21.15.27:0/3054266212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48391b0 msgr2=0x55bdd4bd9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.108+0000 7fca98175740 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48391b0 0x55bdd4bd9970 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fca80007800 tx=0x7fca80007cc0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.110+0000 7fca8b59e640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fca740afcf0 0x7fca740b2200 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fca8001e000 tx=0x7fca80021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.110+0000 7fca98175740 1 -- 172.21.15.27:0/3054266212 shutdown_connections 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.110+0000 7fca98175740 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48184d0 0x55bdd4be0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.110+0000 7fca98175740 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bd9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.110+0000 7fca98175740 1 --2- 172.21.15.27:0/3054266212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48391b0 0x55bdd4bd9970 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.110+0000 7fca98175740 1 -- 172.21.15.27:0/3054266212 >> 172.21.15.27:0/3054266212 conn(0x55bdd482d300 msgr2=0x55bdd4bd5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.111+0000 7fca98175740 1 -- 172.21.15.27:0/3054266212 shutdown_connections 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.111+0000 7fca98175740 1 -- 172.21.15.27:0/3054266212 wait complete. 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.112+0000 7fca98175740 1 -- start start 2024-04-25T13:55:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.112+0000 7fca98175740 1 Processor -- start 2024-04-25T13:55:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca98175740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48102f0 0x55bdd4be36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca98175740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48184d0 0x55bdd4be3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca98175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4beaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca98175740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55bdd4be1840 con 0x55bdd48102f0 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca98175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55bdd4ac6b60 con 0x55bdd48391b0 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca98175740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55bdd4be19c0 con 0x55bdd48184d0 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca8ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48102f0 0x55bdd4be36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca8ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48102f0 0x55bdd4be36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59694/0 (socket says 172.21.15.27:59694) 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca8ad9d640 1 -- 172.21.15.27:0/3503856738 learned_addr learned my addr 172.21.15.27:0/3503856738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca98175740 10 client.0 _ll_register_callbacks cb 0x55bdd4aeb2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.113+0000 7fca8a59c640 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48184d0 0x55bdd4be3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.114+0000 7fca8a59c640 1 -- 172.21.15.27:0/3503856738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 msgr2=0x55bdd4beaae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.114+0000 7fca8a59c640 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4beaae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.114 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.114+0000 7fca8a59c640 1 -- 172.21.15.27:0/3503856738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48102f0 msgr2=0x55bdd4be36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.114+0000 7fca8a59c640 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48102f0 0x55bdd4be36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.114+0000 7fca8a59c640 1 -- 172.21.15.27:0/3503856738 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdd4851210 con 0x55bdd48184d0 2024-04-25T13:55:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.115+0000 7fca8a59c640 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48184d0 0x55bdd4be3bf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fca7c0058c0 tx=0x7fca7c005290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=164e2bce7303adb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.115+0000 7fca6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.115+0000 7fca6eff5640 1 -- 172.21.15.27:0/3503856738 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c01d050 con 0x55bdd48184d0 2024-04-25T13:55:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.115+0000 7fca98175740 1 -- 172.21.15.27:0/3503856738 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca800128a0 con 0x55bdd48184d0 2024-04-25T13:55:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.115+0000 7fca6eff5640 1 -- 172.21.15.27:0/3503856738 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fca7c0a3370 con 0x55bdd48184d0 2024-04-25T13:55:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.115+0000 7fca6eff5640 1 -- 172.21.15.27:0/3503856738 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fca7c0c1450 con 0x55bdd48184d0 2024-04-25T13:55:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.116+0000 7fca98175740 10 client.27994 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:29.116 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.116+0000 7fca98175740 1 -- 172.21.15.27:0/3503856738 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55bdd491d620 con 0x55bdd48184d0 2024-04-25T13:55:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.117+0000 7fca6eff5640 1 -- 172.21.15.27:0/3503856738 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca800128a0 con 0x55bdd48184d0 2024-04-25T13:55:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.117+0000 7fca6eff5640 1 -- 172.21.15.27:0/3503856738 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fca7c0a5a50 con 0x55bdd48184d0 2024-04-25T13:55:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.117+0000 7fca98175740 10 client.27994 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.117+0000 7fca98175740 10 client.27994 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.117+0000 7fca98175740 20 client.27994 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.117+0000 7fca98175740 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bdd47f7d80 0x55bdd4bf1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.117+0000 7fca98175740 4 client.27994 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.117+0000 7fca98175740 1 -- 172.21.15.27:0/3503856738 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdd4b4d070 con 0x55bdd47f7d80 2024-04-25T13:55:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.118+0000 7fca8ad9d640 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bdd47f7d80 0x55bdd4bf1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.119+0000 7fca8ad9d640 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bdd47f7d80 0x55bdd4bf1780 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b556222f7e66f76d server_cookie=ce2a5dc2c63d05af in_seq=0 out_seq=0 2024-04-25T13:55:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.119+0000 7fca6eff5640 10 client.27994 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.120+0000 7fca6eff5640 1 -- 172.21.15.27:0/3503856738 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bdd4b4d070 con 0x55bdd47f7d80 2024-04-25T13:55:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.120+0000 7fca6eff5640 10 client.27994 handle_command_reply: tid=0 2024-04-25T13:55:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.120+0000 7fca98175740 1 client.27994 shutdown 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca98175740 1 -- 172.21.15.27:0/3503856738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48184d0 msgr2=0x55bdd4be3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca98175740 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48184d0 0x55bdd4be3bf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fca7c0058c0 tx=0x7fca7c005290 comp rx=0 tx=0).stop 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca98175740 1 -- 172.21.15.27:0/3503856738 shutdown_connections 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca98175740 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bdd47f7d80 0x55bdd4bf1780 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca98175740 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4beaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca98175740 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48184d0 0x55bdd4be3bf0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca98175740 1 --2- 172.21.15.27:0/3503856738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48102f0 0x55bdd4be36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca98175740 1 -- 172.21.15.27:0/3503856738 >> 172.21.15.27:0/3503856738 conn(0x55bdd482d300 msgr2=0x55bdd4bd8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.121+0000 7fca6eff5640 0 client.27994 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:29.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.122+0000 7fca98175740 1 -- 172.21.15.27:0/3503856738 shutdown_connections 2024-04-25T13:55:29.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.122+0000 7fca98175740 1 -- 172.21.15.27:0/3503856738 wait complete. 2024-04-25T13:55:29.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.122+0000 7fca98175740 20 client.27994 trim_cache size 0 max 16384 2024-04-25T13:55:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.143+0000 7fca98175740 1 Processor -- start 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.143+0000 7fca98175740 1 -- start start 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.143+0000 7fca98175740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bddaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.143+0000 7fca98175740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 0x55bdd4bddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.143+0000 7fca98175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4beaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.143+0000 7fca98175740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800c390 con 0x55bdd48184d0 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.143+0000 7fca98175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800ec10 con 0x55bdd48391b0 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.143+0000 7fca98175740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800ca00 con 0x55bdd48102f0 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bddaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bddaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:56792/0 (socket says 172.21.15.27:56792) 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8ad9d640 1 -- 172.21.15.27:0/900454584 learned_addr learned my addr 172.21.15.27:0/900454584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8b59e640 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4beaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8b59e640 1 -- 172.21.15.27:0/900454584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 msgr2=0x55bdd4bddaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8b59e640 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bddaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8b59e640 1 -- 172.21.15.27:0/900454584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 msgr2=0x55bdd4bddfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8b59e640 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 0x55bdd4bddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8b59e640 1 -- 172.21.15.27:0/900454584 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdd4beaf40 con 0x55bdd48391b0 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8ad9d640 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bddaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca8b59e640 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4beaa00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fca80006ec0 tx=0x7fca80019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36e7739c4f113c5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.144+0000 7fca70ff9640 1 -- 172.21.15.27:0/900454584 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8002c020 con 0x55bdd48391b0 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca70ff9640 1 -- 172.21.15.27:0/900454584 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca80019070 con 0x55bdd48391b0 2024-04-25T13:55:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca70ff9640 1 -- 172.21.15.27:0/900454584 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80015da0 con 0x55bdd48391b0 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca98175740 1 -- 172.21.15.27:0/900454584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 msgr2=0x55bdd4beaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca98175740 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4beaa00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fca80006ec0 tx=0x7fca80019000 comp rx=0 tx=0).stop 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca98175740 1 -- 172.21.15.27:0/900454584 shutdown_connections 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca98175740 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4beaa00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca98175740 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 0x55bdd4bddfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca98175740 1 --2- 172.21.15.27:0/900454584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4bddaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.145+0000 7fca98175740 1 -- 172.21.15.27:0/900454584 >> 172.21.15.27:0/900454584 conn(0x55bdd482d300 msgr2=0x55bdd4be38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 -- 172.21.15.27:0/900454584 shutdown_connections 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 -- 172.21.15.27:0/900454584 wait complete. 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 -- start start 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 Processor -- start 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4be7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 0x55bdd4be7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4c57b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800d0d0 con 0x55bdd48184d0 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800e540 con 0x55bdd48391b0 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca98175740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800d7a0 con 0x55bdd48102f0 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca8ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4be7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca8a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 0x55bdd4be7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca8ad9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4be7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:56796/0 (socket says 172.21.15.27:56796) 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.146+0000 7fca8ad9d640 1 -- 172.21.15.27:0/3192964813 learned_addr learned my addr 172.21.15.27:0/3192964813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca8b59e640 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4c57b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca98175740 10 client.0 _ll_register_callbacks cb 0x7fca7c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca8ad9d640 1 -- 172.21.15.27:0/3192964813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 msgr2=0x55bdd4c57b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca8ad9d640 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4c57b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca8ad9d640 1 -- 172.21.15.27:0/3192964813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 msgr2=0x55bdd4be7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca8ad9d640 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 0x55bdd4be7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca8ad9d640 1 -- 172.21.15.27:0/3192964813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdd4c58060 con 0x55bdd48102f0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca8ad9d640 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4be7950 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fca840307a0 tx=0x7fca84030900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44e34b6d394de1c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca6eff5640 1 -- 172.21.15.27:0/3192964813 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca84003970 con 0x55bdd48102f0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca6eff5640 1 -- 172.21.15.27:0/3192964813 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fca84003b10 con 0x55bdd48102f0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca98175740 1 -- 172.21.15.27:0/3192964813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca80019070 con 0x55bdd48102f0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca6eff5640 1 -- 172.21.15.27:0/3192964813 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fca84003d10 con 0x55bdd48102f0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca98175740 10 client.28000 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.147+0000 7fca98175740 1 -- 172.21.15.27:0/3192964813 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55bdd4c59280 con 0x55bdd48102f0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca6eff5640 1 -- 172.21.15.27:0/3192964813 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca80019070 con 0x55bdd48102f0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca6eff5640 1 -- 172.21.15.27:0/3192964813 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fca840042e0 con 0x55bdd48102f0 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca98175740 10 client.28000 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca98175740 10 client.28000 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca98175740 20 client.28000 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca98175740 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bdd47cd1c0 0x55bdd4c5bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca98175740 4 client.28000 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:55:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca98175740 1 -- 172.21.15.27:0/3192964813 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bdd4b46ee0 con 0x55bdd47cd1c0 2024-04-25T13:55:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.148+0000 7fca8a59c640 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bdd47cd1c0 0x55bdd4c5bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.150+0000 7fca8a59c640 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bdd47cd1c0 0x55bdd4c5bc50 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae1ccb12103fa8fd server_cookie=ee4fa563a4ab7fd8 in_seq=0 out_seq=0 2024-04-25T13:55:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.150+0000 7fca6eff5640 10 client.28000 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.224+0000 7fca6eff5640 1 -- 172.21.15.27:0/3192964813 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bdd4b46ee0 con 0x55bdd47cd1c0 2024-04-25T13:55:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.224+0000 7fca6eff5640 10 client.28000 handle_command_reply: tid=0 2024-04-25T13:55:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.224+0000 7fca98175740 1 client.28000 shutdown 2024-04-25T13:55:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 -- 172.21.15.27:0/3192964813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 msgr2=0x55bdd4be7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4be7950 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fca840307a0 tx=0x7fca84030900 comp rx=0 tx=0).stop 2024-04-25T13:55:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 -- 172.21.15.27:0/3192964813 shutdown_connections 2024-04-25T13:55:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55bdd47cd1c0 0x55bdd4c5bc50 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.225 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bdd48391b0 0x55bdd4c57b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bdd48184d0 0x55bdd4be7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 --2- 172.21.15.27:0/3192964813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdd48102f0 0x55bdd4be7950 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca6eff5640 0 client.28000 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 -- 172.21.15.27:0/3192964813 >> 172.21.15.27:0/3192964813 conn(0x55bdd482d300 msgr2=0x55bdd4be1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 -- 172.21.15.27:0/3192964813 shutdown_connections 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 1 -- 172.21.15.27:0/3192964813 wait complete. 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca98175740 20 client.28000 trim_cache size 0 max 16384 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:55:29.226 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:55:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:55:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca911ee640 1 -- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fca740afcf0 msgr2=0x7fca740b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca911ee640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fca740afcf0 0x7fca740b2200 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fca8001e000 tx=0x7fca80021000 comp rx=0 tx=0).stop 2024-04-25T13:55:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca911ee640 1 -- 172.21.15.27:0/1918589121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca8c10b350 msgr2=0x7fca8c1a96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:29.227 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.225+0000 7fca911ee640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca8c10b350 0x7fca8c1a96a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c890 tx=0x7fca7c00cd50 comp rx=0 tx=0).stop 2024-04-25T13:55:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.226+0000 7fca911ee640 1 -- 172.21.15.27:0/1918589121 shutdown_connections 2024-04-25T13:55:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.226+0000 7fca911ee640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fca740afcf0 0x7fca740b2200 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.226+0000 7fca911ee640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca8c110070 0x7fca8c1b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.226+0000 7fca911ee640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca8c10b350 0x7fca8c1a96a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.226+0000 7fca911ee640 1 --2- 172.21.15.27:0/1918589121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca8c10a420 0x7fca8c1a9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.226+0000 7fca911ee640 1 -- 172.21.15.27:0/1918589121 >> 172.21.15.27:0/1918589121 conn(0x7fca8c070540 msgr2=0x7fca8c10c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.226+0000 7fca911ee640 1 -- 172.21.15.27:0/1918589121 shutdown_connections 2024-04-25T13:55:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:29.226+0000 7fca911ee640 1 -- 172.21.15.27:0/1918589121 wait complete. 2024-04-25T13:55:30.309 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:29 smithi122 ceph-mon[32843]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 1.39k op/s 2024-04-25T13:55:30.330 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:29 smithi045 ceph-mon[32731]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 1.39k op/s 2024-04-25T13:55:30.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:29 smithi027 ceph-mon[29140]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 1.39k op/s 2024-04-25T13:55:31.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:30 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:55:31.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:30 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:55:31.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:30 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:55:32.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:31 smithi122 ceph-mon[32843]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 36 MiB/s wr, 1.47k op/s 2024-04-25T13:55:32.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:55:32.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:55:32.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:31 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:55:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:31 smithi045 ceph-mon[32731]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 36 MiB/s wr, 1.47k op/s 2024-04-25T13:55:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:55:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:55:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:31 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:55:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:31 smithi027 ceph-mon[29140]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 36 MiB/s wr, 1.47k op/s 2024-04-25T13:55:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:55:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:55:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:31 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:55:32.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 865.0 s, 1457.2 tps, lat 68.043 ms stddev 79.378 2024-04-25T13:55:34.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:33 smithi122 ceph-mon[32843]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 38 MiB/s wr, 1.63k op/s 2024-04-25T13:55:34.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:33 smithi045 ceph-mon[32731]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 38 MiB/s wr, 1.63k op/s 2024-04-25T13:55:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:33 smithi027 ceph-mon[29140]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 38 MiB/s wr, 1.63k op/s 2024-04-25T13:55:36.275 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:55:36.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:55:36.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:35 smithi122 ceph-mon[32843]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 37 MiB/s wr, 1.51k op/s 2024-04-25T13:55:36.314 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:35 smithi027 ceph-mon[29140]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 37 MiB/s wr, 1.51k op/s 2024-04-25T13:55:36.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:35 smithi045 ceph-mon[32731]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 37 MiB/s wr, 1.51k op/s 2024-04-25T13:55:36.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.468+0000 7fed94b85640 1 -- 172.21.15.27:0/3048130078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed90103ae0 msgr2=0x7fed901047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.468+0000 7fed94b85640 1 --2- 172.21.15.27:0/3048130078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed90103ae0 0x7fed901047e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fed80009900 tx=0x7fed8002a410 comp rx=0 tx=0).stop 2024-04-25T13:55:36.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.468+0000 7fed94b85640 1 -- 172.21.15.27:0/3048130078 shutdown_connections 2024-04-25T13:55:36.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.468+0000 7fed94b85640 1 --2- 172.21.15.27:0/3048130078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed90109a20 0x7fed9010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.468+0000 7fed94b85640 1 --2- 172.21.15.27:0/3048130078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed90104d20 0x7fed90105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.468+0000 7fed94b85640 1 --2- 172.21.15.27:0/3048130078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed90103ae0 0x7fed901047e0 secure :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fed80009900 tx=0x7fed8002a410 comp rx=0 tx=0).stop 2024-04-25T13:55:36.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.468+0000 7fed94b85640 1 -- 172.21.15.27:0/3048130078 >> 172.21.15.27:0/3048130078 conn(0x7fed900fed10 msgr2=0x7fed90101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:36.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.468+0000 7fed94b85640 1 -- 172.21.15.27:0/3048130078 shutdown_connections 2024-04-25T13:55:36.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.469+0000 7fed94b85640 1 -- 172.21.15.27:0/3048130078 wait complete. 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.470+0000 7fed94b85640 1 Processor -- start 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed94b85640 1 -- start start 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed94b85640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed90104d20 0x7fed9019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed94b85640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed90109a20 0x7fed9019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed94b85640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed9019f7a0 0x7fed901a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed94b85640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fed9010f5c0 con 0x7fed9019f7a0 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed94b85640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed9010f440 con 0x7fed90109a20 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed94b85640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fed9010f740 con 0x7fed90104d20 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed90104d20 0x7fed9019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed9019f7a0 0x7fed901a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8e59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed90104d20 0x7fed9019a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42918/0 (socket says 172.21.15.27:42918) 2024-04-25T13:55:36.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed9019f7a0 0x7fed901a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42254/0 (socket says 172.21.15.27:42254) 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed90109a20 0x7fed9019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8e59c640 1 -- 172.21.15.27:0/1363049079 learned_addr learned my addr 172.21.15.27:0/1363049079 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8e59c640 1 -- 172.21.15.27:0/1363049079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed90109a20 msgr2=0x7fed9019a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8e59c640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed90109a20 0x7fed9019a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8e59c640 1 -- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed9019f7a0 msgr2=0x7fed901a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8e59c640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed9019f7a0 0x7fed901a1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.471+0000 7fed8e59c640 1 -- 172.21.15.27:0/1363049079 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed901a22f0 con 0x7fed90104d20 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.472+0000 7fed8e59c640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed90104d20 0x7fed9019a2e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fed80006010 tx=0x7fed800042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.472+0000 7fed7f7fe640 1 -- 172.21.15.27:0/1363049079 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8003f030 con 0x7fed90104d20 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.472+0000 7fed7f7fe640 1 -- 172.21.15.27:0/1363049079 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed80031910 con 0x7fed90104d20 2024-04-25T13:55:36.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.472+0000 7fed94b85640 1 -- 172.21.15.27:0/1363049079 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed901a2580 con 0x7fed90104d20 2024-04-25T13:55:36.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.472+0000 7fed7f7fe640 1 -- 172.21.15.27:0/1363049079 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80035070 con 0x7fed90104d20 2024-04-25T13:55:36.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.472+0000 7fed94b85640 1 -- 172.21.15.27:0/1363049079 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed901a2a20 con 0x7fed90104d20 2024-04-25T13:55:36.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.474+0000 7fed9ba31740 1 Processor -- start 2024-04-25T13:55:36.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed9ba31740 1 -- start start 2024-04-25T13:55:36.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c5fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed7f7fe640 1 -- 172.21.15.27:0/1363049079 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fed80042030 con 0x7fed90104d20 2024-04-25T13:55:36.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c61080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed9ba31740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55da38bc0d80 con 0x55da388ba390 2024-04-25T13:55:36.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed9ba31740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55da38b7a8b0 con 0x55da388996b0 2024-04-25T13:55:36.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed9ba31740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55da38bbcd40 con 0x55da388914d0 2024-04-25T13:55:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c5fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c5fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42270/0 (socket says 172.21.15.27:42270) 2024-04-25T13:55:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed8ed9d640 1 -- 172.21.15.27:0/1394345098 learned_addr learned my addr 172.21.15.27:0/1394345098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.475+0000 7fed8e59c640 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed8dd9b640 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c61080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed8ed9d640 1 -- 172.21.15.27:0/1394345098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 msgr2=0x55da38c603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed8ed9d640 1 -- 172.21.15.27:0/1394345098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 msgr2=0x55da38c61080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c61080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed8ed9d640 1 -- 172.21.15.27:0/1394345098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da38b4ba40 con 0x55da388ba390 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c5fe70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fed88009fd0 tx=0x7fed880118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3087aa9573b9a74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed7f7fe640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fed780afb70 0x7fed780b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed7cff9640 1 -- 172.21.15.27:0/1394345098 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88004d20 con 0x55da388ba390 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.476+0000 7fed7cff9640 1 -- 172.21.15.27:0/1394345098 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed88004050 con 0x55da388ba390 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.477+0000 7fed8e59c640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fed780afb70 0x7fed780b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.477+0000 7fed7f7fe640 1 -- 172.21.15.27:0/1363049079 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fed800ff060 con 0x7fed90104d20 2024-04-25T13:55:36.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.477+0000 7fed7cff9640 1 -- 172.21.15.27:0/1394345098 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88005280 con 0x55da388ba390 2024-04-25T13:55:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 -- 172.21.15.27:0/1394345098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 msgr2=0x55da38c5fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed8e59c640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fed780afb70 0x7fed780b2080 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fed80009900 tx=0x7fed80102a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c5fe70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fed88009fd0 tx=0x7fed880118d0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 -- 172.21.15.27:0/1394345098 shutdown_connections 2024-04-25T13:55:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c61080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 --2- 172.21.15.27:0/1394345098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c5fe70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 -- 172.21.15.27:0/1394345098 >> 172.21.15.27:0/1394345098 conn(0x55da388ae4e0 msgr2=0x55da38c57270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 -- 172.21.15.27:0/1394345098 shutdown_connections 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 -- 172.21.15.27:0/1394345098 wait complete. 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 -- start start 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.478+0000 7fed9ba31740 1 Processor -- start 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c69d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c6a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed9ba31740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55da38c61c30 con 0x55da388ba390 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed9ba31740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55da38b47d10 con 0x55da388996b0 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed9ba31740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55da38c61db0 con 0x55da388914d0 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed9ba31740 10 client.0 _ll_register_callbacks cb 0x55da38b74210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c711a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42284/0 (socket says 172.21.15.27:42284) 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c6a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8ed9d640 1 -- 172.21.15.27:0/474060 learned_addr learned my addr 172.21.15.27:0/474060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c69d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8dd9b640 1 -- 172.21.15.27:0/474060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 msgr2=0x55da38c69d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8dd9b640 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c69d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8dd9b640 1 -- 172.21.15.27:0/474060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 msgr2=0x55da38c711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8dd9b640 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8dd9b640 1 -- 172.21.15.27:0/474060 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da3899c7d0 con 0x55da388996b0 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed8dd9b640 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c6a240 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fed840126d0 tx=0x7fed84012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9153b62158682acc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.479+0000 7fed75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed75ffb640 1 -- 172.21.15.27:0/474060 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8400f920 con 0x55da388996b0 2024-04-25T13:55:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed75ffb640 1 -- 172.21.15.27:0/474060 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fed8400fac0 con 0x55da388996b0 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 1 -- 172.21.15.27:0/474060 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed88004050 con 0x55da388996b0 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed75ffb640 1 -- 172.21.15.27:0/474060 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fed8400fcc0 con 0x55da388996b0 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 10 client.27941 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 1 -- 172.21.15.27:0/474060 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55da388d23f0 con 0x55da388996b0 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed75ffb640 1 -- 172.21.15.27:0/474060 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed88004050 con 0x55da388996b0 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed75ffb640 1 -- 172.21.15.27:0/474060 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fed84027420 con 0x55da388996b0 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 10 client.27941 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 10 client.27941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 20 client.27941 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55da38878f60 0x55da38c77760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 4 client.27941 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.480+0000 7fed9ba31740 1 -- 172.21.15.27:0/474060 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da38bce990 con 0x55da38878f60 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.481+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55da38878f60 0x55da38c77760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.481+0000 7fed75ffb640 10 client.27941 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:36.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.481+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55da38878f60 0x55da38c77760 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=584df39a6a31db56 server_cookie=56c675bc239e9bf3 in_seq=0 out_seq=0 2024-04-25T13:55:36.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.482+0000 7fed75ffb640 1 -- 172.21.15.27:0/474060 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55da38bce990 con 0x55da38878f60 2024-04-25T13:55:36.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.482+0000 7fed75ffb640 10 client.27941 handle_command_reply: tid=0 2024-04-25T13:55:36.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.482+0000 7fed9ba31740 1 client.27941 shutdown 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed9ba31740 1 -- 172.21.15.27:0/474060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 msgr2=0x55da38c6a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed9ba31740 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c6a240 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fed840126d0 tx=0x7fed84012b90 comp rx=0 tx=0).stop 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed9ba31740 1 -- 172.21.15.27:0/474060 shutdown_connections 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed9ba31740 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55da38878f60 0x55da38c77760 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed9ba31740 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed9ba31740 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388996b0 0x55da38c6a240 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fed840126d0 tx=0x7fed84012b90 comp rx=0 tx=0).stop 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed9ba31740 1 --2- 172.21.15.27:0/474060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c69d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed9ba31740 1 -- 172.21.15.27:0/474060 >> 172.21.15.27:0/474060 conn(0x55da388ae4e0 msgr2=0x55da38c5a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.483+0000 7fed75ffb640 0 client.27941 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.484+0000 7fed9ba31740 1 -- 172.21.15.27:0/474060 shutdown_connections 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.484+0000 7fed9ba31740 1 -- 172.21.15.27:0/474060 wait complete. 2024-04-25T13:55:36.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.484+0000 7fed9ba31740 20 client.27941 trim_cache size 0 max 16384 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.508+0000 7fed9ba31740 1 Processor -- start 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.508+0000 7fed9ba31740 1 -- start start 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.509+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c69d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.509+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c56ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.509+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da3884e3a0 0x55da38c571e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.509+0000 7fed9ba31740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fed901a7610 con 0x55da388ba390 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.509+0000 7fed9ba31740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed901a9e90 con 0x55da3884e3a0 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.509+0000 7fed9ba31740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fed901a7c80 con 0x55da388914d0 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.510+0000 7fed8dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c56ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.510+0000 7fed8dd9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c56ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42288/0 (socket says 172.21.15.27:42288) 2024-04-25T13:55:36.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.510+0000 7fed8dd9b640 1 -- 172.21.15.27:0/3657135795 learned_addr learned my addr 172.21.15.27:0/3657135795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:36.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.512+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c69d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.513+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da3884e3a0 0x55da38c571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.513+0000 7fed8ed9d640 1 -- 172.21.15.27:0/3657135795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 msgr2=0x55da38c69d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:36.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.513+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c69d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.513+0000 7fed8ed9d640 1 -- 172.21.15.27:0/3657135795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 msgr2=0x55da38c56ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:36.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.513+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c56ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.513+0000 7fed8ed9d640 1 -- 172.21.15.27:0/3657135795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da388920f0 con 0x55da3884e3a0 2024-04-25T13:55:36.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.518+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da3884e3a0 0x55da38c571e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fed88005c40 tx=0x7fed88013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c55a5f1e4acf3398 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:36.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.521+0000 7fed77fff640 1 -- 172.21.15.27:0/3657135795 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88004050 con 0x55da3884e3a0 2024-04-25T13:55:36.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.521+0000 7fed77fff640 1 -- 172.21.15.27:0/3657135795 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed88004940 con 0x55da3884e3a0 2024-04-25T13:55:36.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.521+0000 7fed77fff640 1 -- 172.21.15.27:0/3657135795 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88005040 con 0x55da3884e3a0 2024-04-25T13:55:36.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.523+0000 7fed9ba31740 1 -- 172.21.15.27:0/3657135795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da3884e3a0 msgr2=0x55da38c571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.523+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da3884e3a0 0x55da38c571e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fed88005c40 tx=0x7fed88013040 comp rx=0 tx=0).stop 2024-04-25T13:55:36.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.523+0000 7fed9ba31740 1 -- 172.21.15.27:0/3657135795 shutdown_connections 2024-04-25T13:55:36.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.523+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da3884e3a0 0x55da38c571e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.523+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388ba390 0x55da38c56ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.523+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3657135795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da388914d0 0x55da38c69d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.523+0000 7fed9ba31740 1 -- 172.21.15.27:0/3657135795 >> 172.21.15.27:0/3657135795 conn(0x55da388ae4e0 msgr2=0x55da38c68440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:36.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.532+0000 7fed9ba31740 1 -- 172.21.15.27:0/3657135795 shutdown_connections 2024-04-25T13:55:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.536+0000 7fed9ba31740 1 -- 172.21.15.27:0/3657135795 wait complete. 2024-04-25T13:55:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.536+0000 7fed9ba31740 1 -- start start 2024-04-25T13:55:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.536+0000 7fed9ba31740 1 Processor -- start 2024-04-25T13:55:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.537+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da3884e3a0 0x55da38c5de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.537+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388914d0 0x55da38c5e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.537+0000 7fed9ba31740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388ba390 0x55da38c5e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.537+0000 7fed9ba31740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fed901a8350 con 0x55da388914d0 2024-04-25T13:55:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.537+0000 7fed9ba31740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed901a97c0 con 0x55da388ba390 2024-04-25T13:55:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.537+0000 7fed9ba31740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fed901a8a20 con 0x55da3884e3a0 2024-04-25T13:55:36.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.537+0000 7fed9ba31740 10 client.0 _ll_register_callbacks cb 0x7fed84009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.540+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da3884e3a0 0x55da38c5de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.540+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388ba390 0x55da38c5e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.540+0000 7fed8ed9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da3884e3a0 0x55da38c5de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42944/0 (socket says 172.21.15.27:42944) 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.540+0000 7fed8ed9d640 1 -- 172.21.15.27:0/3440867603 learned_addr learned my addr 172.21.15.27:0/3440867603 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.541+0000 7fed8ed9d640 1 -- 172.21.15.27:0/3440867603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388ba390 msgr2=0x55da38c5e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.541+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388ba390 0x55da38c5e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.541+0000 7fed8ed9d640 1 -- 172.21.15.27:0/3440867603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388914d0 msgr2=0x55da38c5e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.541+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388914d0 0x55da38c5e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.541+0000 7fed8ed9d640 1 -- 172.21.15.27:0/3440867603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da38b814b0 con 0x55da3884e3a0 2024-04-25T13:55:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.542+0000 7fed8ed9d640 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da3884e3a0 0x55da38c5de10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fed88003fd0 tx=0x7fed880118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f579661e145cac9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:36.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.543+0000 7fed75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:36.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.545+0000 7fed75ffb640 1 -- 172.21.15.27:0/3440867603 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88004420 con 0x55da3884e3a0 2024-04-25T13:55:36.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.546+0000 7fed9ba31740 1 -- 172.21.15.27:0/3440867603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed88004940 con 0x55da3884e3a0 2024-04-25T13:55:36.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.546+0000 7fed75ffb640 1 -- 172.21.15.27:0/3440867603 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fed88030030 con 0x55da3884e3a0 2024-04-25T13:55:36.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.546+0000 7fed75ffb640 1 -- 172.21.15.27:0/3440867603 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fed880045c0 con 0x55da3884e3a0 2024-04-25T13:55:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.548+0000 7fed9ba31740 10 client.28015 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:36.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.548+0000 7fed9ba31740 1 -- 172.21.15.27:0/3440867603 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55da38c702d0 con 0x55da3884e3a0 2024-04-25T13:55:36.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.549+0000 7fed75ffb640 1 -- 172.21.15.27:0/3440867603 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed88004940 con 0x55da3884e3a0 2024-04-25T13:55:36.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.550+0000 7fed75ffb640 1 -- 172.21.15.27:0/3440867603 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fed8800f540 con 0x55da3884e3a0 2024-04-25T13:55:36.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.551+0000 7fed9ba31740 10 client.28015 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:36.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.551+0000 7fed9ba31740 10 client.28015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:36.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.551+0000 7fed9ba31740 20 client.28015 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:36.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.551+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55da38cd9aa0 0x55da38cdbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.551+0000 7fed9ba31740 4 client.28015 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:55:36.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.551+0000 7fed9ba31740 1 -- 172.21.15.27:0/3440867603 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55da38bc8800 con 0x55da38cd9aa0 2024-04-25T13:55:36.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.551+0000 7fed8dd9b640 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55da38cd9aa0 0x55da38cdbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.552+0000 7fed8dd9b640 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55da38cd9aa0 0x55da38cdbec0 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f09893df9fc9e608 server_cookie=d18dce65a39f0147 in_seq=0 out_seq=0 2024-04-25T13:55:36.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.552+0000 7fed75ffb640 10 client.28015 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.555+0000 7fed75ffb640 1 -- 172.21.15.27:0/3440867603 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55da38bc8800 con 0x55da38cd9aa0 2024-04-25T13:55:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.555+0000 7fed75ffb640 10 client.28015 handle_command_reply: tid=0 2024-04-25T13:55:36.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.555+0000 7fed9ba31740 1 client.28015 shutdown 2024-04-25T13:55:36.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.555+0000 7fed9ba31740 1 -- 172.21.15.27:0/3440867603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da3884e3a0 msgr2=0x55da38c5de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.555+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da3884e3a0 0x55da38c5de10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fed88003fd0 tx=0x7fed880118d0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.558+0000 7fed9ba31740 1 -- 172.21.15.27:0/3440867603 shutdown_connections 2024-04-25T13:55:36.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.558+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55da38cd9aa0 0x55da38cdbec0 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.558+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da388ba390 0x55da38c5e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.558+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55da388914d0 0x55da38c5e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.558+0000 7fed9ba31740 1 --2- 172.21.15.27:0/3440867603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55da3884e3a0 0x55da38c5de10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.558+0000 7fed9ba31740 1 -- 172.21.15.27:0/3440867603 >> 172.21.15.27:0/3440867603 conn(0x55da388ae4e0 msgr2=0x55da38c68440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:36.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.558+0000 7fed75ffb640 0 client.28015 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:36.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.561+0000 7fed9ba31740 1 -- 172.21.15.27:0/3440867603 shutdown_connections 2024-04-25T13:55:36.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.561+0000 7fed9ba31740 1 -- 172.21.15.27:0/3440867603 wait complete. 2024-04-25T13:55:36.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.562+0000 7fed9ba31740 20 client.28015 trim_cache size 0 max 16384 2024-04-25T13:55:36.562 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:55:36.562 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:55:36.562 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:55:36.562 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:55:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 -- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fed780afb70 msgr2=0x7fed780b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fed780afb70 0x7fed780b2080 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fed80009900 tx=0x7fed80102a20 comp rx=0 tx=0).stop 2024-04-25T13:55:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 -- 172.21.15.27:0/1363049079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed90104d20 msgr2=0x7fed9019a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed90104d20 0x7fed9019a2e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fed80006010 tx=0x7fed800042f0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 -- 172.21.15.27:0/1363049079 shutdown_connections 2024-04-25T13:55:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fed780afb70 0x7fed780b2080 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed9019f7a0 0x7fed901a1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed90109a20 0x7fed9019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 --2- 172.21.15.27:0/1363049079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fed90104d20 0x7fed9019a2e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 -- 172.21.15.27:0/1363049079 >> 172.21.15.27:0/1363049079 conn(0x7fed900fed10 msgr2=0x7fed90105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 -- 172.21.15.27:0/1363049079 shutdown_connections 2024-04-25T13:55:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.563+0000 7fed94b85640 1 -- 172.21.15.27:0/1363049079 wait complete. 2024-04-25T13:55:36.578 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:55:36.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:55:36.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.796+0000 7f868b5ba640 1 -- 172.21.15.27:0/3594045535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8684110030 msgr2=0x7f8684110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.796+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3594045535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8684110030 0x7f8684110440 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f867c009ca0 tx=0x7f867c02a8f0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.801+0000 7f868b5ba640 1 -- 172.21.15.27:0/3594045535 shutdown_connections 2024-04-25T13:55:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.801+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3594045535 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8684110030 0x7f8684110440 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.801+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3594045535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f868410b310 0x7f868410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.801+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3594045535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868410a420 0x7f868410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.801+0000 7f868b5ba640 1 -- 172.21.15.27:0/3594045535 >> 172.21.15.27:0/3594045535 conn(0x7f86840704e0 msgr2=0x7f8684072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:36.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.806+0000 7f868b5ba640 1 -- 172.21.15.27:0/3594045535 shutdown_connections 2024-04-25T13:55:36.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.810+0000 7f868b5ba640 1 -- 172.21.15.27:0/3594045535 wait complete. 2024-04-25T13:55:36.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.810+0000 7f868b5ba640 1 Processor -- start 2024-04-25T13:55:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.811+0000 7f868b5ba640 1 -- start start 2024-04-25T13:55:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.811+0000 7f868b5ba640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868410a420 0x7f8684119d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.811+0000 7f868b5ba640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f868410b310 0x7f868411a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.811+0000 7f868b5ba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8684110030 0x7f868411b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.811+0000 7f868b5ba640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8684111f90 con 0x7f868410a420 2024-04-25T13:55:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.811+0000 7f868b5ba640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8684111e10 con 0x7f8684110030 2024-04-25T13:55:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.811+0000 7f868b5ba640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8684112110 con 0x7f868410b310 2024-04-25T13:55:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.812+0000 7f8689b57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8684110030 0x7f868411b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.812+0000 7f8689b57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8684110030 0x7f868411b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34104/0 (socket says 172.21.15.27:34104) 2024-04-25T13:55:36.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.812+0000 7f8689b57640 1 -- 172.21.15.27:0/3033219713 learned_addr learned my addr 172.21.15.27:0/3033219713 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:36.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.812+0000 7f8689b57640 1 -- 172.21.15.27:0/3033219713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f868410b310 msgr2=0x7f868411a270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T13:55:36.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.812+0000 7f8689b57640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f868410b310 0x7f868411a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.812+0000 7f8689b57640 1 -- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868410a420 msgr2=0x7f8684119d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:36.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.812+0000 7f8689b57640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868410a420 0x7f8684119d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:36.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.812+0000 7f8689b57640 1 -- 172.21.15.27:0/3033219713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868411b840 con 0x7f8684110030 2024-04-25T13:55:36.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.813+0000 7f8689b57640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8684110030 0x7f868411b1d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f867c031b00 tx=0x7f867c02fe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:36.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.820+0000 7f867a7fc640 1 -- 172.21.15.27:0/3033219713 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c030a00 con 0x7f8684110030 2024-04-25T13:55:36.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.820+0000 7f867a7fc640 1 -- 172.21.15.27:0/3033219713 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f867c02d070 con 0x7f8684110030 2024-04-25T13:55:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.821+0000 7f867a7fc640 1 -- 172.21.15.27:0/3033219713 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c03b400 con 0x7f8684110030 2024-04-25T13:55:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.821+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868411bad0 con 0x7f8684110030 2024-04-25T13:55:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.821+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f868411bf90 con 0x7f8684110030 2024-04-25T13:55:36.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.825+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86841138d0 con 0x7f8684110030 2024-04-25T13:55:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.827+0000 7f867a7fc640 1 -- 172.21.15.27:0/3033219713 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f867c045610 con 0x7f8684110030 2024-04-25T13:55:36.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.828+0000 7f867a7fc640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f86700afcf0 0x7f86700b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:36.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.829+0000 7f867a7fc640 1 -- 172.21.15.27:0/3033219713 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f867c100200 con 0x7f8684110030 2024-04-25T13:55:36.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.829+0000 7f8689356640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f86700afcf0 0x7f86700b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:36.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.830+0000 7f8689356640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f86700afcf0 0x7f86700b2200 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8684109c60 tx=0x7f8674006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:36.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:36.834+0000 7f867a7fc640 1 -- 172.21.15.27:0/3033219713 <== mon.2 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f867c0c8db0 con 0x7f8684110030 2024-04-25T13:55:37.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.182+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86841b9f00 con 0x7f8684110030 2024-04-25T13:55:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:55:37.186 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:55:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:55:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.186+0000 7f867a7fc640 1 -- 172.21.15.27:0/3033219713 <== mon.2 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f867c0c86b0 con 0x7f8684110030 2024-04-25T13:55:37.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.198+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f86700afcf0 msgr2=0x7f86700b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.198+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f86700afcf0 0x7f86700b2200 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8684109c60 tx=0x7f8674006c90 comp rx=0 tx=0).stop 2024-04-25T13:55:37.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.198+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8684110030 msgr2=0x7f868411b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.198+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8684110030 0x7f868411b1d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f867c031b00 tx=0x7f867c02fe80 comp rx=0 tx=0).stop 2024-04-25T13:55:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.201+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 shutdown_connections 2024-04-25T13:55:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.201+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f86700afcf0 0x7f86700b2200 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.201+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8684110030 0x7f868411b1d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.201+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f868410b310 0x7f868411a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.201+0000 7f868b5ba640 1 --2- 172.21.15.27:0/3033219713 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f868410a420 0x7f8684119d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.201+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 >> 172.21.15.27:0/3033219713 conn(0x7f86840704e0 msgr2=0x7f868410e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.202+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 shutdown_connections 2024-04-25T13:55:37.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.202+0000 7f868b5ba640 1 -- 172.21.15.27:0/3033219713 wait complete. 2024-04-25T13:55:37.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:55:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.404+0000 7f3492ffd640 1 -- 172.21.15.27:0/309792626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347c0318a0 con 0x7f3494110030 2024-04-25T13:55:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.404+0000 7f349a4a8640 1 -- 172.21.15.27:0/309792626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494110030 msgr2=0x7f3494110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.404+0000 7f349a4a8640 1 --2- 172.21.15.27:0/309792626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494110030 0x7f3494110440 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f347c009900 tx=0x7f347c02a440 comp rx=0 tx=0).stop 2024-04-25T13:55:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.405+0000 7f349a4a8640 1 -- 172.21.15.27:0/309792626 shutdown_connections 2024-04-25T13:55:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.405+0000 7f349a4a8640 1 --2- 172.21.15.27:0/309792626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494110030 0x7f3494110440 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.405+0000 7f349a4a8640 1 --2- 172.21.15.27:0/309792626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f349410b310 0x7f349410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.405+0000 7f349a4a8640 1 --2- 172.21.15.27:0/309792626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f349410a420 0x7f349410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.405+0000 7f349a4a8640 1 -- 172.21.15.27:0/309792626 >> 172.21.15.27:0/309792626 conn(0x7f34940704e0 msgr2=0x7f3494072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.406+0000 7f349a4a8640 1 -- 172.21.15.27:0/309792626 shutdown_connections 2024-04-25T13:55:37.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.406+0000 7f349a4a8640 1 -- 172.21.15.27:0/309792626 wait complete. 2024-04-25T13:55:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f349a4a8640 1 Processor -- start 2024-04-25T13:55:37.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f349a4a8640 1 -- start start 2024-04-25T13:55:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f349a4a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f349410a420 0x7f34941a9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f349a4a8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f349410b310 0x7f34941a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f349a4a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3494110030 0x7f34941b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f349a4a8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f349411e5e0 con 0x7f349410b310 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f349a4a8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f349411e460 con 0x7f349410a420 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f349a4a8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f349411e760 con 0x7f3494110030 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.407+0000 7f34937fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f349410b310 0x7f34941a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f34937fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f349410b310 0x7f34941a9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42338/0 (socket says 172.21.15.27:42338) 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f34937fe640 1 -- 172.21.15.27:0/963669325 learned_addr learned my addr 172.21.15.27:0/963669325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f3493fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f349410a420 0x7f34941a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f3498a45640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3494110030 0x7f34941b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f3498a45640 1 -- 172.21.15.27:0/963669325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f349410a420 msgr2=0x7f34941a9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f3498a45640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f349410a420 0x7f34941a9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f3498a45640 1 -- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f349410b310 msgr2=0x7f34941a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f3498a45640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f349410b310 0x7f34941a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.408+0000 7f3498a45640 1 -- 172.21.15.27:0/963669325 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34941b0d60 con 0x7f3494110030 2024-04-25T13:55:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.409+0000 7f3498a45640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3494110030 0x7f34941b06f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f347c009230 tx=0x7f347c003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.409+0000 7f34917fa640 1 -- 172.21.15.27:0/963669325 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347c03f070 con 0x7f3494110030 2024-04-25T13:55:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.409+0000 7f349a4a8640 1 -- 172.21.15.27:0/963669325 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34941b0ff0 con 0x7f3494110030 2024-04-25T13:55:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.409+0000 7f34917fa640 1 -- 172.21.15.27:0/963669325 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f347c031c50 con 0x7f3494110030 2024-04-25T13:55:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.409+0000 7f34917fa640 1 -- 172.21.15.27:0/963669325 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347c035030 con 0x7f3494110030 2024-04-25T13:55:37.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.409+0000 7f349a4a8640 1 -- 172.21.15.27:0/963669325 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34941b14d0 con 0x7f3494110030 2024-04-25T13:55:37.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.412+0000 7f34a1436740 1 Processor -- start 2024-04-25T13:55:37.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.413+0000 7f34917fa640 1 -- 172.21.15.27:0/963669325 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f347c042070 con 0x7f3494110030 2024-04-25T13:55:37.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.413+0000 7f34a1436740 1 -- start start 2024-04-25T13:55:37.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34a1436740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23286e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34a1436740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23286ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34a1436740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a23286fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34a1436740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a2327d2d80 con 0x55a2324ab6b0 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34a1436740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55a23278ca20 con 0x55a2324a34d0 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34a1436740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55a2327ced40 con 0x55a2324cc390 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34917fa640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f34800afbb0 0x7f34800b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23286e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f3493fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23286ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23286e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42966/0 (socket says 172.21.15.27:42966) 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 -- 172.21.15.27:0/1117255990 learned_addr learned my addr 172.21.15.27:0/1117255990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a23286fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34917fa640 1 -- 172.21.15.27:0/963669325 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f347c0ff560 con 0x7f3494110030 2024-04-25T13:55:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f3498a45640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f34800afbb0 0x7f34800b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 -- 172.21.15.27:0/1117255990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 msgr2=0x55a23286ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23286ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 -- 172.21.15.27:0/1117255990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 msgr2=0x55a23286fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a23286fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f34937fe640 1 -- 172.21.15.27:0/1117255990 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a232797c30 con 0x55a2324cc390 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.414+0000 7f3493fff640 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23286ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.415+0000 7f3498a45640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f34800afbb0 0x7f34800b20c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f347c0c3a60 tx=0x7f347c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.415+0000 7f34937fe640 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23286e810 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f34840177a0 tx=0x7f34840095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8cce757c5a0b8c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.415+0000 7f348a7fc640 1 -- 172.21.15.27:0/1117255990 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34840157c0 con 0x55a2324cc390 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.415+0000 7f348a7fc640 1 -- 172.21.15.27:0/1117255990 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348401d070 con 0x55a2324cc390 2024-04-25T13:55:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.415+0000 7f348a7fc640 1 -- 172.21.15.27:0/1117255990 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484018b30 con 0x55a2324cc390 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 -- 172.21.15.27:0/1117255990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 msgr2=0x55a23286e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23286e810 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f34840177a0 tx=0x7f34840095c0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 -- 172.21.15.27:0/1117255990 shutdown_connections 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a23286fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23286ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 --2- 172.21.15.27:0/1117255990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23286e810 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 -- 172.21.15.27:0/1117255990 >> 172.21.15.27:0/1117255990 conn(0x55a2324c04e0 msgr2=0x55a23286a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 -- 172.21.15.27:0/1117255990 shutdown_connections 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 -- 172.21.15.27:0/1117255990 wait complete. 2024-04-25T13:55:37.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.416+0000 7f34a1436740 1 -- start start 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34a1436740 1 Processor -- start 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34a1436740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a232878500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34a1436740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a232878a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34a1436740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23287f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34a1436740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55a232870520 con 0x55a2324ab6b0 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34a1436740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55a232759dc0 con 0x55a2324a34d0 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34a1436740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55a2328706a0 con 0x55a2324cc390 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a232878500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34a1436740 10 client.0 _ll_register_callbacks cb 0x55a232775600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a232878500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34120/0 (socket says 172.21.15.27:34120) 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34937fe640 1 -- 172.21.15.27:0/3789554634 learned_addr learned my addr 172.21.15.27:0/3789554634 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f3493fff640 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a232878a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.417+0000 7f34937fe640 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23287f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34937fe640 1 -- 172.21.15.27:0/3789554634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 msgr2=0x55a23287f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34937fe640 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23287f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34937fe640 1 -- 172.21.15.27:0/3789554634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 msgr2=0x55a232878a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34937fe640 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a232878a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34937fe640 1 -- 172.21.15.27:0/3789554634 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2324e43f0 con 0x55a2324a34d0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34937fe640 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a232878500 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3484018010 tx=0x7f34840069e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89fc3fe4779ebea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34737fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34737fe640 1 -- 172.21.15.27:0/3789554634 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34840159c0 con 0x55a2324a34d0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34a1436740 1 -- 172.21.15.27:0/3789554634 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f348401d070 con 0x55a2324a34d0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.418+0000 7f34737fe640 1 -- 172.21.15.27:0/3789554634 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f3484015b60 con 0x55a2324a34d0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.419+0000 7f34737fe640 1 -- 172.21.15.27:0/3789554634 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f3484037050 con 0x55a2324a34d0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.419+0000 7f34a1436740 10 client.27956 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.419+0000 7f34a1436740 1 -- 172.21.15.27:0/3789554634 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55a2325ae7d0 con 0x55a2324a34d0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.419+0000 7f34737fe640 1 -- 172.21.15.27:0/3789554634 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348401d070 con 0x55a2324a34d0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.419+0000 7f34737fe640 1 -- 172.21.15.27:0/3789554634 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f348402e070 con 0x55a2324a34d0 2024-04-25T13:55:37.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.419+0000 7f34a1436740 10 client.27956 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.419+0000 7f34a1436740 10 client.27956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.419+0000 7f34a1436740 20 client.27956 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.420+0000 7f34a1436740 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a23248af60 0x55a232885e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.420+0000 7f34a1436740 4 client.27956 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.420+0000 7f34a1436740 1 -- 172.21.15.27:0/3789554634 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2327e0990 con 0x55a23248af60 2024-04-25T13:55:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.420+0000 7f3493fff640 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a23248af60 0x55a232885e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.420+0000 7f3493fff640 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a23248af60 0x55a232885e50 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e5d5fda4f0c4e86 server_cookie=89157be99ea0a627 in_seq=0 out_seq=0 2024-04-25T13:55:37.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.420+0000 7f34737fe640 10 client.27956 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.421+0000 7f34737fe640 1 -- 172.21.15.27:0/3789554634 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a2327e0990 con 0x55a23248af60 2024-04-25T13:55:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.421+0000 7f34737fe640 10 client.27956 handle_command_reply: tid=0 2024-04-25T13:55:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.421+0000 7f34a1436740 1 client.27956 shutdown 2024-04-25T13:55:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 -- 172.21.15.27:0/3789554634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 msgr2=0x55a232878500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a232878500 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3484018010 tx=0x7f34840069e0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 -- 172.21.15.27:0/3789554634 shutdown_connections 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a23248af60 0x55a232885e50 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a2324cc390 0x55a23287f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a232878a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34737fe640 0 client.27956 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 --2- 172.21.15.27:0/3789554634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a232878500 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 -- 172.21.15.27:0/3789554634 >> 172.21.15.27:0/3789554634 conn(0x55a2324c04e0 msgr2=0x55a23286cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 -- 172.21.15.27:0/3789554634 shutdown_connections 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 1 -- 172.21.15.27:0/3789554634 wait complete. 2024-04-25T13:55:37.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.422+0000 7f34a1436740 20 client.27956 trim_cache size 0 max 16384 2024-04-25T13:55:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.443+0000 7f34a1436740 1 Processor -- start 2024-04-25T13:55:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.443+0000 7f34a1436740 1 -- start start 2024-04-25T13:55:37.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.443+0000 7f34a1436740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a2328780d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.443+0000 7f34a1436740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324a34d0 0x55a232878610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.443+0000 7f34937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a2328780d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.443+0000 7f34937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a2328780d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:42976/0 (socket says 172.21.15.27:42976) 2024-04-25T13:55:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.443+0000 7f34937fe640 1 -- 172.21.15.27:0/2527223043 learned_addr learned my addr 172.21.15.27:0/2527223043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f34a1436740 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324ab6b0 0x55a23288a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f34a1436740 1 -- 172.21.15.27:0/2527223043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b7fe0 con 0x55a2324a34d0 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f3493fff640 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324a34d0 0x55a232878610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f34a1436740 1 -- 172.21.15.27:0/2527223043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34941ba860 con 0x55a2324ab6b0 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f34a1436740 1 -- 172.21.15.27:0/2527223043 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b8650 con 0x55a23248af60 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f34937fe640 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324ab6b0 0x55a23288a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f3493fff640 1 -- 172.21.15.27:0/2527223043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 msgr2=0x55a2328780d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f3493fff640 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a2328780d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f3493fff640 1 -- 172.21.15.27:0/2527223043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324ab6b0 msgr2=0x55a23288a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f3493fff640 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324ab6b0 0x55a23288a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f3493fff640 1 -- 172.21.15.27:0/2527223043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a232876610 con 0x55a2324a34d0 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.444+0000 7f34937fe640 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a2328780d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.445+0000 7f3493fff640 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324a34d0 0x55a232878610 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x55a2328864e0 tx=0x7f348c009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1246047a9c884f4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.445+0000 7f3489ffb640 1 -- 172.21.15.27:0/2527223043 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c00adb0 con 0x55a2324a34d0 2024-04-25T13:55:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.445+0000 7f3489ffb640 1 -- 172.21.15.27:0/2527223043 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348c017de0 con 0x55a2324a34d0 2024-04-25T13:55:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.445+0000 7f3489ffb640 1 -- 172.21.15.27:0/2527223043 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c024420 con 0x55a2324a34d0 2024-04-25T13:55:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 -- 172.21.15.27:0/2527223043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324a34d0 msgr2=0x55a232878610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324a34d0 0x55a232878610 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x55a2328864e0 tx=0x7f348c009830 comp rx=0 tx=0).stop 2024-04-25T13:55:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 -- 172.21.15.27:0/2527223043 shutdown_connections 2024-04-25T13:55:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324ab6b0 0x55a23288a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324a34d0 0x55a232878610 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 --2- 172.21.15.27:0/2527223043 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a2328780d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 -- 172.21.15.27:0/2527223043 >> 172.21.15.27:0/2527223043 conn(0x55a2324c04e0 msgr2=0x55a232873250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 -- 172.21.15.27:0/2527223043 shutdown_connections 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 -- 172.21.15.27:0/2527223043 wait complete. 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 -- start start 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.446+0000 7f34a1436740 1 Processor -- start 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34a1436740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a23287e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34a1436740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23287e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34a1436740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a2328ec270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34a1436740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b8d20 con 0x55a2324ab6b0 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34a1436740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34941ba190 con 0x55a2324a34d0 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34a1436740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b93f0 con 0x55a23248af60 2024-04-25T13:55:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34a1436740 10 client.0 _ll_register_callbacks cb 0x7f3484009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34937fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a23287e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f3493fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23287e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34937fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a2328ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f3493fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23287e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34140/0 (socket says 172.21.15.27:34140) 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f3493fff640 1 -- 172.21.15.27:0/710060055 learned_addr learned my addr 172.21.15.27:0/710060055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34937fe640 1 -- 172.21.15.27:0/710060055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 msgr2=0x55a23287e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34937fe640 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23287e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34937fe640 1 -- 172.21.15.27:0/710060055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 msgr2=0x55a2328ec270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34937fe640 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a2328ec270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.447+0000 7f34937fe640 1 -- 172.21.15.27:0/710060055 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2328ec860 con 0x55a23248af60 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.448+0000 7f34937fe640 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a23287e2d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3484000be0 tx=0x7f34840069e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0e746fa7c307d27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.448+0000 7f34737fe640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.448+0000 7f34737fe640 1 -- 172.21.15.27:0/710060055 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484022810 con 0x55a23248af60 2024-04-25T13:55:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.448+0000 7f34737fe640 1 -- 172.21.15.27:0/710060055 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f34840229b0 con 0x55a23248af60 2024-04-25T13:55:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.448+0000 7f34a1436740 1 -- 172.21.15.27:0/710060055 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f348c017de0 con 0x55a23248af60 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34737fe640 1 -- 172.21.15.27:0/710060055 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f34840213f0 con 0x55a23248af60 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34a1436740 10 client.28033 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34a1436740 1 -- 172.21.15.27:0/710060055 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55a2328edae0 con 0x55a23248af60 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34737fe640 1 -- 172.21.15.27:0/710060055 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f348c017de0 con 0x55a23248af60 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34737fe640 1 -- 172.21.15.27:0/710060055 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f348402e040 con 0x55a23248af60 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34a1436740 10 client.28033 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34a1436740 10 client.28033 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34a1436740 20 client.28033 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34a1436740 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a2324cc390 0x55a2328f04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34a1436740 4 client.28033 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.449+0000 7f34a1436740 1 -- 172.21.15.27:0/710060055 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a2327da800 con 0x55a2324cc390 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f3493fff640 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a2324cc390 0x55a2328f04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34737fe640 10 client.28033 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f3493fff640 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a2324cc390 0x55a2328f04c0 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9668991900c03ec server_cookie=80791aa79ba7ba3e in_seq=0 out_seq=0 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34737fe640 1 -- 172.21.15.27:0/710060055 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a2327da800 con 0x55a2324cc390 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34737fe640 10 client.28033 handle_command_reply: tid=0 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 client.28033 shutdown 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 -- 172.21.15.27:0/710060055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 msgr2=0x55a23287e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a23287e2d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3484000be0 tx=0x7f34840069e0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 -- 172.21.15.27:0/710060055 shutdown_connections 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55a2324cc390 0x55a2328f04c0 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a2324ab6b0 0x55a2328ec270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a2324a34d0 0x55a23287e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 --2- 172.21.15.27:0/710060055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55a23248af60 0x55a23287e2d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34737fe640 0 client.28033 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 -- 172.21.15.27:0/710060055 >> 172.21.15.27:0/710060055 conn(0x55a2324c04e0 msgr2=0x55a2328780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.450+0000 7f34a1436740 1 -- 172.21.15.27:0/710060055 shutdown_connections 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.451+0000 7f34a1436740 1 -- 172.21.15.27:0/710060055 wait complete. 2024-04-25T13:55:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.451+0000 7f34a1436740 20 client.28033 trim_cache size 0 max 16384 2024-04-25T13:55:37.454 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 -- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f34800afbb0 msgr2=0x7f34800b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f34800afbb0 0x7f34800b20c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f347c0c3a60 tx=0x7f347c009260 comp rx=0 tx=0).stop 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 -- 172.21.15.27:0/963669325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3494110030 msgr2=0x7f34941b06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3494110030 0x7f34941b06f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f347c009230 tx=0x7f347c003cd0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 -- 172.21.15.27:0/963669325 shutdown_connections 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f34800afbb0 0x7f34800b20c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3494110030 0x7f34941b06f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f349410b310 0x7f34941a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 --2- 172.21.15.27:0/963669325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f349410a420 0x7f34941a9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 -- 172.21.15.27:0/963669325 >> 172.21.15.27:0/963669325 conn(0x7f34940704e0 msgr2=0x7f349410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.452+0000 7f349a4a8640 1 -- 172.21.15.27:0/963669325 shutdown_connections 2024-04-25T13:55:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:37.453+0000 7f349a4a8640 1 -- 172.21.15.27:0/963669325 wait complete. 2024-04-25T13:55:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 870.0 s, 1865.2 tps, lat 53.406 ms stddev 60.742 2024-04-25T13:55:38.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:37 smithi122 ceph-mon[32843]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 1.58k op/s 2024-04-25T13:55:38.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:37 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3033219713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:55:38.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:37 smithi045 ceph-mon[32731]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 1.58k op/s 2024-04-25T13:55:38.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:37 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3033219713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:55:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:37 smithi027 ceph-mon[29140]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 1.58k op/s 2024-04-25T13:55:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:37 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3033219713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:55:38.467 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:55:38.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:55:38.468 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:55:38.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.615+0000 7fdaf304a640 1 -- 172.21.15.27:0/3147523996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec103c30 msgr2=0x7fdaec104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.615+0000 7fdaf304a640 1 --2- 172.21.15.27:0/3147523996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec103c30 0x7fdaec104040 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009900 tx=0x7fdad802a410 comp rx=0 tx=0).stop 2024-04-25T13:55:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.616+0000 7fdaf304a640 1 -- 172.21.15.27:0/3147523996 shutdown_connections 2024-04-25T13:55:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.616+0000 7fdaf304a640 1 --2- 172.21.15.27:0/3147523996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaec108930 0x7fdaec10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.616+0000 7fdaf304a640 1 --2- 172.21.15.27:0/3147523996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec103c30 0x7fdaec104040 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.616+0000 7fdaf304a640 1 --2- 172.21.15.27:0/3147523996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaec1029f0 0x7fdaec1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.616+0000 7fdaf304a640 1 -- 172.21.15.27:0/3147523996 >> 172.21.15.27:0/3147523996 conn(0x7fdaec0fdc00 msgr2=0x7fdaec100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.617+0000 7fdaf304a640 1 -- 172.21.15.27:0/3147523996 shutdown_connections 2024-04-25T13:55:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.617+0000 7fdaf304a640 1 -- 172.21.15.27:0/3147523996 wait complete. 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf304a640 1 Processor -- start 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf304a640 1 -- start start 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf304a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec1029f0 0x7fdaec197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf304a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaec103c30 0x7fdaec198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf304a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaec108930 0x7fdaec19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf304a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec10d410 con 0x7fdaec1029f0 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf304a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec10d290 con 0x7fdaec108930 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf304a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec10d590 con 0x7fdaec103c30 2024-04-25T13:55:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf0de6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec1029f0 0x7fdaec197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf0de6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec1029f0 0x7fdaec197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42400/0 (socket says 172.21.15.27:42400) 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf0de6640 1 -- 172.21.15.27:0/1819697139 learned_addr learned my addr 172.21.15.27:0/1819697139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaec108930 0x7fdaec19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.618+0000 7fdaebfff640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaec103c30 0x7fdaec198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf15e7640 1 -- 172.21.15.27:0/1819697139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaec103c30 msgr2=0x7fdaec198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaec103c30 0x7fdaec198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf15e7640 1 -- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec1029f0 msgr2=0x7fdaec197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec1029f0 0x7fdaec197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf15e7640 1 -- 172.21.15.27:0/1819697139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaec19fb30 con 0x7fdaec108930 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec1029f0 0x7fdaec197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaebfff640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaec103c30 0x7fdaec198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaec108930 0x7fdaec19f4c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdae400ed90 tx=0x7fdae4012490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdae9ffb640 1 -- 172.21.15.27:0/1819697139 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae401c070 con 0x7fdaec108930 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdae9ffb640 1 -- 172.21.15.27:0/1819697139 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae4005020 con 0x7fdaec108930 2024-04-25T13:55:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf304a640 1 -- 172.21.15.27:0/1819697139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaec19fe20 con 0x7fdaec108930 2024-04-25T13:55:38.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdae9ffb640 1 -- 172.21.15.27:0/1819697139 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae4018660 con 0x7fdaec108930 2024-04-25T13:55:38.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.619+0000 7fdaf304a640 1 -- 172.21.15.27:0/1819697139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaec1a0230 con 0x7fdaec108930 2024-04-25T13:55:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.622+0000 7fdae9ffb640 1 -- 172.21.15.27:0/1819697139 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdae4017070 con 0x7fdaec108930 2024-04-25T13:55:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.623+0000 7fdaf9fd8740 1 Processor -- start 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf9fd8740 1 -- start start 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49b474a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b47a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 0x562c49b4e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562c49aacb00 con 0x562c497a6410 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562c49a48920 con 0x562c4977d550 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562c49aa8ac0 con 0x562c49785730 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49b474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf0de6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b47a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaebfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49b474a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42416/0 (socket says 172.21.15.27:42416) 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaf0de6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b47a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34176/0 (socket says 172.21.15.27:34176) 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaebfff640 1 -- 172.21.15.27:0/355414573 learned_addr learned my addr 172.21.15.27:0/355414573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.624+0000 7fdaebfff640 1 --2- 172.21.15.27:0/355414573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 0x562c49b4e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdaf0de6640 1 -- 172.21.15.27:0/355414573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 msgr2=0x562c49b4e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/355414573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 0x562c49b4e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdaf0de6640 1 -- 172.21.15.27:0/355414573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 msgr2=0x562c49b474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/355414573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49b474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdaf0de6640 1 -- 172.21.15.27:0/355414573 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c498867e0 con 0x562c4977d550 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/355414573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b47a20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdae000df20 tx=0x7fdae0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=909c4da4c8ec4093 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdae9ffb640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdadc0afb70 0x7fdadc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdad6ffd640 1 -- 172.21.15.27:0/355414573 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae001d070 con 0x562c4977d550 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdad6ffd640 1 -- 172.21.15.27:0/355414573 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae0004250 con 0x562c4977d550 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdad6ffd640 1 -- 172.21.15.27:0/355414573 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0018750 con 0x562c4977d550 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdaebfff640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdadc0afb70 0x7fdadc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.625+0000 7fdae9ffb640 1 -- 172.21.15.27:0/1819697139 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdae40e2b00 con 0x7fdaec108930 2024-04-25T13:55:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/355414573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 msgr2=0x562c49b47a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaebfff640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdadc0afb70 0x7fdadc0b2080 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdad8000c00 tx=0x7fdad8032ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/355414573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b47a20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdae000df20 tx=0x7fdae0005f90 comp rx=0 tx=0).stop 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/355414573 shutdown_connections 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/355414573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 0x562c49b4e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/355414573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b47a20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/355414573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49b474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/355414573 >> 172.21.15.27:0/355414573 conn(0x562c4979a560 msgr2=0x562c49b42e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/355414573 shutdown_connections 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.626+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/355414573 wait complete. 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 1 -- start start 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 1 Processor -- start 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c4977d550 0x562c49b51270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c49785730 0x562c49b517b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c497a6410 0x562c49b58710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562c49b4f370 con 0x562c49785730 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562c49a465d0 con 0x562c497a6410 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562c49b4f4f0 con 0x562c4977d550 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.627+0000 7fdaf9fd8740 10 client.0 _ll_register_callbacks cb 0x562c49a34080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c497a6410 0x562c49b58710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c497a6410 0x562c49b58710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34182/0 (socket says 172.21.15.27:34182) 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 -- 172.21.15.27:0/357445533 learned_addr learned my addr 172.21.15.27:0/357445533 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaebfff640 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c49785730 0x562c49b517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c4977d550 0x562c49b51270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 -- 172.21.15.27:0/357445533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c4977d550 msgr2=0x562c49b51270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c4977d550 0x562c49b51270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 -- 172.21.15.27:0/357445533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c49785730 msgr2=0x562c49b517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c49785730 0x562c49b517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 -- 172.21.15.27:0/357445533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c498e6f70 con 0x562c497a6410 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c497a6410 0x562c49b58710 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdae40aaa70 tx=0x7fdae40ab7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d9a4f32652418e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdacbfff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdacbfff640 1 -- 172.21.15.27:0/357445533 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae40c4020 con 0x562c497a6410 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdacbfff640 1 -- 172.21.15.27:0/357445533 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdae40054d0 con 0x562c497a6410 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/357445533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae0004250 con 0x562c497a6410 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.628+0000 7fdacbfff640 1 -- 172.21.15.27:0/357445533 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdae40b3e10 con 0x562c497a6410 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf9fd8740 10 client.27974 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/357445533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562c4977e170 con 0x562c497a6410 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdacbfff640 1 -- 172.21.15.27:0/357445533 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae0004250 con 0x562c497a6410 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdacbfff640 1 -- 172.21.15.27:0/357445533 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fdae40a9e10 con 0x562c497a6410 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf9fd8740 10 client.27974 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf9fd8740 10 client.27974 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf9fd8740 20 client.27974 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562c497651f0 0x562c49b5ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf9fd8740 4 client.27974 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/357445533 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c49aba710 con 0x562c497651f0 2024-04-25T13:55:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.629+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562c497651f0 0x562c49b5ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.630+0000 7fdacbfff640 10 client.27974 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.630+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562c497651f0 0x562c49b5ec30 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73db181d9aadd01c server_cookie=d5e2ad8dd25940ee in_seq=0 out_seq=0 2024-04-25T13:55:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.630+0000 7fdacbfff640 1 -- 172.21.15.27:0/357445533 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562c49aba710 con 0x562c497651f0 2024-04-25T13:55:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.630+0000 7fdacbfff640 10 client.27974 handle_command_reply: tid=0 2024-04-25T13:55:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.630+0000 7fdaf9fd8740 1 client.27974 shutdown 2024-04-25T13:55:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/357445533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c497a6410 msgr2=0x562c49b58710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c497a6410 0x562c49b58710 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdae40aaa70 tx=0x7fdae40ab7f0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/357445533 shutdown_connections 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562c497651f0 0x562c49b5ec30 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c497a6410 0x562c49b58710 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c49785730 0x562c49b517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/357445533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c4977d550 0x562c49b51270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/357445533 >> 172.21.15.27:0/357445533 conn(0x562c4979a560 msgr2=0x562c49b41ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.631+0000 7fdacbfff640 0 client.27974 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.632+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/357445533 shutdown_connections 2024-04-25T13:55:38.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.632+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/357445533 wait complete. 2024-04-25T13:55:38.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.632+0000 7fdaf9fd8740 20 client.27974 trim_cache size 0 max 16384 2024-04-25T13:55:38.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.667+0000 7fdaf9fd8740 1 Processor -- start 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.667+0000 7fdaf9fd8740 1 -- start start 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.667+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c4977d550 0x562c49b4ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.667+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c49785730 0x562c49b427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.667+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c497a6410 0x562c49b42d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.667+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1a6f90 con 0x562c4977d550 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.667+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1a9810 con 0x562c49785730 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.667+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1a7600 con 0x562c497a6410 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.668+0000 7fdaf15e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c497a6410 0x562c49b42d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.668+0000 7fdaf15e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c497a6410 0x562c49b42d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43028/0 (socket says 172.21.15.27:43028) 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.668+0000 7fdaf15e7640 1 -- 172.21.15.27:0/1328067758 learned_addr learned my addr 172.21.15.27:0/1328067758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.669+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c4977d550 0x562c49b4ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.672+0000 7fdaebfff640 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c49785730 0x562c49b427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.674+0000 7fdaf15e7640 1 -- 172.21.15.27:0/1328067758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c49785730 msgr2=0x562c49b427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.674+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c49785730 0x562c49b427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.674+0000 7fdaf15e7640 1 -- 172.21.15.27:0/1328067758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c4977d550 msgr2=0x562c49b4ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.674+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c4977d550 0x562c49b4ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.674+0000 7fdaf15e7640 1 -- 172.21.15.27:0/1328067758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c49b4c120 con 0x562c497a6410 2024-04-25T13:55:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.678+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c497a6410 0x562c49b42d30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdae4012990 tx=0x7fdae40aa7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a4734c6a897838b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.679+0000 7fdad67fc640 1 -- 172.21.15.27:0/1328067758 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae40c4070 con 0x562c497a6410 2024-04-25T13:55:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.679+0000 7fdad67fc640 1 -- 172.21.15.27:0/1328067758 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae40054d0 con 0x562c497a6410 2024-04-25T13:55:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.679+0000 7fdad67fc640 1 -- 172.21.15.27:0/1328067758 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae40a6b80 con 0x562c497a6410 2024-04-25T13:55:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.680+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/1328067758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c497a6410 msgr2=0x562c49b42d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.680+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c497a6410 0x562c49b42d30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdae4012990 tx=0x7fdae40aa7b0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.684+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/1328067758 shutdown_connections 2024-04-25T13:55:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.684+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c497a6410 0x562c49b42d30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.684+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c49785730 0x562c49b427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.684+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/1328067758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c4977d550 0x562c49b4ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.684+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/1328067758 >> 172.21.15.27:0/1328067758 conn(0x562c4979a560 msgr2=0x562c49b4f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.692+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/1328067758 shutdown_connections 2024-04-25T13:55:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.696+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/1328067758 wait complete. 2024-04-25T13:55:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.696+0000 7fdaf9fd8740 1 -- start start 2024-04-25T13:55:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.696+0000 7fdaf9fd8740 1 Processor -- start 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.697+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b55f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.697+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 0x562c49b564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.697+0000 7fdaf9fd8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49bc4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.697+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1a7cd0 con 0x562c497a6410 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.697+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1a9140 con 0x562c4977d550 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.697+0000 7fdaf9fd8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1a83a0 con 0x562c49785730 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.697+0000 7fdaf9fd8740 10 client.0 _ll_register_callbacks cb 0x7fdae40aab30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.698+0000 7fdaf0de6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b55f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.698+0000 7fdaf0de6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b55f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34198/0 (socket says 172.21.15.27:34198) 2024-04-25T13:55:38.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.698+0000 7fdaf0de6640 1 -- 172.21.15.27:0/3488312740 learned_addr learned my addr 172.21.15.27:0/3488312740 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.698+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49bc4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.700+0000 7fdaebfff640 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 0x562c49b564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.700+0000 7fdaf15e7640 1 -- 172.21.15.27:0/3488312740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 msgr2=0x562c49b564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.700+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 0x562c49b564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.700+0000 7fdaf15e7640 1 -- 172.21.15.27:0/3488312740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 msgr2=0x562c49b55f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.700+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b55f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.700+0000 7fdaf15e7640 1 -- 172.21.15.27:0/3488312740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c49bc54a0 con 0x562c497a6410 2024-04-25T13:55:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.707+0000 7fdaf15e7640 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49bc4f60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fdae40a7550 tx=0x7fdae40aa420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4281f5ff3a73e387 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.709+0000 7fdacbfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:55:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.713+0000 7fdacbfff640 1 -- 172.21.15.27:0/3488312740 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae40c4070 con 0x562c497a6410 2024-04-25T13:55:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.713+0000 7fdacbfff640 1 -- 172.21.15.27:0/3488312740 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fdae40a9e80 con 0x562c497a6410 2024-04-25T13:55:38.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.714+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/3488312740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae40054d0 con 0x562c497a6410 2024-04-25T13:55:38.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.717+0000 7fdacbfff640 1 -- 172.21.15.27:0/3488312740 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdae4005680 con 0x562c497a6410 2024-04-25T13:55:38.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.721+0000 7fdaf9fd8740 10 client.18324 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:38.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.721+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/3488312740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562c49bc54a0 con 0x562c497a6410 2024-04-25T13:55:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.726+0000 7fdacbfff640 1 -- 172.21.15.27:0/3488312740 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae40054d0 con 0x562c497a6410 2024-04-25T13:55:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.726+0000 7fdacbfff640 1 -- 172.21.15.27:0/3488312740 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fdae40b04f0 con 0x562c497a6410 2024-04-25T13:55:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.727+0000 7fdaf9fd8740 10 client.18324 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.727+0000 7fdaf9fd8740 10 client.18324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.727+0000 7fdaf9fd8740 20 client.18324 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:38.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.727+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562c4973a630 0x562c49bc8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.727+0000 7fdaf9fd8740 4 client.18324 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:55:38.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.727+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/3488312740 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562c49ab4580 con 0x562c4973a630 2024-04-25T13:55:38.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.732+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562c4973a630 0x562c49bc8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.736+0000 7fdaf0de6640 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562c4973a630 0x562c49bc8a50 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8814c88bfc822 server_cookie=af75d2ac82dfec88 in_seq=0 out_seq=0 2024-04-25T13:55:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.736+0000 7fdacbfff640 10 client.18324 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:38.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.738+0000 7fdacbfff640 1 -- 172.21.15.27:0/3488312740 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562c49ab4580 con 0x562c4973a630 2024-04-25T13:55:38.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.738+0000 7fdacbfff640 10 client.18324 handle_command_reply: tid=0 2024-04-25T13:55:38.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.739+0000 7fdaf9fd8740 1 client.18324 shutdown 2024-04-25T13:55:38.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.741+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/3488312740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 msgr2=0x562c49bc4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.741+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49bc4f60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fdae40a7550 tx=0x7fdae40aa420 comp rx=0 tx=0).stop 2024-04-25T13:55:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.743+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/3488312740 shutdown_connections 2024-04-25T13:55:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.743+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562c4973a630 0x562c49bc8a50 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.743+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562c497a6410 0x562c49bc4f60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.743+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562c49785730 0x562c49b564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.743+0000 7fdaf9fd8740 1 --2- 172.21.15.27:0/3488312740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562c4977d550 0x562c49b55f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.743+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/3488312740 >> 172.21.15.27:0/3488312740 conn(0x562c4979a560 msgr2=0x562c49b45c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.743+0000 7fdacbfff640 0 client.18324 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.745+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/3488312740 shutdown_connections 2024-04-25T13:55:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.745+0000 7fdaf9fd8740 1 -- 172.21.15.27:0/3488312740 wait complete. 2024-04-25T13:55:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.746+0000 7fdaf9fd8740 20 client.18324 trim_cache size 0 max 16384 2024-04-25T13:55:38.746 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:55:38.746 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:55:38.747 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "ded7a5f7-f87d-4811-9e36-c52d0534fad4", 2024-04-25T13:55:38.747 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:55:38.747 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:55:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.750+0000 7fdaf304a640 1 -- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdadc0afb70 msgr2=0x7fdadc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.750+0000 7fdaf304a640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdadc0afb70 0x7fdadc0b2080 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdad8000c00 tx=0x7fdad8032ed0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.750+0000 7fdaf304a640 1 -- 172.21.15.27:0/1819697139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaec108930 msgr2=0x7fdaec19f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.750+0000 7fdaf304a640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaec108930 0x7fdaec19f4c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdae400ed90 tx=0x7fdae4012490 comp rx=0 tx=0).stop 2024-04-25T13:55:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.751+0000 7fdaf304a640 1 -- 172.21.15.27:0/1819697139 shutdown_connections 2024-04-25T13:55:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.751+0000 7fdaf304a640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fdadc0afb70 0x7fdadc0b2080 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.751+0000 7fdaf304a640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaec108930 0x7fdaec19f4c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.751+0000 7fdaf304a640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdaec103c30 0x7fdaec198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.751+0000 7fdaf304a640 1 --2- 172.21.15.27:0/1819697139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdaec1029f0 0x7fdaec197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.751+0000 7fdaf304a640 1 -- 172.21.15.27:0/1819697139 >> 172.21.15.27:0/1819697139 conn(0x7fdaec0fdc00 msgr2=0x7fdaec104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.752+0000 7fdaf304a640 1 -- 172.21.15.27:0/1819697139 shutdown_connections 2024-04-25T13:55:38.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.752+0000 7fdaf304a640 1 -- 172.21.15.27:0/1819697139 wait complete. 2024-04-25T13:55:38.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:55:38.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:55:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.944+0000 7fb5f52af640 1 -- 172.21.15.27:0/1362820408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0108930 msgr2=0x7fb5f010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 --2- 172.21.15.27:0/1362820408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0108930 0x7fb5f010ad50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009ca0 tx=0x7fb5e802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 -- 172.21.15.27:0/1362820408 shutdown_connections 2024-04-25T13:55:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 --2- 172.21.15.27:0/1362820408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0108930 0x7fb5f010ad50 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 --2- 172.21.15.27:0/1362820408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f0103c30 0x7fb5f0104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 --2- 172.21.15.27:0/1362820408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f01029f0 0x7fb5f01036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 -- 172.21.15.27:0/1362820408 >> 172.21.15.27:0/1362820408 conn(0x7fb5f00fdc00 msgr2=0x7fb5f0100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 -- 172.21.15.27:0/1362820408 shutdown_connections 2024-04-25T13:55:38.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 -- 172.21.15.27:0/1362820408 wait complete. 2024-04-25T13:55:38.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.945+0000 7fb5f52af640 1 Processor -- start 2024-04-25T13:55:38.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5f52af640 1 -- start start 2024-04-25T13:55:38.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5f52af640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f01029f0 0x7fb5f0199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5f52af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0103c30 0x7fb5f01995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5f52af640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0108930 0x7fb5f01a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5f52af640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f010e550 con 0x7fb5f01029f0 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5f52af640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f010e3d0 con 0x7fb5f0108930 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f01029f0 0x7fb5f0199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5f52af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f010e6d0 con 0x7fb5f0103c30 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5ee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0103c30 0x7fb5f01995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f01029f0 0x7fb5f0199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42460/0 (socket says 172.21.15.27:42460) 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5eeffd640 1 -- 172.21.15.27:0/411392126 learned_addr learned my addr 172.21.15.27:0/411392126 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.946+0000 7fb5ef7fe640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0108930 0x7fb5f01a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.947+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/411392126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0108930 msgr2=0x7fb5f01a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.947+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0108930 0x7fb5f01a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.947+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f01029f0 msgr2=0x7fb5f0199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.947+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f01029f0 0x7fb5f0199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.947+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/411392126 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f01a0bd0 con 0x7fb5f0103c30 2024-04-25T13:55:38.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.947+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0103c30 0x7fb5f01995d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009f10 tx=0x7fb5e400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.949+0000 7fb5d7fff640 1 -- 172.21.15.27:0/411392126 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40159e0 con 0x7fb5f0103c30 2024-04-25T13:55:38.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.949+0000 7fb5d7fff640 1 -- 172.21.15.27:0/411392126 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e400cd60 con 0x7fb5f0103c30 2024-04-25T13:55:38.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.949+0000 7fb5d7fff640 1 -- 172.21.15.27:0/411392126 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40108f0 con 0x7fb5f0103c30 2024-04-25T13:55:38.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.950+0000 7fb5f52af640 1 -- 172.21.15.27:0/411392126 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f006b2f0 con 0x7fb5f0103c30 2024-04-25T13:55:38.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.950+0000 7fb5f52af640 1 -- 172.21.15.27:0/411392126 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f006b7f0 con 0x7fb5f0103c30 2024-04-25T13:55:38.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.953+0000 7fb5d7fff640 1 -- 172.21.15.27:0/411392126 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb5e4015b80 con 0x7fb5f0103c30 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.953+0000 7fb5fc167740 1 Processor -- start 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.953+0000 7fb5fc167740 1 -- start start 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.953+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cde390 0x56526607fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cb54d0 0x56526607ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266080c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5fc167740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565265fe4d80 con 0x565265cbd6b0 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5fc167740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x565265f9e8b0 con 0x565265cde390 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5fc167740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x565265fe0d40 con 0x565265cb54d0 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266080c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266080c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42462/0 (socket says 172.21.15.27:42462) 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cde390 0x56526607fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/4044596448 learned_addr learned my addr 172.21.15.27:0/4044596448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cb54d0 0x56526607ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5d7fff640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afc40 0x7fb5d80b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/4044596448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cb54d0 msgr2=0x56526607ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cb54d0 0x56526607ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/4044596448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cde390 msgr2=0x56526607fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cde390 0x56526607fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5d7fff640 1 -- 172.21.15.27:0/411392126 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5e40e30d0 con 0x7fb5f0103c30 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.954+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/4044596448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565265f6fa40 con 0x565265cbd6b0 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.955+0000 7fb5ef7fe640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afc40 0x7fb5d80b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.955+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cb54d0 0x56526607ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.955+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266080c70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40e0c00 tx=0x7fb5e400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2227bfe8b6b2190a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.955+0000 7fb5d57fa640 1 -- 172.21.15.27:0/4044596448 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40ab3b0 con 0x565265cbd6b0 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.955+0000 7fb5d57fa640 1 -- 172.21.15.27:0/4044596448 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e40ab0f0 con 0x565265cbd6b0 2024-04-25T13:55:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.955+0000 7fb5d57fa640 1 -- 172.21.15.27:0/4044596448 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40aad50 con 0x565265cbd6b0 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.956+0000 7fb5ef7fe640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afc40 0x7fb5d80b2150 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb5e802f540 tx=0x7fb5e802a8c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.956+0000 7fb5fc167740 1 -- 172.21.15.27:0/4044596448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 msgr2=0x565266080c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.956+0000 7fb5fc167740 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266080c70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40e0c00 tx=0x7fb5e400cf10 comp rx=0 tx=0).stop 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.957+0000 7fb5fc167740 1 -- 172.21.15.27:0/4044596448 shutdown_connections 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.957+0000 7fb5fc167740 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266080c70 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.957+0000 7fb5fc167740 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cb54d0 0x56526607ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.957+0000 7fb5fc167740 1 --2- 172.21.15.27:0/4044596448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cde390 0x56526607fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.957+0000 7fb5fc167740 1 -- 172.21.15.27:0/4044596448 >> 172.21.15.27:0/4044596448 conn(0x565265cd24e0 msgr2=0x56526607b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.958+0000 7fb5fc167740 1 -- 172.21.15.27:0/4044596448 shutdown_connections 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.958+0000 7fb5fc167740 1 -- 172.21.15.27:0/4044596448 wait complete. 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.958+0000 7fb5fc167740 1 -- start start 2024-04-25T13:55:38.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.958+0000 7fb5fc167740 1 Processor -- start 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.959+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266089900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.959+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cbd6b0 0x565266089e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.959+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cde390 0x565266090da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.959+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266089900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.959+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266089900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34232/0 (socket says 172.21.15.27:34232) 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.959+0000 7fb5ee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cbd6b0 0x565266089e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.959+0000 7fb5ee7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cbd6b0 0x565266089e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43068/0 (socket says 172.21.15.27:43068) 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x565266081720 con 0x565265cde390 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5ef7fe640 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cde390 0x565266090da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5eeffd640 1 -- 172.21.15.27:0/467089868 learned_addr learned my addr 172.21.15.27:0/467089868 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x565265f6bd10 con 0x565265cb54d0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5652660818a0 con 0x565265cbd6b0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5fc167740 10 client.0 _ll_register_callbacks cb 0x565265f98210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/467089868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 msgr2=0x565266089900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266089900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/467089868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cde390 msgr2=0x565266090da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cde390 0x565266090da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5ee7fc640 1 -- 172.21.15.27:0/467089868 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565265dc07d0 con 0x565265cbd6b0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cbd6b0 0x565266089e40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40ab2e0 tx=0x7fb5e40e6900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=596e4ba24a9766ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.960+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/467089868 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40aaf30 con 0x565265cbd6b0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/467089868 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5e40b41b0 con 0x565265cbd6b0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5e40ab0f0 con 0x565265cbd6b0 2024-04-25T13:55:38.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/467089868 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb5e40ab3b0 con 0x565265cbd6b0 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 10 client.28063 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x565265cf63f0 con 0x565265cbd6b0 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/467089868 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e40ab0f0 con 0x565265cbd6b0 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/467089868 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb5e4026030 con 0x565265cbd6b0 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 10 client.28063 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 10 client.28063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 20 client.28063 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565265c9cf60 0x565266097450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 4 client.28063 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.961+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565265ff2990 con 0x565265c9cf60 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.962+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565265c9cf60 0x565266097450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.962+0000 7fb5d2ff5640 10 client.28063 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:38.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.962+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565265c9cf60 0x565266097450 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e493095c504e8530 server_cookie=b52dd2ff523a8364 in_seq=0 out_seq=0 2024-04-25T13:55:38.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.963+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/467089868 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x565265ff2990 con 0x565265c9cf60 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.963+0000 7fb5d2ff5640 10 client.28063 handle_command_reply: tid=0 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.963+0000 7fb5fc167740 1 client.28063 shutdown 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cbd6b0 msgr2=0x565266089e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cbd6b0 0x565266089e40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40ab2e0 tx=0x7fb5e40e6900 comp rx=0 tx=0).stop 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 shutdown_connections 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565265c9cf60 0x565266097450 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cde390 0x565266090da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cbd6b0 0x565266089e40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 --2- 172.21.15.27:0/467089868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266089900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 >> 172.21.15.27:0/467089868 conn(0x565265cd24e0 msgr2=0x56526607a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5d2ff5640 0 client.28063 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 shutdown_connections 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 1 -- 172.21.15.27:0/467089868 wait complete. 2024-04-25T13:55:38.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.964+0000 7fb5fc167740 20 client.28063 trim_cache size 0 max 16384 2024-04-25T13:55:38.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.992+0000 7fb5fc167740 1 Processor -- start 2024-04-25T13:55:38.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.992+0000 7fb5fc167740 1 -- start start 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266088610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266090cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266088610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266088610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34236/0 (socket says 172.21.15.27:34236) 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266091200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5eeffd640 1 -- 172.21.15.27:0/3014482165 learned_addr learned my addr 172.21.15.27:0/3014482165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5fc167740 1 -- 172.21.15.27:0/3014482165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f01a6ea0 con 0x565265cbd6b0 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5ef7fe640 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266091200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5fc167740 1 -- 172.21.15.27:0/3014482165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f01a9740 con 0x565265cb54d0 2024-04-25T13:55:38.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.993+0000 7fb5fc167740 1 -- 172.21.15.27:0/3014482165 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f01a7530 con 0x565265cde390 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.994+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266090cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.994+0000 7fb5eeffd640 1 -- 172.21.15.27:0/3014482165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 msgr2=0x565266091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.994+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266091200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.994+0000 7fb5eeffd640 1 -- 172.21.15.27:0/3014482165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 msgr2=0x565266090cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.994+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266090cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.994+0000 7fb5eeffd640 1 -- 172.21.15.27:0/3014482165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565265cb60f0 con 0x565265cb54d0 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.994+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266090cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.994+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266088610 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000beb0 tx=0x7fb5e000b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d31c2c8e71195e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.995+0000 7fb5d4ff9640 1 -- 172.21.15.27:0/3014482165 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e0004220 con 0x565265cb54d0 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.995+0000 7fb5d4ff9640 1 -- 172.21.15.27:0/3014482165 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e0024030 con 0x565265cb54d0 2024-04-25T13:55:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.995+0000 7fb5d4ff9640 1 -- 172.21.15.27:0/3014482165 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e0004db0 con 0x565265cb54d0 2024-04-25T13:55:38.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 -- 172.21.15.27:0/3014482165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 msgr2=0x565266088610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:38.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266088610 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000beb0 tx=0x7fb5e000b410 comp rx=0 tx=0).stop 2024-04-25T13:55:38.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 -- 172.21.15.27:0/3014482165 shutdown_connections 2024-04-25T13:55:38.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266091200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cbd6b0 0x565266090cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 --2- 172.21.15.27:0/3014482165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cb54d0 0x565266088610 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 -- 172.21.15.27:0/3014482165 >> 172.21.15.27:0/3014482165 conn(0x565265cd24e0 msgr2=0x565266087f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 -- 172.21.15.27:0/3014482165 shutdown_connections 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.996+0000 7fb5fc167740 1 -- 172.21.15.27:0/3014482165 wait complete. 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5fc167740 1 -- start start 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5fc167740 1 Processor -- start 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cb54d0 0x56526608d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5fc167740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cbd6b0 0x56526608dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cb54d0 0x56526608d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cb54d0 0x56526608d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42502/0 (socket says 172.21.15.27:42502) 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5eeffd640 1 -- 172.21.15.27:0/1894291081 learned_addr learned my addr 172.21.15.27:0/1894291081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5fc167740 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266090b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f01a7c00 con 0x565265cb54d0 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cbd6b0 0x56526608dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5ef7fe640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.997+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f01a9070 con 0x565265cbd6b0 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f01a82d0 con 0x565265cde390 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5fc167740 10 client.0 _ll_register_callbacks cb 0x7fb5e40ce030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5eeffd640 1 -- 172.21.15.27:0/1894291081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 msgr2=0x565266090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266090b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5eeffd640 1 -- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cbd6b0 msgr2=0x56526608dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cbd6b0 0x56526608dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5eeffd640 1 -- 172.21.15.27:0/1894291081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565265fa54b0 con 0x565265cb54d0 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cbd6b0 0x56526608dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5ef7fe640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5eeffd640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cb54d0 0x56526608d690 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000bd70 tx=0x7fb5e000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd48281ca3664553 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.998+0000 7fb5d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:55:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.999+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/1894291081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e0004350 con 0x565265cb54d0 2024-04-25T13:55:39.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.999+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/1894291081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb5e00035f0 con 0x565265cb54d0 2024-04-25T13:55:39.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.999+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5e0024030 con 0x565265cb54d0 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:38.999+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/1894291081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb5e0038050 con 0x565265cb54d0 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5fc167740 10 client.18342 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5652660fdeb0 con 0x565265cb54d0 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/1894291081 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5e0024030 con 0x565265cb54d0 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/1894291081 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fb5e00206b0 con 0x565265cb54d0 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5fc167740 10 client.18342 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5fc167740 10 client.18342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5fc167740 20 client.18342 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5fc167740 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565265c723a0 0x5652661008c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5fc167740 4 client.18342 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.000+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565265fec800 con 0x565265c723a0 2024-04-25T13:55:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.001+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565265c723a0 0x5652661008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.002+0000 7fb5ee7fc640 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565265c723a0 0x5652661008c0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9201c5d91d23c8 server_cookie=ce05126125e256da in_seq=0 out_seq=0 2024-04-25T13:55:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.002+0000 7fb5d2ff5640 10 client.18342 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.002+0000 7fb5d2ff5640 1 -- 172.21.15.27:0/1894291081 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x565265fec800 con 0x565265c723a0 2024-04-25T13:55:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.002+0000 7fb5d2ff5640 10 client.18342 handle_command_reply: tid=0 2024-04-25T13:55:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.002+0000 7fb5fc167740 1 client.18342 shutdown 2024-04-25T13:55:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.003+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cb54d0 msgr2=0x56526608d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.003+0000 7fb5fc167740 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cb54d0 0x56526608d690 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000bd70 tx=0x7fb5e000bda0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 shutdown_connections 2024-04-25T13:55:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x565265c723a0 0x5652661008c0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x565265cde390 0x565266090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565265cbd6b0 0x56526608dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5d2ff5640 0 client.18342 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 1 --2- 172.21.15.27:0/1894291081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x565265cb54d0 0x56526608d690 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 >> 172.21.15.27:0/1894291081 conn(0x565265cd24e0 msgr2=0x56526607e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 shutdown_connections 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 1 -- 172.21.15.27:0/1894291081 wait complete. 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.004+0000 7fb5fc167740 20 client.18342 trim_cache size 0 max 16384 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (40 inodes in the stack)", 2024-04-25T13:55:39.006 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:55:39.007 INFO:teuthology.orchestra.run.smithi027.stdout: "ded7a5f7-f87d-4811-9e36-c52d0534fad4": { 2024-04-25T13:55:39.007 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:55:39.007 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "ded7a5f7-f87d-4811-9e36-c52d0534fad4", 2024-04-25T13:55:39.007 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:55:39.007 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:55:39.007 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:55:39.007 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 -- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afc40 msgr2=0x7fb5d80b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afc40 0x7fb5d80b2150 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb5e802f540 tx=0x7fb5e802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 -- 172.21.15.27:0/411392126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0103c30 msgr2=0x7fb5f01995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0103c30 0x7fb5f01995d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009f10 tx=0x7fb5e400ed10 comp rx=0 tx=0).stop 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 -- 172.21.15.27:0/411392126 shutdown_connections 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb5d80afc40 0x7fb5d80b2150 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f0108930 0x7fb5f01a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5f0103c30 0x7fb5f01995d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 --2- 172.21.15.27:0/411392126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5f01029f0 0x7fb5f0199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.005+0000 7fb5f52af640 1 -- 172.21.15.27:0/411392126 >> 172.21.15.27:0/411392126 conn(0x7fb5f00fdc00 msgr2=0x7fb5f0104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.006+0000 7fb5f52af640 1 -- 172.21.15.27:0/411392126 shutdown_connections 2024-04-25T13:55:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:55:39.006+0000 7fb5f52af640 1 -- 172.21.15.27:0/411392126 wait complete. 2024-04-25T13:55:39.019 INFO:tasks.cephfs.filesystem:scrub status for tag:ded7a5f7-f87d-4811-9e36-c52d0534fad4 - {'path': '/', 'tag': 'ded7a5f7-f87d-4811-9e36-c52d0534fad4', 'options': 'recursive,force'} 2024-04-25T13:55:40.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:39 smithi122 ceph-mon[32843]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 30 MiB/s wr, 1.35k op/s 2024-04-25T13:55:40.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:39 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:55:40.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:39 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:40.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:39 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:55:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:39 smithi045 ceph-mon[32731]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 30 MiB/s wr, 1.35k op/s 2024-04-25T13:55:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:39 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:55:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:39 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:40.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:39 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:55:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:39 smithi027 ceph-mon[29140]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 30 MiB/s wr, 1.35k op/s 2024-04-25T13:55:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:39 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:55:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:39 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:39 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:55:41.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:40 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:41.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:40 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:41.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:40 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:55:42.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:41 smithi122 ceph-mon[32843]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 832 op/s 2024-04-25T13:55:42.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:41 smithi045 ceph-mon[32731]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 832 op/s 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.366+0000 7f1a50e05640 1 -- 172.21.15.27:0/8471186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a4c104d90 msgr2=0x7f1a4c1051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.366+0000 7f1a50e05640 1 --2- 172.21.15.27:0/8471186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a4c104d90 0x7f1a4c1051a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009860 tx=0x7f1a4002a340 comp rx=0 tx=0).stop 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.367+0000 7f1a50e05640 1 -- 172.21.15.27:0/8471186 shutdown_connections 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.367+0000 7f1a50e05640 1 --2- 172.21.15.27:0/8471186 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a4c109a90 0x7f1a4c10beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.367+0000 7f1a50e05640 1 --2- 172.21.15.27:0/8471186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a4c104d90 0x7f1a4c1051a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.367+0000 7f1a50e05640 1 --2- 172.21.15.27:0/8471186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a4c103b50 0x7f1a4c104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.367+0000 7f1a50e05640 1 -- 172.21.15.27:0/8471186 >> 172.21.15.27:0/8471186 conn(0x7f1a4c0fed20 msgr2=0x7f1a4c1011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.367+0000 7f1a50e05640 1 -- 172.21.15.27:0/8471186 shutdown_connections 2024-04-25T13:55:42.367 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.367+0000 7f1a50e05640 1 -- 172.21.15.27:0/8471186 wait complete. 2024-04-25T13:55:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a50e05640 1 Processor -- start 2024-04-25T13:55:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a50e05640 1 -- start start 2024-04-25T13:55:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a50e05640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a4c103b50 0x7f1a4c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a50e05640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a4c104d90 0x7f1a4c19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a50e05640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a4c109a90 0x7f1a4c1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a50e05640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c10f680 con 0x7f1a4c103b50 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a50e05640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c10f500 con 0x7f1a4c109a90 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a50e05640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c10f800 con 0x7f1a4c104d90 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a4a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a4c103b50 0x7f1a4c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a4c104d90 0x7f1a4c19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a4c104d90 0x7f1a4c19a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43102/0 (socket says 172.21.15.27:43102) 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a4a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a4c103b50 0x7f1a4c19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42530/0 (socket says 172.21.15.27:42530) 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a49d9b640 1 -- 172.21.15.27:0/1392491849 learned_addr learned my addr 172.21.15.27:0/1392491849 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a4ad9d640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a4c109a90 0x7f1a4c1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a49d9b640 1 -- 172.21.15.27:0/1392491849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a4c109a90 msgr2=0x7f1a4c1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a4c109a90 0x7f1a4c1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a49d9b640 1 -- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a4c103b50 msgr2=0x7f1a4c19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a4c103b50 0x7f1a4c19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.368+0000 7f1a49d9b640 1 -- 172.21.15.27:0/1392491849 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c1a1d10 con 0x7f1a4c104d90 2024-04-25T13:55:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.369+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a4c104d90 0x7f1a4c19a710 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1a40032d50 tx=0x7f1a40032d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.369+0000 7f1a3b7fe640 1 -- 172.21.15.27:0/1392491849 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40037040 con 0x7f1a4c104d90 2024-04-25T13:55:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.369+0000 7f1a50e05640 1 -- 172.21.15.27:0/1392491849 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4c1a1fa0 con 0x7f1a4c104d90 2024-04-25T13:55:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.369+0000 7f1a3b7fe640 1 -- 172.21.15.27:0/1392491849 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a40030c10 con 0x7f1a4c104d90 2024-04-25T13:55:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.369+0000 7f1a3b7fe640 1 -- 172.21.15.27:0/1392491849 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40003850 con 0x7f1a4c104d90 2024-04-25T13:55:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.369+0000 7f1a50e05640 1 -- 172.21.15.27:0/1392491849 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a4c1a23a0 con 0x7f1a4c104d90 2024-04-25T13:55:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.373+0000 7f1a3b7fe640 1 -- 172.21.15.27:0/1392491849 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1a40043020 con 0x7f1a4c104d90 2024-04-25T13:55:42.373 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.374+0000 7f1a3b7fe640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a340afb60 0x7f1a340b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.374+0000 7f1a3b7fe640 1 -- 172.21.15.27:0/1392491849 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a400fe250 con 0x7f1a4c104d90 2024-04-25T13:55:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.374+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a340afb60 0x7f1a340b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.375+0000 7f1a57d92740 1 Processor -- start 2024-04-25T13:55:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.375+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a340afb60 0x7f1a340b2070 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009530 tx=0x7f1a3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.375+0000 7f1a57d92740 1 -- start start 2024-04-25T13:55:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0ef1b0 0x55faef49a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0c62f0 0x55faef49cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.376 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef49f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a57d92740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55faef3f5460 con 0x55faef0c62f0 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a57d92740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55faef3af590 con 0x55faef0ce4d0 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a57d92740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55faef3f1420 con 0x55faef0ef1b0 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef49f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0c62f0 0x55faef49cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef49f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34266/0 (socket says 172.21.15.27:34266) 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 -- 172.21.15.27:0/1782904732 learned_addr learned my addr 172.21.15.27:0/1782904732 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/1782904732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0ef1b0 0x55faef49a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 -- 172.21.15.27:0/1782904732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0ef1b0 msgr2=0x55faef49a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 --2- 172.21.15.27:0/1782904732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0ef1b0 0x55faef49a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 -- 172.21.15.27:0/1782904732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0c62f0 msgr2=0x55faef49cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 --2- 172.21.15.27:0/1782904732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0c62f0 0x55faef49cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 -- 172.21.15.27:0/1782904732 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faef3bc0d0 con 0x55faef0ce4d0 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.376+0000 7f1a4ad9d640 1 --2- 172.21.15.27:0/1782904732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef49f510 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009fd0 tx=0x7f1a44011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f6806f88fa539e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.377+0000 7f1a38ff9640 1 -- 172.21.15.27:0/1782904732 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44004d20 con 0x55faef0ce4d0 2024-04-25T13:55:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.377+0000 7f1a38ff9640 1 -- 172.21.15.27:0/1782904732 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a44004030 con 0x55faef0ce4d0 2024-04-25T13:55:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.377+0000 7f1a38ff9640 1 -- 172.21.15.27:0/1782904732 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44005280 con 0x55faef0ce4d0 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.377+0000 7f1a57d92740 1 -- 172.21.15.27:0/1782904732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 msgr2=0x55faef49f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.377+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1782904732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef49f510 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009fd0 tx=0x7f1a44011810 comp rx=0 tx=0).stop 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 -- 172.21.15.27:0/1782904732 shutdown_connections 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1782904732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef49f510 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1782904732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0c62f0 0x55faef49cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1782904732 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0ef1b0 0x55faef49a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 -- 172.21.15.27:0/1782904732 >> 172.21.15.27:0/1782904732 conn(0x55faef0e3300 msgr2=0x55faef48b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 -- 172.21.15.27:0/1782904732 shutdown_connections 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 -- 172.21.15.27:0/1782904732 wait complete. 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 -- start start 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 Processor -- start 2024-04-25T13:55:42.385 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ce4d0 0x55faef4a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ef1b0 0x55faef4a9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55faef4a0030 con 0x55faef0ce4d0 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55faef37cb60 con 0x55faef0ef1b0 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.378+0000 7f1a57d92740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55faef4a01b0 con 0x55faef0c62f0 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a57d92740 10 client.0 _ll_register_callbacks cb 0x55faef3a12b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ce4d0 0x55faef4a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43110/0 (socket says 172.21.15.27:43110) 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 -- 172.21.15.27:0/4053235506 learned_addr learned my addr 172.21.15.27:0/4053235506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ce4d0 0x55faef4a23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42540/0 (socket says 172.21.15.27:42540) 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4ad9d640 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ef1b0 0x55faef4a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.386 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 -- 172.21.15.27:0/4053235506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ef1b0 msgr2=0x55faef4a9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ef1b0 0x55faef4a9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 -- 172.21.15.27:0/4053235506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ce4d0 msgr2=0x55faef4a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ce4d0 0x55faef4a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 -- 172.21.15.27:0/4053235506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faef107210 con 0x55faef0c62f0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ce4d0 0x55faef4a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a1e70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c01c9f0 tx=0x7f1a3c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48a0d487cb4016c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a31ffb640 1 -- 172.21.15.27:0/4053235506 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c02a060 con 0x55faef0c62f0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a57d92740 1 -- 172.21.15.27:0/4053235506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a44004030 con 0x55faef0c62f0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.379+0000 7f1a31ffb640 1 -- 172.21.15.27:0/4053235506 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a3c009040 con 0x55faef0c62f0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.380+0000 7f1a31ffb640 1 -- 172.21.15.27:0/4053235506 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1a3c00e8b0 con 0x55faef0c62f0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.380+0000 7f1a57d92740 10 client.28081 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.380+0000 7f1a57d92740 1 -- 172.21.15.27:0/4053235506 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55faef1d3620 con 0x55faef0c62f0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.380+0000 7f1a31ffb640 1 -- 172.21.15.27:0/4053235506 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a44004030 con 0x55faef0c62f0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.380+0000 7f1a31ffb640 1 -- 172.21.15.27:0/4053235506 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1a3c024070 con 0x55faef0c62f0 2024-04-25T13:55:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a57d92740 10 client.28081 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a57d92740 10 client.28081 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a57d92740 20 client.28081 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a57d92740 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55faef0add80 0x55faef4afa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a57d92740 4 client.28081 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a57d92740 1 -- 172.21.15.27:0/4053235506 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faef403070 con 0x55faef0add80 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55faef0add80 0x55faef4afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a31ffb640 10 client.28081 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.381+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55faef0add80 0x55faef4afa10 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f23be5c82be98e6b server_cookie=a02f8dfea36dcac in_seq=0 out_seq=0 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a31ffb640 1 -- 172.21.15.27:0/4053235506 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55faef403070 con 0x55faef0add80 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a31ffb640 10 client.28081 handle_command_reply: tid=0 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 client.28081 shutdown 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 -- 172.21.15.27:0/4053235506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 msgr2=0x55faef4a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a1e70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c01c9f0 tx=0x7f1a3c004730 comp rx=0 tx=0).stop 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 -- 172.21.15.27:0/4053235506 shutdown_connections 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55faef0add80 0x55faef4afa10 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ef1b0 0x55faef4a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ce4d0 0x55faef4a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 --2- 172.21.15.27:0/4053235506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a1e70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 -- 172.21.15.27:0/4053235506 >> 172.21.15.27:0/4053235506 conn(0x55faef0e3300 msgr2=0x55faef48a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a31ffb640 0 client.28081 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.382+0000 7f1a57d92740 1 -- 172.21.15.27:0/4053235506 shutdown_connections 2024-04-25T13:55:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.383+0000 7f1a57d92740 1 -- 172.21.15.27:0/4053235506 wait complete. 2024-04-25T13:55:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.383+0000 7f1a57d92740 20 client.28081 trim_cache size 0 max 16384 2024-04-25T13:55:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:41 smithi027 ceph-mon[29140]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 832 op/s 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.405+0000 7f1a57d92740 1 Processor -- start 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.405+0000 7f1a57d92740 1 -- start start 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef4b4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a57d92740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c1a6ef0 con 0x55faef0ef1b0 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a57d92740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c1a9770 con 0x55faef0ce4d0 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a57d92740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c1a7560 con 0x55faef0c62f0 2024-04-25T13:55:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a4a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef4b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34278/0 (socket says 172.21.15.27:34278) 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a49d9b640 1 -- 172.21.15.27:0/2907249148 learned_addr learned my addr 172.21.15.27:0/2907249148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a4a59c640 1 -- 172.21.15.27:0/2907249148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 msgr2=0x55faef4a92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/2907249148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a4a59c640 1 -- 172.21.15.27:0/2907249148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 msgr2=0x55faef4b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/2907249148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef4b4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a4a59c640 1 -- 172.21.15.27:0/2907249148 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faef4ad780 con 0x55faef0c62f0 2024-04-25T13:55:42.407 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.406+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/2907249148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a2150 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c025c10 tx=0x7f1a3c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8593a1bf289ea61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.407+0000 7f1a33fff640 1 -- 172.21.15.27:0/2907249148 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c035030 con 0x55faef0c62f0 2024-04-25T13:55:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.407+0000 7f1a33fff640 1 -- 172.21.15.27:0/2907249148 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a3c0253f0 con 0x55faef0c62f0 2024-04-25T13:55:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.407+0000 7f1a33fff640 1 -- 172.21.15.27:0/2907249148 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c031570 con 0x55faef0c62f0 2024-04-25T13:55:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.408+0000 7f1a57d92740 1 -- 172.21.15.27:0/2907249148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 msgr2=0x55faef4a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.408+0000 7f1a57d92740 1 --2- 172.21.15.27:0/2907249148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a2150 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c025c10 tx=0x7f1a3c009f90 comp rx=0 tx=0).stop 2024-04-25T13:55:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.408+0000 7f1a57d92740 1 -- 172.21.15.27:0/2907249148 shutdown_connections 2024-04-25T13:55:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.408+0000 7f1a57d92740 1 --2- 172.21.15.27:0/2907249148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef4b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.408+0000 7f1a57d92740 1 --2- 172.21.15.27:0/2907249148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.408+0000 7f1a57d92740 1 --2- 172.21.15.27:0/2907249148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a2150 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.408+0000 7f1a57d92740 1 -- 172.21.15.27:0/2907249148 >> 172.21.15.27:0/2907249148 conn(0x55faef0e3300 msgr2=0x55faef4a1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.408+0000 7f1a57d92740 1 -- 172.21.15.27:0/2907249148 shutdown_connections 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 -- 172.21.15.27:0/2907249148 wait complete. 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 -- start start 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 Processor -- start 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef5161b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c1a7c30 con 0x55faef0ef1b0 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c1a90a0 con 0x55faef0ce4d0 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c1a8300 con 0x55faef0c62f0 2024-04-25T13:55:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.409+0000 7f1a57d92740 10 client.0 _ll_register_callbacks cb 0x7f1a3c01c880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef5161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a4ad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef5161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42560/0 (socket says 172.21.15.27:42560) 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a49d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:34294/0 (socket says 172.21.15.27:34294) 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a4ad9d640 1 -- 172.21.15.27:0/1903827127 learned_addr learned my addr 172.21.15.27:0/1903827127 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a49d9b640 1 -- 172.21.15.27:0/1903827127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 msgr2=0x55faef4a8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a49d9b640 1 -- 172.21.15.27:0/1903827127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 msgr2=0x55faef5161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef5161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a49d9b640 1 -- 172.21.15.27:0/1903827127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faef5167a0 con 0x55faef0ce4d0 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a49d9b640 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a8d60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1a400303f0 tx=0x7f1a400e9040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f47ad4cf866b7f64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a31ffb640 1 -- 172.21.15.27:0/1903827127 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a400df070 con 0x55faef0ce4d0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a31ffb640 1 -- 172.21.15.27:0/1903827127 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f1a400fbe30 con 0x55faef0ce4d0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a57d92740 1 -- 172.21.15.27:0/1903827127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a3c0253f0 con 0x55faef0ce4d0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.410+0000 7f1a31ffb640 1 -- 172.21.15.27:0/1903827127 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1a400caab0 con 0x55faef0ce4d0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a57d92740 10 client.28013 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a57d92740 1 -- 172.21.15.27:0/1903827127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55faef517b00 con 0x55faef0ce4d0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a31ffb640 1 -- 172.21.15.27:0/1903827127 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a3c0253f0 con 0x55faef0ce4d0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a31ffb640 1 -- 172.21.15.27:0/1903827127 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f1a400cf0d0 con 0x55faef0ce4d0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a57d92740 10 client.28013 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a57d92740 10 client.28013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a57d92740 20 client.28013 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55faef0831c0 0x55faef51a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a57d92740 4 client.28013 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a57d92740 1 -- 172.21.15.27:0/1903827127 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55faef3fcee0 con 0x55faef0831c0 2024-04-25T13:55:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.411+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55faef0831c0 0x55faef51a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.412+0000 7f1a31ffb640 10 client.28013 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.412+0000 7f1a4a59c640 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55faef0831c0 0x55faef51a460 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6520cf7d05dc1f5 server_cookie=dbc7e4b2d1d326ac in_seq=0 out_seq=0 2024-04-25T13:55:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a31ffb640 1 -- 172.21.15.27:0/1903827127 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55faef3fcee0 con 0x55faef0831c0 2024-04-25T13:55:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a31ffb640 10 client.28013 handle_command_reply: tid=0 2024-04-25T13:55:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 client.28013 shutdown 2024-04-25T13:55:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 -- 172.21.15.27:0/1903827127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 msgr2=0x55faef4a8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a8d60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1a400303f0 tx=0x7f1a400e9040 comp rx=0 tx=0).stop 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 -- 172.21.15.27:0/1903827127 shutdown_connections 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55faef0831c0 0x55faef51a460 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a31ffb640 0 client.28013 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55faef0ef1b0 0x55faef5161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55faef0ce4d0 0x55faef4a8d60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 --2- 172.21.15.27:0/1903827127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55faef0c62f0 0x55faef4a8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 -- 172.21.15.27:0/1903827127 >> 172.21.15.27:0/1903827127 conn(0x55faef0e3300 msgr2=0x55faef489820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 -- 172.21.15.27:0/1903827127 shutdown_connections 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 1 -- 172.21.15.27:0/1903827127 wait complete. 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.531+0000 7f1a57d92740 20 client.28013 trim_cache size 0 max 16384 2024-04-25T13:55:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:55:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:55:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:55:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:55:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 -- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a340afb60 msgr2=0x7f1a340b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a340afb60 0x7f1a340b2070 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009530 tx=0x7f1a3c009260 comp rx=0 tx=0).stop 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 -- 172.21.15.27:0/1392491849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a4c104d90 msgr2=0x7f1a4c19a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a4c104d90 0x7f1a4c19a710 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1a40032d50 tx=0x7f1a40032d80 comp rx=0 tx=0).stop 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 -- 172.21.15.27:0/1392491849 shutdown_connections 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f1a340afb60 0x7f1a340b2070 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a4c109a90 0x7f1a4c1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a4c104d90 0x7f1a4c19a710 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 --2- 172.21.15.27:0/1392491849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a4c103b50 0x7f1a4c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 -- 172.21.15.27:0/1392491849 >> 172.21.15.27:0/1392491849 conn(0x7f1a4c0fed20 msgr2=0x7f1a4c105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.532+0000 7f1a50e05640 1 -- 172.21.15.27:0/1392491849 shutdown_connections 2024-04-25T13:55:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:42.533+0000 7f1a50e05640 1 -- 172.21.15.27:0/1392491849 wait complete. 2024-04-25T13:55:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 875.0 s, 1345.0 tps, lat 73.988 ms stddev 81.569 2024-04-25T13:55:44.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:43 smithi122 ceph-mon[32843]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 11 MiB/s wr, 840 op/s 2024-04-25T13:55:44.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:43 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:55:44.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:43 smithi045 ceph-mon[32731]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 11 MiB/s wr, 840 op/s 2024-04-25T13:55:44.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:43 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:55:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:43 smithi027 ceph-mon[29140]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 11 MiB/s wr, 840 op/s 2024-04-25T13:55:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:43 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:55:46.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:45 smithi122 ceph-mon[32843]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 7.7 MiB/s wr, 639 op/s 2024-04-25T13:55:46.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:45 smithi045 ceph-mon[32731]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 7.7 MiB/s wr, 639 op/s 2024-04-25T13:55:46.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:45 smithi027 ceph-mon[29140]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 7.7 MiB/s wr, 639 op/s 2024-04-25T13:55:47.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 880.0 s, 1941.2 tps, lat 52.277 ms stddev 64.130 2024-04-25T13:55:48.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:47 smithi122 ceph-mon[32843]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-04-25T13:55:48.330 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:47 smithi045 ceph-mon[32731]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-04-25T13:55:48.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:47 smithi027 ceph-mon[29140]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-04-25T13:55:50.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:49 smithi122 ceph-mon[32843]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 8.8 MiB/s wr, 724 op/s 2024-04-25T13:55:50.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:49 smithi045 ceph-mon[32731]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 8.8 MiB/s wr, 724 op/s 2024-04-25T13:55:50.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:49 smithi027 ceph-mon[29140]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 8.8 MiB/s wr, 724 op/s 2024-04-25T13:55:52.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:51 smithi122 ceph-mon[32843]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.4 MiB/s wr, 651 op/s 2024-04-25T13:55:52.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:51 smithi045 ceph-mon[32731]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.4 MiB/s wr, 651 op/s 2024-04-25T13:55:52.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:51 smithi027 ceph-mon[29140]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.4 MiB/s wr, 651 op/s 2024-04-25T13:55:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 885.0 s, 1645.2 tps, lat 59.756 ms stddev 72.047 2024-04-25T13:55:54.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:53 smithi122 ceph-mon[32843]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 8.1 MiB/s wr, 741 op/s 2024-04-25T13:55:54.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:53 smithi045 ceph-mon[32731]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 8.1 MiB/s wr, 741 op/s 2024-04-25T13:55:54.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:53 smithi027 ceph-mon[29140]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 8.1 MiB/s wr, 741 op/s 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.685+0000 7f0e920f5640 1 -- 172.21.15.27:0/3561462414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 msgr2=0x7f0e840965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.685+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3561462414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 0x7f0e840965e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c068690 tx=0x7f0e8c09fe90 comp rx=0 tx=0).stop 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.685+0000 7f0e920f5640 1 -- 172.21.15.27:0/3561462414 shutdown_connections 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.685+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3561462414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e84004780 0x7f0e84004b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.685+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3561462414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e84096b20 0x7f0e84096f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.685+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3561462414 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 0x7f0e840965e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.685+0000 7f0e920f5640 1 -- 172.21.15.27:0/3561462414 >> 172.21.15.27:0/3561462414 conn(0x7f0e84090ab0 msgr2=0x7f0e84092f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.686+0000 7f0e920f5640 1 -- 172.21.15.27:0/3561462414 shutdown_connections 2024-04-25T13:55:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.686+0000 7f0e920f5640 1 -- 172.21.15.27:0/3561462414 wait complete. 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e920f5640 1 Processor -- start 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e920f5640 1 -- start start 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e920f5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e84004780 0x7f0e8412af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e920f5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 0x7f0e8412b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e920f5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e84096b20 0x7f0e84132460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e920f5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e840a0380 con 0x7f0e84096b20 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e920f5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e840a0200 con 0x7f0e84004780 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e920f5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0e840a0500 con 0x7f0e840958e0 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e908f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 0x7f0e8412b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e918f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e84096b20 0x7f0e84132460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e910f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e84004780 0x7f0e8412af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e908f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 0x7f0e8412b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58294/0 (socket says 172.21.15.27:58294) 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e918f4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e84096b20 0x7f0e84132460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49180/0 (socket says 172.21.15.27:49180) 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.687+0000 7f0e908f2640 1 -- 172.21.15.27:0/3278635366 learned_addr learned my addr 172.21.15.27:0/3278635366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.688+0000 7f0e918f4640 1 -- 172.21.15.27:0/3278635366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 msgr2=0x7f0e8412b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.688+0000 7f0e918f4640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 0x7f0e8412b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.688+0000 7f0e918f4640 1 -- 172.21.15.27:0/3278635366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e84004780 msgr2=0x7f0e8412af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.688+0000 7f0e918f4640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e84004780 0x7f0e8412af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.688+0000 7f0e918f4640 1 -- 172.21.15.27:0/3278635366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e84132ad0 con 0x7f0e84096b20 2024-04-25T13:55:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.688+0000 7f0e918f4640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e84096b20 0x7f0e84132460 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00b3e0 tx=0x7f0e7c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.689+0000 7f0e827fc640 1 -- 172.21.15.27:0/3278635366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c015020 con 0x7f0e84096b20 2024-04-25T13:55:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.689+0000 7f0e920f5640 1 -- 172.21.15.27:0/3278635366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e84132dc0 con 0x7f0e84096b20 2024-04-25T13:55:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.689+0000 7f0e827fc640 1 -- 172.21.15.27:0/3278635366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e7c011ce0 con 0x7f0e84096b20 2024-04-25T13:55:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.689+0000 7f0e827fc640 1 -- 172.21.15.27:0/3278635366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c010db0 con 0x7f0e84096b20 2024-04-25T13:55:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.689+0000 7f0e920f5640 1 -- 172.21.15.27:0/3278635366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e841332a0 con 0x7f0e84096b20 2024-04-25T13:55:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.693+0000 7f0e9a20a740 1 Processor -- start 2024-04-25T13:55:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.695+0000 7f0e827fc640 1 -- 172.21.15.27:0/3278635366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0e7c018020 con 0x7f0e84096b20 2024-04-25T13:55:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.695+0000 7f0e9a20a740 1 -- start start 2024-04-25T13:55:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f6081b0 0x55e27f9a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9a9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5e74d0 0x55e27f9aff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e27f90e460 con 0x55e27f5e74d0 2024-04-25T13:55:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e27f8c8590 con 0x55e27f6081b0 2024-04-25T13:55:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e27f90a420 con 0x55e27f5df2f0 2024-04-25T13:55:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e908f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f6081b0 0x55e27f9a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e908f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f6081b0 0x55e27f9a8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56874/0 (socket says 172.21.15.27:56874) 2024-04-25T13:55:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e908f2640 1 -- 172.21.15.27:0/3815470836 learned_addr learned my addr 172.21.15.27:0/3815470836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e827fc640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0e780afbb0 0x7f0e780b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e910f3640 1 --2- 172.21.15.27:0/3815470836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.696+0000 7f0e827fc640 1 -- 172.21.15.27:0/3278635366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0e7c0de300 con 0x7f0e84096b20 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e908f2640 1 -- 172.21.15.27:0/3815470836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 msgr2=0x55e27f9a9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e908f2640 1 --2- 172.21.15.27:0/3815470836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9a9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e908f2640 1 -- 172.21.15.27:0/3815470836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5e74d0 msgr2=0x55e27f9aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e908f2640 1 --2- 172.21.15.27:0/3815470836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5e74d0 0x55e27f9aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e908f2640 1 -- 172.21.15.27:0/3815470836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e27f8d50d0 con 0x55e27f6081b0 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e908f2640 1 --2- 172.21.15.27:0/3815470836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f6081b0 0x55e27f9a8b10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009b70 tx=0x7f0e880105c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb704a84449a7752 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e777fe640 1 -- 172.21.15.27:0/3815470836 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88018020 con 0x55e27f6081b0 2024-04-25T13:55:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e918f4640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0e780afbb0 0x7f0e780b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e777fe640 1 -- 172.21.15.27:0/3815470836 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e88009e60 con 0x55e27f6081b0 2024-04-25T13:55:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.697+0000 7f0e777fe640 1 -- 172.21.15.27:0/3815470836 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88013720 con 0x55e27f6081b0 2024-04-25T13:55:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3815470836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f6081b0 msgr2=0x55e27f9a8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3815470836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f6081b0 0x55e27f9a8b10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009b70 tx=0x7f0e880105c0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e918f4640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0e780afbb0 0x7f0e780b20c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00bda0 tx=0x7f0e7c0107d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3815470836 shutdown_connections 2024-04-25T13:55:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3815470836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5e74d0 0x55e27f9aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3815470836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9a9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3815470836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f6081b0 0x55e27f9a8b10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3815470836 >> 172.21.15.27:0/3815470836 conn(0x55e27f5fc300 msgr2=0x55e27f9a44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3815470836 shutdown_connections 2024-04-25T13:55:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.699+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3815470836 wait complete. 2024-04-25T13:55:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 1 -- start start 2024-04-25T13:55:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 1 Processor -- start 2024-04-25T13:55:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 0x55e27f9b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f6081b0 0x55e27f9b9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55e27f9b09e0 con 0x55e27f5df2f0 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55e27f895b60 con 0x55e27f5e74d0 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55e27f9b0b60 con 0x55e27f6081b0 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.700+0000 7f0e9a20a740 10 client.0 _ll_register_callbacks cb 0x55e27f8ba2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e910f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e910f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:56878/0 (socket says 172.21.15.27:56878) 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e910f3640 1 -- 172.21.15.27:0/3420187990 learned_addr learned my addr 172.21.15.27:0/3420187990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e908f2640 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f6081b0 0x55e27f9b9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e908f2640 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 0x55e27f9b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e910f3640 1 -- 172.21.15.27:0/3420187990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f6081b0 msgr2=0x55e27f9b9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e910f3640 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f6081b0 0x55e27f9b9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e910f3640 1 -- 172.21.15.27:0/3420187990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 msgr2=0x55e27f9b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e910f3640 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 0x55e27f9b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e910f3640 1 -- 172.21.15.27:0/3420187990 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e27f620210 con 0x55e27f5e74d0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.701+0000 7f0e908f2640 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f6081b0 0x55e27f9b9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e910f3640 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9b2df0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0685b0 tx=0x7f0e8c068fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608879b33dbb7201 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e74ff9640 1 -- 172.21.15.27:0/3420187990 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c06b560 con 0x55e27f5e74d0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3420187990 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e88009e60 con 0x55e27f5e74d0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e74ff9640 1 -- 172.21.15.27:0/3420187990 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0e8c0802d0 con 0x55e27f5e74d0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e74ff9640 1 -- 172.21.15.27:0/3420187990 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0e8c07eda0 con 0x55e27f5e74d0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e9a20a740 10 client.28022 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3420187990 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e27f6ec620 con 0x55e27f5e74d0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.702+0000 7f0e74ff9640 1 -- 172.21.15.27:0/3420187990 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e88009e60 con 0x55e27f5e74d0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.703+0000 7f0e74ff9640 1 -- 172.21.15.27:0/3420187990 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f0e8c0a8af0 con 0x55e27f5e74d0 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.703+0000 7f0e9a20a740 10 client.28022 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.703+0000 7f0e9a20a740 10 client.28022 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.703+0000 7f0e9a20a740 20 client.28022 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.703+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27f5c6d80 0x55e27f9c0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.703+0000 7f0e9a20a740 4 client.28022 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:55:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.703+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3420187990 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e27f91c070 con 0x55e27f5c6d80 2024-04-25T13:55:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.703+0000 7f0e908f2640 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27f5c6d80 0x55e27f9c0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.704+0000 7f0e74ff9640 10 client.28022 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.704+0000 7f0e908f2640 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27f5c6d80 0x55e27f9c0af0 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6f45a4b6cda8bc6 server_cookie=6375f5091c7e1dde in_seq=0 out_seq=0 2024-04-25T13:55:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e74ff9640 1 -- 172.21.15.27:0/3420187990 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e27f91c070 con 0x55e27f5c6d80 2024-04-25T13:55:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e74ff9640 10 client.28022 handle_command_reply: tid=0 2024-04-25T13:55:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 client.28022 shutdown 2024-04-25T13:55:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3420187990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 msgr2=0x55e27f9b2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9b2df0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0685b0 tx=0x7f0e8c068fe0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3420187990 shutdown_connections 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27f5c6d80 0x55e27f9c0af0 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f6081b0 0x55e27f9b9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9b2df0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e74ff9640 0 client.28022 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/3420187990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 0x55e27f9b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3420187990 >> 172.21.15.27:0/3420187990 conn(0x55e27f5fc300 msgr2=0x55e27f9a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3420187990 shutdown_connections 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.705+0000 7f0e9a20a740 1 -- 172.21.15.27:0/3420187990 wait complete. 2024-04-25T13:55:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.706+0000 7f0e9a20a740 20 client.28022 trim_cache size 0 max 16384 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e9a20a740 1 Processor -- start 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e9a20a740 1 -- start start 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5c6d80 0x55e27f9c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9c6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84139e60 con 0x55e27f5c6d80 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8413c690 con 0x55e27f5e74d0 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8413a480 con 0x55e27f5df2f0 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e910f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e908f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5c6d80 0x55e27f9c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e910f3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58308/0 (socket says 172.21.15.27:58308) 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e910f3640 1 -- 172.21.15.27:0/1851766192 learned_addr learned my addr 172.21.15.27:0/1851766192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.730+0000 7f0e908f2640 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e910f3640 1 -- 172.21.15.27:0/1851766192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 msgr2=0x55e27f9c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e910f3640 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e910f3640 1 -- 172.21.15.27:0/1851766192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5c6d80 msgr2=0x55e27f9c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e910f3640 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5c6d80 0x55e27f9c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e910f3640 1 -- 172.21.15.27:0/1851766192 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e27f9ac360 con 0x55e27f5df2f0 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e908f2640 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e910f3640 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9c70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c068ac0 tx=0x7f0e8c068520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ba7aacf6ae03e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e76ffd640 1 -- 172.21.15.27:0/1851766192 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0a04d0 con 0x55e27f5df2f0 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e76ffd640 1 -- 172.21.15.27:0/1851766192 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e8c0a4040 con 0x55e27f5df2f0 2024-04-25T13:55:55.732 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.731+0000 7f0e76ffd640 1 -- 172.21.15.27:0/1851766192 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0804f0 con 0x55e27f5df2f0 2024-04-25T13:55:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.732+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1851766192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 msgr2=0x55e27f9b9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.732+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9c70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c068ac0 tx=0x7f0e8c068520 comp rx=0 tx=0).stop 2024-04-25T13:55:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.732+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1851766192 shutdown_connections 2024-04-25T13:55:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.732+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27f9c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.732+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9c70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.732+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1851766192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5c6d80 0x55e27f9c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.732+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1851766192 >> 172.21.15.27:0/1851766192 conn(0x55e27f5fc300 msgr2=0x55e27f9accd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.733+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1851766192 shutdown_connections 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.733+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1851766192 wait complete. 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.733+0000 7f0e9a20a740 1 -- start start 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.733+0000 7f0e9a20a740 1 Processor -- start 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.733+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5c6d80 0x55e27f9b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e9a20a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27fa27060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8413ab50 con 0x55e27f5df2f0 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8413bfc0 con 0x55e27f5e74d0 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e9a20a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8413b220 con 0x55e27f5c6d80 2024-04-25T13:55:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5c6d80 0x55e27f9b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5c6d80 0x55e27f9b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:58322/0 (socket says 172.21.15.27:58322) 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e910f3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 -- 172.21.15.27:0/1818473064 learned_addr learned my addr 172.21.15.27:0/1818473064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27fa27060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e9a20a740 10 client.0 _ll_register_callbacks cb 0x7f0e8c068b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 -- 172.21.15.27:0/1818473064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 msgr2=0x55e27fa27060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27fa27060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 -- 172.21.15.27:0/1818473064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 msgr2=0x55e27f9b9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 -- 172.21.15.27:0/1818473064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e27fa27650 con 0x55e27f5c6d80 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e908f2640 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5c6d80 0x55e27f9b8f50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009b40 tx=0x7f0e8801f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172cddf4288c9f9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e74ff9640 1 -- 172.21.15.27:0/1818473064 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88018040 con 0x55e27f5c6d80 2024-04-25T13:55:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.734+0000 7f0e74ff9640 1 -- 172.21.15.27:0/1818473064 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f0e88014720 con 0x55e27f5c6d80 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1818473064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e8c0a4040 con 0x55e27f5c6d80 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e74ff9640 1 -- 172.21.15.27:0/1818473064 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0e88014920 con 0x55e27f5c6d80 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 10 client.28105 fetch_fsmap learned FSMap version 19 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1818473064 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55e27fa288e0 con 0x55e27f5c6d80 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e74ff9640 1 -- 172.21.15.27:0/1818473064 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e8c0a4040 con 0x55e27f5c6d80 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e74ff9640 1 -- 172.21.15.27:0/1818473064 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f0e8801d420 con 0x55e27f5c6d80 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 10 client.28105 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 10 client.28105 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 20 client.28105 populate_metadata read hostname 'smithi027' 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27f6081b0 0x55e27fa2af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 4 client.28105 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1818473064 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e27f915ee0 con 0x55e27f6081b0 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.735+0000 7f0e910f3640 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27f6081b0 0x55e27fa2af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:55:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.736+0000 7f0e910f3640 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27f6081b0 0x55e27fa2af90 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fae503b3b71c0186 server_cookie=be42743f80b4e934 in_seq=0 out_seq=0 2024-04-25T13:55:55.740 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.736+0000 7f0e74ff9640 10 client.28105 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.756+0000 7f0e74ff9640 1 -- 172.21.15.27:0/1818473064 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e27f915ee0 con 0x55e27f6081b0 2024-04-25T13:55:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.756+0000 7f0e74ff9640 10 client.28105 handle_command_reply: tid=0 2024-04-25T13:55:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.757+0000 7f0e9a20a740 1 client.28105 shutdown 2024-04-25T13:55:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.757+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1818473064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5c6d80 msgr2=0x55e27f9b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.757+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5c6d80 0x55e27f9b8f50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009b40 tx=0x7f0e8801f040 comp rx=0 tx=0).stop 2024-04-25T13:55:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.758+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1818473064 shutdown_connections 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.758+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55e27f6081b0 0x55e27fa2af90 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.758+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e27f5e74d0 0x55e27fa27060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.759+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55e27f5df2f0 0x55e27f9b9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.759+0000 7f0e74ff9640 0 client.28105 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.759+0000 7f0e9a20a740 1 --2- 172.21.15.27:0/1818473064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e27f5c6d80 0x55e27f9b8f50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.759+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1818473064 >> 172.21.15.27:0/1818473064 conn(0x55e27f5fc300 msgr2=0x55e27f9a72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.759+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1818473064 shutdown_connections 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.759+0000 7f0e9a20a740 1 -- 172.21.15.27:0/1818473064 wait complete. 2024-04-25T13:55:55.759 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.759+0000 7f0e9a20a740 20 client.28105 trim_cache size 0 max 16384 2024-04-25T13:55:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:55:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:55:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:55:55.760 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 -- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0e780afbb0 msgr2=0x7f0e780b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0e780afbb0 0x7f0e780b20c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00bda0 tx=0x7f0e7c0107d0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 -- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e84096b20 msgr2=0x7f0e84132460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e84096b20 0x7f0e84132460 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00b3e0 tx=0x7f0e7c00b8a0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 -- 172.21.15.27:0/3278635366 shutdown_connections 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f0e780afbb0 0x7f0e780b20c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e84096b20 0x7f0e84132460 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e840958e0 0x7f0e8412b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 --2- 172.21.15.27:0/3278635366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e84004780 0x7f0e8412af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 -- 172.21.15.27:0/3278635366 >> 172.21.15.27:0/3278635366 conn(0x7f0e84090ab0 msgr2=0x7f0e84092240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 -- 172.21.15.27:0/3278635366 shutdown_connections 2024-04-25T13:55:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:55:55.760+0000 7f0e920f5640 1 -- 172.21.15.27:0/3278635366 wait complete. 2024-04-25T13:55:56.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:55 smithi122 ceph-mon[32843]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 558 op/s 2024-04-25T13:55:56.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:55 smithi045 ceph-mon[32731]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 558 op/s 2024-04-25T13:55:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:55 smithi027 ceph-mon[29140]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 558 op/s 2024-04-25T13:55:57.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 890.0 s, 1528.6 tps, lat 66.738 ms stddev 78.264 2024-04-25T13:55:58.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:57 smithi122 ceph-mon[32843]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.4 MiB/s wr, 637 op/s 2024-04-25T13:55:58.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:57 smithi045 ceph-mon[32731]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.4 MiB/s wr, 637 op/s 2024-04-25T13:55:58.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:57 smithi027 ceph-mon[29140]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.4 MiB/s wr, 637 op/s 2024-04-25T13:56:00.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:55:59 smithi122 ceph-mon[32843]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 657 op/s 2024-04-25T13:56:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:55:59 smithi045 ceph-mon[32731]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 657 op/s 2024-04-25T13:56:00.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:55:59 smithi027 ceph-mon[29140]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 657 op/s 2024-04-25T13:56:02.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:01 smithi122 ceph-mon[32843]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.9 MiB/s wr, 559 op/s 2024-04-25T13:56:02.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:01 smithi045 ceph-mon[32731]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.9 MiB/s wr, 559 op/s 2024-04-25T13:56:02.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:01 smithi027 ceph-mon[29140]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.9 MiB/s wr, 559 op/s 2024-04-25T13:56:02.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 895.0 s, 1980.0 tps, lat 49.678 ms stddev 60.298 2024-04-25T13:56:04.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:03 smithi122 ceph-mon[32843]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 701 op/s 2024-04-25T13:56:04.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:03 smithi045 ceph-mon[32731]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 701 op/s 2024-04-25T13:56:04.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:03 smithi027 ceph-mon[29140]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 701 op/s 2024-04-25T13:56:06.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:05 smithi122 ceph-mon[32843]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.5 MiB/s wr, 596 op/s 2024-04-25T13:56:06.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:05 smithi045 ceph-mon[32731]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.5 MiB/s wr, 596 op/s 2024-04-25T13:56:06.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:05 smithi027 ceph-mon[29140]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.5 MiB/s wr, 596 op/s 2024-04-25T13:56:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:progress: 900.0 s, 1356.4 tps, lat 74.366 ms stddev 81.588 2024-04-25T13:56:08.079 INFO:teuthology.orchestra.run.smithi027.stdout:transaction type: 2024-04-25T13:56:08.079 INFO:teuthology.orchestra.run.smithi027.stdout:scaling factor: 32 2024-04-25T13:56:08.079 INFO:teuthology.orchestra.run.smithi027.stdout:query mode: simple 2024-04-25T13:56:08.080 INFO:teuthology.orchestra.run.smithi027.stdout:number of clients: 100 2024-04-25T13:56:08.080 INFO:teuthology.orchestra.run.smithi027.stdout:number of threads: 4 2024-04-25T13:56:08.080 INFO:teuthology.orchestra.run.smithi027.stdout:duration: 900 s 2024-04-25T13:56:08.080 INFO:teuthology.orchestra.run.smithi027.stdout:number of transactions actually processed: 2190230 2024-04-25T13:56:08.080 INFO:teuthology.orchestra.run.smithi027.stdout:latency average = 41.087 ms 2024-04-25T13:56:08.080 INFO:teuthology.orchestra.run.smithi027.stdout:latency stddev = 52.247 ms 2024-04-25T13:56:08.080 INFO:teuthology.orchestra.run.smithi027.stdout:tps = 2433.144712 (including connections establishing) 2024-04-25T13:56:08.080 INFO:teuthology.orchestra.run.smithi027.stdout:tps = 2433.151845 (excluding connections establishing) 2024-04-25T13:56:08.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-25T13:56:08.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:08 smithi122 ceph-mon[32843]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.4 MiB/s wr, 650 op/s 2024-04-25T13:56:08.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:08 smithi045 ceph-mon[32731]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.4 MiB/s wr, 650 op/s 2024-04-25T13:56:08.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:08 smithi027 ceph-mon[29140]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.4 MiB/s wr, 650 op/s 2024-04-25T13:56:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.936+0000 7f7907138640 1 -- 172.21.15.27:0/2369391849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7900107970 msgr2=0x7f7900107d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.936+0000 7f7907138640 1 --2- 172.21.15.27:0/2369391849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7900107970 0x7f7900107d80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009ca0 tx=0x7f78f802a8d0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.942+0000 7f7907138640 1 -- 172.21.15.27:0/2369391849 shutdown_connections 2024-04-25T13:56:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.942+0000 7f7907138640 1 --2- 172.21.15.27:0/2369391849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7900107970 0x7f7900107d80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.942+0000 7f7907138640 1 --2- 172.21.15.27:0/2369391849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7900102800 0x7f7900102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.942+0000 7f7907138640 1 --2- 172.21.15.27:0/2369391849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7900101dc0 0x7f7900102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.942+0000 7f7907138640 1 -- 172.21.15.27:0/2369391849 >> 172.21.15.27:0/2369391849 conn(0x7f79000fd390 msgr2=0x7f79000ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.946+0000 7f7907138640 1 -- 172.21.15.27:0/2369391849 shutdown_connections 2024-04-25T13:56:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.949+0000 7f7907138640 1 -- 172.21.15.27:0/2369391849 wait complete. 2024-04-25T13:56:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.950+0000 7f7907138640 1 Processor -- start 2024-04-25T13:56:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.951+0000 7f7907138640 1 -- start start 2024-04-25T13:56:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.952+0000 7f7907138640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7900101dc0 0x7f7900071a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.952+0000 7f7907138640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7900102800 0x7f7900071f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.952+0000 7f7907138640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7900107970 0x7f7900072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.952+0000 7f7907138640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7900115c40 con 0x7f7900101dc0 2024-04-25T13:56:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.952+0000 7f7907138640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7900115ac0 con 0x7f7900107970 2024-04-25T13:56:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.952+0000 7f7907138640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f7900115dc0 con 0x7f7900102800 2024-04-25T13:56:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.953+0000 7f79056d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7900107970 0x7f7900072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.954+0000 7f79056d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7900107970 0x7f7900072ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51314/0 (socket says 172.21.15.27:51314) 2024-04-25T13:56:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.954+0000 7f79056d5640 1 -- 172.21.15.27:0/291737136 learned_addr learned my addr 172.21.15.27:0/291737136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.954+0000 7f78fffff640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7900102800 0x7f7900071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.956+0000 7f7904ed4640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7900101dc0 0x7f7900071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.957+0000 7f79056d5640 1 -- 172.21.15.27:0/291737136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7900102800 msgr2=0x7f7900071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:08.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.957+0000 7f79056d5640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7900102800 0x7f7900071f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.957+0000 7f79056d5640 1 -- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7900101dc0 msgr2=0x7f7900071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:08.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.957+0000 7f79056d5640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7900101dc0 0x7f7900071a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.958 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.957+0000 7f79056d5640 1 -- 172.21.15.27:0/291737136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7900073530 con 0x7f7900107970 2024-04-25T13:56:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.962+0000 7f79056d5640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7900107970 0x7f7900072ec0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f78f80098d0 tx=0x7f78f8003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:08.965 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.964+0000 7f78fdffb640 1 -- 172.21.15.27:0/291737136 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8004040 con 0x7f7900107970 2024-04-25T13:56:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.964+0000 7f78fdffb640 1 -- 172.21.15.27:0/291737136 <== mon.2 v2:172.21.15.45:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78f8004690 con 0x7f7900107970 2024-04-25T13:56:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.964+0000 7f78fdffb640 1 -- 172.21.15.27:0/291737136 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8030d20 con 0x7f7900107970 2024-04-25T13:56:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.964+0000 7f7907138640 1 -- 172.21.15.27:0/291737136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79000737c0 con 0x7f7900107970 2024-04-25T13:56:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.965+0000 7f7907138640 1 -- 172.21.15.27:0/291737136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7900073cc0 con 0x7f7900107970 2024-04-25T13:56:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.968+0000 7f790dfef740 1 Processor -- start 2024-04-25T13:56:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.968+0000 7f790dfef740 1 -- start start 2024-04-25T13:56:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.969+0000 7f790dfef740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f5125f9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.969+0000 7f790dfef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f5125f9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.969+0000 7f790dfef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 0x55f512600df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.969+0000 7f790dfef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f51255f460 con 0x55f5122384d0 2024-04-25T13:56:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.969+0000 7f790dfef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f512519590 con 0x55f5122302f0 2024-04-25T13:56:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.969+0000 7f790dfef740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f51255b420 con 0x55f5122591b0 2024-04-25T13:56:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.970+0000 7f78fffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f5125f9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.970+0000 7f78fffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f5125f9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43492/0 (socket says 172.21.15.27:43492) 2024-04-25T13:56:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.970+0000 7f78fffff640 1 -- 172.21.15.27:0/4200091682 learned_addr learned my addr 172.21.15.27:0/4200091682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.970+0000 7f78fffff640 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 0x55f512600df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.975+0000 7f7904ed4640 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f5125f9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.975+0000 7f78fffff640 1 -- 172.21.15.27:0/4200091682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 msgr2=0x55f5125f9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.975+0000 7f78fffff640 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f5125f9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.975+0000 7f78fffff640 1 -- 172.21.15.27:0/4200091682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 msgr2=0x55f512600df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.975+0000 7f78fffff640 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 0x55f512600df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.975+0000 7f78fffff640 1 -- 172.21.15.27:0/4200091682 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5125260d0 con 0x55f5122591b0 2024-04-25T13:56:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.975+0000 7f78fdffb640 1 -- 172.21.15.27:0/291737136 <== mon.2 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f78f80041e0 con 0x7f7900107970 2024-04-25T13:56:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.976+0000 7f78fdffb640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f00afcf0 0x7f78f00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.976+0000 7f78fdffb640 1 -- 172.21.15.27:0/291737136 <== mon.2 v2:172.21.15.45:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f78f81002a0 con 0x7f7900107970 2024-04-25T13:56:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.977+0000 7f78fffff640 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f5125f9970 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f78f400beb0 tx=0x7f78f400b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7c543c6b0b78848 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:08.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.982+0000 7f78eaffd640 1 -- 172.21.15.27:0/4200091682 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4032740 con 0x55f5122591b0 2024-04-25T13:56:08.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.982+0000 7f78eaffd640 1 -- 172.21.15.27:0/4200091682 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78f4032d90 con 0x55f5122591b0 2024-04-25T13:56:08.985 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.982+0000 7f78eaffd640 1 -- 172.21.15.27:0/4200091682 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f403fca0 con 0x55f5122591b0 2024-04-25T13:56:08.986 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.983+0000 7f7904ed4640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f00afcf0 0x7f78f00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.986 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.983+0000 7f790dfef740 1 -- 172.21.15.27:0/4200091682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 msgr2=0x55f5125f9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:08.986 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.983+0000 7f790dfef740 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f5125f9970 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f78f400beb0 tx=0x7f78f400b7c0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.986 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.984+0000 7f790dfef740 1 -- 172.21.15.27:0/4200091682 shutdown_connections 2024-04-25T13:56:08.986 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.984+0000 7f790dfef740 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 0x55f512600df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.986 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.984+0000 7f790dfef740 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f5125f9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.986 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.984+0000 7f790dfef740 1 --2- 172.21.15.27:0/4200091682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f5125f9970 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.986 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.984+0000 7f790dfef740 1 -- 172.21.15.27:0/4200091682 >> 172.21.15.27:0/4200091682 conn(0x55f51224d300 msgr2=0x55f5125f5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.990+0000 7f790dfef740 1 -- 172.21.15.27:0/4200091682 shutdown_connections 2024-04-25T13:56:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.992+0000 7f7904ed4640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f00afcf0 0x7f78f00b2200 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x55f5125f6090 tx=0x7f78ec005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.994+0000 7f790dfef740 1 -- 172.21.15.27:0/4200091682 wait complete. 2024-04-25T13:56:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.994+0000 7f790dfef740 1 -- start start 2024-04-25T13:56:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.994+0000 7f790dfef740 1 Processor -- start 2024-04-25T13:56:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f790dfef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122302f0 0x55f512603790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f790dfef740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122384d0 0x55f512603cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f790dfef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122591b0 0x55f51260ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f790dfef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55f512601840 con 0x55f5122302f0 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f790dfef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f5124e6b60 con 0x55f5122591b0 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f790dfef740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55f5126019c0 con 0x55f5122384d0 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f790dfef740 10 client.0 _ll_register_callbacks cb 0x55f51250b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f78fffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122384d0 0x55f512603cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f78fffff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122384d0 0x55f512603cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43502/0 (socket says 172.21.15.27:43502) 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f78fffff640 1 -- 172.21.15.27:0/1218912778 learned_addr learned my addr 172.21.15.27:0/1218912778 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.995+0000 7f79056d5640 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122591b0 0x55f51260ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.997+0000 7f78fffff640 1 -- 172.21.15.27:0/1218912778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122591b0 msgr2=0x55f51260ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.997+0000 7f78fffff640 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122591b0 0x55f51260ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.997+0000 7f78fffff640 1 -- 172.21.15.27:0/1218912778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122302f0 msgr2=0x55f512603790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.997+0000 7f78fffff640 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122302f0 0x55f512603790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:08.997 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.997+0000 7f78fffff640 1 -- 172.21.15.27:0/1218912778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f512271210 con 0x55f5122384d0 2024-04-25T13:56:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.997+0000 7f78fffff640 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122384d0 0x55f512603cd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f78f403f610 tx=0x7f78f403f640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d25b1b7d6d6a930a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:08.998 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:08.997+0000 7f78dffff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:56:09.002 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.001+0000 7f78dffff640 1 -- 172.21.15.27:0/1218912778 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4034250 con 0x55f5122384d0 2024-04-25T13:56:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.001+0000 7f790dfef740 1 -- 172.21.15.27:0/1218912778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78f4032d90 con 0x55f5122384d0 2024-04-25T13:56:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.001+0000 7f78dffff640 1 -- 172.21.15.27:0/1218912778 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f78f4004040 con 0x55f5122384d0 2024-04-25T13:56:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.001+0000 7f78dffff640 1 -- 172.21.15.27:0/1218912778 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f78f40343f0 con 0x55f5122384d0 2024-04-25T13:56:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.004+0000 7f790dfef740 10 client.28117 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.004+0000 7f790dfef740 1 -- 172.21.15.27:0/1218912778 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f51233d620 con 0x55f5122384d0 2024-04-25T13:56:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.006+0000 7f78dffff640 1 -- 172.21.15.27:0/1218912778 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78f4032d90 con 0x55f5122384d0 2024-04-25T13:56:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.006+0000 7f78dffff640 1 -- 172.21.15.27:0/1218912778 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f78f4049070 con 0x55f5122384d0 2024-04-25T13:56:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.006+0000 7f790dfef740 10 client.28117 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.006+0000 7f790dfef740 10 client.28117 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.006+0000 7f790dfef740 20 client.28117 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.006+0000 7f790dfef740 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f512217d80 0x55f5126338b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.006+0000 7f790dfef740 4 client.28117 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:56:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.006+0000 7f790dfef740 1 -- 172.21.15.27:0/1218912778 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f51256d070 con 0x55f512217d80 2024-04-25T13:56:09.014 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.013+0000 7f7904ed4640 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f512217d80 0x55f5126338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:56:09.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:56:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.023+0000 7f7904ed4640 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f512217d80 0x55f5126338b0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9af0b021f3ef66a server_cookie=e875425232d8e248 in_seq=0 out_seq=0 2024-04-25T13:56:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.024+0000 7f78dffff640 10 client.28117 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:09.029 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.028+0000 7f78dffff640 1 -- 172.21.15.27:0/1218912778 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f51256d070 con 0x55f512217d80 2024-04-25T13:56:09.029 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.028+0000 7f78dffff640 10 client.28117 handle_command_reply: tid=0 2024-04-25T13:56:09.029 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.028+0000 7f790dfef740 1 client.28117 shutdown 2024-04-25T13:56:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.037+0000 7f790dfef740 1 -- 172.21.15.27:0/1218912778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122384d0 msgr2=0x55f512603cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:09.037 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.037+0000 7f790dfef740 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122384d0 0x55f512603cd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f78f403f610 tx=0x7f78f403f640 comp rx=0 tx=0).stop 2024-04-25T13:56:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.041+0000 7f790dfef740 1 -- 172.21.15.27:0/1218912778 shutdown_connections 2024-04-25T13:56:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.041+0000 7f790dfef740 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f512217d80 0x55f5126338b0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.041+0000 7f790dfef740 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122591b0 0x55f51260ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.041+0000 7f790dfef740 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122384d0 0x55f512603cd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.041+0000 7f790dfef740 1 --2- 172.21.15.27:0/1218912778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122302f0 0x55f512603790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.040 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.041+0000 7f78dffff640 0 client.28117 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.041+0000 7f790dfef740 1 -- 172.21.15.27:0/1218912778 >> 172.21.15.27:0/1218912778 conn(0x55f51224d300 msgr2=0x55f5125f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:09.045 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.046+0000 7f790dfef740 1 -- 172.21.15.27:0/1218912778 shutdown_connections 2024-04-25T13:56:09.050 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.050+0000 7f790dfef740 1 -- 172.21.15.27:0/1218912778 wait complete. 2024-04-25T13:56:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.053+0000 7f790dfef740 20 client.28117 trim_cache size 0 max 16384 2024-04-25T13:56:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.096+0000 7f790dfef740 1 Processor -- start 2024-04-25T13:56:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.100+0000 7f790dfef740 1 -- start start 2024-04-25T13:56:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.100+0000 7f790dfef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122302f0 0x55f5125fe250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.100+0000 7f790dfef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122384d0 0x55f512603130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.100+0000 7f790dfef740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f51260ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.100+0000 7f790dfef740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f790010ead0 con 0x55f5122302f0 2024-04-25T13:56:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.100+0000 7f790dfef740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7900111350 con 0x55f5122384d0 2024-04-25T13:56:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.100+0000 7f790dfef740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f790010f140 con 0x55f5122591b0 2024-04-25T13:56:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.101+0000 7f79056d5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f51260ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.101+0000 7f78fffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122384d0 0x55f512603130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.101+0000 7f79056d5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f51260ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43512/0 (socket says 172.21.15.27:43512) 2024-04-25T13:56:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.101+0000 7f79056d5640 1 -- 172.21.15.27:0/4128991284 learned_addr learned my addr 172.21.15.27:0/4128991284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.102+0000 7f79056d5640 1 -- 172.21.15.27:0/4128991284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122384d0 msgr2=0x55f512603130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.102+0000 7f79056d5640 1 --2- 172.21.15.27:0/4128991284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122384d0 0x55f512603130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.102+0000 7f79056d5640 1 -- 172.21.15.27:0/4128991284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122302f0 msgr2=0x55f5125fe250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:09.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.102+0000 7f79056d5640 1 --2- 172.21.15.27:0/4128991284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122302f0 0x55f5125fe250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.102+0000 7f79056d5640 1 -- 172.21.15.27:0/4128991284 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51260b090 con 0x55f5122591b0 2024-04-25T13:56:09.102 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.102+0000 7f79056d5640 1 --2- 172.21.15.27:0/4128991284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f51260ab50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f78f80c5930 tx=0x7f78f80307b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c3d6df8790a4eba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.103+0000 7f78ea7fc640 1 -- 172.21.15.27:0/4128991284 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f80e8070 con 0x55f5122591b0 2024-04-25T13:56:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.104+0000 7f78ea7fc640 1 -- 172.21.15.27:0/4128991284 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78f8036030 con 0x55f5122591b0 2024-04-25T13:56:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.104+0000 7f78ea7fc640 1 -- 172.21.15.27:0/4128991284 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f80e1030 con 0x55f5122591b0 2024-04-25T13:56:09.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.106+0000 7f790dfef740 1 -- 172.21.15.27:0/4128991284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 msgr2=0x55f51260ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:09.108 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.106+0000 7f790dfef740 1 --2- 172.21.15.27:0/4128991284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f51260ab50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f78f80c5930 tx=0x7f78f80307b0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.111+0000 7f790dfef740 1 -- 172.21.15.27:0/4128991284 shutdown_connections 2024-04-25T13:56:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.111+0000 7f790dfef740 1 --2- 172.21.15.27:0/4128991284 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f51260ab50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.111+0000 7f790dfef740 1 --2- 172.21.15.27:0/4128991284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122384d0 0x55f512603130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.111+0000 7f790dfef740 1 --2- 172.21.15.27:0/4128991284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122302f0 0x55f5125fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.111+0000 7f790dfef740 1 -- 172.21.15.27:0/4128991284 >> 172.21.15.27:0/4128991284 conn(0x55f51224d300 msgr2=0x55f512602a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.118+0000 7f790dfef740 1 -- 172.21.15.27:0/4128991284 shutdown_connections 2024-04-25T13:56:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.118+0000 7f790dfef740 1 -- 172.21.15.27:0/4128991284 wait complete. 2024-04-25T13:56:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.119+0000 7f790dfef740 1 -- start start 2024-04-25T13:56:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.119+0000 7f790dfef740 1 Processor -- start 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f790dfef740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f512607910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f790dfef740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 0x55f512607e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f7904ed4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f512607910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f7904ed4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f512607910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:51338/0 (socket says 172.21.15.27:51338) 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f790dfef740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f512699c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f7904ed4640 1 -- 172.21.15.27:0/2753929979 learned_addr learned my addr 172.21.15.27:0/2753929979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f790010f810 con 0x55f5122384d0 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7900110c80 con 0x55f5122302f0 2024-04-25T13:56:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f790010fee0 con 0x55f5122591b0 2024-04-25T13:56:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.120+0000 7f790dfef740 10 client.0 _ll_register_callbacks cb 0x7f78f4034620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.122+0000 7f78fffff640 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 0x55f512607e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.122 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.123+0000 7f79056d5640 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f512699c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.130 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.125+0000 7f7904ed4640 1 -- 172.21.15.27:0/2753929979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 msgr2=0x55f512699c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:09.130 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.125+0000 7f7904ed4640 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f512699c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.130 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.125+0000 7f7904ed4640 1 -- 172.21.15.27:0/2753929979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 msgr2=0x55f512607e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:09.130 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.125+0000 7f7904ed4640 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 0x55f512607e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.130 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.125+0000 7f7904ed4640 1 -- 172.21.15.27:0/2753929979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f512608410 con 0x55f5122302f0 2024-04-25T13:56:09.131 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.130+0000 7f7904ed4640 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f512607910 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00fb80 tx=0x7f78ec004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed8c9b9d1e4a182f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:09.131 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.131+0000 7f78dffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:56:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.135+0000 7f78dffff640 1 -- 172.21.15.27:0/2753929979 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec025250 con 0x55f5122302f0 2024-04-25T13:56:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.135+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78f8036030 con 0x55f5122302f0 2024-04-25T13:56:09.134 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.135+0000 7f78dffff640 1 -- 172.21.15.27:0/2753929979 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f78ec0253f0 con 0x55f5122302f0 2024-04-25T13:56:09.138 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.139+0000 7f78dffff640 1 -- 172.21.15.27:0/2753929979 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f78ec033050 con 0x55f5122302f0 2024-04-25T13:56:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.140+0000 7f790dfef740 10 client.28040 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.140+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55f51269b250 con 0x55f5122302f0 2024-04-25T13:56:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.150+0000 7f78dffff640 1 -- 172.21.15.27:0/2753929979 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78f8036030 con 0x55f5122302f0 2024-04-25T13:56:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.151+0000 7f78dffff640 1 -- 172.21.15.27:0/2753929979 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f78ec025ce0 con 0x55f5122302f0 2024-04-25T13:56:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.151+0000 7f790dfef740 10 client.28040 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.151+0000 7f790dfef740 10 client.28040 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.151+0000 7f790dfef740 20 client.28040 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.151+0000 7f790dfef740 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5121ed1c0 0x55f51269dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.151+0000 7f790dfef740 4 client.28040 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T13:56:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.151+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f512566ee0 con 0x55f5121ed1c0 2024-04-25T13:56:09.154 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.154+0000 7f78fffff640 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5121ed1c0 0x55f51269dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.160+0000 7f78fffff640 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5121ed1c0 0x55f51269dc40 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45b8d0c7ba43be48 server_cookie=c02a09748f00aec7 in_seq=0 out_seq=0 2024-04-25T13:56:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:09.160+0000 7f78dffff640 10 client.28040 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.309+0000 7f6afdc84640 1 -- 172.21.15.27:0/2779106596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 msgr2=0x7f6af8110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.309+0000 7f6afdc84640 1 --2- 172.21.15.27:0/2779106596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 0x7f6af8110440 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009900 tx=0x7f6ae802a440 comp rx=0 tx=0).stop 2024-04-25T13:56:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.314+0000 7f6afdc84640 1 -- 172.21.15.27:0/2779106596 shutdown_connections 2024-04-25T13:56:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.314+0000 7f6afdc84640 1 --2- 172.21.15.27:0/2779106596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 0x7f6af8110440 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.314+0000 7f6afdc84640 1 --2- 172.21.15.27:0/2779106596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6af810b310 0x7f6af810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.314+0000 7f6afdc84640 1 --2- 172.21.15.27:0/2779106596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6af810a420 0x7f6af810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.314+0000 7f6afdc84640 1 -- 172.21.15.27:0/2779106596 >> 172.21.15.27:0/2779106596 conn(0x7f6af80704e0 msgr2=0x7f6af8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:09.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.324+0000 7f6afdc84640 1 -- 172.21.15.27:0/2779106596 shutdown_connections 2024-04-25T13:56:09.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.326+0000 7f6afdc84640 1 -- 172.21.15.27:0/2779106596 wait complete. 2024-04-25T13:56:09.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.327+0000 7f6afdc84640 1 Processor -- start 2024-04-25T13:56:09.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.330+0000 7f6afdc84640 1 -- start start 2024-04-25T13:56:09.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.331+0000 7f6afdc84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6af810a420 0x7f6af81a9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.331+0000 7f6afdc84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6af810b310 0x7f6af81a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.331+0000 7f6afdc84640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 0x7f6af81b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.331+0000 7f6afdc84640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6af811e5e0 con 0x7f6af8110030 2024-04-25T13:56:09.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.331+0000 7f6afdc84640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6af811e460 con 0x7f6af810a420 2024-04-25T13:56:09.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.331+0000 7f6afdc84640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6af811e760 con 0x7f6af810b310 2024-04-25T13:56:09.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.334+0000 7f6af7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 0x7f6af81b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.334+0000 7f6af7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 0x7f6af81b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47678/0 (socket says 172.21.15.27:47678) 2024-04-25T13:56:09.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.334+0000 7f6af7fff640 1 -- 172.21.15.27:0/1852206274 learned_addr learned my addr 172.21.15.27:0/1852206274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:09.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.335+0000 7f6af77fe640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6af810a420 0x7f6af81a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.335+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6af810b310 0x7f6af81a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.336+0000 7f6af7fff640 1 -- 172.21.15.27:0/1852206274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6af810b310 msgr2=0x7f6af81a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:09.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.336+0000 7f6af7fff640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6af810b310 0x7f6af81a95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.336+0000 7f6af7fff640 1 -- 172.21.15.27:0/1852206274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6af810a420 msgr2=0x7f6af81a9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:09.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.336+0000 7f6af7fff640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6af810a420 0x7f6af81a9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.336+0000 7f6af7fff640 1 -- 172.21.15.27:0/1852206274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af8111c90 con 0x7f6af8110030 2024-04-25T13:56:09.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.338+0000 7f6af7fff640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 0x7f6af81b0560 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8030bc0 tx=0x7f6ae8030bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:09.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.342+0000 7f6af4ff9640 1 -- 172.21.15.27:0/1852206274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8040070 con 0x7f6af8110030 2024-04-25T13:56:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.342+0000 7f6af4ff9640 1 -- 172.21.15.27:0/1852206274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ae803cb20 con 0x7f6af8110030 2024-04-25T13:56:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.342+0000 7f6af4ff9640 1 -- 172.21.15.27:0/1852206274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8035030 con 0x7f6af8110030 2024-04-25T13:56:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.344+0000 7f6afdc84640 1 -- 172.21.15.27:0/1852206274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af8111f20 con 0x7f6af8110030 2024-04-25T13:56:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.344+0000 7f6afdc84640 1 -- 172.21.15.27:0/1852206274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af8112440 con 0x7f6af8110030 2024-04-25T13:56:09.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.347+0000 7f6b04c0a740 1 Processor -- start 2024-04-25T13:56:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.354+0000 7f6b04c0a740 1 -- start start 2024-04-25T13:56:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.354+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.354+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14f74d0 0x562be18c2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.354+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 0x562be18c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.354+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562be1826d80 con 0x562be14f74d0 2024-04-25T13:56:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.354+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562be17e08b0 con 0x562be14ff6b0 2024-04-25T13:56:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.354+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562be1822d40 con 0x562be1520390 2024-04-25T13:56:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.355+0000 7f6af4ff9640 1 -- 172.21.15.27:0/1852206274 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6ae803c670 con 0x7f6af8110030 2024-04-25T13:56:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.356+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.356+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 0x562be18c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.356+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18c1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:43532/0 (socket says 172.21.15.27:43532) 2024-04-25T13:56:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.356+0000 7f6af6ffd640 1 -- 172.21.15.27:0/1984657800 learned_addr learned my addr 172.21.15.27:0/1984657800 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.356+0000 7f6af4ff9640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6ae00afcf0 0x7f6ae00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.356+0000 7f6af4ff9640 1 -- 172.21.15.27:0/1852206274 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6ae80ff8a0 con 0x7f6af8110030 2024-04-25T13:56:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.357+0000 7f6af77fe640 1 --2- 172.21.15.27:0/1984657800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14f74d0 0x562be18c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.358+0000 7f6af7fff640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6ae00afcf0 0x7f6ae00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.359+0000 7f6af77fe640 1 -- 172.21.15.27:0/1984657800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 msgr2=0x562be18c1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.359+0000 7f6af77fe640 1 --2- 172.21.15.27:0/1984657800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18c1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.359+0000 7f6af77fe640 1 -- 172.21.15.27:0/1984657800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 msgr2=0x562be18c2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.359+0000 7f6af77fe640 1 --2- 172.21.15.27:0/1984657800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 0x562be18c2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.359+0000 7f6af77fe640 1 -- 172.21.15.27:0/1984657800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be17b1a40 con 0x562be14f74d0 2024-04-25T13:56:09.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.364+0000 7f6af77fe640 1 --2- 172.21.15.27:0/1984657800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14f74d0 0x562be18c2010 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009ed0 tx=0x7f6af000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67efc7893e6c7b75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:09.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.364+0000 7f6af7fff640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6ae00afcf0 0x7f6ae00b2200 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80c4fc0 tx=0x7f6ae803cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:09.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.366+0000 7f6afdc84640 1 -- 172.21.15.27:0/1984657800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af001a070 con 0x562be14f74d0 2024-04-25T13:56:09.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.366+0000 7f6afdc84640 1 -- 172.21.15.27:0/1984657800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6af00158c0 con 0x562be14f74d0 2024-04-25T13:56:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.366+0000 7f6afdc84640 1 -- 172.21.15.27:0/1984657800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af000a8a0 con 0x562be14f74d0 2024-04-25T13:56:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.369+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1984657800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14f74d0 msgr2=0x562be18c2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.369+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/1984657800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14f74d0 0x562be18c2010 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009ed0 tx=0x7f6af000c7f0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.370+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1984657800 shutdown_connections 2024-04-25T13:56:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.370+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/1984657800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 0x562be18c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.370+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/1984657800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14f74d0 0x562be18c2010 secure :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009ed0 tx=0x7f6af000c7f0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.370+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/1984657800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.370+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1984657800 >> 172.21.15.27:0/1984657800 conn(0x562be15144e0 msgr2=0x562be18bd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:09.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.374+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1984657800 shutdown_connections 2024-04-25T13:56:09.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.380+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1984657800 wait complete. 2024-04-25T13:56:09.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.380+0000 7f6b04c0a740 1 -- start start 2024-04-25T13:56:09.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.380+0000 7f6b04c0a740 1 Processor -- start 2024-04-25T13:56:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.383+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.383+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.383+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.383+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x562be18c37b0 con 0x562be14ff6b0 2024-04-25T13:56:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.383+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x562be17add10 con 0x562be14def60 2024-04-25T13:56:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.383+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x562be18c3930 con 0x562be1520390 2024-04-25T13:56:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.383+0000 7f6b04c0a740 10 client.0 _ll_register_callbacks cb 0x562be17da210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.387+0000 7f6af77fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.387+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.387+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.387+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47694/0 (socket says 172.21.15.27:47694) 2024-04-25T13:56:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.387+0000 7f6af6ffd640 1 -- 172.21.15.27:0/4042109436 learned_addr learned my addr 172.21.15.27:0/4042109436 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.389+0000 7f6af6ffd640 1 -- 172.21.15.27:0/4042109436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 msgr2=0x562be18cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.389+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.389+0000 7f6af6ffd640 1 -- 172.21.15.27:0/4042109436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 msgr2=0x562be18cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.389+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.389+0000 7f6af6ffd640 1 -- 172.21.15.27:0/4042109436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be16027d0 con 0x562be14def60 2024-04-25T13:56:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.390+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d2fd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6aec01aee0 tx=0x7f6aec004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b750649721f595 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.390+0000 7f6ad6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-04-25T13:56:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.395+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/4042109436 <== mon.2 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00f040 con 0x562be14def60 2024-04-25T13:56:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.395+0000 7f6b04c0a740 1 -- 172.21.15.27:0/4042109436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6af00158c0 con 0x562be14def60 2024-04-25T13:56:09.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.395+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/4042109436 <== mon.2 v2:172.21.15.45:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6aec018df0 con 0x562be14def60 2024-04-25T13:56:09.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.400+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/4042109436 <== mon.2 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6aec025450 con 0x562be14def60 2024-04-25T13:56:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.401+0000 7f6b04c0a740 10 client.28049 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:09.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.401+0000 7f6b04c0a740 1 -- 172.21.15.27:0/4042109436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562be15383f0 con 0x562be14def60 2024-04-25T13:56:09.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.404+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/4042109436 <== mon.2 v2:172.21.15.45:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6af00158c0 con 0x562be14def60 2024-04-25T13:56:09.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.404+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/4042109436 <== mon.2 v2:172.21.15.45:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6aec015070 con 0x562be14def60 2024-04-25T13:56:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.409+0000 7f6b04c0a740 10 client.28049 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.409+0000 7f6b04c0a740 10 client.28049 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.409+0000 7f6b04c0a740 20 client.28049 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.409+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562be14b43a0 0x562be18d9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.409+0000 7f6b04c0a740 4 client.28049 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:56:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.409+0000 7f6b04c0a740 1 -- 172.21.15.27:0/4042109436 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562be1834990 con 0x562be14b43a0 2024-04-25T13:56:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.411+0000 7f6af77fe640 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562be14b43a0 0x562be18d9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.420+0000 7f6af77fe640 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562be14b43a0 0x562be18d9450 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37935a10f7af7838 server_cookie=6e7ea1773d3c4afb in_seq=0 out_seq=0 2024-04-25T13:56:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:09.421+0000 7f6ad6ffd640 10 client.28049 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:10.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:10 smithi122 ceph-mon[32843]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.5 MiB/s wr, 669 op/s 2024-04-25T13:56:10.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:10 smithi045 ceph-mon[32731]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.5 MiB/s wr, 669 op/s 2024-04-25T13:56:10.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:10 smithi027 ceph-mon[29140]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.5 MiB/s wr, 669 op/s 2024-04-25T13:56:12.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:12 smithi122 ceph-mon[32843]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.2 MiB/s wr, 631 op/s 2024-04-25T13:56:12.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:12 smithi045 ceph-mon[32731]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.2 MiB/s wr, 631 op/s 2024-04-25T13:56:12.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:12 smithi027 ceph-mon[29140]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.2 MiB/s wr, 631 op/s 2024-04-25T13:56:14.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:14 smithi122 ceph-mon[32843]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 913 op/s 2024-04-25T13:56:14.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:14 smithi045 ceph-mon[32731]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 913 op/s 2024-04-25T13:56:14.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:14 smithi027 ceph-mon[29140]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 913 op/s 2024-04-25T13:56:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f78dffff640 1 -- 172.21.15.27:0/2753929979 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f512566ee0 con 0x55f5121ed1c0 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f78dffff640 10 client.28040 handle_command_reply: tid=0 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 client.28040 shutdown 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 msgr2=0x55f512607910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f512607910 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00fb80 tx=0x7f78ec004730 comp rx=0 tx=0).stop 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 shutdown_connections 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55f5121ed1c0 0x55f51269dc40 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f5122591b0 0x55f512699c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f78dffff640 0 client.28040 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55f5122384d0 0x55f512607e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 --2- 172.21.15.27:0/2753929979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f5122302f0 0x55f512607910 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 >> 172.21.15.27:0/2753929979 conn(0x55f51224d300 msgr2=0x55f5125f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 shutdown_connections 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.072+0000 7f790dfef740 1 -- 172.21.15.27:0/2753929979 wait complete. 2024-04-25T13:56:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f790dfef740 20 client.28040 trim_cache size 0 max 16384 2024-04-25T13:56:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:{ 2024-04-25T13:56:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "message": "", 2024-04-25T13:56:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout: "return_code": 0 2024-04-25T13:56:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stdout:} 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/4042109436 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562be1834990 con 0x562be14b43a0 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6ad6ffd640 10 client.28049 handle_command_reply: tid=0 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 client.28049 shutdown 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 -- 172.21.15.27:0/4042109436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 msgr2=0x562be18d2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d2fd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6aec01aee0 tx=0x7f6aec004750 comp rx=0 tx=0).stop 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 -- 172.21.15.27:0/4042109436 shutdown_connections 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562be14b43a0 0x562be18d9450 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d2fd0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6ad6ffd640 0 client.28049 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/4042109436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 -- 172.21.15.27:0/4042109436 >> 172.21.15.27:0/4042109436 conn(0x562be15144e0 msgr2=0x562be18c04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 -- 172.21.15.27:0/4042109436 shutdown_connections 2024-04-25T13:56:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 -- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f00afcf0 msgr2=0x7f78f00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f00afcf0 0x7f78f00b2200 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x55f5125f6090 tx=0x7f78ec005e40 comp rx=0 tx=0).stop 2024-04-25T13:56:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 -- 172.21.15.27:0/291737136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7900107970 msgr2=0x7f7900072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7900107970 0x7f7900072ec0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f78f80098d0 tx=0x7f78f8003ba0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 1 -- 172.21.15.27:0/4042109436 wait complete. 2024-04-25T13:56:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 -- 172.21.15.27:0/291737136 shutdown_connections 2024-04-25T13:56:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f00afcf0 0x7f78f00b2200 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7900107970 0x7f7900072ec0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7900102800 0x7f7900071f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 --2- 172.21.15.27:0/291737136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7900101dc0 0x7f7900071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f6b04c0a740 20 client.28049 trim_cache size 0 max 16384 2024-04-25T13:56:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.073+0000 7f7907138640 1 -- 172.21.15.27:0/291737136 >> 172.21.15.27:0/291737136 conn(0x7f79000fd390 msgr2=0x7f7900103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.074+0000 7f7907138640 1 -- 172.21.15.27:0/291737136 shutdown_connections 2024-04-25T13:56:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi027.front.sepia.ceph.com.smithi027.stderr:2024-04-25T13:56:16.074+0000 7f7907138640 1 -- 172.21.15.27:0/291737136 wait complete. 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.099+0000 7f6b04c0a740 1 Processor -- start 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.099+0000 7f6b04c0a740 1 -- start start 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.099+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14def60 0x562be18ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.099+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 0x562be18bd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.099+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.099+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6af8119080 con 0x562be14def60 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.099+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81c3040 con 0x562be14ff6b0 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.099+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81196f0 con 0x562be1520390 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14def60 0x562be18ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14def60 0x562be18ca7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60830/0 (socket says 172.21.15.27:60830) 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 -- 172.21.15.27:0/1533180389 learned_addr learned my addr 172.21.15.27:0/1533180389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af77fe640 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 0x562be18bd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 -- 172.21.15.27:0/1533180389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 msgr2=0x562be18d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18d2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 -- 172.21.15.27:0/1533180389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 msgr2=0x562be18bd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 0x562be18bd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 -- 172.21.15.27:0/1533180389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be14f80f0 con 0x562be14def60 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14def60 0x562be18ca7c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0164f0 tx=0x7f6aec01a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13eab2b05b484e9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6add7fa640 1 -- 172.21.15.27:0/1533180389 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec016cd0 con 0x562be14def60 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6add7fa640 1 -- 172.21.15.27:0/1533180389 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6aec035070 con 0x562be14def60 2024-04-25T13:56:16.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.100+0000 7f6add7fa640 1 -- 172.21.15.27:0/1533180389 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0038f0 con 0x562be14def60 2024-04-25T13:56:16.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1533180389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14def60 msgr2=0x562be18ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14def60 0x562be18ca7c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0164f0 tx=0x7f6aec01a7b0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1533180389 shutdown_connections 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18d2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14ff6b0 0x562be18bd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/1533180389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14def60 0x562be18ca7c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1533180389 >> 172.21.15.27:0/1533180389 conn(0x562be15144e0 msgr2=0x562be18bcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1533180389 shutdown_connections 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.101+0000 7f6b04c0a740 1 -- 172.21.15.27:0/1533180389 wait complete. 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 1 -- start start 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 1 Processor -- start 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18d0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6af8119dc0 con 0x562be14ff6b0 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81c2970 con 0x562be14def60 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81c1bd0 con 0x562be1520390 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6b04c0a740 10 client.0 _ll_register_callbacks cb 0x7f6aec00b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6af77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18d0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6af77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18d0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60832/0 (socket says 172.21.15.27:60832) 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6af77fe640 1 -- 172.21.15.27:0/2373917629 learned_addr learned my addr 172.21.15.27:0/2373917629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6af6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.102+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6af77fe640 1 -- 172.21.15.27:0/2373917629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 msgr2=0x562be18d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6af77fe640 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18d1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6af77fe640 1 -- 172.21.15.27:0/2373917629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 msgr2=0x562be18d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6af77fe640 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6af77fe640 1 -- 172.21.15.27:0/2373917629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be17e74b0 con 0x562be14ff6b0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6af77fe640 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18d0fa0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6af0018040 tx=0x7f6af0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f07779eca749b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6ad6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/2373917629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af001a040 con 0x562be14ff6b0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6b04c0a740 1 -- 172.21.15.27:0/2373917629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6aec035070 con 0x562be14ff6b0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/2373917629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6af0003a10 con 0x562be14ff6b0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.103+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/2373917629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f6af00095d0 con 0x562be14ff6b0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6b04c0a740 10 client.18402 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6b04c0a740 1 -- 172.21.15.27:0/2373917629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x562be18d3580 con 0x562be14ff6b0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/2373917629 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6aec035070 con 0x562be14ff6b0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/2373917629 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f6af00155b0 con 0x562be14ff6b0 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6b04c0a740 10 client.18402 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6b04c0a740 10 client.18402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6b04c0a740 20 client.18402 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:16.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562be1941790 0x562be1943bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6b04c0a740 4 client.18402 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6b04c0a740 1 -- 172.21.15.27:0/2373917629 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562be182e800 con 0x562be1941790 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.104+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562be1941790 0x562be1943bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.105+0000 7f6af6ffd640 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562be1941790 0x562be1943bb0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aec5ca46bbb14c57 server_cookie=90d70b0fe8d711a in_seq=0 out_seq=0 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.105+0000 7f6ad6ffd640 10 client.18402 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6ad6ffd640 1 -- 172.21.15.27:0/2373917629 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562be182e800 con 0x562be1941790 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6ad6ffd640 10 client.18402 handle_command_reply: tid=0 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 client.18402 shutdown 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 -- 172.21.15.27:0/2373917629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 msgr2=0x562be18d0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18d0fa0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6af0018040 tx=0x7f6af0009920 comp rx=0 tx=0).stop 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 -- 172.21.15.27:0/2373917629 shutdown_connections 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x562be1941790 0x562be1943bb0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562be1520390 0x562be18d1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562be14ff6b0 0x562be18d0fa0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 --2- 172.21.15.27:0/2373917629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562be14def60 0x562be18d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 -- 172.21.15.27:0/2373917629 >> 172.21.15.27:0/2373917629 conn(0x562be15144e0 msgr2=0x562be18c0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6ad6ffd640 0 client.18402 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 -- 172.21.15.27:0/2373917629 shutdown_connections 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 1 -- 172.21.15.27:0/2373917629 wait complete. 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.106+0000 7f6b04c0a740 20 client.18402 trim_cache size 0 max 16384 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:56:16.110 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:56:16.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 -- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6ae00afcf0 msgr2=0x7f6ae00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6ae00afcf0 0x7f6ae00b2200 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80c4fc0 tx=0x7f6ae803cf90 comp rx=0 tx=0).stop 2024-04-25T13:56:16.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 -- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 msgr2=0x7f6af81b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 0x7f6af81b0560 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8030bc0 tx=0x7f6ae8030bf0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 -- 172.21.15.27:0/1852206274 shutdown_connections 2024-04-25T13:56:16.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f6ae00afcf0 0x7f6ae00b2200 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6af8110030 0x7f6af81b0560 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6af810b310 0x7f6af81a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 --2- 172.21.15.27:0/1852206274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6af810a420 0x7f6af81a9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 -- 172.21.15.27:0/1852206274 >> 172.21.15.27:0/1852206274 conn(0x7f6af80704e0 msgr2=0x7f6af810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 -- 172.21.15.27:0/1852206274 shutdown_connections 2024-04-25T13:56:16.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.107+0000 7f6ade7fc640 1 -- 172.21.15.27:0/1852206274 wait complete. 2024-04-25T13:56:16.118 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:56:16.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:56:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 -- 172.21.15.27:0/147675327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474103b00 msgr2=0x7fc474104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 --2- 172.21.15.27:0/147675327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474103b00 0x7fc474104800 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc470009900 tx=0x7fc47002a410 comp rx=0 tx=0).stop 2024-04-25T13:56:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 -- 172.21.15.27:0/147675327 shutdown_connections 2024-04-25T13:56:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 --2- 172.21.15.27:0/147675327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc474109a40 0x7fc47410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 --2- 172.21.15.27:0/147675327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc474104d40 0x7fc474105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 --2- 172.21.15.27:0/147675327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474103b00 0x7fc474104800 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 -- 172.21.15.27:0/147675327 >> 172.21.15.27:0/147675327 conn(0x7fc4740fecf0 msgr2=0x7fc4741011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 -- 172.21.15.27:0/147675327 shutdown_connections 2024-04-25T13:56:16.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.287+0000 7fc47beaa640 1 -- 172.21.15.27:0/147675327 wait complete. 2024-04-25T13:56:16.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47beaa640 1 Processor -- start 2024-04-25T13:56:16.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47beaa640 1 -- start start 2024-04-25T13:56:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47beaa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc474103b00 0x7fc474076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47beaa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc474104d40 0x7fc474074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47beaa640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474109a40 0x7fc474075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47beaa640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc47410f5e0 con 0x7fc474109a40 2024-04-25T13:56:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47beaa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc47410f460 con 0x7fc474103b00 2024-04-25T13:56:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47beaa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fc47410f760 con 0x7fc474104d40 2024-04-25T13:56:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47a447640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474109a40 0x7fc474075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc479445640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc474104d40 0x7fc474074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47a447640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474109a40 0x7fc474075430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60862/0 (socket says 172.21.15.27:60862) 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc47a447640 1 -- 172.21.15.27:0/1083319513 learned_addr learned my addr 172.21.15.27:0/1083319513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.288+0000 7fc479c46640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc474103b00 0x7fc474076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc47a447640 1 -- 172.21.15.27:0/1083319513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc474104d40 msgr2=0x7fc474074ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc47a447640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc474104d40 0x7fc474074ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc47a447640 1 -- 172.21.15.27:0/1083319513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc474103b00 msgr2=0x7fc474076850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc47a447640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc474103b00 0x7fc474076850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc47a447640 1 -- 172.21.15.27:0/1083319513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc474075b60 con 0x7fc474109a40 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc479445640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc474104d40 0x7fc474074ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc47a447640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474109a40 0x7fc474075430 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00efd0 tx=0x7fc46c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc46affd640 1 -- 172.21.15.27:0/1083319513 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c003b70 con 0x7fc474109a40 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc46affd640 1 -- 172.21.15.27:0/1083319513 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc46c003d10 con 0x7fc474109a40 2024-04-25T13:56:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4741a42e0 con 0x7fc474109a40 2024-04-25T13:56:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc46affd640 1 -- 172.21.15.27:0/1083319513 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c0108a0 con 0x7fc474109a40 2024-04-25T13:56:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.289+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4741a4750 con 0x7fc474109a40 2024-04-25T13:56:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.290+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc474108850 con 0x7fc474109a40 2024-04-25T13:56:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.292+0000 7fc46affd640 1 -- 172.21.15.27:0/1083319513 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc46c004b40 con 0x7fc474109a40 2024-04-25T13:56:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.293+0000 7fc46affd640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4600afaf0 0x7fc4600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.293+0000 7fc479c46640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4600afaf0 0x7fc4600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.293+0000 7fc46affd640 1 -- 172.21.15.27:0/1083319513 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fc46c0df3f0 con 0x7fc474109a40 2024-04-25T13:56:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.293+0000 7fc479c46640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4600afaf0 0x7fc4600b2000 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc470005ab0 tx=0x7fc470031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.297+0000 7fc46affd640 1 -- 172.21.15.27:0/1083319513 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc46c0a8000 con 0x7fc474109a40 2024-04-25T13:56:16.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:16 smithi122 ceph-mon[32843]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.8 MiB/s wr, 818 op/s 2024-04-25T13:56:16.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:16 smithi045 ceph-mon[32731]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.8 MiB/s wr, 818 op/s 2024-04-25T13:56:16.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:16 smithi027 ceph-mon[29140]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.8 MiB/s wr, 818 op/s 2024-04-25T13:56:16.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.524+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc47402db50 con 0x7fc474109a40 2024-04-25T13:56:16.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.525+0000 7fc46affd640 1 -- 172.21.15.27:0/1083319513 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fc46c0a7900 con 0x7fc474109a40 2024-04-25T13:56:16.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:16.525 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:56:16.525 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4600afaf0 msgr2=0x7fc4600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4600afaf0 0x7fc4600b2000 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc470005ab0 tx=0x7fc470031400 comp rx=0 tx=0).stop 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474109a40 msgr2=0x7fc474075430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474109a40 0x7fc474075430 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00efd0 tx=0x7fc46c00c6a0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 shutdown_connections 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fc4600afaf0 0x7fc4600b2000 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc474109a40 0x7fc474075430 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc474104d40 0x7fc474074ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 --2- 172.21.15.27:0/1083319513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc474103b00 0x7fc474076850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 >> 172.21.15.27:0/1083319513 conn(0x7fc4740fecf0 msgr2=0x7fc474105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 shutdown_connections 2024-04-25T13:56:16.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.527+0000 7fc47beaa640 1 -- 172.21.15.27:0/1083319513 wait complete. 2024-04-25T13:56:16.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:56:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 -- 172.21.15.27:0/3191466388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c1029e0 msgr2=0x7f2d4c1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 --2- 172.21.15.27:0/3191466388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c1029e0 0x7f2d4c1036e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009900 tx=0x7f2d4802a410 comp rx=0 tx=0).stop 2024-04-25T13:56:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 -- 172.21.15.27:0/3191466388 shutdown_connections 2024-04-25T13:56:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 --2- 172.21.15.27:0/3191466388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4c108920 0x7f2d4c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 --2- 172.21.15.27:0/3191466388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d4c103c20 0x7f2d4c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 --2- 172.21.15.27:0/3191466388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c1029e0 0x7f2d4c1036e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 -- 172.21.15.27:0/3191466388 >> 172.21.15.27:0/3191466388 conn(0x7f2d4c0fdbd0 msgr2=0x7f2d4c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 -- 172.21.15.27:0/3191466388 shutdown_connections 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.653+0000 7f2d54334640 1 -- 172.21.15.27:0/3191466388 wait complete. 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d54334640 1 Processor -- start 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d54334640 1 -- start start 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d54334640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d4c1029e0 0x7f2d4c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d54334640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4c103c20 0x7f2d4c19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d54334640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c108920 0x7f2d4c1a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d54334640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c10f610 con 0x7f2d4c108920 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d54334640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c10f490 con 0x7f2d4c103c20 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d54334640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c10f790 con 0x7f2d4c1029e0 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c108920 0x7f2d4c1a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d4c1029e0 0x7f2d4c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c108920 0x7f2d4c1a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60882/0 (socket says 172.21.15.27:60882) 2024-04-25T13:56:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d518cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4c103c20 0x7f2d4c19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d4c1029e0 0x7f2d4c19a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47876/0 (socket says 172.21.15.27:47876) 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 -- 172.21.15.27:0/1497838106 learned_addr learned my addr 172.21.15.27:0/1497838106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 -- 172.21.15.27:0/1497838106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d4c1029e0 msgr2=0x7f2d4c19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d4c1029e0 0x7f2d4c19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 -- 172.21.15.27:0/1497838106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4c103c20 msgr2=0x7f2d4c19aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4c103c20 0x7f2d4c19aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 -- 172.21.15.27:0/1497838106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c1a20b0 con 0x7f2d4c108920 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.654+0000 7f2d528d1640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c108920 0x7f2d4c1a1a40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400eb10 tx=0x7f2d4400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.655+0000 7f2d437fe640 1 -- 172.21.15.27:0/1497838106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44004b20 con 0x7f2d4c108920 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.655+0000 7f2d437fe640 1 -- 172.21.15.27:0/1497838106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d44004cc0 con 0x7f2d4c108920 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.655+0000 7f2d437fe640 1 -- 172.21.15.27:0/1497838106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44010670 con 0x7f2d4c108920 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.655+0000 7f2d54334640 1 -- 172.21.15.27:0/1497838106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4c1a23a0 con 0x7f2d4c108920 2024-04-25T13:56:16.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.655+0000 7f2d54334640 1 -- 172.21.15.27:0/1497838106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d4c1a28a0 con 0x7f2d4c108920 2024-04-25T13:56:16.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.658+0000 7f2d5b1e8740 1 Processor -- start 2024-04-25T13:56:16.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.658+0000 7f2d437fe640 1 -- 172.21.15.27:0/1497838106 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2d44005020 con 0x7f2d4c108920 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.658+0000 7f2d5b1e8740 1 -- start start 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x561473907810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56147353c4d0 0x561473907d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614735446b0 0x561473908a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x56147386bd80 con 0x56147353c4d0 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561473825a20 con 0x5614735446b0 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x561473867d40 con 0x561473565390 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56147353c4d0 0x561473907d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d518cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x561473907810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56147353c4d0 0x561473907d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60892/0 (socket says 172.21.15.27:60892) 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d520d0640 1 -- 172.21.15.27:0/2688543003 learned_addr learned my addr 172.21.15.27:0/2688543003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d518cf640 1 --2- 172.21.15.27:0/2688543003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614735446b0 0x561473908a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d437fe640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2d380afaf0 0x7f2d380b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d518cf640 1 -- 172.21.15.27:0/2688543003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614735446b0 msgr2=0x561473908a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d518cf640 1 --2- 172.21.15.27:0/2688543003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614735446b0 0x561473908a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d518cf640 1 -- 172.21.15.27:0/2688543003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56147353c4d0 msgr2=0x561473907d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d518cf640 1 --2- 172.21.15.27:0/2688543003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56147353c4d0 0x561473907d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d518cf640 1 -- 172.21.15.27:0/2688543003 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561473830c30 con 0x561473565390 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d528d1640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2d380afaf0 0x7f2d380b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d518cf640 1 --2- 172.21.15.27:0/2688543003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x561473907810 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0058c0 tx=0x7f2d3c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2993a62c92b77c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.659+0000 7f2d437fe640 1 -- 172.21.15.27:0/1497838106 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2d440df1b0 con 0x7f2d4c108920 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.660+0000 7f2d40bfe640 1 -- 172.21.15.27:0/2688543003 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c005020 con 0x561473565390 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.660+0000 7f2d40bfe640 1 -- 172.21.15.27:0/2688543003 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c0047a0 con 0x561473565390 2024-04-25T13:56:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.660+0000 7f2d40bfe640 1 -- 172.21.15.27:0/2688543003 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c00f9b0 con 0x561473565390 2024-04-25T13:56:16.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.660+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/2688543003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 msgr2=0x561473907810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.660+0000 7f2d528d1640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2d380afaf0 0x7f2d380b2000 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2d440dd050 tx=0x7f2d440ca040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.660+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/2688543003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x561473907810 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0058c0 tx=0x7f2d3c005850 comp rx=0 tx=0).stop 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/2688543003 shutdown_connections 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/2688543003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614735446b0 0x561473908a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/2688543003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56147353c4d0 0x561473907d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/2688543003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x561473907810 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/2688543003 >> 172.21.15.27:0/2688543003 conn(0x5614735594e0 msgr2=0x561473903020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/2688543003 shutdown_connections 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/2688543003 wait complete. 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 -- start start 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 Processor -- start 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 0x561473911590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x561473911ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 0x561473918a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x561473909530 con 0x5614735446b0 2024-04-25T13:56:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x5614737f2dc0 con 0x561473565390 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x5614739096b0 con 0x56147353c4d0 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.661+0000 7f2d5b1e8740 10 client.0 _ll_register_callbacks cb 0x56147380e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x561473911ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d518cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 0x561473911590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x561473911ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60902/0 (socket says 172.21.15.27:60902) 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 -- 172.21.15.27:0/3662229796 learned_addr learned my addr 172.21.15.27:0/3662229796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d518cf640 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 0x561473918a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 -- 172.21.15.27:0/3662229796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 msgr2=0x561473911590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 0x561473911590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 -- 172.21.15.27:0/3662229796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 msgr2=0x561473918a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 0x561473918a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 -- 172.21.15.27:0/3662229796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56147357d3f0 con 0x5614735446b0 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d520d0640 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x561473911ad0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2d48005ab0 tx=0x7f2d48031db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a7cf75a9ac6a9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3662229796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48035040 con 0x5614735446b0 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3662229796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2d48004040 con 0x5614735446b0 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3662229796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d3c0047a0 con 0x5614735446b0 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3662229796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2d4804d070 con 0x5614735446b0 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d5b1e8740 10 client.18420 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.662+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3662229796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x5614736477d0 con 0x5614735446b0 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3662229796 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d3c0047a0 con 0x5614735446b0 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3662229796 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2d48044020 con 0x5614735446b0 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d5b1e8740 10 client.18420 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d5b1e8740 10 client.18420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d5b1e8740 20 client.18420 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561473523f60 0x56147391f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d5b1e8740 4 client.18420 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3662229796 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561473879990 con 0x561473523f60 2024-04-25T13:56:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d518cf640 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561473523f60 0x56147391f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d35ffb640 10 client.18420 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.663+0000 7f2d518cf640 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561473523f60 0x56147391f2e0 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67b08b8fd0747895 server_cookie=99d55a6624f0fe78 in_seq=0 out_seq=0 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.664+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3662229796 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561473879990 con 0x561473523f60 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.664+0000 7f2d35ffb640 10 client.18420 handle_command_reply: tid=0 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.664+0000 7f2d5b1e8740 1 client.18420 shutdown 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3662229796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 msgr2=0x561473911ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x561473911ad0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2d48005ab0 tx=0x7f2d48031db0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3662229796 shutdown_connections 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x561473523f60 0x56147391f2e0 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 0x561473918a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x561473911ad0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3662229796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 0x561473911590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d35ffb640 0 client.18420 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3662229796 >> 172.21.15.27:0/3662229796 conn(0x5614735594e0 msgr2=0x561473902040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3662229796 shutdown_connections 2024-04-25T13:56:16.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3662229796 wait complete. 2024-04-25T13:56:16.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.665+0000 7f2d5b1e8740 20 client.18420 trim_cache size 0 max 16384 2024-04-25T13:56:16.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d5b1e8740 1 Processor -- start 2024-04-25T13:56:16.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d5b1e8740 1 -- start start 2024-04-25T13:56:16.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56147353c4d0 0x56147390c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x56147390c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x56147390fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1a91d0 con 0x5614735446b0 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1aba50 con 0x56147353c4d0 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1a9840 con 0x561473565390 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x56147390c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x56147390c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60906/0 (socket says 172.21.15.27:60906) 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 -- 172.21.15.27:0/4158407780 learned_addr learned my addr 172.21.15.27:0/4158407780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d518cf640 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x56147390fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d518cf640 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56147353c4d0 0x56147390c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 -- 172.21.15.27:0/4158407780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 msgr2=0x56147390fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x56147390fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 -- 172.21.15.27:0/4158407780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56147353c4d0 msgr2=0x56147390c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56147353c4d0 0x56147390c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 -- 172.21.15.27:0/4158407780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614739126d0 con 0x5614735446b0 2024-04-25T13:56:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d518cf640 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56147353c4d0 0x56147390c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.679+0000 7f2d520d0640 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x56147390c850 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2d48031940 tx=0x7f2d4802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b1176b982e7a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d37fff640 1 -- 172.21.15.27:0/4158407780 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48035070 con 0x5614735446b0 2024-04-25T13:56:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d37fff640 1 -- 172.21.15.27:0/4158407780 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d48057070 con 0x5614735446b0 2024-04-25T13:56:16.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d37fff640 1 -- 172.21.15.27:0/4158407780 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48000e60 con 0x5614735446b0 2024-04-25T13:56:16.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/4158407780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 msgr2=0x56147390c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x56147390c850 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2d48031940 tx=0x7f2d4802a410 comp rx=0 tx=0).stop 2024-04-25T13:56:16.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/4158407780 shutdown_connections 2024-04-25T13:56:16.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561473565390 0x56147390fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x56147390c850 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/4158407780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56147353c4d0 0x56147390c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/4158407780 >> 172.21.15.27:0/4158407780 conn(0x5614735594e0 msgr2=0x5614739077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/4158407780 shutdown_connections 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.680+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/4158407780 wait complete. 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 1 -- start start 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 1 Processor -- start 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 0x5614739152a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x5614739157e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 0x561473916710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1a9f10 con 0x5614735446b0 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1ab380 con 0x561473565390 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1aa5e0 con 0x56147353c4d0 2024-04-25T13:56:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d5b1e8740 10 client.0 _ll_register_callbacks cb 0x7f2d48009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x5614739157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d518cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 0x5614739152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d520d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x5614739157e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60918/0 (socket says 172.21.15.27:60918) 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d520d0640 1 -- 172.21.15.27:0/3978863300 learned_addr learned my addr 172.21.15.27:0/3978863300 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d518cf640 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 0x561473916710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d520d0640 1 -- 172.21.15.27:0/3978863300 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 msgr2=0x5614739152a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d520d0640 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 0x5614739152a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d520d0640 1 -- 172.21.15.27:0/3978863300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 msgr2=0x561473916710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d520d0640 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 0x561473916710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.681+0000 7f2d520d0640 1 -- 172.21.15.27:0/3978863300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561473916cd0 con 0x5614735446b0 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d520d0640 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x5614739157e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2d48003e50 tx=0x7f2d48046740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9644911e8a1ac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3978863300 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48035040 con 0x5614735446b0 2024-04-25T13:56:16.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3978863300 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f2d48003fc0 con 0x5614735446b0 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3978863300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d48057070 con 0x5614735446b0 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3978863300 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2d480041c0 con 0x5614735446b0 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d5b1e8740 10 client.18426 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3978863300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x561473917f50 con 0x5614735446b0 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3978863300 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d48057070 con 0x5614735446b0 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3978863300 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f2d48044030 con 0x5614735446b0 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d5b1e8740 10 client.18426 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d5b1e8740 10 client.18426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d5b1e8740 20 client.18426 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.682+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5614734f93a0 0x5614739899d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 4 client.18426 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3978863300 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561473873800 con 0x5614734f93a0 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d518cf640 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5614734f93a0 0x5614739899d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:16.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d35ffb640 10 client.18426 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d518cf640 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5614734f93a0 0x5614739899d0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0c9ba0eb487e5cf server_cookie=d49b9e66970da7b3 in_seq=0 out_seq=0 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d35ffb640 1 -- 172.21.15.27:0/3978863300 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561473873800 con 0x5614734f93a0 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d35ffb640 10 client.18426 handle_command_reply: tid=0 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 client.18426 shutdown 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3978863300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 msgr2=0x5614739157e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x5614739157e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2d48003e50 tx=0x7f2d48046740 comp rx=0 tx=0).stop 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3978863300 shutdown_connections 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x5614734f93a0 0x5614739899d0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561473565390 0x561473916710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5614735446b0 0x5614739157e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 --2- 172.21.15.27:0/3978863300 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56147353c4d0 0x5614739152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d35ffb640 0 client.18426 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.683+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3978863300 >> 172.21.15.27:0/3978863300 conn(0x5614735594e0 msgr2=0x5614739077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3978863300 shutdown_connections 2024-04-25T13:56:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d5b1e8740 1 -- 172.21.15.27:0/3978863300 wait complete. 2024-04-25T13:56:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d5b1e8740 20 client.18426 trim_cache size 0 max 16384 2024-04-25T13:56:16.690 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 -- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2d380afaf0 msgr2=0x7f2d380b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2d380afaf0 0x7f2d380b2000 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2d440dd050 tx=0x7f2d440ca040 comp rx=0 tx=0).stop 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 -- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c108920 msgr2=0x7f2d4c1a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c108920 0x7f2d4c1a1a40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400eb10 tx=0x7f2d4400c2d0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 -- 172.21.15.27:0/1497838106 shutdown_connections 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f2d380afaf0 0x7f2d380b2000 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d4c108920 0x7f2d4c1a1a40 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d4c103c20 0x7f2d4c19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 --2- 172.21.15.27:0/1497838106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d4c1029e0 0x7f2d4c19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 -- 172.21.15.27:0/1497838106 >> 172.21.15.27:0/1497838106 conn(0x7f2d4c0fdbd0 msgr2=0x7f2d4c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 -- 172.21.15.27:0/1497838106 shutdown_connections 2024-04-25T13:56:16.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:16.684+0000 7f2d54334640 1 -- 172.21.15.27:0/1497838106 wait complete. 2024-04-25T13:56:17.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:17 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1083319513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:17.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:17 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1083319513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:17.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:17 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1083319513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:17.696 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T13:56:17.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:56:17.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T13:56:17.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.900+0000 7fa977fff640 1 -- 172.21.15.27:0/1513659692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97000ad50 msgr2=0x7fa97000b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:17.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.900+0000 7fa977fff640 1 --2- 172.21.15.27:0/1513659692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97000ad50 0x7fa97000b160 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009860 tx=0x7fa96c02a380 comp rx=0 tx=0).stop 2024-04-25T13:56:17.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.901+0000 7fa977fff640 1 -- 172.21.15.27:0/1513659692 shutdown_connections 2024-04-25T13:56:17.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.901+0000 7fa977fff640 1 --2- 172.21.15.27:0/1513659692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa97000ba50 0x7fa97000be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.901+0000 7fa977fff640 1 --2- 172.21.15.27:0/1513659692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97000ad50 0x7fa97000b160 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.901+0000 7fa977fff640 1 --2- 172.21.15.27:0/1513659692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa970009ac0 0x7fa97000a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.901+0000 7fa977fff640 1 -- 172.21.15.27:0/1513659692 >> 172.21.15.27:0/1513659692 conn(0x7fa970093dc0 msgr2=0x7fa970096250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.901+0000 7fa977fff640 1 -- 172.21.15.27:0/1513659692 shutdown_connections 2024-04-25T13:56:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.901+0000 7fa977fff640 1 -- 172.21.15.27:0/1513659692 wait complete. 2024-04-25T13:56:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.902+0000 7fa977fff640 1 Processor -- start 2024-04-25T13:56:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.904+0000 7fa977fff640 1 -- start start 2024-04-25T13:56:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.904+0000 7fa977fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa970009ac0 0x7fa9700a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.904+0000 7fa977fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97000ad50 0x7fa9700a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.904+0000 7fa977fff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa97000ba50 0x7fa9700a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.904+0000 7fa977fff640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa9700a8ac0 con 0x7fa970009ac0 2024-04-25T13:56:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.904+0000 7fa977fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9700a8940 con 0x7fa97000ad50 2024-04-25T13:56:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.904+0000 7fa977fff640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa9700a8c40 con 0x7fa97000ba50 2024-04-25T13:56:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.906+0000 7fa976ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa970009ac0 0x7fa9700a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.906+0000 7fa976ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa970009ac0 0x7fa9700a37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60938/0 (socket says 172.21.15.27:60938) 2024-04-25T13:56:17.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.906+0000 7fa976ffd640 1 -- 172.21.15.27:0/2148511859 learned_addr learned my addr 172.21.15.27:0/2148511859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:17.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.907+0000 7fa9767fc640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97000ad50 0x7fa9700a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.908+0000 7fa9777fe640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa97000ba50 0x7fa9700a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.909+0000 7fa976ffd640 1 -- 172.21.15.27:0/2148511859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa97000ba50 msgr2=0x7fa9700a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.909+0000 7fa976ffd640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa97000ba50 0x7fa9700a4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.909+0000 7fa976ffd640 1 -- 172.21.15.27:0/2148511859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97000ad50 msgr2=0x7fa9700a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.909+0000 7fa976ffd640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97000ad50 0x7fa9700a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.909+0000 7fa976ffd640 1 -- 172.21.15.27:0/2148511859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9700a52b0 con 0x7fa970009ac0 2024-04-25T13:56:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.910+0000 7fa976ffd640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa970009ac0 0x7fa9700a37a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa96800e9f0 tx=0x7fa96800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.911+0000 7fa95ffff640 1 -- 172.21.15.27:0/2148511859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96800ccc0 con 0x7fa970009ac0 2024-04-25T13:56:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.911+0000 7fa95ffff640 1 -- 172.21.15.27:0/2148511859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9680108e0 con 0x7fa970009ac0 2024-04-25T13:56:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.911+0000 7fa95ffff640 1 -- 172.21.15.27:0/2148511859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968018880 con 0x7fa970009ac0 2024-04-25T13:56:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.911+0000 7fa977fff640 1 -- 172.21.15.27:0/2148511859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9700a55a0 con 0x7fa970009ac0 2024-04-25T13:56:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.911+0000 7fa977fff640 1 -- 172.21.15.27:0/2148511859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9700a5aa0 con 0x7fa970009ac0 2024-04-25T13:56:17.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.916+0000 7fa984671740 1 Processor -- start 2024-04-25T13:56:17.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.918+0000 7fa984671740 1 -- start start 2024-04-25T13:56:17.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa984671740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 0x55c8903796a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa984671740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c890379c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa984671740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ffb7730 0x55c890380b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa984671740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c8902deb00 con 0x55c88ffb7730 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa984671740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c89027a920 con 0x55c88ffd8410 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa984671740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55c8902daac0 con 0x55c88ffaf550 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa95ffff640 1 -- 172.21.15.27:0/2148511859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa968010430 con 0x7fa970009ac0 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa976ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c890379c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa976ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c890379c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47962/0 (socket says 172.21.15.27:47962) 2024-04-25T13:56:17.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa976ffd640 1 -- 172.21.15.27:0/1372893361 learned_addr learned my addr 172.21.15.27:0/1372893361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa976ffd640 1 -- 172.21.15.27:0/1372893361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 msgr2=0x55c8903796a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:56:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa976ffd640 1 --2- 172.21.15.27:0/1372893361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 0x55c8903796a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa976ffd640 1 -- 172.21.15.27:0/1372893361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ffb7730 msgr2=0x55c890380b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:56:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa976ffd640 1 --2- 172.21.15.27:0/1372893361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ffb7730 0x55c890380b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.919+0000 7fa976ffd640 1 -- 172.21.15.27:0/1372893361 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8900b87e0 con 0x55c88ffaf550 2024-04-25T13:56:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.920+0000 7fa976ffd640 1 --2- 172.21.15.27:0/1372893361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c890379c20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa9680e2ef0 tx=0x7fa9680e7810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78ca2d97dfc7ba15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:17.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.921+0000 7fa95ffff640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa9600afcc0 0x7fa9600b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.921+0000 7fa9777fe640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa9600afcc0 0x7fa9600b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.922+0000 7fa95d7fa640 1 -- 172.21.15.27:0/1372893361 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680f4070 con 0x55c88ffaf550 2024-04-25T13:56:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.922+0000 7fa95d7fa640 1 -- 172.21.15.27:0/1372893361 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9680e5d40 con 0x55c88ffaf550 2024-04-25T13:56:17.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.922+0000 7fa95d7fa640 1 -- 172.21.15.27:0/1372893361 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680efd10 con 0x55c88ffaf550 2024-04-25T13:56:17.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.923+0000 7fa984671740 1 -- 172.21.15.27:0/1372893361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 msgr2=0x55c890379c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:17.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.923+0000 7fa984671740 1 --2- 172.21.15.27:0/1372893361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c890379c20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa9680e2ef0 tx=0x7fa9680e7810 comp rx=0 tx=0).stop 2024-04-25T13:56:17.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.923+0000 7fa9777fe640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa9600afcc0 0x7fa9600b21d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa97802e0e0 tx=0x7fa978064120 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:17.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.924+0000 7fa95ffff640 1 -- 172.21.15.27:0/2148511859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa9680dee30 con 0x7fa970009ac0 2024-04-25T13:56:17.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.925+0000 7fa984671740 1 -- 172.21.15.27:0/1372893361 shutdown_connections 2024-04-25T13:56:17.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.925+0000 7fa984671740 1 --2- 172.21.15.27:0/1372893361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ffb7730 0x55c890380b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.925+0000 7fa984671740 1 --2- 172.21.15.27:0/1372893361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c890379c20 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.925+0000 7fa984671740 1 --2- 172.21.15.27:0/1372893361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 0x55c8903796a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.925+0000 7fa984671740 1 -- 172.21.15.27:0/1372893361 >> 172.21.15.27:0/1372893361 conn(0x55c88ffcc560 msgr2=0x55c890375030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:17.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.929+0000 7fa984671740 1 -- 172.21.15.27:0/1372893361 shutdown_connections 2024-04-25T13:56:17.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.929+0000 7fa984671740 1 -- 172.21.15.27:0/1372893361 wait complete. 2024-04-25T13:56:17.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.930+0000 7fa984671740 1 -- start start 2024-04-25T13:56:17.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.930+0000 7fa984671740 1 Processor -- start 2024-04-25T13:56:17.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.930+0000 7fa984671740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ffaf550 0x55c890383520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa984671740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890383a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa984671740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 0x55c89038a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa984671740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55c890381570 con 0x55c88ffaf550 2024-04-25T13:56:17.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa984671740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55c8902785d0 con 0x55c88ffd8410 2024-04-25T13:56:17.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9767fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890383a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9767fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890383a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47968/0 (socket says 172.21.15.27:47968) 2024-04-25T13:56:17.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9767fc640 1 -- 172.21.15.27:0/3483694679 learned_addr learned my addr 172.21.15.27:0/3483694679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:17.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9777fe640 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 0x55c89038a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55c8903816f0 con 0x55c88ffb7730 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9767fc640 1 -- 172.21.15.27:0/3483694679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 msgr2=0x55c89038a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9767fc640 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 0x55c89038a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9767fc640 1 -- 172.21.15.27:0/3483694679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ffaf550 msgr2=0x55c890383520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9767fc640 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ffaf550 0x55c890383520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa9767fc640 1 -- 172.21.15.27:0/3483694679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c890118f70 con 0x55c88ffb7730 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.931+0000 7fa984671740 10 client.0 _ll_register_callbacks cb 0x55c890266080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.932+0000 7fa9767fc640 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890383a60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa96c02a970 tx=0x7fa96c030fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=759e1fe56a0969bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.932+0000 7fa95aff5640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.932+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9680e5d40 con 0x55c88ffb7730 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.933+0000 7fa95aff5640 1 -- 172.21.15.27:0/3483694679 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c037040 con 0x55c88ffb7730 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.933+0000 7fa95aff5640 1 -- 172.21.15.27:0/3483694679 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa96c004230 con 0x55c88ffb7730 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.933+0000 7fa95aff5640 1 -- 172.21.15.27:0/3483694679 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa96c004430 con 0x55c88ffb7730 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.933+0000 7fa984671740 10 client.28171 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:17.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.933+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({fsmap=19}) -- 0x55c88ffb0170 con 0x55c88ffb7730 2024-04-25T13:56:17.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.934+0000 7fa95aff5640 1 -- 172.21.15.27:0/3483694679 <== mon.1 v2:172.21.15.122:3300/0 4 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa96c005040 con 0x55c88ffb7730 2024-04-25T13:56:17.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.934+0000 7fa984671740 10 client.28171 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:17.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.934+0000 7fa984671740 10 client.28171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:17.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.934+0000 7fa984671740 20 client.28171 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:17.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.934+0000 7fa984671740 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c88ff971f0 0x55c890390d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.934+0000 7fa984671740 4 client.28171 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:56:17.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.934+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa9680e5d40 con 0x55c88ff971f0 2024-04-25T13:56:17.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.934+0000 7fa976ffd640 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c88ff971f0 0x55c890390d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.938+0000 7fa976ffd640 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c88ff971f0 0x55c890390d10 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ebaec4f88f7ad7d server_cookie=f5d7cf7fff94494f in_seq=0 out_seq=0 2024-04-25T13:56:17.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.938+0000 7fa95aff5640 10 client.28171 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:17.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.940+0000 7fa95aff5640 1 -- 172.21.15.27:0/3483694679 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fa9680e5d40 con 0x55c88ff971f0 2024-04-25T13:56:17.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.940+0000 7fa95aff5640 10 client.28171 handle_command_reply: tid=0 2024-04-25T13:56:17.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.942+0000 7fa984671740 1 client.28171 shutdown 2024-04-25T13:56:17.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.943+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 msgr2=0x55c890383a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:17.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.943+0000 7fa984671740 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890383a60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa96c02a970 tx=0x7fa96c030fb0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.944+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 shutdown_connections 2024-04-25T13:56:17.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.944+0000 7fa984671740 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c88ff971f0 0x55c890390d10 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.944+0000 7fa95aff5640 0 client.28171 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:17.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.944+0000 7fa984671740 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffd8410 0x55c89038a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.944+0000 7fa984671740 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890383a60 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.944+0000 7fa984671740 1 --2- 172.21.15.27:0/3483694679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ffaf550 0x55c890383520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.945+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 >> 172.21.15.27:0/3483694679 conn(0x55c88ffcc560 msgr2=0x55c890374050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.946+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 shutdown_connections 2024-04-25T13:56:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.946+0000 7fa984671740 1 -- 172.21.15.27:0/3483694679 wait complete. 2024-04-25T13:56:17.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.947+0000 7fa984671740 20 client.28171 trim_cache size 0 max 16384 2024-04-25T13:56:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.987+0000 7fa984671740 1 Processor -- start 2024-04-25T13:56:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.987+0000 7fa984671740 1 -- start start 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.987+0000 7fa984671740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c8903749a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.987+0000 7fa984671740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffaf550 0x55c89038a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.987+0000 7fa984671740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890374ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.987+0000 7fa984671740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa97015bf40 con 0x55c88ff971f0 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.987+0000 7fa984671740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa97015e7c0 con 0x55c88ffaf550 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.987+0000 7fa984671740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa97015c5b0 con 0x55c88ffb7730 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890374ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa9777fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890374ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47970/0 (socket says 172.21.15.27:47970) 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa9777fe640 1 -- 172.21.15.27:0/44749314 learned_addr learned my addr 172.21.15.27:0/44749314 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa976ffd640 1 --2- 172.21.15.27:0/44749314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c8903749a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa9777fe640 1 -- 172.21.15.27:0/44749314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffaf550 msgr2=0x55c89038a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T13:56:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa9777fe640 1 --2- 172.21.15.27:0/44749314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffaf550 0x55c89038a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa9777fe640 1 -- 172.21.15.27:0/44749314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 msgr2=0x55c8903749a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa9777fe640 1 --2- 172.21.15.27:0/44749314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c8903749a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.988+0000 7fa9777fe640 1 -- 172.21.15.27:0/44749314 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c890380cf0 con 0x55c88ffb7730 2024-04-25T13:56:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.989+0000 7fa9777fe640 1 --2- 172.21.15.27:0/44749314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890374ee0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa97802e110 tx=0x7fa978074000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6509168edb2d2092 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.989+0000 7fa95cff9640 1 -- 172.21.15.27:0/44749314 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa978070e60 con 0x55c88ffb7730 2024-04-25T13:56:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.991+0000 7fa95cff9640 1 -- 172.21.15.27:0/44749314 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa97807e040 con 0x55c88ffb7730 2024-04-25T13:56:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.992+0000 7fa95cff9640 1 -- 172.21.15.27:0/44749314 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9780853f0 con 0x55c88ffb7730 2024-04-25T13:56:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.995+0000 7fa984671740 1 -- 172.21.15.27:0/44749314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 msgr2=0x55c890374ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.995+0000 7fa984671740 1 --2- 172.21.15.27:0/44749314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890374ee0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa97802e110 tx=0x7fa978074000 comp rx=0 tx=0).stop 2024-04-25T13:56:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.995+0000 7fa984671740 1 -- 172.21.15.27:0/44749314 shutdown_connections 2024-04-25T13:56:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.995+0000 7fa984671740 1 --2- 172.21.15.27:0/44749314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffb7730 0x55c890374ee0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.995+0000 7fa984671740 1 --2- 172.21.15.27:0/44749314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffaf550 0x55c89038a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.995+0000 7fa984671740 1 --2- 172.21.15.27:0/44749314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c8903749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.995+0000 7fa984671740 1 -- 172.21.15.27:0/44749314 >> 172.21.15.27:0/44749314 conn(0x55c88ffcc560 msgr2=0x55c89037d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.996+0000 7fa984671740 1 -- 172.21.15.27:0/44749314 shutdown_connections 2024-04-25T13:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.997+0000 7fa984671740 1 -- 172.21.15.27:0/44749314 wait complete. 2024-04-25T13:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.997+0000 7fa984671740 1 -- start start 2024-04-25T13:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.997+0000 7fa984671740 1 Processor -- start 2024-04-25T13:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa984671740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c890387db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa984671740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c8903882f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa984671740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffb7730 0x55c8903890f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa984671740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa97015cc80 con 0x55c88ff971f0 2024-04-25T13:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa984671740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa97015e0f0 con 0x55c88ffb7730 2024-04-25T13:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa984671740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa97015d350 con 0x55c88ffaf550 2024-04-25T13:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa984671740 10 client.0 _ll_register_callbacks cb 0x7fa96c0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa976ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c890387db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa976ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c890387db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60968/0 (socket says 172.21.15.27:60968) 2024-04-25T13:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.998+0000 7fa976ffd640 1 -- 172.21.15.27:0/96529614 learned_addr learned my addr 172.21.15.27:0/96529614 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.999+0000 7fa9777fe640 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffb7730 0x55c8903890f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.999+0000 7fa976ffd640 1 -- 172.21.15.27:0/96529614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 msgr2=0x55c8903882f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.999+0000 7fa976ffd640 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c8903882f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.999+0000 7fa976ffd640 1 -- 172.21.15.27:0/96529614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffb7730 msgr2=0x55c8903890f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.999+0000 7fa976ffd640 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffb7730 0x55c8903890f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:17.999+0000 7fa976ffd640 1 -- 172.21.15.27:0/96529614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8903896b0 con 0x55c88ff971f0 2024-04-25T13:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.000+0000 7fa976ffd640 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c890387db0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa9680a3360 tx=0x7fa9680096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4458fa0b49eb43d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.000+0000 7fa95aff5640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.000+0000 7fa95aff5640 1 -- 172.21.15.27:0/96529614 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680a31a0 con 0x55c88ff971f0 2024-04-25T13:56:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.000+0000 7fa95aff5640 1 -- 172.21.15.27:0/96529614 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa968010a90 con 0x55c88ff971f0 2024-04-25T13:56:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.001+0000 7fa984671740 1 -- 172.21.15.27:0/96529614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa97807e040 con 0x55c88ff971f0 2024-04-25T13:56:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.001+0000 7fa95aff5640 1 -- 172.21.15.27:0/96529614 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa9680a6a50 con 0x55c88ff971f0 2024-04-25T13:56:18.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.001+0000 7fa984671740 10 client.18438 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:18.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.001+0000 7fa984671740 1 -- 172.21.15.27:0/96529614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55c89038a9d0 con 0x55c88ff971f0 2024-04-25T13:56:18.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.002+0000 7fa95aff5640 1 -- 172.21.15.27:0/96529614 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa97807e040 con 0x55c88ff971f0 2024-04-25T13:56:18.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.002+0000 7fa95aff5640 1 -- 172.21.15.27:0/96529614 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa9680a3570 con 0x55c88ff971f0 2024-04-25T13:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.002+0000 7fa984671740 10 client.18438 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.002+0000 7fa984671740 10 client.18438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.002+0000 7fa984671740 20 client.18438 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.002+0000 7fa984671740 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c88ffd8410 0x55c8903fb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.002+0000 7fa984671740 4 client.18438 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T13:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.002+0000 7fa984671740 1 -- 172.21.15.27:0/96529614 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c8902ec710 con 0x55c88ffd8410 2024-04-25T13:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.003+0000 7fa9767fc640 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c88ffd8410 0x55c8903fb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.004+0000 7fa9767fc640 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c88ffd8410 0x55c8903fb1a0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18304a7b341314aa server_cookie=58dbc542257b3a1 in_seq=0 out_seq=0 2024-04-25T13:56:18.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.004+0000 7fa95aff5640 10 client.18438 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:18.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.005+0000 7fa95aff5640 1 -- 172.21.15.27:0/96529614 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c8902ec710 con 0x55c88ffd8410 2024-04-25T13:56:18.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.005+0000 7fa95aff5640 10 client.18438 handle_command_reply: tid=0 2024-04-25T13:56:18.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.005+0000 7fa984671740 1 client.18438 shutdown 2024-04-25T13:56:18.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.006+0000 7fa984671740 1 -- 172.21.15.27:0/96529614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 msgr2=0x55c890387db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.006+0000 7fa984671740 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c890387db0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa9680a3360 tx=0x7fa9680096a0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa984671740 1 -- 172.21.15.27:0/96529614 shutdown_connections 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa984671740 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55c88ffd8410 0x55c8903fb1a0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa984671740 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c88ffb7730 0x55c8903890f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa984671740 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c88ffaf550 0x55c8903882f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa984671740 1 --2- 172.21.15.27:0/96529614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55c88ff971f0 0x55c890387db0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa984671740 1 -- 172.21.15.27:0/96529614 >> 172.21.15.27:0/96529614 conn(0x55c88ffcc560 msgr2=0x55c890383420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa95aff5640 0 client.18438 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa984671740 1 -- 172.21.15.27:0/96529614 shutdown_connections 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.007+0000 7fa984671740 1 -- 172.21.15.27:0/96529614 wait complete. 2024-04-25T13:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.008+0000 7fa984671740 20 client.18438 trim_cache size 0 max 16384 2024-04-25T13:56:18.008 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:56:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "return_code": 0, 2024-04-25T13:56:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "scrub_tag": "a625dc08-2888-457f-9a38-74fe9858248a", 2024-04-25T13:56:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "asynchronous" 2024-04-25T13:56:18.008 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:56:18.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.011+0000 7fa977fff640 1 -- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa9600afcc0 msgr2=0x7fa9600b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.011+0000 7fa977fff640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa9600afcc0 0x7fa9600b21d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa97802e0e0 tx=0x7fa978064120 comp rx=0 tx=0).stop 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.012+0000 7fa977fff640 1 -- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa970009ac0 msgr2=0x7fa9700a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.012+0000 7fa977fff640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa970009ac0 0x7fa9700a37a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa96800e9f0 tx=0x7fa96800eeb0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.012+0000 7fa977fff640 1 -- 172.21.15.27:0/2148511859 shutdown_connections 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.012+0000 7fa977fff640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa9600afcc0 0x7fa9600b21d0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.012+0000 7fa977fff640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa97000ba50 0x7fa9700a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.012+0000 7fa977fff640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa97000ad50 0x7fa9700a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.012+0000 7fa977fff640 1 --2- 172.21.15.27:0/2148511859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa970009ac0 0x7fa9700a37a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.012+0000 7fa977fff640 1 -- 172.21.15.27:0/2148511859 >> 172.21.15.27:0/2148511859 conn(0x7fa970093dc0 msgr2=0x7fa970094410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:18.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.013+0000 7fa977fff640 1 -- 172.21.15.27:0/2148511859 shutdown_connections 2024-04-25T13:56:18.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.013+0000 7fa977fff640 1 -- 172.21.15.27:0/2148511859 wait complete. 2024-04-25T13:56:18.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:56:18.030 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:56:18.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 -- 172.21.15.27:0/556320203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc106c20 msgr2=0x7fa5cc109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/556320203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc106c20 0x7fa5cc109100 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009900 tx=0x7fa5b802a410 comp rx=0 tx=0).stop 2024-04-25T13:56:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 -- 172.21.15.27:0/556320203 shutdown_connections 2024-04-25T13:56:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/556320203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5cc109790 0x7fa5cc10bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/556320203 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc106c20 0x7fa5cc109100 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/556320203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5cc101e90 0x7fa5cc1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 -- 172.21.15.27:0/556320203 >> 172.21.15.27:0/556320203 conn(0x7fa5cc0fd340 msgr2=0x7fa5cc0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 -- 172.21.15.27:0/556320203 shutdown_connections 2024-04-25T13:56:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.193+0000 7fa5d2e97640 1 -- 172.21.15.27:0/556320203 wait complete. 2024-04-25T13:56:18.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d2e97640 1 Processor -- start 2024-04-25T13:56:18.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d2e97640 1 -- start start 2024-04-25T13:56:18.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d2e97640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc101e90 0x7fa5cc19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d2e97640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5cc106c20 0x7fa5cc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d2e97640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5cc109790 0x7fa5cc1a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d2e97640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc10f650 con 0x7fa5cc106c20 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d2e97640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc10f4d0 con 0x7fa5cc109790 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d2e97640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc10f7d0 con 0x7fa5cc101e90 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d0c33640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc101e90 0x7fa5cc19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d0c33640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc101e90 0x7fa5cc19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:47988/0 (socket says 172.21.15.27:47988) 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d0c33640 1 -- 172.21.15.27:0/3694124247 learned_addr learned my addr 172.21.15.27:0/3694124247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5cc106c20 0x7fa5cc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.194+0000 7fa5d1434640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5cc109790 0x7fa5cc1a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5d0c33640 1 -- 172.21.15.27:0/3694124247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5cc109790 msgr2=0x7fa5cc1a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5cc109790 0x7fa5cc1a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5d0c33640 1 -- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5cc106c20 msgr2=0x7fa5cc19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5cc106c20 0x7fa5cc19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5d0c33640 1 -- 172.21.15.27:0/3694124247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc1a1d30 con 0x7fa5cc101e90 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5d1434640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5cc109790 0x7fa5cc1a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc101e90 0x7fa5cc19a1f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000e980 tx=0x7fa5c000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:18.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5c9ffb640 1 -- 172.21.15.27:0/3694124247 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c000cd80 con 0x7fa5cc101e90 2024-04-25T13:56:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5c9ffb640 1 -- 172.21.15.27:0/3694124247 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c00108e0 con 0x7fa5cc101e90 2024-04-25T13:56:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.195+0000 7fa5c9ffb640 1 -- 172.21.15.27:0/3694124247 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0018990 con 0x7fa5cc101e90 2024-04-25T13:56:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.196+0000 7fa5d2e97640 1 -- 172.21.15.27:0/3694124247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5cc1a1fc0 con 0x7fa5cc101e90 2024-04-25T13:56:18.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.196+0000 7fa5d2e97640 1 -- 172.21.15.27:0/3694124247 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5cc1a23d0 con 0x7fa5cc101e90 2024-04-25T13:56:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.199+0000 7fa5c9ffb640 1 -- 172.21.15.27:0/3694124247 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa5c001f020 con 0x7fa5cc101e90 2024-04-25T13:56:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d9e22740 1 Processor -- start 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d9e22740 1 -- start start 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5c9ffb640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5bc0afb70 0x7fa5bc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa308e390 0x55dfa342fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa30654d0 0x55dfa342ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3430c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dfa3394d80 con 0x55dfa306d6b0 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55dfa334e8b0 con 0x55dfa30654d0 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55dfa3390d40 con 0x55dfa308e390 2024-04-25T13:56:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5c9ffb640 1 -- 172.21.15.27:0/3694124247 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5c00ded80 con 0x7fa5cc101e90 2024-04-25T13:56:18.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.200+0000 7fa5d1434640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5bc0afb70 0x7fa5bc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3430c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3430c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60988/0 (socket says 172.21.15.27:60988) 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 -- 172.21.15.27:0/2707637473 learned_addr learned my addr 172.21.15.27:0/2707637473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa308e390 0x55dfa342fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa30654d0 0x55dfa342ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 -- 172.21.15.27:0/2707637473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa308e390 msgr2=0x55dfa342fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa308e390 0x55dfa342fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 -- 172.21.15.27:0/2707637473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa30654d0 msgr2=0x55dfa342ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa30654d0 0x55dfa342ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 -- 172.21.15.27:0/2707637473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa331fa40 con 0x55dfa306d6b0 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5bc0afb70 0x7fa5bc0b2080 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4005c40 tx=0x7fa5c4009e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5d1434640 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3430c70 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4010bd0 tx=0x7fa5c401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3985e041a2755762 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.201+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/2707637473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400cb40 con 0x55dfa306d6b0 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.202+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/2707637473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c400cce0 con 0x55dfa306d6b0 2024-04-25T13:56:18.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.202+0000 7fa5b6ffd640 1 -- 172.21.15.27:0/2707637473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4005840 con 0x55dfa306d6b0 2024-04-25T13:56:18.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.202+0000 7fa5d9e22740 1 -- 172.21.15.27:0/2707637473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 msgr2=0x55dfa3430c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.202+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3430c70 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4010bd0 tx=0x7fa5c401c000 comp rx=0 tx=0).stop 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.202+0000 7fa5d9e22740 1 -- 172.21.15.27:0/2707637473 shutdown_connections 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.202+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3430c70 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.202+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa30654d0 0x55dfa342ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.202+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/2707637473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa308e390 0x55dfa342fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.203+0000 7fa5d9e22740 1 -- 172.21.15.27:0/2707637473 >> 172.21.15.27:0/2707637473 conn(0x55dfa30824e0 msgr2=0x55dfa342b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.203+0000 7fa5d9e22740 1 -- 172.21.15.27:0/2707637473 shutdown_connections 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.203+0000 7fa5d9e22740 1 -- 172.21.15.27:0/2707637473 wait complete. 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.203+0000 7fa5d9e22740 1 -- start start 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.203+0000 7fa5d9e22740 1 Processor -- start 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa343dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa343e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa308e390 0x55dfa34451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55dfa3431720 con 0x55dfa306d6b0 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55dfa331bd10 con 0x55dfa308e390 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55dfa34318a0 con 0x55dfa30654d0 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d0c33640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa343dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5cbfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa343e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d0c33640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa343dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:48006/0 (socket says 172.21.15.27:48006) 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d0c33640 1 -- 172.21.15.27:0/3950147461 learned_addr learned my addr 172.21.15.27:0/3950147461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d9e22740 10 client.0 _ll_register_callbacks cb 0x55dfa3348210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.204+0000 7fa5d1434640 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa308e390 0x55dfa34451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5cbfff640 1 -- 172.21.15.27:0/3950147461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 msgr2=0x55dfa343dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa343dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5cbfff640 1 -- 172.21.15.27:0/3950147461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa308e390 msgr2=0x55dfa34451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa308e390 0x55dfa34451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5cbfff640 1 -- 172.21.15.27:0/3950147461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa31707d0 con 0x55dfa306d6b0 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa343e280 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa5b802aa00 tx=0x7fa5b80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887cc45ccc8e6fc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5abfff640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5abfff640 1 -- 172.21.15.27:0/3950147461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8035070 con 0x55dfa306d6b0 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5abfff640 1 -- 172.21.15.27:0/3950147461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5b8003a00 con 0x55dfa306d6b0 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.205+0000 7fa5d9e22740 1 -- 172.21.15.27:0/3950147461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5c400cce0 con 0x55dfa306d6b0 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.206+0000 7fa5abfff640 1 -- 172.21.15.27:0/3950147461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa5b804c050 con 0x55dfa306d6b0 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.206+0000 7fa5d9e22740 10 client.18450 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.206+0000 7fa5d9e22740 1 -- 172.21.15.27:0/3950147461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55dfa30a63f0 con 0x55dfa306d6b0 2024-04-25T13:56:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.206+0000 7fa5abfff640 1 -- 172.21.15.27:0/3950147461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c400cce0 con 0x55dfa306d6b0 2024-04-25T13:56:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.206+0000 7fa5abfff640 1 -- 172.21.15.27:0/3950147461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa5b8043070 con 0x55dfa306d6b0 2024-04-25T13:56:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.206+0000 7fa5d9e22740 10 client.18450 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.206+0000 7fa5d9e22740 10 client.18450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.206+0000 7fa5d9e22740 20 client.18450 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.207+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dfa304cf60 0x55dfa344baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.207+0000 7fa5d9e22740 4 client.18450 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.207+0000 7fa5d9e22740 1 -- 172.21.15.27:0/3950147461 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfa33a2990 con 0x55dfa304cf60 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.207+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dfa304cf60 0x55dfa344baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.207+0000 7fa5abfff640 10 client.18450 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.207+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dfa304cf60 0x55dfa344baf0 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adb15eae78bb694c server_cookie=44472f00a9a22447 in_seq=0 out_seq=0 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5abfff640 1 -- 172.21.15.27:0/3950147461 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dfa33a2990 con 0x55dfa304cf60 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5abfff640 10 client.18450 handle_command_reply: tid=0 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 client.18450 shutdown 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 -- 172.21.15.27:0/3950147461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 msgr2=0x55dfa343e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa343e280 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa5b802aa00 tx=0x7fa5b80043b0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 -- 172.21.15.27:0/3950147461 shutdown_connections 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dfa304cf60 0x55dfa344baf0 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa308e390 0x55dfa34451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa343e280 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5abfff640 0 client.18450 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/3950147461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa343dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.209+0000 7fa5d9e22740 1 -- 172.21.15.27:0/3950147461 >> 172.21.15.27:0/3950147461 conn(0x55dfa30824e0 msgr2=0x55dfa342a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.210+0000 7fa5d9e22740 1 -- 172.21.15.27:0/3950147461 shutdown_connections 2024-04-25T13:56:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.210+0000 7fa5d9e22740 1 -- 172.21.15.27:0/3950147461 wait complete. 2024-04-25T13:56:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.210+0000 7fa5d9e22740 20 client.18450 trim_cache size 0 max 16384 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d9e22740 1 Processor -- start 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d9e22740 1 -- start start 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa304cf60 0x55dfa344ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3434640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa306d6b0 0x55dfa3434b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1a7000 con 0x55dfa304cf60 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1a9880 con 0x55dfa306d6b0 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1a7670 con 0x55dfa30654d0 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d0c33640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa304cf60 0x55dfa344ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d0c33640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa304cf60 0x55dfa344ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32774/0 (socket says 172.21.15.27:32774) 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d0c33640 1 -- 172.21.15.27:0/1449849773 learned_addr learned my addr 172.21.15.27:0/1449849773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.231+0000 7fa5d1434640 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa306d6b0 0x55dfa3434b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3434640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5d0c33640 1 -- 172.21.15.27:0/1449849773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 msgr2=0x55dfa3434640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3434640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5d0c33640 1 -- 172.21.15.27:0/1449849773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa306d6b0 msgr2=0x55dfa3434b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa306d6b0 0x55dfa3434b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5d0c33640 1 -- 172.21.15.27:0/1449849773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa34300c0 con 0x55dfa304cf60 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3434640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa304cf60 0x55dfa344ba50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0000c00 tx=0x7fa5c001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16d1657fbfa3036 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5b67fc640 1 -- 172.21.15.27:0/1449849773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c00c0070 con 0x55dfa304cf60 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5b67fc640 1 -- 172.21.15.27:0/1449849773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c0010d40 con 0x55dfa304cf60 2024-04-25T13:56:18.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.232+0000 7fa5b67fc640 1 -- 172.21.15.27:0/1449849773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c00afc60 con 0x55dfa304cf60 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1449849773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa304cf60 msgr2=0x55dfa344ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa304cf60 0x55dfa344ba50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0000c00 tx=0x7fa5c001a040 comp rx=0 tx=0).stop 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1449849773 shutdown_connections 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa306d6b0 0x55dfa3434b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3434640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1449849773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa304cf60 0x55dfa344ba50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1449849773 >> 172.21.15.27:0/1449849773 conn(0x55dfa30824e0 msgr2=0x55dfa343dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1449849773 shutdown_connections 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.233+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1449849773 wait complete. 2024-04-25T13:56:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 1 -- start start 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 1 Processor -- start 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa304cf60 0x55dfa3441aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3441fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3442f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1a7d40 con 0x55dfa306d6b0 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1a91b0 con 0x55dfa304cf60 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc1a8410 con 0x55dfa30654d0 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d9e22740 10 client.0 _ll_register_callbacks cb 0x7fa5b8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5cbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3441fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.234+0000 7fa5d0c33640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa304cf60 0x55dfa3441aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5cbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3441fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:48034/0 (socket says 172.21.15.27:48034) 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5d0c33640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa304cf60 0x55dfa3441aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.27:39212/0 (socket says 172.21.15.27:39212) 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5cbfff640 1 -- 172.21.15.27:0/1133830465 learned_addr learned my addr 172.21.15.27:0/1133830465 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5d1434640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3442f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5cbfff640 1 -- 172.21.15.27:0/1133830465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa304cf60 msgr2=0x55dfa3441aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa304cf60 0x55dfa3441aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5cbfff640 1 -- 172.21.15.27:0/1133830465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 msgr2=0x55dfa3442f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3442f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5cbfff640 1 -- 172.21.15.27:0/1133830465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa30660f0 con 0x55dfa30654d0 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa304cf60 0x55dfa3441aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5d1434640 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3442f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5cbfff640 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3441fe0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8005f70 tx=0x7fa5b8045770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adab73de6fdd1926 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5abfff640 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.235+0000 7fa5abfff640 1 -- 172.21.15.27:0/1133830465 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8035040 con 0x55dfa30654d0 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5abfff640 1 -- 172.21.15.27:0/1133830465 <== mon.1 v2:172.21.15.122:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa5b8045d50 con 0x55dfa30654d0 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1133830465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5c0010d40 con 0x55dfa30654d0 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5abfff640 1 -- 172.21.15.27:0/1133830465 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa5b80315a0 con 0x55dfa30654d0 2024-04-25T13:56:18.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 10 client.28192 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1133830465 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55dfa33554b0 con 0x55dfa30654d0 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5abfff640 1 -- 172.21.15.27:0/1133830465 <== mon.1 v2:172.21.15.122:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c0010d40 con 0x55dfa30654d0 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5abfff640 1 -- 172.21.15.27:0/1133830465 <== mon.1 v2:172.21.15.122:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7fa5b8043040 con 0x55dfa30654d0 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 10 client.28192 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 10 client.28192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 20 client.28192 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dfa308e390 0x55dfa34b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 4 client.28192 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.236+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1133830465 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dfa339c800 con 0x55dfa308e390 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.237+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dfa308e390 0x55dfa34b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.237+0000 7fa5d0c33640 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dfa308e390 0x55dfa34b5fa0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e7900fd3a93357 server_cookie=b24d645424b8c05d in_seq=0 out_seq=0 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.237+0000 7fa5abfff640 10 client.28192 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.237+0000 7fa5abfff640 1 -- 172.21.15.27:0/1133830465 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55dfa339c800 con 0x55dfa308e390 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.237+0000 7fa5abfff640 10 client.28192 handle_command_reply: tid=0 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.237+0000 7fa5d9e22740 1 client.28192 shutdown 2024-04-25T13:56:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.237+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1133830465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 msgr2=0x55dfa3441fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.237+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3441fe0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8005f70 tx=0x7fa5b8045770 comp rx=0 tx=0).stop 2024-04-25T13:56:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1133830465 shutdown_connections 2024-04-25T13:56:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55dfa308e390 0x55dfa34b5fa0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55dfa306d6b0 0x55dfa3442f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5abfff640 0 client.28192 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dfa30654d0 0x55dfa3441fe0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 1 --2- 172.21.15.27:0/1133830465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dfa304cf60 0x55dfa3441aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1133830465 >> 172.21.15.27:0/1133830465 conn(0x55dfa30824e0 msgr2=0x55dfa343c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1133830465 shutdown_connections 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 1 -- 172.21.15.27:0/1133830465 wait complete. 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.238+0000 7fa5d9e22740 20 client.28192 trim_cache size 0 max 16384 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:56:18.242 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "scrub active (904 inodes in the stack)", 2024-04-25T13:56:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": { 2024-04-25T13:56:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "a625dc08-2888-457f-9a38-74fe9858248a": { 2024-04-25T13:56:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "path": "/", 2024-04-25T13:56:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "tag": "a625dc08-2888-457f-9a38-74fe9858248a", 2024-04-25T13:56:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: "options": "recursive,force" 2024-04-25T13:56:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:18.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:18.243 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 -- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5bc0afb70 msgr2=0x7fa5bc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5bc0afb70 0x7fa5bc0b2080 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4005c40 tx=0x7fa5c4009e40 comp rx=0 tx=0).stop 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 -- 172.21.15.27:0/3694124247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc101e90 msgr2=0x7fa5cc19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc101e90 0x7fa5cc19a1f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000e980 tx=0x7fa5c000ee40 comp rx=0 tx=0).stop 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 -- 172.21.15.27:0/3694124247 shutdown_connections 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa5bc0afb70 0x7fa5bc0b2080 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5cc109790 0x7fa5cc1a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5cc106c20 0x7fa5cc19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 --2- 172.21.15.27:0/3694124247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa5cc101e90 0x7fa5cc19a1f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 -- 172.21.15.27:0/3694124247 >> 172.21.15.27:0/3694124247 conn(0x7fa5cc0fd340 msgr2=0x7fa5cc107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 -- 172.21.15.27:0/3694124247 shutdown_connections 2024-04-25T13:56:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:18.239+0000 7fa5d2e97640 1 -- 172.21.15.27:0/3694124247 wait complete. 2024-04-25T13:56:18.250 INFO:tasks.cephfs.filesystem:scrub status for tag:a625dc08-2888-457f-9a38-74fe9858248a - {'path': '/', 'tag': 'a625dc08-2888-457f-9a38-74fe9858248a', 'options': 'recursive,force'} 2024-04-25T13:56:18.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:18 smithi122 ceph-mon[32843]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 841 op/s 2024-04-25T13:56:18.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:18 smithi045 ceph-mon[32731]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 841 op/s 2024-04-25T13:56:18.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:18 smithi027 ceph-mon[29140]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 841 op/s 2024-04-25T13:56:18.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T13:56:18.795 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/: 2024-04-25T13:56:18.795 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.795 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 .. 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 data 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data: 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:total 36 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 . 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 .. 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 13:41 base 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Apr 25 13:41 current_logfiles 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 25 13:41 global 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:41 log 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_commit_ts 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_dynshmem 2024-04-25T13:56:18.796 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 25 13:40 pg_hba.conf 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 25 13:40 pg_ident.conf 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:56 pg_logical 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 pg_multixact 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_notify 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_replslot 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_serial 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_snapshots 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:56 pg_stat 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:56 pg_stat_tmp 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:56 pg_subtrans 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_tblspc 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 pg_twophase 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 PG_VERSION 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 13:56 pg_wal 2024-04-25T13:56:18.797 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:56 pg_xact 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 25 13:40 postgresql.auto.conf 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 25 13:40 postgresql.conf 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Apr 25 13:41 postmaster.opts 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base: 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 13:41 . 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:40 1 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:40 13493 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 25 13:44 13494 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 pgsql_tmp 2024-04-25T13:56:18.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:total 7777 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:40 . 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 13:41 .. 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 112 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 113 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:40 1247 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1247_fsm 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1247_vm 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 13:40 1249 2024-04-25T13:56:18.799 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1249_fsm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1249_vm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 13:40 1255 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1255_fsm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1255_vm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 13:40 1259 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1259_fsm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1259_vm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 13:40 13344 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13344_fsm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13344_vm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13346 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13348 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13349_fsm 2024-04-25T13:56:18.800 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349_vm 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13351 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13353 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13354 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13354_fsm 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13354_vm 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13356 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13358 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13359_fsm 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359_vm 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13361 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13363 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1417 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1418 2024-04-25T13:56:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 174 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 175 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2187 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2224 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2328 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2336 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2337 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2579 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2600 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2600_fsm 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2600_vm 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2601_fsm 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601_vm 2024-04-25T13:56:18.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2602 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2602_fsm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2602_vm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2603 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2603_fsm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2603_vm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2604 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2605 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2605_fsm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2605_vm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2606 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2606_fsm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2606_vm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2607 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2607_fsm 2024-04-25T13:56:18.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2607_vm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 13:40 2608 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2608_fsm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2608_vm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 13:40 2609 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2609_fsm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2609_vm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2610 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2610_fsm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2610_vm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2611 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2612_fsm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612_vm 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2613 2024-04-25T13:56:18.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2615_fsm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615_vm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2616 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2616_fsm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2616_vm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 13:40 2617 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2617_fsm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2617_vm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 13:40 2618 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2618_fsm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2618_vm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 13:40 2619 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2619_fsm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2619_vm 2024-04-25T13:56:18.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2620 2024-04-25T13:56:18.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2650 2024-04-25T13:56:18.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2651 2024-04-25T13:56:18.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2652 2024-04-25T13:56:18.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2653 2024-04-25T13:56:18.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2654 2024-04-25T13:56:18.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2655 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2656 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2657 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 13:40 2658 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:40 2659 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2660 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2661 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2662 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2663 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2664 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2665 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2666 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2667 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2668 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2669 2024-04-25T13:56:18.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2670 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 13:40 2673 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 13:40 2674 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 13:40 2675 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2678 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2679 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2680 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2681 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2682 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2683 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2684 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2685 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2686 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2687 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2688 2024-04-25T13:56:18.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2689 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 13:40 2690 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 13:40 2691 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2692 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2693 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2696 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2699 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2701 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2702 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2703 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2704 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2753 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2753_fsm 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2753_vm 2024-04-25T13:56:18.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2754 2024-04-25T13:56:18.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2755 2024-04-25T13:56:18.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2756 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2757 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2830 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2831 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2832 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2833 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2834 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2835 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2836 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2837 2024-04-25T13:56:18.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 13:40 2838 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2838_fsm 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2838_vm 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2839 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2840 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2840_fsm 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2840_vm 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2841 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2995 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2996 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3079 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3079_fsm 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3079_vm 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3080 2024-04-25T13:56:18.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3081 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 3085 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3118 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3119 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 13:40 3164 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3256 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3257 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3258 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3350 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3351 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3379 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3380 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3381 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3394 2024-04-25T13:56:18.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3394_fsm 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3394_vm 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3395 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3429 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3430 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3431 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3433 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3439 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3440 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 3455 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 13:40 3456 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3456_fsm 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3456_vm 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3466 2024-04-25T13:56:18.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3467 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3468 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3501 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3502 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3503 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3534 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3541_fsm 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541_vm 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3542 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3574 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3575 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3576 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3596 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3597 2024-04-25T13:56:18.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3598 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3599 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3600_fsm 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600_vm 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3601_fsm 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601_vm 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3602_fsm 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602_vm 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3603 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3603_fsm 2024-04-25T13:56:18.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3603_vm 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3604 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3605 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3606 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3607 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3608 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 3609 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3712 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3764 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3764_fsm 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3764_vm 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3766 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3767 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3997 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4143 2024-04-25T13:56:18.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4144 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4145 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4146 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4147 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4148 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4149 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4150 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4151 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4152 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4153 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4154 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4155 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4156 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4157 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4158 2024-04-25T13:56:18.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4159 2024-04-25T13:56:18.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4160 2024-04-25T13:56:18.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4161 2024-04-25T13:56:18.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4162 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4163 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4164 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4165 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4166 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4167 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4168 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4169 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4170 2024-04-25T13:56:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4171 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4172 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4173 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4174 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 5002 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 548 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 549 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6102 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6104 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6106 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6110 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6111 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6112 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6113 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6117 2024-04-25T13:56:18.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 826 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 827 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 828 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 13:40 pg_filenode.map 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 PG_VERSION 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:total 7777 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 13:40 . 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 13:41 .. 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 112 2024-04-25T13:56:18.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 113 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:40 1247 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1247_fsm 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1247_vm 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 13:40 1249 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1249_fsm 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1249_vm 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 13:40 1255 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1255_fsm 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1255_vm 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 13:40 1259 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1259_fsm 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1259_vm 2024-04-25T13:56:18.827 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 13:40 13344 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13344_fsm 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13344_vm 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13346 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13348 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13349_fsm 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349_vm 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13351 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13353 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13354 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13354_fsm 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13354_vm 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13356 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13358 2024-04-25T13:56:18.828 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13359_fsm 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359_vm 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13361 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13363 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1417 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1418 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 174 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 175 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2187 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2224 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2328 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2336 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2337 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2579 2024-04-25T13:56:18.829 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2600 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2600_fsm 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2600_vm 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2601_fsm 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601_vm 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2602 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2602_fsm 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2602_vm 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2603 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2603_fsm 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2603_vm 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2604 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2605 2024-04-25T13:56:18.830 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2605_fsm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2605_vm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2606 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2606_fsm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2606_vm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2607 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2607_fsm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2607_vm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 13:40 2608 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2608_fsm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2608_vm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 13:40 2609 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2609_fsm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2609_vm 2024-04-25T13:56:18.831 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2610 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2610_fsm 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2610_vm 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2611 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2612_fsm 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612_vm 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2613 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2615_fsm 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615_vm 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2616 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2616_fsm 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2616_vm 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 13:40 2617 2024-04-25T13:56:18.832 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2617_fsm 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2617_vm 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 13:40 2618 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2618_fsm 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2618_vm 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 13:40 2619 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2619_fsm 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2619_vm 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2620 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2650 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2651 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2652 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2653 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2654 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2655 2024-04-25T13:56:18.833 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2656 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2657 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 13:40 2658 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:40 2659 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2660 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2661 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2662 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2663 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2664 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2665 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2666 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2667 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2668 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2669 2024-04-25T13:56:18.834 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2670 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 13:40 2673 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 13:40 2674 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 13:40 2675 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2678 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2679 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2680 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2681 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2682 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2683 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2684 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2685 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2686 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2687 2024-04-25T13:56:18.835 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2688 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2689 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 13:40 2690 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 13:40 2691 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2692 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2693 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2696 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2699 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2701 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2702 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2703 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2704 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2753 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2753_fsm 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2753_vm 2024-04-25T13:56:18.836 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2754 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2755 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2756 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2757 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2830 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2831 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2832 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2833 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2834 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2835 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2836 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2837 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 13:40 2838 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2838_fsm 2024-04-25T13:56:18.837 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2838_vm 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2839 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2840 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2840_fsm 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2840_vm 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2841 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2995 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2996 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3079 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3079_fsm 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3079_vm 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3080 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3081 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 3085 2024-04-25T13:56:18.838 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3118 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3119 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 13:40 3164 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3256 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3257 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3258 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3350 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3351 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3379 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3380 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3381 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3394 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3394_fsm 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3394_vm 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3395 2024-04-25T13:56:18.839 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3429 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3430 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3431 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3433 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3439 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3440 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 3455 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 13:40 3456 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3456_fsm 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3456_vm 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3466 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3467 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3468 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3501 2024-04-25T13:56:18.840 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3502 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3503 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3534 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3541_fsm 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541_vm 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3542 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3574 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3575 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3576 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3596 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3597 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3598 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3599 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600 2024-04-25T13:56:18.841 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3600_fsm 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600_vm 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3601_fsm 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601_vm 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3602_fsm 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602_vm 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3603 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3603_fsm 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3603_vm 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3604 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3605 2024-04-25T13:56:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3606 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3607 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3608 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 3609 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3712 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3764 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3764_fsm 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3764_vm 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3766 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3767 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3997 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4143 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4144 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4145 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4146 2024-04-25T13:56:18.843 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4147 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4148 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4149 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4150 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4151 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4152 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4153 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4154 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4155 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4156 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4157 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4158 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4159 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4160 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4161 2024-04-25T13:56:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4162 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4163 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4164 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4165 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4166 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4167 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4168 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4169 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4170 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4171 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4172 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4173 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4174 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 5002 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 548 2024-04-25T13:56:18.845 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 549 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6102 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6104 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6106 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6110 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6111 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6112 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6113 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6117 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 826 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 827 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 828 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 13:40 pg_filenode.map 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:40 PG_VERSION 2024-04-25T13:56:18.846 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:total 628886 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 25 13:44 . 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 13:41 .. 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 112 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 113 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:41 1247 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1247_fsm 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 1247_vm 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 13:41 1249 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 1249_fsm 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 1249_vm 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 13:40 1255 2024-04-25T13:56:18.847 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 1255_fsm 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1255_vm 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 13:56 1259 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 1259_fsm 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 1259_vm 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 13:40 13344 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13344_fsm 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13344_vm 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13346 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13348 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13349 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13349_fsm 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13349_vm 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 13351 2024-04-25T13:56:18.848 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13353 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13354 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 13354_fsm 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 13354_vm 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 13356 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13358 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 13359_fsm 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13359_vm 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 13361 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 13363 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 1417 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 1418 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 445849600 Apr 25 13:56 16396 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 25 13:56 16396_fsm 2024-04-25T13:56:18.849 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:43 16396_vm 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1310720 Apr 25 13:56 16397 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:56 16397_fsm 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:56 16397_vm 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1908736 Apr 25 13:56 16399 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:55 16399_fsm 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:55 16399_vm 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:56 16400 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 13:56 16402 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Apr 25 13:56 16404 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114507776 Apr 25 13:56 16406 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:56 16406_fsm 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:54 16406_vm 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 174 2024-04-25T13:56:18.850 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 175 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2187 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2224 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2328 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2336 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2337 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2579 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2600 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2600_fsm 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2600_vm 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2601_fsm 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2601_vm 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2602 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2602_fsm 2024-04-25T13:56:18.851 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2602_vm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2603 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2603_fsm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2603_vm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2604 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2605 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2605_fsm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2605_vm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2606 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2606_fsm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2606_vm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2607 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2607_fsm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2607_vm 2024-04-25T13:56:18.852 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 13:41 2608 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2608_fsm 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2608_vm 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 13:40 2609 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2609_fsm 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2609_vm 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2610 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2610_fsm 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2610_vm 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2611 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2612_fsm 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2612_vm 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2613 2024-04-25T13:56:18.853 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2615_fsm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2615_vm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2616 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2616_fsm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2616_vm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 13:40 2617 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2617_fsm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2617_vm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 13:40 2618 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2618_fsm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2618_vm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 172032 Apr 25 13:56 2619 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:45 2619_fsm 2024-04-25T13:56:18.854 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2619_vm 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2620 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2650 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2651 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2652 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 2653 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 2654 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2655 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2656 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2657 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 13:41 2658 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 13:41 2659 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2660 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2661 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2662 2024-04-25T13:56:18.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:41 2663 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2664 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2665 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2666 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2667 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2668 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2669 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2670 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 13:41 2673 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 13:41 2674 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 13:40 2675 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2678 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2679 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2680 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2681 2024-04-25T13:56:18.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2682 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2683 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2684 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2685 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2686 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2687 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:40 2688 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 13:40 2689 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 13:41 2690 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 13:41 2691 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2692 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2693 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:46 2696 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2699 2024-04-25T13:56:18.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2701 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2702 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:41 2703 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 2704 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2753 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2753_fsm 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2753_vm 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2754 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2755 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2756 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 2757 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2830 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2831 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2832 2024-04-25T13:56:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2833 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2834 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2835 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 2836 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2837 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 13:41 2838 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2838_fsm 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2838_vm 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2839 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2840 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 2840_fsm 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2840_vm 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2841 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2995 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2996 2024-04-25T13:56:18.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3079 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3079_fsm 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3079_vm 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3080 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3081 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 3085 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3118 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3119 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 13:41 3164 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3256 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3257 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3258 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3350 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3351 2024-04-25T13:56:18.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3379 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3380 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3381 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3394 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3394_fsm 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3394_vm 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3395 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3429 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3430 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3431 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3433 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3439 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3440 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 13:41 3455 2024-04-25T13:56:18.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 13:40 3456 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3456_fsm 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3456_vm 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3466 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3467 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3468 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3501 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3502 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3503 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3534 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3541 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 3541_fsm 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3541_vm 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3542 2024-04-25T13:56:18.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3574 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3575 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 3576 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3596 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3597 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3598 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3599 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3600_fsm 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3600_vm 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3601 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3601_fsm 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3601_vm 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3602 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 3602_fsm 2024-04-25T13:56:18.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3602_vm 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 3603 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:41 3603_fsm 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3603_vm 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3604 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3605 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3606 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3607 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3608 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 13:40 3609 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 3712 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3764 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 3764_fsm 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3764_vm 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3766 2024-04-25T13:56:18.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 3767 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 3997 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4143 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4144 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4145 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4146 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4147 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4148 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4149 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4150 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4151 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4152 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4153 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4154 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4155 2024-04-25T13:56:18.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4156 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4157 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4158 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4159 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4160 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4161 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4162 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4163 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4164 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4165 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4166 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 4167 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4168 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4169 2024-04-25T13:56:18.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4170 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4171 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4172 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4173 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 4174 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 5002 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 548 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 549 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6102 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6104 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 6106 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 6110 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 6111 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 6112 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6113 2024-04-25T13:56:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6117 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 826 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 827 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 828 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 13:41 pg_filenode.map 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Apr 25 13:41 pg_internal.init 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:41 PG_VERSION 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:41 . 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 13:41 .. 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/global: 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:total 560 2024-04-25T13:56:18.868 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 25 13:41 . 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1213 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1213_fsm 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1213_vm 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1214 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1214_fsm 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1214_vm 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 1232 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 1233 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1260 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1260_fsm 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1260_vm 2024-04-25T13:56:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1261 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1261_fsm 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1261_vm 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 1262 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 1262_fsm 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 1262_vm 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:41 2396 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:40 2396_fsm 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2396_vm 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:41 2397 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2671 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2672 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2676 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2677 2024-04-25T13:56:18.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2694 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2695 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2697 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 13:40 2698 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2846 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2847 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2964 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2965 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 2966 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 2967 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 3592 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 3593 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4060 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4061 2024-04-25T13:56:18.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4175 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4176 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4177 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4178 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4181 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4182 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4183 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4184 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 4185 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 4186 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6000 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6001 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6002 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 6100 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6114 2024-04-25T13:56:18.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 6115 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:56 pg_control 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 13:40 pg_filenode.map 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Apr 25 13:41 pg_internal.init 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/log: 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:total 2 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:41 . 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1222 Apr 25 13:56 postgresql-Thu.log 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.873 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:total 1 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:56 . 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 mappings 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 25 13:56 replorigin_checkpoint 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 snapshots 2024-04-25T13:56:18.874 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:56 .. 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:56 .. 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 . 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.875 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 members 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 offsets 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:total 8 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 .. 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:40 0000 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:total 8 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:40 . 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 13:40 .. 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 13:43 0000 2024-04-25T13:56:18.876 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:total 15 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:56 . 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Apr 25 13:56 db_0.stat 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Apr 25 13:56 db_13494.stat 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Apr 25 13:56 global.stat 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.878 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:56 . 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:total 112 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 13:56 . 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 13:56 0021 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:total 1048576 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 13:56 . 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:56 0000000100000000000000C6 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:51 0000000100000000000000C7 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000C8 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000C9 2024-04-25T13:56:18.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000CA 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:53 0000000100000000000000CB 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000CC 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000CD 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000CE 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:51 0000000100000000000000CF 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:52 0000000100000000000000D0 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:53 0000000100000000000000D1 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000D2 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:51 0000000100000000000000D3 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000D4 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:51 0000000100000000000000D5 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000D6 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:51 0000000100000000000000D7 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000D8 2024-04-25T13:56:18.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:53 0000000100000000000000D9 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000DA 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000DB 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000DC 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000DD 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000DE 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000DF 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:51 0000000100000000000000E0 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000E1 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:51 0000000100000000000000E2 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:52 0000000100000000000000E3 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000E4 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000E5 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000E6 2024-04-25T13:56:18.882 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:50 0000000100000000000000E7 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000E8 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:56 0000000100000000000000E9 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000EA 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000EB 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000EC 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000ED 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000EE 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000EF 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000F0 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000F1 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000F2 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000F3 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000F4 2024-04-25T13:56:18.883 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000F5 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000F6 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000F7 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000F8 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000F9 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000FA 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000FB 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000FC 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000FD 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 0000000100000000000000FE 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 0000000100000000000000FF 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 000000010000000100000000 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 000000010000000100000001 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 000000010000000100000002 2024-04-25T13:56:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 000000010000000100000003 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:55 000000010000000100000004 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 13:54 000000010000000100000005 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 archive_status 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:total 0 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 13:40 . 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 13:56 .. 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:total 536 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 13:56 . 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 13:56 .. 2024-04-25T13:56:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Apr 25 13:46 0000 2024-04-25T13:56:18.886 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Apr 25 13:56 0001 2024-04-25T13:56:18.886 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 13:56 0002 2024-04-25T13:56:18.887 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T13:56:18.898 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T13:56:18.906 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T13:56:18.916 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T13:56:19.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:19 smithi122 ceph-mon[32843]: scrub queued for path: / 2024-04-25T13:56:19.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:19 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:56:19.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:19 smithi122 ceph-mon[32843]: scrub summary: active paths [/] 2024-04-25T13:56:19.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:19 smithi122 ceph-mon[32843]: scrub summary: idle+waiting paths [/] 2024-04-25T13:56:19.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:19 smithi122 ceph-mon[32843]: scrub summary: idle 2024-04-25T13:56:19.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:19 smithi045 ceph-mon[32731]: scrub queued for path: / 2024-04-25T13:56:19.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:19 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:56:19.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:19 smithi045 ceph-mon[32731]: scrub summary: active paths [/] 2024-04-25T13:56:19.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:19 smithi045 ceph-mon[32731]: scrub summary: idle+waiting paths [/] 2024-04-25T13:56:19.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:19 smithi045 ceph-mon[32731]: scrub summary: idle 2024-04-25T13:56:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:19 smithi027 ceph-mon[29140]: scrub queued for path: / 2024-04-25T13:56:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:19 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:56:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:19 smithi027 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T13:56:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:19 smithi027 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T13:56:19.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:19 smithi027 ceph-mon[29140]: scrub summary: idle 2024-04-25T13:56:20.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:20 smithi122 ceph-mon[32843]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-25T13:56:20.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:20 smithi045 ceph-mon[32731]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-25T13:56:20.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:20 smithi027 ceph-mon[29140]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-25T13:56:22.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:22 smithi122 ceph-mon[32843]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 1.39k op/s 2024-04-25T13:56:22.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:22 smithi045 ceph-mon[32731]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 1.39k op/s 2024-04-25T13:56:22.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:22 smithi027 ceph-mon[29140]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 1.39k op/s 2024-04-25T13:56:24.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:24 smithi122 ceph-mon[32843]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 22 MiB/s wr, 1.96k op/s 2024-04-25T13:56:24.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:24 smithi045 ceph-mon[32731]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 22 MiB/s wr, 1.96k op/s 2024-04-25T13:56:24.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:24 smithi027 ceph-mon[29140]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 22 MiB/s wr, 1.96k op/s 2024-04-25T13:56:26.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:26 smithi045 ceph-mon[32731]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 19 MiB/s wr, 1.69k op/s 2024-04-25T13:56:26.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:26 smithi027 ceph-mon[29140]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 19 MiB/s wr, 1.69k op/s 2024-04-25T13:56:26.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:26 smithi122 ceph-mon[32843]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 19 MiB/s wr, 1.69k op/s 2024-04-25T13:56:28.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:28 smithi045 ceph-mon[32731]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-25T13:56:28.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:28 smithi027 ceph-mon[29140]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-25T13:56:28.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:28 smithi122 ceph-mon[32843]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-04-25T13:56:30.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:30 smithi045 ceph-mon[32731]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-04-25T13:56:30.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:30 smithi027 ceph-mon[29140]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-04-25T13:56:30.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:30 smithi122 ceph-mon[32843]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-04-25T13:56:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:32 smithi045 ceph-mon[32731]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 12 MiB/s wr, 988 op/s 2024-04-25T13:56:32.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:32 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:56:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:32 smithi027 ceph-mon[29140]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 12 MiB/s wr, 988 op/s 2024-04-25T13:56:32.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:32 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:56:32.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:32 smithi122 ceph-mon[32843]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 12 MiB/s wr, 988 op/s 2024-04-25T13:56:32.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:32 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T13:56:33.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:56:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:56:33.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:33 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:56:33.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:56:33.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:56:33.401 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:33 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:56:33.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T13:56:33.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T13:56:33.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:33 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' 2024-04-25T13:56:34.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:34 smithi027 ceph-mon[29140]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 8.0 MiB/s wr, 692 op/s 2024-04-25T13:56:34.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:34 smithi122 ceph-mon[32843]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 8.0 MiB/s wr, 692 op/s 2024-04-25T13:56:34.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:34 smithi045 ceph-mon[32731]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 8.0 MiB/s wr, 692 op/s 2024-04-25T13:56:36.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:36 smithi027 ceph-mon[29140]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 380 KiB/s wr, 55 op/s 2024-04-25T13:56:36.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:36 smithi122 ceph-mon[32843]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 380 KiB/s wr, 55 op/s 2024-04-25T13:56:36.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:36 smithi045 ceph-mon[32731]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 380 KiB/s wr, 55 op/s 2024-04-25T13:56:38.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:38 smithi027 ceph-mon[29140]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.9 KiB/s wr, 3 op/s 2024-04-25T13:56:38.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:38 smithi122 ceph-mon[32843]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.9 KiB/s wr, 3 op/s 2024-04-25T13:56:38.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:38 smithi045 ceph-mon[32731]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.9 KiB/s wr, 3 op/s 2024-04-25T13:56:40.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:40 smithi027 ceph-mon[29140]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-25T13:56:40.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:40 smithi122 ceph-mon[32843]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-25T13:56:40.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:40 smithi045 ceph-mon[32731]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2024-04-25T13:56:42.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:42 smithi027 ceph-mon[29140]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6.7 KiB/s wr, 4 op/s 2024-04-25T13:56:42.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:42 smithi122 ceph-mon[32843]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6.7 KiB/s wr, 4 op/s 2024-04-25T13:56:42.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:42 smithi045 ceph-mon[32731]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6.7 KiB/s wr, 4 op/s 2024-04-25T13:56:44.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:44 smithi027 ceph-mon[29140]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 20 KiB/s wr, 5 op/s 2024-04-25T13:56:44.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:44 smithi122 ceph-mon[32843]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 20 KiB/s wr, 5 op/s 2024-04-25T13:56:44.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:44 smithi045 ceph-mon[32731]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 20 KiB/s wr, 5 op/s 2024-04-25T13:56:46.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:46 smithi027 ceph-mon[29140]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 31 KiB/s wr, 6 op/s 2024-04-25T13:56:46.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:46 smithi122 ceph-mon[32843]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 31 KiB/s wr, 6 op/s 2024-04-25T13:56:46.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:46 smithi045 ceph-mon[32731]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 31 KiB/s wr, 6 op/s 2024-04-25T13:56:48.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T13:56:48.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T13:56:48.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 -- 172.21.15.27:0/3367688368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538108920 msgr2=0x7f653810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 --2- 172.21.15.27:0/3367688368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538108920 0x7f653810ad40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6530009ca0 tx=0x7f653002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 -- 172.21.15.27:0/3367688368 shutdown_connections 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 --2- 172.21.15.27:0/3367688368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538108920 0x7f653810ad40 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 --2- 172.21.15.27:0/3367688368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6538103c20 0x7f6538104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 --2- 172.21.15.27:0/3367688368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65381029e0 0x7f65381036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 -- 172.21.15.27:0/3367688368 >> 172.21.15.27:0/3367688368 conn(0x7f65380fdbd0 msgr2=0x7f6538100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 -- 172.21.15.27:0/3367688368 shutdown_connections 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.382+0000 7f653e427640 1 -- 172.21.15.27:0/3367688368 wait complete. 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f653e427640 1 Processor -- start 2024-04-25T13:56:48.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f653e427640 1 -- start start 2024-04-25T13:56:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f653e427640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65381029e0 0x7f653819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f653e427640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538103c20 0x7f653819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f653e427640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6538108920 0x7f65381a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f653e427640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f653810f610 con 0x7f6538103c20 2024-04-25T13:56:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f653e427640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f653810f490 con 0x7f6538108920 2024-04-25T13:56:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f653e427640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f653810f790 con 0x7f65381029e0 2024-04-25T13:56:48.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.383+0000 7f65377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538103c20 0x7f653819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538103c20 0x7f653819aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52538/0 (socket says 172.21.15.27:52538) 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65377fe640 1 -- 172.21.15.27:0/2776585013 learned_addr learned my addr 172.21.15.27:0/2776585013 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f6537fff640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65381029e0 0x7f653819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f653c9c4640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6538108920 0x7f65381a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65377fe640 1 -- 172.21.15.27:0/2776585013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65381029e0 msgr2=0x7f653819a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65377fe640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65381029e0 0x7f653819a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65377fe640 1 -- 172.21.15.27:0/2776585013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6538108920 msgr2=0x7f65381a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65377fe640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6538108920 0x7f65381a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65377fe640 1 -- 172.21.15.27:0/2776585013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65381a2020 con 0x7f6538103c20 2024-04-25T13:56:48.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f6537fff640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65381029e0 0x7f653819a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:56:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65377fe640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538103c20 0x7f653819aa20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e9f0 tx=0x7f652c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65357fa640 1 -- 172.21.15.27:0/2776585013 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c00cd80 con 0x7f6538103c20 2024-04-25T13:56:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65357fa640 1 -- 172.21.15.27:0/2776585013 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f652c0054d0 con 0x7f6538103c20 2024-04-25T13:56:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f653e427640 1 -- 172.21.15.27:0/2776585013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65381a2310 con 0x7f6538103c20 2024-04-25T13:56:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f65357fa640 1 -- 172.21.15.27:0/2776585013 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c010690 con 0x7f6538103c20 2024-04-25T13:56:48.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.384+0000 7f653e427640 1 -- 172.21.15.27:0/2776585013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65381a2810 con 0x7f6538103c20 2024-04-25T13:56:48.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65453b9740 1 Processor -- start 2024-04-25T13:56:48.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65357fa640 1 -- 172.21.15.27:0/2776585013 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f652c004e90 con 0x7f6538103c20 2024-04-25T13:56:48.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65453b9740 1 -- start start 2024-04-25T13:56:48.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65453b9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33d8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65453b9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 0x55fed33d8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65453b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33d9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65453b9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fed333dd80 con 0x55fed3037390 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65453b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55fed32f78b0 con 0x55fed30166b0 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f65453b9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55fed3339d40 con 0x55fed300e4d0 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f653c9c4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33d8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f653c9c4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33d8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52550/0 (socket says 172.21.15.27:52550) 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f653c9c4640 1 -- 172.21.15.27:0/1712440980 learned_addr learned my addr 172.21.15.27:0/1712440980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.387+0000 7f6537fff640 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 0x55fed33d8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f65377fe640 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33d9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f653c9c4640 1 -- 172.21.15.27:0/1712440980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 msgr2=0x55fed33d8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f653c9c4640 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 0x55fed33d8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f653c9c4640 1 -- 172.21.15.27:0/1712440980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 msgr2=0x55fed33d9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f653c9c4640 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33d9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f653c9c4640 1 -- 172.21.15.27:0/1712440980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fed32c8a40 con 0x55fed3037390 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f65357fa640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65200afaf0 0x7f65200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f653c9c4640 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33d8a60 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f653002f390 tx=0x7f653002f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3334ffd86c3e75e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f6537fff640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65200afaf0 0x7f65200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f65357fa640 1 -- 172.21.15.27:0/2776585013 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f652c0df110 con 0x7f6538103c20 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f651e7fc640 1 -- 172.21.15.27:0/1712440980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6530031650 con 0x55fed3037390 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f651e7fc640 1 -- 172.21.15.27:0/1712440980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6530036040 con 0x55fed3037390 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.388+0000 7f651e7fc640 1 -- 172.21.15.27:0/1712440980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65300046e0 con 0x55fed3037390 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 -- 172.21.15.27:0/1712440980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 msgr2=0x55fed33d8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f6537fff640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65200afaf0 0x7f65200b2000 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x55fed33d8780 tx=0x7f6528005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33d8a60 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f653002f390 tx=0x7f653002f3c0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 -- 172.21.15.27:0/1712440980 shutdown_connections 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33d9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 0x55fed33d8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 --2- 172.21.15.27:0/1712440980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33d8a60 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 -- 172.21.15.27:0/1712440980 >> 172.21.15.27:0/1712440980 conn(0x55fed302b4e0 msgr2=0x55fed33d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 -- 172.21.15.27:0/1712440980 shutdown_connections 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 -- 172.21.15.27:0/1712440980 wait complete. 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.389+0000 7f65453b9740 1 -- start start 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65453b9740 1 Processor -- start 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65453b9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 0x55fed33e2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65453b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65453b9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33e9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65453b9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55fed33da720 con 0x55fed3037390 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65453b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55fed32c4d10 con 0x55fed30166b0 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65453b9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55fed33da8a0 con 0x55fed300e4d0 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65453b9740 10 client.0 _ll_register_callbacks cb 0x55fed32f1210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f653c9c4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33e9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f653c9c4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33e9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52558/0 (socket says 172.21.15.27:52558) 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f653c9c4640 1 -- 172.21.15.27:0/2455950775 learned_addr learned my addr 172.21.15.27:0/2455950775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f65377fe640 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.390+0000 7f6537fff640 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 0x55fed33e2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f653c9c4640 1 -- 172.21.15.27:0/2455950775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 msgr2=0x55fed33e2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f653c9c4640 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 0x55fed33e2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f653c9c4640 1 -- 172.21.15.27:0/2455950775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 msgr2=0x55fed33e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f653c9c4640 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f653c9c4640 1 -- 172.21.15.27:0/2455950775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fed31197d0 con 0x55fed3037390 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f653c9c4640 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33e9da0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f653002a900 tx=0x7f6530003fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca33de0efddcae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f65177fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f65177fe640 1 -- 172.21.15.27:0/2455950775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6530052020 con 0x55fed3037390 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f65177fe640 1 -- 172.21.15.27:0/2455950775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f6530031650 con 0x55fed3037390 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f65453b9740 1 -- 172.21.15.27:0/2455950775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6530036040 con 0x55fed3037390 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f65177fe640 1 -- 172.21.15.27:0/2455950775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f653004a040 con 0x55fed3037390 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f65453b9740 10 client.18468 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f65453b9740 1 -- 172.21.15.27:0/2455950775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55fed304f3f0 con 0x55fed3037390 2024-04-25T13:56:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.391+0000 7f65177fe640 1 -- 172.21.15.27:0/2455950775 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6530036040 con 0x55fed3037390 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f65177fe640 1 -- 172.21.15.27:0/2455950775 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f653004d030 con 0x55fed3037390 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f65453b9740 10 client.18468 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f65453b9740 10 client.18468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f65453b9740 20 client.18468 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f65453b9740 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fed2ff5f60 0x55fed33efe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f65453b9740 4 client.18468 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f65453b9740 1 -- 172.21.15.27:0/2455950775 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fed334b990 con 0x55fed2ff5f60 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f6537fff640 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fed2ff5f60 0x55fed33efe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f65177fe640 10 client.18468 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.392+0000 7f6537fff640 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fed2ff5f60 0x55fed33efe60 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d875e15c6468e8c3 server_cookie=a4a71541982f5f84 in_seq=0 out_seq=0 2024-04-25T13:56:48.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65177fe640 1 -- 172.21.15.27:0/2455950775 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fed334b990 con 0x55fed2ff5f60 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65177fe640 10 client.18468 handle_command_reply: tid=0 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 client.18468 shutdown 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 -- 172.21.15.27:0/2455950775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 msgr2=0x55fed33e9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33e9da0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f653002a900 tx=0x7f6530003fa0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 -- 172.21.15.27:0/2455950775 shutdown_connections 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fed2ff5f60 0x55fed33efe60 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed3037390 0x55fed33e9da0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65177fe640 0 client.18468 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 --2- 172.21.15.27:0/2455950775 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed300e4d0 0x55fed33e2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 -- 172.21.15.27:0/2455950775 >> 172.21.15.27:0/2455950775 conn(0x55fed302b4e0 msgr2=0x55fed33d7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 -- 172.21.15.27:0/2455950775 shutdown_connections 2024-04-25T13:56:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.393+0000 7f65453b9740 1 -- 172.21.15.27:0/2455950775 wait complete. 2024-04-25T13:56:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.394+0000 7f65453b9740 20 client.18468 trim_cache size 0 max 16384 2024-04-25T13:56:48.404 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:48 smithi027 ceph-mon[29140]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 31 KiB/s wr, 5 op/s 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65453b9740 1 Processor -- start 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65453b9740 1 -- start start 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65453b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed2ff5f60 0x55fed33e1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65453b9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33d4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65453b9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed30166b0 0x55fed33e9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65453b9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65381a9170 con 0x55fed300e4d0 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65453b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65381ab9f0 con 0x55fed2ff5f60 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65453b9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f65381a97e0 con 0x55fed30166b0 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33d4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33d4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52570/0 (socket says 172.21.15.27:52570) 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65377fe640 1 -- 172.21.15.27:0/2325646060 learned_addr learned my addr 172.21.15.27:0/2325646060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f653c9c4640 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed30166b0 0x55fed33e9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f6537fff640 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed2ff5f60 0x55fed33e1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65377fe640 1 -- 172.21.15.27:0/2325646060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed30166b0 msgr2=0x55fed33e9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65377fe640 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed30166b0 0x55fed33e9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65377fe640 1 -- 172.21.15.27:0/2325646060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed2ff5f60 msgr2=0x55fed33e1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65377fe640 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed2ff5f60 0x55fed33e1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f65377fe640 1 -- 172.21.15.27:0/2325646060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fed300f0f0 con 0x55fed300e4d0 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f653c9c4640 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed30166b0 0x55fed33e9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.412+0000 7f6537fff640 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed2ff5f60 0x55fed33e1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:56:48.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.413+0000 7f65377fe640 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33d4320 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f652c0a36a0 tx=0x7f652c0a6f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c130ea65e019087e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.413+0000 7f651dffb640 1 -- 172.21.15.27:0/2325646060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0a7130 con 0x55fed300e4d0 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.413+0000 7f651dffb640 1 -- 172.21.15.27:0/2325646060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f652c005680 con 0x55fed300e4d0 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.413+0000 7f651dffb640 1 -- 172.21.15.27:0/2325646060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0046c0 con 0x55fed300e4d0 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.413+0000 7f65453b9740 1 -- 172.21.15.27:0/2325646060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 msgr2=0x55fed33d4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.413+0000 7f65453b9740 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33d4320 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f652c0a36a0 tx=0x7f652c0a6f70 comp rx=0 tx=0).stop 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 -- 172.21.15.27:0/2325646060 shutdown_connections 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed30166b0 0x55fed33e9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33d4320 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 --2- 172.21.15.27:0/2325646060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed2ff5f60 0x55fed33e1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 -- 172.21.15.27:0/2325646060 >> 172.21.15.27:0/2325646060 conn(0x55fed302b4e0 msgr2=0x55fed33d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 -- 172.21.15.27:0/2325646060 shutdown_connections 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 -- 172.21.15.27:0/2325646060 wait complete. 2024-04-25T13:56:48.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 -- start start 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 Processor -- start 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed2ff5f60 0x55fed33e8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33e8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f65381a9eb0 con 0x55fed300e4d0 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65381ab320 con 0x55fed30166b0 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f65381aa580 con 0x55fed2ff5f60 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.414+0000 7f65453b9740 10 client.0 _ll_register_callbacks cb 0x7f652c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33e8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33e8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52582/0 (socket says 172.21.15.27:52582) 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 -- 172.21.15.27:0/357021378 learned_addr learned my addr 172.21.15.27:0/357021378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f6537fff640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed2ff5f60 0x55fed33e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f653c9c4640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 -- 172.21.15.27:0/357021378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed2ff5f60 msgr2=0x55fed33e8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed2ff5f60 0x55fed33e8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 -- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 msgr2=0x55fed33e9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 -- 172.21.15.27:0/357021378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fed32fe4b0 con 0x55fed300e4d0 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f6537fff640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed2ff5f60 0x55fed33e8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f653c9c4640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65377fe640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33e8760 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f652c0a77e0 tx=0x7f652c0a70a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a1dff6ec6896d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65177fe640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65177fe640 1 -- 172.21.15.27:0/357021378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0c1060 con 0x55fed300e4d0 2024-04-25T13:56:48.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65453b9740 1 -- 172.21.15.27:0/357021378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f652c005680 con 0x55fed300e4d0 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65177fe640 1 -- 172.21.15.27:0/357021378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f652c0afff0 con 0x55fed300e4d0 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65177fe640 1 -- 172.21.15.27:0/357021378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f652c0b0270 con 0x55fed300e4d0 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65453b9740 10 client.18474 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.415+0000 7f65453b9740 1 -- 172.21.15.27:0/357021378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55fed3458300 con 0x55fed300e4d0 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f65177fe640 1 -- 172.21.15.27:0/357021378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f652c005680 con 0x55fed300e4d0 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f65177fe640 1 -- 172.21.15.27:0/357021378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f652c0a2f90 con 0x55fed300e4d0 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f65453b9740 10 client.18474 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f65453b9740 10 client.18474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f65453b9740 20 client.18474 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f65453b9740 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fed3037390 0x55fed345abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f65453b9740 4 client.18474 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T13:56:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f65453b9740 1 -- 172.21.15.27:0/357021378 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fed3345800 con 0x55fed3037390 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.416+0000 7f6537fff640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fed3037390 0x55fed345abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65177fe640 10 client.18474 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f6537fff640 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fed3037390 0x55fed345abe0 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f9b1381bdd0f4b7 server_cookie=5503cf1406127595 in_seq=0 out_seq=0 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65177fe640 1 -- 172.21.15.27:0/357021378 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fed3345800 con 0x55fed3037390 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65177fe640 10 client.18474 handle_command_reply: tid=0 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 client.18474 shutdown 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 -- 172.21.15.27:0/357021378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 msgr2=0x55fed33e8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33e8760 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f652c0a77e0 tx=0x7f652c0a70a0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 -- 172.21.15.27:0/357021378 shutdown_connections 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55fed3037390 0x55fed345abe0 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fed30166b0 0x55fed33e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65177fe640 0 client.18474 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55fed300e4d0 0x55fed33e8760 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 --2- 172.21.15.27:0/357021378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55fed2ff5f60 0x55fed33e8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 -- 172.21.15.27:0/357021378 >> 172.21.15.27:0/357021378 conn(0x55fed302b4e0 msgr2=0x55fed33d29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 -- 172.21.15.27:0/357021378 shutdown_connections 2024-04-25T13:56:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 1 -- 172.21.15.27:0/357021378 wait complete. 2024-04-25T13:56:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.417+0000 7f65453b9740 20 client.18474 trim_cache size 0 max 16384 2024-04-25T13:56:48.425 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:56:48.425 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "no active scrubs running", 2024-04-25T13:56:48.426 INFO:teuthology.orchestra.run.smithi027.stdout: "scrubs": {} 2024-04-25T13:56:48.426 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 -- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65200afaf0 msgr2=0x7f65200b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65200afaf0 0x7f65200b2000 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x55fed33d8780 tx=0x7f6528005f90 comp rx=0 tx=0).stop 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 -- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538103c20 msgr2=0x7f653819aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538103c20 0x7f653819aa20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f652c00e9f0 tx=0x7f652c00edf0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 -- 172.21.15.27:0/2776585013 shutdown_connections 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f65200afaf0 0x7f65200b2000 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6538108920 0x7f65381a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6538103c20 0x7f653819aa20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 --2- 172.21.15.27:0/2776585013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65381029e0 0x7f653819a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 -- 172.21.15.27:0/2776585013 >> 172.21.15.27:0/2776585013 conn(0x7f65380fdbd0 msgr2=0x7f6538104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 -- 172.21.15.27:0/2776585013 shutdown_connections 2024-04-25T13:56:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.418+0000 7f653e427640 1 -- 172.21.15.27:0/2776585013 wait complete. 2024-04-25T13:56:48.429 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T13:56:48.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:56:48.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 -- 172.21.15.27:0/3296351156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 msgr2=0x7fd114106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 --2- 172.21.15.27:0/3296351156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 0x7fd114106610 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd110009900 tx=0x7fd11002a410 comp rx=0 tx=0).stop 2024-04-25T13:56:48.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 -- 172.21.15.27:0/3296351156 shutdown_connections 2024-04-25T13:56:48.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 --2- 172.21.15.27:0/3296351156 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd114109750 0x7fd11410bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 --2- 172.21.15.27:0/3296351156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd114106be0 0x7fd1141090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 --2- 172.21.15.27:0/3296351156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 0x7fd114106610 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 -- 172.21.15.27:0/3296351156 >> 172.21.15.27:0/3296351156 conn(0x7fd1140fdbd0 msgr2=0x7fd114100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 -- 172.21.15.27:0/3296351156 shutdown_connections 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.542+0000 7fd11ba49640 1 -- 172.21.15.27:0/3296351156 wait complete. 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd11ba49640 1 Processor -- start 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd11ba49640 1 -- start start 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd11ba49640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 0x7fd11419a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd11ba49640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd114106be0 0x7fd11419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd11ba49640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd114109750 0x7fd1141a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd11ba49640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd11410f590 con 0x7fd1141041f0 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd11ba49640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd11410f410 con 0x7fd114109750 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd11ba49640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fd11410f710 con 0x7fd114106be0 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 0x7fd11419a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 0x7fd11419a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52600/0 (socket says 172.21.15.27:52600) 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 -- 172.21.15.27:0/445957375 learned_addr learned my addr 172.21.15.27:0/445957375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd118fe4640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd114106be0 0x7fd11419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd119fe6640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd114109750 0x7fd1141a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 -- 172.21.15.27:0/445957375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd114106be0 msgr2=0x7fd11419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd114106be0 0x7fd11419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 -- 172.21.15.27:0/445957375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd114109750 msgr2=0x7fd1141a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd114109750 0x7fd1141a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 -- 172.21.15.27:0/445957375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1141a1c70 con 0x7fd1141041f0 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.543+0000 7fd1197e5640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 0x7fd11419a1e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd110006010 tx=0x7fd110031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.544+0000 7fd10a7fc640 1 -- 172.21.15.27:0/445957375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11003f030 con 0x7fd1141041f0 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.544+0000 7fd10a7fc640 1 -- 172.21.15.27:0/445957375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd110003aa0 con 0x7fd1141041f0 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.544+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1141a1f00 con 0x7fd1141041f0 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.544+0000 7fd10a7fc640 1 -- 172.21.15.27:0/445957375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd110035070 con 0x7fd1141041f0 2024-04-25T13:56:48.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.544+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1141a2310 con 0x7fd1141041f0 2024-04-25T13:56:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.545+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1140ff6f0 con 0x7fd1141041f0 2024-04-25T13:56:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.547+0000 7fd10a7fc640 1 -- 172.21.15.27:0/445957375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd110042030 con 0x7fd1141041f0 2024-04-25T13:56:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.548+0000 7fd10a7fc640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd1000afaf0 0x7fd1000b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.548+0000 7fd118fe4640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd1000afaf0 0x7fd1000b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.548+0000 7fd10a7fc640 1 -- 172.21.15.27:0/445957375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fd1100ff570 con 0x7fd1141041f0 2024-04-25T13:56:48.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.548+0000 7fd118fe4640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd1000afaf0 0x7fd1000b2000 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd11419b280 tx=0x7fd10400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.552+0000 7fd10a7fc640 1 -- 172.21.15.27:0/445957375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd1100c8120 con 0x7fd1141041f0 2024-04-25T13:56:48.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:48 smithi122 ceph-mon[32843]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 31 KiB/s wr, 5 op/s 2024-04-25T13:56:48.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:48 smithi045 ceph-mon[32731]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 31 KiB/s wr, 5 op/s 2024-04-25T13:56:48.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.780+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd11402dcb0 con 0x7fd1141041f0 2024-04-25T13:56:48.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.780+0000 7fd10a7fc640 1 -- 172.21.15.27:0/445957375 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7fd1100c3070 con 0x7fd1141041f0 2024-04-25T13:56:48.780 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:48.780 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:56:48.781 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:56:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd1000afaf0 msgr2=0x7fd1000b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd1000afaf0 0x7fd1000b2000 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd11419b280 tx=0x7fd10400a2d0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 msgr2=0x7fd11419a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 0x7fd11419a1e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd110006010 tx=0x7fd110031a50 comp rx=0 tx=0).stop 2024-04-25T13:56:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 shutdown_connections 2024-04-25T13:56:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fd1000afaf0 0x7fd1000b2000 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd114109750 0x7fd1141a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd114106be0 0x7fd11419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 --2- 172.21.15.27:0/445957375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1141041f0 0x7fd11419a1e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 >> 172.21.15.27:0/445957375 conn(0x7fd1140fdbd0 msgr2=0x7fd114106210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 shutdown_connections 2024-04-25T13:56:48.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.783+0000 7fd11ba49640 1 -- 172.21.15.27:0/445957375 wait complete. 2024-04-25T13:56:48.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T13:56:48.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 -- 172.21.15.27:0/3194519491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80109a30 msgr2=0x7f9b8010be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 --2- 172.21.15.27:0/3194519491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80109a30 0x7f9b8010be50 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009ca0 tx=0x7f9b7002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 -- 172.21.15.27:0/3194519491 shutdown_connections 2024-04-25T13:56:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 --2- 172.21.15.27:0/3194519491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80109a30 0x7f9b8010be50 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 --2- 172.21.15.27:0/3194519491 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b80104d30 0x7f9b80105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 --2- 172.21.15.27:0/3194519491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b80103af0 0x7f9b801047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 -- 172.21.15.27:0/3194519491 >> 172.21.15.27:0/3194519491 conn(0x7f9b800fece0 msgr2=0x7f9b80101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 -- 172.21.15.27:0/3194519491 shutdown_connections 2024-04-25T13:56:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 -- 172.21.15.27:0/3194519491 wait complete. 2024-04-25T13:56:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 Processor -- start 2024-04-25T13:56:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.909+0000 7f9b87dab640 1 -- start start 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b87dab640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80103af0 0x7f9b8019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b87dab640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b80104d30 0x7f9b8019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b87dab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b80109a30 0x7f9b801a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b87dab640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010f620 con 0x7f9b80103af0 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b87dab640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010f4a0 con 0x7f9b80109a30 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b87dab640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010f7a0 con 0x7f9b80104d30 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80103af0 0x7f9b8019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80103af0 0x7f9b8019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52616/0 (socket says 172.21.15.27:52616) 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 -- 172.21.15.27:0/4125936107 learned_addr learned my addr 172.21.15.27:0/4125936107 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b86348640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b80109a30 0x7f9b801a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85346640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b80104d30 0x7f9b8019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 -- 172.21.15.27:0/4125936107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b80104d30 msgr2=0x7f9b8019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b80104d30 0x7f9b8019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 -- 172.21.15.27:0/4125936107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b80109a30 msgr2=0x7f9b801a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b80109a30 0x7f9b801a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 -- 172.21.15.27:0/4125936107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b801a1cd0 con 0x7f9b80103af0 2024-04-25T13:56:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b85b47640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80103af0 0x7f9b8019a190 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800d740 tx=0x7f9b7800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.910+0000 7f9b7effd640 1 -- 172.21.15.27:0/4125936107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78017070 con 0x7f9b80103af0 2024-04-25T13:56:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.911+0000 7f9b7effd640 1 -- 172.21.15.27:0/4125936107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b7800de50 con 0x7f9b80103af0 2024-04-25T13:56:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.911+0000 7f9b87dab640 1 -- 172.21.15.27:0/4125936107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b801a1fc0 con 0x7f9b80103af0 2024-04-25T13:56:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.911+0000 7f9b7effd640 1 -- 172.21.15.27:0/4125936107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78003c20 con 0x7f9b80103af0 2024-04-25T13:56:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.911+0000 7f9b87dab640 1 -- 172.21.15.27:0/4125936107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b801a2420 con 0x7f9b80103af0 2024-04-25T13:56:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.913+0000 7f9b8ec58740 1 Processor -- start 2024-04-25T13:56:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.913+0000 7f9b8ec58740 1 -- start start 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.913+0000 7f9b7effd640 1 -- 172.21.15.27:0/4125936107 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9b78003dc0 con 0x7f9b80103af0 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.913+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 0x55802abc6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.913+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7fb4d0 0x55802abc6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abc7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55802ab2ad80 con 0x55802a8036b0 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55802aae4a20 con 0x55802a7fb4d0 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55802ab26d40 con 0x55802a824390 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abc7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abc7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52618/0 (socket says 172.21.15.27:52618) 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 -- 172.21.15.27:0/2789125604 learned_addr learned my addr 172.21.15.27:0/2789125604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85b47640 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7fb4d0 0x55802abc6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b86348640 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 0x55802abc6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 -- 172.21.15.27:0/2789125604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 msgr2=0x55802abc6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 0x55802abc6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 -- 172.21.15.27:0/2789125604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7fb4d0 msgr2=0x55802abc6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7fb4d0 0x55802abc6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 -- 172.21.15.27:0/2789125604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55802aaefc30 con 0x55802a8036b0 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b85346640 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abc7a20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9b74004650 tx=0x7f9b7400bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3913333a52c47c41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b7effd640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b680afaf0 0x7f9b680b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b67fff640 1 -- 172.21.15.27:0/2789125604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74003bd0 con 0x55802a8036b0 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.914+0000 7f9b86348640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b680afaf0 0x7f9b680b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b67fff640 1 -- 172.21.15.27:0/2789125604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b74015070 con 0x55802a8036b0 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b67fff640 1 -- 172.21.15.27:0/2789125604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74004d80 con 0x55802a8036b0 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b7effd640 1 -- 172.21.15.27:0/4125936107 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9b780de550 con 0x7f9b80103af0 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b86348640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b680afaf0 0x7f9b680b2000 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9b700094e0 tx=0x7f9b70009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2789125604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 msgr2=0x55802abc7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abc7a20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9b74004650 tx=0x7f9b7400bba0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2789125604 shutdown_connections 2024-04-25T13:56:48.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abc7a20 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7fb4d0 0x55802abc6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2789125604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 0x55802abc6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.915+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2789125604 >> 172.21.15.27:0/2789125604 conn(0x55802a8184e0 msgr2=0x55802abc2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2789125604 shutdown_connections 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2789125604 wait complete. 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 -- start start 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 Processor -- start 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802abd0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 0x55802abd7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x55802abc8530 con 0x55802a7fb4d0 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55802aab1dc0 con 0x55802a8036b0 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.916+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x55802abc86b0 con 0x55802a824390 2024-04-25T13:56:48.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b8ec58740 10 client.0 _ll_register_callbacks cb 0x55802aacd600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52628/0 (socket says 172.21.15.27:52628) 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 -- 172.21.15.27:0/684525077 learned_addr learned my addr 172.21.15.27:0/684525077 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85346640 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802abd0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b86348640 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 0x55802abd7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 -- 172.21.15.27:0/684525077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 msgr2=0x55802abd7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 0x55802abd7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 -- 172.21.15.27:0/684525077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 msgr2=0x55802abd0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802abd0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 -- 172.21.15.27:0/684525077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55802a83c3f0 con 0x55802a7fb4d0 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b85b47640 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd0560 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9b78000c00 tx=0x7f9b78009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f1b9b8e9ebaf6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b657fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b657fa640 1 -- 172.21.15.27:0/684525077 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78020070 con 0x55802a7fb4d0 2024-04-25T13:56:48.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b657fa640 1 -- 172.21.15.27:0/684525077 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9b780a2da0 con 0x55802a7fb4d0 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b8ec58740 1 -- 172.21.15.27:0/684525077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b74015070 con 0x55802a7fb4d0 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b657fa640 1 -- 172.21.15.27:0/684525077 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9b780a6460 con 0x55802a7fb4d0 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.917+0000 7f9b8ec58740 10 client.18492 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b8ec58740 1 -- 172.21.15.27:0/684525077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55802a9067d0 con 0x55802a7fb4d0 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b657fa640 1 -- 172.21.15.27:0/684525077 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b74015070 con 0x55802a7fb4d0 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b657fa640 1 -- 172.21.15.27:0/684525077 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9b780de160 con 0x55802a7fb4d0 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b8ec58740 10 client.18492 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b8ec58740 10 client.18492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b8ec58740 20 client.18492 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55802a7e2f60 0x55802abde290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b8ec58740 4 client.18492 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b8ec58740 1 -- 172.21.15.27:0/684525077 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55802ab38990 con 0x55802a7e2f60 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.918+0000 7f9b85346640 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55802a7e2f60 0x55802abde290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.919+0000 7f9b85346640 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55802a7e2f60 0x55802abde290 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f284f31e22da6b92 server_cookie=e8542b2f3a714908 in_seq=0 out_seq=0 2024-04-25T13:56:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.919+0000 7f9b657fa640 10 client.18492 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:48.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.919+0000 7f9b657fa640 1 -- 172.21.15.27:0/684525077 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55802ab38990 con 0x55802a7e2f60 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.919+0000 7f9b657fa640 10 client.18492 handle_command_reply: tid=0 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 client.18492 shutdown 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 -- 172.21.15.27:0/684525077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 msgr2=0x55802abd0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd0560 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9b78000c00 tx=0x7f9b78009cb0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 -- 172.21.15.27:0/684525077 shutdown_connections 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55802a7e2f60 0x55802abde290 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a824390 0x55802abd7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802abd0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b657fa640 0 client.18492 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/684525077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd0560 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 -- 172.21.15.27:0/684525077 >> 172.21.15.27:0/684525077 conn(0x55802a8184e0 msgr2=0x55802abc1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 -- 172.21.15.27:0/684525077 shutdown_connections 2024-04-25T13:56:48.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 1 -- 172.21.15.27:0/684525077 wait complete. 2024-04-25T13:56:48.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.920+0000 7f9b8ec58740 20 client.18492 trim_cache size 0 max 16384 2024-04-25T13:56:48.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b8ec58740 1 Processor -- start 2024-04-25T13:56:48.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b8ec58740 1 -- start start 2024-04-25T13:56:48.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7e2f60 0x55802abd0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7fb4d0 0x55802abd7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abd0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801a9150 con 0x55802a8036b0 2024-04-25T13:56:48.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801ab9d0 con 0x55802a7e2f60 2024-04-25T13:56:48.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801a97c0 con 0x55802a7fb4d0 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abd0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abd0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52644/0 (socket says 172.21.15.27:52644) 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 -- 172.21.15.27:0/1147391379 learned_addr learned my addr 172.21.15.27:0/1147391379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b85b47640 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7e2f60 0x55802abd0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 -- 172.21.15.27:0/1147391379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7fb4d0 msgr2=0x55802abd7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7fb4d0 0x55802abd7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 -- 172.21.15.27:0/1147391379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7e2f60 msgr2=0x55802abd0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7e2f60 0x55802abd0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 -- 172.21.15.27:0/1147391379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55802abc2bb0 con 0x55802a8036b0 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b85b47640 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7e2f60 0x55802abd0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.934+0000 7f9b86348640 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abd0670 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9b70035000 tx=0x7f9b70044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114b20d5f1855340 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b677fe640 1 -- 172.21.15.27:0/1147391379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b70004040 con 0x55802a8036b0 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b677fe640 1 -- 172.21.15.27:0/1147391379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b70036050 con 0x55802a8036b0 2024-04-25T13:56:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b677fe640 1 -- 172.21.15.27:0/1147391379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7002fd10 con 0x55802a8036b0 2024-04-25T13:56:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 -- 172.21.15.27:0/1147391379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 msgr2=0x55802abd0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abd0670 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9b70035000 tx=0x7f9b70044000 comp rx=0 tx=0).stop 2024-04-25T13:56:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 -- 172.21.15.27:0/1147391379 shutdown_connections 2024-04-25T13:56:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a8036b0 0x55802abd0670 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7fb4d0 0x55802abd7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/1147391379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a7e2f60 0x55802abd0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 -- 172.21.15.27:0/1147391379 >> 172.21.15.27:0/1147391379 conn(0x55802a8184e0 msgr2=0x55802abcb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 -- 172.21.15.27:0/1147391379 shutdown_connections 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.935+0000 7f9b8ec58740 1 -- 172.21.15.27:0/1147391379 wait complete. 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 1 -- start start 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 1 Processor -- start 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7e2f60 0x55802abd47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802ac445f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801a9e90 con 0x55802a7fb4d0 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801ab300 con 0x55802a8036b0 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801aa560 con 0x55802a7e2f60 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b8ec58740 10 client.0 _ll_register_callbacks cb 0x7f9b780e1d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b85346640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b85346640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52658/0 (socket says 172.21.15.27:52658) 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.936+0000 7f9b85346640 1 -- 172.21.15.27:0/2221838065 learned_addr learned my addr 172.21.15.27:0/2221838065 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b86348640 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802ac445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b85346640 1 -- 172.21.15.27:0/2221838065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7e2f60 msgr2=0x55802abd47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b85b47640 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7e2f60 0x55802abd47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b85346640 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7e2f60 0x55802abd47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b85346640 1 -- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 msgr2=0x55802ac445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b85346640 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802ac445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b85346640 1 -- 172.21.15.27:0/2221838065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55802abd5340 con 0x55802a7fb4d0 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b86348640 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802ac445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b85346640 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd4d30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9b74004620 tx=0x7f9b740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d940a59ae531bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b657fa640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b657fa640 1 -- 172.21.15.27:0/2221838065 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74003fe0 con 0x55802a7fb4d0 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b657fa640 1 -- 172.21.15.27:0/2221838065 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9b7402a1d0 con 0x55802a7fb4d0 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2221838065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b70036050 con 0x55802a7fb4d0 2024-04-25T13:56:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b657fa640 1 -- 172.21.15.27:0/2221838065 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9b74004180 con 0x55802a7fb4d0 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b8ec58740 10 client.18498 fetch_fsmap learned FSMap version 19 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2221838065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) -- 0x55802ac45b20 con 0x55802a7fb4d0 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.937+0000 7f9b657fa640 1 -- 172.21.15.27:0/2221838065 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b70036050 con 0x55802a7fb4d0 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b657fa640 1 -- 172.21.15.27:0/2221838065 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 19) ==== 8322+0+0 (secure 0 0 0) 0x7f9b74034a00 con 0x55802a7fb4d0 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b8ec58740 10 client.18498 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b8ec58740 10 client.18498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b8ec58740 20 client.18498 populate_metadata read hostname 'smithi027' 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55802a824390 0x55802ac48400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b8ec58740 4 client.18498 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2221838065 --> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55802ab32800 con 0x55802a824390 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b85b47640 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55802a824390 0x55802ac48400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b85b47640 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55802a824390 0x55802ac48400 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f70272c9a95084 server_cookie=7e73e7f65d35fab5 in_seq=0 out_seq=0 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.938+0000 7f9b657fa640 10 client.18498 ms_handle_connect on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b657fa640 1 -- 172.21.15.27:0/2221838065 <== mds.0 v2:172.21.15.45:6838/3661968964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55802ab32800 con 0x55802a824390 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b657fa640 10 client.18498 handle_command_reply: tid=0 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 client.18498 shutdown 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2221838065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 msgr2=0x55802abd4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd4d30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9b74004620 tx=0x7f9b740047c0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2221838065 shutdown_connections 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:6838/3661968964,v1:172.21.15.45:6839/3661968964] conn(0x55802a824390 0x55802ac48400 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55802a8036b0 0x55802ac445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55802a7fb4d0 0x55802abd4d30 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b657fa640 0 client.18498 ms_handle_reset on v2:172.21.15.45:6838/3661968964 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 --2- 172.21.15.27:0/2221838065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55802a7e2f60 0x55802abd47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2221838065 >> 172.21.15.27:0/2221838065 conn(0x55802a8184e0 msgr2=0x55802abd0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2221838065 shutdown_connections 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 1 -- 172.21.15.27:0/2221838065 wait complete. 2024-04-25T13:56:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b8ec58740 20 client.18498 trim_cache size 0 max 16384 2024-04-25T13:56:48.945 INFO:teuthology.orchestra.run.smithi027.stdout:[] 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b87dab640 1 -- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b680afaf0 msgr2=0x7f9b680b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b87dab640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b680afaf0 0x7f9b680b2000 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9b700094e0 tx=0x7f9b70009470 comp rx=0 tx=0).stop 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b87dab640 1 -- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80103af0 msgr2=0x7f9b8019a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.939+0000 7f9b87dab640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80103af0 0x7f9b8019a190 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800d740 tx=0x7f9b7800dc00 comp rx=0 tx=0).stop 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.940+0000 7f9b87dab640 1 -- 172.21.15.27:0/4125936107 shutdown_connections 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.940+0000 7f9b87dab640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9b680afaf0 0x7f9b680b2000 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.940+0000 7f9b87dab640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b80109a30 0x7f9b801a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.940+0000 7f9b87dab640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b80104d30 0x7f9b8019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.940+0000 7f9b87dab640 1 --2- 172.21.15.27:0/4125936107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b80103af0 0x7f9b8019a190 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.940+0000 7f9b87dab640 1 -- 172.21.15.27:0/4125936107 >> 172.21.15.27:0/4125936107 conn(0x7f9b800fece0 msgr2=0x7f9b80105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.940+0000 7f9b87dab640 1 -- 172.21.15.27:0/4125936107 shutdown_connections 2024-04-25T13:56:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:48.940+0000 7f9b87dab640 1 -- 172.21.15.27:0/4125936107 wait complete. 2024-04-25T13:56:49.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:49 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/445957375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:49.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:49 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/445957375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:49.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:49 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/445957375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:49.951 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-25T13:56:49.952 INFO:tasks.fwd_scrub:done joining 2024-04-25T13:56:49.952 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T13:56:49.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:56:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 -- 172.21.15.27:0/2404750228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174103c40 msgr2=0x7fa174104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 --2- 172.21.15.27:0/2404750228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174103c40 0x7fa174104050 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa15c009900 tx=0x7fa15c02a410 comp rx=0 tx=0).stop 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 -- 172.21.15.27:0/2404750228 shutdown_connections 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 --2- 172.21.15.27:0/2404750228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa174108940 0x7fa17410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 --2- 172.21.15.27:0/2404750228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174103c40 0x7fa174104050 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 --2- 172.21.15.27:0/2404750228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa174102a00 0x7fa174103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 -- 172.21.15.27:0/2404750228 >> 172.21.15.27:0/2404750228 conn(0x7fa1740fdbf0 msgr2=0x7fa1741000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 -- 172.21.15.27:0/2404750228 shutdown_connections 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 -- 172.21.15.27:0/2404750228 wait complete. 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 Processor -- start 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.092+0000 7fa17926e640 1 -- start start 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa17926e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa174102a00 0x7fa174075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa17926e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa174103c40 0x7fa17419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa17926e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174108940 0x7fa1741a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa17926e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa17410e530 con 0x7fa174108940 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa17926e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa17410e3b0 con 0x7fa174103c40 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa17926e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa17410e6b0 con 0x7fa174102a00 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174108940 0x7fa1741a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174108940 0x7fa1741a14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52670/0 (socket says 172.21.15.27:52670) 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 -- 172.21.15.27:0/532672811 learned_addr learned my addr 172.21.15.27:0/532672811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa16bfff640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa174103c40 0x7fa17419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa172ffd640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa174102a00 0x7fa174075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 -- 172.21.15.27:0/532672811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa174102a00 msgr2=0x7fa174075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa174102a00 0x7fa174075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 -- 172.21.15.27:0/532672811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa174103c40 msgr2=0x7fa17419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa174103c40 0x7fa17419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 -- 172.21.15.27:0/532672811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1741a1cf0 con 0x7fa174108940 2024-04-25T13:56:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa172ffd640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa174102a00 0x7fa174075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.093+0000 7fa1737fe640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174108940 0x7fa1741a14d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00eaa0 tx=0x7fa16c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.094+0000 7fa170ff9640 1 -- 172.21.15.27:0/532672811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c00cdb0 con 0x7fa174108940 2024-04-25T13:56:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.094+0000 7fa170ff9640 1 -- 172.21.15.27:0/532672811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa16c005020 con 0x7fa174108940 2024-04-25T13:56:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.094+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1741a1fe0 con 0x7fa174108940 2024-04-25T13:56:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.094+0000 7fa170ff9640 1 -- 172.21.15.27:0/532672811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c0116d0 con 0x7fa174108940 2024-04-25T13:56:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.094+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1741a2440 con 0x7fa174108940 2024-04-25T13:56:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.095+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa17406a190 con 0x7fa174108940 2024-04-25T13:56:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.097+0000 7fa170ff9640 1 -- 172.21.15.27:0/532672811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa16c0051d0 con 0x7fa174108940 2024-04-25T13:56:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.098+0000 7fa170ff9640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1600afaf0 0x7fa1600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.098+0000 7fa172ffd640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1600afaf0 0x7fa1600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.098+0000 7fa170ff9640 1 -- 172.21.15.27:0/532672811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa16c0df190 con 0x7fa174108940 2024-04-25T13:56:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.098+0000 7fa172ffd640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1600afaf0 0x7fa1600b2000 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa174103600 tx=0x7fa164005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:50.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.102+0000 7fa170ff9640 1 -- 172.21.15.27:0/532672811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa16c016030 con 0x7fa174108940 2024-04-25T13:56:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.290+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa17402dc10 con 0x7fa174108940 2024-04-25T13:56:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.291+0000 7fa170ff9640 1 -- 172.21.15.27:0/532672811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7fa16c0a7d40 con 0x7fa174108940 2024-04-25T13:56:50.291 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:50.291 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:56:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1600afaf0 msgr2=0x7fa1600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1600afaf0 0x7fa1600b2000 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa174103600 tx=0x7fa164005f90 comp rx=0 tx=0).stop 2024-04-25T13:56:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174108940 msgr2=0x7fa1741a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174108940 0x7fa1741a14d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00eaa0 tx=0x7fa16c00ef60 comp rx=0 tx=0).stop 2024-04-25T13:56:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 shutdown_connections 2024-04-25T13:56:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa1600afaf0 0x7fa1600b2000 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa174108940 0x7fa1741a14d0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa174103c40 0x7fa17419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 --2- 172.21.15.27:0/532672811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa174102a00 0x7fa174075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 >> 172.21.15.27:0/532672811 conn(0x7fa1740fdbf0 msgr2=0x7fa174104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 shutdown_connections 2024-04-25T13:56:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.294+0000 7fa17926e640 1 -- 172.21.15.27:0/532672811 wait complete. 2024-04-25T13:56:50.304 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:50 smithi027 ceph-mon[29140]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 32 KiB/s wr, 6 op/s 2024-04-25T13:56:50.305 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:56:50.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:56:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 -- 172.21.15.27:0/2558296380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358108930 msgr2=0x7fb35810ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/2558296380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358108930 0x7fb35810ad50 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb350009ca0 tx=0x7fb35002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 -- 172.21.15.27:0/2558296380 shutdown_connections 2024-04-25T13:56:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/2558296380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358108930 0x7fb35810ad50 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/2558296380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb358103c30 0x7fb358104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/2558296380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3581029f0 0x7fb3581036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 -- 172.21.15.27:0/2558296380 >> 172.21.15.27:0/2558296380 conn(0x7fb3580fdbc0 msgr2=0x7fb358100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:50.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 -- 172.21.15.27:0/2558296380 shutdown_connections 2024-04-25T13:56:50.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.421+0000 7fb35d0ca640 1 -- 172.21.15.27:0/2558296380 wait complete. 2024-04-25T13:56:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.422+0000 7fb35d0ca640 1 Processor -- start 2024-04-25T13:56:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.422+0000 7fb35d0ca640 1 -- start start 2024-04-25T13:56:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.422+0000 7fb35d0ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3581029f0 0x7fb35819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.422+0000 7fb35d0ca640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358103c30 0x7fb35819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.422+0000 7fb35d0ca640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb358108930 0x7fb3581a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.422+0000 7fb35d0ca640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810e500 con 0x7fb358103c30 2024-04-25T13:56:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.422+0000 7fb35d0ca640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810e380 con 0x7fb3581029f0 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.422+0000 7fb35d0ca640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810e680 con 0x7fb358108930 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358103c30 0x7fb35819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358103c30 0x7fb35819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52690/0 (socket says 172.21.15.27:52690) 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 -- 172.21.15.27:0/458425908 learned_addr learned my addr 172.21.15.27:0/458425908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb356d9d640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3581029f0 0x7fb35819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35759e640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb358108930 0x7fb3581a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 -- 172.21.15.27:0/458425908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb358108930 msgr2=0x7fb3581a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb358108930 0x7fb3581a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 -- 172.21.15.27:0/458425908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3581029f0 msgr2=0x7fb35819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3581029f0 0x7fb35819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 -- 172.21.15.27:0/458425908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3581a1ce0 con 0x7fb358103c30 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35659c640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358103c30 0x7fb35819a6e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c890 tx=0x7fb34c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb33ffff640 1 -- 172.21.15.27:0/458425908 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004220 con 0x7fb358103c30 2024-04-25T13:56:50.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3581a1fd0 con 0x7fb358103c30 2024-04-25T13:56:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb33ffff640 1 -- 172.21.15.27:0/458425908 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb34c0043c0 con 0x7fb358103c30 2024-04-25T13:56:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3581a2490 con 0x7fb358103c30 2024-04-25T13:56:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.423+0000 7fb33ffff640 1 -- 172.21.15.27:0/458425908 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c01d060 con 0x7fb358103c30 2024-04-25T13:56:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.425+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3581a2750 con 0x7fb358103c30 2024-04-25T13:56:50.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.426+0000 7fb33ffff640 1 -- 172.21.15.27:0/458425908 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb34c0079b0 con 0x7fb358103c30 2024-04-25T13:56:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.427+0000 7fb33ffff640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3400afaf0 0x7fb3400b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.427+0000 7fb33ffff640 1 -- 172.21.15.27:0/458425908 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fb34c0dec10 con 0x7fb358103c30 2024-04-25T13:56:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.427+0000 7fb356d9d640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3400afaf0 0x7fb3400b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.427+0000 7fb356d9d640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3400afaf0 0x7fb3400b2000 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb348005fd0 tx=0x7fb34800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.431+0000 7fb33ffff640 1 -- 172.21.15.27:0/458425908 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb34c014030 con 0x7fb358103c30 2024-04-25T13:56:50.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:50 smithi122 ceph-mon[32843]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 32 KiB/s wr, 6 op/s 2024-04-25T13:56:50.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:50 smithi045 ceph-mon[32731]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 32 KiB/s wr, 6 op/s 2024-04-25T13:56:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.617+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb3581a2920 con 0x7fb358103c30 2024-04-25T13:56:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.617+0000 7fb33ffff640 1 -- 172.21.15.27:0/458425908 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7fb34c0a71d0 con 0x7fb358103c30 2024-04-25T13:56:50.617 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3400afaf0 msgr2=0x7fb3400b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3400afaf0 0x7fb3400b2000 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb348005fd0 tx=0x7fb34800b410 comp rx=0 tx=0).stop 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358103c30 msgr2=0x7fb35819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358103c30 0x7fb35819a6e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c890 tx=0x7fb34c00cd50 comp rx=0 tx=0).stop 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 shutdown_connections 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fb3400afaf0 0x7fb3400b2000 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb358108930 0x7fb3581a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb358103c30 0x7fb35819a6e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 --2- 172.21.15.27:0/458425908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3581029f0 0x7fb35819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 >> 172.21.15.27:0/458425908 conn(0x7fb3580fdbc0 msgr2=0x7fb358109d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:50.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 shutdown_connections 2024-04-25T13:56:50.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.619+0000 7fb35d0ca640 1 -- 172.21.15.27:0/458425908 wait complete. 2024-04-25T13:56:50.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:56:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.746+0000 7ff8ab572640 1 -- 172.21.15.27:0/4265988175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4103c30 msgr2=0x7ff8a4104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.746+0000 7ff8ab572640 1 --2- 172.21.15.27:0/4265988175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4103c30 0x7ff8a4104040 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff894009900 tx=0x7ff89402a410 comp rx=0 tx=0).stop 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.746+0000 7ff8ab572640 1 -- 172.21.15.27:0/4265988175 shutdown_connections 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.746+0000 7ff8ab572640 1 --2- 172.21.15.27:0/4265988175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8a4108930 0x7ff8a410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.746+0000 7ff8ab572640 1 --2- 172.21.15.27:0/4265988175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4103c30 0x7ff8a4104040 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.746+0000 7ff8ab572640 1 --2- 172.21.15.27:0/4265988175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a41029f0 0x7ff8a41036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.746+0000 7ff8ab572640 1 -- 172.21.15.27:0/4265988175 >> 172.21.15.27:0/4265988175 conn(0x7ff8a40fdbc0 msgr2=0x7ff8a4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 -- 172.21.15.27:0/4265988175 shutdown_connections 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 -- 172.21.15.27:0/4265988175 wait complete. 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 Processor -- start 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 -- start start 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8a41029f0 0x7ff8a41990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4103c30 0x7ff8a4199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4108930 0x7ff8a41a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a410e520 con 0x7ff8a4108930 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a410e3a0 con 0x7ff8a4103c30 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.747+0000 7ff8ab572640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a410e6a0 con 0x7ff8a41029f0 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4108930 0x7ff8a41a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4108930 0x7ff8a41a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52710/0 (socket says 172.21.15.27:52710) 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 -- 172.21.15.27:0/299398415 learned_addr learned my addr 172.21.15.27:0/299398415 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a930e640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8a41029f0 0x7ff8a41990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a8b0d640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4103c30 0x7ff8a4199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 -- 172.21.15.27:0/299398415 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8a41029f0 msgr2=0x7ff8a41990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8a41029f0 0x7ff8a41990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 -- 172.21.15.27:0/299398415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4103c30 msgr2=0x7ff8a4199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4103c30 0x7ff8a4199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 -- 172.21.15.27:0/299398415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a41a0c30 con 0x7ff8a4108930 2024-04-25T13:56:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8a9b0f640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4108930 0x7ff8a41a05c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00eb10 tx=0x7ff89c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff89a7fc640 1 -- 172.21.15.27:0/299398415 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c004b20 con 0x7ff8a4108930 2024-04-25T13:56:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff89a7fc640 1 -- 172.21.15.27:0/299398415 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff89c004cc0 con 0x7ff8a4108930 2024-04-25T13:56:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a41a0f20 con 0x7ff8a4108930 2024-04-25T13:56:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff89a7fc640 1 -- 172.21.15.27:0/299398415 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c010670 con 0x7ff8a4108930 2024-04-25T13:56:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.748+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a41a1380 con 0x7ff8a4108930 2024-04-25T13:56:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.749+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a406a160 con 0x7ff8a4108930 2024-04-25T13:56:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.751+0000 7ff89a7fc640 1 -- 172.21.15.27:0/299398415 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff89c005000 con 0x7ff8a4108930 2024-04-25T13:56:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.751+0000 7ff89a7fc640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8900afaf0 0x7ff8900b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.752+0000 7ff8a930e640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8900afaf0 0x7ff8900b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.752+0000 7ff89a7fc640 1 -- 172.21.15.27:0/299398415 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff89c0df3e0 con 0x7ff8a4108930 2024-04-25T13:56:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.752+0000 7ff8a930e640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8900afaf0 0x7ff8900b2000 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff8a41035f0 tx=0x7ff8a000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.755+0000 7ff89a7fc640 1 -- 172.21.15.27:0/299398415 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff89c053ff0 con 0x7ff8a4108930 2024-04-25T13:56:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.945+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff8a402dbd0 con 0x7ff8a4108930 2024-04-25T13:56:50.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.946+0000 7ff89a7fc640 1 -- 172.21.15.27:0/299398415 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7ff89c0a7ff0 con 0x7ff8a4108930 2024-04-25T13:56:50.946 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8900afaf0 msgr2=0x7ff8900b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8900afaf0 0x7ff8900b2000 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff8a41035f0 tx=0x7ff8a000b390 comp rx=0 tx=0).stop 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4108930 msgr2=0x7ff8a41a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4108930 0x7ff8a41a05c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00eb10 tx=0x7ff89c00c2d0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 shutdown_connections 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8900afaf0 0x7ff8900b2000 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8a4108930 0x7ff8a41a05c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4103c30 0x7ff8a4199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 --2- 172.21.15.27:0/299398415 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8a41029f0 0x7ff8a41990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:50.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 >> 172.21.15.27:0/299398415 conn(0x7ff8a40fdbc0 msgr2=0x7ff8a4104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:50.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 shutdown_connections 2024-04-25T13:56:50.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:50.948+0000 7ff8ab572640 1 -- 172.21.15.27:0/299398415 wait complete. 2024-04-25T13:56:50.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:56:51.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.077+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3003234002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8103af0 msgr2=0x7fbaf81047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.077+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3003234002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8103af0 0x7fbaf81047f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009900 tx=0x7fbae802a410 comp rx=0 tx=0).stop 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3003234002 shutdown_connections 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3003234002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf8109a30 0x7fbaf810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3003234002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaf8104d30 0x7fbaf8105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3003234002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8103af0 0x7fbaf81047f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3003234002 >> 172.21.15.27:0/3003234002 conn(0x7fbaf80fece0 msgr2=0x7fbaf8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3003234002 shutdown_connections 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3003234002 wait complete. 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 Processor -- start 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 -- start start 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaf8103af0 0x7fbaf819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8104d30 0x7fbaf819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.078+0000 7fbafd7c0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf8109a30 0x7fbaf81a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbafd7c0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf810f5f0 con 0x7fbaf8104d30 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbafd7c0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf810f470 con 0x7fbaf8109a30 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbafd7c0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf810f770 con 0x7fbaf8103af0 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8104d30 0x7fbaf819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8104d30 0x7fbaf819a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52736/0 (socket says 172.21.15.27:52736) 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 -- 172.21.15.27:0/3620373493 learned_addr learned my addr 172.21.15.27:0/3620373493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaf6ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaf8103af0 0x7fbaf819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaf77fe640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf8109a30 0x7fbaf81a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 -- 172.21.15.27:0/3620373493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaf8103af0 msgr2=0x7fbaf819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaf8103af0 0x7fbaf819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 -- 172.21.15.27:0/3620373493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf8109a30 msgr2=0x7fbaf81a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf8109a30 0x7fbaf81a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 -- 172.21.15.27:0/3620373493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf81a1cf0 con 0x7fbaf8104d30 2024-04-25T13:56:51.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaf77fe640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf8109a30 0x7fbaf81a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:51.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaeffff640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8104d30 0x7fbaf819a7a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbae000e9f0 tx=0x7fbae000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:51.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaf4ff9640 1 -- 172.21.15.27:0/3620373493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae000cd80 con 0x7fbaf8104d30 2024-04-25T13:56:51.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaf4ff9640 1 -- 172.21.15.27:0/3620373493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae00054d0 con 0x7fbaf8104d30 2024-04-25T13:56:51.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbaf4ff9640 1 -- 172.21.15.27:0/3620373493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0010690 con 0x7fbaf8104d30 2024-04-25T13:56:51.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf81a1fe0 con 0x7fbaf8104d30 2024-04-25T13:56:51.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.079+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf81a23f0 con 0x7fbaf8104d30 2024-04-25T13:56:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.080+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf8108840 con 0x7fbaf8104d30 2024-04-25T13:56:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.082+0000 7fbaf4ff9640 1 -- 172.21.15.27:0/3620373493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbae0004e30 con 0x7fbaf8104d30 2024-04-25T13:56:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.083+0000 7fbaf4ff9640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbae40afaf0 0x7fbae40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.083+0000 7fbaf6ffd640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbae40afaf0 0x7fbae40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.083+0000 7fbaf4ff9640 1 -- 172.21.15.27:0/3620373493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbae00df250 con 0x7fbaf8104d30 2024-04-25T13:56:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.084+0000 7fbaf6ffd640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbae40afaf0 0x7fbae40b2000 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbae8005ab0 tx=0x7fbae8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:51.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.087+0000 7fbaf4ff9640 1 -- 172.21.15.27:0/3620373493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbae0053ff0 con 0x7fbaf8104d30 2024-04-25T13:56:51.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.272+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbaf80637d0 con 0x7fbaf8104d30 2024-04-25T13:56:51.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.273+0000 7fbaf4ff9640 1 -- 172.21.15.27:0/3620373493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7fbae00a7e60 con 0x7fbaf8104d30 2024-04-25T13:56:51.273 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:56:51.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbae40afaf0 msgr2=0x7fbae40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:51.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbae40afaf0 0x7fbae40b2000 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbae8005ab0 tx=0x7fbae8031400 comp rx=0 tx=0).stop 2024-04-25T13:56:51.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8104d30 msgr2=0x7fbaf819a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:51.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8104d30 0x7fbaf819a7a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbae000e9f0 tx=0x7fbae000edf0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 shutdown_connections 2024-04-25T13:56:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbae40afaf0 0x7fbae40b2000 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf8109a30 0x7fbaf81a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbaf8104d30 0x7fbaf819a7a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 --2- 172.21.15.27:0/3620373493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbaf8103af0 0x7fbaf819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 >> 172.21.15.27:0/3620373493 conn(0x7fbaf80fece0 msgr2=0x7fbaf8105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 shutdown_connections 2024-04-25T13:56:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.275+0000 7fbafd7c0640 1 -- 172.21.15.27:0/3620373493 wait complete. 2024-04-25T13:56:51.285 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:51 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/532672811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:51.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:51 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/458425908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:56:51.286 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:51 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/299398415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:56:51.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 -- 172.21.15.27:0/1736327493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 msgr2=0x7f875410a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 --2- 172.21.15.27:0/1736327493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 0x7f875410a4c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f874c009ca0 tx=0x7f874c02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 -- 172.21.15.27:0/1736327493 shutdown_connections 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 --2- 172.21.15.27:0/1736327493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 0x7f875410a4c0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 --2- 172.21.15.27:0/1736327493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87541033a0 0x7f87541037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 --2- 172.21.15.27:0/1736327493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8754102160 0x7f8754102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 -- 172.21.15.27:0/1736327493 >> 172.21.15.27:0/1736327493 conn(0x7f87540fd330 msgr2=0x7f87540ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 -- 172.21.15.27:0/1736327493 shutdown_connections 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.399+0000 7f875b93c640 1 -- 172.21.15.27:0/1736327493 wait complete. 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f875b93c640 1 Processor -- start 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f875b93c640 1 -- start start 2024-04-25T13:56:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f875b93c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8754102160 0x7f87541990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f875b93c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87541033a0 0x7f87541995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f875b93c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 0x7f87541a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f875b93c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f875410dc80 con 0x7f87541080a0 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f875b93c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f875410db00 con 0x7f87541033a0 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f875b93c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f875410de00 con 0x7f8754102160 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8759ed9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 0x7f87541a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8759ed9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 0x7f87541a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52762/0 (socket says 172.21.15.27:52762) 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8759ed9640 1 -- 172.21.15.27:0/2734065923 learned_addr learned my addr 172.21.15.27:0/2734065923 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f87596d8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8754102160 0x7f87541990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8758ed7640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87541033a0 0x7f87541995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8759ed9640 1 -- 172.21.15.27:0/2734065923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8754102160 msgr2=0x7f87541990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8759ed9640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8754102160 0x7f87541990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8759ed9640 1 -- 172.21.15.27:0/2734065923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87541033a0 msgr2=0x7f87541995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8759ed9640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87541033a0 0x7f87541995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8759ed9640 1 -- 172.21.15.27:0/2734065923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87541a0bf0 con 0x7f87541080a0 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.400+0000 7f8758ed7640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87541033a0 0x7f87541995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.401+0000 7f8759ed9640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 0x7f87541a0580 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f874c0097b0 tx=0x7f874c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.401+0000 7f874a7fc640 1 -- 172.21.15.27:0/2734065923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c04a070 con 0x7f87541080a0 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.401+0000 7f874a7fc640 1 -- 172.21.15.27:0/2734065923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f874c0317c0 con 0x7f87541080a0 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.401+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87541094c0 con 0x7f87541080a0 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.401+0000 7f874a7fc640 1 -- 172.21.15.27:0/2734065923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c02fcc0 con 0x7f87541080a0 2024-04-25T13:56:51.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.401+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8754109a10 con 0x7f87541080a0 2024-04-25T13:56:51.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.402+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f875406a190 con 0x7f87541080a0 2024-04-25T13:56:51.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.403+0000 7f874a7fc640 1 -- 172.21.15.27:0/2734065923 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f874c04d070 con 0x7f87541080a0 2024-04-25T13:56:51.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.404+0000 7f874a7fc640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87400afaf0 0x7f87400b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:51.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.404+0000 7f87596d8640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87400afaf0 0x7f87400b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:51.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.404+0000 7f874a7fc640 1 -- 172.21.15.27:0/2734065923 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f874c1018e0 con 0x7f87541080a0 2024-04-25T13:56:51.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.405+0000 7f87596d8640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87400afaf0 0x7f87400b2000 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8754102d60 tx=0x7f874400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:51.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.408+0000 7f874a7fc640 1 -- 172.21.15.27:0/2734065923 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f874c081ff0 con 0x7f87541080a0 2024-04-25T13:56:51.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:51 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/532672811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:51.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:51 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/458425908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:56:51.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:51 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/299398415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:56:51.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:51 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/532672811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:51.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:51 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/458425908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:56:51.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:51 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/299398415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:56:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.647+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f8754107210 con 0x7f87541080a0 2024-04-25T13:56:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.656+0000 7f874a7fc640 1 -- 172.21.15.27:0/2734065923 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+803127 (secure 0 0 0) 0x7f874c0ca490 con 0x7f87541080a0 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 18, 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "active_gid": 14152, 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "active_name": "x", 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "active_addrs": { 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6800", 2024-04-25T13:56:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1716439151 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6801", 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1716439151 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: "active_addr": "172.21.15.27:6801/1716439151", 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: "active_change": "2024-04-25T13:31:30.235662+0000", 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-04-25T13:56:51.669 INFO:teuthology.orchestra.run.smithi027.stdout: "standbys": [ 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 24109, 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "z", 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_features": 4540701547738038271, 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T13:56:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T13:56:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T13:56:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T13:56:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T13:56:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T13:56:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T13:56:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.730 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.731 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T13:56:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T13:56:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T13:56:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.765 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.766 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.767 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.768 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.769 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.770 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.771 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.772 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.773 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.774 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.775 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.776 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.777 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T13:56:51.778 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.779 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T13:56:51.780 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T13:56:51.781 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T13:56:51.782 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T13:56:51.783 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.785 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.786 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.787 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.788 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.790 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.792 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.793 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T13:56:51.794 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.795 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.796 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.797 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T13:56:51.798 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T13:56:51.799 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "[]", 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "{}", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T13:56:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T13:56:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T13:56:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T13:56:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T13:56:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T13:56:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T13:56:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T13:56:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T13:56:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T13:56:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T13:56:51.915 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.916 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T13:56:51.917 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.918 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.920 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.921 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.922 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.923 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.924 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.925 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.926 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.927 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.928 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.929 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.930 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.931 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.932 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.933 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.934 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.936 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.938 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.940 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.941 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.942 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T13:56:51.943 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.944 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.945 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.946 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.949 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.950 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.952 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.953 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.955 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.956 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.958 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.959 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.960 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.961 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.962 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:51.963 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.964 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.965 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T13:56:51.966 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T13:56:51.967 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.968 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.969 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.971 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.972 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.974 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.976 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.977 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.978 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.979 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.980 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.981 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.982 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.983 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.983 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.983 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.983 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.983 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.983 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.983 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.985 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:51.985 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:51.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.985 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.986 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.987 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:51.988 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.989 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.991 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T13:56:51.992 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T13:56:51.993 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:51.994 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.996 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:51.997 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:51.998 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:51.999 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.000 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.001 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.002 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.003 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.004 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T13:56:52.005 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.006 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.007 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.008 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.009 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.010 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.011 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.012 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.013 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.014 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.015 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.016 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.017 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.018 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.021 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.022 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.023 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.024 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.025 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.026 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.027 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.028 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.029 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.030 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.031 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.032 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.033 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T13:56:52.034 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.035 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.036 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.037 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.038 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.039 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.040 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.041 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.042 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.043 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.044 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.045 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.046 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T13:56:52.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.048 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.049 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.050 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.051 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.052 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.053 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.054 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.056 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.057 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.058 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.059 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.060 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.061 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.062 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.063 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.064 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.065 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.066 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.068 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.069 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.070 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.071 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.072 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.075 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.076 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.077 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.078 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.079 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.080 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.081 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.082 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.083 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.084 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.085 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.086 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.087 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.089 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.090 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.092 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T13:56:52.093 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T13:56:52.094 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.095 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.096 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.097 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.098 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.099 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.100 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.101 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.102 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.103 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.104 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.105 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.106 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.107 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.108 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 24116, 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "y", 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_features": 4540701547738038271, 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T13:56:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.138 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.139 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T13:56:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T13:56:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T13:56:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T13:56:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T13:56:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T13:56:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T13:56:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T13:56:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T13:56:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T13:56:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T13:56:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T13:56:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T13:56:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T13:56:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T13:56:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T13:56:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T13:56:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T13:56:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T13:56:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.232 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.233 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.234 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.235 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T13:56:52.236 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.237 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T13:56:52.238 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T13:56:52.239 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.240 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.241 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.242 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.243 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.244 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.245 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T13:56:52.246 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T13:56:52.247 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.248 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.249 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.250 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:52.251 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.252 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.253 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.254 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.255 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.256 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.257 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.258 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "[]", 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.259 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "{}", 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.260 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.261 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.262 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.263 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.264 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.265 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.266 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.268 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.269 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.270 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.271 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.272 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.273 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.275 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.276 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.277 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.278 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.279 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.280 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.281 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.284 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T13:56:52.285 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T13:56:52.286 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.287 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.288 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.290 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.291 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T13:56:52.292 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.293 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.294 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.295 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.296 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.297 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.298 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.299 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.300 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.301 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.302 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.303 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.304 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.305 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.306 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.307 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T13:56:52.308 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.309 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.312 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.313 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.314 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T13:56:52.315 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.316 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.317 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.318 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.319 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.320 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T13:56:52.321 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.322 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.323 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.324 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.326 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.327 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.328 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.329 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.330 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.331 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.332 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.336 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T13:56:52.337 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T13:56:52.338 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T13:56:52.339 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.340 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.341 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.342 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.343 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.345 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.346 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.347 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.348 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.349 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.351 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.352 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.353 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.354 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.356 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.357 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.358 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.359 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.360 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.361 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.362 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.363 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.364 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.365 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.366 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.367 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.368 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.369 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.370 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T13:56:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.374 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.375 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.376 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.377 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.378 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.379 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.380 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.381 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.382 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.383 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.384 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.385 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T13:56:52.386 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.388 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.390 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.391 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.392 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.393 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.394 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.396 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.397 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.398 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.399 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.400 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.401 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.402 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.403 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.404 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.406 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.407 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.411 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.412 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.413 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.414 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.415 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.416 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.418 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.419 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.420 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.421 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.423 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.424 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.425 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.426 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.427 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.428 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.429 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.430 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.431 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.432 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.433 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.434 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.435 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.436 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.438 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.439 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.440 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.441 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.442 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.443 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.444 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.445 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.446 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T13:56:52.447 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T13:56:52.448 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.449 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.450 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.451 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.452 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.453 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.454 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.455 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.458 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.460 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.461 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.462 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.463 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.464 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.465 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.466 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.467 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.468 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.469 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.470 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.471 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.472 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.473 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.474 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.475 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.476 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.477 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.478 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.479 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.481 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.482 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.483 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.485 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.486 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.487 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T13:56:52.488 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T13:56:52.489 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T13:56:52.490 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.491 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.492 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.493 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.494 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.495 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.496 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.498 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.499 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.500 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.501 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.502 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.503 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.504 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.505 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.506 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.507 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.508 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.509 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.510 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.511 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.514 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.515 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.516 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.517 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.518 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.519 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T13:56:52.520 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.521 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.522 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.523 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.524 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.526 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.529 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.531 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T13:56:52.532 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.533 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.534 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.535 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.536 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.537 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.538 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.541 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.542 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.543 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.544 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T13:56:52.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T13:56:52.546 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.547 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.548 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.549 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.550 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.551 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.552 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.553 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.554 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.555 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.556 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.557 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.558 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.559 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T13:56:52.560 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T13:56:52.561 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T13:56:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T13:56:52.563 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T13:56:52.565 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.566 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.567 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.568 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.569 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.571 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T13:56:52.574 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T13:56:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "dashboard", 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "available_modules": [ 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "alerts", 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_destination": { 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_destination", 2024-04-25T13:56:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email address to send alerts to", 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_from_name": { 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_from_name", 2024-04-25T13:56:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "Ceph", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Email From: name", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_host": { 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_host", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP server", 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_password": { 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_password", 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Password to authenticate with", 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_port": { 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_port", 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "465", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP port", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_sender": { 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_sender", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SMTP envelope sender", 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_ssl": { 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_ssl", 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "smtp_user": { 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "smtp_user", 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "User to authenticate as", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "balancer", 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "active": { 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "active", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_time": { 2024-04-25T13:56:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_time", 2024-04-25T13:56:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0000", 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "begin_weekday": { 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "begin_weekday", 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_max_iterations": { 2024-04-25T13:56:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_max_iterations", 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "25", 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "250", 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_metrics": { 2024-04-25T13:56:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_metrics", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_compat_step": { 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crush_compat_step", 2024-04-25T13:56:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.5", 2024-04-25T13:56:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0.001", 2024-04-25T13:56:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "0.999", 2024-04-25T13:56:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "aggressiveness of optimization", 2024-04-25T13:56:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T13:56:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "end_time": { 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_time", 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2359", 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T13:56:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "end_weekday": { 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "end_weekday", 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "0", 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "6", 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T13:56:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min_score": { 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_score", 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "upmap", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-compat", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "none", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "read", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap", 2024-04-25T13:56:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap-read" 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Balancer mode", 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ids": { 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_ids", 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_deviation": { 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_deviation", 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "upmap_max_optimizations": { 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "upmap_max_optimizations", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T13:56:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm", 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_down_multiplier": { 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_down_multiplier", 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_refresh_rate": { 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_refresh_rate", 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "20", 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "agent_starting_port": { 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "agent_starting_port", 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4721", 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_ptrace": { 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_ptrace", 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_interval": { 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_interval", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "autotune_memory_target_ratio": { 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0.7", 2024-04-25T13:56:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_log_destination": { 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephadm_log_destination", 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "file", 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "file,syslog", 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "syslog" 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "cgroups_split": { 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cgroups_split", 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "config_checks_enabled": { 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_checks_enabled", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "config_dashboard": { 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_dashboard", 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_alertmanager": { 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_alertmanager", 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_base": { 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_base", 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Container image name, without the tag", 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_elasticsearch": { 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_elasticsearch", 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "elasticsearch container image", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_grafana": { 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_grafana", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T13:56:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_haproxy": { 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_haproxy", 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T13:56:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "HAproxy container image", 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_agent": { 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_agent", 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger agent container image", 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_collector": { 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_collector", 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger collector container image", 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_jaeger_query": { 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_jaeger_query", 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Jaeger query container image", 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_keepalived": { 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_keepalived", 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Keepalived container image", 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_loki": { 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_loki", 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Loki container image", 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_node_exporter": { 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_node_exporter", 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_nvmeof": { 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_nvmeof", 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Nvme-of container image", 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_prometheus": { 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_prometheus", 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Prometheus container image", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_promtail": { 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_promtail", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Promtail container image", 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_samba": { 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_samba", 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Samba/SMB container image", 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image_snmp_gateway": { 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_image_snmp_gateway", 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "SNMP Gateway container image", 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "container_init": { 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "container_init", 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "daemon_cache_timeout": { 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "daemon_cache_timeout", 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "default_cephadm_command_timeout": { 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "900", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "default_registry": { 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "default_registry", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "docker.io", 2024-04-25T13:56:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "device_cache_timeout": { 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_cache_timeout", 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1800", 2024-04-25T13:56:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache device inventory", 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "device_enhanced_scan": { 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_enhanced_scan", 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "facts_cache_timeout": { 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "facts_cache_timeout", 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "seconds to cache host facts data", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "grafana_dashboards_path": { 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "grafana_dashboards_path", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "host_check_interval": { 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "host_check_interval", 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to perform a host check", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "hw_monitoring": { 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hw_monitoring", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "inventory_list_all": { 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "inventory_list_all", 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "log_refresh_metadata": { 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_refresh_metadata", 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T13:56:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "*", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "max_count_per_host": { 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_count_per_host", 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of daemons per service per host", 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd_draining_count": { 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_osd_draining_count", 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "migration_current": { 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "migration_current", 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "internal - do not modify", 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": { 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mode", 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "root", 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm-package", 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "root" 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "oob_default_addr": { 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "oob_default_addr", 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "169.254.1.1", 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "prometheus_alerts_path": { 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus_alerts_path", 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_insecure": { 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_insecure", 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_password": { 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_password", 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_url": { 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_url", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T13:56:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "registry_username": { 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "registry_username", 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T13:56:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "secure_monitoring_stack": { 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "secure_monitoring_stack", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "service_discovery_port": { 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "service_discovery_port", 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8765", 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "cephadm service discovery port", 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "ssh_config_file": { 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssh_config_file", 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "use_agent": { 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_agent", 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "use_repo_digest": { 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "use_repo_digest", 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_failed_host_check": { 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_failed_host_check", 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_daemons": { 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_daemons", 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T13:56:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_on_stray_hosts": { 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_on_stray_hosts", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crash", 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "retain_interval": { 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retain_interval", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "31536000", 2024-04-25T13:56:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_recent_interval": { 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_recent_interval", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1209600", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T13:56:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dashboard", 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_ENABLED": { 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.697 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.698 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T13:56:52.699 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.700 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.702 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_URL": { 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_URL", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.703 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.704 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.705 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.706 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.707 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.708 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "[]", 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.709 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "{}", 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.710 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.711 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.712 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.713 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.714 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.715 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.716 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.717 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.718 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.719 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.720 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8", 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T13:56:52.723 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "45", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T13:56:52.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.725 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "admin", 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.726 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.727 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.728 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.729 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.730 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.731 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "cross_origin_url": { 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cross_origin_url", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.732 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "crt_file": { 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "crt_file", 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.733 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "debug": { 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "debug", 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.734 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Enable/disable debug options", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "jwt_token_ttl": { 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "jwt_token_ttl", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "28800", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.736 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.737 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.738 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.739 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "motd": { 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "motd", 2024-04-25T13:56:52.741 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "The message of the day", 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect_resolve_ip_addr": { 2024-04-25T13:56:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.743 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8080", 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.745 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl_server_port": { 2024-04-25T13:56:52.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl_server_port", 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8443", 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.748 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "redirect", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "redirect" 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "url_prefix": { 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url_prefix", 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_monitoring": { 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_monitoring", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "monitor device health metrics", 2024-04-25T13:56:52.752 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.753 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.754 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.755 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.756 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "mark_out_threshold": { 2024-04-25T13:56:52.757 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mark_out_threshold", 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "2419200", 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": { 2024-04-25T13:56:52.758 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pool_name", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "device_health_metrics", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "retention_period": { 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "retention_period", 2024-04-25T13:56:52.759 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15552000", 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long to retain device health metrics", 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_frequency": { 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_frequency", 2024-04-25T13:56:52.760 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "self_heal": { 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "self_heal", 2024-04-25T13:56:52.761 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.763 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "warn_threshold": { 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "warn_threshold", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.764 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7257600", 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "diskprediction_local", 2024-04-25T13:56:52.765 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.766 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.767 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.768 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.769 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.770 INFO:teuthology.orchestra.run.smithi027.stdout: "predict_interval": { 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predict_interval", 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "86400", 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "predictor_model": { 2024-04-25T13:56:52.771 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "predictor_model", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "prophetstor", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "600", 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.773 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "influx", 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": false, 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "influxdb python module not found", 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "batch_size": { 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "batch_size", 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5000", 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.775 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "database": { 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "database", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "ceph", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T13:56:52.776 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": { 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "hostname", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server hostname", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.777 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "30", 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "5", 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T13:56:52.778 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.779 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.780 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.781 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.782 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "password": { 2024-04-25T13:56:52.783 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "password", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "password of InfluxDB server user", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "port": { 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "port", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.784 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "8086", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "InfluxDB server port", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.785 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "ssl": { 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ssl", 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.786 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "false", 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "threads": { 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threads", 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.787 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "32", 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "username": { 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "username", 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.788 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "username of InfluxDB server user", 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "verify_ssl": { 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "verify_ssl", 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.789 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "true", 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "insights", 2024-04-25T13:56:52.790 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.791 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.792 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.793 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.795 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.796 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "iostat", 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.797 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.798 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.800 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.801 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.802 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "k8sevents", 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.803 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_event_retention_days": { 2024-04-25T13:56:52.804 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ceph_event_retention_days", 2024-04-25T13:56:52.805 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.805 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.805 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.805 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "7", 2024-04-25T13:56:52.805 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "config_check_secs": { 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "config_check_secs", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "10", 2024-04-25T13:56:52.806 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.807 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.809 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.810 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.811 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.812 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "localpool", 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "failure_domain": { 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "failure_domain", 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.813 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "host", 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "failure domain for any created local pool", 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.814 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.817 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.818 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": { 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "min_size", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.819 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default min_size for any created local pool", 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "num_rep": { 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "num_rep", 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3", 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.820 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default replica count for any created local pool", 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": { 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_num", 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "128", 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.821 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "prefix": { 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prefix", 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.822 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "name prefix for any created local pool", 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.823 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "subtree": { 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subtree", 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "rack", 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.824 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mds_autoscaler", 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.825 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.826 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.827 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.828 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.829 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.830 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirroring", 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.832 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.834 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.835 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.836 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.837 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "nfs", 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.838 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.841 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.842 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.843 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "fail_fs": { 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "fail_fs", 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.845 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.846 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.847 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.848 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.849 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.850 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator": { 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "orchestrator", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm", 2024-04-25T13:56:52.851 INFO:teuthology.orchestra.run.smithi027.stdout: "rook", 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "test_orchestrator" 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Orchestrator backend", 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.852 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_perf_query", 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.854 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.855 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.856 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.857 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.858 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "osd_support", 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.860 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.862 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.863 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.864 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.865 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "pg_autoscaler", 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.867 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.868 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.869 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.870 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.871 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.872 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "threshold": { 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "threshold", 2024-04-25T13:56:52.873 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "3.0", 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1.0", 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "scaling threshold", 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "progress", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_pg_recovery_event": { 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_pg_recovery_event", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T13:56:52.875 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.876 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.877 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.878 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.879 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "max_completed_events": { 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_completed_events", 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "50", 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "number of past completed events to remember", 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.882 INFO:teuthology.orchestra.run.smithi027.stdout: "sleep_interval": { 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sleep_interval", 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "5", 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "how long the module is going to sleep", 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.883 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "prometheus", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "cache": { 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cache", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.885 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "exclude_perf_counters": { 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "exclude_perf_counters", 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T13:56:52.886 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.887 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.888 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.889 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.890 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools": { 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools", 2024-04-25T13:56:52.891 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.892 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "300", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "scrape_interval": { 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "scrape_interval", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.893 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15.0", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "::", 2024-04-25T13:56:52.894 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "9283", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.895 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.896 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "stale_cache_strategy": { 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stale_cache_strategy", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "log", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.897 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_behaviour": { 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_behaviour", 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "default", 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "default", 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "error" 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.898 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "standby_error_status_code": { 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "standby_error_status_code", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "500", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "400", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "599", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.899 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.900 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.902 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.903 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.904 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_snap_create": { 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_snap_create", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.905 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "mirror_snapshot_schedule": { 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.906 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "trash_purge_schedule": { 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "trash_purge_schedule", 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.908 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "restful", 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_auth": { 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enable_auth", 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.909 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "key_file": { 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "key_file", 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.910 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.911 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.912 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.913 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.915 INFO:teuthology.orchestra.run.smithi027.stdout: "server_addr": { 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_addr", 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "server_port": { 2024-04-25T13:56:52.916 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "server_port", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.917 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rgw", 2024-04-25T13:56:52.918 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.919 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.921 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.922 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.923 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rook", 2024-04-25T13:56:52.924 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.925 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.926 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.927 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.928 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.929 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "storage_class": { 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "storage_class", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "local", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.930 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "selftest", 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.931 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.933 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.935 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "roption1": { 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption1", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.936 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "roption2": { 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "roption2", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "xyz", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.937 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption1": { 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption1", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.938 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption2": { 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption2", 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.939 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption3": { 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption3", 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "float", 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.940 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption4": { 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption4", 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption5": { 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption5", 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.942 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption6": { 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption6", 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.943 INFO:teuthology.orchestra.run.smithi027.stdout: "rwoption7": { 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rwoption7", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "1", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "42", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "testkey": { 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testkey", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.945 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "testlkey": { 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testlkey", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "testnewline": { 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "testnewline", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.947 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snap_schedule", 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_m_granularity": { 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "allow_m_granularity", 2024-04-25T13:56:52.948 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "dump_on_update": { 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "dump_on_update", 2024-04-25T13:56:52.949 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "dump database to debug log on update", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.950 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.951 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.953 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.954 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.955 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "stats", 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.956 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.957 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.958 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.959 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.960 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.961 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "status", 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.962 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.963 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.965 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.966 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.967 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telegraf", 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "address": { 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "address", 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.968 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "15", 2024-04-25T13:56:52.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.970 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.971 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.972 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.974 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.975 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "telemetry", 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_basic": { 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_basic", 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.976 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_crash": { 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_crash", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.977 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_device": { 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_device", 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.978 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_ident": { 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_ident", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "channel_perf": { 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "channel_perf", 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.980 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "contact": { 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "contact", 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.981 INFO:teuthology.orchestra.run.smithi027.stdout: "description": { 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "description", 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "device_url": { 2024-04-25T13:56:52.982 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "device_url", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "enabled": { 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "enabled", 2024-04-25T13:56:52.983 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.984 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "24", 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "8", 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "last_opt_revision": { 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "last_opt_revision", 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.985 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "1", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard": { 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.986 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "leaderboard_description": { 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "leaderboard_description", 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.987 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.988 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.989 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:52.990 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.991 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.992 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "organization": { 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "organization", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.993 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "proxy": { 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "proxy", 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.994 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.995 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "url": { 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "url", 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.996 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "test_orchestrator", 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:52.997 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:52.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:53.000 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:53.001 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:53.002 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:53.003 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:53.004 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.005 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.006 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:53.007 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "max_concurrent_clones": { 2024-04-25T13:56:53.008 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "max_concurrent_clones", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "4", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "periodic_async_work": { 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "periodic_async_work", 2024-04-25T13:56:53.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Periodically check for async work", 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_delay": { 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_delay", 2024-04-25T13:56:53.010 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot_clone_no_wait": { 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T13:56:53.011 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "True", 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:53.012 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.013 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "can_run": true, 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "error_string": "", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "module_options": { 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "discovery_interval": { 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "discovery_interval", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "uint", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "100", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.014 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "identifier": { 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "identifier", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.015 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "interval": { 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "interval", 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "secs", 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "60", 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.016 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "log_level": { 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_level", 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:53.017 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster": { 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster", 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.018 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_cluster_level": { 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_cluster_level", 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.019 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "info", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [ 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "critical", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "debug", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "error", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "info", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "warning" 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.020 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": { 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "log_to_file", 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "bool", 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "False", 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "sqlite3_killpoint": { 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sqlite3_killpoint", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "dev", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "0", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.022 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_host": { 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_host", 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "", 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.023 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_port": { 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_port", 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "int", 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "10051", 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "zabbix_sender": { 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "zabbix_sender", 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "str", 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "level": "advanced", 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 0, 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "min": "", 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "max": "", 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "enum_allowed": [], 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "desc": "", 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "long_desc": "", 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "tags": [], 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: "see_also": [] 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.025 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "services": { 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "dashboard": "https://172.21.15.27:8443/" 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "always_on_modules": { 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "octopus": [ 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T13:56:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "pacific": [ 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T13:56:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "quincy": [ 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "reef": [ 2024-04-25T13:56:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "squid": [ 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "balancer", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "devicehealth", 2024-04-25T13:56:53.029 INFO:teuthology.orchestra.run.smithi027.stdout: "orchestrator", 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscaler", 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "progress", 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_support", 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "status", 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "telemetry", 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "volumes" 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "last_failure_osd_epoch": 3, 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "active_clients": [ 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "devicehealth", 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.030 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4276929155 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "libcephsqlite", 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1102216915 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.031 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "rbd_support", 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3865533517 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "volumes", 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-04-25T13:56:53.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-04-25T13:56:53.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-04-25T13:56:53.033 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-04-25T13:56:53.033 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3278365795 2024-04-25T13:56:53.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.033 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:53.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-25T13:56:53.033 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-25T13:56:53.033 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-25T13:56:53.034 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:52 smithi045 ceph-mon[32731]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-25T13:56:53.034 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3620373493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:56:53.034 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:52 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2734065923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T13:56:53.034 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:52 smithi122 ceph-mon[32843]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-25T13:56:53.034 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3620373493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:56:53.034 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:52 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2734065923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87400afaf0 msgr2=0x7f87400b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87400afaf0 0x7f87400b2000 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8754102d60 tx=0x7f874400a2d0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 msgr2=0x7f87541a0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 0x7f87541a0580 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f874c0097b0 tx=0x7f874c003ba0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 shutdown_connections 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f87400afaf0 0x7f87400b2000 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87541080a0 0x7f87541a0580 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87541033a0 0x7f87541995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 --2- 172.21.15.27:0/2734065923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8754102160 0x7f87541990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.661+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 >> 172.21.15.27:0/2734065923 conn(0x7f87540fd330 msgr2=0x7f87540fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.662+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 shutdown_connections 2024-04-25T13:56:53.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:51.662+0000 7f875b93c640 1 -- 172.21.15.27:0/2734065923 wait complete. 2024-04-25T13:56:53.043 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T13:56:53.054 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T13:56:53.054 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:56:53.054 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:52 smithi027 ceph-mon[29140]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-25T13:56:53.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3620373493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:56:53.055 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:52 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2734065923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T13:56:53.075 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-25T13:56:53.075 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T13:56:53.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 -- 172.21.15.27:0/778945759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac106f50 msgr2=0x7fe5ac109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/778945759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac106f50 0x7fe5ac109370 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009900 tx=0x7fe5a002a410 comp rx=0 tx=0).stop 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 -- 172.21.15.27:0/778945759 shutdown_connections 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/778945759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5ac1098b0 0x7fe5ac10bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/778945759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac106f50 0x7fe5ac109370 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/778945759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5ac1045f0 0x7fe5ac106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 -- 172.21.15.27:0/778945759 >> 172.21.15.27:0/778945759 conn(0x7fe5ac0fdbf0 msgr2=0x7fe5ac1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 -- 172.21.15.27:0/778945759 shutdown_connections 2024-04-25T13:56:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.295+0000 7fe5b0f96640 1 -- 172.21.15.27:0/778945759 wait complete. 2024-04-25T13:56:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5b0f96640 1 Processor -- start 2024-04-25T13:56:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5b0f96640 1 -- start start 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5b0f96640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5ac1045f0 0x7fe5ac103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5b0f96640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5ac106f50 0x7fe5ac1020e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5b0f96640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac1098b0 0x7fe5ac102620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5b0f96640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac10f610 con 0x7fe5ac1098b0 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5b0f96640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac10f490 con 0x7fe5ac106f50 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5b0f96640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac10f790 con 0x7fe5ac1045f0 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac1098b0 0x7fe5ac102620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aad9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac1098b0 0x7fe5ac102620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52784/0 (socket says 172.21.15.27:52784) 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aad9d640 1 -- 172.21.15.27:0/2256620646 learned_addr learned my addr 172.21.15.27:0/2256620646 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aa59c640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5ac1045f0 0x7fe5ac103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5a9d9b640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5ac106f50 0x7fe5ac1020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aad9d640 1 -- 172.21.15.27:0/2256620646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5ac1045f0 msgr2=0x7fe5ac103a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aad9d640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5ac1045f0 0x7fe5ac103a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aad9d640 1 -- 172.21.15.27:0/2256620646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5ac106f50 msgr2=0x7fe5ac1020e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aad9d640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5ac106f50 0x7fe5ac1020e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.296+0000 7fe5aad9d640 1 -- 172.21.15.27:0/2256620646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ac102cc0 con 0x7fe5ac1098b0 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.297+0000 7fe5aad9d640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac1098b0 0x7fe5ac102620 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400e9d0 tx=0x7fe5a400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.297+0000 7fe59b7fe640 1 -- 172.21.15.27:0/2256620646 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4004d20 con 0x7fe5ac1098b0 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.297+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ac1a42d0 con 0x7fe5ac1098b0 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.297+0000 7fe59b7fe640 1 -- 172.21.15.27:0/2256620646 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5a4005020 con 0x7fe5ac1098b0 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.297+0000 7fe59b7fe640 1 -- 172.21.15.27:0/2256620646 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40116b0 con 0x7fe5ac1098b0 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.297+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ac1a46e0 con 0x7fe5ac1098b0 2024-04-25T13:56:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.298+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ac0ff710 con 0x7fe5ac1098b0 2024-04-25T13:56:53.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.299+0000 7fe59b7fe640 1 -- 172.21.15.27:0/2256620646 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe5a40051d0 con 0x7fe5ac1098b0 2024-04-25T13:56:53.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.300+0000 7fe59b7fe640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe5940afaf0 0x7fe5940b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.300+0000 7fe5aa59c640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe5940afaf0 0x7fe5940b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.300+0000 7fe59b7fe640 1 -- 172.21.15.27:0/2256620646 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fe5a40df210 con 0x7fe5ac1098b0 2024-04-25T13:56:53.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.301+0000 7fe5aa59c640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe5940afaf0 0x7fe5940b2000 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac0757f0 tx=0x7fe59c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:53.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.304+0000 7fe59b7fe640 1 -- 172.21.15.27:0/2256620646 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5a4016030 con 0x7fe5ac1098b0 2024-04-25T13:56:53.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.491+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5ac0637d0 con 0x7fe5ac1098b0 2024-04-25T13:56:53.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.493+0000 7fe59b7fe640 1 -- 172.21.15.27:0/2256620646 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7fe5a40a7e20 con 0x7fe5ac1098b0 2024-04-25T13:56:53.493 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:53.493 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe5940afaf0 msgr2=0x7fe5940b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe5940afaf0 0x7fe5940b2000 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac0757f0 tx=0x7fe59c00a2d0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac1098b0 msgr2=0x7fe5ac102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac1098b0 0x7fe5ac102620 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400e9d0 tx=0x7fe5a400edd0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 shutdown_connections 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fe5940afaf0 0x7fe5940b2000 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac1098b0 0x7fe5ac102620 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5ac106f50 0x7fe5ac1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 --2- 172.21.15.27:0/2256620646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5ac1045f0 0x7fe5ac103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 >> 172.21.15.27:0/2256620646 conn(0x7fe5ac0fdbf0 msgr2=0x7fe5ac105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 shutdown_connections 2024-04-25T13:56:53.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.495+0000 7fe5b0f96640 1 -- 172.21.15.27:0/2256620646 wait complete. 2024-04-25T13:56:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T13:33:56.328828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T13:38:42.470969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T13:38:43.683894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T13:56:53.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T13:56:53.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 -- 172.21.15.27:0/2900927305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 msgr2=0x7fa890103380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 --2- 172.21.15.27:0/2900927305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 0x7fa890103380 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa884009900 tx=0x7fa88402a410 comp rx=0 tx=0).stop 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 -- 172.21.15.27:0/2900927305 shutdown_connections 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 --2- 172.21.15.27:0/2900927305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa890104470 0x7fa89010abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 --2- 172.21.15.27:0/2900927305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa890103950 0x7fa890103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 --2- 172.21.15.27:0/2900927305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 0x7fa890103380 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 -- 172.21.15.27:0/2900927305 >> 172.21.15.27:0/2900927305 conn(0x7fa8900fd350 msgr2=0x7fa8900ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 -- 172.21.15.27:0/2900927305 shutdown_connections 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.624+0000 7fa896b3f640 1 -- 172.21.15.27:0/2900927305 wait complete. 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa896b3f640 1 Processor -- start 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa896b3f640 1 -- start start 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa896b3f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 0x7fa890102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa896b3f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa890103950 0x7fa89019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa896b3f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa890104470 0x7fa8901a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa896b3f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010e510 con 0x7fa890102f70 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa896b3f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010e390 con 0x7fa890104470 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa896b3f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010e690 con 0x7fa890103950 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa8948db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 0x7fa890102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa8948db640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 0x7fa890102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52810/0 (socket says 172.21.15.27:52810) 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.625+0000 7fa8948db640 1 -- 172.21.15.27:0/1486227184 learned_addr learned my addr 172.21.15.27:0/1486227184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa8950dc640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa890104470 0x7fa8901a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa88ffff640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa890103950 0x7fa89019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa8948db640 1 -- 172.21.15.27:0/1486227184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa890103950 msgr2=0x7fa89019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa8948db640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa890103950 0x7fa89019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa8948db640 1 -- 172.21.15.27:0/1486227184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa890104470 msgr2=0x7fa8901a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa8948db640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa890104470 0x7fa8901a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa8948db640 1 -- 172.21.15.27:0/1486227184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8901a1d60 con 0x7fa890102f70 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa88ffff640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa890103950 0x7fa89019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa8948db640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 0x7fa890102210 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fa884006010 tx=0x7fa884004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa88dffb640 1 -- 172.21.15.27:0/1486227184 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88403f030 con 0x7fa890102f70 2024-04-25T13:56:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa88dffb640 1 -- 172.21.15.27:0/1486227184 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa884003aa0 con 0x7fa890102f70 2024-04-25T13:56:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8901a1ff0 con 0x7fa890102f70 2024-04-25T13:56:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa88dffb640 1 -- 172.21.15.27:0/1486227184 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa884035070 con 0x7fa890102f70 2024-04-25T13:56:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.626+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8901a2400 con 0x7fa890102f70 2024-04-25T13:56:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.627+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa890107910 con 0x7fa890102f70 2024-04-25T13:56:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.629+0000 7fa88dffb640 1 -- 172.21.15.27:0/1486227184 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa884042030 con 0x7fa890102f70 2024-04-25T13:56:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.630+0000 7fa88dffb640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8800afaf0 0x7fa8800b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.630+0000 7fa88ffff640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8800afaf0 0x7fa8800b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.630+0000 7fa88dffb640 1 -- 172.21.15.27:0/1486227184 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fa8840ff680 con 0x7fa890102f70 2024-04-25T13:56:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.630+0000 7fa88ffff640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8800afaf0 0x7fa8800b2000 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa890101bb0 tx=0x7fa87c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.634+0000 7fa88dffb640 1 -- 172.21.15.27:0/1486227184 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8840c8290 con 0x7fa890102f70 2024-04-25T13:56:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.821+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa89002dda0 con 0x7fa890102f70 2024-04-25T13:56:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.822+0000 7fa88dffb640 1 -- 172.21.15.27:0/1486227184 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7fa8840c7b90 con 0x7fa890102f70 2024-04-25T13:56:53.821 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-25T13:56:53.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8800afaf0 msgr2=0x7fa8800b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8800afaf0 0x7fa8800b2000 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa890101bb0 tx=0x7fa87c005f90 comp rx=0 tx=0).stop 2024-04-25T13:56:53.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 msgr2=0x7fa890102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 0x7fa890102210 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fa884006010 tx=0x7fa884004270 comp rx=0 tx=0).stop 2024-04-25T13:56:53.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 shutdown_connections 2024-04-25T13:56:53.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fa8800afaf0 0x7fa8800b2000 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa890104470 0x7fa8901a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa890103950 0x7fa89019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 --2- 172.21.15.27:0/1486227184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa890102f70 0x7fa890102210 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 >> 172.21.15.27:0/1486227184 conn(0x7fa8900fd350 msgr2=0x7fa890104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:53.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 shutdown_connections 2024-04-25T13:56:53.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.824+0000 7fa896b3f640 1 -- 172.21.15.27:0/1486227184 wait complete. 2024-04-25T13:56:53.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.951+0000 7fbbd17ee640 1 -- 172.21.15.27:0/1671422173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 msgr2=0x7fbbcc1047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.951+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/1671422173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 0x7fbbcc1047f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009900 tx=0x7fbbbc02a410 comp rx=0 tx=0).stop 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.951+0000 7fbbd17ee640 1 -- 172.21.15.27:0/1671422173 shutdown_connections 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.951+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/1671422173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc109a30 0x7fbbcc10be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.951+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/1671422173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbcc104d30 0x7fbbcc105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.951+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/1671422173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 0x7fbbcc1047f0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.951+0000 7fbbd17ee640 1 -- 172.21.15.27:0/1671422173 >> 172.21.15.27:0/1671422173 conn(0x7fbbcc0fece0 msgr2=0x7fbbcc101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.951+0000 7fbbd17ee640 1 -- 172.21.15.27:0/1671422173 shutdown_connections 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 -- 172.21.15.27:0/1671422173 wait complete. 2024-04-25T13:56:53.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 Processor -- start 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 -- start start 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 0x7fbbcc19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbcc104d30 0x7fbbcc19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc109a30 0x7fbbcc1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc10f5f0 con 0x7fbbcc103af0 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc10f470 con 0x7fbbcc109a30 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbd17ee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc10f770 con 0x7fbbcc104d30 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbcaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 0x7fbbcc19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbcaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 0x7fbbcc19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43590/0 (socket says 172.21.15.27:43590) 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.952+0000 7fbbcaffd640 1 -- 172.21.15.27:0/519538628 learned_addr learned my addr 172.21.15.27:0/519538628 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbcb7fe640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc109a30 0x7fbbcc1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbc3fff640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbcc104d30 0x7fbbcc19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbcaffd640 1 -- 172.21.15.27:0/519538628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbcc104d30 msgr2=0x7fbbcc19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbcaffd640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbcc104d30 0x7fbbcc19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbcaffd640 1 -- 172.21.15.27:0/519538628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc109a30 msgr2=0x7fbbcc1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbcaffd640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc109a30 0x7fbbcc1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbcaffd640 1 -- 172.21.15.27:0/519538628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc1a1d80 con 0x7fbbcc103af0 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbcaffd640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 0x7fbbcc19a2b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc006010 tx=0x7fbbbc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbc8ff9640 1 -- 172.21.15.27:0/519538628 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc03f030 con 0x7fbbcc103af0 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbcc1a2010 con 0x7fbbcc103af0 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbc8ff9640 1 -- 172.21.15.27:0/519538628 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbbc003aa0 con 0x7fbbcc103af0 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbc8ff9640 1 -- 172.21.15.27:0/519538628 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc035070 con 0x7fbbcc103af0 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.953+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbcc1a2550 con 0x7fbbcc103af0 2024-04-25T13:56:53.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.954+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbcc108840 con 0x7fbbcc103af0 2024-04-25T13:56:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.956+0000 7fbbc8ff9640 1 -- 172.21.15.27:0/519538628 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbbbc042030 con 0x7fbbcc103af0 2024-04-25T13:56:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.956+0000 7fbbc8ff9640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbbb80afaf0 0x7fbbb80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.956+0000 7fbbc8ff9640 1 -- 172.21.15.27:0/519538628 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7fbbbc0ff5c0 con 0x7fbbcc103af0 2024-04-25T13:56:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.956+0000 7fbbc3fff640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbbb80afaf0 0x7fbbb80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:53.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.957+0000 7fbbc3fff640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbbb80afaf0 0x7fbbb80b2000 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4005fd0 tx=0x7fbbb4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:53.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:53.960+0000 7fbbc8ff9640 1 -- 172.21.15.27:0/519538628 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbbbc0c81d0 con 0x7fbbcc103af0 2024-04-25T13:56:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.147+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbbcc0637a0 con 0x7fbbcc103af0 2024-04-25T13:56:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.147+0000 7fbbc8ff9640 1 -- 172.21.15.27:0/519538628 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7fbbbc0d0250 con 0x7fbbcc103af0 2024-04-25T13:56:54.147 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-25T13:56:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbbb80afaf0 msgr2=0x7fbbb80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbbb80afaf0 0x7fbbb80b2000 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4005fd0 tx=0x7fbbb4005850 comp rx=0 tx=0).stop 2024-04-25T13:56:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 msgr2=0x7fbbcc19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 0x7fbbcc19a2b0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc006010 tx=0x7fbbbc031a50 comp rx=0 tx=0).stop 2024-04-25T13:56:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 shutdown_connections 2024-04-25T13:56:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fbbb80afaf0 0x7fbbb80b2000 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbcc109a30 0x7fbbcc1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbbcc104d30 0x7fbbcc19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 --2- 172.21.15.27:0/519538628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbcc103af0 0x7fbbcc19a2b0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.149+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 >> 172.21.15.27:0/519538628 conn(0x7fbbcc0fece0 msgr2=0x7fbbcc105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.150+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 shutdown_connections 2024-04-25T13:56:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.150+0000 7fbbd17ee640 1 -- 172.21.15.27:0/519538628 wait complete. 2024-04-25T13:56:54.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T13:56:54.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 -- 172.21.15.27:0/1758785376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 msgr2=0x7f2328105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 --2- 172.21.15.27:0/1758785376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 0x7f2328105140 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f2324009900 tx=0x7f232402a410 comp rx=0 tx=0).stop 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 -- 172.21.15.27:0/1758785376 shutdown_connections 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 --2- 172.21.15.27:0/1758785376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2328109a30 0x7f232810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 --2- 172.21.15.27:0/1758785376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 0x7f2328105140 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 --2- 172.21.15.27:0/1758785376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328103af0 0x7f23281047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 -- 172.21.15.27:0/1758785376 >> 172.21.15.27:0/1758785376 conn(0x7f23280fece0 msgr2=0x7f2328101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 -- 172.21.15.27:0/1758785376 shutdown_connections 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.276+0000 7f232f35e640 1 -- 172.21.15.27:0/1758785376 wait complete. 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232f35e640 1 Processor -- start 2024-04-25T13:56:54.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232f35e640 1 -- start start 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232f35e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328103af0 0x7f232819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232f35e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 0x7f232819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232f35e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2328109a30 0x7f23281a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232f35e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f232810f620 con 0x7f2328104d30 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232f35e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f232810f4a0 con 0x7f2328103af0 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232f35e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f232810f7a0 con 0x7f2328109a30 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232c8f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 0x7f232819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232c8f9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 0x7f232819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43610/0 (socket says 172.21.15.27:43610) 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232c8f9640 1 -- 172.21.15.27:0/2037505302 learned_addr learned my addr 172.21.15.27:0/2037505302 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.277+0000 7f232d0fa640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328103af0 0x7f232819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232d8fb640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2328109a30 0x7f23281a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232c8f9640 1 -- 172.21.15.27:0/2037505302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2328109a30 msgr2=0x7f23281a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232c8f9640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2328109a30 0x7f23281a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232c8f9640 1 -- 172.21.15.27:0/2037505302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328103af0 msgr2=0x7f232819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232c8f9640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328103af0 0x7f232819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232c8f9640 1 -- 172.21.15.27:0/2037505302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23281a1d60 con 0x7f2328104d30 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232d8fb640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2328109a30 0x7f23281a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232c8f9640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 0x7f232819a760 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2324006010 tx=0x7f23240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f231a7fc640 1 -- 172.21.15.27:0/2037505302 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232403f030 con 0x7f2328104d30 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23281a1ff0 con 0x7f2328104d30 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f231a7fc640 1 -- 172.21.15.27:0/2037505302 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2324031910 con 0x7f2328104d30 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f231a7fc640 1 -- 172.21.15.27:0/2037505302 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324035070 con 0x7f2328104d30 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.278+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23281a2400 con 0x7f2328104d30 2024-04-25T13:56:54.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.279+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2328108840 con 0x7f2328104d30 2024-04-25T13:56:54.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.281+0000 7f231a7fc640 1 -- 172.21.15.27:0/2037505302 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2324042030 con 0x7f2328104d30 2024-04-25T13:56:54.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.281+0000 7f231a7fc640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23140afaf0 0x7f23140b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:54.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.281+0000 7f232d0fa640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23140afaf0 0x7f23140b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:54.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.281+0000 7f231a7fc640 1 -- 172.21.15.27:0/2037505302 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f23240ff7f0 con 0x7f2328104d30 2024-04-25T13:56:54.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.282+0000 7f232d0fa640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23140afaf0 0x7f23140b2000 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f23281046f0 tx=0x7f231c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:54.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.285+0000 7f231a7fc640 1 -- 172.21.15.27:0/2037505302 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23240c8400 con 0x7f2328104d30 2024-04-25T13:56:54.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.470+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f232802dce0 con 0x7f2328104d30 2024-04-25T13:56:54.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.471+0000 7f231a7fc640 1 -- 172.21.15.27:0/2037505302 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f23240c7d00 con 0x7f2328104d30 2024-04-25T13:56:54.471 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-04-25T13:56:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23140afaf0 msgr2=0x7f23140b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23140afaf0 0x7f23140b2000 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f23281046f0 tx=0x7f231c005f90 comp rx=0 tx=0).stop 2024-04-25T13:56:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 msgr2=0x7f232819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 0x7f232819a760 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2324006010 tx=0x7f23240042f0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 shutdown_connections 2024-04-25T13:56:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f23140afaf0 0x7f23140b2000 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2328109a30 0x7f23281a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2328104d30 0x7f232819a760 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 --2- 172.21.15.27:0/2037505302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328103af0 0x7f232819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 >> 172.21.15.27:0/2037505302 conn(0x7f23280fece0 msgr2=0x7f2328105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 shutdown_connections 2024-04-25T13:56:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.473+0000 7f232f35e640 1 -- 172.21.15.27:0/2037505302 wait complete. 2024-04-25T13:56:54.483 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:54 smithi027 ceph-mon[29140]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-25T13:56:54.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2256620646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:54.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1486227184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:56:54.484 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:54 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/519538628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:56:54.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T13:56:54.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:54 smithi122 ceph-mon[32843]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-25T13:56:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:54 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2256620646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:54 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1486227184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:56:54.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:54 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/519538628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:56:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:54 smithi045 ceph-mon[32731]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-25T13:56:54.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:54 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2256620646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:54.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:54 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1486227184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T13:56:54.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:54 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/519538628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T13:56:54.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 -- 172.21.15.27:0/373500189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8104d30 msgr2=0x7f79e8105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/373500189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8104d30 0x7f79e8105140 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009900 tx=0x7f79dc02a410 comp rx=0 tx=0).stop 2024-04-25T13:56:54.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 -- 172.21.15.27:0/373500189 shutdown_connections 2024-04-25T13:56:54.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/373500189 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79e8109a30 0x7f79e810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/373500189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8104d30 0x7f79e8105140 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/373500189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8103af0 0x7f79e81047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 -- 172.21.15.27:0/373500189 >> 172.21.15.27:0/373500189 conn(0x7f79e80fece0 msgr2=0x7f79e8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 -- 172.21.15.27:0/373500189 shutdown_connections 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.598+0000 7f79ee8b6640 1 -- 172.21.15.27:0/373500189 wait complete. 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79ee8b6640 1 Processor -- start 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79ee8b6640 1 -- start start 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79ee8b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8103af0 0x7f79e819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79ee8b6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79e8104d30 0x7f79e819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79ee8b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8109a30 0x7f79e81a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79ee8b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810f640 con 0x7f79e8103af0 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79ee8b6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810f4c0 con 0x7f79e8109a30 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79ee8b6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f79e810f7c0 con 0x7f79e8104d30 2024-04-25T13:56:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8103af0 0x7f79e819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79e7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8103af0 0x7f79e819a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43628/0 (socket says 172.21.15.27:43628) 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79e7fff640 1 -- 172.21.15.27:0/2443922641 learned_addr learned my addr 172.21.15.27:0/2443922641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.599+0000 7f79e77fe640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79e8104d30 0x7f79e819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79ece53640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8109a30 0x7f79e81a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e7fff640 1 -- 172.21.15.27:0/2443922641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79e8104d30 msgr2=0x7f79e819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e7fff640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79e8104d30 0x7f79e819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e7fff640 1 -- 172.21.15.27:0/2443922641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8109a30 msgr2=0x7f79e81a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e7fff640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8109a30 0x7f79e81a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e7fff640 1 -- 172.21.15.27:0/2443922641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e81a1d40 con 0x7f79e8103af0 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e7fff640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8103af0 0x7f79e819a270 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f79d800c970 tx=0x7f79d800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e57fa640 1 -- 172.21.15.27:0/2443922641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8004220 con 0x7f79e8103af0 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e81a1fd0 con 0x7f79e8103af0 2024-04-25T13:56:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e57fa640 1 -- 172.21.15.27:0/2443922641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79d80043c0 con 0x7f79e8103af0 2024-04-25T13:56:54.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79e57fa640 1 -- 172.21.15.27:0/2443922641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d800f450 con 0x7f79e8103af0 2024-04-25T13:56:54.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.600+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e81a24e0 con 0x7f79e8103af0 2024-04-25T13:56:54.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.601+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e8108840 con 0x7f79e8103af0 2024-04-25T13:56:54.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.603+0000 7f79e57fa640 1 -- 172.21.15.27:0/2443922641 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f79d80079b0 con 0x7f79e8103af0 2024-04-25T13:56:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.603+0000 7f79e57fa640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79d00afaf0 0x7f79d00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.604+0000 7f79e77fe640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79d00afaf0 0x7f79d00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.604+0000 7f79e57fa640 1 -- 172.21.15.27:0/2443922641 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f79d80deda0 con 0x7f79e8103af0 2024-04-25T13:56:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.604+0000 7f79e77fe640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79d00afaf0 0x7f79d00b2000 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f79dc006010 tx=0x7f79dc031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:54.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.607+0000 7f79e57fa640 1 -- 172.21.15.27:0/2443922641 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79d8014030 con 0x7f79e8103af0 2024-04-25T13:56:54.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.808+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f79e802db50 con 0x7f79e8103af0 2024-04-25T13:56:54.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.808+0000 7f79e57fa640 1 -- 172.21.15.27:0/2443922641 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v78) ==== 81+0+592 (secure 0 0 0) 0x7f79d80a79b0 con 0x7f79e8103af0 2024-04-25T13:56:54.808 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4203555014 2024-04-26T13:31:30.235539+0000 2024-04-25T13:56:54.808 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/1748672481 2024-04-26T13:31:30.235539+0000 2024-04-25T13:56:54.808 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4198333021 2024-04-26T13:31:03.359904+0000 2024-04-25T13:56:54.808 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2249397755 2024-04-26T13:31:30.235539+0000 2024-04-25T13:56:54.809 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/4149968369 2024-04-26T13:31:03.359904+0000 2024-04-25T13:56:54.809 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1628315942 2024-04-26T13:31:03.359904+0000 2024-04-25T13:56:54.809 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3607663760 2024-04-26T13:31:03.359904+0000 2024-04-25T13:56:54.809 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/1748672481 2024-04-26T13:31:30.235539+0000 2024-04-25T13:56:54.809 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/4149968369 2024-04-26T13:31:03.359904+0000 2024-04-25T13:56:54.809 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3971294279 2024-04-26T13:31:30.235539+0000 2024-04-25T13:56:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:listed 10 entries 2024-04-25T13:56:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79d00afaf0 msgr2=0x7f79d00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79d00afaf0 0x7f79d00b2000 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f79dc006010 tx=0x7f79dc031400 comp rx=0 tx=0).stop 2024-04-25T13:56:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8103af0 msgr2=0x7f79e819a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8103af0 0x7f79e819a270 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f79d800c970 tx=0x7f79d800ce30 comp rx=0 tx=0).stop 2024-04-25T13:56:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 shutdown_connections 2024-04-25T13:56:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f79d00afaf0 0x7f79d00b2000 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8109a30 0x7f79e81a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79e8104d30 0x7f79e819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 --2- 172.21.15.27:0/2443922641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79e8103af0 0x7f79e819a270 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.810+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 >> 172.21.15.27:0/2443922641 conn(0x7f79e80fece0 msgr2=0x7f79e8105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.811+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 shutdown_connections 2024-04-25T13:56:54.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:54.811+0000 7f79ee8b6640 1 -- 172.21.15.27:0/2443922641 wait complete. 2024-04-25T13:56:54.822 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T13:56:54.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:56:54.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /tmp/cephfs 2024-04-25T13:56:55.173 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T13:56:55.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:56:55.174 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-04-25T13:56:55.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:56:55.197 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-25T13:56:55.254 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-04-25T13:56:55.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:56:55.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-04-25T13:56:55.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T13:56:55.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-04-25T13:56:55.405 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T13:56:55.419 INFO:tasks.cephadm:Teardown begin 2024-04-25T13:56:55.419 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T13:56:55.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:56:55.423 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:55 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2037505302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:56:55.423 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:55 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2443922641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T13:56:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 -- 172.21.15.27:0/3910256790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78106be0 msgr2=0x7f9d781090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:55.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/3910256790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78106be0 0x7f9d781090c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009900 tx=0x7f9d6402a410 comp rx=0 tx=0).stop 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 -- 172.21.15.27:0/3910256790 shutdown_connections 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/3910256790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d78109750 0x7f9d7810bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/3910256790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78106be0 0x7f9d781090c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/3910256790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d781041f0 0x7f9d78106610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 -- 172.21.15.27:0/3910256790 >> 172.21.15.27:0/3910256790 conn(0x7f9d780fdbd0 msgr2=0x7f9d78100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 -- 172.21.15.27:0/3910256790 shutdown_connections 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 -- 172.21.15.27:0/3910256790 wait complete. 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 Processor -- start 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 -- start start 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d781041f0 0x7f9d7819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.536+0000 7f9d7eabd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d78106be0 0x7f9d7819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7eabd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78109750 0x7f9d781a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:55.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7eabd640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7810f5a0 con 0x7f9d78109750 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7eabd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7810f420 con 0x7f9d78106be0 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7eabd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7810f720 con 0x7f9d781041f0 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78109750 0x7f9d781a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78109750 0x7f9d781a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43644/0 (socket says 172.21.15.27:43644) 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 -- 172.21.15.27:0/2972179921 learned_addr learned my addr 172.21.15.27:0/2972179921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d77fff640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d78106be0 0x7f9d7819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7c859640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d781041f0 0x7f9d7819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 -- 172.21.15.27:0/2972179921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d781041f0 msgr2=0x7f9d7819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d781041f0 0x7f9d7819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 -- 172.21.15.27:0/2972179921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d78106be0 msgr2=0x7f9d7819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d78106be0 0x7f9d7819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 -- 172.21.15.27:0/2972179921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d781a1d40 con 0x7f9d78109750 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7d05a640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78109750 0x7f9d781a16d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009fd0 tx=0x7f9d7000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:55.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d75ffb640 1 -- 172.21.15.27:0/2972179921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70017070 con 0x7f9d78109750 2024-04-25T13:56:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d781a1fd0 con 0x7f9d78109750 2024-04-25T13:56:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d75ffb640 1 -- 172.21.15.27:0/2972179921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d70003e20 con 0x7f9d78109750 2024-04-25T13:56:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.537+0000 7f9d75ffb640 1 -- 172.21.15.27:0/2972179921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70010460 con 0x7f9d78109750 2024-04-25T13:56:55.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.538+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d781a2440 con 0x7f9d78109750 2024-04-25T13:56:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.538+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d780ff6f0 con 0x7f9d78109750 2024-04-25T13:56:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.540+0000 7f9d75ffb640 1 -- 172.21.15.27:0/2972179921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9d70003970 con 0x7f9d78109750 2024-04-25T13:56:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.540+0000 7f9d75ffb640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9d680afaf0 0x7f9d680b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.541+0000 7f9d7c859640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9d680afaf0 0x7f9d680b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.541+0000 7f9d75ffb640 1 -- 172.21.15.27:0/2972179921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f9d700de8a0 con 0x7f9d78109750 2024-04-25T13:56:55.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.541+0000 7f9d7c859640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9d680afaf0 0x7f9d680b2000 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9d7806a850 tx=0x7f9d6c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:55.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.545+0000 7f9d75ffb640 1 -- 172.21.15.27:0/2972179921 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d700a6e60 con 0x7f9d78109750 2024-04-25T13:56:55.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:55 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2037505302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:56:55.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:55 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2443922641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T13:56:55.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:55 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2037505302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T13:56:55.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:55 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2443922641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T13:56:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.773+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d7802db50 con 0x7f9d78109750 2024-04-25T13:56:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.773+0000 7f9d75ffb640 1 -- 172.21.15.27:0/2972179921 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f9d700ab160 con 0x7f9d78109750 2024-04-25T13:56:55.773 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:55.774 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:56:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9d680afaf0 msgr2=0x7f9d680b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9d680afaf0 0x7f9d680b2000 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9d7806a850 tx=0x7f9d6c00b390 comp rx=0 tx=0).stop 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78109750 msgr2=0x7f9d781a16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78109750 0x7f9d781a16d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009fd0 tx=0x7f9d7000b260 comp rx=0 tx=0).stop 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 shutdown_connections 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f9d680afaf0 0x7f9d680b2000 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d78109750 0x7f9d781a16d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d78106be0 0x7f9d7819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 --2- 172.21.15.27:0/2972179921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d781041f0 0x7f9d7819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 >> 172.21.15.27:0/2972179921 conn(0x7f9d780fdbd0 msgr2=0x7f9d78104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 shutdown_connections 2024-04-25T13:56:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.776+0000 7f9d7eabd640 1 -- 172.21.15.27:0/2972179921 wait complete. 2024-04-25T13:56:55.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T13:56:55.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 -- 172.21.15.27:0/896096510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 msgr2=0x7f843810ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 --2- 172.21.15.27:0/896096510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 0x7f843810ad20 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8430009ca0 tx=0x7f843002a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 -- 172.21.15.27:0/896096510 shutdown_connections 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 --2- 172.21.15.27:0/896096510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 0x7f843810ad20 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 --2- 172.21.15.27:0/896096510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8438103c00 0x7f8438104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 --2- 172.21.15.27:0/896096510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84381029c0 0x7f84381036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 -- 172.21.15.27:0/896096510 >> 172.21.15.27:0/896096510 conn(0x7f84380fdbd0 msgr2=0x7f8438100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 -- 172.21.15.27:0/896096510 shutdown_connections 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 -- 172.21.15.27:0/896096510 wait complete. 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 Processor -- start 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.900+0000 7f843f234640 1 -- start start 2024-04-25T13:56:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843f234640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84381029c0 0x7f843806a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843f234640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8438103c00 0x7f843819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843f234640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 0x7f84381a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843f234640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f843810e4d0 con 0x7f8438108900 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843f234640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f843810e350 con 0x7f8438103c00 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843f234640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f843810e650 con 0x7f84381029c0 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 0x7f84381a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 0x7f84381a14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43660/0 (socket says 172.21.15.27:43660) 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 -- 172.21.15.27:0/860688883 learned_addr learned my addr 172.21.15.27:0/860688883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843cfd0640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84381029c0 0x7f843806a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f8437fff640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8438103c00 0x7f843819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 -- 172.21.15.27:0/860688883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84381029c0 msgr2=0x7f843806a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84381029c0 0x7f843806a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 -- 172.21.15.27:0/860688883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8438103c00 msgr2=0x7f843819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8438103c00 0x7f843819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 -- 172.21.15.27:0/860688883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84381a1c40 con 0x7f8438108900 2024-04-25T13:56:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f843d7d1640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 0x7f84381a14b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f84300097b0 tx=0x7f8430003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.901+0000 7f8435ffb640 1 -- 172.21.15.27:0/860688883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8430004240 con 0x7f8438108900 2024-04-25T13:56:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.902+0000 7f8435ffb640 1 -- 172.21.15.27:0/860688883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84300317c0 con 0x7f8438108900 2024-04-25T13:56:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.902+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84381a1ed0 con 0x7f8438108900 2024-04-25T13:56:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.902+0000 7f8435ffb640 1 -- 172.21.15.27:0/860688883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843002fde0 con 0x7f8438108900 2024-04-25T13:56:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.902+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84381a2330 con 0x7f8438108900 2024-04-25T13:56:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.903+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f843806afa0 con 0x7f8438108900 2024-04-25T13:56:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.904+0000 7f8435ffb640 1 -- 172.21.15.27:0/860688883 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f84300043e0 con 0x7f8438108900 2024-04-25T13:56:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.905+0000 7f8435ffb640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84280afaf0 0x7f84280b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.905+0000 7f843cfd0640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84280afaf0 0x7f84280b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.905+0000 7f8435ffb640 1 -- 172.21.15.27:0/860688883 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f8430100670 con 0x7f8438108900 2024-04-25T13:56:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.906+0000 7f843cfd0640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84280afaf0 0x7f84280b2000 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f84381035c0 tx=0x7f842c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:55.909+0000 7f8435ffb640 1 -- 172.21.15.27:0/860688883 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f84300c9280 con 0x7f8438108900 2024-04-25T13:56:56.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.135+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8438107710 con 0x7f8438108900 2024-04-25T13:56:56.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.135+0000 7f8435ffb640 1 -- 172.21.15.27:0/860688883 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+11807 (secure 0 0 0) 0x7f84300c8b80 con 0x7f8438108900 2024-04-25T13:56:56.135 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:56.135 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3166244728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3166244728},{"type":"v1","addr":"172.21.15.27:6835","nonce":3166244728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/481277630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":481277630},{"type":"v1","addr":"172.21.15.27:6837","nonce":481277630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6839/2432690809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6838","nonce":2432690809},{"type":"v1","addr":"172.21.15.27:6839","nonce":2432690809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6841/1048885133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6840","nonce":1048885133},{"type":"v1","addr":"172.21.15.27:6841","nonce":1048885133}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/3322601046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3322601046},{"type":"v1","addr":"172.21.15.122:6833","nonce":3322601046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6833/3792078836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3792078836},{"type":"v1","addr":"172.21.15.45:6833","nonce":3792078836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/4225728799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4225728799},{"type":"v1","addr":"172.21.15.45:6835","nonce":4225728799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/1340027477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":1340027477},{"type":"v1","addr":"172.21.15.122:6835","nonce":1340027477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24493,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/1997914176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":1997914176},{"type":"v1","addr":"172.21.15.122:6837","nonce":1997914176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24512,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/4218693179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":4218693179},{"type":"v1","addr":"172.21.15.45:6837","nonce":4218693179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T13:38:44.716138+0000","modified":"2024-04-25T13:38:48.756882+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6839/3397333136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":3397333136},{"type":"v1","addr":"172.21.15.122:6839","nonce":3397333136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6839/3661968964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":3661968964},{"type":"v1","addr":"172.21.15.45:6839","nonce":3661968964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-04-25T13:56:56.136 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-25T13:56:56.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.137+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84280afaf0 msgr2=0x7f84280b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.137+0000 7f843f234640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84280afaf0 0x7f84280b2000 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f84381035c0 tx=0x7f842c00a2d0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.137+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 msgr2=0x7f84381a14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.137+0000 7f843f234640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 0x7f84381a14b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f84300097b0 tx=0x7f8430003c20 comp rx=0 tx=0).stop 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.137+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 shutdown_connections 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.137+0000 7f843f234640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f84280afaf0 0x7f84280b2000 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.138+0000 7f843f234640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8438108900 0x7f84381a14b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.138+0000 7f843f234640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8438103c00 0x7f843819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.138+0000 7f843f234640 1 --2- 172.21.15.27:0/860688883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84381029c0 0x7f843806a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.138+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 >> 172.21.15.27:0/860688883 conn(0x7f84380fdbd0 msgr2=0x7f84381048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.138+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 shutdown_connections 2024-04-25T13:56:56.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.138+0000 7f843f234640 1 -- 172.21.15.27:0/860688883 wait complete. 2024-04-25T13:56:56.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T13:56:56.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 -- 172.21.15.27:0/3252016892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908104d40 msgr2=0x7f7908105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 --2- 172.21.15.27:0/3252016892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908104d40 0x7f7908105150 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009900 tx=0x7f78f002a410 comp rx=0 tx=0).stop 2024-04-25T13:56:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 -- 172.21.15.27:0/3252016892 shutdown_connections 2024-04-25T13:56:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 --2- 172.21.15.27:0/3252016892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7908109a40 0x7f790810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 --2- 172.21.15.27:0/3252016892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908104d40 0x7f7908105150 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 --2- 172.21.15.27:0/3252016892 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7908103b00 0x7f7908104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 -- 172.21.15.27:0/3252016892 >> 172.21.15.27:0/3252016892 conn(0x7f79080fed10 msgr2=0x7f79081011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 -- 172.21.15.27:0/3252016892 shutdown_connections 2024-04-25T13:56:56.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.262+0000 7f790c8d8640 1 -- 172.21.15.27:0/3252016892 wait complete. 2024-04-25T13:56:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790c8d8640 1 Processor -- start 2024-04-25T13:56:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790c8d8640 1 -- start start 2024-04-25T13:56:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790c8d8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908103b00 0x7f790819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790c8d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7908104d40 0x7f790819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790c8d8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7908109a40 0x7f79081a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790c8d8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f790810f600 con 0x7f7908103b00 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790c8d8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f790810f480 con 0x7f7908104d40 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790c8d8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f790810f780 con 0x7f7908109a40 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908103b00 0x7f790819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908103b00 0x7f790819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43684/0 (socket says 172.21.15.27:43684) 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790659c640 1 -- 172.21.15.27:0/2949924029 learned_addr learned my addr 172.21.15.27:0/2949924029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f7906d9d640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7908109a40 0x7f79081a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f78fdd9b640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7908104d40 0x7f790819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790659c640 1 -- 172.21.15.27:0/2949924029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7908109a40 msgr2=0x7f79081a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790659c640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7908109a40 0x7f79081a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790659c640 1 -- 172.21.15.27:0/2949924029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7908104d40 msgr2=0x7f790819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790659c640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7908104d40 0x7f790819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f790659c640 1 -- 172.21.15.27:0/2949924029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79081a1d80 con 0x7f7908103b00 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.263+0000 7f78fdd9b640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7908104d40 0x7f790819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:56:56.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.264+0000 7f790659c640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908103b00 0x7f790819a240 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f78f8005fd0 tx=0x7f78f800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:56.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.264+0000 7f78fffff640 1 -- 172.21.15.27:0/2949924029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8017070 con 0x7f7908103b00 2024-04-25T13:56:56.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.264+0000 7f78fffff640 1 -- 172.21.15.27:0/2949924029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78f8009c30 con 0x7f7908103b00 2024-04-25T13:56:56.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.264+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79081a2070 con 0x7f7908103b00 2024-04-25T13:56:56.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.264+0000 7f78fffff640 1 -- 172.21.15.27:0/2949924029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8003d10 con 0x7f7908103b00 2024-04-25T13:56:56.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.264+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79081a25c0 con 0x7f7908103b00 2024-04-25T13:56:56.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.265+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7908108850 con 0x7f7908103b00 2024-04-25T13:56:56.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.266+0000 7f78fffff640 1 -- 172.21.15.27:0/2949924029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f78f8003eb0 con 0x7f7908103b00 2024-04-25T13:56:56.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.267+0000 7f78fffff640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f40afaf0 0x7f78f40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.267+0000 7f78fdd9b640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f40afaf0 0x7f78f40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.267+0000 7f78fffff640 1 -- 172.21.15.27:0/2949924029 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7f78f80de8f0 con 0x7f7908103b00 2024-04-25T13:56:56.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.267+0000 7f78fdd9b640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f40afaf0 0x7f78f40b2000 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f78f0005f70 tx=0x7f78f0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:56.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.272+0000 7f78fffff640 1 -- 172.21.15.27:0/2949924029 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78f8020030 con 0x7f7908103b00 2024-04-25T13:56:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:56 smithi027 ceph-mon[29140]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 15 KiB/s wr, 4 op/s 2024-04-25T13:56:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:56 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2972179921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:56.400 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:56 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/860688883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.459+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79080637a0 con 0x7f7908103b00 2024-04-25T13:56:56.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.461+0000 7f78fffff640 1 -- 172.21.15.27:0/2949924029 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22576 (secure 0 0 0) 0x7f78f8009de0 con 0x7f7908103b00 2024-04-25T13:56:56.461 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-25T13:56:56.461 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"fsid":"db01fc44-0307-11ef-bc93-c7b262605968","created":"2024-04-25T13:30:33.620592+0000","modified":"2024-04-25T13:39:31.481282+0000","last_up_change":"2024-04-25T13:37:53.396328+0000","last_in_change":"2024-04-25T13:37:32.836260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T13:33:56.328828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T13:38:42.470969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T13:38:43.683894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24cfe666-151a-4e3b-b9eb-448622218f5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6803","nonce":342047280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6805","nonce":342047280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6809","nonce":342047280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":342047280},{"type":"v1","addr":"172.21.15.27:6807","nonce":342047280}]},"public_addr":"172.21.15.27:6803/342047280","cluster_addr":"172.21.15.27:6805/342047280","heartbeat_back_addr":"172.21.15.27:6809/342047280","heartbeat_front_addr":"172.21.15.27:6807/342047280","state":["exists","up"]},{"osd":1,"uuid":"6f666f7a-74de-468f-8d21-f9aaeab6d18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6801","nonce":4202033932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6803","nonce":4202033932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6807","nonce":4202033932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":4202033932},{"type":"v1","addr":"172.21.15.45:6805","nonce":4202033932}]},"public_addr":"172.21.15.45:6801/4202033932","cluster_addr":"172.21.15.45:6803/4202033932","heartbeat_back_addr":"172.21.15.45:6807/4202033932","heartbeat_front_addr":"172.21.15.45:6805/4202033932","state":["exists","up"]},{"osd":2,"uuid":"a9efabc5-f820-4cd2-9e55-956219ccb103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6801","nonce":1296998909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6803","nonce":1296998909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6807","nonce":1296998909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1296998909},{"type":"v1","addr":"172.21.15.122:6805","nonce":1296998909}]},"public_addr":"172.21.15.122:6801/1296998909","cluster_addr":"172.21.15.122:6803/1296998909","heartbeat_back_addr":"172.21.15.122:6807/1296998909","heartbeat_front_addr":"172.21.15.122:6805/1296998909","state":["exists","up"]},{"osd":3,"uuid":"d0b3dc7e-9e5d-4cee-8395-0a83219ee52d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6811","nonce":1699526500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6813","nonce":1699526500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6817","nonce":1699526500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":1699526500},{"type":"v1","addr":"172.21.15.27:6815","nonce":1699526500}]},"public_addr":"172.21.15.27:6811/1699526500","cluster_addr":"172.21.15.27:6813/1699526500","heartbeat_back_addr":"172.21.15.27:6817/1699526500","heartbeat_front_addr":"172.21.15.27:6815/1699526500","state":["exists","up"]},{"osd":4,"uuid":"75840f44-2190-409f-802e-0479ec705f0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6809","nonce":2006568001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6811","nonce":2006568001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6815","nonce":2006568001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2006568001},{"type":"v1","addr":"172.21.15.45:6813","nonce":2006568001}]},"public_addr":"172.21.15.45:6809/2006568001","cluster_addr":"172.21.15.45:6811/2006568001","heartbeat_back_addr":"172.21.15.45:6815/2006568001","heartbeat_front_addr":"172.21.15.45:6813/2006568001","state":["exists","up"]},{"osd":5,"uuid":"0e2af772-371f-40d5-8ff8-541551a67e38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6809","nonce":3058238339}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6811","nonce":3058238339}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6815","nonce":3058238339}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":3058238339},{"type":"v1","addr":"172.21.15.122:6813","nonce":3058238339}]},"public_addr":"172.21.15.122:6809/3058238339","cluster_addr":"172.21.15.122:6811/3058238339","heartbeat_back_addr":"172.21.15.122:6815/3058238339","heartbeat_front_addr":"172.21.15.122:6813/3058238339","state":["exists","up"]},{"osd":6,"uuid":"8e18badd-2f57-42b1-a367-e88ecaa3c0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6819","nonce":100381961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6821","nonce":100381961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6825","nonce":100381961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":100381961},{"type":"v1","addr":"172.21.15.27:6823","nonce":100381961}]},"public_addr":"172.21.15.27:6819/100381961","cluster_addr":"172.21.15.27:6821/100381961","heartbeat_back_addr":"172.21.15.27:6825/100381961","heartbeat_front_addr":"172.21.15.27:6823/100381961","state":["exists","up"]},{"osd":7,"uuid":"30034bf9-dc47-49be-a9c5-e8ff56a32d8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6817","nonce":1966399238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6819","nonce":1966399238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6823","nonce":1966399238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1966399238},{"type":"v1","addr":"172.21.15.45:6821","nonce":1966399238}]},"public_addr":"172.21.15.45:6817/1966399238","cluster_addr":"172.21.15.45:6819/1966399238","heartbeat_back_addr":"172.21.15.45:6823/1966399238","heartbeat_front_addr":"172.21.15.45:6821/1966399238","state":["exists","up"]},{"osd":8,"uuid":"1ce8ba2c-eab3-41d7-9e48-afd6a5d58a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6817","nonce":1308119675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6819","nonce":1308119675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6823","nonce":1308119675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1308119675},{"type":"v1","addr":"172.21.15.122:6821","nonce":1308119675}]},"public_addr":"172.21.15.122:6817/1308119675","cluster_addr":"172.21.15.122:6819/1308119675","heartbeat_back_addr":"172.21.15.122:6823/1308119675","heartbeat_front_addr":"172.21.15.122:6821/1308119675","state":["exists","up"]},{"osd":9,"uuid":"b43d7183-1100-4873-9968-72ba295cef58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6827","nonce":3822838727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6829","nonce":3822838727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6833","nonce":3822838727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":3822838727},{"type":"v1","addr":"172.21.15.27:6831","nonce":3822838727}]},"public_addr":"172.21.15.27:6827/3822838727","cluster_addr":"172.21.15.27:6829/3822838727","heartbeat_back_addr":"172.21.15.27:6833/3822838727","heartbeat_front_addr":"172.21.15.27:6831/3822838727","state":["exists","up"]},{"osd":10,"uuid":"8438652f-3c76-4b50-8995-cd190d70f017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6825","nonce":374944878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6827","nonce":374944878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6831","nonce":374944878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":374944878},{"type":"v1","addr":"172.21.15.45:6829","nonce":374944878}]},"public_addr":"172.21.15.45:6825/374944878","cluster_addr":"172.21.15.45:6827/374944878","heartbeat_back_addr":"172.21.15.45:6831/374944878","heartbeat_front_addr":"172.21.15.45:6829/374944878","state":["exists","up"]},{"osd":11,"uuid":"23f3a823-363d-42ca-b03c-64cca2ba75d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6825","nonce":3308484009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6827","nonce":3308484009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6831","nonce":3308484009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":3308484009},{"type":"v1","addr":"172.21.15.122:6829","nonce":3308484009}]},"public_addr":"172.21.15.122:6825/3308484009","cluster_addr":"172.21.15.122:6827/3308484009","heartbeat_back_addr":"172.21.15.122:6831/3308484009","heartbeat_front_addr":"172.21.15.122:6829/3308484009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:03.452097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:28.008073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:33:52.665076+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:18.481099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:34:44.636385+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:10.232908+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:35:37.092182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:03.924665+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:30.531299+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:36:57.903927+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:24.844196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T13:37:51.700796+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/4203555014":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:0/4198333021":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/2249397755":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6800/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/1628315942":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3607663760":"2024-04-26T13:31:03.359904+0000","172.21.15.27:6800/1748672481":"2024-04-26T13:31:30.235539+0000","172.21.15.27:6801/4149968369":"2024-04-26T13:31:03.359904+0000","172.21.15.27:0/3971294279":"2024-04-26T13:31:30.235539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f40afaf0 msgr2=0x7f78f40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f40afaf0 0x7f78f40b2000 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f78f0005f70 tx=0x7f78f0031400 comp rx=0 tx=0).stop 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908103b00 msgr2=0x7f790819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908103b00 0x7f790819a240 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f78f8005fd0 tx=0x7f78f800dde0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 shutdown_connections 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f78f40afaf0 0x7f78f40b2000 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7908109a40 0x7f79081a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7908104d40 0x7f790819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 --2- 172.21.15.27:0/2949924029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7908103b00 0x7f790819a240 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 >> 172.21.15.27:0/2949924029 conn(0x7f79080fed10 msgr2=0x7f7908105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 shutdown_connections 2024-04-25T13:56:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.463+0000 7f790c8d8640 1 -- 172.21.15.27:0/2949924029 wait complete. 2024-04-25T13:56:56.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T13:56:56.557 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:56 smithi122 ceph-mon[32843]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 15 KiB/s wr, 4 op/s 2024-04-25T13:56:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:56 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2972179921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:56.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:56 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/860688883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:56 smithi045 ceph-mon[32731]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 15 KiB/s wr, 4 op/s 2024-04-25T13:56:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:56 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2972179921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:56.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:56 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/860688883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3227574635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b4104690 msgr2=0x7ff8b410ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3227574635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b4104690 0x7ff8b410ade0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009ca0 tx=0x7ff8ac02a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3227574635 shutdown_connections 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3227574635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b4104690 0x7ff8b410ade0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3227574635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8b4103b30 0x7ff8b4104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3227574635 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8b4102fd0 0x7ff8b41033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3227574635 >> 172.21.15.27:0/3227574635 conn(0x7ff8b40fd350 msgr2=0x7ff8b40ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3227574635 shutdown_connections 2024-04-25T13:56:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.589+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3227574635 wait complete. 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8bc2f2640 1 Processor -- start 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8bc2f2640 1 -- start start 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8bc2f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8b4102fd0 0x7ff8b4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8bc2f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8b4103b30 0x7ff8b419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8bc2f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b419f480 0x7ff8b41a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8bc2f2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b410f860 con 0x7ff8b419f480 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8bc2f2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b410f6e0 con 0x7ff8b4102fd0 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8bc2f2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b410f9e0 con 0x7ff8b4103b30 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8ba88f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b419f480 0x7ff8b41a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8ba88f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b419f480 0x7ff8b41a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43706/0 (socket says 172.21.15.27:43706) 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8ba88f640 1 -- 172.21.15.27:0/3074420375 learned_addr learned my addr 172.21.15.27:0/3074420375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.590+0000 7ff8ba08e640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8b4102fd0 0x7ff8b4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8b988d640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8b4103b30 0x7ff8b419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8ba88f640 1 -- 172.21.15.27:0/3074420375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8b4103b30 msgr2=0x7ff8b419a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8ba88f640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8b4103b30 0x7ff8b419a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8ba88f640 1 -- 172.21.15.27:0/3074420375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8b4102fd0 msgr2=0x7ff8b4199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8ba88f640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8b4102fd0 0x7ff8b4199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8ba88f640 1 -- 172.21.15.27:0/3074420375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b41a1f10 con 0x7ff8b419f480 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8b988d640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8b4103b30 0x7ff8b419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8ba88f640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b419f480 0x7ff8b41a18a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0097b0 tx=0x7ff8ac003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8a77fe640 1 -- 172.21.15.27:0/3074420375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac004240 con 0x7ff8b419f480 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b41a21a0 con 0x7ff8b419f480 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8a77fe640 1 -- 172.21.15.27:0/3074420375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8ac02f380 con 0x7ff8b419f480 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8a77fe640 1 -- 172.21.15.27:0/3074420375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac02fc30 con 0x7ff8b419f480 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.591+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b41a2740 con 0x7ff8b419f480 2024-04-25T13:56:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.592+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b406b310 con 0x7ff8b419f480 2024-04-25T13:56:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.594+0000 7ff8a77fe640 1 -- 172.21.15.27:0/3074420375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff8ac0043e0 con 0x7ff8b419f480 2024-04-25T13:56:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.595+0000 7ff8a77fe640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8a00afaf0 0x7ff8a00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.595+0000 7ff8ba08e640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8a00afaf0 0x7ff8a00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.595+0000 7ff8a77fe640 1 -- 172.21.15.27:0/3074420375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8190+0+0 (secure 0 0 0) 0x7ff8ac100550 con 0x7ff8b419f480 2024-04-25T13:56:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.595+0000 7ff8ba08e640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8a00afaf0 0x7ff8a00b2000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005d00 tx=0x7ff8a8005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:56.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.599+0000 7ff8a77fe640 1 -- 172.21.15.27:0/3074420375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8ac0c9160 con 0x7ff8b419f480 2024-04-25T13:56:56.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.825+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff8b402dda0 con 0x7ff8b419f480 2024-04-25T13:56:56.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.828+0000 7ff8a77fe640 1 -- 172.21.15.27:0/3074420375 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) ==== 161+0+0 (secure 0 0 0) 0x7ff8ac0d11e0 con 0x7ff8b419f480 2024-04-25T13:56:56.828 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T13:56:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.830+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8a00afaf0 msgr2=0x7ff8a00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.830+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8a00afaf0 0x7ff8a00b2000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005d00 tx=0x7ff8a8005c50 comp rx=0 tx=0).stop 2024-04-25T13:56:56.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.830+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b419f480 msgr2=0x7ff8b41a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.830+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b419f480 0x7ff8b41a18a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0097b0 tx=0x7ff8ac003c20 comp rx=0 tx=0).stop 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.831+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 shutdown_connections 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.831+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7ff8a00afaf0 0x7ff8a00b2000 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.831+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8b419f480 0x7ff8b41a18a0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.831+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8b4103b30 0x7ff8b419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.831+0000 7ff8bc2f2640 1 --2- 172.21.15.27:0/3074420375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8b4102fd0 0x7ff8b4199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.831+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 >> 172.21.15.27:0/3074420375 conn(0x7ff8b40fd350 msgr2=0x7ff8b4106cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.831+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 shutdown_connections 2024-04-25T13:56:56.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.831+0000 7ff8bc2f2640 1 -- 172.21.15.27:0/3074420375 wait complete. 2024-04-25T13:56:56.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T13:56:56.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 -- 172.21.15.27:0/1967142226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 msgr2=0x7f889c10abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 --2- 172.21.15.27:0/1967142226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 0x7f889c10abc0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8894009ca0 tx=0x7f889402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 -- 172.21.15.27:0/1967142226 shutdown_connections 2024-04-25T13:56:56.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 --2- 172.21.15.27:0/1967142226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 0x7f889c10abc0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 --2- 172.21.15.27:0/1967142226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f889c103980 0x7f889c103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 --2- 172.21.15.27:0/1967142226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f889c102fa0 0x7f889c1033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 -- 172.21.15.27:0/1967142226 >> 172.21.15.27:0/1967142226 conn(0x7f889c0fd380 msgr2=0x7f889c0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:56.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 -- 172.21.15.27:0/1967142226 shutdown_connections 2024-04-25T13:56:56.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.958+0000 7f88a251f640 1 -- 172.21.15.27:0/1967142226 wait complete. 2024-04-25T13:56:56.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a251f640 1 Processor -- start 2024-04-25T13:56:56.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a251f640 1 -- start start 2024-04-25T13:56:56.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a251f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f889c102fa0 0x7f889c102240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a251f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f889c103980 0x7f889c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a251f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 0x7f889c1a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a251f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f889c10e590 con 0x7f889c104480 2024-04-25T13:56:56.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a251f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f889c10e410 con 0x7f889c102fa0 2024-04-25T13:56:56.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a251f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f889c10e710 con 0x7f889c103980 2024-04-25T13:56:56.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a0abc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 0x7f889c1a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a0abc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 0x7f889c1a1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43714/0 (socket says 172.21.15.27:43714) 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f88a0abc640 1 -- 172.21.15.27:0/2080856308 learned_addr learned my addr 172.21.15.27:0/2080856308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f889b7fe640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f889c103980 0x7f889c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.959+0000 7f889bfff640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f889c102fa0 0x7f889c102240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88a0abc640 1 -- 172.21.15.27:0/2080856308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f889c103980 msgr2=0x7f889c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88a0abc640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f889c103980 0x7f889c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88a0abc640 1 -- 172.21.15.27:0/2080856308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f889c102fa0 msgr2=0x7f889c102240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88a0abc640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f889c102fa0 0x7f889c102240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88a0abc640 1 -- 172.21.15.27:0/2080856308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c1a1d20 con 0x7f889c104480 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f889bfff640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f889c102fa0 0x7f889c102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88a0abc640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 0x7f889c1a1500 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8894009430 tx=0x7f88940317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88997fa640 1 -- 172.21.15.27:0/2080856308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894003bf0 con 0x7f889c104480 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88997fa640 1 -- 172.21.15.27:0/2080856308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8894003d90 con 0x7f889c104480 2024-04-25T13:56:56.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889c1a1fb0 con 0x7f889c104480 2024-04-25T13:56:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88997fa640 1 -- 172.21.15.27:0/2080856308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889402fc80 con 0x7f889c104480 2024-04-25T13:56:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.960+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f889c1a2410 con 0x7f889c104480 2024-04-25T13:56:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.961+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f889c06a190 con 0x7f889c104480 2024-04-25T13:56:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.963+0000 7f88997fa640 1 -- 172.21.15.27:0/2080856308 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8894004710 con 0x7f889c104480 2024-04-25T13:56:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.963+0000 7f88997fa640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88880afaf0 0x7f88880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.963+0000 7f889bfff640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88880afaf0 0x7f88880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.963+0000 7f88997fa640 1 -- 172.21.15.27:0/2080856308 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8276+0+0 (secure 0 0 0) 0x7f88941004f0 con 0x7f889c104480 2024-04-25T13:56:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.964+0000 7f889bfff640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88880afaf0 0x7f88880b2000 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f889c06b290 tx=0x7f888c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:56.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:56.967+0000 7f88997fa640 1 -- 172.21.15.27:0/2080856308 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f889c06a190 con 0x7f889c104480 2024-04-25T13:56:57.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a[29136]: 2024-04-25T13:56:56.826+0000 7f6aeda2c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T13:56:57.182 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:56:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b[63598]: ignoring --setuser ceph since I am not root 2024-04-25T13:56:57.182 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:56:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b[63598]: ignoring --setgroup ceph since I am not root 2024-04-25T13:56:57.182 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:56:56 smithi045 ceph-mds[63602]: -- 172.21.15.45:0/1158493103 <== mon.2 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558fd44e9a40 con 0x558fd4531a80 2024-04-25T13:56:57.182 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:56:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b[63598]: starting mds.b at 2024-04-25T13:56:57.182 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:56:56 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i[64292]: ignoring --setuser ceph since I am not root 2024-04-25T13:56:57.183 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:56:56 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i[64292]: ignoring --setgroup ceph since I am not root 2024-04-25T13:56:57.183 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:56:56 smithi122 ceph-mds[64296]: -- 172.21.15.122:0/4162636868 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560ad3b3a40 con 0x5560ad3fba80 2024-04-25T13:56:57.183 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:56:56 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i[64292]: starting mds.i at 2024-04-25T13:56:57.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.200+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f889c02dbd0 con 0x7f889c104480 2024-04-25T13:56:57.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:57 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2949924029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:57.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:57 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3074420375' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T13:56:57.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:57 smithi122 ceph-mon[32843]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T13:56:57.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:57 smithi122 ceph-mon[32843]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T13:56:57.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:57 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/3074420375' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T13:56:57.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:57 smithi122 ceph-mon[32843]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T13:56:57.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:57 smithi122 ceph-mon[32843]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-25T13:56:57.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:57 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2949924029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:57.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:57 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3074420375' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T13:56:57.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:57 smithi045 ceph-mon[32731]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T13:56:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:57 smithi045 ceph-mon[32731]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T13:56:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:57 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/3074420375' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T13:56:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:57 smithi045 ceph-mon[32731]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T13:56:57.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:57 smithi045 ceph-mon[32731]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-25T13:56:57.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:57 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2949924029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T13:56:57.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:57 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3074420375' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T13:56:57.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:57 smithi027 ceph-mon[29140]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T13:56:57.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:57 smithi027 ceph-mon[29140]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T13:56:57.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:57 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/3074420375' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T13:56:57.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:57 smithi027 ceph-mon[29140]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T13:56:57.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:57 smithi027 ceph-mon[29140]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-25T13:56:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.830+0000 7f88997fa640 1 -- 172.21.15.27:0/2080856308 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f88940c90a0 con 0x7f889c104480 2024-04-25T13:56:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.833+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88880afaf0 msgr2=0x7f88880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.833+0000 7f88a251f640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88880afaf0 0x7f88880b2000 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f889c06b290 tx=0x7f888c005f90 comp rx=0 tx=0).stop 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.833+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 msgr2=0x7f889c1a1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.833+0000 7f88a251f640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 0x7f889c1a1500 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8894009430 tx=0x7f88940317c0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.834+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 shutdown_connections 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.834+0000 7f88a251f640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f88880afaf0 0x7f88880b2000 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.834+0000 7f88a251f640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f889c104480 0x7f889c1a1500 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.834+0000 7f88a251f640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f889c103980 0x7f889c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.834+0000 7f88a251f640 1 --2- 172.21.15.27:0/2080856308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f889c102fa0 0x7f889c102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.834+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 >> 172.21.15.27:0/2080856308 conn(0x7f889c0fd380 msgr2=0x7f889c106b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.834+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 shutdown_connections 2024-04-25T13:56:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.834+0000 7f88a251f640 1 -- 172.21.15.27:0/2080856308 wait complete. 2024-04-25T13:56:57.849 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T13:56:57.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.972+0000 7f8c73334640 1 -- 172.21.15.27:0/2913799005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c102250 msgr2=0x7f8c6c10bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:57.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.972+0000 7f8c73334640 1 --2- 172.21.15.27:0/2913799005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c102250 0x7f8c6c10bcb0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009ca0 tx=0x7f8c6402a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.972+0000 7f8c73334640 1 -- 172.21.15.27:0/2913799005 shutdown_connections 2024-04-25T13:56:57.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.972+0000 7f8c73334640 1 --2- 172.21.15.27:0/2913799005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c102250 0x7f8c6c10bcb0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.972+0000 7f8c73334640 1 --2- 172.21.15.27:0/2913799005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c6c101750 0x7f8c6c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.972+0000 7f8c73334640 1 --2- 172.21.15.27:0/2913799005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c6c1050a0 0x7f8c6c1054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.972+0000 7f8c73334640 1 -- 172.21.15.27:0/2913799005 >> 172.21.15.27:0/2913799005 conn(0x7f8c6c0fd350 msgr2=0x7f8c6c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 -- 172.21.15.27:0/2913799005 shutdown_connections 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 -- 172.21.15.27:0/2913799005 wait complete. 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 Processor -- start 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 -- start start 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c6c101750 0x7f8c6c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c6c102250 0x7f8c6c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c1050a0 0x7f8c6c1a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10f630 con 0x7f8c6c1050a0 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10f4b0 con 0x7f8c6c101750 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.973+0000 7f8c73334640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10f7b0 con 0x7f8c6c102250 2024-04-25T13:56:57.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c1050a0 0x7f8c6c1a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c1050a0 0x7f8c6c1a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43726/0 (socket says 172.21.15.27:43726) 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 -- 172.21.15.27:0/968429872 learned_addr learned my addr 172.21.15.27:0/968429872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c708cf640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c6c102250 0x7f8c6c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c710d0640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c6c101750 0x7f8c6c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 -- 172.21.15.27:0/968429872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c6c102250 msgr2=0x7f8c6c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c6c102250 0x7f8c6c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 -- 172.21.15.27:0/968429872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c6c101750 msgr2=0x7f8c6c19a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c6c101750 0x7f8c6c19a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 -- 172.21.15.27:0/968429872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6c1a1d00 con 0x7f8c6c1050a0 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c710d0640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c6c101750 0x7f8c6c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c718d1640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c1050a0 0x7f8c6c1a1690 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8c640097b0 tx=0x7f8c64031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c627fc640 1 -- 172.21.15.27:0/968429872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64003bf0 con 0x7f8c6c1050a0 2024-04-25T13:56:57.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c627fc640 1 -- 172.21.15.27:0/968429872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c64003d90 con 0x7f8c6c1050a0 2024-04-25T13:56:57.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c6c1a1f90 con 0x7f8c6c1050a0 2024-04-25T13:56:57.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c627fc640 1 -- 172.21.15.27:0/968429872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6402fc30 con 0x7f8c6c1050a0 2024-04-25T13:56:57.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.974+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c6c1a2450 con 0x7f8c6c1050a0 2024-04-25T13:56:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.975+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c6c10aa40 con 0x7f8c6c1050a0 2024-04-25T13:56:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.977+0000 7f8c627fc640 1 -- 172.21.15.27:0/968429872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8c64004040 con 0x7f8c6c1050a0 2024-04-25T13:56:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.978+0000 7f8c627fc640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8c580afb70 0x7f8c580b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.978+0000 7f8c710d0640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8c580afb70 0x7f8c580b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.978+0000 7f8c627fc640 1 -- 172.21.15.27:0/968429872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8276+0+0 (secure 0 0 0) 0x7f8c641004d0 con 0x7f8c6c1050a0 2024-04-25T13:56:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.978+0000 7f8c710d0640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8c580afb70 0x7f8c580b2080 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c06b290 tx=0x7f8c68005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:57.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:57.982+0000 7f8c627fc640 1 -- 172.21.15.27:0/968429872 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c640046a0 con 0x7f8c6c1050a0 2024-04-25T13:56:58.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.169+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8c6c02db50 con 0x7f8c6c1050a0 2024-04-25T13:56:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-25T13:56:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2080856308' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:56:58.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: Cluster is now healthy 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/2080856308' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.122:6838/1677660235,v1:172.21.15.122:6839/1677660235] up:boot 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: mds.? [v2:172.21.15.45:6838/2332645818,v1:172.21.15.45:6839/2332645818] up:boot 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: fsmap 12 up:standby 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T13:56:58.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:58 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/968429872' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T13:56:58.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-25T13:56:58.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2080856308' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:56:58.577 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: Cluster is now healthy 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/2080856308' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.122:6838/1677660235,v1:172.21.15.122:6839/1677660235] up:boot 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: mds.? [v2:172.21.15.45:6838/2332645818,v1:172.21.15.45:6839/2332645818] up:boot 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: fsmap 12 up:standby 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T13:56:58.578 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:58 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/968429872' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T13:56:58.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-25T13:56:58.650 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2080856308' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: Cluster is now healthy 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/2080856308' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.122:6838/1677660235,v1:172.21.15.122:6839/1677660235] up:boot 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: mds.? [v2:172.21.15.45:6838/2332645818,v1:172.21.15.45:6839/2332645818] up:boot 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: fsmap 12 up:standby 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: from='mgr.14152 172.21.15.27:0/3035764718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T13:56:58.651 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:58 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/968429872' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T13:56:58.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.831+0000 7f8c627fc640 1 -- 172.21.15.27:0/968429872 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v80) ==== 181+0+0 (secure 0 0 0) 0x7f8c640c9060 con 0x7f8c6c1050a0 2024-04-25T13:56:58.831 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-04-25T13:56:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.835+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8c580afb70 msgr2=0x7f8c580b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.835+0000 7f8c73334640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8c580afb70 0x7f8c580b2080 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c06b290 tx=0x7f8c68005e70 comp rx=0 tx=0).stop 2024-04-25T13:56:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.835+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c1050a0 msgr2=0x7f8c6c1a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.836+0000 7f8c73334640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c1050a0 0x7f8c6c1a1690 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8c640097b0 tx=0x7f8c64031840 comp rx=0 tx=0).stop 2024-04-25T13:56:58.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.836+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 shutdown_connections 2024-04-25T13:56:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.836+0000 7f8c73334640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7f8c580afb70 0x7f8c580b2080 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.836+0000 7f8c73334640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c6c1050a0 0x7f8c6c1a1690 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.836+0000 7f8c73334640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c6c102250 0x7f8c6c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.836+0000 7f8c73334640 1 --2- 172.21.15.27:0/968429872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c6c101750 0x7f8c6c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.836+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 >> 172.21.15.27:0/968429872 conn(0x7f8c6c0fd350 msgr2=0x7f8c6c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.837+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 shutdown_connections 2024-04-25T13:56:58.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.837+0000 7f8c73334640 1 -- 172.21.15.27:0/968429872 wait complete. 2024-04-25T13:56:58.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 -- 172.21.15.27:0/908013490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd801045b0 msgr2=0x7fcd801069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 --2- 172.21.15.27:0/908013490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd801045b0 0x7fcd801069d0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcd78009ca0 tx=0x7fcd7802a8c0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 -- 172.21.15.27:0/908013490 shutdown_connections 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 --2- 172.21.15.27:0/908013490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd80109870 0x7fcd8010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 --2- 172.21.15.27:0/908013490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd80106f10 0x7fcd80109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 --2- 172.21.15.27:0/908013490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd801045b0 0x7fcd801069d0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 -- 172.21.15.27:0/908013490 >> 172.21.15.27:0/908013490 conn(0x7fcd800fdbd0 msgr2=0x7fcd80100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 -- 172.21.15.27:0/908013490 shutdown_connections 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 -- 172.21.15.27:0/908013490 wait complete. 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 Processor -- start 2024-04-25T13:56:58.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.970+0000 7fcd871e3640 1 -- start start 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd871e3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd801045b0 0x7fcd80103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd871e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80106f10 0x7fcd801020a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd871e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd80109870 0x7fcd801025e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd871e3640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8010f5c0 con 0x7fcd80106f10 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd871e3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8010f440 con 0x7fcd80109870 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd871e3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8010f740 con 0x7fcd801045b0 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80106f10 0x7fcd801020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80106f10 0x7fcd801020a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43730/0 (socket says 172.21.15.27:43730) 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 -- 172.21.15.27:0/1816808526 learned_addr learned my addr 172.21.15.27:0/1816808526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd85780640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd80109870 0x7fcd801025e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd84f7f640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd801045b0 0x7fcd80103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 -- 172.21.15.27:0/1816808526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd801045b0 msgr2=0x7fcd80103a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:58.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd801045b0 0x7fcd80103a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 -- 172.21.15.27:0/1816808526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd80109870 msgr2=0x7fcd801025e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd80109870 0x7fcd801025e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 -- 172.21.15.27:0/1816808526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd80102d10 con 0x7fcd80106f10 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.971+0000 7fcd7ffff640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80106f10 0x7fcd801020a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c970 tx=0x7fcd7400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.972+0000 7fcd7dffb640 1 -- 172.21.15.27:0/1816808526 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74007bb0 con 0x7fcd80106f10 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.972+0000 7fcd7dffb640 1 -- 172.21.15.27:0/1816808526 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd74007d50 con 0x7fcd80106f10 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.972+0000 7fcd7dffb640 1 -- 172.21.15.27:0/1816808526 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7400f450 con 0x7fcd80106f10 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.972+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd801a42e0 con 0x7fcd80106f10 2024-04-25T13:56:58.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.972+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd801a4750 con 0x7fcd80106f10 2024-04-25T13:56:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.973+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd800ff6f0 con 0x7fcd80106f10 2024-04-25T13:56:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.974+0000 7fcd7dffb640 1 -- 172.21.15.27:0/1816808526 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcd74004f20 con 0x7fcd80106f10 2024-04-25T13:56:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.975+0000 7fcd7dffb640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcd680afb70 0x7fcd680b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T13:56:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.975+0000 7fcd84f7f640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcd680afb70 0x7fcd680b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T13:56:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.975+0000 7fcd7dffb640 1 -- 172.21.15.27:0/1816808526 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7743+0+0 (secure 0 0 0) 0x7fcd740de800 con 0x7fcd80106f10 2024-04-25T13:56:58.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.975+0000 7fcd84f7f640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcd680afb70 0x7fcd680b2080 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcd78031b50 tx=0x7fcd780093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T13:56:58.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:58.979+0000 7fcd7dffb640 1 -- 172.21.15.27:0/1816808526 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd74014030 con 0x7fcd80106f10 2024-04-25T13:56:59.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.168+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcd8002db50 con 0x7fcd80106f10 2024-04-25T13:56:59.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.838+0000 7fcd7dffb640 1 -- 172.21.15.27:0/1816808526 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v81) ==== 169+0+0 (secure 0 0 0) 0x7fcd740a75c0 con 0x7fcd80106f10 2024-04-25T13:56:59.838 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-04-25T13:56:59.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.842+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcd680afb70 msgr2=0x7fcd680b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:59.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.842+0000 7fcd871e3640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcd680afb70 0x7fcd680b2080 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcd78031b50 tx=0x7fcd780093f0 comp rx=0 tx=0).stop 2024-04-25T13:56:59.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.842+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80106f10 msgr2=0x7fcd801020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T13:56:59.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.842+0000 7fcd871e3640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80106f10 0x7fcd801020a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c970 tx=0x7fcd7400ce30 comp rx=0 tx=0).stop 2024-04-25T13:56:59.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.843+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 shutdown_connections 2024-04-25T13:56:59.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.843+0000 7fcd871e3640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:6800/1716439151,v1:172.21.15.27:6801/1716439151] conn(0x7fcd680afb70 0x7fcd680b2080 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:59.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.843+0000 7fcd871e3640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd80109870 0x7fcd801025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:59.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.843+0000 7fcd871e3640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd80106f10 0x7fcd801020a0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:59.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.843+0000 7fcd871e3640 1 --2- 172.21.15.27:0/1816808526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcd801045b0 0x7fcd80103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T13:56:59.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.843+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 >> 172.21.15.27:0/1816808526 conn(0x7fcd800fdbd0 msgr2=0x7fcd80105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T13:56:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.844+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 shutdown_connections 2024-04-25T13:56:59.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-25T13:56:59.844+0000 7fcd871e3640 1 -- 172.21.15.27:0/1816808526 wait complete. 2024-04-25T13:56:59.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:56:59.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:56:59.932 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:56:59.963 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T13:56:59.963 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T13:56:59.981 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T13:56:59.998 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T13:57:00.018 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T13:57:00.019 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T13:57:00.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.a 2024-04-25T13:57:00.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:59 smithi027 ceph-mon[29140]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 2 op/s 2024-04-25T13:57:00.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:59 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/968429872' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T13:57:00.150 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:59 smithi027 ceph-mon[29140]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T13:57:00.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:56:59 smithi027 ceph-mon[29140]: from='client.? 172.21.15.27:0/1816808526' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T13:57:00.151 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:57:00 smithi027 systemd[1]: Stopping Ceph mon.a for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:00.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:59 smithi122 ceph-mon[32843]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 2 op/s 2024-04-25T13:57:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:59 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/968429872' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T13:57:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:59 smithi122 ceph-mon[32843]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T13:57:00.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:56:59 smithi122 ceph-mon[32843]: from='client.? 172.21.15.27:0/1816808526' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T13:57:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:59 smithi045 ceph-mon[32731]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 2 op/s 2024-04-25T13:57:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:59 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/968429872' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T13:57:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:59 smithi045 ceph-mon[32731]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T13:57:00.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:56:59 smithi045 ceph-mon[32731]: from='client.? 172.21.15.27:0/1816808526' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T13:57:00.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:57:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a[29136]: 2024-04-25T13:57:00.415+0000 7f6af3237700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:00.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:57:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a[29136]: 2024-04-25T13:57:00.415+0000 7f6af3237700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T13:57:00.900 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:57:00 smithi027 podman[95596]: 2024-04-25 13:57:00.652224621 +0000 UTC m=+0.372501493 container died d3ac28c1a0c9b3cb569a0c68e9252c201386d0464c1ab1d7d9b17c9c3dc12225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:01.382 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:57:01 smithi027 podman[95596]: 2024-04-25 13:57:01.15750426 +0000 UTC m=+0.877781131 container cleanup d3ac28c1a0c9b3cb569a0c68e9252c201386d0464c1ab1d7d9b17c9c3dc12225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:57:01.383 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:57:01 smithi027 bash[95596]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a 2024-04-25T13:57:01.639 INFO:journalctl@ceph.mon.a.smithi027.stdout:Apr 25 13:57:01 smithi027 podman[95608]: 2024-04-25 13:57:01.384092244 +0000 UTC m=+0.732553364 container remove d3ac28c1a0c9b3cb569a0c68e9252c201386d0464c1ab1d7d9b17c9c3dc12225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T13:57:01.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T13:57:01.701 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:01.702 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T13:57:01.702 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T13:57:01.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.b 2024-04-25T13:57:02.041 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:57:01 smithi045 systemd[1]: Stopping Ceph mon.b for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:02.327 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:57:02 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-b[32727]: 2024-04-25T13:57:02.038+0000 7f0e8b2c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:02.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:57:02 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-b[32727]: 2024-04-25T13:57:02.038+0000 7f0e8b2c8700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T13:57:02.328 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:57:02 smithi045 podman[67957]: 2024-04-25 13:57:02.110603664 +0000 UTC m=+0.189480350 container died b396fa368b634dc01c985307591b7e6d34acdaaa0b60b9e2306900c82027f840 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:03.574 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:57:03 smithi045 podman[67957]: 2024-04-25 13:57:03.307327344 +0000 UTC m=+1.386204025 container cleanup b396fa368b634dc01c985307591b7e6d34acdaaa0b60b9e2306900c82027f840 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:57:03.574 INFO:journalctl@ceph.mon.b.smithi045.stdout:Apr 25 13:57:03 smithi045 bash[67957]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-b 2024-04-25T13:57:03.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T13:57:03.745 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:03.745 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T13:57:03.745 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T13:57:03.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.c 2024-04-25T13:57:04.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:57:03 smithi122 systemd[1]: Stopping Ceph mon.c for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:04.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:57:04 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c[32839]: 2024-04-25T13:57:04.199+0000 7f97cba8a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:04.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:57:04 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c[32839]: 2024-04-25T13:57:04.199+0000 7f97cba8a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T13:57:04.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:57:04 smithi122 podman[69144]: 2024-04-25 13:57:04.433666412 +0000 UTC m=+0.462871699 container died 3705a47103b402683d0da385748395bc402db88327a2f94f38865eb38cc0e678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T13:57:06.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:57:05 smithi122 podman[69144]: 2024-04-25 13:57:05.767252684 +0000 UTC m=+1.796457960 container cleanup 3705a47103b402683d0da385748395bc402db88327a2f94f38865eb38cc0e678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T13:57:06.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:57:05 smithi122 bash[69144]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c 2024-04-25T13:57:06.058 INFO:journalctl@ceph.mon.c.smithi122.stdout:Apr 25 13:57:06 smithi122 podman[69157]: 2024-04-25 13:57:06.002505762 +0000 UTC m=+1.569571038 container remove 3705a47103b402683d0da385748395bc402db88327a2f94f38865eb38cc0e678 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T13:57:06.247 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T13:57:06.296 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:06.297 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T13:57:06.297 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T13:57:06.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.x 2024-04-25T13:57:06.620 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:57:06 smithi027 systemd[1]: Stopping Ceph mgr.x for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:06.900 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:57:06 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:57:06.619+0000 7f107e3ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:06.900 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:57:06 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x[29380]: 2024-04-25T13:57:06.619+0000 7f107e3ac700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-25T13:57:06.900 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:57:06 smithi027 podman[95716]: 2024-04-25 13:57:06.650989744 +0000 UTC m=+0.155693843 container died e2e0c7abc52382931cf5e11ed4a1e221850cfe21c41e0a6c098882ff870501a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:07.400 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:57:07 smithi027 podman[95716]: 2024-04-25 13:57:07.154173027 +0000 UTC m=+0.658877126 container cleanup e2e0c7abc52382931cf5e11ed4a1e221850cfe21c41e0a6c098882ff870501a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T13:57:07.400 INFO:journalctl@ceph.mgr.x.smithi027.stdout:Apr 25 13:57:07 smithi027 bash[95716]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-x 2024-04-25T13:57:07.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T13:57:07.626 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:07.626 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T13:57:07.626 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T13:57:07.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.y 2024-04-25T13:57:07.973 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:57:07 smithi045 systemd[1]: Stopping Ceph mgr.y for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:08.327 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:57:07 smithi045 podman[68080]: 2024-04-25 13:57:07.974654004 +0000 UTC m=+0.148366316 container died d464e8b57aed8a456891f62e48f8cfbb434f178f6fa6eac35de0dbeb3a8ef1be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS) 2024-04-25T13:57:08.828 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:57:08 smithi045 podman[68080]: 2024-04-25 13:57:08.445858906 +0000 UTC m=+0.619571224 container cleanup d464e8b57aed8a456891f62e48f8cfbb434f178f6fa6eac35de0dbeb3a8ef1be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:57:08.828 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Apr 25 13:57:08 smithi045 bash[68080]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-y 2024-04-25T13:57:09.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T13:57:09.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:09.067 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T13:57:09.067 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T13:57:09.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.z 2024-04-25T13:57:09.408 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:57:09 smithi122 systemd[1]: Stopping Ceph mgr.z for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:09.714 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:57:09 smithi122 podman[69264]: 2024-04-25 13:57:09.409771231 +0000 UTC m=+0.147236798 container died 2068ea4a75cf2b1415b87d71dff2cd339c1b37c67ab103db4f01c77f7528cda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:57:10.057 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:57:09 smithi122 podman[69264]: 2024-04-25 13:57:09.814827084 +0000 UTC m=+0.552292652 container cleanup 2068ea4a75cf2b1415b87d71dff2cd339c1b37c67ab103db4f01c77f7528cda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T13:57:10.058 INFO:journalctl@ceph.mgr.z.smithi122.stdout:Apr 25 13:57:09 smithi122 bash[69264]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mgr-z 2024-04-25T13:57:10.195 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T13:57:10.227 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:10.227 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T13:57:10.228 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T13:57:10.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.0 2024-04-25T13:57:10.540 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:10 smithi027 systemd[1]: Stopping Ceph osd.0 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:10.900 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:10 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0[41693]: 2024-04-25T13:57:10.541+0000 7fdae89e3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:10.900 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:10 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0[41693]: 2024-04-25T13:57:10.541+0000 7fdae89e3700 -1 osd.0 81 *** Got signal Terminated *** 2024-04-25T13:57:10.900 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:10 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0[41693]: 2024-04-25T13:57:10.541+0000 7fdae89e3700 -1 osd.0 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:57:16.292 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:16 smithi027 podman[95840]: 2024-04-25 13:57:16.019829949 +0000 UTC m=+5.594866440 container died a8527e9e426e68b01d1c8804e15f19cee90a840b21e96b5c3201c5dd09d7a435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-25T13:57:16.642 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:16 smithi027 podman[95840]: 2024-04-25 13:57:16.39196724 +0000 UTC m=+5.967003726 container cleanup a8527e9e426e68b01d1c8804e15f19cee90a840b21e96b5c3201c5dd09d7a435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T13:57:16.642 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:16 smithi027 bash[95840]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0 2024-04-25T13:57:16.900 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:16 smithi027 podman[95857]: 2024-04-25 13:57:16.643161485 +0000 UTC m=+0.622529132 container remove a8527e9e426e68b01d1c8804e15f19cee90a840b21e96b5c3201c5dd09d7a435 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T13:57:17.152 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:16 smithi027 podman[95946]: 2024-04-25 13:57:16.868187884 +0000 UTC m=+0.018396204 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:57:17.152 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:17 smithi027 podman[95946]: 2024-04-25 13:57:17.011204995 +0000 UTC m=+0.161413342 container create 632b0d9b066e8479a61689eadb6b9e4aae8b01b103501d9c576c4a98c16d17c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-25T13:57:17.446 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:17 smithi027 podman[95946]: 2024-04-25 13:57:17.372925866 +0000 UTC m=+0.523134176 container init 632b0d9b066e8479a61689eadb6b9e4aae8b01b103501d9c576c4a98c16d17c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T13:57:17.447 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:17 smithi027 podman[95946]: 2024-04-25 13:57:17.376581375 +0000 UTC m=+0.526789687 container start 632b0d9b066e8479a61689eadb6b9e4aae8b01b103501d9c576c4a98c16d17c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T13:57:17.447 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:17 smithi027 podman[95946]: 2024-04-25 13:57:17.447374797 +0000 UTC m=+0.597583123 container attach 632b0d9b066e8479a61689eadb6b9e4aae8b01b103501d9c576c4a98c16d17c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:17.900 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 25 13:57:17 smithi027 podman[95946]: 2024-04-25 13:57:17.684781654 +0000 UTC m=+0.834989967 container died 632b0d9b066e8479a61689eadb6b9e4aae8b01b103501d9c576c4a98c16d17c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:18.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T13:57:18.386 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:18.386 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T13:57:18.386 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T13:57:18.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.1 2024-04-25T13:57:18.700 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:18 smithi045 systemd[1]: Stopping Ceph osd.1 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:18.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1[37314]: 2024-04-25T13:57:18.700+0000 7fbea0250700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:18.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1[37314]: 2024-04-25T13:57:18.700+0000 7fbea0250700 -1 osd.1 81 *** Got signal Terminated *** 2024-04-25T13:57:18.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1[37314]: 2024-04-25T13:57:18.700+0000 7fbea0250700 -1 osd.1 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:57:24.534 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:24 smithi045 podman[68208]: 2024-04-25 13:57:24.247893442 +0000 UTC m=+5.667724184 container died fb9efe77cef8baf1a29dd3671a342f4bbd12be38f32553abc8f44d3364f85506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:24.827 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:24 smithi045 podman[68208]: 2024-04-25 13:57:24.633725294 +0000 UTC m=+6.053556038 container cleanup fb9efe77cef8baf1a29dd3671a342f4bbd12be38f32553abc8f44d3364f85506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True) 2024-04-25T13:57:24.828 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:24 smithi045 bash[68208]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1 2024-04-25T13:57:25.270 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:24 smithi045 podman[68237]: 2024-04-25 13:57:24.910099488 +0000 UTC m=+0.662840779 container remove fb9efe77cef8baf1a29dd3671a342f4bbd12be38f32553abc8f44d3364f85506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T13:57:25.578 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:25 smithi045 podman[68328]: 2024-04-25 13:57:25.173854582 +0000 UTC m=+0.018716355 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:57:25.578 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:25 smithi045 podman[68328]: 2024-04-25 13:57:25.278040675 +0000 UTC m=+0.122902436 container create 1e0e9db8aa97467e2a7f84c8ba9d1614ce002ad6668b3dfc2df4daf2a99bf180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T13:57:25.578 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:25 smithi045 podman[68328]: 2024-04-25 13:57:25.514242198 +0000 UTC m=+0.359103959 container init 1e0e9db8aa97467e2a7f84c8ba9d1614ce002ad6668b3dfc2df4daf2a99bf180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T13:57:25.578 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:25 smithi045 podman[68328]: 2024-04-25 13:57:25.517521884 +0000 UTC m=+0.362383645 container start 1e0e9db8aa97467e2a7f84c8ba9d1614ce002ad6668b3dfc2df4daf2a99bf180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T13:57:26.078 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:25 smithi045 podman[68328]: 2024-04-25 13:57:25.614312518 +0000 UTC m=+0.459174291 container attach 1e0e9db8aa97467e2a7f84c8ba9d1614ce002ad6668b3dfc2df4daf2a99bf180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T13:57:26.078 INFO:journalctl@ceph.osd.1.smithi045.stdout:Apr 25 13:57:25 smithi045 podman[68328]: 2024-04-25 13:57:25.820785707 +0000 UTC m=+0.665647467 container died 1e0e9db8aa97467e2a7f84c8ba9d1614ce002ad6668b3dfc2df4daf2a99bf180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:57:26.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T13:57:26.635 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:26.635 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T13:57:26.635 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T13:57:26.636 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.2 2024-04-25T13:57:26.951 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:26 smithi122 systemd[1]: Stopping Ceph osd.2 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:27.307 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:26 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2[37980]: 2024-04-25T13:57:26.951+0000 7f9521ab0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:27.308 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:26 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2[37980]: 2024-04-25T13:57:26.951+0000 7f9521ab0700 -1 osd.2 81 *** Got signal Terminated *** 2024-04-25T13:57:27.308 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:26 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2[37980]: 2024-04-25T13:57:26.951+0000 7f9521ab0700 -1 osd.2 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:57:32.769 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:32 smithi122 podman[69389]: 2024-04-25 13:57:32.458289392 +0000 UTC m=+5.627657832 container died 68daadcd3a72c943976f00bf93b53fec112fb28c4b87c408b95d8fc44b273b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:57:33.057 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:32 smithi122 podman[69389]: 2024-04-25 13:57:32.869526953 +0000 UTC m=+6.038895368 container cleanup 68daadcd3a72c943976f00bf93b53fec112fb28c4b87c408b95d8fc44b273b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T13:57:33.058 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:32 smithi122 bash[69389]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2 2024-04-25T13:57:33.414 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:33 smithi122 podman[69418]: 2024-04-25 13:57:33.104303506 +0000 UTC m=+0.647054741 container remove 68daadcd3a72c943976f00bf93b53fec112fb28c4b87c408b95d8fc44b273b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T13:57:33.707 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:33 smithi122 podman[69510]: 2024-04-25 13:57:33.318041794 +0000 UTC m=+0.018505031 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:57:33.707 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:33 smithi122 podman[69510]: 2024-04-25 13:57:33.430187788 +0000 UTC m=+0.130651012 container create ba9382978a0cff200c7ef250494c50e3aed8c191c42ef4c67902bad94b0669a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T13:57:33.707 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:33 smithi122 podman[69510]: 2024-04-25 13:57:33.708243723 +0000 UTC m=+0.408706953 container init ba9382978a0cff200c7ef250494c50e3aed8c191c42ef4c67902bad94b0669a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T13:57:34.005 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:33 smithi122 podman[69510]: 2024-04-25 13:57:33.711948446 +0000 UTC m=+0.412411667 container start ba9382978a0cff200c7ef250494c50e3aed8c191c42ef4c67902bad94b0669a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T13:57:34.005 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:33 smithi122 podman[69510]: 2024-04-25 13:57:33.816523333 +0000 UTC m=+0.516986572 container attach ba9382978a0cff200c7ef250494c50e3aed8c191c42ef4c67902bad94b0669a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:57:34.307 INFO:journalctl@ceph.osd.2.smithi122.stdout:Apr 25 13:57:34 smithi122 podman[69510]: 2024-04-25 13:57:34.006016263 +0000 UTC m=+0.706479489 container died ba9382978a0cff200c7ef250494c50e3aed8c191c42ef4c67902bad94b0669a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T13:57:34.704 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T13:57:34.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:34.743 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T13:57:34.744 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T13:57:34.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.3 2024-04-25T13:57:35.044 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:34 smithi027 systemd[1]: Stopping Ceph osd.3 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:35.401 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:35 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3[48395]: 2024-04-25T13:57:35.044+0000 7f7f81eb9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:35.401 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:35 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3[48395]: 2024-04-25T13:57:35.044+0000 7f7f81eb9700 -1 osd.3 81 *** Got signal Terminated *** 2024-04-25T13:57:35.401 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:35 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3[48395]: 2024-04-25T13:57:35.044+0000 7f7f81eb9700 -1 osd.3 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:57:36.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:36 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:36.318+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:37.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:37 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:37.291+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:37.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:37 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:37.530+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:38.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:37 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:37.947+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:38.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:38 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:38.309+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:38.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:38.664+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:38.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:38.516+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:39.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:38 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:38.987+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:39.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:39 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:39.327+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:39.400 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:39 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3[48395]: 2024-04-25T13:57:39.106+0000 7f7f7a467700 -1 osd.3 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:13.619127+0000 front 2024-04-25T13:57:13.619179+0000 (oldest deadline 2024-04-25T13:57:38.919374+0000) 2024-04-25T13:57:39.807 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:39.647+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:39.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:39.470+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:40.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:39 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:39.985+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:40.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:40 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:40.278+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:40.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:40 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:40.299+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:40.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:40.518+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:40.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:40.652+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:41.070 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:40 smithi027 podman[96057]: 2024-04-25 13:57:40.668166176 +0000 UTC m=+5.717977404 container died 74f63193d5622060e3da85f612bca06748c4b4131235c220b1f253036dcc0de7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:57:41.329 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:40 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:40.948+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:41.400 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:41 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:41.281+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:41.400 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:41 smithi027 podman[96057]: 2024-04-25 13:57:41.162201637 +0000 UTC m=+6.212012852 container cleanup 74f63193d5622060e3da85f612bca06748c4b4131235c220b1f253036dcc0de7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:41.401 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:41 smithi027 bash[96057]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3 2024-04-25T13:57:41.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:41 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:41.237+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:41.744 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:41 smithi027 podman[96088]: 2024-04-25 13:57:41.413326252 +0000 UTC m=+0.740800677 container remove 74f63193d5622060e3da85f612bca06748c4b4131235c220b1f253036dcc0de7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T13:57:41.807 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:41 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:41.493+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:41.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:41 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:41.678+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:42.065 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:41 smithi027 podman[96179]: 2024-04-25 13:57:41.648348496 +0000 UTC m=+0.018545577 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:57:42.066 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:41 smithi027 podman[96179]: 2024-04-25 13:57:41.764537449 +0000 UTC m=+0.134734503 container create 70368aa63952ac4f851ec028f4e3cd2fca6e78a8f55e7ca16abaf84341233de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T13:57:42.066 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:41 smithi027 podman[96179]: 2024-04-25 13:57:41.991850063 +0000 UTC m=+0.362047118 container init 70368aa63952ac4f851ec028f4e3cd2fca6e78a8f55e7ca16abaf84341233de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T13:57:42.066 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:41 smithi027 podman[96179]: 2024-04-25 13:57:41.995106329 +0000 UTC m=+0.365303405 container start 70368aa63952ac4f851ec028f4e3cd2fca6e78a8f55e7ca16abaf84341233de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:57:42.066 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:42 smithi027 podman[96179]: 2024-04-25 13:57:42.066862457 +0000 UTC m=+0.437059522 container attach 70368aa63952ac4f851ec028f4e3cd2fca6e78a8f55e7ca16abaf84341233de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:57:42.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:41 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:41.955+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:42.400 INFO:journalctl@ceph.osd.3.smithi027.stdout:Apr 25 13:57:42 smithi027 podman[96179]: 2024-04-25 13:57:42.298472096 +0000 UTC m=+0.668669168 container died 70368aa63952ac4f851ec028f4e3cd2fca6e78a8f55e7ca16abaf84341233de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T13:57:42.401 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:42 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:42.269+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:42.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:42 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:42.203+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:42.722 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:42.467+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:42.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T13:57:42.897 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:42.897 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T13:57:42.897 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T13:57:42.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.4 2024-04-25T13:57:43.057 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:42.722+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:43.220 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:42 smithi045 systemd[1]: Stopping Ceph osd.4 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:43.220 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:42 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:42.996+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:43.577 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:43 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4[44261]: 2024-04-25T13:57:43.220+0000 7fbb55bcf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:43.577 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:43 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4[44261]: 2024-04-25T13:57:43.220+0000 7fbb55bcf700 -1 osd.4 81 *** Got signal Terminated *** 2024-04-25T13:57:43.577 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:43 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4[44261]: 2024-04-25T13:57:43.220+0000 7fbb55bcf700 -1 osd.4 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:57:43.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:43 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:43.236+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:43.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:43 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:43.218+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:43.772 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:43.508+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:44.057 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:43.773+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:44.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:44 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:44.034+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:44.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:44 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:44.257+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:44.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:44 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:44.218+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:44.807 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:44 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:44.518+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:44.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:44 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:44.732+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:45.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:45 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:45.066+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:45.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:45 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:45.066+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:45.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:45 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:45.249+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:45.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:45 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:45.216+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:45.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:45 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:45.216+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:45.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:45 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:45.497+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:45.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:45 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:45.741+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:45.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:45 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:45.741+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:45.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:45 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:45.512+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:46.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:46.059+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:46.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:46.059+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:46.615 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:46 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:46.286+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:46.615 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:46 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:46.286+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:46.615 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:46 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:46.260+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:46.615 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:46 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:46.260+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:46.769 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:46 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:46.450+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:46.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:46 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:46.528+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:47.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:46 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:46.784+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:47.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:46 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:46.770+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:47.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:46 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:46.770+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:47.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:47 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:47.049+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:47.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:47 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:47.049+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:47.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:47.285+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:47.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:47.285+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:47.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:47.268+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:47.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:47.268+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:47.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:47 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:47.433+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:47.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:47 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:47.519+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:48.057 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:47 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:47.785+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:48.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:47 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:47.757+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:48.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:47 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:47.757+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:48.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:48 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:48.045+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:48.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:48 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:48.045+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:48.614 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:48 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:48.530+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:48.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:48.283+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:48.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:48.283+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:48.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:48.307+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:48.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:48 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:48.307+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:48.770 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:48 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:48.429+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:48.770 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:48 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:48.429+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.755404+0000 front 2024-04-25T13:57:21.755314+0000 (oldest deadline 2024-04-25T13:57:47.555720+0000) 2024-04-25T13:57:48.887 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:48 smithi045 podman[68433]: 2024-04-25 13:57:48.614635456 +0000 UTC m=+5.514846070 container died 2b8f3d445f647d895960b5c9c31d95c5947264e259501e59c9ec437cf8b93a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T13:57:49.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:48 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:48.803+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:49.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:48 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:48.771+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:49.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:48 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:48.771+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:49.263 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:49 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:49.019+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:49.263 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:49 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:49.019+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:49.263 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:48 smithi045 podman[68433]: 2024-04-25 13:57:48.987548521 +0000 UTC m=+5.887759142 container cleanup 2b8f3d445f647d895960b5c9c31d95c5947264e259501e59c9ec437cf8b93a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T13:57:49.263 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:48 smithi045 bash[68433]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4 2024-04-25T13:57:49.516 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:49 smithi045 podman[68451]: 2024-04-25 13:57:49.263873827 +0000 UTC m=+0.647071037 container remove 2b8f3d445f647d895960b5c9c31d95c5947264e259501e59c9ec437cf8b93a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:49.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:49.238+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:49.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:49.238+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:49.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:49.276+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:49.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:49 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:49.276+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:49.795 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:49 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:49.472+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:49.795 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:49 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:49.472+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.755404+0000 front 2024-04-25T13:57:21.755314+0000 (oldest deadline 2024-04-25T13:57:47.555720+0000) 2024-04-25T13:57:49.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:49 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:49.516+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:49.828 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:49 smithi045 podman[68542]: 2024-04-25 13:57:49.60624658 +0000 UTC m=+0.111984823 container create b6174e52697beb34a5b453e538bb81ffc0c4639ce5df39ff78483e589c3b41b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T13:57:49.828 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:49 smithi045 podman[68542]: 2024-04-25 13:57:49.512816382 +0000 UTC m=+0.018554647 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:57:50.057 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:49 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:49.808+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:50.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:49 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:49.796+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:50.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:49 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:49.796+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:50.185 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:50 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:50.054+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:50.185 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:50 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:50.054+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:50.185 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:49 smithi045 podman[68542]: 2024-04-25 13:57:49.876711711 +0000 UTC m=+0.382449962 container init b6174e52697beb34a5b453e538bb81ffc0c4639ce5df39ff78483e589c3b41b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T13:57:50.185 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:49 smithi045 podman[68542]: 2024-04-25 13:57:49.880025439 +0000 UTC m=+0.385763690 container start b6174e52697beb34a5b453e538bb81ffc0c4639ce5df39ff78483e589c3b41b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:57:50.185 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:49 smithi045 podman[68542]: 2024-04-25 13:57:49.959139099 +0000 UTC m=+0.464877364 container attach b6174e52697beb34a5b453e538bb81ffc0c4639ce5df39ff78483e589c3b41b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T13:57:50.527 INFO:journalctl@ceph.osd.4.smithi045.stdout:Apr 25 13:57:50 smithi045 podman[68542]: 2024-04-25 13:57:50.186117972 +0000 UTC m=+0.691856221 container died b6174e52697beb34a5b453e538bb81ffc0c4639ce5df39ff78483e589c3b41b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T13:57:50.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:50 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:50.314+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:50.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:50 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:50.314+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:50.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:50 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:50.218+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:50.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:50 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:50.218+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:50.807 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:50 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:50.512+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:50.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:50 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:50.512+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.755404+0000 front 2024-04-25T13:57:21.755314+0000 (oldest deadline 2024-04-25T13:57:47.555720+0000) 2024-04-25T13:57:50.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:50 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:50.529+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:50.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T13:57:50.908 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:50.909 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T13:57:50.909 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T13:57:50.909 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.5 2024-04-25T13:57:51.216 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:50 smithi122 systemd[1]: Stopping Ceph osd.5 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:51.216 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:50 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:50.812+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:51.217 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:50 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:50.824+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:51.217 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:50 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:50.824+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:51.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:51 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:51.064+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:51.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:51 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:51.064+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:51.474 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:51 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:51.217+0000 7ff70f304700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:51.474 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:51 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:51.217+0000 7ff70f304700 -1 osd.5 81 *** Got signal Terminated *** 2024-04-25T13:57:51.474 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:51 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:51.217+0000 7ff70f304700 -1 osd.5 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:57:51.474 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:51 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:51.475+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:51.474 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:51 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:51.475+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.755404+0000 front 2024-04-25T13:57:21.755314+0000 (oldest deadline 2024-04-25T13:57:47.555720+0000) 2024-04-25T13:57:51.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:51 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:51.268+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:51.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:51 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:51.268+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:51.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:51 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:51.308+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:51.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:51 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:51.308+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:51.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:51 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:51.505+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:52.057 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:51 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:51.845+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:52.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:51 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:51.845+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:52.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:51 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:51.768+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:52.489 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:52 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:52.106+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:52.489 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:52 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:52.106+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:52.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:52 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:52.294+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:52.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:52 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:52.294+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:52.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:52 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:52.352+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:52.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:52 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:52.352+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:52.721 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:52 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:52.452+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:52.721 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:52 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:52.452+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.755404+0000 front 2024-04-25T13:57:21.755314+0000 (oldest deadline 2024-04-25T13:57:47.555720+0000) 2024-04-25T13:57:52.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:52 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:52.489+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:53.057 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:52 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:52.721+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:53.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:52 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:52.855+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:53.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:52 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:52.855+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:53.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:52 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:52.855+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:57:53.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:53 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:53.067+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:53.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:53 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:53.067+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:53.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:53 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:53.337+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:53.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:53 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:53.337+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:53.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:53 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:53.334+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:53.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:53 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:53.334+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:53.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:53 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:53.334+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:57:53.721 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:53 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:53.432+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:53.721 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:53 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:53.432+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.755404+0000 front 2024-04-25T13:57:21.755314+0000 (oldest deadline 2024-04-25T13:57:47.555720+0000) 2024-04-25T13:57:53.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:53 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:53.470+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:54.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:53 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:53.722+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:54.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:53 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:53.885+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:54.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:53 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:53.885+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:54.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:53 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:53.885+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:57:54.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:54 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:54.062+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:54.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:54 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:54.062+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:54.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:54 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:54.366+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:54.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:54 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:54.366+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:54.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:54 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:54.366+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:57:54.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:54 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:54.320+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:54.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:54 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:54.320+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:54.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:54 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:54.320+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:57:54.808 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:54 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:54.678+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:54.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:54 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:54.432+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:54.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:54 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:54.432+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.755404+0000 front 2024-04-25T13:57:21.755314+0000 (oldest deadline 2024-04-25T13:57:47.555720+0000) 2024-04-25T13:57:54.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:54 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:54.432+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.056751+0000 front 2024-04-25T13:57:31.056726+0000 (oldest deadline 2024-04-25T13:57:53.857412+0000) 2024-04-25T13:57:54.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:54 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:54.472+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:54.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:54 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:54.472+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:57:55.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:54 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:54.921+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:55.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:54 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:54.921+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:55.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:54 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:54.921+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:57:55.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:55 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:55.025+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:55.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:55 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:55.025+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:55.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:55 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:55.025+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:57:55.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:55 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:55.372+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:55.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:55 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:55.372+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:55.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:55 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:55.372+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:57:55.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:55 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:55.305+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:55.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:55 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:55.305+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:55.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:55 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:55.305+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:57:55.807 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:55 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:55.654+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:55.808 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:55 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:55.654+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:57:55.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:55 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:55.437+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.352654+0000 front 2024-04-25T13:57:11.352757+0000 (oldest deadline 2024-04-25T13:57:37.252752+0000) 2024-04-25T13:57:55.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:55 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:55.437+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.755404+0000 front 2024-04-25T13:57:21.755314+0000 (oldest deadline 2024-04-25T13:57:47.555720+0000) 2024-04-25T13:57:55.808 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:55 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5[44660]: 2024-04-25T13:57:55.437+0000 7ff70b8ba700 -1 osd.5 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.056751+0000 front 2024-04-25T13:57:31.056726+0000 (oldest deadline 2024-04-25T13:57:53.857412+0000) 2024-04-25T13:57:55.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:55 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:55.467+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:55.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:55 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:55.467+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:57:56.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:55 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:55.953+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:56.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:55 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:55.953+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:56.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:55 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:55.953+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:57:56.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:56.014+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:56.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:56.014+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:56.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:56.014+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:57:56.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:56.405+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:56.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:56.405+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:56.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:56.405+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:57:56.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:56.285+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:56.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:56.285+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:56.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:56.285+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:57:56.828 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:56.420+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:56.828 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:56.420+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:57:57.033 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:56 smithi122 podman[69618]: 2024-04-25 13:57:56.701241619 +0000 UTC m=+5.600179327 container died 9082a834a57b7d357af765987fd50cb6fa19b152baf8e6b06fc2ac4e2be8b502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:57:57.034 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:56 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:56.689+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:57.034 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:56 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:56.689+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:57:57.034 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:56 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:56.909+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:57.034 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:56 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:56.909+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:57.034 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:56 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:56.909+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:57:57.308 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:57 smithi122 podman[69618]: 2024-04-25 13:57:57.134119337 +0000 UTC m=+6.033057041 container cleanup 9082a834a57b7d357af765987fd50cb6fa19b152baf8e6b06fc2ac4e2be8b502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T13:57:57.308 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:57 smithi122 bash[69618]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5 2024-04-25T13:57:57.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:56.977+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:57.329 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:56.977+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:57.329 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:56 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:56.977+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:57:57.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:57 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:57.389+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:57.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:57 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:57.389+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:57.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:57 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:57.389+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:57:57.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:57 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:57.327+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:57.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:57 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:57.327+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:57.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:57 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:57.327+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:57:57.710 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:57 smithi122 podman[69634]: 2024-04-25 13:57:57.402509854 +0000 UTC m=+0.696510047 container remove 9082a834a57b7d357af765987fd50cb6fa19b152baf8e6b06fc2ac4e2be8b502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T13:57:57.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:57 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:57.448+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:57.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:57 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:57.448+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:57:57.975 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:57 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:57.711+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:57.975 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:57 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:57.711+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:57:57.976 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:57 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:57.958+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:57.976 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:57 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:57.958+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:57.976 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:57 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:57.958+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:57:57.976 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:57 smithi122 podman[69729]: 2024-04-25 13:57:57.705132487 +0000 UTC m=+0.018287987 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:57:57.976 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:57 smithi122 podman[69729]: 2024-04-25 13:57:57.836584097 +0000 UTC m=+0.149739592 container create 26e44dc176c0b858ac4c27ef70f2fbc216091418314da85629d9ae373557c4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T13:57:58.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:57 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:57.989+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:58.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:57 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:57.989+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:58.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:57 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:57.989+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:57:58.384 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:58 smithi122 podman[69729]: 2024-04-25 13:57:58.089780932 +0000 UTC m=+0.402936419 container init 26e44dc176c0b858ac4c27ef70f2fbc216091418314da85629d9ae373557c4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T13:57:58.384 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:58 smithi122 podman[69729]: 2024-04-25 13:57:58.092840042 +0000 UTC m=+0.405995531 container start 26e44dc176c0b858ac4c27ef70f2fbc216091418314da85629d9ae373557c4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:57:58.385 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:58 smithi122 podman[69729]: 2024-04-25 13:57:58.189728732 +0000 UTC m=+0.502884232 container attach 26e44dc176c0b858ac4c27ef70f2fbc216091418314da85629d9ae373557c4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T13:57:58.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:58.358+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:58.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:58.358+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:58.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:58.358+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:57:58.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:58.363+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:58.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:58.363+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:58.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:58 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:58.363+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:57:58.684 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 25 13:57:58 smithi122 podman[69729]: 2024-04-25 13:57:58.386003185 +0000 UTC m=+0.699158666 container died 26e44dc176c0b858ac4c27ef70f2fbc216091418314da85629d9ae373557c4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T13:57:58.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:58 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:58.408+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:58.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:58 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:58.408+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:57:58.934 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:58 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:58.720+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:57:58.935 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:58 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:58.720+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:57:59.038 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T13:57:59.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:57:59.074 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T13:57:59.074 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T13:57:59.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.6 2024-04-25T13:57:59.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:58 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:58.935+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:57:59.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:58 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:58.935+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:57:59.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:58 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:58.935+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:57:59.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:59 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:59.030+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:57:59.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:59 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:59.030+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:57:59.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:57:59 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:57:59.030+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:57:59.364 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:59 smithi027 systemd[1]: Stopping Ceph osd.6 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:57:59.364 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:59.320+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:57:59.364 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:59.320+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:57:59.364 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:57:59.320+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:57:59.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:59.364+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:57:59.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:59.364+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:57:59.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:59.364+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:57:59.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:59.398+0000 7f1ced7be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:57:59.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:59.398+0000 7f1ced7be700 -1 osd.6 81 *** Got signal Terminated *** 2024-04-25T13:57:59.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:57:59 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:57:59.398+0000 7f1ced7be700 -1 osd.6 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:57:59.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:59 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:59.412+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:57:59.828 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:57:59 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:57:59.412+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:00.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:59 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:59.748+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:00.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:57:59 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:57:59.748+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:00.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:59 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:59.924+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:00.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:59 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:59.924+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:00.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:57:59 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:57:59.924+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:00.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:00 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:00.067+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:00.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:00 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:00.067+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:00.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:00 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:00.067+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:00.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:00.347+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:58:00.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:00.347+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:58:00.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:00.347+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:58:00.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:00.287+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:00.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:00.287+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:00.651 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:00 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:00.287+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:00.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:00 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:00.381+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:00.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:00 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:00.381+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:01.057 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:00 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:00.701+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:01.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:00 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:00.701+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:01.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:00 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:00.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:01.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:00 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:00.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:01.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:00 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:00.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:01.352 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:01 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:01.080+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:01.353 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:01 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:01.080+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:01.353 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:01 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:01.080+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:01.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:01.241+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:01.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:01.241+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:01.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:01.241+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:01.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:01.380+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:58:01.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:01.380+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:58:01.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:01 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:01.380+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:58:01.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:01 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:01.352+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:01.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:01 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:01.352+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:01.993 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:01 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:01.738+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:01.993 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:01 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:01.738+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:02.307 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:01 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:01.993+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:02.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:01 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:01.993+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:02.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:01 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:01.993+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:02.392 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:02 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:02.106+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:02.392 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:02 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:02.106+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:02.393 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:02 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:02.106+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:02.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:02.200+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:02.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:02.200+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:02.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:02.200+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:02.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:02.343+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:58:02.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:02.343+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:58:02.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:02.343+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:58:02.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:02 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:02.393+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:02.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:02 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:02.393+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:03.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:02 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:02.989+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:03.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:02 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:02.989+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:03.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:02 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:02.989+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:03.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:02 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:02.758+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:03.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:02 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:02.758+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:03.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:02 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:02.758+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:03.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:03 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:03.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:03.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:03 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:03.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:03.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:03 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:03.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:03.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:03 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:03.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:03.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:03.208+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:03.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:03.208+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:03.650 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:03.208+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:03.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:03.342+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:58:03.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:03.342+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:58:03.651 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:03 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:03.342+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:58:03.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:03 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:03.401+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:03.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:03 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:03.401+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:04.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:03 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:03.808+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:04.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:03 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:03.808+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:04.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:03 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:03.808+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:04.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:03 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:03.948+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:04.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:03 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:03.948+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:04.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:03 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:03.948+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:04.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:04.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:04.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:04.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:04.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:04.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:04.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:04.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:04.552 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:04 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:04.306+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.894490+0000 front 2024-04-25T13:57:11.894649+0000 (oldest deadline 2024-04-25T13:57:40.095482+0000) 2024-04-25T13:58:04.552 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:04 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:04.306+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.096150+0000 front 2024-04-25T13:57:20.096114+0000 (oldest deadline 2024-04-25T13:57:45.396048+0000) 2024-04-25T13:58:04.552 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:04 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:04.306+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.097087+0000 front 2024-04-25T13:57:30.097264+0000 (oldest deadline 2024-04-25T13:57:52.997270+0000) 2024-04-25T13:58:04.552 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:04 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6[55101]: 2024-04-25T13:58:04.306+0000 7f1ce5d6c700 -1 osd.6 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.398495+0000 front 2024-04-25T13:57:39.398553+0000 (oldest deadline 2024-04-25T13:58:04.098775+0000) 2024-04-25T13:58:04.552 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:04 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:04.201+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:04.552 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:04 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:04.201+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:04.552 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:04 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:04.201+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:04.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:04.353+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:04.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:04.353+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:04.827 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:04 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:04.353+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:05.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:04 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:04.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:05.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:04 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:04.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:05.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:04 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:04.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:05.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:04 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:04.794+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:05.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:04 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:04.794+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:05.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:04 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:04.794+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:05.321 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:05.062+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:05.321 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:05.062+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:05.321 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:05.062+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:05.321 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:05.062+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:05.382 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:04 smithi027 podman[96286]: 2024-04-25 13:58:04.979505368 +0000 UTC m=+5.709206987 container died dd37d4fcf84775603235c3e96a9db31893d948dfdb1744b5dc0ed39cc2843e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T13:58:05.382 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:05 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:05.163+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:05.382 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:05 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:05.163+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:05.382 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:05 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:05.163+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:05.383 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:05 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:05.163+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:05.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:05.320+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:05.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:05.320+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:05.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:05.321+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:05.645 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:05 smithi027 podman[96286]: 2024-04-25 13:58:05.48198422 +0000 UTC m=+6.211685848 container cleanup dd37d4fcf84775603235c3e96a9db31893d948dfdb1744b5dc0ed39cc2843e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T13:58:05.646 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:05 smithi027 bash[96286]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6 2024-04-25T13:58:05.900 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:05 smithi027 podman[96304]: 2024-04-25 13:58:05.650076435 +0000 UTC m=+0.671207638 container remove dd37d4fcf84775603235c3e96a9db31893d948dfdb1744b5dc0ed39cc2843e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T13:58:06.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:05 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:05.755+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:06.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:05 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:05.755+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:06.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:05 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:05.755+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:06.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:05 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:05.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:06.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:05 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:05.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:06.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:05 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:05.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:06.369 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:06 smithi027 podman[96406]: 2024-04-25 13:58:05.986475092 +0000 UTC m=+0.020014918 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:58:06.369 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:06 smithi027 podman[96406]: 2024-04-25 13:58:06.125916452 +0000 UTC m=+0.159456250 container create bf0aa5e80b09d33c5606448178049b2ac0d107e538f8d36a9e7d2910b8763d11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T13:58:06.369 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:06 smithi027 podman[96406]: 2024-04-25 13:58:06.36998427 +0000 UTC m=+0.403524060 container init bf0aa5e80b09d33c5606448178049b2ac0d107e538f8d36a9e7d2910b8763d11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T13:58:06.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:06 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:06.152+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:06.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:06 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:06.152+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:06.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:06 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:06.152+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:06.369 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:06 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:06.152+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:06.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:06 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:06.288+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:06.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:06 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:06.288+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:06.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:06 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:06.288+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:06.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:06 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:06.083+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:06.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:06 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:06.083+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:06.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:06 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:06.083+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:06.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:06 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:06.083+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:06.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:06 smithi027 podman[96406]: 2024-04-25 13:58:06.373434416 +0000 UTC m=+0.406974209 container start bf0aa5e80b09d33c5606448178049b2ac0d107e538f8d36a9e7d2910b8763d11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-25T13:58:06.650 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:06 smithi027 podman[96406]: 2024-04-25 13:58:06.478548955 +0000 UTC m=+0.512088750 container attach bf0aa5e80b09d33c5606448178049b2ac0d107e538f8d36a9e7d2910b8763d11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T13:58:06.964 INFO:journalctl@ceph.osd.6.smithi027.stdout:Apr 25 13:58:06 smithi027 podman[96406]: 2024-04-25 13:58:06.679799038 +0000 UTC m=+0.713338838 container died bf0aa5e80b09d33c5606448178049b2ac0d107e538f8d36a9e7d2910b8763d11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T13:58:07.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:06 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:06.787+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:07.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:06 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:06.787+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:07.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:06 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:06.787+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:07.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:06 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:06.950+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:07.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:06 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:06.950+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:07.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:06 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:06.950+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:07.264 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:07 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:07.113+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:07.265 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:07 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:07.113+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:07.265 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:07 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:07.113+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:07.265 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:07 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:07.113+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:07.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T13:58:07.327 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:07.296+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:07.327 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:07.296+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:07.328 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:07.296+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:07.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:07.076+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:07.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:07.076+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:07.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:07.076+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:07.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:07.076+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:07.334 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:58:07.335 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T13:58:07.335 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T13:58:07.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.7 2024-04-25T13:58:07.656 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:07 smithi045 systemd[1]: Stopping Ceph osd.7 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:58:08.043 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:07.656+0000 7f19f9681700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:58:08.043 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:07.656+0000 7f19f9681700 -1 osd.7 81 *** Got signal Terminated *** 2024-04-25T13:58:08.043 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:07 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:07.656+0000 7f19f9681700 -1 osd.7 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:58:08.307 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:07 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:07.984+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:08.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:07 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:07.984+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:08.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:07 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:07.984+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:08.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:07 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:07.811+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:08.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:07 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:07.811+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:08.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:07 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:07.811+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:08.309 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:08 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:08.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:08.309 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:08 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:08.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:08.309 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:08 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:08.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:08.309 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:08 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:08.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:08.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:08.089+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:08.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:08.089+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:08.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:08.089+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:08.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:08 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:08.089+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:08.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:08 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:08.309+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:08.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:08 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:08.309+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:08.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:08 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:08.309+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:09.294 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:09.016+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:09.295 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:09.016+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:09.295 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:09.016+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:09.295 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:09.016+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:09.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:08 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:08.825+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:09.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:08 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:08.825+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:09.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:08 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:08.825+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:09.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:08 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:08.975+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:09.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:08 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:08.975+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:09.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:08 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:08.975+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:09.402 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:09 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:09.087+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:09.402 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:09 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:09.087+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:09.403 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:09 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:09.087+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:09.403 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:09 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:09.087+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:09.403 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:09 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:09.087+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:09.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:09.295+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:09.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:09.295+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:09.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:09.295+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:10.057 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:09 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:09.969+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:10.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:09 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:09.969+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:10.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:09 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:09.969+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:10.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:09 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:09.799+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:10.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:09 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:09.799+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:10.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:09 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:09.799+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:10.267 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:09.967+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:10.267 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:09.967+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:10.267 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:09.967+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:10.267 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:09 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:09.967+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:10.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:10 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:10.110+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:10.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:10 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:10.110+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:10.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:10 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:10.110+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:10.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:10 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:10.110+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:10.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:10 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:10.110+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:10.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:10.267+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:10.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:10.267+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:10.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:10.267+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:11.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:11.005+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:11.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:11.005+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:11.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:11.005+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:11.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:10 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:10.787+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:11.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:10 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:10.787+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:11.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:10 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:10.787+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:11.270 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:10.961+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:11.270 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:10.961+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:11.270 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:10.961+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:11.270 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:10.961+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:11.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:11 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:11.119+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:11.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:11 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:11.119+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:11.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:11 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:11.119+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:11.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:11 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:11.119+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:11.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:11 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:11.119+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:11.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:11 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:11.270+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:11.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:11 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:11.270+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:11.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:11 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:11.270+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:12.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:11.998+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:12.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:11.998+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:12.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:11.998+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:12.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:11.998+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:12.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:11.758+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:12.059 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:11.758+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:12.059 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:11 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:11.758+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:12.281 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:11 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:11.929+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:12.281 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:11 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:11.929+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:12.281 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:11 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:11.929+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:12.281 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:11 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:11.929+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:12.281 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:11 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:11.929+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:12.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:12 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:12.073+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:12.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:12 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:12.073+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:12.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:12 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:12.073+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:12.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:12 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:12.073+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:12.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:12 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:12.073+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:12.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:12.281+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.447407+0000 front 2024-04-25T13:57:20.447446+0000 (oldest deadline 2024-04-25T13:57:44.547510+0000) 2024-04-25T13:58:12.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:12.281+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.349747+0000 front 2024-04-25T13:57:30.349486+0000 (oldest deadline 2024-04-25T13:57:54.449499+0000) 2024-04-25T13:58:12.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:12.281+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:37.851482+0000 front 2024-04-25T13:57:37.851620+0000 (oldest deadline 2024-04-25T13:58:03.751438+0000) 2024-04-25T13:58:12.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7[51277]: 2024-04-25T13:58:12.281+0000 7f19f1c2f700 -1 osd.7 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:45.452670+0000 front 2024-04-25T13:57:45.452646+0000 (oldest deadline 2024-04-25T13:58:11.352825+0000) 2024-04-25T13:58:12.990 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:12 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:12.740+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:12.991 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:12 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:12.740+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:12.991 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:12 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:12.740+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:12.991 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:12 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:12.740+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.572531+0000 front 2024-04-25T13:57:47.572422+0000 (oldest deadline 2024-04-25T13:58:12.272353+0000) 2024-04-25T13:58:13.307 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:12 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:12.991+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:13.307 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:12 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:12.991+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:13.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:12 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:12.991+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:13.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:12 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:12.991+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:13.327 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:13 smithi045 podman[68646]: 2024-04-25 13:58:13.101897354 +0000 UTC m=+5.566130856 container died 5273e10d557da2d71d3bd740d3707a4641b933aae1ca2bed97c75da96600441a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T13:58:13.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:12.951+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:13.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:12.951+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:13.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:12.951+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:13.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:12.951+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:13.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:12 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:12.951+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:13.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:13 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:13.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:13.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:13 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:13.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:13.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:13 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:13.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:13.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:13 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:13.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:13.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:13 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:13.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:13.731 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:13 smithi045 podman[68646]: 2024-04-25 13:58:13.456635509 +0000 UTC m=+5.920868999 container cleanup 5273e10d557da2d71d3bd740d3707a4641b933aae1ca2bed97c75da96600441a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T13:58:13.731 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:13 smithi045 bash[68646]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7 2024-04-25T13:58:13.987 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:13 smithi045 podman[68663]: 2024-04-25 13:58:13.732772147 +0000 UTC m=+0.625940419 container remove 5273e10d557da2d71d3bd740d3707a4641b933aae1ca2bed97c75da96600441a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:58:14.057 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:13 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:13.986+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:14.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:13 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:13.986+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:14.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:13 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:13.986+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:14.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:13 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:13.986+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:14.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:13 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:13.770+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:14.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:13 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:13.770+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:14.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:13 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:13.770+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:14.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:13 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:13.770+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.572531+0000 front 2024-04-25T13:57:47.572422+0000 (oldest deadline 2024-04-25T13:58:12.272353+0000) 2024-04-25T13:58:14.244 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:14 smithi045 podman[68758]: 2024-04-25 13:58:13.992390088 +0000 UTC m=+0.018831117 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:58:14.244 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:14 smithi045 podman[68758]: 2024-04-25 13:58:14.117241069 +0000 UTC m=+0.143682140 container create e2926538d0b9b343d7ad4f167784852764c7b18f45bf3746beb313afe7c21e66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T13:58:14.245 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:13 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:13.987+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:14.245 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:13 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:13.987+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:14.245 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:13 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:13.987+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:14.245 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:13 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:13.987+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:14.245 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:13 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:13.987+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:14.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:14 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:14.075+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:14.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:14 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:14.075+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:14.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:14 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:14.075+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:14.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:14 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:14.075+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:14.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:14 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:14.075+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:14.577 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:14 smithi045 podman[68758]: 2024-04-25 13:58:14.378513586 +0000 UTC m=+0.404954626 container init e2926538d0b9b343d7ad4f167784852764c7b18f45bf3746beb313afe7c21e66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-25T13:58:14.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:14 smithi045 podman[68758]: 2024-04-25 13:58:14.381939473 +0000 UTC m=+0.408380505 container start e2926538d0b9b343d7ad4f167784852764c7b18f45bf3746beb313afe7c21e66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T13:58:14.578 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:14 smithi045 podman[68758]: 2024-04-25 13:58:14.461240466 +0000 UTC m=+0.487681500 container attach e2926538d0b9b343d7ad4f167784852764c7b18f45bf3746beb313afe7c21e66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-25T13:58:15.017 INFO:journalctl@ceph.osd.7.smithi045.stdout:Apr 25 13:58:14 smithi045 podman[68758]: 2024-04-25 13:58:14.687970893 +0000 UTC m=+0.714411924 container died e2926538d0b9b343d7ad4f167784852764c7b18f45bf3746beb313afe7c21e66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T13:58:15.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:14 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:14.775+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:15.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:14 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:14.775+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:15.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:14 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:14.775+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:15.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:14 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:14.775+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.572531+0000 front 2024-04-25T13:57:47.572422+0000 (oldest deadline 2024-04-25T13:58:12.272353+0000) 2024-04-25T13:58:15.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:14 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:14.956+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:15.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:14 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:14.956+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:15.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:14 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:14.956+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:15.059 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:14 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:14.956+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:15.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:15.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:15.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:15.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:15.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:15.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T13:58:15.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:15 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:15.047+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:15.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:15 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:15.047+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:15.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:15 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:15.047+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:15.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:15 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:15.047+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:15.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:15 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:15.047+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:15.403 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:58:15.404 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T13:58:15.404 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T13:58:15.404 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.8 2024-04-25T13:58:15.730 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:15 smithi122 systemd[1]: Stopping Ceph osd.8 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:58:16.006 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:15 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:15.730+0000 7f0f4989f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:58:16.006 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:15 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:15.730+0000 7f0f4989f700 -1 osd.8 81 *** Got signal Terminated *** 2024-04-25T13:58:16.006 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:15 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:15.730+0000 7f0f4989f700 -1 osd.8 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:58:16.006 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:15 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:15.798+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:16.006 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:15 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:15.798+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:16.006 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:15 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:15.798+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:16.006 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:15 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:15.798+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.572531+0000 front 2024-04-25T13:57:47.572422+0000 (oldest deadline 2024-04-25T13:58:12.272353+0000) 2024-04-25T13:58:16.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:16 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:16.006+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:16.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:16 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:16.006+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:16.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:16 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:16.006+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:16.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:16 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:16.006+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:16.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.973+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:16.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.973+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:16.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.973+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:16.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.973+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:16.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:15.973+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:16.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.013+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:16.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.013+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:16.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.013+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:16.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.013+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:16.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.013+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:17.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.002+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:17.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.002+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:17.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.002+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:17.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.002+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:17.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:16 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:16.778+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:17.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:16 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:16.778+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:17.059 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:16 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:16.778+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:17.059 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:16 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:16.778+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.572531+0000 front 2024-04-25T13:57:47.572422+0000 (oldest deadline 2024-04-25T13:58:12.272353+0000) 2024-04-25T13:58:17.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:16 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:16.994+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:17.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:16 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:16.994+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:17.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:16 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:16.994+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:17.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:16 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:16.994+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:17.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:16 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:16.994+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:17.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.992+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:17.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.992+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:17.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.992+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:17.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.992+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:17.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:16 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:16.992+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:17.993 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:17.733+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:17.993 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:17.733+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:17.993 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:17.733+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:17.994 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:17.733+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.572531+0000 front 2024-04-25T13:57:47.572422+0000 (oldest deadline 2024-04-25T13:58:12.272353+0000) 2024-04-25T13:58:17.994 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:17.733+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:53.973421+0000 front 2024-04-25T13:57:53.973402+0000 (oldest deadline 2024-04-25T13:58:17.473586+0000) 2024-04-25T13:58:18.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.993+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:18.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.993+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:18.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.993+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:18.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.993+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:18.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:17 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:17.993+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:18.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:18.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:18.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:18.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:18.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:18.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:18.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:18.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:18.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:18 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:18.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:18.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:18 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:18.005+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:18.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:18 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:18.005+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:18.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:18 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:18.005+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:18.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:18 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:18.005+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:18.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:18 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:18.005+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:18.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:18 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:18.005+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:19.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:18.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:19.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:18.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:19.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:18.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:19.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:18.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:19.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:18.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:19.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:18.781+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:19.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:18.781+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:19.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:18.781+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:19.059 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:18.781+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.572531+0000 front 2024-04-25T13:57:47.572422+0000 (oldest deadline 2024-04-25T13:58:12.272353+0000) 2024-04-25T13:58:19.059 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:18 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:18.781+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:53.973421+0000 front 2024-04-25T13:57:53.973402+0000 (oldest deadline 2024-04-25T13:58:17.473586+0000) 2024-04-25T13:58:19.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:19 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:19.010+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:19.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:19 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:19.010+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:19.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:19 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:19.010+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:19.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:19 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:19.010+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:19.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:19 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:19.010+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:19.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:19 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:19.036+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:19.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:19 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:19.036+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:19.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:19 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:19.036+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:19.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:19 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:19.036+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:19.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:19 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:19.036+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:19.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:19 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:19.036+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:20.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:19.971+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:20.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:19.971+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:20.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:19.971+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:20.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:19.971+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:20.058 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:19.971+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:20.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:19.768+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:22.467769+0000 front 2024-04-25T13:57:22.467780+0000 (oldest deadline 2024-04-25T13:57:45.967799+0000) 2024-04-25T13:58:20.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:19.768+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:31.769553+0000 front 2024-04-25T13:57:31.769600+0000 (oldest deadline 2024-04-25T13:57:55.269798+0000) 2024-04-25T13:58:20.058 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:19.768+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:35.270270+0000 front 2024-04-25T13:57:35.270410+0000 (oldest deadline 2024-04-25T13:58:02.271000+0000) 2024-04-25T13:58:20.059 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:19.768+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.572531+0000 front 2024-04-25T13:57:47.572422+0000 (oldest deadline 2024-04-25T13:58:12.272353+0000) 2024-04-25T13:58:20.059 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:19 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8[51621]: 2024-04-25T13:58:19.768+0000 7f0f41e4d700 -1 osd.8 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:53.973421+0000 front 2024-04-25T13:57:53.973402+0000 (oldest deadline 2024-04-25T13:58:17.473586+0000) 2024-04-25T13:58:20.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:20 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:20.020+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:20.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:20 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:20.020+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:20.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:20 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:20.020+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:20.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:20 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:20.020+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:20.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:20 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:20.020+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:20.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:20 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:20.020+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:20.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:20 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:20.030+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:20.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:20 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:20.030+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:20.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:20 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:20.030+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:20.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:20 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:20.030+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:20.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:20 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:20.030+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:20.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:20 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:20.030+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:21.248 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:20 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:20.952+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:21.249 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:20 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:20.952+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:21.249 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:20 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:20.952+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:21.249 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:20 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:20.952+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:21.249 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:20 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:20.952+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:21.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:21 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:21.042+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:21.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:21 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:21.042+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:21.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:21 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:21.042+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:21.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:21 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:21.042+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:21.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:21 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:21.042+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:21.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:21 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:21.042+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:21.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:21 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:21.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:21.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:21 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:21.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:21.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:21 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:21.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:21.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:21 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:21.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:21.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:21 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:21.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:21.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:21 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:21.068+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:21.547 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:21 smithi122 podman[69835]: 2024-04-25 13:58:21.249781132 +0000 UTC m=+5.647230005 container died 833f4ece13f22f92474ca055cb0e9e209d92cfe90606e4316a4828739436d4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-25T13:58:21.807 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:21 smithi122 podman[69835]: 2024-04-25 13:58:21.655718796 +0000 UTC m=+6.053167647 container cleanup 833f4ece13f22f92474ca055cb0e9e209d92cfe90606e4316a4828739436d4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:58:21.808 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:21 smithi122 bash[69835]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8 2024-04-25T13:58:22.308 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:21 smithi122 podman[69854]: 2024-04-25 13:58:21.882273629 +0000 UTC m=+0.627226618 container remove 833f4ece13f22f92474ca055cb0e9e209d92cfe90606e4316a4828739436d4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:58:22.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:21 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:21.955+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:22.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:21 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:21.955+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:22.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:21 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:21.955+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:22.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:21 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:21.955+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:22.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:21 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:21.955+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:22.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:22 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:22.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:22.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:22 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:22.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:22.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:22 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:22.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:22.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:22 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:22.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:22.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:22 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:22.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:22.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:22 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:22.017+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:22.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:22 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:22.049+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:22.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:22 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:22.049+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:22.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:22 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:22.049+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:22.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:22 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:22.049+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:22.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:22 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:22.049+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:22.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:22 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:22.049+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:22.601 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:22 smithi122 podman[69945]: 2024-04-25 13:58:22.221462596 +0000 UTC m=+0.018339186 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:58:22.601 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:22 smithi122 podman[69945]: 2024-04-25 13:58:22.3497465 +0000 UTC m=+0.146623092 container create 27d4471a6bf2959e3d17a975e24f9430360273828b960cda796333e39a9706d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-04-25T13:58:22.602 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:22 smithi122 podman[69945]: 2024-04-25 13:58:22.602817351 +0000 UTC m=+0.399693937 container init 27d4471a6bf2959e3d17a975e24f9430360273828b960cda796333e39a9706d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:58:22.898 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:22 smithi122 podman[69945]: 2024-04-25 13:58:22.605934749 +0000 UTC m=+0.402811344 container start 27d4471a6bf2959e3d17a975e24f9430360273828b960cda796333e39a9706d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-25T13:58:22.898 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:22 smithi122 podman[69945]: 2024-04-25 13:58:22.711197305 +0000 UTC m=+0.508073908 container attach 27d4471a6bf2959e3d17a975e24f9430360273828b960cda796333e39a9706d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:58:23.255 INFO:journalctl@ceph.osd.8.smithi122.stdout:Apr 25 13:58:22 smithi122 podman[69945]: 2024-04-25 13:58:22.899603038 +0000 UTC m=+0.696479632 container died 27d4471a6bf2959e3d17a975e24f9430360273828b960cda796333e39a9706d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T13:58:23.256 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:22 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:22.918+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:23.256 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:22 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:22.918+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:23.256 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:22 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:22.918+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:23.256 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:22 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:22.918+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:23.256 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:22 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:22.918+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:23.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:23 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:23.061+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:23.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:23 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:23.061+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:23.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:23 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:23.061+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:23.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:23 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:23.061+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:23.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:23 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:23.061+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:23.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:23 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:23.061+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:23.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.041+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:23.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.041+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:23.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.041+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:23.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.041+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:23.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.041+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:23.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.041+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:23.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T13:58:23.623 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:58:23.624 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T13:58:23.624 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T13:58:23.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.9 2024-04-25T13:58:23.984 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 systemd[1]: Stopping Ceph osd.9 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:58:24.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:23 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:23.896+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:24.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:23 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:23.896+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:24.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:23 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:23.896+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:24.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:23 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:23.896+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:24.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:23 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:23.896+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:24.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.984+0000 7f98c5a0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:58:24.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.984+0000 7f98c5a0c700 -1 osd.9 81 *** Got signal Terminated *** 2024-04-25T13:58:24.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:23 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:23.984+0000 7f98c5a0c700 -1 osd.9 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:58:24.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:24 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:24.090+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:24.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:24 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:24.090+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:24.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:24 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:24.090+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:24.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:24 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:24.090+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:24.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:24 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:24.090+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:24.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:24 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:24.090+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:24.577 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:24 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:24.079+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:24.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:24 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:24.079+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:24.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:24 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:24.079+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:24.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:24 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:24.079+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:24.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:24 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:24.079+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:24.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:24 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:24.079+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:25.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:24 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:24.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:25.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:24 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:24.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:25.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:24 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:24.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:25.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:24 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:24.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:25.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:24 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:24.944+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:25.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:25 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:25.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:25.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:25 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:25.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:25.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:25 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:25.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:25.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:25 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:25.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:25.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:25 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:25.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:25.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:25 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:25.058+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:25.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:25.135+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:25.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:25.135+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:25.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:25.135+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:25.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:25.135+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:25.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:25.135+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:25.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:25 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:25.135+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:26.209 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:25 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:25.926+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:26.209 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:25 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:25.926+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:26.209 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:25 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:25.926+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:26.209 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:25 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:25.926+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:26.209 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:25 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:25.926+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:26.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:26 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:26.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:26.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:26 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:26.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:26.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:26 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:26.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:26.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:26 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:26.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:26.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:26 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:26.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:26.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:26 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:26.043+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:26.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:26 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:26.126+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:26.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:26 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:26.126+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:26.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:26 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:26.126+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:26.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:26 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:26.126+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:26.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:26 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:26.126+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:26.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:26 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:26.126+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:27.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:26 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:26.903+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:27.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:26 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:26.903+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:27.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:26 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:26.903+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:27.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:26 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:26.903+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:27.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:26 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:26.903+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:27.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.038+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:27.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.038+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:27.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.038+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:27.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.038+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:27.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.038+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:27.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.038+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:27.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:27.091+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:27.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:27.091+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:27.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:27.091+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:27.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:27.091+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:27.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:27.091+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:27.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:27 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:27.091+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:28.307 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:27 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:27.930+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:28.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:27 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:27.930+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:28.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:27 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:27.930+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:28.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:27 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:27.930+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:28.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:27 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:27.930+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:28.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:27 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:27.930+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:28.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.991+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:28.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.991+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:28.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.991+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:28.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.991+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:28.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.991+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:28.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:27 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:27.991+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:28.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:28.129+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:10.343828+0000 front 2024-04-25T13:57:10.343847+0000 (oldest deadline 2024-04-25T13:57:36.243994+0000) 2024-04-25T13:58:28.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:28.129+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:21.544967+0000 front 2024-04-25T13:57:21.544924+0000 (oldest deadline 2024-04-25T13:57:45.045061+0000) 2024-04-25T13:58:28.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:28.129+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:30.346376+0000 front 2024-04-25T13:57:30.346594+0000 (oldest deadline 2024-04-25T13:57:53.846310+0000) 2024-04-25T13:58:28.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:28.129+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.147496+0000 front 2024-04-25T13:57:39.147556+0000 (oldest deadline 2024-04-25T13:58:04.447594+0000) 2024-04-25T13:58:28.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:28.129+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.748846+0000 front 2024-04-25T13:57:46.748778+0000 (oldest deadline 2024-04-25T13:58:08.949324+0000) 2024-04-25T13:58:28.401 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:28 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9[61892]: 2024-04-25T13:58:28.129+0000 7f98bdfba700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.250320+0000 front 2024-04-25T13:57:54.250343+0000 (oldest deadline 2024-04-25T13:58:17.150185+0000) 2024-04-25T13:58:29.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:28 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:28.960+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:29.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:28 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:28.960+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:29.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:28 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:28.960+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:29.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:28 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:28.960+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:29.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:28 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:28.960+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:29.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:28 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:28.960+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:29.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:29 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:29.026+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:29.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:29 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:29.026+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:29.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:29 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:29.026+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:29.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:29 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:29.026+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:29.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:29 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:29.026+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:29.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:29 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:29.026+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:29.759 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:29 smithi027 podman[96514]: 2024-04-25 13:58:29.49249916 +0000 UTC m=+5.661572672 container died 81a7e02bbb165de125dae757d3da9b5160d5dfc90eb3523bc2d9d5f4a936929b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:58:30.118 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:29 smithi027 podman[96514]: 2024-04-25 13:58:29.859452667 +0000 UTC m=+6.028526169 container cleanup 81a7e02bbb165de125dae757d3da9b5160d5dfc90eb3523bc2d9d5f4a936929b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux ) 2024-04-25T13:58:30.118 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:29 smithi027 bash[96514]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9 2024-04-25T13:58:30.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:29 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:29.943+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:30.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:29 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:29.943+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:30.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:29 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:29.943+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:30.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:29 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:29.943+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:30.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:29 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:29.943+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:30.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:29 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:29.943+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:30.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:30 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:30.056+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:30.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:30 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:30.056+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:30.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:30 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:30.056+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:30.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:30 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:30.056+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:30.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:30 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:30.056+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:30.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:30 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:30.056+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:30.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:30 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:30.056+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:57:59.406055+0000 front 2024-04-25T13:57:59.406104+0000 (oldest deadline 2024-04-25T13:58:30.006560+0000) 2024-04-25T13:58:30.400 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:30 smithi027 podman[96531]: 2024-04-25 13:58:30.119093599 +0000 UTC m=+0.620248830 container remove 81a7e02bbb165de125dae757d3da9b5160d5dfc90eb3523bc2d9d5f4a936929b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:58:30.738 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:30 smithi027 podman[96616]: 2024-04-25 13:58:30.389832049 +0000 UTC m=+0.018361821 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:58:30.738 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:30 smithi027 podman[96616]: 2024-04-25 13:58:30.511558811 +0000 UTC m=+0.140088610 container create fc287e075e6a0322c15751557550f3432f6a6401c5d7b66f13060fbe45644212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T13:58:30.738 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:30 smithi027 podman[96616]: 2024-04-25 13:58:30.739629655 +0000 UTC m=+0.368159418 container init fc287e075e6a0322c15751557550f3432f6a6401c5d7b66f13060fbe45644212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T13:58:31.048 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:30 smithi027 podman[96616]: 2024-04-25 13:58:30.742999294 +0000 UTC m=+0.371529058 container start fc287e075e6a0322c15751557550f3432f6a6401c5d7b66f13060fbe45644212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T13:58:31.049 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:30 smithi027 podman[96616]: 2024-04-25 13:58:30.814145268 +0000 UTC m=+0.442675025 container attach fc287e075e6a0322c15751557550f3432f6a6401c5d7b66f13060fbe45644212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T13:58:31.307 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:30 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:30.959+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:31.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:30 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:30.959+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:31.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:30 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:30.959+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:31.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:30 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:30.959+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:31.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:30 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:30.959+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:31.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:30 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:30.959+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:31.331 INFO:journalctl@ceph.osd.9.smithi027.stdout:Apr 25 13:58:31 smithi027 podman[96616]: 2024-04-25 13:58:31.050244041 +0000 UTC m=+0.678773800 container died fc287e075e6a0322c15751557550f3432f6a6401c5d7b66f13060fbe45644212 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T13:58:31.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.088+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:31.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.088+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:31.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.088+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:31.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.088+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:31.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.088+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:31.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.088+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:31.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.088+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:57:59.406055+0000 front 2024-04-25T13:57:59.406104+0000 (oldest deadline 2024-04-25T13:58:30.006560+0000) 2024-04-25T13:58:31.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T13:58:31.666 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:58:31.666 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T13:58:31.667 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T13:58:31.667 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.10 2024-04-25T13:58:31.958 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 systemd[1]: Stopping Ceph osd.10 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:58:31.959 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.958+0000 7f393da67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:58:31.959 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.958+0000 7f393da67700 -1 osd.10 81 *** Got signal Terminated *** 2024-04-25T13:58:31.959 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:31 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:31.958+0000 7f393da67700 -1 osd.10 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:58:32.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:31 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:31.941+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:32.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:31 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:31.941+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:32.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:31 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:31.941+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:32.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:31 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:31.941+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:32.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:31 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:31.941+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:32.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:31 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:31.941+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:32.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:32 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:32.121+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:32.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:32 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:32.121+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:32.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:32 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:32.121+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:32.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:32 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:32.121+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:32.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:32 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:32.121+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:32.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:32 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:32.121+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:32.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:32 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:32.121+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:57:59.406055+0000 front 2024-04-25T13:57:59.406104+0000 (oldest deadline 2024-04-25T13:58:30.006560+0000) 2024-04-25T13:58:33.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:32 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:32.898+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:33.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:32 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:32.898+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:33.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:32 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:32.898+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:33.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:32 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:32.898+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:33.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:32 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:32.898+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:33.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:32 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:32.898+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:33.577 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:33 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:33.123+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:33.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:33 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:33.123+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:33.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:33 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:33.123+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:33.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:33 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:33.123+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:33.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:33 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:33.123+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:33.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:33 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:33.123+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:33.578 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:33 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:33.123+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:57:59.406055+0000 front 2024-04-25T13:57:59.406104+0000 (oldest deadline 2024-04-25T13:58:30.006560+0000) 2024-04-25T13:58:34.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:33 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:33.906+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:34.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:33 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:33.906+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:34.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:33 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:33.906+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:34.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:33 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:33.906+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:34.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:33 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:33.906+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:34.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:33 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:33.906+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:34.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:34 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:34.075+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:34.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:34 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:34.075+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:34.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:34 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:34.075+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:34.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:34 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:34.075+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:34.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:34 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:34.075+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:34.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:34 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:34.075+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:34.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:34 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:34.075+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:57:59.406055+0000 front 2024-04-25T13:57:59.406104+0000 (oldest deadline 2024-04-25T13:58:30.006560+0000) 2024-04-25T13:58:35.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:34 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:34.899+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:35.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:34 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:34.899+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:35.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:34 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:34.899+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:35.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:34 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:34.899+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:35.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:34 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:34.899+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:35.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:34 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:34.899+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:35.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:35 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:35.065+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:35.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:35 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:35.065+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:35.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:35 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:35.065+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:35.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:35 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:35.065+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:35.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:35 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:35.065+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:35.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:35 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:35.065+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:35.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:35 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:35.065+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:57:59.406055+0000 front 2024-04-25T13:57:59.406104+0000 (oldest deadline 2024-04-25T13:58:30.006560+0000) 2024-04-25T13:58:36.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:35 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:35.917+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:36.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:35 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:35.917+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:36.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:35 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:35.917+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:36.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:35 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:35.917+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:36.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:35 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:35.917+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:36.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:35 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:35.917+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:36.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:35 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:35.917+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:36.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:36 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:36.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:11.796682+0000 front 2024-04-25T13:57:11.796798+0000 (oldest deadline 2024-04-25T13:57:37.096734+0000) 2024-04-25T13:58:36.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:36 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:36.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:18.698391+0000 front 2024-04-25T13:57:18.698363+0000 (oldest deadline 2024-04-25T13:57:44.598525+0000) 2024-04-25T13:58:36.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:36 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:36.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:29.800364+0000 front 2024-04-25T13:57:29.800318+0000 (oldest deadline 2024-04-25T13:57:55.000933+0000) 2024-04-25T13:58:36.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:36 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:36.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6814 osd.3 since back 2024-04-25T13:57:39.102275+0000 front 2024-04-25T13:57:39.102377+0000 (oldest deadline 2024-04-25T13:58:02.602141+0000) 2024-04-25T13:58:36.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:36 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:36.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:46.603771+0000 front 2024-04-25T13:57:46.603811+0000 (oldest deadline 2024-04-25T13:58:11.303923+0000) 2024-04-25T13:58:36.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:36 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:36.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:54.705642+0000 front 2024-04-25T13:57:54.705696+0000 (oldest deadline 2024-04-25T13:58:19.405628+0000) 2024-04-25T13:58:36.328 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:36 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10[58343]: 2024-04-25T13:58:36.071+0000 7f393a01d700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:57:59.406055+0000 front 2024-04-25T13:57:59.406104+0000 (oldest deadline 2024-04-25T13:58:30.006560+0000) 2024-04-25T13:58:37.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:36 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:36.914+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:37.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:36 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:36.914+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:37.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:36 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:36.914+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:37.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:36 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:36.914+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:37.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:36 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:36.914+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:37.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:36 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:36.914+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:37.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:36 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:36.914+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:37.683 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:37 smithi045 podman[68876]: 2024-04-25 13:58:37.392658428 +0000 UTC m=+5.526497462 container died 0e8867f9af34ec9aed0141381323dd9b2aeefc24fda08933c24496d64e520846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T13:58:38.025 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:37 smithi045 podman[68876]: 2024-04-25 13:58:37.791423491 +0000 UTC m=+5.925262516 container cleanup 0e8867f9af34ec9aed0141381323dd9b2aeefc24fda08933c24496d64e520846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:58:38.025 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:37 smithi045 bash[68876]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10 2024-04-25T13:58:38.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:37 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:37.942+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:38.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:37 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:37.942+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:38.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:37 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:37.942+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:38.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:37 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:37.942+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:38.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:37 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:37.942+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:38.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:37 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:37.942+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:38.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:37 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:37.942+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:38.327 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:38 smithi045 podman[68893]: 2024-04-25 13:58:38.025960549 +0000 UTC m=+0.632231238 container remove 0e8867f9af34ec9aed0141381323dd9b2aeefc24fda08933c24496d64e520846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T13:58:38.628 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:38 smithi045 podman[68985]: 2024-04-25 13:58:38.235168251 +0000 UTC m=+0.018506444 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:58:38.628 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:38 smithi045 podman[68985]: 2024-04-25 13:58:38.351956355 +0000 UTC m=+0.135294578 container create 4f1e5e5b24f209b92d5acc06cb36f390bfa0508f1385a69cb5a1295d293e62ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T13:58:38.629 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:38 smithi045 podman[68985]: 2024-04-25 13:58:38.537475669 +0000 UTC m=+0.320813853 container init 4f1e5e5b24f209b92d5acc06cb36f390bfa0508f1385a69cb5a1295d293e62ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T13:58:38.629 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:38 smithi045 podman[68985]: 2024-04-25 13:58:38.541244687 +0000 UTC m=+0.324582881 container start 4f1e5e5b24f209b92d5acc06cb36f390bfa0508f1385a69cb5a1295d293e62ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:58:38.629 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:38 smithi045 podman[68985]: 2024-04-25 13:58:38.629119756 +0000 UTC m=+0.412457949 container attach 4f1e5e5b24f209b92d5acc06cb36f390bfa0508f1385a69cb5a1295d293e62ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T13:58:39.077 INFO:journalctl@ceph.osd.10.smithi045.stdout:Apr 25 13:58:38 smithi045 podman[68985]: 2024-04-25 13:58:38.847228545 +0000 UTC m=+0.630566735 container died 4f1e5e5b24f209b92d5acc06cb36f390bfa0508f1385a69cb5a1295d293e62ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T13:58:39.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:38.946+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:39.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:38.946+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:39.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:38.946+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:39.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:38.946+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:39.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:38.946+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:39.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:38.946+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:39.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:38 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:38.946+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:39.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T13:58:39.503 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:58:39.503 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T13:58:39.503 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T13:58:39.504 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.11 2024-04-25T13:58:39.807 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 systemd[1]: Stopping Ceph osd.11 for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.810+0000 7fc9f4f13700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.810+0000 7fc9f4f13700 -1 osd.11 81 *** Got signal Terminated *** 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.810+0000 7fc9f4f13700 -1 osd.11 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.996+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.996+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.996+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.996+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.996+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.996+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:40.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:39 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:39.996+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:41.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:40.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:41.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:40.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:41.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:40.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:41.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:40.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:41.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:40.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:41.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:40.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:41.309 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:40 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:40.982+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:42.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.023+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:42.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.023+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:42.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.023+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:42.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.023+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:42.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.023+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:42.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.023+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:42.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.023+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:43.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.988+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:43.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.988+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:43.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.988+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:43.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.988+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:43.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.988+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:43.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.988+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:43.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:42 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:42.988+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:44.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:43.954+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-25T13:57:15.406484+0000 front 2024-04-25T13:57:15.406518+0000 (oldest deadline 2024-04-25T13:57:37.706607+0000) 2024-04-25T13:58:44.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:43.954+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6804 osd.1 since back 2024-04-25T13:57:20.007839+0000 front 2024-04-25T13:57:20.007977+0000 (oldest deadline 2024-04-25T13:57:45.208462+0000) 2024-04-25T13:58:44.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:43.954+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6804 osd.2 since back 2024-04-25T13:57:28.709502+0000 front 2024-04-25T13:57:28.709400+0000 (oldest deadline 2024-04-25T13:57:52.809525+0000) 2024-04-25T13:58:44.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:43.954+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6812 osd.4 since back 2024-04-25T13:57:47.813044+0000 front 2024-04-25T13:57:47.812848+0000 (oldest deadline 2024-04-25T13:58:11.213582+0000) 2024-04-25T13:58:44.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:43.954+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6812 osd.5 since back 2024-04-25T13:57:51.213986+0000 front 2024-04-25T13:57:51.214078+0000 (oldest deadline 2024-04-25T13:58:17.114216+0000) 2024-04-25T13:58:44.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:43.954+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.27:6822 osd.6 since back 2024-04-25T13:58:03.015443+0000 front 2024-04-25T13:58:03.015475+0000 (oldest deadline 2024-04-25T13:58:27.715350+0000) 2024-04-25T13:58:44.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:43.954+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.45:6820 osd.7 since back 2024-04-25T13:58:09.416446+0000 front 2024-04-25T13:58:09.416410+0000 (oldest deadline 2024-04-25T13:58:35.316294+0000) 2024-04-25T13:58:44.308 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:43 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11[58665]: 2024-04-25T13:58:43.954+0000 7fc9ed4c1700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.122:6820 osd.8 since back 2024-04-25T13:58:19.817981+0000 front 2024-04-25T13:58:19.817999+0000 (oldest deadline 2024-04-25T13:58:43.317949+0000) 2024-04-25T13:58:45.518 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:45 smithi122 podman[70066]: 2024-04-25 13:58:45.239428558 +0000 UTC m=+5.536167843 container died 0c1a60953590b9facd6c3b4f7919be0ccff673fa877eb1dad916a01c507b3259 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T13:58:45.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:45 smithi122 podman[70066]: 2024-04-25 13:58:45.618395035 +0000 UTC m=+5.915134312 container cleanup 0c1a60953590b9facd6c3b4f7919be0ccff673fa877eb1dad916a01c507b3259 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T13:58:45.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:45 smithi122 bash[70066]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11 2024-04-25T13:58:46.177 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:45 smithi122 podman[70084]: 2024-04-25 13:58:45.861305961 +0000 UTC m=+0.622488650 container remove 0c1a60953590b9facd6c3b4f7919be0ccff673fa877eb1dad916a01c507b3259 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T13:58:46.472 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:46 smithi122 podman[70174]: 2024-04-25 13:58:46.178720301 +0000 UTC m=+0.111954890 container create db1d217f2c9bef18732088eb4347d068d490ed55b65fe295b54a532b3da208b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:58:46.472 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:46 smithi122 podman[70174]: 2024-04-25 13:58:46.084773341 +0000 UTC m=+0.018007943 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T13:58:46.472 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:46 smithi122 podman[70174]: 2024-04-25 13:58:46.397908085 +0000 UTC m=+0.331142674 container init db1d217f2c9bef18732088eb4347d068d490ed55b65fe295b54a532b3da208b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T13:58:46.472 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:46 smithi122 podman[70174]: 2024-04-25 13:58:46.40093978 +0000 UTC m=+0.334174369 container start db1d217f2c9bef18732088eb4347d068d490ed55b65fe295b54a532b3da208b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-25T13:58:46.472 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:46 smithi122 podman[70174]: 2024-04-25 13:58:46.472840093 +0000 UTC m=+0.406074692 container attach db1d217f2c9bef18732088eb4347d068d490ed55b65fe295b54a532b3da208b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T13:58:46.808 INFO:journalctl@ceph.osd.11.smithi122.stdout:Apr 25 13:58:46 smithi122 podman[70174]: 2024-04-25 13:58:46.698111729 +0000 UTC m=+0.631346318 container died db1d217f2c9bef18732088eb4347d068d490ed55b65fe295b54a532b3da208b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T13:58:47.332 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T13:58:47.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:58:47.370 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T13:58:47.371 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T13:58:47.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.a 2024-04-25T13:58:47.650 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:58:47 smithi027 systemd[1]: Stopping Ceph mds.a for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:58:48.150 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:58:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a[68430]: 2024-04-25T13:58:47.677+0000 7f5051232700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:58:48.151 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:58:47 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a[68430]: 2024-04-25T13:58:47.677+0000 7f5051232700 -1 mds.a *** got signal Terminated *** 2024-04-25T13:58:51.969 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:58:51 smithi027 podman[96733]: 2024-04-25 13:58:51.691511852 +0000 UTC m=+4.112209861 container died e939bfeeb51a5280087e0df981dd8231554dd90f76b7d0e55513a736a5caecf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T13:58:52.319 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:58:52 smithi027 podman[96733]: 2024-04-25 13:58:52.044289538 +0000 UTC m=+4.464987540 container cleanup e939bfeeb51a5280087e0df981dd8231554dd90f76b7d0e55513a736a5caecf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T13:58:52.319 INFO:journalctl@ceph.mds.a.smithi027.stdout:Apr 25 13:58:52 smithi027 bash[96733]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-a 2024-04-25T13:58:52.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T13:58:52.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:58:52.509 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T13:58:52.510 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T13:58:52.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.d 2024-04-25T13:58:52.900 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:58:52 smithi027 systemd[1]: Stopping Ceph mds.d for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:58:53.400 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:58:52 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-d[67473]: 2024-04-25T13:58:52.955+0000 7fca2ca52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:58:53.400 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:58:52 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-d[67473]: 2024-04-25T13:58:52.955+0000 7fca2ca52700 -1 mds.d *** got signal Terminated *** 2024-04-25T13:58:55.900 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:58:55 smithi027 podman[96852]: 2024-04-25 13:58:55.64540316 +0000 UTC m=+2.914887532 container died 1ee835dc342cae2854fe494609282c24e52ea9b067670606382dde9310b5af4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T13:58:56.281 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:58:55 smithi027 podman[96852]: 2024-04-25 13:58:55.997801361 +0000 UTC m=+3.267285734 container cleanup 1ee835dc342cae2854fe494609282c24e52ea9b067670606382dde9310b5af4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-25T13:58:56.281 INFO:journalctl@ceph.mds.d.smithi027.stdout:Apr 25 13:58:55 smithi027 bash[96852]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-d 2024-04-25T13:58:56.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T13:58:56.465 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:58:56.466 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T13:58:56.466 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T13:58:56.466 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.g 2024-04-25T13:58:56.900 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:58:56 smithi027 systemd[1]: Stopping Ceph mds.g for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:58:56.900 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:58:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g[68071]: 2024-04-25T13:58:56.792+0000 7f5ec0cb1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:58:56.900 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:58:56 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g[68071]: 2024-04-25T13:58:56.792+0000 7f5ec0cb1700 -1 mds.g *** got signal Terminated *** 2024-04-25T13:59:01.150 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:59:00 smithi027 podman[96973]: 2024-04-25 13:59:00.806895608 +0000 UTC m=+4.121642031 container died c73ebae03d8ef50df78c404e0fe16d955d3eb4d003767c21592c4418f54d8dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:59:01.568 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:59:01 smithi027 podman[96973]: 2024-04-25 13:59:01.259578955 +0000 UTC m=+4.574325370 container cleanup c73ebae03d8ef50df78c404e0fe16d955d3eb4d003767c21592c4418f54d8dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T13:59:01.568 INFO:journalctl@ceph.mds.g.smithi027.stdout:Apr 25 13:59:01 smithi027 bash[96973]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-g 2024-04-25T13:59:01.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T13:59:01.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:01.725 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T13:59:01.726 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T13:59:01.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.j 2024-04-25T13:59:01.900 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:59:01 smithi027 systemd[1]: Stopping Ceph mds.j for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:02.400 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:59:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j[67115]: 2024-04-25T13:59:02.062+0000 7f37ffad0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:02.400 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:59:02 smithi027 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j[67115]: 2024-04-25T13:59:02.063+0000 7f37ffad0700 -1 mds.j *** got signal Terminated *** 2024-04-25T13:59:05.021 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:59:04 smithi027 podman[97092]: 2024-04-25 13:59:04.762820862 +0000 UTC m=+2.808085974 container died 54604d426a1e208e050f1642c75655b09d1b899c2606ac938054d1ecf46f0df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T13:59:05.379 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:59:05 smithi027 podman[97092]: 2024-04-25 13:59:05.121650395 +0000 UTC m=+3.166915505 container cleanup 54604d426a1e208e050f1642c75655b09d1b899c2606ac938054d1ecf46f0df4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T13:59:05.380 INFO:journalctl@ceph.mds.j.smithi027.stdout:Apr 25 13:59:05 smithi027 bash[97092]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-j 2024-04-25T13:59:05.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T13:59:05.536 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:05.537 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T13:59:05.537 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T13:59:05.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.b 2024-04-25T13:59:05.828 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:59:05 smithi045 systemd[1]: Stopping Ceph mds.b for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:06.327 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:59:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b[63598]: 2024-04-25T13:59:05.848+0000 7fb2d4287700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:06.328 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:59:05 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b[63598]: 2024-04-25T13:59:05.848+0000 7fb2d4287700 -1 mds.b *** got signal Terminated *** 2024-04-25T13:59:09.190 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:59:08 smithi045 podman[69094]: 2024-04-25 13:59:08.927723986 +0000 UTC m=+3.188343314 container died 172f6a43af0244f31bca7830bde97da5bb22a93b0266befd86211bb434dcfe7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-25T13:59:09.557 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:59:09 smithi045 podman[69094]: 2024-04-25 13:59:09.298618715 +0000 UTC m=+3.559238056 container cleanup 172f6a43af0244f31bca7830bde97da5bb22a93b0266befd86211bb434dcfe7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T13:59:09.557 INFO:journalctl@ceph.mds.b.smithi045.stdout:Apr 25 13:59:09 smithi045 bash[69094]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-b 2024-04-25T13:59:09.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T13:59:09.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:09.723 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T13:59:09.723 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T13:59:09.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.e 2024-04-25T13:59:09.827 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:59:09 smithi045 systemd[1]: Stopping Ceph mds.e for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:10.327 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:59:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e[62979]: 2024-04-25T13:59:10.043+0000 7f701ffcd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:10.327 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:59:10 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e[62979]: 2024-04-25T13:59:10.043+0000 7f701ffcd700 -1 mds.e *** got signal Terminated *** 2024-04-25T13:59:14.322 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:59:14 smithi045 podman[69228]: 2024-04-25 13:59:14.055445779 +0000 UTC m=+4.108398664 container died fae0190ee89d5d4c646ac917326de0ba5a044940d935b9fe2165a0177ad433b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:59:14.577 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:59:14 smithi045 podman[69228]: 2024-04-25 13:59:14.418679273 +0000 UTC m=+4.471632156 container cleanup fae0190ee89d5d4c646ac917326de0ba5a044940d935b9fe2165a0177ad433b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T13:59:14.578 INFO:journalctl@ceph.mds.e.smithi045.stdout:Apr 25 13:59:14 smithi045 bash[69228]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-e 2024-04-25T13:59:14.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T13:59:14.889 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:14.890 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T13:59:14.890 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T13:59:14.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.h 2024-04-25T13:59:15.238 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:59:14 smithi045 systemd[1]: Stopping Ceph mds.h for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:15.577 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:59:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h[63290]: 2024-04-25T13:59:15.238+0000 7ff3e3e26700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:15.577 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:59:15 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h[63290]: 2024-04-25T13:59:15.238+0000 7ff3e3e26700 -1 mds.h *** got signal Terminated *** 2024-04-25T13:59:19.413 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:59:19 smithi045 podman[69348]: 2024-04-25 13:59:19.149872562 +0000 UTC m=+4.036445630 container died e41878edd41fb86eced882d80a5f53d0415673491ca19e75c4f7b80f3c3e690b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:59:19.789 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:59:19 smithi045 podman[69348]: 2024-04-25 13:59:19.513899404 +0000 UTC m=+4.400472469 container cleanup e41878edd41fb86eced882d80a5f53d0415673491ca19e75c4f7b80f3c3e690b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T13:59:19.789 INFO:journalctl@ceph.mds.h.smithi045.stdout:Apr 25 13:59:19 smithi045 bash[69348]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-h 2024-04-25T13:59:19.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T13:59:19.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:19.988 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T13:59:19.988 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T13:59:19.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.k 2024-04-25T13:59:20.300 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:59:20 smithi045 systemd[1]: Stopping Ceph mds.k for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:20.577 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:59:20 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k[62656]: 2024-04-25T13:59:20.300+0000 7f07984f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:20.577 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:59:20 smithi045 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k[62656]: 2024-04-25T13:59:20.300+0000 7f07984f0700 -1 mds.k *** got signal Terminated *** 2024-04-25T13:59:22.077 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:59:21 smithi045 podman[69469]: 2024-04-25 13:59:21.809666031 +0000 UTC m=+1.602079412 container died 4235b5a6ead6b3f98817da1e7dd606c8c366e064ab08b4b3a2d758926176b2a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T13:59:22.442 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:59:22 smithi045 podman[69469]: 2024-04-25 13:59:22.159262711 +0000 UTC m=+1.951676100 container cleanup 4235b5a6ead6b3f98817da1e7dd606c8c366e064ab08b4b3a2d758926176b2a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T13:59:22.442 INFO:journalctl@ceph.mds.k.smithi045.stdout:Apr 25 13:59:22 smithi045 bash[69469]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-k 2024-04-25T13:59:22.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T13:59:22.630 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:22.631 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T13:59:22.631 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T13:59:22.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.c 2024-04-25T13:59:22.934 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:59:22 smithi122 systemd[1]: Stopping Ceph mds.c for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:23.194 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:59:22 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c[63984]: 2024-04-25T13:59:22.934+0000 7f6e53880700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:23.194 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:59:22 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c[63984]: 2024-04-25T13:59:22.934+0000 7f6e53880700 -1 mds.c *** got signal Terminated *** 2024-04-25T13:59:24.408 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:59:24 smithi122 podman[70279]: 2024-04-25 13:59:24.124472497 +0000 UTC m=+1.295457604 container died 83de7562972e04926abfe00a192555ad3cfe64f09bac17d5a317b332dda3e083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-25T13:59:24.751 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:59:24 smithi122 podman[70279]: 2024-04-25 13:59:24.501042291 +0000 UTC m=+1.672027389 container cleanup 83de7562972e04926abfe00a192555ad3cfe64f09bac17d5a317b332dda3e083 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T13:59:24.751 INFO:journalctl@ceph.mds.c.smithi122.stdout:Apr 25 13:59:24 smithi122 bash[70279]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-c 2024-04-25T13:59:24.913 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T13:59:24.947 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:24.947 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T13:59:24.947 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T13:59:24.947 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.f 2024-04-25T13:59:25.279 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:59:25 smithi122 systemd[1]: Stopping Ceph mds.f for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:25.557 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:59:25 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f[63673]: 2024-04-25T13:59:25.280+0000 7f2b2d568700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:25.558 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:59:25 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f[63673]: 2024-04-25T13:59:25.280+0000 7f2b2d568700 -1 mds.f *** got signal Terminated *** 2024-04-25T13:59:29.546 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:59:29 smithi122 podman[70412]: 2024-04-25 13:59:29.29321507 +0000 UTC m=+4.119015820 container died f8e414b7cf09bd3642fe92273573d9110cce4682d81d104d2f80f427e3c8215a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T13:59:29.807 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:59:29 smithi122 podman[70412]: 2024-04-25 13:59:29.646347796 +0000 UTC m=+4.472148551 container cleanup f8e414b7cf09bd3642fe92273573d9110cce4682d81d104d2f80f427e3c8215a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T13:59:29.808 INFO:journalctl@ceph.mds.f.smithi122.stdout:Apr 25 13:59:29 smithi122 bash[70412]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-f 2024-04-25T13:59:30.077 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T13:59:30.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:30.109 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T13:59:30.109 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T13:59:30.109 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.i 2024-04-25T13:59:30.449 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:59:30 smithi122 systemd[1]: Stopping Ceph mds.i for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:30.807 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:59:30 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i[64292]: 2024-04-25T13:59:30.450+0000 7fc5b1bf9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:30.808 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:59:30 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i[64292]: 2024-04-25T13:59:30.450+0000 7fc5b1bf9700 -1 mds.i *** got signal Terminated *** 2024-04-25T13:59:33.199 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:59:32 smithi122 podman[70535]: 2024-04-25 13:59:32.918523431 +0000 UTC m=+2.596449801 container died 115a3ae05213681ad8840899f72b34901c9903e2e03d04e35d61900d9f1f98ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-25T13:59:33.558 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:59:33 smithi122 podman[70535]: 2024-04-25 13:59:33.300246488 +0000 UTC m=+2.978172846 container cleanup 115a3ae05213681ad8840899f72b34901c9903e2e03d04e35d61900d9f1f98ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T13:59:33.558 INFO:journalctl@ceph.mds.i.smithi122.stdout:Apr 25 13:59:33 smithi122 bash[70535]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-i 2024-04-25T13:59:33.724 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T13:59:33.755 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:33.756 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T13:59:33.756 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T13:59:33.756 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.l 2024-04-25T13:59:34.069 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:59:33 smithi122 systemd[1]: Stopping Ceph mds.l for db01fc44-0307-11ef-bc93-c7b262605968... 2024-04-25T13:59:34.557 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:59:34 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l[63359]: 2024-04-25T13:59:34.069+0000 7f665d05d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T13:59:34.558 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:59:34 smithi122 ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l[63359]: 2024-04-25T13:59:34.069+0000 7f665d05d700 -1 mds.l *** got signal Terminated *** 2024-04-25T13:59:38.454 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:59:38 smithi122 podman[70654]: 2024-04-25 13:59:38.085116093 +0000 UTC m=+4.117714251 container died d2d867b1f50e9a8eab65322b7ed510fc02ebd6aa429a638a261f4a7ec0823f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T13:59:38.807 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:59:38 smithi122 podman[70654]: 2024-04-25 13:59:38.537136778 +0000 UTC m=+4.569734936 container cleanup d2d867b1f50e9a8eab65322b7ed510fc02ebd6aa429a638a261f4a7ec0823f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T13:59:38.808 INFO:journalctl@ceph.mds.l.smithi122.stdout:Apr 25 13:59:38 smithi122 bash[70654]: ceph-db01fc44-0307-11ef-bc93-c7b262605968-mds-l 2024-04-25T13:59:38.969 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-db01fc44-0307-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T13:59:39.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T13:59:39.000 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T13:59:39.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid db01fc44-0307-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T13:59:39.199 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:59:41.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid db01fc44-0307-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T13:59:41.859 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:59:44.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid db01fc44-0307-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T13:59:44.552 INFO:teuthology.orchestra.run.smithi122.stdout:Deleting cluster with fsid: db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T13:59:46.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:59:46.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:59:46.932 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T13:59:46.958 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T13:59:46.960 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi027/crash 2024-04-25T13:59:46.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash -- . 2024-04-25T13:59:47.024 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T13:59:47.024 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-04-25T13:59:47.026 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi045/crash 2024-04-25T13:59:47.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash -- . 2024-04-25T13:59:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:tar: /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T13:59:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:tar: Error is not recoverable: exiting now 2024-04-25T13:59:47.063 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi122/crash 2024-04-25T13:59:47.064 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash -- . 2024-04-25T13:59:47.098 INFO:teuthology.orchestra.run.smithi122.stderr:tar: /var/lib/ceph/db01fc44-0307-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T13:59:47.098 INFO:teuthology.orchestra.run.smithi122.stderr:tar: Error is not recoverable: exiting now 2024-04-25T13:59:47.099 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T13:59:47.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T13:59:47.141 INFO:tasks.cephadm:Compressing logs... 2024-04-25T13:59:47.141 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T13:59:47.184 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T13:59:47.187 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T13:59:47.209 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T13:59:47.209 INFO:teuthology.orchestra.run.smithi027.stderr:: No such file or directory 2024-04-25T13:59:47.209 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log 2024-04-25T13:59:47.209 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T13:59:47.210 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T13:59:47.219 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.g.log: /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.x.log: 90.2% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log.gz 87.2% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.6.log: 94.7% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr: 94.6% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T13:59:47.220 INFO:teuthology.orchestra.run.smithi027.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:: No such file or directory 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.f.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T13:59:47.227 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.2% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.3% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.11.log: 94.6% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr: 94.5% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T13:59:47.228 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log: 93.2% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T13:59:47.234 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T13:59:47.234 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T13:59:47.234 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.a.log: 82.7% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T13:59:47.234 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T13:59:47.235 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T13:59:47.235 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.d.log: /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log: 94.6% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T13:59:47.235 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:: No such file or directory 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.7.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr: 90.4% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T13:59:47.236 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.h.log: 87.2% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.4% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.e.log: 94.4% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log: 93.1% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.b.log: 94.5% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T13:59:47.237 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T13:59:47.238 INFO:teuthology.orchestra.run.smithi027.stderr: 94.5% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T13:59:47.239 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T13:59:47.239 INFO:teuthology.orchestra.run.smithi122.stderr: 94.5% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T13:59:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.9.log: 91.1% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T13:59:47.545 INFO:teuthology.orchestra.run.smithi027.stderr: 90.2% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T13:59:47.684 INFO:teuthology.orchestra.run.smithi122.stderr: 94.3% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T13:59:47.786 INFO:teuthology.orchestra.run.smithi122.stderr: 91.8% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T13:59:47.803 INFO:teuthology.orchestra.run.smithi045.stderr: 91.9% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T13:59:48.697 INFO:teuthology.orchestra.run.smithi027.stderr: 90.2% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T13:59:54.304 INFO:teuthology.orchestra.run.smithi045.stderr: 94.1% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T14:00:21.304 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T14:00:25.954 INFO:teuthology.orchestra.run.smithi122.stderr: 91.1% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T14:00:27.140 INFO:teuthology.orchestra.run.smithi045.stderr: 91.5% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T14:00:31.728 INFO:teuthology.orchestra.run.smithi122.stderr: 91.1% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T14:00:31.803 INFO:teuthology.orchestra.run.smithi027.stderr: 91.0% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T14:00:32.567 INFO:teuthology.orchestra.run.smithi122.stderr: 91.1% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T14:00:33.721 INFO:teuthology.orchestra.run.smithi045.stderr: 91.0% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T14:00:43.428 INFO:teuthology.orchestra.run.smithi122.stderr: 90.9% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T14:00:43.430 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-25T14:00:43.430 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m56.229s 2024-04-25T14:00:43.430 INFO:teuthology.orchestra.run.smithi122.stderr:user 2m59.996s 2024-04-25T14:00:43.430 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m5.583s 2024-04-25T14:00:43.757 INFO:teuthology.orchestra.run.smithi027.stderr: 90.9% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T14:00:44.577 INFO:teuthology.orchestra.run.smithi045.stderr: 91.0% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T14:00:46.876 INFO:teuthology.orchestra.run.smithi027.stderr: 90.9% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T14:00:46.878 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-04-25T14:00:46.878 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m59.680s 2024-04-25T14:00:46.878 INFO:teuthology.orchestra.run.smithi027.stderr:user 3m10.935s 2024-04-25T14:00:46.878 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m5.416s 2024-04-25T14:00:52.040 INFO:teuthology.orchestra.run.smithi045.stderr: 91.0% -- replaced with /var/log/ceph/db01fc44-0307-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T14:00:52.042 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-25T14:00:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:real 1m4.841s 2024-04-25T14:00:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:user 3m30.229s 2024-04-25T14:00:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m5.732s 2024-04-25T14:00:52.042 INFO:tasks.cephadm:Archiving logs... 2024-04-25T14:00:52.043 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi027/log 2024-04-25T14:00:52.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T14:01:25.280 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi045/log 2024-04-25T14:01:25.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T14:02:01.858 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi122/log 2024-04-25T14:02:01.859 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T14:02:36.385 INFO:tasks.cephadm:Removing cluster... 2024-04-25T14:02:36.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid db01fc44-0307-11ef-bc93-c7b262605968 --force 2024-04-25T14:02:36.589 INFO:teuthology.orchestra.run.smithi027.stdout:Deleting cluster with fsid: db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T14:02:37.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid db01fc44-0307-11ef-bc93-c7b262605968 --force 2024-04-25T14:02:37.366 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T14:02:37.951 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid db01fc44-0307-11ef-bc93-c7b262605968 --force 2024-04-25T14:02:38.147 INFO:teuthology.orchestra.run.smithi122.stdout:Deleting cluster with fsid: db01fc44-0307-11ef-bc93-c7b262605968 2024-04-25T14:02:38.688 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T14:02:38.689 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T14:02:38.708 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T14:02:38.726 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T14:02:38.743 INFO:tasks.cephadm:Teardown complete 2024-04-25T14:02:38.743 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T14:02:38.755 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T14:02:38.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T14:02:38.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T14:02:38.770 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T14:02:38.887 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T14:02:38.898 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T14:02:38.898 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T14:02:38.901 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T14:02:38.903 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T14:02:38.915 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-25T14:02:38.917 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-25T14:02:38.917 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-04-25T14:02:38.921 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T14:02:38.921 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-25T14:02:38.921 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 +1011us[+1024us] +/- 59ms 2024-04-25T14:02:38.921 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 -1674us[-1674us] +/- 69ms 2024-04-25T14:02:38.921 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +596us[ +609us] +/- 75ms 2024-04-25T14:02:38.921 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T14:02:38.921 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T14:02:38.922 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-25T14:02:38.922 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 +991us[ +936us] +/- 59ms 2024-04-25T14:02:38.922 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 7 -1713us[-1768us] +/- 69ms 2024-04-25T14:02:38.922 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +584us[ +584us] +/- 75ms 2024-04-25T14:02:38.922 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T14:02:38.923 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T14:02:38.923 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-04-25T14:02:38.923 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 +1020us[+1053us] +/- 59ms 2024-04-25T14:02:38.923 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -1682us[-1682us] +/- 69ms 2024-04-25T14:02:38.924 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +602us[ +602us] +/- 75ms 2024-04-25T14:02:38.924 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T14:02:38.924 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T14:02:38.934 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T14:02:38.935 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T14:02:38.945 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T14:02:39.042 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T14:02:39.112 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T14:02:39.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T14:02:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:39.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T14:02:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:39.236 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T14:02:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:39.263 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T14:02:39.276 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T14:02:39.288 INFO:teuthology.task.internal:Duration was 2317.843077 seconds 2024-04-25T14:02:39.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T14:02:39.299 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T14:02:39.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T14:02:39.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T14:02:39.304 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T14:02:39.338 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T14:02:39.340 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T14:02:39.343 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T14:02:39.691 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T14:02:39.692 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-04-25T14:02:39.692 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T14:02:39.753 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-04-25T14:02:39.754 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T14:02:39.783 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-04-25T14:02:39.783 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T14:02:39.805 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T14:02:39.805 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T14:02:39.807 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T14:02:39.826 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T14:02:39.869 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T14:02:39.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T14:02:39.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T14:02:39.894 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T14:02:40.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T14:02:40.261 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T14:02:40.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T14:02:40.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T14:02:40.313 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T14:02:40.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T14:02:40.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:02:40.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:02:40.355 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:02:40.372 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-04-25T14:02:40.376 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-04-25T14:02:40.400 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-04-25T14:02:40.436 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:02:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:40.450 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:02:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:40.465 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:02:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:40.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T14:02:40.506 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T14:02:40.507 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi027 2024-04-25T14:02:40.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T14:02:40.548 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi045 2024-04-25T14:02:40.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T14:02:40.586 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673154/remote/smithi122 2024-04-25T14:02:40.586 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T14:02:40.621 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T14:02:40.621 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T14:02:40.624 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T14:02:40.626 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T14:02:40.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T14:02:40.684 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T14:02:40.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T14:02:40.696 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T14:02:40.696 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T14:02:40.698 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T14:02:40.704 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T14:02:40.710 INFO:teuthology.orchestra.run.smithi027.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 14:02 /home/ubuntu/cephtest 2024-04-25T14:02:40.717 INFO:teuthology.orchestra.run.smithi045.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 14:02 /home/ubuntu/cephtest 2024-04-25T14:02:40.724 INFO:teuthology.orchestra.run.smithi122.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 14:02 /home/ubuntu/cephtest 2024-04-25T14:02:40.725 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T14:02:40.737 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T14:02:40.762 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} duration: 2317.843076944351 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T14:02:40.762 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T14:02:40.843 INFO:teuthology.run:pass